CN114299252A - Universal three-dimensional model reconstruction method and device, storage medium and electronic equipment - Google Patents
Universal three-dimensional model reconstruction method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN114299252A CN114299252A CN202111651821.8A CN202111651821A CN114299252A CN 114299252 A CN114299252 A CN 114299252A CN 202111651821 A CN202111651821 A CN 202111651821A CN 114299252 A CN114299252 A CN 114299252A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- cloud data
- characteristic information
- dimensional model
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 105
- 238000003860 storage Methods 0.000 title claims abstract description 23
- 238000013528 artificial neural network Methods 0.000 claims abstract description 66
- 238000012549 training Methods 0.000 claims abstract description 28
- 230000006870 function Effects 0.000 claims description 18
- 238000004891 communication Methods 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 13
- 238000007781 pre-processing Methods 0.000 claims description 10
- 238000005070 sampling Methods 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 description 17
- 238000010606 normalization Methods 0.000 description 9
- 238000013527 convolutional neural network Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 238000013135 deep learning Methods 0.000 description 5
- 238000010801 machine learning Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000002759 z-score normalization Methods 0.000 description 2
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 1
- 241000282414 Homo sapiens Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 238000011425 standardization method Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
Abstract
The disclosure provides a universal three-dimensional model reconstruction method and device, a storage medium and electronic equipment, and relates to the technical field of three-dimensional reconstruction. The universal three-dimensional model reconstruction method comprises the following steps: acquiring first point cloud data to be reconstructed; extracting basic characteristic information and shape characteristic information of the first point cloud data; inputting the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data, wherein the dynamic neural network is obtained by taking the basic characteristic information and the shape characteristic information of the point cloud data and a symbol distance function SDF value as training data. The method uses the symbol distance function SDF value obtained in real time and combines the characteristic information to form a dynamic structure, improves the reusability of a dynamic neural network, and enables the three-dimensional reconstruction method to have universality and instantaneity.
Description
Technical Field
The present disclosure relates to the field of three-dimensional reconstruction technologies, and in particular, to a method and an apparatus for reconstructing a general three-dimensional model, a storage medium, and an electronic device.
Background
With the development of information technology, three-dimensional communication will be developed to be applied to the ground in the future. The three-dimensional reconstruction technology is a key technology in the field of three-dimensional communication, and the existing three-dimensional reconstruction technology has a series of problems of low real-time performance, low reconstruction precision and the like, and the problems become the technical difficulty of three-dimensional communication and prevent the three-dimensional communication from falling to the ground.
The existing three-dimensional reconstruction method has no universality, is complicated in reconstruction process and long in time consumption, and is difficult to meet the requirements on real-time performance in the fields of three-dimensional communication, automatic driving and the like.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The present disclosure provides a universal three-dimensional model reconstruction method and apparatus, a storage medium, and an electronic device, which at least overcome the problem that the three-dimensional reconstruction method in the related art has no versatility and real-time property to some extent.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to an aspect of the present disclosure, there is provided a general three-dimensional model reconstruction method including: acquiring first point cloud data to be reconstructed; extracting basic feature information and shape feature information of the first point cloud data; inputting the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data, wherein the dynamic neural network is obtained by taking the basic characteristic information and the shape characteristic information of the point cloud data and a Symbol Distance Function (SDF) value as training data.
In one embodiment of the present disclosure, the general three-dimensional model reconstruction method further includes: acquiring grid data under a preset three-dimensional coordinate; inputting the grid data under the preset three-dimensional coordinate and the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data under the preset three-dimensional coordinate.
In an embodiment of the present disclosure, before extracting the basic feature information and the shape feature information of the first point cloud data, the method for reconstructing a general three-dimensional model further includes: preprocessing the first point cloud data by at least one of the following steps: and (6) normalizing.
In one embodiment of the present disclosure, the general three-dimensional model reconstruction method further includes: acquiring a plurality of groups of second point cloud data serving as samples; extracting basic characteristic information of each group of second point cloud data from each group of second point cloud data; extracting shape feature information of each group of second point cloud data from the basic feature information of each group of second point cloud data; and training a neural network by taking the basic characteristic information, the shape characteristic information and the corresponding SDF values of the multiple groups of second point cloud data as sample data to obtain the dynamic neural network.
In one embodiment of the present disclosure, extracting the basic feature information of each set of second point cloud data from each set of second point cloud data includes: and extracting the basic characteristic information of each group of second point cloud data from each group of second point cloud data by using a pre-trained residual error network.
In one embodiment of the present disclosure, extracting shape feature information of each set of second point cloud data from the basic feature information of each set of second point cloud data includes: and extracting the shape characteristic information of each group of second point cloud data from the basic characteristic information of each group of second point cloud data by using a pre-trained fully-connected neural network.
In an embodiment of the present disclosure, acquiring first point cloud data to be reconstructed further includes: and acquiring first point cloud data to be reconstructed by using a sparse sampling method.
In one embodiment of the present disclosure, before acquiring the first point cloud data to be reconstructed, the method includes: acquiring the network state of the current communication network; when the network state is a first state, acquiring a first amount of first point cloud data; when the network state is a second state, acquiring a second quantity of first point cloud data; wherein the network condition in the first state is better than the network condition in the second state, and the first number is greater than the second number.
According to another aspect of the present disclosure, there is provided a general three-dimensional model reconstruction apparatus, including: the data acquisition module is used for acquiring first point cloud data to be reconstructed; the extraction characteristic module is used for extracting basic characteristic information and shape characteristic information of the first point cloud data; and the model establishing module is used for inputting the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network and outputting a three-dimensional model of the first point cloud data, wherein the dynamic neural network is obtained by taking the basic characteristic information and the shape characteristic information of the point cloud data and a symbol distance function SDF value as training data.
According to still another aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described universal three-dimensional model reconstruction method via execution of the executable instructions.
According to yet another aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the general three-dimensional model reconstruction method described above.
The embodiment of the disclosure provides a universal three-dimensional model reconstruction method and device, a storage medium and electronic equipment.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.
FIG. 1 is a flow chart illustrating a general three-dimensional model reconstruction method according to an embodiment of the present disclosure;
FIG. 2 is a flow chart illustrating an embodiment of a general three-dimensional model reconstruction method according to an embodiment of the present disclosure;
FIG. 3 is a schematic diagram illustrating a general three-dimensional model reconstruction apparatus according to an embodiment of the disclosure;
FIG. 4a shows a flow chart of a prior art three-dimensional model reconstruction method;
FIG. 4b shows a flowchart of a three-dimensional model reconstruction method in an embodiment of the disclosure;
FIG. 5a shows a flow chart of a prior art three-dimensional model reconstruction system;
FIG. 5b illustrates a flow chart of a three-dimensional model reconstruction system in an embodiment of the present disclosure;
FIG. 6 is a flow chart illustrating an exemplary embodiment of a general three-dimensional model reconstruction method according to an embodiment of the present disclosure;
FIG. 7 is a schematic diagram illustrating an embodiment of a dynamic neural network according to the present disclosure;
FIG. 8 is a flow chart illustrating an embodiment of a general three-dimensional model reconstruction method according to an embodiment of the present disclosure;
fig. 9 shows a block diagram of an electronic device in an embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
For ease of understanding, the following first explains several terms to which the disclosure relates:
point cloud data: point closed data refers to a collection of vectors in a three-dimensional coordinate system. The scan data is recorded in the form of dots, each dot containing three-dimensional coordinates, some of which may contain color information (RGB) or Intensity information (Intensity).
Grid data: data over a grid is defined, and scattered point data can be gridded using some digital model to obtain grid data.
And (3) SDF: sign distance function, a symbol distance function. Also referred to as oriented distance function (oriented distance function), determines the distance of a point to a region boundary over a limited region in space and simultaneously defines the sign of the distance: the point is positive inside the region boundary, negative outside, and 0 when located on the boundary.
And (3) rolling layers: the conditional layer. Each convolution layer in the convolutional neural network consists of a plurality of convolution units, and the parameters of each convolution unit are optimized through a back propagation algorithm. The convolution operation aims to extract different input features, the convolution layer at the first layer can only extract some low-level features such as edges, lines, angles and other levels, and more layers of networks can iteratively extract more complex features from the low-level features.
A convolutional neural network: convolitional Neural Network, CNN. The method is a feedforward neural network, and the artificial neurons of the feedforward neural network can respond to peripheral units in a part of coverage range and have excellent performance on large-scale image processing. The convolutional neural network consists of one or more convolutional layers and a top fully connected layer (corresponding to the classical neural network), and also includes associated weights and pooling layers (pooling layers). This structure enables the convolutional neural network to utilize a two-dimensional structure of the input data. Convolutional neural networks can give better results in terms of image and speech recognition than other deep learning structures. This model can also be trained using a back propagation algorithm. Compared with other deep and feedforward neural networks, the convolutional neural network needs fewer considered parameters, so that the convolutional neural network becomes an attractive deep learning structure.
Residual error network: residual Network, ResNet. The residual network is characterized by easy optimization and can improve accuracy by adding considerable depth. The inner residual block uses jump connection, and the problem of gradient disappearance caused by depth increase in a deep neural network is relieved.
Normalization: normalization. Data is generally mapped to a specified range for removing dimensions and dimension units of data of different dimensions. Common mapping ranges are [0,1] and [ -1,1], with the most common Normalization method being Min-Max Normalization (Min-Max Normalization).
And (3) standardization: normalization, normalized and normalized english translation are consistent but understood (or translated) differently depending on its use (or formula). Normalization is the processing of data according to the columns of the feature matrix. There are various methods for data normalization, such as: linear methods (e.g., extreme methods, standard deviation methods), polygonal methods (e.g., triple-polygonal methods), and curvilinear methods (e.g., semi-normal distributions). Different standardization methods have different influences on the evaluation result of the system. Among them, Z-Score Normalization (Z-Score Normalization) is most commonly used.
Machine learning: machine Learning. The method is a multi-field cross discipline and relates to a plurality of disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and the like. The special research on how a computer simulates or realizes the learning behavior of human beings so as to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve the performance of the computer. Is the core of artificial intelligence, and is the fundamental way for making computers have intelligence.
The present exemplary embodiment will be described in detail below with reference to the drawings and examples.
First, the embodiment of the present disclosure provides a general three-dimensional model reconstruction method, which can be executed by any electronic device with computing processing capability.
Fig. 1 shows a flowchart of a general three-dimensional model reconstruction method in an embodiment of the present disclosure, and as shown in fig. 1, the general three-dimensional model reconstruction method provided in the embodiment of the present disclosure includes the following steps:
s102, acquiring first point cloud data to be reconstructed;
s104, extracting basic feature information and shape feature information of the first point cloud data;
and S106, inputting the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data, wherein the dynamic neural network is obtained by taking the basic characteristic information and the shape characteristic information of the point cloud data and a Symbol Distance Function (SDF) value as training data.
The method uses the symbol distance function SDF value obtained in real time and combines the characteristic information to form a dynamic structure, improves the reusability of a dynamic neural network, does not need to train each model once, and can establish three-dimensional models with a plurality of shapes only by training once. The three-dimensional reconstruction method has universality and instantaneity.
For S102, in one embodiment, it may be: first point cloud data of an object to be reconstructed is acquired.
In one embodiment, before extracting the basic feature information and the shape feature information of the first point cloud data, the general three-dimensional model reconstruction method may further include: preprocessing the first point cloud data by at least one of the following steps: and (6) normalizing.
In one embodiment, before acquiring the first point cloud data to be reconstructed, the method may include: acquiring the network state of the current communication network; when the network state is a first state, first point cloud data of a first quantity are acquired; when the network state is a second state, acquiring a second quantity of first point cloud data; the network condition of the first state is better than that of the second state, and the first number is larger than the second number.
For example, when the network state of the communication network is good, some first point cloud data are acquired; and when the network state of the communication network is poor, less first point cloud data is acquired.
In one embodiment, acquiring the first point cloud data to be reconstructed may further include: and acquiring first point cloud data to be reconstructed by using a sparse sampling method.
For example, first point cloud data of the three-dimensional model object to be reconstructed is obtained, and the first point cloud data may be data obtained by sparse sampling or data content is incomplete (data content is missing).
For S104, in one embodiment, it may be: and extracting basic feature information of the first point cloud data, then extracting shape feature information of the basic feature information of the first point cloud data, and finally obtaining the basic feature information and the shape feature credit of the first point cloud data.
For S106, in one embodiment, it may be: inputting the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data, wherein the dynamic neural network is obtained by taking the basic characteristic information and the shape characteristic information of the point cloud data and a symbol distance function SDF value as training data.
In one embodiment, grid data under a preset three-dimensional coordinate is acquired; inputting the grid data and the basic characteristic information and the shape characteristic information of the first point cloud data under the preset three-dimensional coordinate into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data under the preset three-dimensional coordinate.
Fig. 2 is a flowchart illustrating a specific example of a general three-dimensional model reconstruction method in an embodiment of the present disclosure, and as shown in fig. 2, the general three-dimensional model reconstruction method provided in the embodiment of the present disclosure may further include the following steps:
s202, acquiring a plurality of groups of second point cloud data serving as samples;
s204, extracting basic feature information of each group of second point cloud data from each group of second point cloud data;
s206, extracting shape characteristic information of each group of second point cloud data from the basic characteristic information of each group of second point cloud data;
and S208, training the neural network by taking the basic characteristic information, the shape characteristic information and the corresponding SDF values of the multiple groups of second point cloud data as sample data to obtain the dynamic neural network.
For S202, in one embodiment, it may be: and acquiring a plurality of groups of second point cloud data as sample data for training the dynamic neural network.
For S204, in one embodiment, it may be: and respectively extracting respective basic feature information of each group of second point cloud data from each group of second point cloud data.
In one embodiment, extracting the basic feature information of each group of second point cloud data from each group of second point cloud data may include: and extracting the basic characteristic information of each group of second point cloud data from each group of second point cloud data by using a pre-trained residual error network.
For example, the residual error network is trained by a machine learning method, so as to obtain a trained residual error network. And then, extracting the basic characteristic information of each group of second point cloud data from each group of second point cloud data by using the trained residual error network.
For S206, in one embodiment, it may be: and extracting the shape characteristic information of each group of second point cloud data from the basic characteristic information of each group of second point cloud data of the plurality of groups of second point cloud data.
In one embodiment, extracting shape feature information of each set of second point cloud data from the basic feature information of each set of second point cloud data may include: and extracting the shape characteristic information of each group of second point cloud data from the basic characteristic information of each group of second point cloud data by using a pre-trained fully-connected neural network.
For example, the fully-connected neural network is trained by a machine learning method, so that the trained fully-connected neural network is obtained. And then extracting the shape characteristic information of each group of second point cloud data from the basic characteristic information of each group of second point cloud data by using the trained fully-connected neural network.
For S208, in one embodiment, it may be: and training the neural network by using a machine learning method by taking the basic characteristic information, the shape characteristic information and the corresponding SDF values of the multiple groups of second point cloud data as sample data to obtain the dynamic neural network.
Based on the same inventive concept, the embodiment of the present disclosure further provides a general three-dimensional model reconstruction device, as described in the following embodiments. Because the principle of the embodiment of the apparatus for solving the problem is similar to that of the embodiment of the method, the embodiment of the apparatus can be implemented by referring to the implementation of the embodiment of the method, and repeated details are not described again.
Fig. 3 is a schematic diagram of a general three-dimensional model reconstruction apparatus in an embodiment of the present disclosure, and as shown in fig. 3, the apparatus includes:
the data obtaining module 301 is configured to obtain first point cloud data to be reconstructed;
an extraction feature module 302, configured to extract basic feature information and shape feature information of the first point cloud data;
the model establishing module 303 is configured to input the basic feature information and the shape feature information of the first point cloud data into a pre-trained dynamic neural network, and output a three-dimensional model of the first point cloud data, where the dynamic neural network is obtained by using the basic feature information and the shape feature information of the point cloud data and a symbol distance function SDF value as training data.
In one embodiment, the model building module 303 of the universal three-dimensional model reconstruction device further comprises: acquiring grid data under a preset three-dimensional coordinate; inputting the grid data and the basic characteristic information and the shape characteristic information of the first point cloud data under the preset three-dimensional coordinate into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data under the preset three-dimensional coordinate.
In one embodiment, the universal three-dimensional model reconstruction apparatus may further include:
the preprocessing module is used for preprocessing the first point cloud data by at least one of the following steps: and (6) normalizing.
In one embodiment, the universal three-dimensional model reconstruction apparatus may further include:
the sample data acquisition module is used for acquiring a plurality of groups of second point cloud data serving as samples;
the basic feature extraction module is used for extracting basic feature information of each group of second point cloud data from each group of second point cloud data;
the extraction shape feature module is used for extracting the information of the shape features of each group of second point cloud data from the basic feature information of each group of second point cloud data;
and the training neural network module is used for training the neural network by taking the basic characteristic information, the shape characteristic information and the corresponding SDF values of the multiple groups of second point cloud data as sample data to obtain the dynamic neural network.
In one embodiment, the module for extracting basic features of the universal three-dimensional model reconstruction device may further include: and extracting the basic characteristic signature of each group of second point cloud data from each group of second point cloud data by using a pre-trained residual error network.
In one embodiment, the shape feature extracting module of the universal three-dimensional model reconstruction apparatus may further include: and extracting the shape characteristic information of each group of second point cloud data from the basic characteristic information of each group of second point cloud data by using a pre-trained fully-connected neural network.
In one embodiment, the data acquiring module 301 of the universal three-dimensional model reconstruction device further includes: and acquiring first point cloud data to be reconstructed by using a sparse sampling method.
Fig. 4a shows a flowchart of a three-dimensional model reconstruction method in the prior art, and as shown in fig. 4a, the flowchart of the three-dimensional model reconstruction method in the prior art includes the following steps:
s41: inputting a key frame;
s42: point cloud data preprocessing;
s43: network training learns continuous SDF functions;
s44: reconstructing according to the network;
s45: and outputting the three-dimensional fine model.
The method is different from the three-dimensional model reconstruction method in the prior art. Fig. 4b shows a flowchart of a three-dimensional model reconstruction method in the embodiment of the present disclosure, and as shown in fig. 4b, a flowchart of a three-dimensional model reconstruction method in the prior art includes the following steps:
s402: inputting a key frame;
s404: point cloud data preprocessing;
s406: extracting basic features of the point cloud data;
s408: extracting shape features;
s410: acquiring Grid data;
s412: three-dimensional reconstruction;
s414: and outputting the three-dimensional fine model.
S406 is included in the dynamic neural network, and the basic features of the input preprocessed three-dimensional point cloud data are extracted, and high-dimensional semantic information is mainly extracted from the point cloud data.
S408, the shape features of the point cloud data are extracted, the shape information of the three-dimensional object in a higher dimension can be extracted, and the dynamic neural network can uniquely identify the three-dimensional target according to the shape features and is used for guiding the subsequent three-dimensional reconstruction.
And S412, performing three-dimensional reconstruction, including inputting basic characteristics and shape characteristics of point cloud data in the dynamic neural network, and performing three-dimensional reconstruction on the point cloud data by combining Grid data generated in a three-dimensional coordinate system.
According to the method, a dynamic neural network is constructed and trained through point cloud data basic feature extraction, shape feature extraction and three-dimensional reconstruction. The trained dynamic neural network can be used for reconstructing various types of three-dimensional objects and can realize reconstruction with any precision. This accuracy is determined by the Grid mesh data.
When the three-dimensional model reconstruction method is used, the result is directly inferred, additional training is not needed, the whole inference process hardly consumes time, and the application with high real-time requirements can be met.
Fig. 5a shows a flow chart of a three-dimensional model reconstruction system in the prior art, and as shown in fig. 5a, the flow chart of the three-dimensional model reconstruction system in the prior art includes the following steps:
s51: inputting a key frame 1;
s52: sparse sampling flag SDF;
s53: inputting a network for training;
s54: reconstructing the model according to the network;
s55: model 1 is output.
A network model needs to be trained for each key frame, and the method is complex in calculation and poor in real-time performance.
Fig. 5b shows a flowchart of a three-dimensional model reconstruction system in an embodiment of the present disclosure, and as shown in fig. 5b, a flow of a three-dimensional model reconstruction system in the prior art includes the following steps:
s502: inputting a key frame;
s504: sparse sampling flag SDF;
s506: a dynamic network reconstruction model;
s508: and outputting the model.
The method and the system solve the reconstruction of all shapes by using one dynamic network, and have high reasoning speed and high real-time property.
Fig. 6 is a flowchart illustrating a specific example of a general three-dimensional model reconstruction method according to an embodiment of the present disclosure, and as shown in fig. 6, the flowchart illustrating a specific example of a general three-dimensional model reconstruction method includes the following steps:
s612: constructing a data set, constructing a training set in a training stage, acquiring a large amount of different three-dimensional point cloud data and corresponding SDF values to form training data;
s614: extracting basic characteristics of point cloud data, wherein the basic characteristics of the point cloud data need to be extracted after the point cloud data is input, and the basic characteristic information of the point cloud data can influence the quality of three-dimensional reconstruction of the point cloud data; in this embodiment, the preprocessed point cloud data is input into a basic feature extraction module for constructing the point cloud data by using ResNet, so as to obtain basic features of the point cloud data;
s616: the shape features of the point cloud data are extracted, and in the three-dimensional reconstruction process, the shape features can help a network to uniquely identify the object in the point cloud data and can be used for guiding the subsequent three-dimensional reconstruction, so that the extraction of the shape features of the three-dimensional object is a key part in the reconstruction process. In the system, the extracted basic characteristics of the point cloud data are input into a shape characteristic module for secondary extraction to obtain the shape characteristics of the point cloud data, and the module is realized by using a fully-connected neural network in the embodiment;
s618: and training a network according to the features and the marked SDF function values, and training the network by using the SDF function as a label according to the basic features of the point cloud data and the shape features of the point cloud data obtained in the two steps (S614 and S616). The finally obtained network can identify and model various three-dimensional shapes;
s622: acquiring point cloud data, and acquiring input point cloud data which can be sparsely sampled or incomplete;
s624: preprocessing point cloud data, namely performing preprocessing such as standardization, normalization and the like on the point cloud data;
s626: and generating a three-dimensional reconstruction model by using Grid mesh data and the trained dynamic neural network, and inputting the preprocessed point cloud data and the Grid mesh data generated under the three-dimensional coordinate system into the trained dynamic neural network to generate the three-dimensional reconstruction model of the original point cloud data.
Wherein, S612, S614, S616 and S618 are training phases; s622, S624, S626 are inference phases.
Fig. 7 is a schematic structural diagram of a specific example of the dynamic neural network in the embodiment of the present disclosure, and as shown in fig. 7, the structure includes:
convs: a convolution layer; 1x1 conv: 1x1 convolution; instructor: as a parameter of 3 1x1 conv; grid grids; feature extraction: the output can be modulated into parameters to guide the subsequent three-dimensional reconstruction process by utilizing the meta-learning thought, a higher-order learning process is essentially modeled, and compared with the traditional deep learning, the method learns to select more suitable functions for fitting different input data, so that the method has the advantages of shape recognition of a three-dimensional model and guiding the reconstruction process.
The Convs and the Instructor construct a dynamic shape feature extraction module, and the physical significance of the dynamic shape feature extraction module lies in that various shape information can be extracted, and the shape code is represented in a parameterization mode and used for guiding the subsequent model reconstruction process. Compared with the existing three-dimensional reconstruction method based on deep learning, the method is applicable to various three-dimensional shapes, has relatively simple process, and can meet the application field with requirements on real-time performance.
Fig. 8 is a flowchart illustrating a specific example of a general three-dimensional model reconstruction method according to an embodiment of the present disclosure, and as shown in fig. 8, the flowchart illustrating a specific example of a general three-dimensional model reconstruction method includes the following steps:
and S802, acquiring point cloud data (equivalent to the first point cloud data) with sparse input and sampling spatial three-dimensional points in Grid grids.
And S804, inputting the point cloud data into a D module to extract shape features and perform parametric representation.
S806, the Grid sample data and the output of D are input to the neural network W (corresponding to the dynamic neural network) to obtain the prediction of the SDF value.
And S808, synthesizing the final three-dimensional model by the Grid and the predicted SDF value.
In one example, input x, x is dynamically branched by D to obtain D (x), D (x) is used as parameter W of the intermediate network, and the parameter W is operated with Grid sample data to obtain final SDF output. And when grid gives one coordinate x, y and z, the trained dynamic neural network can output one SDF value. Therefore, the final point cloud data (i.e., the three-dimensional model) is obtained as long as the SDF value is extracted and displayed to be equal to or less than 0 (points in and on the body surface of the object).
The method and the device improve the real-time three-dimensional reconstruction speed and precision of the three-dimensional communication scene. The core technology is to use a deep learning method to extract basic features of point cloud data, then extract shape features of the point cloud data, and guide the three-dimensional reconstruction process of the point cloud data by using the extracted basic features and shape feature information. The system can achieve the effect of fast reasoning and reconstruction of the three-dimensional object with any shape without repeated training.
As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
An electronic device 900 according to this embodiment of the disclosure is described below with reference to fig. 9. The electronic device 900 shown in fig. 9 is only an example and should not bring any limitations to the functionality or scope of use of the embodiments of the present disclosure.
As shown in fig. 9, the electronic device 900 is embodied in the form of a general purpose computing device. Components of electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one memory unit 920, and a bus 930 that couples various system components including the memory unit 920 and the processing unit 910.
Wherein the storage unit stores program code that is executable by the processing unit 910 to cause the processing unit 910 to perform steps according to various exemplary embodiments of the present disclosure described in the above section "exemplary method" of the present specification.
For example, the processing unit 910 may perform the following steps of the above method embodiments: acquiring first point cloud data to be reconstructed; extracting basic characteristic information and shape characteristic information of the first point cloud data; inputting the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data, wherein the dynamic neural network is obtained by taking the basic characteristic information and the shape characteristic information of the point cloud data and a symbol distance function SDF value as training data.
For example, the processing unit 910 may perform the following steps of the above method embodiment: acquiring grid data under a preset three-dimensional coordinate; inputting the grid data and the basic characteristic information and the shape characteristic information of the first point cloud data under the preset three-dimensional coordinate into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data under the preset three-dimensional coordinate.
For example, the processing unit 910 may perform the following steps of the above method embodiment: before extracting the basic feature information and the shape feature information of the first point cloud data, performing at least one of the following preprocessing on the first point cloud data: and (6) normalizing.
For example, the processing unit 910 may perform the following steps of the above method embodiment: acquiring a plurality of groups of second point cloud data serving as samples; extracting basic characteristic information of each group of second point cloud data from each group of second point cloud data; extracting shape feature information of each group of second point cloud data from the basic feature information of each group of second point cloud data; and training the neural network by taking the basic characteristic information, the shape characteristic information and the corresponding SDF values of the multiple groups of second point cloud data as sample data to obtain the dynamic neural network.
For example, the processing unit 910 may perform the following steps of the above method embodiment: and extracting the basic characteristic information of each group of second point cloud data from each group of second point cloud data by using a pre-trained residual error network.
For example, the processing unit 910 may perform the following steps of the above method embodiment: and extracting the shape characteristic information of each group of second point cloud data from the basic characteristic information of each group of second point cloud data by using a pre-trained fully-connected neural network.
For example, the processing unit 910 may perform the following steps of the above method embodiment: and acquiring first point cloud data to be reconstructed by using a sparse sampling method.
For example, the processing unit 910 may perform the following steps of the above method embodiment: acquiring the network state of the current communication network; when the network state is a first state, first point cloud data of a first quantity are acquired; when the network state is a second state, acquiring a second quantity of first point cloud data; the network condition of the first state is better than that of the second state, and the first number is larger than the second number.
The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM)9201 and/or a cache memory unit 9202, and may further include a read only memory unit (ROM) 9203.
The electronic device 900 may also communicate with one or more external devices 940 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 900, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 900 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interface 950. Also, the electronic device 900 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet) via the network adapter 960. As shown, the network adapter 960 communicates with the other modules of the electronic device 900 via the bus 930. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium, which may be a readable signal medium or a readable storage medium. On which a program product capable of implementing the above-described method of the present disclosure is stored. In some possible embodiments, various aspects of the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the disclosure described in the "exemplary methods" section above of this specification, when the program product is run on the terminal device.
More specific examples of the computer-readable storage medium in the present disclosure may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
In the present disclosure, a computer readable storage medium may include a propagated data signal with readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Alternatively, program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
In particular implementations, program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Moreover, although the steps of the methods of the present disclosure are depicted in the drawings in a particular order, this does not require or imply that the steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
Claims (11)
1. A method for reconstructing a generic three-dimensional model, comprising:
acquiring first point cloud data to be reconstructed;
extracting basic feature information and shape feature information of the first point cloud data;
inputting the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data, wherein the dynamic neural network is obtained by taking the basic characteristic information and the shape characteristic information of the point cloud data and a Symbol Distance Function (SDF) value as training data.
2. The method for universal three-dimensional model reconstruction according to claim 1, further comprising:
acquiring grid data under a preset three-dimensional coordinate;
inputting the grid data under the preset three-dimensional coordinate and the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network, and outputting a three-dimensional model of the first point cloud data under the preset three-dimensional coordinate.
3. The method of universal three-dimensional model reconstruction according to claim 1, wherein prior to extracting the basic feature information and the shape feature information of the first point cloud data, the method further comprises:
preprocessing the first point cloud data by at least one of the following steps: and (6) normalizing.
4. The method for universal three-dimensional model reconstruction according to claim 1, further comprising:
acquiring a plurality of groups of second point cloud data serving as samples;
extracting basic characteristic information of each group of second point cloud data from each group of second point cloud data;
extracting shape feature information of each group of second point cloud data from the basic feature information of each group of second point cloud data;
and training a neural network by taking the basic characteristic information, the shape characteristic information and the corresponding SDF values of the multiple groups of second point cloud data as sample data to obtain the dynamic neural network.
5. The universal three-dimensional model reconstruction method according to claim 4, wherein extracting the basic feature information of each group of second point cloud data from each group of second point cloud data comprises:
and extracting the basic characteristic information of each group of second point cloud data from each group of second point cloud data by using a pre-trained residual error network.
6. The general three-dimensional model reconstruction method according to claim 4, wherein extracting shape feature information of each group of second point cloud data from the basic feature information of each group of second point cloud data comprises:
and extracting the shape characteristic information of each group of second point cloud data from the basic characteristic information of each group of second point cloud data by using a pre-trained fully-connected neural network.
7. The universal three-dimensional model reconstruction method according to claim 1, wherein obtaining first point cloud data to be reconstructed further comprises:
and acquiring first point cloud data to be reconstructed by using a sparse sampling method.
8. The universal three-dimensional model reconstruction method according to claim 1, comprising, before acquiring the first point cloud data to be reconstructed:
acquiring the network state of the current communication network;
when the network state is a first state, acquiring a first amount of first point cloud data;
when the network state is a second state, acquiring a second quantity of first point cloud data; wherein the network condition in the first state is better than the network condition in the second state, and the first number is greater than the second number.
9. A universal three-dimensional model reconstruction apparatus, comprising:
the data acquisition module is used for acquiring first point cloud data to be reconstructed;
the extraction characteristic module is used for extracting basic characteristic information and shape characteristic information of the first point cloud data;
and the model establishing module is used for inputting the basic characteristic information and the shape characteristic information of the first point cloud data into a pre-trained dynamic neural network and outputting a three-dimensional model of the first point cloud data, wherein the dynamic neural network is obtained by taking the basic characteristic information and the shape characteristic information of the point cloud data and a symbol distance function SDF value as training data.
10. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the method of universal three-dimensional model reconstruction according to any one of claims 1-8 via execution of the executable instructions.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method for universal three-dimensional model reconstruction according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111651821.8A CN114299252B (en) | 2021-12-30 | 2021-12-30 | Method and device for reconstructing universal three-dimensional model, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111651821.8A CN114299252B (en) | 2021-12-30 | 2021-12-30 | Method and device for reconstructing universal three-dimensional model, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114299252A true CN114299252A (en) | 2022-04-08 |
| CN114299252B CN114299252B (en) | 2024-04-02 |
Family
ID=80973622
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111651821.8A Active CN114299252B (en) | 2021-12-30 | 2021-12-30 | Method and device for reconstructing universal three-dimensional model, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114299252B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116797726A (en) * | 2023-05-20 | 2023-09-22 | 北京大学 | Organ three-dimensional reconstruction method, device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210074012A1 (en) * | 2018-02-23 | 2021-03-11 | Tsinghua University | Dynamic scene three-dimensional reconstruction method, apparatus and system, server, and medium |
| CN112891945A (en) * | 2021-03-26 | 2021-06-04 | 腾讯科技（深圳）有限公司 | Data processing method and device, electronic equipment and storage medium |
-
2021
- 2021-12-30 CN CN202111651821.8A patent/CN114299252B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210074012A1 (en) * | 2018-02-23 | 2021-03-11 | Tsinghua University | Dynamic scene three-dimensional reconstruction method, apparatus and system, server, and medium |
| CN112891945A (en) * | 2021-03-26 | 2021-06-04 | 腾讯科技（深圳）有限公司 | Data processing method and device, electronic equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116797726A (en) * | 2023-05-20 | 2023-09-22 | 北京大学 | Organ three-dimensional reconstruction method, device, electronic equipment and storage medium |
| CN116797726B (en) * | 2023-05-20 | 2024-05-07 | 北京大学 | Organ three-dimensional reconstruction method, device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114299252B (en) | 2024-04-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111444340B (en) | Text classification method, device, equipment and storage medium | |
| CN111897941B (en) | Dialogue generation method, network training method, device, storage medium and equipment | |
| CN111915540B (en) | Rubbing oracle character image augmentation method, rubbing oracle character image augmentation system, computer equipment and medium | |
| CN111260740A (en) | Text-to-image generation method based on generation countermeasure network | |
| WO2023160472A1 (en) | Model training method and related device | |
| CN110781413B (en) | Method and device for determining interest points, storage medium and electronic equipment | |
| CN112101045B (en) | Multi-mode semantic integrity recognition method and device and electronic equipment | |
| CN113761153A (en) | Question and answer processing method and device based on picture, readable medium and electronic equipment | |
| CN110874591B (en) | Image positioning method, device, equipment and storage medium | |
| CN111597815A (en) | Multi-embedded named entity identification method, device, equipment and storage medium | |
| CN110991195A (en) | Machine translation model training method, device and storage medium | |
| CN111597341A (en) | Document level relation extraction method, device, equipment and storage medium | |
| CN111858878B (en) | Method, system and storage medium for automatically extracting answer from natural language text | |
| CN114299252B (en) | Method and device for reconstructing universal three-dimensional model, storage medium and electronic equipment | |
| CN113392197A (en) | Question-answer reasoning method and device, storage medium and electronic equipment | |
| CN111653270A (en) | Voice processing method and device, computer readable storage medium and electronic equipment | |
| CN114925320B (en) | Data processing method and related device | |
| Wang et al. | A tooth surface design method combining semantic guidance, confidence, and structural coherence | |
| CN117094365A (en) | Training method and device for image-text generation model, electronic equipment and medium | |
| CN115712739B (en) | Dance motion generation method, computer device and storage medium | |
| CN116957921A (en) | Image rendering method, device, equipment and storage medium | |
| CN112749797B (en) | Pruning method and device for neural network model | |
| CN112132269B (en) | Model processing method, device, equipment and storage medium | |
| CN111062477B (en) | Data processing method, device and storage medium | |
| CN113762061A (en) | Quantitative perception training method and device for neural network and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |