CN114255328A - Three-dimensional reconstruction method for ancient cultural relics based on single view and deep learning - Google Patents
Three-dimensional reconstruction method for ancient cultural relics based on single view and deep learning Download PDFInfo
- Publication number
- CN114255328A CN114255328A CN202111510170.0A CN202111510170A CN114255328A CN 114255328 A CN114255328 A CN 114255328A CN 202111510170 A CN202111510170 A CN 202111510170A CN 114255328 A CN114255328 A CN 114255328A
- Authority
- CN
- China
- Prior art keywords
- convolution
- dimensional
- network
- ancient cultural
- cultural relic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000013135 deep learning Methods 0.000 title claims abstract description 14
- 238000000605 extraction Methods 0.000 claims abstract description 21
- 238000005065 mining Methods 0.000 claims abstract description 10
- 238000012549 training Methods 0.000 claims abstract description 10
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 9
- 230000003044 adaptive effect Effects 0.000 claims description 4
- 238000004088 simulation Methods 0.000 claims description 3
- 230000006872 improvement Effects 0.000 claims description 2
- 230000000717 retained effect Effects 0.000 claims 1
- 238000005457 optimization Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000008901 benefit Effects 0.000 description 5
- 241000282414 Homo sapiens Species 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000012468 concentrated sample Substances 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 238000005034 decoration Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000007480 spreading Effects 0.000 description 1
- 238000003892 spreading Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Geometry (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computer Graphics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Medical Informatics (AREA)
- Computer Hardware Design (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention relates to the technical field of three-dimensional reconstruction of ancient cultural relic pictures, and discloses a three-dimensional reconstruction method of ancient cultural relics based on single view and deep learning, which comprises the following steps: step 1: inputting a data set of ancient cultural relics and artworks similar to the ancient cultural relics; step 2: according to a network structure of an encoder-decoder, an encoder with a 3D-ResVGG network and a multipath channel attention module is used for carrying out deep information mining and feature extraction on the data set, and an ancient cultural relic three-dimensional network model is generated; and step 3: preprocessing an ancient cultural relic picture needing three-dimensional reconstruction through an intelligent AI tool to generate a preset type picture; and 4, step 4: and inputting the preset type picture as a single view into the ancient cultural relic three-dimensional network model to generate a complete ancient cultural relic three-dimensional model of the ancient cultural relic picture needing three-dimensional reconstruction. The invention provides a novel 3D-ResVGG network, which improves the over-fitting problem of the original VGG framework through multi-branch model training.
Description
Technical Field
The invention relates to the technical field of three-dimensional reconstruction of ancient cultural relic pictures, in particular to a three-dimensional reconstruction method of ancient cultural relics based on single view and deep learning.
Background
The ancient cultural relics are relics and traces which are left in social activities by human beings and have historical, artistic and scientific values, and are valuable historical cultural heritages of the human beings. Many times, people only observe and understand ancient cultural relics through pictures, and thus the ancient cultural relics are not deeply understood. Because we live in a three-dimensional world, but what the human eye sees is a two-dimensional projection of an object, and the form of the object captured by a camera or the like is usually a two-dimensional image, image-based three-dimensional reconstruction is a core problem in many fields such as computer vision, computer animation, industrial manufacturing, and the like. In the real world, all objects exist in three dimensions and have characteristics such as shape, appearance, and texture. For the engineering surveying and mapping technology, the three-dimensional reconstruction can help a remote sensing system to acquire required earth surface space information, and the real-time performance is realized; in the field of human-computer interaction, a three-dimensional reconstruction technology is a foundation for laying a high-level human-computer interface, and an immersive and interactive virtual reality technology is really realized; in the field of building informatization, the method is an important tool for cultural relic building protection, engineering quality detection and management, building removal management and building transformation or decoration.
Meanwhile, the cultural relic data is used as a carrier for spreading, storing and displaying historical culture and social landscape to the public, and the protection work of the cultural relic data is gradually valued by the public. The three-dimensional reconstruction technology is developed to the present day and gradually becomes the most leading application technology in the aspect of historical relic data preservation. The method can store the geometric information of the real cultural relics for a long time, and provides great convenience for promoting the application research and the deep data analysis of the cultural relics.
In addition, in the technical field of visualization, the digitization of entity cultural relic information and the visualization technology in the field of computers are both applied in a larger scale, and the related industries of cultural education have more opportunities to exert the social education functions of the cultural relics so as to further expand and meet the appreciation requirements of the masses.
The traditional method in three-dimensional reconstruction is mature in the theory of stereoscopic vision, and the stereoscopic vision is a scientific method for calculating parallax and further deriving depth by simulating the geometric rule of human eyes to watch objects through a camera based on the principle of pinhole imaging. In recent years, with the popularization of smart phones and the pursuit of mobile phone manufacturers for lens blurring, binocular vision-based three-dimensional reconstruction is more and more mature, a disparity map is calculated through double photographing of a camera, a depth map is calculated through the disparity map, and then a three-dimensional scene is restored. While the development of three-dimensional reconstruction technology, a large number of stereoscopic vision devices such as stereo cameras, Kinect and the like are produced. However, despite the emergence of a large number of stereoscopic devices, most ancient cultural relic picture materials which are easily acquired in a network environment are still mainly monocular single-view images, are not binocular images containing depth RGBD or given camera parameters, and single-view pictures lacking associated information are difficult to reconstruct through a stereoscopic vision theory, so that a new computer vision problem is caused: three-dimensional reconstruction of single views.
The task of single-view three-dimensional reconstruction is a very difficult computer vision task. The traditional method for extracting features mainly comprises the steps of constructing manual features, extracting textures and contours of images by using operators, calculating optical flows, extracting motion information, and judging a light-dark relation through gray levels, wherein the manual features are commonly used for extracting the manual features, the reconstruction of a three-dimensional model is completed by using the features and auxiliary assumed conditions, the characteristics are limited by information content contained in the features, and the effect of the traditional method is not ideal.
Disclosure of Invention
Aiming at the problems of the lack of ancient cultural relic view pictures and the lack of three-dimensional model data sets at present, on the basis of the existing research and the realization of the ancient cultural relic three-dimensional model data sets, a three-dimensional ancient cultural relic reconstruction method based on single view and deep learning is provided, a model can be generated by inputting a single view picture after the training of an ancient cultural relic three-dimensional network model is completed, so that the mapping from a two-dimensional image to a three-dimensional model is realized, the three-dimensional reconstruction is carried out under the condition of a single view picture, in addition, the algorithm does not need image annotation or classification labels for training, and the problems of lack of textures, wide baseline characteristic matching and the like which cannot be solved in the past are overcome.
The invention is realized by the following technical scheme:
a three-dimensional ancient cultural relic reconstruction method based on single view and deep learning comprises the following steps:
step 1: inputting a data set of an ancient cultural relic and a handicraft similar to the ancient cultural relic, wherein the data set comprises a multi-angle multi-view picture of the ancient cultural relic and the handicraft similar to the ancient cultural relic and a voxel model for training;
step 2: according to a network structure of an encoder-decoder, an encoder with a 3D-ResVGG network and a multipath channel attention module is used for carrying out deep information mining and feature extraction on the data set, and an ancient cultural relic three-dimensional network model is generated;
and step 3: preprocessing an ancient cultural relic picture needing three-dimensional reconstruction through an intelligent AI tool to generate a preset type picture;
and 4, step 4: and inputting the preset type picture as a single view into the ancient cultural relic three-dimensional network model to generate a complete ancient cultural relic three-dimensional model of the ancient cultural relic picture needing three-dimensional reconstruction.
As an optimization, in step 2, the ancient cultural relic three-dimensional network model comprises an encoder, an iterative convolution module and a decoder, wherein the encoder is composed of a residual network structure, a multi-path channel attention module and a 3D-ResVGG network, the iterative convolution module is composed of a group of iterative convolution units, the iterative convolution units are distributed in a three-dimensional grid structure in space, and each iterative convolution unit is responsible for reconstructing a final output voxel probability result.
As an optimization, the 3D-ResVGG network adds two groups of 1 × 1 convolutions to the forward propagation part of the ResNet network, and is obtained by combining a ResNet residual module and a VGG network architecture.
As an optimization, the multipath channel attention module functions as: and performing grouping convolution in the process of performing feature extraction on the 3D-ResVGG network by using the multipath channel attention module, and performing deep mining on data information in the form of multipath multi-group convolution kernels to obtain the depth information of the data.
As optimization, the encoder in step 2 specifically performs depth information mining and feature extraction on the data set, and the specific steps are as follows:
2.1, the 3D-ResVGG network receives a data set and carries out feature extraction on the data set to generate a feature map;
step 2.2, the multipath channel attention module performs grouping extraction on the feature map generated by the 3D-ResVGG network: dividing the characteristic graph into a plurality of groups according to the characteristics of the characteristic graph and convolution kernels, and selecting a plurality of groups of adaptive convolution kernels to perform grouping convolution on the characteristic graph;
2.3, performing a group of 1 × 1 convolution on each group of the feature map simulation residual error network structure after convolution to obtain a plurality of corresponding groups of feature maps;
and 2.4, splicing and combining the plurality of groups of feature maps obtained in the step 2.3 to obtain a feature map with high scale, high dimensionality and richer information content.
As an optimization, in step 2.2, 3 groups of information convolution kernels are selected to perform grouping convolution on the feature map.
And as optimization, the iterative convolution unit is composed of a long-short convolution network, the long-short convolution network selects specific convolution unit quantity according to specific format of the input feature diagram, the feature diagram extracted by the encoder is subjected to iterative convolution from a three-dimensional angle, the convolution voxel probability result of each convolution in the feature diagram is obtained in a classification network mode, and a group of three-dimensional probability information is collected and reserved.
As an optimization, in step 2, the specific process of performing convolution operation on the input multiple groups of feature maps by the iterative convolution unit is as follows: and the iterative convolution unit performs iterative convolution operation on a plurality of groups of multi-angle characteristic images to circularly extract the information of the characteristic images, dynamically updates the voxel probability related to the iterative convolution unit, and finally combines the voxel probability data of all the iterative convolution units to synthesize a group of three-dimensional voxel probability models.
As an optimization, in the iterative convolution module, the specific structure of the iterative convolution unit is determined by the resolution of the feature map, and the iterative convolution unit of X is used for processing data of the feature map with X resolution.
And as optimization, verifying whether the ancient cultural relic three-dimensional network model is reliable or not through a loss function, wherein the loss function is a characteristic regularization softmax function after a cross entropy function is improved.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the method can generate a corresponding three-dimensional voxel model by inputting a single ancient cultural relic picture, finish the training operation of the ancient cultural relic three-dimensional network model by a plurality of two-dimensional images, and then perform reconstruction operation on the two-dimensional image to be detected by utilizing the ancient cultural relic three-dimensional network model.
2. The invention provides a new 3D-ResVGG network aiming at the problems of low video memory occupation and low flexibility in the traditional ResNet network and the over-fitting problem in the traditional VGG network under the condition of a deeper network layer number, wherein the network improves a 3 x 3 residual operator in the ResNet into a form of combining a 3 x 3 convolution with a 1 x 1 convolution branch through multi-branch model training on the basis of the traditional ResNet and VGG network architecture, and adds a Relu form after each group of large branches, thereby perfecting the over-fitting problem of the traditional VGG network.
3. The invention improves the problems that the traditional encoder network cannot carry out deep extraction on information, has lower training speed and the like, and improves the precision and the depth of the integral three-dimensional reconstruction.
Drawings
In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and that for those skilled in the art, other related drawings can be obtained from these drawings without inventive effort. In the drawings:
FIG. 1 is a schematic structural diagram of an ancient cultural relic three-dimensional model of an ancient cultural relic three-dimensional reconstruction method based on a single view and deep learning according to the invention;
FIG. 2 is a flow chart of a three-dimensional ancient cultural relic reconstruction method based on single view and deep learning according to the invention;
FIG. 3 is a flow chart of an encoder in a three-dimensional ancient cultural relic reconstruction method based on single view and deep learning according to the present invention;
FIG. 4 is a block diagram of a multi-path channel attention module in accordance with the present invention;
fig. 5 is a structure diagram of a 3D-ResVGG network in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to examples and accompanying drawings, and the exemplary embodiments and descriptions thereof are only used for explaining the present invention and are not meant to limit the present invention.
Example 1
A three-dimensional ancient cultural relic reconstruction method based on single view and deep learning comprises the following steps:
step 1: inputting ancient cultural relics and handicraft similar to the ancient cultural relics (how to judge whether the ancient cultural relics are similar; the data set herein may use existing ShapeNetVox data sets, including voxel models similar to ancient cultural relics, such as china, lamps, bowls, tables and chairs.
Step 2: according to a network structure of an encoder-decoder, an encoder with a 3D-ResVGG network and a multipath channel attention module is used for carrying out deep information mining and feature extraction on the data set, and an ancient cultural relic three-dimensional network model is generated;
and step 3: preprocessing an ancient cultural relic picture needing three-dimensional reconstruction through an intelligent AI tool to generate a preset type picture; the intelligent AI tool herein may include but is not limited to PhotoScissor, which uses a photo processing tool such as PhotoScissor to perform preprocessing such as dividing, clipping, etc. on a picture.
And 4, step 4: and inputting the preset type picture as a single view into the ancient cultural relic three-dimensional network model to generate a complete ancient cultural relic three-dimensional model of the ancient cultural relic picture needing three-dimensional reconstruction.
In this embodiment, in step 2, the ancient cultural relic three-dimensional network model includes an encoder, an iterative convolution module and a decoder. The encoder is composed of a residual error network structure, a multi-path channel attention module and a 3D-ResVGG network, wherein the multi-path channel attention module is an efficient multi-scale channel attention mechanism network, the input of the encoder is a two-dimensional image of a single view, the output of the encoder is a two-dimensional feature vector, and the two-dimensional feature vector needs to be converted into three-dimensional information;
the iterative convolution module uses a set of iterative convolution units, each iterative convolution unit is composed of a set of long and short convolution networks, namely continuous long and short time and LRU networks, the iterative convolution units are distributed in a three-dimensional grid structure in space, each iterative convolution unit is responsible for reconstructing a voxel probability module which is finally output, specifically, the iterative convolution units select specific convolution unit number according to the specific format of a network input picture, for example, the specific convolution unit number is determined according to the spatial resolution of the picture, if the spatial resolution is 32, 32 × 32 convolution units are provided, the feature map extracted in the above mode is subjected to iterative convolution from the three-dimensional angle, the voxel probability result of each convolution in the feature map is obtained in the form of a classification network, and a set of three-dimensional probability information is collected and reserved, wherein the specific format of the feature map is determined according to the size of the picture, the method is equivalent to cutting a picture into a plurality of small blocks, wherein each small block represents the voxel probability of the position of the small block; the input of the decoder is three-dimensional information obtained by processing of the iterative convolution module, and the output of the decoder is three-dimensional prediction voxel occupation of a single image.
As shown in fig. 5, in this embodiment, the 3D-ResVGG network is obtained by adding two groups of 1 × 1 convolutions to the forward propagation portion of the ResNet network and combining the ResNet residual module and the VGG network architecture. The 3D-ResVGG network realizes high-precision feature extraction on the basis of only using 3 x 3 convolution and 1 x 1 convolution by combining the advantages of a ResNet residual module and a VGG network architecture.
In this embodiment, the multi-path channel attention module functions as: and performing grouping convolution in the process of performing feature extraction on the 3D-ResVGG network by using the multi-path channel attention module, activating by an activation function Relu after the convolution, performing deep mining on data information in the form of multi-path multi-group convolution kernels, and acquiring the depth information of the data.
Specifically, as shown in fig. 3 to 4, in this embodiment, the specific steps of the encoder in step 2 for performing depth information mining and feature extraction on the data set are as follows:
2.1, the 3D-ResVGG network receives a data set and carries out feature extraction on the data set to generate a feature map;
step 2.2, the multipath channel attention module performs grouping extraction on the feature map generated by the 3D-ResVGG network: dividing the characteristic graph into a plurality of groups according to the characteristics of the characteristic graph and convolution kernels, and selecting a plurality of groups of adaptive convolution kernels to perform grouping convolution on the characteristic graph, so that the parameter quantity can be reduced and the extraction precision of the characteristic graph can be improved; the feature map is specifically determined to be divided into three groups or four groups according to the number of the feature maps, the convolution kernels are selected according to actual training requirements, for example, convolution kernels with high accuracy requirements and low number of selectable kernels are selected, for example, convolution kernels with high kernel number are selected according to time requirements, for example, the feature maps are divided into multiple groups according to the characteristics of the feature maps according to the convolution kernels 1 × 1, 3 × 3, 5 × 5, 7 × 7 and 9, and three adaptive groups are selected for carrying out grouping convolution; the parameter reduction here means that 12 feature maps are generated if the three-dimensional neural network is not grouped, and the 12 feature maps may be too large to be processed by the server at one time, but the grouping may be four groups of 3, 3 and 3, and the server may process 3 at one time.
2.3, in order to prevent the result from being over-fitted, performing a group of 1-by-1 convolution on each group of the feature map simulation residual error network structures after convolution to obtain a plurality of corresponding groups of feature maps; as a specific example according to step 2.2, three sets of feature maps may be obtained,
and 2.4, splicing and combining the plurality of groups of feature maps subjected to the weighting and residual error processing in the steps 2.1-2.3 to obtain a feature map with high scale, high dimensionality and richer information content.
In fig. 4, "input" is a plurality of sets of feature maps, the node Split is to group the plurality of sets of feature maps, conv is a convolution kernel, path is a path, SE is a residual module, and "output" is a feature map after combination and splicing.
In this embodiment, in step 2, a specific process of performing convolution operation on the input multiple sets of feature maps by the iterative convolution unit is as follows: the iterative convolution unit carries out iterative convolution operation on a plurality of groups of multi-angle feature maps to circularly extract the information of the feature maps, the voxel probability related to the iterative convolution unit is dynamically updated, under the condition of multiple views, the processing of each view updates the currently generated voxel probability model to generate a more accurate probability model, and finally the voxel probability data of all the iterative convolution units are combined to synthesize a group of three-dimensional voxel probability models. It should be noted that, the information of the feature map is extracted circularly here, which means that a hidden layer between the input layer and the output layer of the iterative convolution unit is used to perform a second processing on the generated information of the feature map, wherein the hidden layer of the iterative convolution unit (long-short time network) is arranged in the three-dimensional grid
In this embodiment, in the iterative convolution module, a specific structure of the iterative convolution unit is determined by the resolution of the feature map, the iterative convolution unit of X × X is configured to process data of the feature map with X resolution, and X is a positive integer.
In this embodiment, whether the ancient cultural relic three-dimensional network model is reliable is verified through a loss function, where the loss function is a feature regularization softmax function after a cross entropy function is improved. The normalization processing of data is added on the basis of the existing A-softmax and L-softmax, when the quality of a plurality of groups of data is different, feature extraction is carried out according to the specific picture data condition, and the problems of concentrated sample attention failure and the like caused by picture quality are solved.
The specific formula is as follows:
wherein, theta is defined as an interval angle, m is an introduced parameter factor, and is defined as an angle characteristic distance parameter, the distance between the characteristics is adjusted through m, and is fixed to be 4, s is a scaling factor, and is fixed to be 30.
The encoder is a 3D-ResVGG network improved on the existing ResNet network, combines the advantages of a ResNet residual module and a VGG network architecture, and realizes high-precision feature extraction on the basis of only using 3 x 3 convolution and 1 x 1 convolution; the iterative convolution module is composed of a group of iterative convolution units, each iterative convolution unit generates a voxel probability, and a complete three-dimensional voxel probability model is formed by multi-view image input; the decoder is composed of a traditional resnet network, the resnet network model is realized by using the prior art, the traditional resnet network model can prevent the overfitting problem caused by too deep network learning, meanwhile, the problem that the voxel probability model accuracy possibly caused by an encoder and an iterative convolution module is unstable is solved, and finally, a convolution neural network is formed. The input of the encoder is a single-view two-dimensional image, the output of the encoder is a two-dimensional feature vector, and the two-dimensional feature vector needs to be converted into three-dimensional information; the iterative convolution module completes a voxel probability model distributed in the three-dimensional grid. The input of the decoder is three-dimensional information obtained by processing of the iterative convolution module, and the output of the decoder is three-dimensional prediction voxel occupation of a single image.
The feasibility of the protocol of example 1 was verified in conjunction with specific experiments, as described in detail below:
1) experimental data set
ShapeNet data set
This data set consists of a three-dimensional CAD model of an object, the largest three-dimensional model data set that contains rich annotations to date. The three-dimensional model is organized under WordNet classification, rich semantic annotations are provided for each three-dimensional model, the semantic annotations comprise physical dimensions, keywords and the like, and the annotations can be provided through a Web-based interface so as to realize data visualization of object attributes. ShapeNet contains over 300 ten thousand models in total, of which 22 ten thousand models are classified into 3135 categories.
2) Evaluation criteria
IoU:Intersection-over-Union
IoU is used as an evaluation index, and the larger the evaluation index is, the better the evaluation index is. The formula is as follows, wherein I (x) is an indicator function and t represents a threshold value. If the probability is greater than the threshold, the voxel exists in the position, and the specific formula is as follows:
3) training process
The invention trains the ancient cultural relic three-dimensional network model by using a ShapeNet data set. The picture of 127 × 3 is input, and after passing through the ancient cultural relic three-dimensional network model, the voxel space of 32 × 32 is output.
In summary, the embodiment of the present invention verifies the feasibility of the scheme in embodiment 1 through the experimental process, the experimental data, and the experimental results, and the three-dimensional object reconstruction algorithm provided by the embodiment of the present invention has a good three-dimensional reconstruction capability for the two-dimensional image.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A three-dimensional ancient cultural relic reconstruction method based on single view and deep learning is characterized by comprising the following steps:
step 1: inputting a data set of an ancient cultural relic and a handicraft similar to the ancient cultural relic, wherein the data set comprises a multi-angle multi-view picture of the ancient cultural relic and the handicraft similar to the ancient cultural relic and a voxel model for training;
step 2: according to a network structure of an encoder-decoder, an encoder with a 3D-ResVGG network and a multipath channel attention module is used for carrying out deep information mining and feature extraction on the data set, and an ancient cultural relic three-dimensional network model is generated;
and step 3: preprocessing an ancient cultural relic picture needing three-dimensional reconstruction through an intelligent AI tool to generate a preset type picture;
and 4, step 4: and inputting the preset type picture as a single view into the ancient cultural relic three-dimensional network model to generate a complete ancient cultural relic three-dimensional model of the ancient cultural relic picture needing three-dimensional reconstruction.
2. The method for reconstructing the ancient cultural relics based on the single view and the deep learning as claimed in claim 1, wherein in the step 2, the ancient cultural relics three-dimensional network model comprises an encoder, an iterative convolution module and a decoder, the encoder is composed of a residual network structure combined with a multipath channel attention module and a 3D-ResVGG network, the iterative convolution module is composed of a group of iterative convolution units, the iterative convolution units are spatially distributed in a three-dimensional grid structure, and each iterative convolution unit is responsible for reconstructing a final output voxel probability result.
3. The method of claim 1, wherein the 3D-ResVGG network is obtained by adding two groups of 1 x 1 convolutions to a forward propagation part of a ResNet network and combining a ResNet residual module and a VGG network architecture.
4. The method as claimed in claim 3, wherein the multipath channel attention module is used for: and performing grouping convolution in the process of performing feature extraction on the 3D-ResVGG network by using the multipath channel attention module, and performing deep mining on data information in the form of multipath multi-group convolution kernels to obtain the depth information of the data.
5. The method as claimed in claim 4, wherein the encoder in step 2 performs depth information mining and feature extraction on the data set by the following specific steps:
2.1, the 3D-ResVGG network receives a data set and carries out feature extraction on the data set to generate a feature map;
step 2.2, the multipath channel attention module performs grouping extraction on the feature map generated by the 3D-ResVGG network: dividing the characteristic graph into a plurality of groups according to the characteristics of the characteristic graph and convolution kernels, and selecting a plurality of groups of adaptive convolution kernels to perform grouping convolution on the characteristic graph;
2.3, performing a group of 1 × 1 convolution on each group of the feature map simulation residual error network structure after convolution to obtain a plurality of corresponding groups of feature maps;
and 2.4, splicing and combining the plurality of groups of feature maps obtained in the step 2.3 to obtain a feature map with high scale, high dimensionality and richer information content.
6. The method as claimed in claim 5, wherein in step 2.2, 3 groups of information convolution kernels are selected to perform the grouping convolution on the feature map.
7. The ancient cultural relic three-dimensional reconstruction method based on the single view and the deep learning according to claim 6, wherein the iterative convolution unit is composed of a long and short convolution network, the long and short convolution network selects specific convolution unit quantity according to specific format of the input feature map, the feature map extracted by an encoder is subjected to iterative convolution from a three-dimensional angle, a voxel probability result of each convolution in the feature map is obtained in a classification network mode, and a group of three-dimensional probability information is collected and retained.
8. The method according to claim 7, wherein in step 2, the iterative convolution unit performs convolution operation on the input multiple groups of feature maps by using a specific process that is: and the iterative convolution unit performs iterative convolution operation on a plurality of groups of multi-angle characteristic images to circularly extract the information of the characteristic images, dynamically updates the voxel probability related to the iterative convolution unit, and finally combines the voxel probability data of all the iterative convolution units to synthesize a group of three-dimensional voxel probability models.
9. The method according to claim 8, wherein in the iterative convolution module, the specific structure of the iterative convolution unit is determined by the resolution of the feature map, and the iterative convolution unit of X is used for processing data of the feature map of X resolution.
10. The ancient cultural relic three-dimensional reconstruction method based on the single view and the deep learning as claimed in any one of claims 1 to 9, wherein whether the ancient cultural relic three-dimensional network model is reliable or not is verified through a loss function, wherein the loss function is a feature regularization softmax function after cross entropy function improvement.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111510170.0A CN114255328A (en) | 2021-12-10 | 2021-12-10 | Three-dimensional reconstruction method for ancient cultural relics based on single view and deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111510170.0A CN114255328A (en) | 2021-12-10 | 2021-12-10 | Three-dimensional reconstruction method for ancient cultural relics based on single view and deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114255328A true CN114255328A (en) | 2022-03-29 |
Family
ID=80794841
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111510170.0A Pending CN114255328A (en) | 2021-12-10 | 2021-12-10 | Three-dimensional reconstruction method for ancient cultural relics based on single view and deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114255328A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116502548A (en) * | 2023-06-29 | 2023-07-28 | 湖北工业大学 | Three-dimensional toy design method based on deep learning |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2011075082A1 (en) * | 2009-12-14 | 2011-06-23 | Agency For Science, Technology And Research | Method and system for single view image 3 d face synthesis |
| ES2400277A2 (en) * | 2009-05-21 | 2013-04-08 | Intel Corporation | Techniques for rapid stereo reconstruction from images |
| CN112862949A (en) * | 2021-01-18 | 2021-05-28 | 北京航空航天大学 | Object 3D shape reconstruction method based on multiple views |
| CN113724361A (en) * | 2021-08-23 | 2021-11-30 | 西安工业大学 | Single-view three-dimensional reconstruction method based on deep learning |
-
2021
- 2021-12-10 CN CN202111510170.0A patent/CN114255328A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| ES2400277A2 (en) * | 2009-05-21 | 2013-04-08 | Intel Corporation | Techniques for rapid stereo reconstruction from images |
| WO2011075082A1 (en) * | 2009-12-14 | 2011-06-23 | Agency For Science, Technology And Research | Method and system for single view image 3 d face synthesis |
| CN112862949A (en) * | 2021-01-18 | 2021-05-28 | 北京航空航天大学 | Object 3D shape reconstruction method based on multiple views |
| CN113724361A (en) * | 2021-08-23 | 2021-11-30 | 西安工业大学 | Single-view three-dimensional reconstruction method based on deep learning |
Non-Patent Citations (3)
| Title |
|---|
| CHIA-HO HSU: "Fast Single-View 3D Object Reconstruction with Fine Details Through Dilated Downsample and Multi-Path Upsample Deep Neural Network", ICASSP 2020 - 2020 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 9 April 2020 (2020-04-09) * |
| 叶成庆: "基于单视图和深度学习的古文物三维重建技术研究", CNKI优秀硕士学位论文全文库, 15 June 2023 (2023-06-15) * |
| 朱莉;陈辉;: "基于深度学习的单幅图像三维重建算法", 吉林化工学院学报, no. 01, 15 January 2020 (2020-01-15) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116502548A (en) * | 2023-06-29 | 2023-07-28 | 湖北工业大学 | Three-dimensional toy design method based on deep learning |
| CN116502548B (en) * | 2023-06-29 | 2023-09-15 | 湖北工业大学 | Three-dimensional toy design method based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Pittaluga et al. | Revealing scenes by inverting structure from motion reconstructions | |
| CN108921926B (en) | End-to-end three-dimensional face reconstruction method based on single image | |
| US11328481B2 (en) | Multi-resolution voxel meshing | |
| CN115100339A (en) | Image generation method and device, electronic equipment and storage medium | |
| CN113159232A (en) | Three-dimensional target classification and segmentation method | |
| Lu et al. | Design and implementation of virtual interactive scene based on unity 3D | |
| CN112634456B (en) | Real-time high-realism drawing method of complex three-dimensional model based on deep learning | |
| CN116977531A (en) | Three-dimensional texture image generation method, three-dimensional texture image generation device, computer equipment and storage medium | |
| Choi et al. | Balanced spherical grid for egocentric view synthesis | |
| CN114255328A (en) | Three-dimensional reconstruction method for ancient cultural relics based on single view and deep learning | |
| Chen et al. | Fast virtual view synthesis for an 8k 3d light-field display based on cutoff-nerf and 3d voxel rendering | |
| CN117557721A (en) | Method, system, equipment and medium for reconstructing detail three-dimensional face of single image | |
| CN116935008A (en) | Display interaction method and device based on mixed reality | |
| Wang et al. | Integrated design system of voice-visual VR based on multi-dimensional information analysis | |
| CN115686202A (en) | Three-dimensional model interactive rendering method across Unity/Optix platform | |
| JP2019149112A (en) | Composition device, method, and program | |
| Xu et al. | Real-time panoramic map modeling method based on multisource image fusion and three-dimensional rendering | |
| Yao et al. | Neural Radiance Field-based Visual Rendering: A Comprehensive Review | |
| Xie | Dance Performance in New Rural Areas Based on 3D Image Reconstruction Technology | |
| Gao et al. | Research on digital protection of bamboo tube based on hani nationality | |
| Liu | [Retracted] Ming‐Style Furniture Display Design Based on Immersive 5G Virtual Reality | |
| Rebollo et al. | Three-dimensional trees for virtual globes | |
| Liao et al. | Multi-View Image Reconstruction Algorithm Based on Virtual Reality Technology | |
| Jiang et al. | PMPI: Patch-Based Multiplane Images for Real-Time Rendering of Neural Radiance Fields | |
| He | Interactive Virtual Reality Indoor Space Roaming System Based on 3D Vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |