CN117132501B - Human body point cloud cavity repairing method and system based on depth camera - Google Patents
Human body point cloud cavity repairing method and system based on depth camera Download PDFInfo
- Publication number
- CN117132501B CN117132501B CN202311189283.4A CN202311189283A CN117132501B CN 117132501 B CN117132501 B CN 117132501B CN 202311189283 A CN202311189283 A CN 202311189283A CN 117132501 B CN117132501 B CN 117132501B
- Authority
- CN
- China
- Prior art keywords
- point cloud
- human body
- feature
- deep
- depth camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000000605 extraction Methods 0.000 claims abstract description 34
- 238000013135 deep learning Methods 0.000 claims abstract description 16
- 239000011159 matrix material Substances 0.000 claims description 48
- 238000011176 pooling Methods 0.000 claims description 20
- 230000004913 activation Effects 0.000 claims description 13
- 230000006870 function Effects 0.000 claims description 10
- 230000008439 repair process Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 7
- 238000010606 normalization Methods 0.000 claims description 4
- 230000003213 activating effect Effects 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 abstract description 6
- 230000000007 visual effect Effects 0.000 abstract description 4
- 238000010586 diagram Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 230000000295 complement effect Effects 0.000 description 4
- 239000013598 vector Substances 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 241000196324 Embryophyta Species 0.000 description 1
- 241000282412 Homo Species 0.000 description 1
- 230000001174 ascending effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
- G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
- G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/7715—Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Multimedia (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Biodiversity & Conservation Biology (AREA)
- Image Processing (AREA)
Abstract
The invention provides a human body point cloud cavity repairing method and system based on a depth camera, wherein the repairing method comprises the following steps: s1, acquiring human body point cloud by using a depth camera, wherein the human body point cloud has cavities; s2, inputting human body point cloud into a point cloud feature extraction network to extract point cloud deep feature information; s3, deep learning is carried out on the deep characteristic information, and the human body point cloud cavity is completed by combining with a 2D point cloud grid; and S4, outputting the complemented human body point cloud. According to the invention, deep characteristic information is extracted through deep learning and combined with the 2D point cloud grid, so that the integrity of point cloud data can be improved, the visual effect of the point cloud data is improved, more accurate and complete point cloud data is provided for subsequent human modeling and animation production, the human modeling and animation production is more real and vivid, holes can be effectively filled, the point cloud model is complete and unbreakable, and the shape and appearance of a human body can be more truly presented.
Description
Technical Field
The invention relates to the field of point cloud model processing, in particular to a human body point cloud cavity repairing method and system based on a depth camera.
Background
In recent years, with the progressive maturity of computer vision computing, a point cloud is a three-dimensional data type that is composed of a large amount of point data, each of which contains a position in a three-dimensional space and other possible attribute information such as color, normal vector, transparency, reflectance, and the like. The relationship and attribute information between these points can accurately represent objects and scenes in the real world, such as buildings, roads, vehicles, plants, humans, and the like. The real scene data can be acquired through laser scanning or camera capturing and is expressed as a point cloud form, so that further analysis and processing are facilitated. Meanwhile, the point cloud data type is commonly used for terrain modeling, building modeling, human modeling and the like; in addition, point cloud data types are also widely used in virtual reality and game development, for example, point cloud data can be used to construct virtual scenes, virtual characters, and virtual objects, thereby achieving a more realistic and immersive virtual reality and game experience.
The 3D human body point cloud model is a computer image technology for representing the shape of a human body, can capture the appearance and posture change of the human body in a three-dimensional form, and can realize high-precision estimation of the posture of the human body by capturing key points and posture change. In addition, the virtual garment fitting method can also be used for virtual fitting, and the virtual garment is mapped onto the human body point cloud model so as to display and evaluate the fitting effect. The human body point cloud model is widely applied to medical image analysis, can be used for surgical design, and helps doctors to plan surgical schemes and operation paths better. However, in practical use, due to the existence of the shielding object and the light, the human body point cloud model may have holes after being acquired by the depth camera, and the holes may cause incomplete or distorted point cloud data, which brings difficulty to subsequent data processing and analysis.
Chinese patent publication No. CN115330632a discloses a method for repairing a point cloud hole of a welded plate based on point cloud rasterization, which inputs a point cloud image to be completed, creates a plurality of bounding boxes, performs equation fitting according to the point cloud in each bounding box, traverses the whole human body point cloud, fits the hole with an equation when the hole exists, and traverses the next when the hole does not exist. However, the method for repairing the cavity by using the equation fitting method needs to create a bounding box and fitting an equation for the whole point cloud, so that the time efficiency is low, the transformation amplitude of the edge information of the human point cloud is large, and the requirements on precision and authenticity are difficult to meet by using the method of equation fitting only.
Disclosure of Invention
In order to achieve the technical purpose, the invention provides a human body point cloud cavity repairing method based on a depth camera, which aims to solve the cavity defect of a human body point cloud model in the prior art and further improve the precision and the authenticity of the point cloud model, and specifically comprises the following steps:
s1, acquiring human body point cloud by using a depth camera, wherein the human body point cloud has cavities;
s2, inputting human body point cloud into a point cloud feature extraction network to extract point cloud deep feature information;
s3, deep learning is carried out on the deep characteristic information, and the human body point cloud cavity is completed by combining with a 2D point cloud grid;
and S4, outputting the complemented human body point cloud.
Further, the depth camera is placed at a position 1-3 meters in front of the human body, and human body point cloud information is automatically acquired by the depth camera.
Further, the step S2 is to input the human body point cloud into the point cloud feature extraction network to extract the deep feature information of the point cloud, including inputting the point cloud, then to perform multiple dimension lifting on the human body point cloud, and finally to perform feature stitching on the multiple dimension lifting results, and to obtain the deep feature information extraction result after feature stitching.
Further, the step S2 of inputting the human body point cloud into the point cloud feature extraction network to extract the deep feature information of the point cloud specifically includes the following steps:
the method comprises the steps of inputting a point cloud, wherein the characteristic of the input point cloud is Nx 3;
first order point Yun Sheng dimension, feature dimension up to n×64;
a second point Yun Shengwei, where feature extraction is performed without changing dimensions, and the feature is kept as n×64;
third order point Yun Sheng dimension, feature upscales to n×128;
the fourth time point Yun Sheng dimension, feature up dimension to nx256;
and finally, performing feature stitching on the results of four point cloud feature dimension lifting, wherein the output result is N multiplied by 512, namely the result of extracting deep feature information is N multiplied by 512.
Further, the first secondary point Yun Shengwei, the second secondary point Yun Shengwei, the third secondary point cloud ascending and the fourth secondary point Yun Shengwei all go through the multi-layer perceptron to ascend, and the multi-layer perceptron ascends and then performs the normalization operation and the activation operation.
Further, the point cloud feature extraction network in step S2 further includes a fused self-attention network, and a global feature and context information in the fused self-attention network capturing point cloud are set after each point cloud dimension increase.
Further, the structure of the fused self-attention network is as follows:
inputting the characteristics of the Yun Sheng-dimensional points into a fused self-attention network, wherein the input characteristics are N multiplied by C, N represents the number of point clouds, and C represents the dimension of each point;
extracting features by 3 different multi-layer perceptrons, but not changing dimensions, to obtain 3N multiplied by C feature matrixes, namely a first feature matrix, a second feature matrix and a third feature matrix;
after the first feature matrix is transposed, multiplying the first feature matrix by the second feature matrix, and activating the multiplied result through a Sigmoid activation function to obtain a probability matrix, wherein the probability matrix is a similar probability matrix, and the probability matrix is N multiplied by N;
respectively carrying out maximum pooling and average pooling on the probability matrix, adding the result features after the maximum pooling and the average pooling, and copying N copies of the result of feature addition to obtain a new matrix of NxN;
and multiplying the third characteristic matrix with the obtained new matrix to obtain an output result, wherein the output result is NxC.
Further, the step S3 performs deep learning on the deep feature information and combines with the 2D point cloud grid to complement the human point cloud cavity, and includes:
copying the result characteristic of the deep characteristic information for 4 times to obtain 4Nx512;
reducing the dimension of the result of extracting the deep feature information to N multiplied by 3 through a multi-layer perceptron, and then copying the feature for 4 times to be changed into 4N multiplied by 3;
forming a 2D point cloud grid according to the acquired human body point cloud;
and (3) performing feature splicing on the 4N multiplied by 512 and 4N multiplied by 3 and 2D point cloud grids which are obtained by the previous two copies, and reducing the dimension of the feature spliced result to 4N multiplied by 3 by adopting a multi-layer perceptron to finish the completion of the human point cloud cavities.
The invention also provides a human body point cloud cavity repairing system based on the depth camera, which comprises the following steps:
the human body point cloud acquisition module is a depth camera and is used for acquiring human body point cloud, and the human body point cloud has a cavity;
the deep feature information extraction module is used for inputting human point cloud into the point cloud feature extraction network to extract point cloud deep feature information;
the human body point cloud cavity complementing module is used for carrying out deep learning on the deep characteristic information and complementing the human body point cloud cavity by combining a 2D point cloud grid;
and the output module outputs the complemented human body point cloud.
The invention also provides a computer device comprising a memory and a processor; the memory has stored thereon a computer program executable by the processor; and when the processor runs the computer program, executing the human body point cloud cavity repairing method based on the depth camera.
Compared with the prior art, the invention has the following beneficial effects:
(1) The deep characteristic information is extracted through deep learning and combined with the 2D point cloud grid, the integrity of the point cloud data can be improved, the visual effect of the point cloud data is improved, more accurate and complete point cloud data is provided for subsequent human modeling and animation production, the human modeling and animation production is more real and vivid, holes can be effectively filled, the point cloud model is complete, and the shape and appearance of a human body can be more truly presented.
(2) The deep learning of the deep feature information can adaptively adjust the model parameters of the network, and the model parameters are continuously adjusted and optimized, so that the stability and the robustness of hole repair are improved, the hole repair result is more stable and reliable, and meanwhile, the deep learning method can adapt to human body point cloud models of different types and scales, and is widely applied.
(3) Interaction and information transfer between various points in the point cloud can be realized based on the fusion self-attention feature extraction network, so that global features of the point cloud can be captured better, and meanwhile, feature aggregation and screening can be carried out on the whole point cloud, so that the robustness and the expression capability of the point cloud features are improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic block flow diagram of a human body point cloud cavity repairing method based on a depth camera according to an embodiment of the present invention.
Fig. 2 is a diagram of a human body cavity repairing network structure of a human body point cloud cavity repairing method based on a depth camera according to an embodiment of the invention.
Fig. 3 is a diagram of a point cloud deep layer feature extraction network structure of a human body point cloud cavity restoration method based on a depth camera according to an embodiment of the present invention.
Fig. 4 is a network structure diagram of a fused self-attention module of a human body point cloud hole repairing method based on a depth camera provided by the embodiment of the invention.
Fig. 5 is a block diagram of a human body point cloud cavity repairing system based on a depth camera according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described with reference to the accompanying drawings.
The terms first and second and the like in the description, the claims and the drawings of the present invention are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the terms "comprising," "including," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion. Such as a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to the list of steps or elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those skilled in the art will appreciate explicitly and implicitly that the described embodiments of the invention may be combined with other embodiments.
In the present invention, "at least one (item)" means one or more, "a plurality" means two or more, "at least two (items)" means two or three and more, "and/or" for describing an association relationship of an association object, and three kinds of relationships may exist, for example, "a and/or B" may represent: only a, only B and both a and B are present, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of (a) or a similar expression thereof means any combination of these items. For example, at least one (one) of a, b or c may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c".
When the human body point cloud is obtained by using the depth camera, due to the reasons of shielding, light rays and the like, the human body point cloud shot by the depth camera is provided with holes, so that the point cloud model is provided with holes, the holes can enable the point cloud data to be incomplete or distorted, the visual effect is affected, and therefore great difficulty is brought to subsequent data processing and analysis, and the holes are required to be repaired and completed.
As shown in fig. 1-2, the invention provides a human body point cloud cavity repairing method based on a depth camera, which comprises the following steps:
s1, acquiring human body point cloud by using a depth camera, wherein the human body point cloud has cavities;
s2, inputting human body point cloud into a point cloud feature extraction network to extract point cloud deep feature information;
s3, deep learning is carried out on the deep characteristic information, and the human body point cloud cavity is completed by combining with a 2D point cloud grid;
and S4, outputting the complemented human body point cloud.
Optionally, the depth camera is placed at a position 1-3 meters in front of the human body, and the depth camera automatically acquires human body point cloud information; further preferably, the depth camera is placed 1.5 meters in front of the human body to acquire the point cloud data.
Step S2, inputting human body point cloud into a point cloud feature extraction network to extract point cloud deep feature information, wherein the point cloud deep feature information comprises input point cloud, then carrying out multiple dimension lifting on the human body point cloud, and finally carrying out feature splicing on the multiple dimension lifting results to obtain a result of extracting deep feature information after feature splicing. And (2) extracting deep point cloud features of the human point cloud with the cavity through the step (S2), and converting the low-dimensional features of the input point cloud into high-dimensional feature representations so as to better learn and model the point cloud. By means of the point cloud feature dimension improvement, detail information and local features in the point cloud can be better captured, and therefore the expression capacity and the expression capacity of the point cloud are improved; the feature after dimension lifting can better represent the cavity and the missing area in the point cloud, and the network is helped to better complement the point cloud.
As shown in fig. 3, the step S2 specifically includes the following steps:
the method comprises the steps of inputting a point cloud, wherein the characteristic of the input point cloud is Nx 3;
first order point Yun Sheng dimension, feature dimension up to n×64;
a second point Yun Shengwei, where feature extraction is performed without changing dimensions, and the feature is kept as n×64;
third order point Yun Sheng dimension, feature upscales to n×128;
the fourth time point Yun Sheng dimension, feature up dimension to nx256;
and finally, performing feature stitching on the results of four point cloud feature dimension lifting, wherein the output result is N multiplied by 512, namely the result of extracting deep feature information is N multiplied by 512.
The input point cloud is characterized by n×3, N represents the number of point clouds, and 3 represents the (x, y, z) coordinates of the point clouds, and the dimension is 3 since only the coordinates are used as the characteristics of the point clouds. In addition, the point cloud is also characterized by curvature, color, and the like, and only coordinate information is used here as the point cloud feature for convenience, so the dimension is 3.
The second time point Yun Shengwei is a special point Yun Shengwei step, only the feature extraction is performed on the result of the first time point Yun Sheng dimension, but the dimension is not changed, and the step is named as a second time point Yun Shengwei so as to be convenient for unifying the names of all the steps in the point cloud feature extraction network; the feature extraction is performed under the condition of not changing the dimension, so that the expression capability of the feature is improved and the computational complexity is reduced while the data structure and the information are reserved.
The first secondary point Yun Shengwei, the second secondary point Yun Shengwei, the third secondary point cloud lifting and the fourth secondary point Yun Shengwei all perform feature extraction and dimension lifting through the multi-layer perceptron, and the multi-layer perceptron performs normalization operation and activation operation after dimension lifting.
Specifically, taking the first time point Yun Shengwei as an example, the human point cloud with a cavity is input into the point cloud feature extraction network, and then the dimension of the human point cloud is increased to 64 through a multi-layer perceptron, so that the feature of the point cloud becomes n×64, and then the multi-layer perceptron operates to perform normalization operation and activation operation, wherein the activation function used in the activation operation is an activation function, and thus one dimension increasing operation is completed. Other activation functions can of course be used for the activation operation, but the effect of extracting features is better when the activation function used is.
The point cloud can be converted into the feature vector with higher dimensionality from the original three-dimensional coordinate data through the dimension increasing operation, so that the shape, the property and the feature of the point cloud can be better described, more abundant and accurate feature information can be better extracted from the original point cloud, and a more accurate and reliable basis is provided for subsequent cavity repair and human modeling.
In other embodiments, the human point cloud performs multiple dimension lifting, the number of dimension lifting times and the dimension of each dimension lifting can be set according to actual needs, and finally, feature stitching is performed on the results of multiple dimension lifting, so that the output result is usually ensured to be n×512.
The point cloud feature extraction network in step S2 further includes a fused self-attention network, and a global feature and context information in the point cloud captured by the fused self-attention network are set after each point cloud dimension increase. As shown in fig. 4, the structure of the fused self-attention network is as follows:
inputting the characteristics of the Yun Sheng-dimensional points into a fused self-attention network, wherein the input characteristics are N multiplied by C, N represents the number of point clouds, and C represents the dimension of each point;
extracting features by 3 different multi-layer perceptrons, but not changing dimensions, to obtain 3N multiplied by C feature matrixes, namely a first feature matrix, a second feature matrix and a third feature matrix;
after the first feature matrix is transposed, multiplying the first feature matrix by the second feature matrix, and activating the multiplied result through a Sigmoid activation function to obtain a probability matrix, wherein the probability matrix is a similar probability matrix, and the probability matrix is N multiplied by N;
respectively carrying out maximum pooling and average pooling on the probability matrix, adding the result features after the maximum pooling and the average pooling, and copying N copies of the result of feature addition to obtain a new matrix of NxN;
and multiplying the third characteristic matrix with the obtained new matrix to obtain an output result, wherein the output result is NxC.
The 3 different multi-layer perceptrons can also be other numbers of the plurality of different multi-layer perceptrons, and the 3 different multi-layer perceptrons are the best choice in consideration of comprehensive calculation force and calculation precision. The probability matrix is subjected to maximum pooling and average pooling respectively, wherein the maximum pooling can reflect the most prominent features, but other features are ignored, all features are considered in the average pooling, the result features after the maximum pooling and the average pooling are added, and the most prominent features and all features are comprehensively considered.
The calculation formula of the fused self-attention network is as follows:
（1）
（2）
（3）
（4）
wherein,for the first feature matrix of feature extraction, < +.>A second feature matrix extracted for the feature, < >>A third feature matrix extracted for the feature, < +.>For the first multi-layer perceptron, +.>For the second multi-layer perceptron +.>For the third multi-layer perceptron +.>For the dimension of the input feature +.>Is a similarity probability matrix>For characteristic replication, ++>For maximum pooling, ++>For average pooling, +.>For the new matrix obtained ∈>To output the result.
Interaction and information transfer between various points in the point cloud can be realized based on the fusion self-attention feature extraction network, so that global features of the point cloud can be captured better, and meanwhile, feature aggregation and screening can be carried out on the whole point cloud, so that the robustness and the expression capability of the point cloud features are improved.
Carrying out dimension lifting on the point cloud characteristics through repeated dimension lifting operation and self-attention fusion operation, and lifting the point cloud characteristics to NX 64 for the first time; extracting features to NX 64 for the second time without changing the dimension; third dimension up feature to nx128; fourth dimension up feature to nx256; and finally, performing feature stitching on all the dimension-up results to obtain 64+64+128+256=512, wherein the output dimension is N×512. From here on, the point cloud feature extraction network ends.
Further, step S3 performs deep learning on the deep feature information and combines with the 2D point cloud grid to complement the human point cloud cavity, as shown in fig. 2, including:
copying the result characteristic of the deep characteristic information for 4 times to obtain 4Nx512;
reducing the dimension of the result of extracting the deep feature information to N multiplied by 3 through a multi-layer perceptron, and then copying the feature for 4 times to be changed into 4N multiplied by 3;
forming a 2D point cloud grid according to the acquired human body point cloud;
and performing feature splicing on the 4N multiplied by 512 and 4N multiplied by 3 and 2D point cloud grids which are obtained by the previous two feature copying results, and reducing the dimension of the feature splicing results to 4N multiplied by 3 by adopting a multi-layer perceptron to finish the point cloud hole completion of the human body.
The deep learning method can adaptively adjust the model parameters of the network, continuously adjust and optimize the model parameters, thereby improving the stability and the robustness of hole repair, enabling the hole repair result to be more stable and reliable, being applicable to human body point cloud models of different types and scales and realizing wider application.
The specific formula of step S3 is:
（5）
wherein,for characteristic splice->Is a multi-layer perceptron, is a->For characteristic replication, ++>For the extraction of deep characteristic information, the result is +.>Is a 2D grid of point clouds.
Wherein,the function of the method is to project and encode the point cloud so as to extract the local characteristics of the point cloud. />Projecting the point cloud onto a two-dimensional grid, and dividing the grid into a plurality of small areas; then, for each small region, calculating the mean and variance of the internal point cloud of each small region respectively, so as to obtain a local feature vector; these local feature vectors can be used to describe the local shape and spatial distribution information of the point cloud, thereby improving the expressive power and robustness of the point cloud.
The point cloud complement with the holes is expanded from the original N points to 4N points, and the areas with the holes are complemented by using the point cloud.
According to the invention, deep characteristic information is extracted through deep learning and combined with the 2D point cloud grid, so that the integrity of point cloud data can be improved, the visual effect of the point cloud data is improved, more accurate and complete point cloud data is provided for subsequent human modeling and animation production, the human modeling and animation production is more real and vivid, holes can be effectively filled, the point cloud model is complete and unbreakable, and the shape and appearance of a human body can be more truly presented.
Based on the human body point cloud cavity repairing method, the invention also provides a human body point cloud cavity repairing system based on a depth camera, as shown in fig. 5, comprising:
the human body point cloud acquisition module is a depth camera and is used for acquiring human body point cloud, and the human body point cloud has a cavity;
the deep feature information extraction module is used for inputting human point cloud into the point cloud feature extraction network to extract point cloud deep feature information;
the human body point cloud cavity complementing module is used for carrying out deep learning on the deep characteristic information and complementing the human body point cloud cavity by combining a 2D point cloud grid;
and the output module outputs the complemented human body point cloud.
The invention also provides a computer device comprising a memory and a processor; the memory has stored thereon a computer program executable by the processor; and when the processor runs the computer program, executing the human body point cloud cavity repairing method based on the depth camera.
The present invention also provides a computer readable storage medium comprising program code for causing an electronic device to perform the steps of the depth camera based human point cloud hole repair method described above when the program code is run on the electronic device.
The electronic device, the computer storage medium, the computer program product, or the chip provided in this embodiment are used to execute the corresponding methods provided above, so that the beneficial effects thereof can be referred to the beneficial effects in the corresponding methods provided above, and will not be described herein.
From the foregoing description of the embodiments, it will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of functional modules is illustrated, and in practical application, the above-described functional allocation may be implemented by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to implement all or part of the functions described above.
In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another apparatus, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and the parts displayed as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated unit may be stored in a readable storage medium if implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, the technical solution of the embodiments of the present invention may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions for causing a device (may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps of the methods of the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.
Claims (6)
1. A human body point cloud cavity repairing method based on a depth camera is characterized by comprising the following steps of:
s1, acquiring human body point cloud by using a depth camera, wherein the human body point cloud has cavities;
s2, inputting human body point cloud into a point cloud feature extraction network to extract point cloud deep feature information;
s3, deep learning is carried out on the deep characteristic information, and the human body point cloud cavity is completed by combining with a 2D point cloud grid;
s4, outputting the complemented human body point cloud;
step S2 is to input human body point cloud into a point cloud feature extraction network to extract point cloud deep feature information, wherein the point cloud deep feature information comprises input point cloud, then to carry out multiple dimension lifting on the human body point cloud, finally to carry out feature stitching on the multiple dimension lifting results, and to obtain the deep feature information extraction result after feature stitching, specifically as follows:
the method comprises the steps of inputting a point cloud, wherein the characteristic of the input point cloud is Nx 3;
first order point Yun Sheng dimension, feature dimension up to n×64;
a second point Yun Shengwei, where feature extraction is performed without changing dimensions, and the feature is kept as n×64;
third order point Yun Sheng dimension, feature upscales to n×128;
the fourth time point Yun Sheng dimension, feature up dimension to nx256;
finally, feature stitching is carried out on the results of four point cloud feature dimension lifting, the output result is N multiplied by 512, namely the result of deep feature information extraction is N multiplied by 512;
the first secondary point Yun Shengwei, the second secondary point Yun Shengwei, the third secondary point cloud lifting and the fourth secondary point Yun Shengwei are all lifted by the multi-layer perceptron, and the normalization operation and the activation operation are all performed after the lifting of the multi-layer perceptron;
step S3 carries out deep learning on deep characteristic information and completes human body point cloud cavities by combining 2D point cloud grids, and comprises the following steps:
copying the result characteristic of the deep characteristic information for 4 times to obtain 4Nx512;
reducing the dimension of the result of extracting the deep feature information to N multiplied by 3 through a multi-layer perceptron, and then copying the feature for 4 times to be changed into 4N multiplied by 3;
forming a 2D point cloud grid according to the acquired human body point cloud;
and (3) performing feature splicing on the 4N multiplied by 512 and 4N multiplied by 3 and 2D point cloud grids which are obtained by the previous two copies, and reducing the dimension of the feature spliced result to 4N multiplied by 3 by adopting a multi-layer perceptron to finish the completion of the human point cloud cavities.
2. The human body point cloud cavity repairing method based on the depth camera according to claim 1, wherein the depth camera is placed at a position 1-3 meters in front of a human body, and human body point cloud information is automatically acquired by the depth camera.
3. The depth camera-based human body point cloud hole repairing method according to claim 1, wherein the point cloud feature extraction network in the step S2 further comprises a fused self-attention network, and global features and context information in the point cloud are captured by setting one fused self-attention network after each point cloud dimension increase.
4. The depth camera-based human body point cloud cavity restoration method as recited in claim 3, wherein the structure of the fused self-attention network is as follows:
inputting the characteristics of the Yun Sheng-dimensional points into a fused self-attention network, wherein the input characteristics are N multiplied by C, N represents the number of point clouds, and C represents the dimension of each point;
extracting features by 3 different multi-layer perceptrons, but not changing dimensions, to obtain 3N multiplied by C feature matrixes, namely a first feature matrix, a second feature matrix and a third feature matrix;
after the first feature matrix is transposed, multiplying the first feature matrix by the second feature matrix, and activating the multiplied result through a Sigmoid activation function to obtain a probability matrix, wherein the probability matrix is a similar probability matrix, and the probability matrix is N multiplied by N;
respectively carrying out maximum pooling and average pooling on the probability matrix, adding the result features after the maximum pooling and the average pooling, and copying N copies of the result of feature addition to obtain a new matrix of NxN;
and multiplying the third characteristic matrix with the obtained new matrix to obtain an output result, wherein the output result is NxC.
5. A depth camera-based human point cloud hole repair system, wherein the depth camera-based human point cloud hole repair method of any one of claims 1 to 4 is performed, the depth camera-based human point cloud hole repair system comprising:
the human body point cloud acquisition module is a depth camera and is used for acquiring human body point cloud, and the human body point cloud has a cavity;
the deep feature information extraction module is used for inputting human point cloud into the point cloud feature extraction network to extract point cloud deep feature information;
the human body point cloud cavity complementing module is used for carrying out deep learning on the deep characteristic information and complementing the human body point cloud cavity by combining a 2D point cloud grid;
and the output module outputs the complemented human body point cloud.
6. A computer device comprising a memory and a processor; the memory has stored thereon a computer program executable by the processor; the processor, when running the computer program, performs the depth camera-based human point cloud hole repair method of any of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311189283.4A CN117132501B (en) | 2023-09-14 | 2023-09-14 | Human body point cloud cavity repairing method and system based on depth camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311189283.4A CN117132501B (en) | 2023-09-14 | 2023-09-14 | Human body point cloud cavity repairing method and system based on depth camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117132501A CN117132501A (en) | 2023-11-28 |
| CN117132501B true CN117132501B (en) | 2024-02-23 |
Family
ID=88850844
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311189283.4A Active CN117132501B (en) | 2023-09-14 | 2023-09-14 | Human body point cloud cavity repairing method and system based on depth camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117132501B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114743007A (en) * | 2022-04-20 | 2022-07-12 | 湘潭大学 | Three-dimensional semantic segmentation method based on channel attention and multi-scale fusion |
| CN115439694A (en) * | 2022-09-19 | 2022-12-06 | 南京邮电大学 | High-precision point cloud completion method and device based on deep learning |
| WO2022252274A1 (en) * | 2021-05-31 | 2022-12-08 | 北京理工大学 | Point cloud segmentation and virtual environment generation method and apparatus based on pointnet network |
| CN116137059A (en) * | 2023-04-17 | 2023-05-19 | 宁波大学科学技术学院 | Three-dimensional point cloud quality evaluation method based on multi-level feature extraction network model |
| CN116310104A (en) * | 2023-03-08 | 2023-06-23 | 武汉纺织大学 | Human body three-dimensional reconstruction method, system and storage medium under complex scene |
| CN116452757A (en) * | 2023-06-15 | 2023-07-18 | 武汉纺织大学 | Human body surface reconstruction method and system under complex scene |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20230206603A1 (en) * | 2022-09-19 | 2023-06-29 | Nanjing University Of Posts And Telecommunications | High-precision point cloud completion method based on deep learning and device thereof |
-
2023
- 2023-09-14 CN CN202311189283.4A patent/CN117132501B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022252274A1 (en) * | 2021-05-31 | 2022-12-08 | 北京理工大学 | Point cloud segmentation and virtual environment generation method and apparatus based on pointnet network |
| CN114743007A (en) * | 2022-04-20 | 2022-07-12 | 湘潭大学 | Three-dimensional semantic segmentation method based on channel attention and multi-scale fusion |
| CN115439694A (en) * | 2022-09-19 | 2022-12-06 | 南京邮电大学 | High-precision point cloud completion method and device based on deep learning |
| CN116310104A (en) * | 2023-03-08 | 2023-06-23 | 武汉纺织大学 | Human body three-dimensional reconstruction method, system and storage medium under complex scene |
| CN116137059A (en) * | 2023-04-17 | 2023-05-19 | 宁波大学科学技术学院 | Three-dimensional point cloud quality evaluation method based on multi-level feature extraction network model |
| CN116452757A (en) * | 2023-06-15 | 2023-07-18 | 武汉纺织大学 | Human body surface reconstruction method and system under complex scene |
Non-Patent Citations (2)
| Title |
|---|
| 一种基于深度学习的点云修复模型;贝子勒;赵杰煜;;无线通信技术(第02期);全文 * |
| 唐煜程 ; 张明君 ; 王浩宇 ; 谢珍珠 ; 康今朝 ; 詹曙 ; .基于GPU的三维人脸数据动态线性快速修复.电子测量与仪器学报.2016,(第06期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117132501A (en) | 2023-11-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108509848B (en) | The real-time detection method and system of three-dimension object | |
| CN110458939B (en) | Indoor scene modeling method based on visual angle generation | |
| CN109147048B (en) | Three-dimensional mesh reconstruction method by utilizing single-sheet colorful image | |
| CN113012293B (en) | Stone carving model construction method, device, equipment and storage medium | |
| CN109978984A (en) | Face three-dimensional rebuilding method and terminal device | |
| CN111028343B (en) | Three-dimensional face model generation method, device, equipment and medium | |
| CN112950775A (en) | Three-dimensional face model reconstruction method and system based on self-supervision learning | |
| CN108898665A (en) | Three-dimensional facial reconstruction method, device, equipment and computer readable storage medium | |
| CN113496507A (en) | Human body three-dimensional model reconstruction method | |
| CN112085835B (en) | Three-dimensional cartoon face generation method and device, electronic equipment and storage medium | |
| CN112102480B (en) | Image data processing method, apparatus, device and medium | |
| Denninger et al. | 3d scene reconstruction from a single viewport | |
| CN114067057A (en) | Human body reconstruction method, model and device based on attention mechanism | |
| CN112530005B (en) | Three-dimensional model linear structure recognition and automatic restoration method | |
| CN112132739A (en) | 3D reconstruction and human face posture normalization method, device, storage medium and equipment | |
| CN110176079A (en) | A kind of three-dimensional model deformation algorithm based on quasi- Conformal | |
| CN116310045B (en) | Three-dimensional face texture creation method, device and equipment | |
| CN109979013A (en) | Three-dimensional face chart pasting method and terminal device | |
| CN114742956B (en) | Model processing method, device, equipment and computer readable storage medium | |
| CN115222917A (en) | Training method, device and equipment for three-dimensional reconstruction model and storage medium | |
| CN115115805A (en) | Training method, device and equipment for three-dimensional reconstruction model and storage medium | |
| CN115496862A (en) | Real-time three-dimensional reconstruction method and system based on SPIN model | |
| US20220375163A1 (en) | Computationally-Efficient Generation of Simulations of Cloth-Like Materials Using Bilinear Element Models | |
| Yin et al. | [Retracted] Virtual Reconstruction Method of Regional 3D Image Based on Visual Transmission Effect | |
| CN116385619B (en) | Object model rendering method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |