WO2020087564A1 - 三维物体重建方法、计算机设备及存储介质 - Google Patents

三维物体重建方法、计算机设备及存储介质 Download PDF

Info

Publication number
WO2020087564A1
WO2020087564A1 PCT/CN2018/114808 CN2018114808W WO2020087564A1 WO 2020087564 A1 WO2020087564 A1 WO 2020087564A1 CN 2018114808 W CN2018114808 W CN 2018114808W WO 2020087564 A1 WO2020087564 A1 WO 2020087564A1
Authority
WO
WIPO (PCT)
Prior art keywords
label
scanned
predicted semantic
dimensional
confidence
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/114808
Other languages
English (en)
French (fr)
Inventor
胡瑞珍
黄惠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen University
Original Assignee
Shenzhen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen University filed Critical Shenzhen University
Priority to US16/491,544 priority Critical patent/US11403812B2/en
Publication of WO2020087564A1 publication Critical patent/WO2020087564A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
    • G06F18/2193—Validation; Performance evaluation; Active pattern learning techniques based on specific statistical tests
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/64—Three-dimensional [3D] objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present invention relates to the technical field of neural networks, and in particular, to a three-dimensional object reconstruction method, device, equipment, and storage medium.
  • a common three-dimensional model reconstruction method Without calibration, use a handheld scanning device to continuously scan around the object to obtain a scanned image of the object, and then register the image by depth or color information to reconstruct the three-dimensional model of the object .
  • the handheld scanning device in the process of acquiring scanned images, the handheld scanning device is not accurately tracked, and the noise in the data can easily lead to incorrect registration of the scan results, which ultimately leads to the three-dimensional model of the object
  • the reconstruction is not real.
  • a three-dimensional object reconstruction method characterized in that the method includes:
  • the sequence of scanned pictures includes at least one frame of scanned pictures, and the scanned pictures include depth information;
  • a neural network algorithm is used to predict the scanned pictures in the scanned picture sequence to obtain the predicted semantic labels of the scanned pictures
  • the three-dimensional model of the target object is reconstructed according to the predicted semantic tags and the scanned pictures in the scanned picture sequence.
  • the terminal obtains the scanned picture sequence in the scanned picture sequence by acquiring the scanned picture sequence of the target object and uses a neural network algorithm to obtain the predicted semantic label of each scanned picture, based on the predicted semantic label and the scanned picture sequence Scan the picture in to reconstruct the three-dimensional model of the target object.
  • the terminal uses a neural network algorithm to obtain the predicted semantic label of the scanned image, and reconstructs the three-dimensional model of the target object based on the predicted semantic label and the scanned image, which makes the reconstructed three-dimensional model more accurate and improves the accuracy of the reconstructed three-dimensional model.
  • a computer device in a second aspect, includes a memory and a processor.
  • the memory stores a computer program.
  • the processor implements the computer program to implement the following steps:
  • the sequence of scanned pictures includes at least one frame of scanned pictures, and the scanned pictures include depth information;
  • a neural network algorithm is used to predict the scanned pictures in the scanned picture sequence to obtain the predicted semantic labels of the scanned pictures
  • the three-dimensional model of the target object is reconstructed according to the predicted semantic tags and the scanned pictures in the scanned picture sequence.
  • a computer-readable storage medium has stored thereon a computer program, which when executed by a processor implements the following steps:
  • the sequence of scanned pictures includes at least one frame of scanned pictures, and the scanned pictures include depth information;
  • a neural network algorithm is used to predict the scanned pictures in the scanned picture sequence to obtain the predicted semantic labels of the scanned pictures
  • the three-dimensional model of the target object is reconstructed according to the predicted semantic tags and the scanned pictures in the scanned picture sequence.
  • the terminal terminal obtains the predicted semantic labels of each scanned picture by acquiring the scanned picture sequence of the target object and using a neural network algorithm to predict the scanned picture in the scanned picture sequence. Reconstruct the three-dimensional model of the target object according to the predicted semantic tags and the scanned pictures in the scanned picture sequence.
  • the terminal uses a neural network algorithm to obtain the predicted semantic label of the scanned image, and reconstructs the three-dimensional model of the target object based on the predicted semantic label and the scanned image, which makes the reconstructed three-dimensional model more accurate and improves the accuracy of the reconstructed three-dimensional model.
  • FIG. 1 is a schematic diagram of a three-dimensional object reconstruction method provided by an embodiment
  • FIG. 2 is a schematic flowchart of a three-dimensional object reconstruction method in an embodiment
  • FIG. 3 is a schematic flowchart of a three-dimensional object reconstruction method in another embodiment
  • 3a is a schematic diagram of three-dimensional object reconstruction in an embodiment
  • FIG. 4 is a schematic flowchart of a three-dimensional object reconstruction method in another embodiment
  • FIG. 4a is a comparison diagram of three-dimensional object reconstruction reality in an embodiment
  • FIG. 4b is a comparison diagram of the reality of three-dimensional object reconstruction in another embodiment
  • FIG. 4c is a comparison diagram of the reconstruction reality of the three-dimensional object in another embodiment
  • FIG. 5 is a schematic flowchart of a three-dimensional object reconstruction method in another embodiment
  • FIG. 6 is a schematic flowchart of a three-dimensional object reconstruction method in another embodiment
  • FIG. 7 is a schematic flowchart of a three-dimensional object reconstruction method in another embodiment
  • FIG. 7a is a schematic diagram of a three-dimensional object reconstruction method in another embodiment
  • FIG. 8 is a schematic diagram of a three-dimensional object reconstruction method provided by an embodiment
  • FIG. 9 is an internal structure diagram of a computing junction device provided by an embodiment.
  • a common three-dimensional object reconstruction method Without calibration, a handheld scanning device is used to continuously scan around the object to obtain a scanned image of the object, and then a neural network algorithm is used to reconstruct the scanned object to obtain a three-dimensional image of the object .
  • the handheld scanning device is not accurately tracked, and the noise in the data can easily lead to incorrect registration of the scan results, which ultimately leads to the three-dimensional scanning of the object The image is not real.
  • a three-dimensional object reconstruction method, device, equipment and storage medium provided by the present application are intended to solve the problem of unreality of three-dimensional object reconstruction.
  • the three-dimensional object reconstruction method provided by this application can be applied in the application environment shown in FIG. 1.
  • the terminal 102 communicates with the server 104 via the network.
  • the terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices.
  • the server 104 may be implemented by an independent server or a server cluster composed of multiple servers.
  • the three-dimensional object reconstruction method provided by the present invention can be applied in the process of reconstruction of various scenes, for example, it can be applied to human-computer interaction; virtual reality; unmanned driving; path planning; navigation and positioning, etc. In the process, the embodiments of the present application do not limit this.
  • the three-dimensional object reconstruction method provided in this embodiment can be applied to a terminal.
  • the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, or a personal digital assistant and other electronic devices with data processing functions.
  • the form is not limited.
  • the execution subject may be a three-dimensional object reconstruction device, which may be implemented as part or all of a terminal for three-dimensional object reconstruction through software, hardware, or a combination of software and hardware .
  • FIG. 2 is a schematic flowchart of a three-dimensional object reconstruction method in an embodiment. This embodiment relates to a specific process of acquiring a three-dimensional image according to the semantic labels of parts. As shown in FIG. 2, the method includes the following steps:
  • the scan picture sequence includes at least one frame of scan pictures, and the scan pictures include depth information.
  • the depth information may refer to the distance of the object corresponding to each pixel from the camera.
  • Scanned pictures generally contain depth information and color information, but can be stored in different ways, which is not limited in the embodiments of the present application.
  • the scan picture sequence can scan the same object and sort it according to the scan time, or it can be a sequence of scan pictures obtained by sorting different angles of the same object.
  • This embodiment of the present application No restrictions.
  • the scanning device may continuously scan the target object to obtain the scanning picture sequence, or the scanning device may obtain the scanning picture sequence obtained by scanning the target object at multiple fixed angles. There are no restrictions.
  • a neural network algorithm is used to predict the scanned pictures in the scanned picture sequence to obtain predicted semantic tags of the scanned pictures.
  • the semantic label may be an identifier used to represent the physical meaning of the actual object in the scanned picture, which may refer to an identifier of what object the picture is, or it may refer to an identifier of a certain part of the object.
  • the semantic label may be the names of objects such as "chair back”, “armrest”, “blade” and so on.
  • the predicted semantic label may refer to a prediction result of the scanned picture, which may indicate an identifier of an object corresponding to the scanned picture, or may refer to a probability that the scanned picture is a specific object, which is not limited in the embodiments of the present application.
  • a neural network algorithm may be used to predict the scanned pictures in the scanned picture sequence to obtain the predicted semantic tags of the scanned pictures.
  • a stochastic gradient descent back propagation algorithm is used to train the neural network.
  • the input of the network is a set of RGBD images and corresponding segmented images with semantic labels of components.
  • the amount of training data is very small.
  • the network is pre-trained to find a set of network parameters W s that minimizes the object classification loss according to the following formula
  • Loss represents the loss function
  • N refers to the number of training images
  • I i , W s ) refers to the probability that the image I i is marked as the real object label y i .
  • the labeled RGBD image obtained in this embodiment fine-tunes the network parameters W s .
  • the goal of fine-tuning is to minimize the classification loss at the pixel level.
  • the loss function is defined as follows:
  • the prediction semantic label is obtained by minimizing the sum of pixel errors of all pixels of all training images.
  • the present invention uses the learned network parameters to calculate the label probability of the pixel, which is a form of predicting the semantic label, as shown in the following formula:
  • K refers to the number of possible labels
  • m k refers to the kth label.
  • the reconstruction may be three-dimensional digital processing representing a real object, which may reconstruct a three-dimensional model of the target object by predicting the semantic tags and the scanned pictures in the scanned picture sequence.
  • the scanned pictures in the scanned picture sequence can be mapped to the predicted positions in the predicted semantic tags according to the predicted semantic tags, Obtain the three-dimensional image of the target object; you can also map the scan pictures in the scan sequence on the corresponding grid of the three-dimensional preset model according to the three-dimensional preset model composed of multiple grids and the predicted semantic label to reconstruct the target object.
  • Three-dimensional model The embodiments of the present application do not limit this.
  • the terminal obtains the scanned picture sequence in the scanned picture sequence by acquiring the scanned picture sequence of the target object and uses a neural network algorithm to obtain the predicted semantic label of each scanned picture, based on the predicted semantic label and the scanned picture sequence Scan the picture in to reconstruct the three-dimensional model of the target object.
  • the terminal uses a neural network algorithm to obtain the predicted semantic label of the scanned image, and reconstructs the three-dimensional model of the target object based on the predicted semantic label and the scanned image, making the reconstructed three-dimensional model more accurate and improving the accuracy of the reconstructed three-dimensional model .
  • the projected label can be reconstructed from the three-dimensional object, and then the predicted semantic label can be updated through the confidence between the projected label and the predicted semantic label to obtain a more accurate three-dimensional object reconstruction.
  • the predicted semantic label can be updated through the confidence between the projected label and the predicted semantic label to obtain a more accurate three-dimensional object reconstruction.
  • FIG. 3 is a schematic flowchart of a three-dimensional object reconstruction method in another embodiment. This embodiment relates to the specific process of updating the predicted semantic label according to the projection label. As shown in FIG. 3, the method further includes the following steps:
  • the projection label may be a scanned image corresponding to the predicted semantic label in the obtained three-dimensional model, the three-dimensional image is projected onto the frame image where the original image is located, and the obtained image identifier.
  • a picture generated by projecting the obtained three-dimensional model, and identifying the picture is a projection label.
  • the component of the obtained three-dimensional image can be identified, and then the projected picture after the identification can be obtained by projection, and the carried label is the projection label; the three-dimensional picture can also be directly Project, obtain a projected picture, identify the projected picture, and obtain a projected label; this embodiment of the present application does not limit this.
  • the confidence can be used to indicate the consistency between the predicted semantic label and the projection label. Since the projection label is obtained based on the projection of the obtained three-dimensional image, the projection label can identify the target object more realistically, according to the projection By comparing the label with the predicted semantic label, the confidence of the predicted semantic label can be obtained. It can be obtained by comparing the projection tags with the pictures corresponding to the predicted semantic tags to obtain the similarity result, which is the confidence of the predicted semantic tags; or by dividing the pictures corresponding to the projection tags and the predicted semantic tags into For multiple pixel grids, obtaining a consistent ratio of each pixel grid is the confidence level for predicting semantic labels. The embodiments of the present application do not limit this.
  • the predicted semantic label may be updated according to the confidence. It may be that the predicted semantic labels are divided into high-confidence and low-confidence predicted semantic labels according to the confidence of the predicted semantic labels, and then self-learning is performed on the scanned pictures corresponding to the high-confidence predicted semantic labels to obtain more predicted semantics Scanned pictures of tags, re-labeling the scanned pictures corresponding to the predicted semantic tags with low confidence to obtain updated predicted semantic tags; since the scanned pictures in the sequence of scanned pictures are continuous, between the scanned pictures of adjacent frames The difference is very small, therefore, it is also possible to obtain a high-confidence sequence based on the confidence of the predicted semantic label, extend the self-learning training set according to the high-confidence sequence, to update the predicted semantic label, and to scan the scanned image of the high-confidence prediction semantic label The corresponding sequence is set to a high-confidence sequence, and a part of the scanned image is selected from the high-confidence sequence to expand the self-
  • the predicted semantic label may be to re-identify the original predicted semantic label to obtain the updated predicted semantic label, or to increase or decrease the existing predicted semantic label to obtain a new predicted semantic
  • the label set is the updated predicted semantic label, which is not limited in the embodiments of the present application
  • a scanned image sequence A which includes multiple scanned images, respectively a1, a2, a3, a4 ... an
  • the predicted semantic label of the scanned image a3 is a high-confidence predicted semantic label
  • the predicted semantic label of a3 is a high-confidence prediction semantic label. It is determined that the scanned image sequence A is a high-confidence sequence, and the sequence A is evenly sampled.
  • One frame is extracted every 25 frames as self-learning training data. After self-learning, Obtaining more predicted semantic tags is the updated predicted semantic tags.
  • the terminal obtains the projection labels of each scanned image in the scanned image sequence according to the three-dimensional model, and then obtains the confidence of the predicted semantic label based on the projection label and the predicted semantic label, and updates it according to the confidence of the predicted semantic label Predict semantic tags.
  • the terminal obtains the confidence of the predicted semantic label according to the projection label and the predicted semantic label, and updates the predicted semantic label according to the confidence of the predicted semantic label, so that the updated predicted semantic label is more accurate, which in turn makes the prediction based on the prediction
  • the 3D images obtained by semantic tags and scanned pictures are more accurate, which further improves the authenticity of the 3D model of the reconstructed target object.
  • S203 "update the predicted semantic label according to the confidence of the predicted semantic label” includes:
  • the confidence of the predicted semantic label is obtained, and if the confidence of the predicted semantic label is greater than a preset threshold, the predicted semantic label is set as a high-confidence label.
  • a neural network algorithm may be used, which may be a self-learning algorithm to perform machine learning on the high-confidence label and the corresponding scanned image to obtain the updated predicted semantic label.
  • the predicted semantic label is set to a low confidence label.
  • an active learning method is used to re-label the scanned image corresponding to the low-confidence label to obtain the updated predicted semantic label.
  • the terminal sets the predicted semantic label whose predicted semantic label has a confidence greater than a preset threshold as a high-confidence label, and uses a neural network algorithm to scan the sequence of scanned images based on the high-confidence label and the corresponding scanned image.
  • the scanned image in is used for prediction, the updated predicted semantic label is obtained, and the predicted semantic label whose predicted semantic label's confidence is not greater than the preset threshold is marked as a low-confidence label, and the method of active learning is used to reduce the low-confidence level.
  • the scanned image corresponding to the label is relabeled to obtain the updated predicted semantic label.
  • the terminal obtains a more accurate updated predicted semantic label according to the confidence of the predicted semantic label, so as to reconstruct a three-dimensional model of the target object with higher authenticity based on the predicted semantic label and the scanned image with higher confidence.
  • the terminal can use the neural network learning method to predict the high-confidence label and the corresponding scanned image, and predict the scanned image in the scanned image sequence to obtain the updated predicted semantic label, or it can use the active learning method. , Re-label the scanned image corresponding to the low-confidence label to obtain the updated predicted semantic label. You can also use the above two methods to update the predicted semantic label at the same time. This application does not limit this.
  • FIG. 5 is a schematic flowchart of a three-dimensional object reconstruction method in another embodiment. This embodiment relates to the specific process of how the terminal obtains the confidence of the label by projecting the label and predicting the semantic label.
  • S202 "Acquiring the confidence of the predicted semantic label based on the projection label and the prediction semantic label"
  • a possible implementation method includes:
  • pixel consistency may indicate the similarity between pictures, which may be the percentage of pixels in the picture that are consistent among all pixels in the picture.
  • the scanned image corresponding to the projection label can be compared with the pixel label in the scanned image of the corresponding predicted semantic label to obtain the percentage of pixels that are consistent in the two scanned images, which is Pixel consistency.
  • the confidence of the predicted semantic label is determined according to the pixel consistency. It may be that the pixel consistency of the projection label and the predicted semantic label is greater than the preset consistency threshold for high confidence labels, and the one that is not greater than the preset consistency threshold is low confidence labels.
  • the terminal obtains pixel consistency by comparing pixel tags in the scanned image corresponding to the projection tags and the predicted semantic tags, and determines the confidence of the predicted semantic tags.
  • the terminal determines the confidence of the predicted semantic label through the consistency of the pixels in the scanned image corresponding to the predicted label and the predicted semantic label, and then updates the predicted semantic label according to the confidence of the predicted semantic label, thereby according to the higher confidence Predictive semantic tags and scanned pictures to obtain a more realistic three-dimensional image of the target object.
  • the above embodiment describes in detail the detailed process in which the terminal reconstructs the projection label from the three-dimensional object, and then updates the predicted semantic label through the confidence between the projection label and the predicted semantic label to obtain a more accurate three-dimensional image.
  • the specific process of the terminal acquiring the three-dimensional image of the target object according to the predicted semantic label and the scanned picture will be described in detail below.
  • S103 "reconstruct the three-dimensional model of the target object based on the predicted semantic label and the scanned image in the scanned image sequence" includes: according to the label distribution and predicted semantic label corresponding to each voxel of the three-dimensional preset model, each scanned image Map to the corresponding position of the three-dimensional preset model to obtain a three-dimensional image.
  • a voxel may be an abbreviation of a volume element (Volume Pixel), and a volume containing a voxel may be represented by a stereo rendering or a polygon isosurface with a given threshold contour extracted, which may be digital data in a three-dimensional space The smallest unit on the split.
  • the three-dimensional preset model may represent the three-dimensional preset model according to the truncated symbol distance function, and the three-dimensional preset model may include multiple voxels.
  • the label corresponding to each voxel can be used to represent the coordinate position of the voxel, and map the scan picture corresponding to the predicted semantic label at the same coordinate position to the position of the voxel to reconstruct a three-dimensional object.
  • the three-dimensional preset model can also be updated to obtain the updated three-dimensional preset model, and then the three-dimensional model can be obtained according to the updated three-dimensional preset model.
  • the three-dimensional model can be obtained according to the updated three-dimensional preset model.
  • map each scanned picture at the corresponding position of the three-dimensional preset model to obtain a three-dimensional model includes:
  • a preset registration algorithm is used to update the three-dimensional preset model to obtain an updated three-dimensional preset model.
  • a plurality of frame pictures in the scanning picture sequence may be obtained by continuously scanning around the target object by the scanning device, and the object information included in the adjacent frame pictures is less different, which may be different angles of the same part Picture, therefore, after the previous picture is mapped to the three-dimensional preset model, the next picture is used to register the previous picture to obtain more accurate object parts, thereby obtaining a more accurate three-dimensional model.
  • TSDF Truncation Sign Distance Function
  • the intermediate iterative process includes registering the frame F t using the sensor pose estimation registration method to the existing model M t-1 , which is constructed from the frames ⁇ F 1 , ..., F t-1 ⁇ fused by the previous registration It should be pointed out that this embodiment uses this method, considering the semantic tags of the frame picture, the pixels of F t and the voxels of M t-1 have different semantic tags, and pixels and pixels with the same semantic tag The voxels are mapped to each other, and the transformation from F t registration to M t-1 is obtained. The detailed description is made by the following description.
  • the pixels in F t are grouped into a semantic set S i , and each possible partial label is i. In each set, only pixels with high confidence are kept. If the information entropy H (P) of a pixel's label probability distribution P is lower than the threshold ⁇ , the pixel is defined as a pixel with high confidence, and the information entropy is usually calculated as follows:
  • P i refers to the probability that a pixel has a label i.
  • the present invention divides the voxels of M t-1 into the semantic group S ′ i , and contains only voxels with high confidence.
  • ICP iterative closest point method
  • conf i refers to the average confidence predicted by the S i label and is given by the average H (P) of all pixels in S i ; size i is the percentage of pixels assigned with the S i label in F t and var i is the reconstruction
  • the reason why the normal change of preference is large is because pixels with small normal change often lead to indeterminate matching, for example: pixel matching on a flat desktop is not clear, because adding one before mapping any such pixel Both translations result in the same loss value.
  • the global transformation T * can be calculated by solving the following optimization problem:
  • T * minimizes the weighted distance and sum of all sets.
  • the Gauss-Newton method can be used to optimize this objective function, and the iterative method linearizes the objective function and solves the above equation.
  • T * and ⁇ T i ⁇ are constrained to rigid transformations consisting only of translation and rotation. Then, the present invention uses T * to align F t to M t-1 and combine the current frame with the model to generate a new model M t .
  • the updated three-dimensional preset model maps the scanned image corresponding to the predicted semantic label at the same coordinate position on the position of the voxel, and a more accurate three-dimensional model of the target object can be obtained.
  • the terminal uses the preset registration algorithm to update the three-dimensional preset model based on the frame sequence of the scanned picture sequence and the corresponding frame picture to obtain the updated three-dimensional preset model, and according to the update
  • the distribution of labels and predicted semantic labels corresponding to each voxel of the three-dimensional preset model are mapped to the corresponding positions of the three-dimensional preset model to obtain the three-dimensional model.
  • the terminal maps each scanned image to the corresponding position of the three-dimensional preset model through the correspondence between the label distribution corresponding to each voxel of the three-dimensional preset model and the predicted semantic label to obtain the three-dimensional model, so that the reconstructed Three-dimensional objects are more realistic.
  • the three-dimensional object reconstruction method of the above embodiment only considers semantic components in the registration process, but the background is also marked and stored in the reconstructed three-dimensional model grid.
  • the background consists of data that has nothing to do with the target object, for example, the wall behind the target object.
  • the terminal achieves this by deleting the background and keeping only the voxels of the target object.
  • simply deleting the voxel with the greatest background confidence may introduce some holes in the final 3D model.
  • the terminal deletes the background of the target object by deleting some voxels, and then obtains a more realistic three-dimensional model. The following is a detailed description through FIG. 7.
  • FIG. 7 is a schematic flowchart of a three-dimensional object reconstruction method in another embodiment. As shown in FIG. 7, the illustrated method further includes:
  • the super voxel may be a volume element obtained according to the set resolution parameter, which may change according to the change of the resolution.
  • the higher the resolution the smaller the volume of the super voxel and the lower the resolution.
  • the preset resolution parameter can be modified according to the preset iteration method.
  • the preset iteration method can be to set a low resolution value first, modify the resolution value according to the step of the preset resolution parameter, and obtain New resolution parameters.
  • S602 Delete the target voxel in the super voxel according to the resolution parameter; the target voxel includes: voxels whose proportion of non-background labels is less than the preset first threshold, and the proportion of background labels of adjacent voxels is greater than Voxels with a preset second threshold.
  • the first threshold and the second threshold may be set fixed values, or may be values obtained according to machine learning, which is not limited in the embodiment of the present application.
  • the resolution parameter is increased to split the results from previous iterations into smaller super-voxels.
  • the iteration process ends.
  • the three-dimensional image obtained after deleting the target voxel is shown in FIG. 7a.
  • the model is divided into very fine-grained super voxels, where each super voxel is associated with the average label distribution of the voxels contained within it. Based on these distributions, the present invention uses the graph cut method to smooth super voxel labels.
  • each super voxel is a node connected to its adjacent super voxel, and the data items used to label the node are based on the label distribution of the corresponding super voxel, by converting the probability value to
  • the energy cost (energy cost) follows the Potts model smoothly.
  • the terminal In the above three-dimensional object reconstruction method, the terminal generates a set of super voxels according to the preset resolution parameters and the preset iteration method, and deletes the target voxels in the super voxels; the target voxels include: the proportion of non-background labels is less than Voxels with a preset first threshold, and the proportion of background labels of adjacent voxels is greater than voxels with a preset second threshold.
  • the terminal deletes the background image in the three-dimensional model of the target object by deleting the target voxel in the super voxel, so that the obtained three-dimensional model of the target object is more realistic.
  • the terminal obtains a sequence of scanned pictures.
  • the scanned pictures in the scanning sequence may be RGBD pictures.
  • a neural network algorithm is used to obtain the predicted semantic tags of the scanned pictures. According to the predicted semantic tags and Scan the picture to reconstruct the three-dimensional model of the target object. Further, the corresponding projection label can be obtained according to the acquired three-dimensional model of the target object, and then the confidence between the projection label and the predicted semantic label can be obtained. According to the confidence, self-learning and active learning can be performed on the scanned image, respectively. Get a more accurate 3D model of the target object. It should be noted that FIG. 8 is only an example.
  • steps in the flowcharts of FIGS. 2-7 are displayed in order according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless clearly stated in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least some of the steps in FIGS. 2-7 may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but may be executed at different times. These sub-steps or stages The order of execution is not necessarily sequential.
  • the three-dimensional object reconstruction device includes: an acquisition module 10, a prediction module 20, and a reconstruction module 30, wherein:
  • An obtaining module configured to obtain a sequence of scanned pictures of a target object; the sequence of scanned pictures includes at least one frame of scanned pictures, and the scanned pictures include depth information;
  • a prediction module used for predicting the scanned pictures in the scanned picture sequence by using a neural network algorithm, and obtaining predicted semantic tags of the scanned pictures;
  • the reconstruction module is configured to reconstruct the three-dimensional model of the target object according to the predicted semantic label and the scan pictures in the scan picture sequence.
  • the three-dimensional object reconstruction device provided by the embodiment of the present invention can execute the above method embodiments, and its implementation principles and technical effects are similar, and will not be repeated here.
  • Each module in the above three-dimensional object reconstruction device may be implemented in whole or in part by software, hardware, or a combination thereof.
  • the above modules may be embedded in the hardware or independent of the processor in the computer device, or may be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
  • a computer device is provided.
  • the computer device may be a terminal, and an internal structure diagram thereof may be as shown in FIG.
  • the computer equipment includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus.
  • the processor of the computer device is used to provide computing and control capabilities.
  • the memory of the computer device includes a non-volatile storage medium and an internal memory.
  • the non-volatile storage medium stores an operating system and computer programs.
  • the internal memory provides an environment for the operating system and computer programs in the non-volatile storage medium.
  • the network interface of the computer device is used to communicate with external terminals through a network connection. When the computer device is executed by the processor to implement a three-dimensional object reconstruction method.
  • the display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen
  • the input device of the computer device may be a touch layer covered on the display screen, or may be a button, a trackball or a touchpad provided on the computer device housing , Can also be an external keyboard, touchpad or mouse.
  • FIG. 10 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied.
  • the specific computer equipment may It includes more or fewer components than shown in the figure, or some components are combined, or have a different component arrangement.
  • a computer device which includes a memory and a processor, and a computer program is stored in the memory, and the processor implements the following steps when the processor executes the computer program:
  • the sequence of scanned pictures includes at least one frame of scanned pictures, and the scanned pictures include depth information;
  • a neural network algorithm is used to predict the scanned pictures in the scanned picture sequence to obtain the predicted semantic labels of the scanned pictures
  • the three-dimensional model of the target object is reconstructed according to the predicted semantic tags and the scanned pictures in the scanned picture sequence.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are realized:
  • the sequence of scanned pictures includes at least one frame of scanned pictures, and the scanned pictures include depth information;
  • a neural network algorithm is used to predict the scanned pictures in the scanned picture sequence to obtain the predicted semantic labels of the scanned pictures
  • the three-dimensional model of the target object is reconstructed according to the predicted semantic tags and the scanned pictures in the scanned picture sequence.
  • the computer-readable storage medium provided in this embodiment has similar implementation principles and technical effects to the foregoing method embodiments, and will not be repeated here.
  • Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory can include random access memory (RAM) or external cache memory.
  • RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain (Synchl) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
  • SRAM static RAM
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDRSDRAM double data rate SDRAM
  • ESDRAM enhanced SDRAM
  • SLDRAM synchronous chain (Synchl) DRAM
  • SLDRAM synchronous chain (Synchl) DRAM
  • Rambus direct RAM
  • DRAM direct memory bus dynamic RAM
  • RDRAM memory bus dynamic RAM

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Data Mining & Analysis (AREA)
  • Probability & Statistics with Applications (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

一种三维物体重建方法、设备和存储介质。终端通过获取目标物体的扫描图片序列，其中，扫描图片序列包括至少一帧扫描图片，扫描图片包括深度信息（S101），并采用神经网络算法，对扫描图片序列中的扫描图片进行预测，获取各扫描图片的预测语义标签（S102），进而根据预测语义标签和扫描图片序列中的扫描图片，重建目标物体的三维模型（S103）。终端采用神经网络算法，获取扫描图片的预测语义标签，根据预测语义标签和扫描图片重建目标物体的三维模型，使得重建的三维模型更加准确，提高了重建的三维模型的准确度。

Description

三维物体重建方法、计算机设备及存储介质
相关申请的交叉引用
本发明要求于2018年10月31日提交中国专利局，申请号为2018112885377，申请名称为“三维物体重建方法、装置、设备和存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本发明中。
技术领域
本发明涉及神经网络的技术领域，特别是涉及了一种三维物体重建方法、装置、设备和存储介质。
背景技术
近年来，扫描技术取得了长足的进步。但是，三维场景获取仍然是一项具有挑战性的工作，尤其是创建大量高质量模型这一领域。
一种常见的三维模型重建方法，在没有校准的前提下，使用手持扫描设备围绕着物体连续扫描，获取物体的扫描图片，进而通过深度或者颜色信息对图片进行配准，重建该物体的三维模型。
然而，采用上述方法，在获取扫描图片的过程中，手持的扫描设备并没有被精确的追踪，并且数据中带有的噪声，很容易导致扫描结果的错误配准，最终导致该物体的三维模型重建不真实。
发明内容
基于此，有必要针对物体的三维模型重建不真实的问题，提供一种三维物体重建方法、装置、设备和存储介质。
第一方面，一种三维物体重建方法，其特征在于，所述方法包括：
获取目标物体的扫描图片序列；所述扫描图片序列包括至少一帧扫描图片，所述扫描图片包括深度信息；
采用神经网络算法，对所述扫描图片序列中的扫描图片进行预测，获取各所述扫描图片的预测语义标签；
根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型。
上述三维物体重建方法，终端通过获取目标物体的扫描图片序列，并采用神经网络算法，对扫描图片序列中的扫描图片进行预测，获取各扫描图片的预测语义标签，根据预测语义标签和扫描图片序列中的扫描图片，重建目标物体的三维模型。本实施例中，终端采 用神经网络算法，获取扫描图片的预测语义标签，根据预测语义标签和扫描图片重建目标物体的三维模型，使得重建的三维模型更加准确，提高了重建的三维模型的准确度。
第二方面，一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，所述处理器执行所述计算机程序时实现以下步骤：
获取目标物体的扫描图片序列；所述扫描图片序列包括至少一帧扫描图片，所述扫描图片包括深度信息；
采用神经网络算法，对所述扫描图片序列中的扫描图片进行预测，获取各所述扫描图片的预测语义标签；
根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型。
第三方面，一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现以下步骤：
获取目标物体的扫描图片序列；所述扫描图片序列包括至少一帧扫描图片，所述扫描图片包括深度信息；
采用神经网络算法，对所述扫描图片序列中的扫描图片进行预测，获取各所述扫描图片的预测语义标签；
根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型。
上述三维物体重建方法、装置、设备和存储介质，终端终端通过获取目标物体的扫描图片序列，并采用神经网络算法，对扫描图片序列中的扫描图片进行预测，获取各扫描图片的预测语义标签，根据预测语义标签和扫描图片序列中的扫描图片，重建目标物体的三维模型。本实施例中，终端采用神经网络算法，获取扫描图片的预测语义标签，根据预测语义标签和扫描图片重建目标物体的三维模型，使得重建的三维模型更加准确，提高了重建的三维模型的准确度。
附图说明
图1为一个实施例提供的三维物体重建方法的示意图；
图2为一个实施例中三维物体重建方法的流程示意图；
图3为另一个实施例中三维物体重建方法的流程示意图；
图3a为一个实施例中三维物体重建的示意图；
图4为另一个实施例中三维物体重建方法的流程示意图；
图4a为一个实施例中三维物体重建真实度的对比图；
图4b为另一个实施例中三维物体重建真实度的对比图；
图4c为另一个实施例中三维物体重建真实度的对比图；
图5为另一个实施例中三维物体重建方法的流程示意图；
图6为另一个实施例中三维物体重建方法的流程示意图；
图7为另一个实施例中三维物体重建方法的流程示意图；
图7a为另一个实施例中三维物体重建方法的示意图；
图8为一个实施例提供的三维物体重建方法的示意图；
图9为一个实施例提供的计算结设备的内部结构图。
具体实施例方式
近年来，扫描技术取得了长足的进步。但是，三维场景获取仍然是一项具有挑战性的工作，尤其是创建大量高质量模型这一领域。一种常见的三维物体重建方法，在没有校准的前提下，使用手持扫描设备围绕着物体连续扫描，获取物体的扫描图片，进而通过神经网络算法对扫描物体进行图片重建，获取该物体的三维图像。然而，采用上述方法，在获取扫描图片的过程中，手持的扫描设备并没有被精确的追踪，并且数据中带有的噪声，很容易导致扫描结果的错误配准，最终导致该物体的三维扫描图像不真实。本申请提供的一种三维物体重建方法、装置、设备和存储介质，旨在解决三维物体重建不真实的问题。
本申请提供的三维物体重建方法，可以应用于如图1所示的应用环境中。其中，终端102通过网络与服务器104通过网络进行通信。其中，终端102可以但不限于是各种个人计算机、笔记本电脑、智能手机、平板电脑和便携式可穿戴设备，服务器104可以用独立的服务器或者是多个服务器组成的服务器集群来实现。
本发明提供的三维物体重建方法，可以应用于多种场景重建的过程中，例如，其可以应用于人机交互；虚拟现实；无人驾驶；路径规划；导航定位等中需用进行场景重建的过程中，本申请实施例对此不做限制。
本实施例提供的三维物体重建方法，可以适用于终端中，终端可以为智能手机、平板电脑、笔记本电脑、台式电脑或个人数字助理等具有数据处理功能的电子设备，本实施例对终端的具体形式不做限定。
需要说明的是，本发明实施例提供的三维物体重建方法，其执行主体可以是三维物体重建装置，该装置可以通过软件、硬件或者软硬件结合的方式实现成为三维物体重建的终端的部分或者全部。
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
图2为一个实施例中三维物体重建方法的流程示意图。本实施例涉及的是根据部件语义标签获取三维图像的具体过程，如图2所示，该方法包括以下步骤：
S101、获取目标物体的扫描图片序列，扫描图片序列包括至少一帧扫描图片，扫描图片包括深度信息。
具体的，深度信息可以是指每个像素点对应的物体距离相机的距离。扫描图片一般均包含深度信息和色彩信息，但是可以不同的方式存储，本申请实施例对此不做限制。扫描图片序列可以对同一个物体进行扫描，并按照扫描时间进行排序获得的扫描图片序列，也可以是根据对同一个物体扫描的不同角度进行排序获得的扫描图片的序列，本申请实施例对此不做限制。在具体获取目标物体的扫描图片序列的过程中，可以通过扫描设备连续扫描目标物体获得扫描图片序列，也可以扫描设备在多个固定的角度扫描目标物体获得的扫描图片序列，本申请实施例对此不做限制。
S102、采用神经网络算法，对扫描图片序列中的扫描图片进行预测，获取各扫描图片的预测语义标签。
具体的，语义标签可以是用来表示扫描图片中的实际物体的物理意义的标识，其可以是指图片是什么物体的标识，也可以是指图片是物体的某一部分的标识，本申请实施例对此不做限制。例如，语义标签可以是“椅背”、“扶手”、“叶片”等物体名称。在具体获取扫描图片序列中的部分扫描图片对应的语义标签的过程中，可以通过人工对部分扫描图片添加部件语义标签；也可以通过机器学习的算法对扫描图片进行学习，获取扫描图片的物理意义，以获得扫描图片的语义标签；还可以是通过人工对少量扫描图片进行物理意义的标注，采用机器学习的方法对标注后的扫描图片和部分扫描图片进行机器学习，获取部分扫描图片的部件语义标签，本申请实施例对此不做限制。
进一步地，预测语义标签可以是指对扫描图片预测的结果，其可以指示扫描图片对应物体的标识，也可以是指扫描图片是具体物体的概率，本申请实施例对此不做限制。在具体的获取各扫描图片的预测语义标签的过程中，可以采用神经网络算法，对扫描图片序列中的扫描图片进行预测，以获取扫描图片的预测语义标签。
示例性的，以采用VGG-16的架构对神经网络算法进行建模为例，使用随机梯度下降的反向传播算法来训练神经网络。在训练阶段，网络的输入是一组RGBD图像和与之对应的带有部件语义标签的分割图像。在第一次迭代中，训练数据量非常小。本实施例预先训练网络以下述公式找到使物体分类损失最小化的一组网络参数W s
Figure PCTCN2018114808-appb-000001
其中，Loss表示损失函数，N指训练图像的数目，P(y i|I i,W s)指图像I i被标记为真实物体标签y i的概率。在预训练之后，本实施例获得的带有标注的RGBD图像微调网络参数W s。微调的目标是最小化像素层面的分类损失，损失函数的定义如下：
Figure PCTCN2018114808-appb-000002
其中，指图像I i输入图像，m ij指图像I i中像素P ij的真实标签，|I i|指图像图像I i中 像素的数目，P(m ij|I i,P ij,W s)指图像I i的第j个像素被标记为真实部件标签m ij的概率。本实施例通过最小化所有训练图像的所有像素的像素误差之和来获取预测语义标签。
在测试阶段，给定图像I'的一个像素p，本发明使用学得的网络参数来计算像素的标签概率，即为预测语义标签的一种表现形式，如下式所示：
P(m k|p)＝P(m k|I′,p,W s),for all k∈{1,...,K}
其中，K指可能的标签的数目，m k指第k个标签。本实施例可以通过上述步骤，获取各扫描图片的预测语义标签。
S103、根据预测语义标签和扫描图片序列中的扫描图片，重建目标物体的三维模型。
具体的，重建可以是表示真实物体的三维数字化处理，其可以通过预测语义标签和扫描图片序列中的扫描图片，来重建目标物体的三维模型。在具体的通过预测语义标签和扫描图片序列中的扫描图片来重建目标物体的三维模型的过程中，可以根据预测语义标签将扫描图片序列中的扫描图片映射到预测语义标签中预测的位置上，获取目标物体的三维图像；也可以根据由多个网格组成的三维预设模型，根据预测语义标签将扫描序列中的扫描图片映射在三维预设模型的对应的网格上，重建目标物体的三维模型。本申请实施例对此不做限制。
上述三维物体重建方法，终端通过获取目标物体的扫描图片序列，并采用神经网络算法，对扫描图片序列中的扫描图片进行预测，获取各扫描图片的预测语义标签，根据预测语义标签和扫描图片序列中的扫描图片，重建目标物体的三维模型。本实施例中，终端采用神经网络算法，获取扫描图片的预测语义标签，根据预测语义标签和扫描图片重建目标物体的三维模型，使得重建的三维模型更加准确，提高了重建的三维模型的准确度。
在上述实施例的基础上，进一步地，可以通过三维物体重建投影标签，进而通过投影标签与预测语义标签之间的置信度，更新预测语义标签，以获得更加准确的三维物体重建。下面通过图3-6来详细说明。
图3为另一个实施例中三维物体重建方法的流程示意图。本实施例涉及的是根据投影标签更新预测语义标签的具体过程，如图3所示，所述方法还包括以下步骤：
S201、根据三维模型，获取扫描图片序列中各扫描图片的投影标签。
具体的，投影标签可以是已获得的三维模型中预测语义标签的对应的扫描图片，将三维图像投影至原图片所在的帧图片上，获得的图片的标识。如图3a所示，对已获得的三维模型投影生成的图片，对该图片进行标识即为投影标签。
在具体的获取投影标签的过程中，可以是通过已获得的三维图像的部件进行标识，再通过投影，获得标识后的投影图片，其中携带的标识即为投影标签；也可以直接将三维图片进行投影，获得投影图片，对投影图片进行标识，获得投影标签；本申请实施例对此不做限制。
S202、根据投影标签和预测语义标签，获取预测语义标签的置信度。
具体的，置信度可以用来表示预测语义标签与投影标签之间的一致性，由于投影标签是根据已获得的三维图像的投影得到的，因此，投影标签能更加真实的标识目标物体，根据投影标签与预测语义标签之间进行比较，可以获得预测语义标签的置信度。其可以是通过将投影标签与预测语义标签对应的图片进行之间的对比，获得相似度结果，即为预测语义标签的置信度；也可以是通过将投影标签和预测语义标签对应的图片划分为多个像素格，获得各像素格一致的比列，即为预测语义标签的置信度。本申请实施例对此不做限制。
S203、根据预测语义标签的置信度，更新预测语义标签。
具体的，在上述S202的基础上，在获取了预测语义标签的置信度之后，可以根据置信度，更新预测语义标签。其可以是根据预测语义标签的置信度将预测语义标签划分为高置信度和低置信度的预测语义标签，进而对高置信度的预测语义标签对应的扫描图片进行自学习，获取更多预测语义标签的扫描图片，对低置信度的预测语义标签对应的扫描图片进行重新标签，以获得更新的预测语义标签；由于扫描图片序列中的扫描图片是连续的，相邻帧的扫描图片之间的差异非常小，因此，还可以是根据预测语义标签的置信度获取高置信度序列，根据高置信度序列扩展自学习的训练集，以更新预测语义标签，将高置信度预测语义标签的扫描图片对应的序列设置为高置信度序列，通过高置信度序列中选择一部分扫描图片扩展自学习的训练集，进而获得更多的预测语义标签；本申请实施例对此不做限制。
在具体的更新预测语义标签的过程中，其可以是将原有的预测语义标签进行重新标识，获取更新的预测语义标签，也可以增加或减少已有的预测语义标签，以获得新的预测语义标签集合，即为更新预测语义标签，本申请实施例对此不做限制
例如，在一个扫描图片序列A中，其中包括多个扫描图片，分别a1、a2、a3、a4…an，其中，扫描图片a3的预测语义标签为高置信度预测语义标签，则可以根据扫描图片a3的预测语义标签为高置信度预测语义标签，确定扫描图片序列A为高置信度序列，并对序列A进行均匀抽样，每25帧抽取1帧，作为自学习的训练数据，经过自学习，获取更多预测语义标签，即为更新的预测语义标签。
上述三维物体重建方法，终端根据三维模型，获取扫描图片序列中各扫描图片的投影标签，进而根据投影标签和预测语义标签，获取预测语义标签的置信度，并根据预测语义标签的置信度，更新预测语义标签。本实施例中，终端根据投影标签和预测语义标签，获取预测语义标签的置信度，并根据预测语义标签的置信度，更新预测语义标签，使得更新后的预测语义标签更加准确，进而使得根据预测语义标签和扫描图片获取的三维图像更加准确，进一步地提高了重建目标物体的三维模型的真实度。
可选地，如图4所示，S203“根据预测语义标签的置信度，更新预测语义标签”包括：
S301、判断预测语义标签的置信度与预设阈值之间的大小，若预测语义标签的置信度大于预设阈值，则执行S302；若预测语义标签的置信度不大于预设阈值，则执行S303。
S302、将预测语义标签设置为高置信度标签，采用神经网络算法，根据高置信度标签和对应的扫描图片，对扫描图片序列中的扫描图片进行预测，获取更新后的预测语义标签。
具体的，在上述S202的基础上，获得预测语义标签的置信度，若预测语义标签的置信度大于预设阈值，则将预测语义标签设置为高置信度标签。在确定了高置信度标签后，采用神经网络算法，可以是采用自学习的算法，对高置信度标签和对应的扫描图片进行机器学习，获取更新后的预测语义标签。
S303、将预测语义标签标记为低置信度标签，采用主动学习的方法，对低置信度标签对应的扫描图片重新标注，获取更新后的预测语义标签。
继续在在上述S202的基础上，获得预测语义标签的置信度，若预测语义标签的置信度不大于预设阈值，则将预测语义标签设置为低置信度标签。在确定了低置信度标签后，采用主动学习的方法，对低置信度标签对应的扫描图片重新标注，获取更新后的预测语义标签。
上述三维物体重建方法，终端将预测语义标签的置信度大于预设阈值的预测语义标签设置为高置信度标签，并采用神经网络算法，根据高置信度标签和对应的扫描图片，对扫描图片序列中的扫描图片进行预测，获取更新后的预测语义标签，并将预测语义标签的置信度不大于预设阈值的预测语义标签标记为低置信度标签，并采用主动学习的方法，对低置信度标签对应的扫描图片重新标注，获取更新后的预测语义标签。本实施例中，终端根据预测语义标签的置信度，获取更加准确的更新预测语义标签，从而根据更高置信度的预 测语义标签和扫描图片，重建更高真实度的目标物体的三维模型。
需要说明的是，终端可以采用神经网络学习的方法对高置信度标签和对应的扫描图片，对扫描图片序列中的扫描图片进行预测，获取更新后的预测语义标签，也可以采用主动学习的方法，对低置信度标签对应的扫描图片重新标注，获取更新后的预测语义标签，还可以同时采用上述两种方法来更新预测语义标签，本申请对此不做限制。
在具体的通过预测语义标签的置信度，更新预测语义标签后，得到更加准确的预测语义标签。如图4a、图4b和图4c所示，通过对高置信度标签进行自学习，获得更新的预测语义标签，及对低置信度标签进行主动学习，获得更新的预测语义标签；通过这两种方法，对目标物体，包括桌子、椅子及摩托车获得的预测语义标签的准确度明显高于传统的主动学习的方法获得预测语义标签的准确度。
在具体根据投影标签和预测语义标签，获取预测语义标签的置信度的过程中，可以通过图5所示的方法获取。图5为另一个实施例中三维物体重建方法的流程示意图。本实施例涉及的是终端如何通过投影标签和预测语义标签，获取标签的置信度的具体过程，如图5所示，S202“根据投影标签和预测语义标签，获取预测语义标签的置信度”的一种可能的实现方法包括：
S401、对投影标签与预测语义标签对应的扫描图片中的像素进行对比，获取像素一致性。
具体的，像素一致性可以表示图片之间的相似度，其可以是图片内一致的像素占图片内所有像素的百分比。在具体获取像素一致性的过程中，可以是将投影标签对应的扫描图片，与对应的预测语义标签的扫描图片中的像素标签进行对比，获取两种扫描图片中一致的像素的百分比，即为像素一致性。
S402、根据像素一致性，确定预测语义标签的置信度。
具体的，在上述实施例的基础上，在对投影标签与预测语义标签对应的扫描图片中的像素标签进行对比，获取像素一致性后，根据该像素一致性，确定预测语义标签的置信度。其可以是投影标签与预测语义标签像素一致性大于预设的一致性阈值的为高置信度标签，不大于预设的一致性阈值的为低置信度标签。
上述三维物体重建方法，终端通过对投影标签与预测语义标签对应的扫描图片中的像素标签进行对比，获取像素一致性，确定预测语义标签的置信度。本实施例中，终端通过投影标签与预测语义标签对应的扫描图片中的像素一致性，确定预测语义标签的置信度，进而根据预测语义标签的置信度更新预测语义标签，从而根据更高置信度的预测语义标签和扫描图片，获得更高真实度的目标物体的三维图像。
上述实施例详细描述了终端通过三维物体重建投影标签，进而通过投影标签与预测语义标签之间的置信度，更新预测语义标签，以获得更加准确的三维图像的详细过程。下文将详细描述终端根据预测语义标签和扫描图片获取目标物体的三维图像的具体过程。
可选地，S103“根据预测语义标签和扫描图片序列中的扫描图片，重建目标物体的三维模型”包括：根据三维预设模型的各体素对应的标签分布和预测语义标签，将各扫描图片映射在三维预设模型的对应位置，得到三维图像。
具体的，体素可以是体素是体积元素(Volume Pixel)的简称，包含体素的立体可以通过立体渲染或者提取给定阈值轮廓的多边形等值面表现出来，其可以是数字数据在三维空间分割上的最小单位。三维预设模型中可以根据截断符号距离函数来表示三维预设模型，该三维预设模型中可以包括多个体素。各体素对应的标签，其可以用来表示该体素的坐标位置，将相同坐标位置的预测语义标签对应的扫描图片映射在该体素的位置上，以重建三维物体。
进一步地，还可以对三维预设模型更新，获得更新的三维预设模型，进而根据更新的三维预设模型，得到三维模型。可选地，如图6所示，“根据预设的三维预设模型的各体 素对应的标签分布和预测语义标签，将各扫描图片映射在三维预设模型的对应位置，得到三维模型”一种可能的实现方式包括：
S501、根据扫描图片序列的帧顺序和对应的帧图片，采用预设的配准算法，对三维预设模型进行更新，得到更新后的三维预设模型。
具体的，扫描图片序列中的多个帧图片，可以是通过扫描设备围绕目标物体进行连续扫描而获得扫描图片，相邻帧图片内包括的物体信息相差较小，其可以是同一部件不同角度的图片，因此，在前一帧图片映射到三维预设模型后，采用后一帧图片对前一帧图片进行配准，以获得更准确的物体部件，进而获得更加准确的三维模型。
以使用截断符号距离函数(Truncation Sign Distance Function，TSDF)来表示三维预设模型为例，从TSDF值为零的空体素开始，使用单位变换矩阵将第一帧配准到该网格中。中间迭代过程包括将帧F t采用传感器位姿估计配准方法配准到现有模型M t-1，该模型由先前配准融合的帧{F 1,...,F t-1}构建而成需要指出的是，本实施例采用这种方法，考虑到帧图片的语义标签，F t的像素和M t-1的体素带有不同的语义标签，将拥有相同语义标签的像素和体素映射到彼此，得到F t配准到M t-1的变换。通过下述描述来进行详细说明。
首先，将F t中像素分组成语义集合S i，每个可能的部分标签为i。在每个集合中，只保留具有高置信度的像素。如果像素的标签概率分布P的信息熵H(P)低于阈值θ，则该像素被定义为具有高置信度的像素，信息熵通常计算如下：
Figure PCTCN2018114808-appb-000003
其中，P i指像素拥有标签i的概率。类似的，本发明将M t-1的体素划分为语义组S' i，只包含高置信度的体素。
其次，可以将计算语义集合的候选变换，并通过优化所有的候选变换来获得F t和M t-1之间的最优变换。更详细地说，可以使用迭代最近点方法(ICP)将F t中每个集合S i对齐到它在M t-1中对应的集合S' i，ICP提供了对齐两个集合的刚性变换T i。
给定所有部件标签的变换集合{T i}，目标是找到一个全局变换T，它最好地结合了{T i}中所有的变换。由于部件标签是通过网络预测获得的，并且不同部件可能具有不同的几何属性，因此对应到不同语义集合的变换在优化中应该具有不同的重要性。根据这个目标，为每个集合S i定义一个权重w i：w i＝conf i+size i+var i。其中，conf i指S i标签预测的平均置信度，由S i中所有像素的均值H(P)给出；size i是在F t内分配有S i标签的像素的百分比，var i是重建模型中对应于集合S i的集合S' i中体素的法向量的变化。因此，权重体现了更高置信度，更大尺寸和更多法向变化的集合应该更多地影响全局变换。偏好法向变化 大的原因是因为具有小的法向变化的像素往往会导致匹配的不确定，例如：平坦桌面上的像素匹配是不明确的，因为将任何一个这样的像素进行映射之前增加一个平移都导致相同的损失值。
由每个集合定义的权重，可以通过求解下面的优化问题来计算全局变换T*：
Figure PCTCN2018114808-appb-000004
其中，p i,j指集合i的第j个像素。目标函数表明，最优变换T*，使所有集合的加权距离和最小，可以使用高斯-牛顿法来最优化这个目标函数，用迭代方法将目标函数线性化并求解上述方程。具体的，将T*和{T i}约束为只由平移和旋转组成的刚性变换。然后，本发明用T*将F t对齐到M t-1，并将当前帧与模型相结合，生成一个新的模型M t。
对齐后，我们通过累积F t中相应像素的标签分布来更新M t中每个体素的标签概率分布：
Figure PCTCN2018114808-appb-000005
其中，
Figure PCTCN2018114808-appb-000006
指第t次迭代模型中一个体素的标签分布，
Figure PCTCN2018114808-appb-000007
指第t帧中对应像素的标签分布。
S502、根据更新后的三维预设模型的各体素对应的标签分布和预测语义标签，将各扫描图片映射在三维预设模型的对应位置，得到三维模型。
具体的，在上述S501的基础上，获得更新后的三维预设模型后，由于更新后的三维预设模型是根据扫描图片序列中多个帧图片相互配准获得的，因此，更新后的三维预设模型，将相同坐标位置的预测语义标签对应的扫描图片映射在该体素的位置上，可以获得更加准确的目标物体的三维模型。
上述三维物体重建方法，终端根根据扫描图片序列的帧顺序和对应的帧图片，采用预设的配准算法，对三维预设模型进行更新，得到更新后的三维预设模型，并根据根据更新后的三维预设模型的各体素对应的标签分布和预测语义标签，将各扫描图片映射在三维预设模型的对应位置，得到三维模型。本实施例中，终端通过三维预设模型的各体素对应的标签分布和预测语义标签之间的对应关系，将各扫描图片映射在三维预设模型的对应位置，得到三维模型，使得重建的三维物体更加真实。
上述实施例的三维物体重建方法，在配准过程中只考虑语义部件，但是背景也被标记并存储在重建的三维模型网格中。背景由与目标物体无关的数据组成，例如，目标物体后面的墙。为了重建更加准确的三维物体，终端通过删除背景，只保留目标物体的体素来实现。然而，由于标签预测的累积不确定性，简单地删除具有最大背景置信度的体素可能会在最终的三维模型中引入一些空洞。终端通过删除部分体素的方法，来删除目标物体的背景，进而获取更真实的三维模型。下面通过图7来详细说明。
图7为另一个实施例中三维物体重建方法的流程示意图。如图7所示，所示方法还包括：
S601、根据预设的分辨率参数和预设的迭代方法，生成一组超体素。
具体的，超体素可以是根据设定的分辨率参数获得的体积元素，其可以根据分辨率的 变化而变化，分辨率越高，则超体素的体积越小，分辨率越低，则超体素的体积越大。预设的分辨率参数可以根据预设的迭代方法进行修改，预设的迭代方法可以是先设置一个低分辨率的值，根据预设的分辨率参数的步进，修改分辨率的值，获得新的分辨率参数。
S602、根据分辨率参数，删除超体素中的目标体素；目标体素包括：非背景标签的比例小于预设的第一阈值的体素，和，相邻体素的背景标签的比例大于预设的第二阈值的体素。
具体的，可以首先指定一个低分辨率的值，并生成第一组超体素，删除非背景标签的比例小于预设的第一阈值的体素，和，相邻体素的背景标签的比例大于预设的第二阈值的体素。其中，第一阈值和第二阈值可以是设定的固定值，也可以是根据机器学习获得的数值，本申请实施例对此不做限制。例如，可以删除非背景标签体素所占的比例不到5％的目标体素，和，与该超体素相邻的所有超体素中，不到一半的超体素是由部件标签所主导的目标体素。迭代性地重复这个过程。在每次迭代过程中，增大分辨率参数，以将来自先前的迭代结果分割为更小的超体素。当每个超体素的平均置信度大于给定阈值或者当达到最大迭代次数时，则迭代过程结束。在一个实施例中，删除目标体素后获得的三维图像如图7a所示。在背景去除的最后阶段，模型划被分为粒度很细的超体素，其中每个超体素与其内包含的体素的平均标签分布相关联。基于这些分布，本发明使用图割的方法来平滑超体素标记。具体来说，可以通过定义图，其中每个超体素是连接到其相邻超体素的节点，用于标记节点的数据项是基于相应超体素的标签分布，通过将概率值转换为能量代价(energy cost)，平滑项地遵循Potts模型。
上述三维物体重建方法，终端根据预设的分辨率参数和预设的迭代方法，生成一组超体素，并删除超体素中的目标体素；目标体素包括：非背景标签的比例小于预设的第一阈值的体素，和，相邻体素的背景标签的比例大于预设的第二阈值的体素。本实施例中，终端通过删除超体素中的目标体素，删除了目标物体三维模型中的背景图像，使得获得的目标物体的三维模型更加真实。
在上述实施例的基础上，如图8所示，终端获取扫描图片序列，该扫描序列中的扫描图片可以是RGBD图片，采用神经网络算法，获取扫描图片的预测语义标签，根据预测语义标签和扫描图片，重建目标物体的三维模型。进一步地，可以根据获取的目标物体的三维模型，获得对应的投影标签，进而获取投影标签与预测语义标签的之间的置信度，根据置信度，分别对扫描图片进行自学习和主动学习，已获得更加准确的目标物体的三维模型。需要说明的是，图8仅是一种示例。
应该理解的是，虽然图2-7的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，这些步骤可以以其它的顺序执行。而且，图2-7中的至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些子步骤或者阶段的执行顺序也不必然是依次进行。
图9为一个实施例提供的三维物体重建装置的结构示意图。如图9所示，该三维物体重建装置，包括：获取模块10、预测模块20和重建模块30，其中：
获取模块，用于获取目标物体的扫描图片序列；所述扫描图片序列包括至少一帧扫描图片，所述扫描图片包括深度信息；
预测模块，用于采用神经网络算法，对所述扫描图片序列中的扫描图片进行预测，获取各所述扫描图片的预测语义标签；
重建模块，用于根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型。
本发明实施例提供的三维物体重建装置，可以执行上述方法实施例，其实现原理和技 术效果类似，在此不再赘述。
关于一种三维物体重建装置的具体限定可以参见上文中对于三维物体重建方法的限定，在此不再赘述。上述三维物体重建装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。
在一个实施例中，提供了一种计算机设备，该计算机设备可以是终端，其内部结构图可以如图10所示。该计算机设备包括通过系统总线连接的处理器、存储器、网络接口、显示屏和输入装置。其中，该计算机设备的处理器用于提供计算和控制能力。该计算机设备的存储器包括非易失性存储介质、内存储器。该非易失性存储介质存储有操作系统和计算机程序。该内存储器为非易失性存储介质中的操作系统和计算机程序的运行提供环境。该计算机设备的网络接口用于与外部的终端通过网络连接通信。该计算机设备被处理器执行时以实现一种三维物体重建方法。该计算机设备的显示屏可以是液晶显示屏或者电子墨水显示屏，该计算机设备的输入装置可以是显示屏上覆盖的触摸层，也可以是计算机设备外壳上设置的按键、轨迹球或触控板，还可以是外接的键盘、触控板或鼠标等。
本领域技术人员可以理解，图10中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
在一个实施例中，提供了一种计算机设备，包括存储器和处理器，存储器中存储有计算机程序，该处理器执行计算机程序时实现以下步骤：
获取目标物体的扫描图片序列；所述扫描图片序列包括至少一帧扫描图片，所述扫描图片包括深度信息；
采用神经网络算法，对所述扫描图片序列中的扫描图片进行预测，获取各所述扫描图片的预测语义标签；
根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型。
本实施例提供的计算机设备，其实现原理和技术效果与上述方法实施例类似，在此不再赘述。
在一个实施例中，提供了一种计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现以下步骤：
获取目标物体的扫描图片序列；所述扫描图片序列包括至少一帧扫描图片，所述扫描图片包括深度信息；
采用神经网络算法，对所述扫描图片序列中的扫描图片进行预测，获取各所述扫描图片的预测语义标签；
根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型。
本实施例提供的计算机可读存储介质，其实现原理和技术效果与上述方法实施例类似，在此不再赘述。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一非易失性计算机可读取存储介质中，该计算机程序在执行时，可包括如上述各方法的实施例的流程。其中，本申请所提供的各实施例中所使用的对存储器、存储、数据库或其它介质的任何引用，均可包括非易失性和/或易失性存储器。非易失性存储器可包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)或者外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDRSDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchl ink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)等。
以上所述实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本发明的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本发明构思的前提下，还可以做出若干变形和改进，这些都属于本发明的保护范围。因此，本发明专利的保护范围应以所附权利要求为准。

Claims (10)

  1. 一种三维物体重建方法，其特征在于，所述方法包括：
    获取目标物体的扫描图片序列；所述扫描图片序列包括至少一帧扫描图片，所述扫描图片包括深度信息；
    采用神经网络算法，对所述扫描图片序列中的扫描图片进行预测，获取各所述扫描图片的预测语义标签；
    根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型。
  2. 根据权利要求1所述方法，其特征在于，所述根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型之后，所述方法还包括：
    根据所述三维模型，获取所述扫描图片序列中各扫描图片的投影标签；
    根据所述投影标签和所述预测语义标签，获取所述预测语义标签的置信度；
    根据所述预测语义标签的置信度，更新所述预测语义标签。
  3. 根据权利要求2所述方法，其特征在于，所述根据所述预测语义标签的置信度，更新所述预测语义标签，包括：
    若所述预测语义标签的置信度大于预设阈值，则将所述预测语义标签设置为高置信度标签；
    采用所述神经网络算法，根据所述高置信度标签和对应的扫描图片，对所述扫描图片序列中的扫描图片进行预测，获取更新后的预测语义标签。
  4. 根据权利要求2所述方法，其特征在于，所述根据所述预测语义标签的置信度，更新所述预测语义标签，还包括：
    若所述预测语义标签的置信度不大于预设阈值，则将所述预测语义标签标记为低置信度标签；
    采用主动学习的方法，对所述低置信度标签对应的扫描图片重新标注，获取更新后的预测语义标签。
  5. 根据权利要求2所述方法，其特征在于，所述根据所述投影标签和所述预测语义标签，获取所述预测语义标签的置信度，包括：
    对所述投影标签与所述预测语义标签对应的扫描图片中的像素进行对比，获取像素一致性；
    根据所述像素一致性，获取所述预测语义标签的置信度。
  6. 根据权利要求1-5任一项所述方法，其特征在于，所述根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型，包括：
    根据三维预设模型的各体素对应的标签分布和所述预测语义标签，将各所述扫描图片映射在所述三维预设模型的对应位置，得到所述三维模型。
  7. 根据权利要求6所述方法，其特征在于，所述根据三维预设模型的各体素对应的标签 分布和所述预测语义标签，将各所述扫描图片映射在所述三维预设模型的对应位置，得到所述三维模型，包括：
    根据所述扫描图片序列的帧顺序和对应的帧图片，采用预设的配准算法，对所述三维预设模型，进行更新，得到更新后的三维预设模型；
    根据所述更新后的三维预设模型的各体素对应的标签分布和所述预测语义标签，将各所述扫描图片映射在所述三维预设模型的对应位置，得到所述三维模型。
  8. 根据权利要求1-5任一项所述方法，其特征在于，所述根据所述预测语义标签和所述扫描图片序列中的扫描图片，重建所述目标物体的三维模型之后，所述方法还包括：
    根据预设的分辨率参数和预设的迭代方法，生成第一组超体素；
    根据所述分辨率参数，删除第一组超体素中的目标体素；所述目标体素包括：非背景标签的比例小于预设的第一阈值的体素，和，相邻体素的背景标签的比例大于预设的第二阈值的体素。
  9. 一种计算机设备，包括存储器和处理器，所述存储器存储有计算机程序，其特征在于，所述处理器执行所述计算机程序时实现权利要求1-8中任一项所述方法的步骤。
  10. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1-8中任一项所述的方法的步骤。
PCT/CN2018/114808 2018-10-31 2018-11-09 三维物体重建方法、计算机设备及存储介质 Ceased WO2020087564A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/491,544 US11403812B2 (en) 2018-10-31 2018-11-09 3D object reconstruction method, computer apparatus and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811288537.7A CN109493417B (zh) 2018-10-31 2018-10-31 三维物体重建方法、装置、设备和存储介质
CN201811288537.7 2018-10-31

Publications (1)

Publication Number Publication Date
WO2020087564A1 true WO2020087564A1 (zh) 2020-05-07

Family

ID=65693557

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/114808 Ceased WO2020087564A1 (zh) 2018-10-31 2018-11-09 三维物体重建方法、计算机设备及存储介质

Country Status (3)

Country Link
US (1) US11403812B2 (zh)
CN (1) CN109493417B (zh)
WO (1) WO2020087564A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114185853A (zh) * 2021-11-25 2022-03-15 武汉尺子科技有限公司 一种模型文件封装方法、装置和存储介质
CN114782321A (zh) * 2022-03-24 2022-07-22 北京医准智能科技有限公司 胸部ct影像选择方法、装置、设备及存储介质

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110047152B (zh) * 2019-04-12 2022-05-17 腾讯科技（深圳）有限公司 基于虚拟环境的物体构建方法、装置及可读存储介质
CN110287764B (zh) * 2019-05-06 2022-01-11 深圳大学 姿势预测方法、装置、计算机设备和存储介质
CN112733579B (zh) * 2019-10-28 2026-04-17 华为技术有限公司 对象重建的方法、装置、设备及存储介质
CN113688851B (zh) * 2020-05-18 2023-09-15 华为云计算技术有限公司 数据标注方法和装置和精细粒度识别方法和装置
CN111862343B (zh) * 2020-07-17 2024-02-02 歌尔科技有限公司 一种三维重建方法、装置、设备及计算机可读存储介质
CN112161619B (zh) * 2020-09-16 2022-11-15 思看科技(杭州)股份有限公司 位姿检测方法、三维扫描路径规划方法和检测系统
EP4053800A1 (en) * 2021-03-04 2022-09-07 Kardiolytics Inc. Autonomous reconstruction of vessels on computed tomography images
CN113706543B (zh) * 2021-08-18 2024-07-12 北京达佳互联信息技术有限公司 一种三维位姿的构建方法、设备及存储介质
CN113792623B (zh) * 2021-08-27 2022-12-13 同方威视技术股份有限公司 安检ct目标物识别方法和装置
CN114067118B (zh) * 2022-01-12 2022-04-15 湖北晓雲科技有限公司 一种航空摄影测量数据的处理方法
CN115375857B (zh) * 2022-10-26 2023-01-03 深圳市其域创新科技有限公司 三维场景重建方法、装置、设备及存储介质
CN118283278B (zh) * 2024-05-09 2024-09-03 深圳市其域创新科技有限公司 三维数据压缩方法、数据解压方法、电子设备及存储介质
CN118864849B (zh) * 2024-07-03 2025-04-15 中国科学院南京土壤研究所 一种分割土壤ct图像中生物性孔隙的方法
CN118736207B (zh) * 2024-09-03 2025-01-28 同方威视技术股份有限公司 三维ct数据的目标识别方法、装置及射线扫描检测系统
CN119850992B (zh) * 2025-03-18 2025-06-03 雷鸟创新技术(深圳)有限公司 场景理解方法、装置、电子设备及计算机可读存储介质
CN120355848B (zh) * 2025-04-08 2025-12-02 北京数原数字化城市研究中心 一种场景表面重建方法、装置、设备及介质
CN120107514B (zh) * 2025-05-07 2025-08-01 广东省人民医院 基于流域分析的多级血管网络重建与分段方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107358648A (zh) * 2017-07-17 2017-11-17 中国科学技术大学 基于单张人脸图像的实时全自动高质量三维人脸重建方法
CN107945269A (zh) * 2017-12-26 2018-04-20 清华大学 基于多视点视频的复杂动态人体对象三维重建方法及系统
CN108090960A (zh) * 2017-12-25 2018-05-29 北京航空航天大学 一种基于几何约束的目标重建方法
CN108510573A (zh) * 2018-04-03 2018-09-07 南京大学 一种基于深度学习的多视点人脸三维模型重建的方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9773302B2 (en) * 2015-10-08 2017-09-26 Hewlett-Packard Development Company, L.P. Three-dimensional object model tagging
US10580140B2 (en) * 2016-05-23 2020-03-03 Intel Corporation Method and system of real-time image segmentation for image processing
US20180322623A1 (en) * 2017-05-08 2018-11-08 Aquifi, Inc. Systems and methods for inspection and defect detection using 3-d scanning
US20190138786A1 (en) * 2017-06-06 2019-05-09 Sightline Innovation Inc. System and method for identification and classification of objects
US10186038B1 (en) * 2017-07-18 2019-01-22 Siemens Healthcare Gmbh Segmentation and representation network for pose discrimination
WO2019075276A1 (en) * 2017-10-11 2019-04-18 Aquifi, Inc. SYSTEMS AND METHODS FOR IDENTIFYING OBJECT
EP3503038A1 (en) * 2017-12-22 2019-06-26 Promaton Holding B.V. Automated 3d root shape prediction using deep learning methods
US10685446B2 (en) * 2018-01-12 2020-06-16 Intel Corporation Method and system of recurrent semantic segmentation for image processing
CN108389251B (zh) * 2018-03-21 2020-04-17 南京大学 基于融合多视角特征的投影全卷积网络三维模型分割方法
CN108491880B (zh) * 2018-03-23 2021-09-03 西安电子科技大学 基于神经网络的物体分类和位姿估计方法
CN108564126B (zh) * 2018-04-19 2022-04-19 郑州大学 一种融合语义控制的特定场景生成方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107358648A (zh) * 2017-07-17 2017-11-17 中国科学技术大学 基于单张人脸图像的实时全自动高质量三维人脸重建方法
CN108090960A (zh) * 2017-12-25 2018-05-29 北京航空航天大学 一种基于几何约束的目标重建方法
CN107945269A (zh) * 2017-12-26 2018-04-20 清华大学 基于多视点视频的复杂动态人体对象三维重建方法及系统
CN108510573A (zh) * 2018-04-03 2018-09-07 南京大学 一种基于深度学习的多视点人脸三维模型重建的方法

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
SANJAYAN, JAY G. ET AL.: "Effect of Surface Moisture on Inter-Layer Strength of 3D Printed Concrete", ELSEVIER: CONSTRUCTION AND BUILDING MATERIALS, vol. 172, 30 May 2018 (2018-05-30), pages 468 - 475, XP055699234, ISSN: 0950-0618, DOI: 10.1016/j.conbuildmat.2018.03.232 *

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114185853A (zh) * 2021-11-25 2022-03-15 武汉尺子科技有限公司 一种模型文件封装方法、装置和存储介质
CN114782321A (zh) * 2022-03-24 2022-07-22 北京医准智能科技有限公司 胸部ct影像选择方法、装置、设备及存储介质
CN114782321B (zh) * 2022-03-24 2022-12-06 北京医准智能科技有限公司 胸部ct影像选择方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN109493417A (zh) 2019-03-19
US11403812B2 (en) 2022-08-02
US20210327126A1 (en) 2021-10-21
CN109493417B (zh) 2023-04-07

Similar Documents

Publication Publication Date Title
CN109493417B (zh) 三维物体重建方法、装置、设备和存储介质
US20250005797A1 (en) Processing images using self-attention based neural networks
US10733431B2 (en) Systems and methods for optimizing pose estimation
US10796452B2 (en) Optimizations for structure mapping and up-sampling
US11443481B1 (en) Reconstructing three-dimensional scenes portrayed in digital images utilizing point cloud machine-learning models
US10586350B2 (en) Optimizations for dynamic object instance detection, segmentation, and structure mapping
CN111670457B (zh) 动态对象实例检测、分割和结构映射的优化
EP3493106B1 (en) Optimizations for dynamic object instance detection, segmentation, and structure mapping
CN111627065A (zh) 一种视觉定位方法及装置、存储介质
WO2019108250A1 (en) Optimizations for dynamic object instance detection, segmentation, and structure mapping
US20210365718A1 (en) Object functionality predication methods, computer device, and storage medium
CN116342887A (zh) 用于图像分割的方法、装置、设备和存储介质
CN113140034A (zh) 基于房间布局的全景新视角生成方法、装置、设备和介质
CN111583264A (zh) 图像分割网络的训练方法、图像分割方法和存储介质
CN111652831B (zh) 对象融合方法、装置、计算机可读存储介质及电子设备
CN110717405A (zh) 人脸特征点定位方法、装置、介质及电子设备
CN119942100A (zh) 基于跨域伪标签生成的点云分割方法、装置、介质及设备
CN121120906B (zh) 图像渲染方法、装置及设备
Kang The applications of robust estimation method BaySAC in indoor point cloud processing
CN115984092A (zh) 一种图像处理方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18938586

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 30.08.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 18938586

Country of ref document: EP

Kind code of ref document: A1