WO2022016867A1 - 三维网格模型的重建方法及其装置、设备、存储介质 - Google Patents

三维网格模型的重建方法及其装置、设备、存储介质 Download PDF

Info

Publication number
WO2022016867A1
WO2022016867A1 PCT/CN2021/078485 CN2021078485W WO2022016867A1 WO 2022016867 A1 WO2022016867 A1 WO 2022016867A1 CN 2021078485 W CN2021078485 W CN 2021078485W WO 2022016867 A1 WO2022016867 A1 WO 2022016867A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
feature information
target
subdivision
mesh model
Prior art date
Application number
PCT/CN2021/078485
Other languages
English (en)
French (fr)
Inventor
李海
章国锋
鲍虎军
王楠
谢卫健
Original Assignee
浙江商汤科技开发有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浙江商汤科技开发有限公司 filed Critical 浙江商汤科技开发有限公司
Priority to KR1020227002922A priority Critical patent/KR20220028010A/ko
Priority to JP2021568963A priority patent/JP7395617B2/ja
Publication of WO2022016867A1 publication Critical patent/WO2022016867A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G06T17/205—Re-meshing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

  • the present disclosure relates to the technical field of artificial intelligence, and in particular, to a method for reconstructing a three-dimensional mesh model and its device, equipment, and storage medium.
  • Three-dimensional reconstruction usually refers to the technology of restoring and reconstructing objects or scenes in three-dimensional space, and the reconstructed model can be easily represented, processed and displayed by the computer.
  • the three-dimensional model of the three-dimensional object is recovered by using the multi-view image of the object, the details of the reflected object are relatively blurred.
  • the present disclosure provides at least a reconstruction method of a three-dimensional mesh model, an apparatus, a device, and a storage medium thereof.
  • a first aspect of the present disclosure provides a method for reconstructing a three-dimensional mesh model, including: extracting features from a target image to obtain target feature information; wherein the target image includes an object to be reconstructed; and based on the target feature information, determining The saliency area of the object to be reconstructed; based on the saliency area, a final three-dimensional mesh model of the object to be reconstructed is constructed.
  • the final 3D mesh model of the object to be reconstructed is constructed by using the salient region reflecting more details of the object to be reconstructed, which can be flexibly reconstructed to the most obvious details. 3D mesh model.
  • determining the salient region of the object to be reconstructed based on the target feature information includes: using the target feature information to transform a basic point cloud model into a target point cloud model corresponding to the object to be reconstructed; A saliency region of the target point cloud model is determined.
  • the saliency area corresponding to the detail area of the object to be reconstructed can be obtained by using the target feature information, and the saliency area can be obtained by deforming the point cloud model.
  • using the feature information to transform the basic point cloud model into a target point cloud model corresponding to the object to be reconstructed includes: projecting the basic point cloud model to the plane where the target image is located to determine the target point cloud model. the target feature information corresponding to each point in the basic point cloud model; use the first neural network to process the target feature information corresponding to each point in the basic point cloud model, and obtain the basic point cloud model deformed into the The position information of each point behind the target point cloud model; the determining the salient region of the target point cloud model includes: acquiring the point distribution of the target point cloud model; finding out all the points in the target point cloud model The point cloud area whose distribution of the points satisfies the requirement of saliency distribution is used as the saliency area.
  • the deformation of the point cloud model is realized through the first neural network, and the saliency region is determined by using the point distribution of the target point cloud model.
  • the method before projecting the basic point cloud model to the plane where the target image is located, the method further includes: uniformly sampling points in a unit sphere to obtain the basic point cloud model; the basic point cloud
  • the position information of each point after the model is deformed into the target point cloud model is: the position offset of each point after the basic point cloud model is deformed into the target point cloud model; the saliency distribution requirements include points The distribution density is greater than the preset density value.
  • the basic point cloud model can be obtained by uniformly sampling the unit sphere;
  • the point position information output by the first neural network is the offset, and the uniform sampling point and the position offset can be used to obtain the point cloud model of the target point.
  • location information; and the saliency area may be determined by, but not limited to, the point distribution density being greater than the preset density value, so that the points in the saliency area are densely distributed and can better reflect the details of the object to be reconstructed.
  • the method further includes the following steps to obtain the first neural network by training: acquiring a real three-dimensional mesh model of a sample image and a sample object, wherein the sample image includes the sample object; Perform feature extraction to obtain sample feature information; project the basic point cloud model to the plane where the sample image is located to determine the sample feature information corresponding to each point in the basic point cloud model; use the first neural network to
  • the sample feature information corresponding to each point in the basic point cloud model is processed to obtain the position information of each point after the basic point cloud model is deformed into the predicted point cloud model; Simplify the grid to obtain a simplified three-dimensional mesh model; find out the points in the predicted point cloud model that match the vertices of the simplified three-dimensional mesh model, and obtain several groups of matching point pairs; use the position difference of each group of matching point pairs , and adjust the parameters of the first neural network.
  • the real 3D mesh model is monitored to ensure that there are fewer patches in the flat area, and then the vertices of the simplified real 3D mesh model are used as supervision signals for training, and the first neural network obtained by training can output the target point. Position information of each point of the cloud model.
  • the constructing the final three-dimensional mesh model of the object to be reconstructed based on the saliency region includes: constructing and obtaining an initial three-dimensional mesh model of the object to be reconstructed by using the target feature information;
  • the three-dimensional mesh model is subjected to mesh subdivision to obtain the final three-dimensional mesh model of the object to be reconstructed, wherein the mesh subdivision includes local mesh subdivision corresponding to the salient region.
  • the salient region is subdivided to reduce the number of vertices of the mesh model, and the mesh model has a certain richness in the details of the saliency region. That is, when performing local mesh subdivision corresponding to the saliency area, since the saliency area is an area that reflects more details of the object to be reconstructed, the mesh subdivision for the saliency area not only reduces the number of vertices of the mesh model. , which can reduce the storage space required for the data of the 3D mesh model, and make the reconstructed 3D mesh model corresponding to the salient region not cause excessive smoothing, and can better reflect the details, so that more detailed 3D mesh can be reconstructed. grid model, and reduce the storage space required for the data of the 3D grid model.
  • performing grid subdivision on the initial 3D grid model to obtain the final 3D grid model of the object to be reconstructed includes: taking the 3D grid model before the current grid subdivision as the first three-dimensional mesh model; project the first three-dimensional mesh model to the plane where the target image is located to determine the target feature information corresponding to each vertex in the first three-dimensional mesh model; At least one new vertex is added to the target area of the mesh model; wherein, the target area includes at least the saliency area; and the subdivision edge is obtained by using the target feature information of the original vertex of the first three-dimensional mesh model Target feature information of the corresponding new vertex; based on the target feature information of the original vertex and the new vertex of the first three-dimensional mesh model, the second three-dimensional mesh model after this mesh subdivision is obtained.
  • the target feature information can be obtained by first projecting the first 3D mesh model, then adding new vertices, and using the original vertices and new vertices of the first 3D mesh model to obtain a subdivided second 3D mesh model to realize the mesh Subdivision, which reflects the details of the object to be reconstructed.
  • the adding at least one new vertex in the target area of the first three-dimensional mesh model includes: adding at least one new vertex to the first three-dimensional mesh In the model, at least one edge located in the salient region is used as a subdivision edge; at least one new vertex is determined on the subdivision edge.
  • At least one edge of the saliency region is used as a subdivision edge, and new vertices are obtained on the subdivision edge, so as to determine the new vertex when the local mesh is subdivided.
  • the saliency area includes several salient points; in the first three-dimensional mesh model, at least one edge located in the saliency area is used as a subdivision edge, including: in the first three-dimensional mesh In the lattice model, the edge whose position satisfies the preset position condition is found for each of the significant points as the edge to be subdivided; each edge in the first three-dimensional mesh model is counted as the edge to be subdivided. number of times; the edge whose number of times of the edge to be subdivided satisfies the preset subdivision condition is taken as the subdivision edge.
  • the preset position condition is that it is closest to the position of the salient point;
  • the preset subdivision condition is that the number of times of the edge to be subdivided is greater than the preset number of times, or, in the first three-dimensional mesh model In the ordering of all the edges from most to least, the number of the edges to be subdivided is within the previous preset number or the previous preset ratio.
  • the mesh subdivision further includes global mesh subdivision corresponding to the entire three-dimensional mesh model; if the current mesh subdivision is the global mesh subdivision, the first three-dimensional mesh Adding at least one new vertex to the target area of the mesh model includes: taking each edge in the first three-dimensional mesh model as a subdivision edge; and determining at least one new vertex on the subdivision edge.
  • the mesh subdivision also includes global mesh subdivision corresponding to the entire 3D mesh model, and the mesh model obtained by the global mesh subdivision is more detailed as a whole.
  • determining at least one new vertex on the subdivision edge includes: taking the midpoint of the subdivision edge as the new vertex; using the target of the original vertex of the first three-dimensional mesh model feature information, obtaining the target feature information of the new vertex corresponding to the subdivision edge, including: using the target feature information of the two original vertices corresponding to the subdivision edge to obtain the target feature information of the new vertex corresponding to the subdivision edge target feature information.
  • obtaining the second three-dimensional mesh model after this mesh subdivision based on the target feature information of the original vertex and the new vertex of the first three-dimensional mesh model includes: using the second neural network to The target feature information of the original vertex and the new vertex of the first three-dimensional mesh model is processed to obtain the position information of each vertex after the first three-dimensional mesh model is deformed into the first three-dimensional mesh model.
  • the first three-dimensional mesh model is deformed into a second three-dimensional mesh model using the second neural network.
  • using the target feature information to construct and obtain the initial 3D mesh model of the object to be reconstructed includes: projecting the basic 3D mesh model to the plane where the target image is located to determine the basic 3D mesh model
  • the target feature information corresponding to each vertex in the basic 3D mesh model is processed by using the second neural network to process the target feature information corresponding to each vertex in the basic 3D mesh model to obtain the basic 3D mesh model deformed into the initial 3D mesh
  • the basic three-dimensional grid model can be deformed into an initial three-dimensional grid model by using the second neural network, so as to complete the initialization of the object to be reconstructed and reflect the initial shape of the object to be reconstructed.
  • the target image is a two-dimensional image; and/or, performing feature extraction on the target image to obtain target feature information includes: using a third neural network to perform feature extraction on the target image to obtain feature information of several dimensions;
  • the target feature information is obtained by fusing the feature information of the several dimensions, wherein the target feature information is a feature tensor.
  • the third neural network is used to perform feature extraction on the two-dimensional target image, and a feature tensor indicating the target feature information is obtained.
  • a second aspect of the present disclosure provides a reconstruction device for a three-dimensional mesh model, comprising: a feature extraction module configured to perform feature extraction on a target image to obtain target feature information; wherein the target image includes an object to be reconstructed; saliency The region determination module is configured to determine the saliency region of the object to be reconstructed based on the target feature information; the model building module is configured to construct the final three-dimensional mesh model of the object to be reconstructed based on the saliency region.
  • a third aspect of the present disclosure provides an electronic device including a memory and a processor coupled to each other, and the processor is configured to execute program instructions stored in the memory to implement the method for reconstructing a three-dimensional mesh model in the first aspect.
  • a fourth aspect of the present disclosure provides a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, implement the method for reconstructing a three-dimensional mesh model in the above-mentioned first aspect.
  • the target feature information of the target image is used to determine the salient area of the object to be reconstructed, and when the three-dimensional mesh model is reconstructed, the salient area is meshed to reduce the number of vertices of the mesh model and make the mesh
  • the details of the model in the saliency region have a certain richness. That is, when performing local mesh subdivision corresponding to the saliency area, since the saliency area is an area that reflects more details of the object to be reconstructed, the mesh subdivision for the saliency area not only reduces the number of vertices of the mesh model.
  • 1A is a schematic flowchart of an embodiment of a method for reconstructing a 3D mesh model of the present disclosure
  • 1B is a schematic diagram of a network architecture of a method for reconstructing a 3D mesh model according to an embodiment of the present application
  • FIG. 2 is a schematic flowchart of step S12 of an embodiment of the method for reconstructing a 3D mesh model of the present disclosure
  • FIG. 3 is a schematic flowchart of another embodiment of the method for reconstructing a three-dimensional mesh model of the present disclosure
  • step S24 is a schematic flowchart of step S24 of another embodiment of the method for reconstructing a 3D mesh model of the present disclosure
  • FIG. 6 is a schematic frame diagram of an embodiment of an apparatus for reconstructing a three-dimensional mesh model of the present disclosure
  • FIG. 7 is a schematic frame diagram of another embodiment of the apparatus for reconstructing a three-dimensional mesh model of the present disclosure.
  • FIG. 8 is a schematic diagram of a framework of an embodiment of an electronic device of the present disclosure.
  • FIG. 9 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present disclosure.
  • FIG. 1A is a schematic flowchart of an embodiment of a method for reconstructing a 3D mesh model of the present disclosure. Specifically, the following steps can be included:
  • Step S11 Perform feature extraction on the target image to obtain target feature information.
  • the target image includes the object to be reconstructed.
  • the execution body of the method for reconstructing a three-dimensional mesh model may be a device for reconstructing a three-dimensional mesh model.
  • the method for reconstructing a three-dimensional mesh model may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment ( User Equipment, UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, Personal Digital Assistant (PDA), handheld devices, computing devices, in-vehicle devices, wearable devices, etc.
  • the method for reconstructing the three-dimensional mesh model may be implemented by the processor calling computer-readable instructions stored in the memory.
  • the object to be reconstructed may be an airplane, a building, a person, an animal, etc., which is not specifically limited in this embodiment of the present disclosure.
  • the target image may be a two-dimensional image, such as a single view shot from a single perspective, etc.
  • the specific form of the target image is not specifically limited, as long as it can contain feature information of the object to be reconstructed. Since the target image contains the object to be reconstructed, feature extraction is performed on the target image to obtain target feature information corresponding to the object to be reconstructed.
  • the target feature information indicates the features of the object to be reconstructed, that is, the target feature information may include the overall feature information and local feature information of the object to be reconstructed.
  • the expression form of the target feature information is not specifically limited, and it only needs to include the feature information of the object to be reconstructed.
  • a third neural network is used to perform feature extraction on the target image to obtain feature information of several dimensions; the feature information of several dimensions is obtained by fusing the feature information of several dimensions.
  • the third neural network may be any neural network capable of image feature extraction, including but not limited to convolutional neural networks. Therefore, the third neural network is used to perform feature extraction on the target image, and the target feature information reflecting the feature information is obtained.
  • the third neural network is a convolutional neural network
  • the convolutional neural network includes several convolutional layers and a pooling layer, and a pooling layer is used after each preset number of convolutional layers, so as to realize the feature extraction.
  • Feature reduction. Input the target image into the convolutional neural network.
  • the first convolutional layer of the convolutional neural network obtains the feature information of the same size as the target image
  • the second convolutional layer obtains the feature of half the size of the target image.
  • the size of the feature information obtained by the convolutional layer of the next layer is half of the size of the feature information obtained by the adjacent convolutional layer of the previous layer, so as to obtain the feature information of several dimensions.
  • the target feature information may be obtained by fusing feature information of all dimensions, or may be obtained by fusing feature information of some dimensions, and the implementation manner of feature fusion is not specifically limited.
  • the feature information of several dimensions corresponds to different sizes, in order to facilitate the fusion of feature information, the feature information of several dimensions is normalized to feature information of the same size as the target image, and the normalized feature information is fused. Get target feature information.
  • the target feature information is a feature tensor, and when the target image is a two-dimensional image, in order to facilitate subsequent use of the feature tensor to achieve feature information fusion, the feature tensor has the same size as the two-dimensional image.
  • the feature information of several dimensions is normalized to the feature information of the same size as the target image, and the normalized feature information is fused to obtain a feature tensor, so that the feature information of different sizes is fused to the size of the two-dimensional image. feature tensors of the same size.
  • Step S12 Determine the saliency area of the object to be reconstructed based on the target feature information.
  • the saliency area is indicated in the area where the curvature of the object to be reconstructed changes greatly and there are many details.
  • the salient area can be the area corresponding to the propeller, the connection area between the wing and the fuselage, and the bending curvature changes greatly.
  • the smooth areas such as the fuselage are non-salient areas; the object to be reconstructed is a car, and the salient areas can be corresponding areas such as tires, headlights, and rear-view mirrors.
  • the target feature information includes the feature information of the object to be reconstructed, and is used for subsequent fusion of the feature information.
  • the saliency area can be represented by two-dimensional coordinate information, three-dimensional coordinate information and color identification information.
  • the coordinate-intensive area can be used as the salient area
  • the salient area is color identification information
  • the depth of the color can be used to indicate the amount of detail
  • the area where the dark color identification information is located can be used as the salient area.
  • FIG. 2 is a schematic flowchart of step S12 of an embodiment of the method for reconstructing a 3D mesh model of the present disclosure. Specifically, step S12 may include the following steps:
  • Step S121 Using the target feature information, transform the basic point cloud model into a target point cloud model corresponding to the object to be reconstructed.
  • the basic point cloud model is a pre-acquired or set model, and the points on the basic point cloud model are evenly distributed, and the basic point cloud model corresponding to any target image is consistent.
  • the basic point cloud model is obtained from uniform sampling points such as a unit sphere and an ellipsoid three-dimensional grid, and is composed of points with no connection relationship. After the uniformly distributed points of the basic point cloud model use the target feature information to fuse the feature information, the positions of the points are shifted, so that the basic point cloud model is deformed into the target point cloud model.
  • the distribution of points on the target point cloud model corresponds to the object to be reconstructed, and can reflect the characteristics of the object to be reconstructed, so that the target point cloud model indicates the overall shape and details of the object to be reconstructed.
  • the basic point cloud model is projected to the plane where the target image is located to determine the target feature information corresponding to each point in the basic point cloud model;
  • the target feature information is processed to obtain the position information of each point after the basic point cloud model is deformed into the target point cloud model, so as to realize the deformation of the point cloud model through the first neural network.
  • the first neural network is any neural network that obtains the position information of each point of the target point cloud model through deep learning, and can realize the deformation of the point cloud model.
  • the first neural network includes but is not limited to Convolutional Neural Networks.
  • the position information of each point after the basic point cloud model is deformed into the target point cloud model may be, but not limited to, the position offset of each point, the position offset path, and other data used to update the point position.
  • the position information of each point after the basic point cloud model is deformed into the target point cloud model may be the position offset of each point.
  • the position information of the point on the target point cloud model is obtained by calculation; in another disclosed embodiment, the position information of each point after the basic point cloud model is deformed into the target point cloud model may be the position offset path of each point, for example, the position The offset path is, but is not limited to, a vector value with direction and length.
  • the points of the basic point cloud model follow the position offset path to the position information of each point on the target point cloud model.
  • the first neural network is obtained by training data sets such as sample images and real 3D mesh models of sample objects.
  • a sample image and a real 3D mesh model of the sample object are obtained, wherein the sample image contains the sample object; feature extraction is performed on the sample image to obtain sample feature information; the basic point cloud model is projected onto the sample image to determine the sample feature information corresponding to each point in the basic point cloud model; use the first neural network to process the sample feature information corresponding to each point in the basic point cloud model, and obtain the basic point cloud model deformed into a predicted point cloud model
  • the related descriptions of the above steps are similar to those of step S11 and step S12, and are not repeated here.
  • the real 3D mesh model is indicated in the 3D mesh model of the sample object, and the actual sample image is used as the target image. model, indicating that the reconstruction method of the 3D mesh model has a higher degree of restoration.
  • the real 3D mesh model is monitored to ensure that there are fewer patches in the flat area, and then the vertices of the simplified real 3D mesh model are used as supervision signals for training.
  • the first neural network can output position information of each point of the target point cloud model.
  • Step S122 Determine the saliency area of the target point cloud model.
  • the target point cloud model is distributed with points of varying degrees of density, and the saliency area can be determined according to the distribution of the points.
  • the point distribution of the target point cloud model is obtained; the point cloud area in the target point cloud model whose point distribution meets the requirements of saliency distribution is found as a saliency area, so as to express the point cloud through the point cloud.
  • the point cloud is used to predict the detailed feature distribution of the object to be reconstructed, and the point cloud area that meets the requirements of the saliency distribution can be used as the saliency area.
  • the saliency distribution requirements can be set as required, including but not limited to taking the area with a point distribution density greater than a preset density value as the saliency area, where the preset density value can be set by yourself.
  • the basic point cloud model is projected to the plane where the target image is located to determine the target feature information corresponding to each point in the basic point cloud model;
  • the convolutional neural network is used to process the target feature information corresponding to each point in the basic point cloud model, and the position offset of each point after the basic point cloud model is deformed into the target point cloud model is obtained;
  • the position information and position offset are calculated to obtain the position information of the points on the target point cloud model; the point distribution of the target point cloud model is obtained; the area where the point distribution density in the target point cloud model is greater than the preset density value is found as the saliency Therefore, the distribution of points in the saliency area is dense, which can better reflect the details of the object to be reconstructed.
  • the basic point cloud model is transformed into the target point cloud model corresponding to the object to be reconstructed, and then the saliency region of the target point cloud model is determined, so that the salient region corresponding to the detail region of the object to be reconstructed can be obtained by using the target feature information.
  • the salient area is obtained by deforming the point cloud model.
  • Step S13 constructing a final three-dimensional mesh model of the object to be reconstructed according to the saliency region.
  • the final 3D mesh model can be obtained by a single or combined 3D mesh model construction method such as mesh subdivision, salient point cloud, etc., which is not limited here. Since the saliency area is an area that reflects more details of the object to be reconstructed, the final 3D mesh model can be constructed flexibly according to the saliency area, for example, but not limited to, mesh subdivision only for the salient area; The target feature information is constructed to obtain an initial 3D mesh model of the object to be reconstructed, and then the initial 3D mesh model is subdivided.
  • a single or combined 3D mesh model construction method such as mesh subdivision, salient point cloud, etc., which is not limited here. Since the saliency area is an area that reflects more details of the object to be reconstructed, the final 3D mesh model can be constructed flexibly according to the saliency area, for example, but not limited to, mesh subdivision only for the salient area; The target feature information is constructed to obtain an initial 3D mesh model of the object to be reconstructed, and then
  • feature extraction is performed on the target image containing the object to be reconstructed, and target feature information is obtained, so that the target feature information is used to determine the saliency area of the object to be reconstructed, and after the salient area is obtained, the saliency area can be used.
  • the final 3D mesh model of the object to be reconstructed is constructed, and the salient area can be used flexibly to reconstruct the 3D mesh model with obvious details.
  • the three-dimensional reconstruction of the object to be reconstructed can be realized through the network architecture as shown in FIG. 1B .
  • FIG. It includes: the user terminal 201, the network 202 and the reconstruction terminal 203 of the three-dimensional mesh model.
  • the user terminal 201 supporting an exemplary application and the reconstruction terminal 203 of the three-dimensional mesh model have a communication connection established through the network 202, when the user terminal 201 needs to perform three-dimensional reconstruction of the target image including the object to be reconstructed, first, the target image is passed through the network 202.
  • the network 202 sends the data to the reconstruction terminal 203 of the three-dimensional mesh model; then, the reconstruction terminal 203 of the three-dimensional mesh model obtains the target feature information by extracting the features of the target image, and based on the target feature information, determines the significant features of the object to be reconstructed. Finally, the reconstruction terminal 203 of the 3D mesh model realizes the 3D reconstruction of the object to be reconstructed through the salient region, and obtains the final 3D mesh model of the object to be reconstructed. In this way, after determining the salient region of the object to be reconstructed on the target image, the final 3D mesh model of the object to be reconstructed is constructed based on the salient region with more details, and the final 3D mesh model with more obvious details can be obtained.
  • FIG. 3 is a schematic flowchart of another embodiment of the method for reconstructing a 3D mesh model of the present disclosure. Specifically, the following steps can be included:
  • Step S21 Perform feature extraction on the target image to obtain target feature information.
  • the target image contains the object to be reconstructed.
  • Step S22 Determine the saliency area of the object to be reconstructed based on the target feature information.
  • Step S23 constructing an initial three-dimensional mesh model of the object to be reconstructed by using the target feature information.
  • the initial 3D mesh model is a simple 3D mesh model before mesh deformation, and reflects the initial overall shape and details of the object to be reconstructed.
  • the initial 3D mesh model consists of vertices, edges and faces. It can be understood that when the saliency area determined based on the target feature information in step S22, the initial overall shape and details of the object to be reconstructed are reflected by a number of point distributions without a connection relationship, and in step S23, the target feature information is used to construct and obtain.
  • the initial three-dimensional mesh model of the object to be reconstructed is the initial overall shape and details of the object to be reconstructed by vertices, edges and faces.
  • the basic three-dimensional grid model is projected to the plane where the target image is located to determine the target feature information corresponding to each vertex in the basic three-dimensional grid model;
  • the network processes the target feature information corresponding to each vertex in the basic three-dimensional mesh model, and obtains the position information of each vertex after the basic three-dimensional mesh model is deformed into the initial three-dimensional mesh model; Therefore, the basic 3D mesh model can be deformed into an initial 3D mesh model by using the second neural network, the initialization of the object to be reconstructed is completed, and the initial shape of the object to be reconstructed is reflected.
  • Both the basic 3D mesh model and the initial 3D mesh model are composed of vertices, edges and faces, and the positions of the vertices on the basic 3D mesh model are offset to form the positions of the vertices on the initial 3D mesh model, so that the originally evenly distributed Vertices are offset to positions that approximate the overall shape and details of the object to be reconstructed.
  • the second neural network is any neural network that obtains the position information of each vertex of the three-dimensional grid model through deep learning, and can realize the deformation of the grid model.
  • the second neural network includes but is not limited to a graph convolutional neural network.
  • the number of dimensions of the input layer, hidden layer, and output layer included in the graph convolutional neural network can be customized, which is not specifically limited here.
  • the second neural network is a neural network that can obtain the position information of each vertex.
  • the basic 3D mesh model can be deformed multiple times according to the target feature information, so that the vertex positions of the initial 3D mesh model are constantly approaching the real object to be reconstructed. The position of the vertex.
  • steps S22 and S23 may be performed in sequence, for example, step S22 is performed first, and then step S23 is performed; or, step S23 is performed first, and then step S22 is performed.
  • the above-mentioned steps S22 and S23 may also be performed simultaneously, which may be specifically set according to the actual application, which is not limited herein.
  • Step S24 Perform grid subdivision on the initial three-dimensional grid model to obtain a final three-dimensional grid model of the object to be reconstructed.
  • the mesh subdivision includes local mesh subdivision corresponding to the saliency region.
  • the initial three-dimensional mesh model and the saliency area are obtained, so that the saliency area is used to guide the mesh subdivision of the initial three-dimensional mesh model. Since the saliency area corresponds to the partial area of the object to be reconstructed, Therefore, only the area corresponding to the salient area can be used as the object of grid subdivision, which can better reflect the detailed information of the corresponding salient area, and the relatively flat area is represented by a larger grid to reduce memory consumption; During local mesh subdivision, mesh subdivision is introduced only in the salient areas indicated by many details, which more effectively reflects the inherent characteristics of the object to be reconstructed, and will not cause excessive smoothing.
  • the salient region is the point cloud distribution obtained by using the target feature information
  • the initial three-dimensional grid model is the grid distribution obtained by using the target feature information.
  • the saliency region-guided mesh subdivision of the initial 3D mesh model combines two model representations: point cloud representation and grid table.
  • global mesh subdivision and local mesh subdivision may be performed, wherein the global mesh subdivision corresponds to the entire three-dimensional mesh. model, and local mesh subdivision corresponds to a saliency area, and the order of global mesh subdivision and local mesh subdivision and the number of times of each mesh subdivision are not specifically limited.
  • the target feature information of the target image is used to determine the salient area of the object to be reconstructed, and when the three-dimensional mesh model is reconstructed, the salient area is meshed to reduce the number of vertices of the mesh model and make the mesh
  • the details of the model in the saliency region have a certain richness. That is, when performing local mesh subdivision corresponding to the saliency area, since the saliency area is an area that reflects more details of the object to be reconstructed, the mesh subdivision for the saliency area not only reduces the number of vertices of the mesh model.
  • the mesh subdivision may be global mesh subdivision and/or local mesh subdivision.
  • FIG. 4 is a three-dimensional mesh
  • FIG. 5 is another schematic flowchart of step S24 of another embodiment of the method for reconstructing a three-dimensional mesh model of the present disclosure, and FIG. 4 corresponds to local mesh subdivision, Figure 5 corresponds to the global mesh subdivision.
  • the step S24 of performing local mesh subdivision on the salient area includes the following steps:
  • Step S241a Take the three-dimensional mesh model before the current mesh subdivision as the first three-dimensional mesh model.
  • the first 3D mesh model is the initial 3D mesh model, and in each subsequent mesh subdivision, the 3D mesh model before this mesh subdivision is used as the first 3D mesh model.
  • 3D mesh model It can be understood that the 3D mesh model before this mesh subdivision may be the result of local mesh subdivision, or may be the result of global mesh subdivision.
  • Step S242a Project the first three-dimensional mesh model to the plane where the target image is located to determine target feature information corresponding to each vertex in the first three-dimensional mesh model.
  • the plane where the target image is located is configured to match and fuse each projected vertex with the corresponding target feature information, so as to obtain target feature information corresponding to each vertex.
  • Step S243a Add at least one new vertex in the target area of the first three-dimensional mesh model.
  • the first three-dimensional mesh model consists of vertices, edges and faces. If this mesh subdivision is a local mesh subdivision, the target area corresponds to a saliency area reflecting the detail area of the object to be reconstructed. After the target area of the first three-dimensional mesh model is determined, at least one new vertex is added to the target area of the first three-dimensional mesh model.
  • the number of new vertices and the specific positions of the new vertices are not specifically limited, and the new vertices are configured to form new edges and faces after being connected. Therefore, the local mesh subdivision of the present disclosure selectively performs mesh subdivision on the detail area of the object to be reconstructed, which can reduce the number of new vertices when the mesh model is deformed.
  • the first three-dimensional mesh model when at least one new vertex is added in the target area of the first three-dimensional mesh model, in the first three-dimensional mesh model, at least one edge located in the saliency area is used as a subdivision edge; At least one new vertex is determined on the split edge to obtain the new vertex of the local mesh subdivision. From several edges in the salient region, at least one edge is selected as a subdivision edge, and at least one new vertex is determined on each subdivision edge. The number of new vertices determined on different subdivision edges can be the same or different. Once connected, new edges and faces are formed and can be used for mesh subdivision. In an application scenario, the subdivision edges may be all subdivision edges of the saliency region, or may be part of the subdivision edges in the saliency region.
  • the final 3D mesh model has better details.
  • a position that satisfies the predetermined conditions is found for each salient point in the first three-dimensional mesh model.
  • the edge of the position condition as the edge to be subdivided, and the salient area includes several salient points, so as to obtain several edges to be subdivided; count the number of times that each edge in the first three-dimensional mesh model is determined as the edge to be subdivided; Edges whose number of times of the edges to be subdivided satisfies the preset subdivision conditions are used as subdivision edges, so as to find the edges to be subdivided for each salient point in the saliency area, and vote for the edges that satisfy a certain positional relationship with the salient points. Edges that meet certain subdivision conditions are used as subdivision edges, thereby further reducing the number of new vertices and reducing the memory usage required for mesh model deformation.
  • the preset position conditions include, but are not limited to, being the closest to the position of the salient point, that is, the edge closest to the position of the salient point as the edge to be subdivided; coincident with the corresponding position of the salient point, that is, the edge where the corresponding position of the salient point is located is used as the edge to be subdivided. Edge to be subdivided. Therefore, to vote for the edge closest to the salient point, a certain proportion or number of edges before the votes are used as the subdivision edge, so that the final subdivision edge is close to the salient point, and the new vertex is closer to the detail position of the object to be reconstructed.
  • the edge to be subdivided determined by different salient points may be the same or different. After all the edges to be subdivided are determined, all the edges to be subdivided may be used as the subdivision edges, or some of the edges to be subdivided may be selected as the subdivision edges, which is not limited herein.
  • the preset subdivision condition is that the number of times of the edge to be subdivided is greater than the preset number of times, or, in the order of times of all edges of the first three-dimensional mesh model from most to least, the number of times of the edge to be subdivided is greater than The number of times is within the previous preset number or the previous preset ratio.
  • Step S244a Using the target feature information of the original vertex of the first three-dimensional mesh model, obtain the target feature information of the new vertex corresponding to the subdivision edge.
  • the new vertex can be determined according to the preset rules.
  • the preset rules include but are not limited to taking the midpoint of the subdivision edge as the new vertex, and the position one third from the left vertex as the new vertex.
  • the average value of the target feature information of the two original vertices is used as the target feature information of the new vertex. Therefore, taking the midpoint of the subdivision edge as the new vertex, it is convenient to obtain the target feature information of the new vertex by using the target feature information of the two original vertices corresponding to the subdivided edge.
  • Step S245a Based on the target feature information of the original vertices and the new vertices of the first 3D mesh model, obtain a second 3D mesh model after this mesh subdivision.
  • the first three-dimensional mesh model is deformed into a second three-dimensional mesh model.
  • the number of vertices of the second three-dimensional mesh model is larger than that of the first three-dimensional mesh model, that is, the The two-dimensional mesh model includes new vertices and the original vertices of the first three-dimensional mesh model, and more vertices reflect the characteristics of the object to be reconstructed, thereby realizing the deformation from coarse to fine. It is understandable that the deformation of the mesh model can continuously take the 3D mesh model before this mesh subdivision as the first 3D mesh model, deform the first 3D mesh model into the second 3D mesh model, and iterate continuously. Mesh subdivision for detail.
  • the second three-dimensional mesh model after this mesh subdivision can be obtained.
  • the second neural network may be used to process the target feature information of the original vertices and the new vertices of the first 3D mesh model to obtain after the first 3D mesh model is deformed into the first 3D mesh model
  • the position information of each vertex is used to deform the first three-dimensional mesh model into a second three-dimensional mesh model by using the second neural network.
  • the position information includes but is not limited to the position offset.
  • the second neural network is any neural network that obtains the position information of each vertex of the three-dimensional grid model through deep learning, and can realize the deformation of the grid model.
  • the second neural network includes but is not limited to a graph convolutional neural network.
  • the number of dimensions of the input layer, hidden layer, and output layer included in the graph convolutional neural network can be customized, which is not specifically limited here.
  • the second neural network is a neural network that can obtain the position information of each vertex.
  • the first three-dimensional mesh model can be deformed multiple times according to the target feature information, so that the vertex positions of the second three-dimensional mesh model are continuously approached to the waiting area. Reconstructs the position of the object's true vertices.
  • the target feature information can be obtained by first projecting the first 3D mesh model, then adding new vertices, and using the original vertices and new vertices of the first 3D mesh model to obtain a subdivided second 3D mesh model to realize the mesh Subdivision, which reflects the details of the object to be reconstructed.
  • mesh subdivision in addition to local mesh subdivision, mesh subdivision also includes global mesh subdivision corresponding to the entire 3D mesh model. As shown in FIG. 5 , after the present disclosure uses the feature information of the target image to determine the saliency area of the object to be reconstructed, when the three-dimensional mesh model is reconstructed, the step S24 of performing global mesh subdivision on the saliency area includes the following steps:
  • Step S241b Take the three-dimensional mesh model before the current mesh subdivision as the first three-dimensional mesh model.
  • Step S242b Project the first three-dimensional mesh model to the plane where the target image is located to determine target feature information corresponding to each vertex in the first three-dimensional mesh model.
  • Step S243b Take each edge in the first three-dimensional mesh model as a subdivision edge respectively; and determine at least one new vertex on the subdivision edge.
  • the target area includes at least a saliency area, this mesh subdivision is a global mesh subdivision, and the target area corresponds to the entire first three-dimensional mesh model. Therefore, each edge in the first three-dimensional mesh model is used as a subdivision edge; at least one new vertex can be determined on the subdivision edge.
  • Step S244b Using the target feature information of the original vertex of the first three-dimensional mesh model, obtain the target feature information of the new vertex corresponding to the subdivision edge.
  • Step S245b Based on the target feature information of the original vertex and the new vertex of the first three-dimensional mesh model, obtain the second three-dimensional mesh model after this mesh subdivision.
  • the target area of the global mesh subdivision and the local mesh subdivision is different, and the target area indicates the area to be meshed. If this mesh subdivision is a global mesh subdivision, the target area corresponds to the entire first three-dimensional mesh model; if this mesh subdivision is a local mesh subdivision, the target area at least includes a saliency area. Except that in step S243b, each edge in the first three-dimensional mesh model is used as a subdivision edge, and at least one new vertex is determined on the subdivision edge, the remaining steps S241b, S242b, S244b and steps of global mesh subdivision are performed S245b can refer to the relevant description of the local mesh subdivision, which is not repeated here.
  • the entire 3D mesh model can be evenly subdivided, and the overall detail accuracy of the 3D mesh model is mentioned.
  • 3D mesh deformation is performed based on local mesh subdivision, only the salient region is subdivided from coarse to fine mesh, which can reduce the memory consumption required for mesh model deformation, and better solve the problem of uniform subdivision leading to final.
  • the problem of over-smoothing of the 3D mesh model makes the details in the salient region have a certain richness.
  • global mesh subdivision and/or local mesh subdivision may be performed, wherein the global mesh subdivision corresponds to the entire 3D mesh model, and the local mesh subdivision corresponds to In the saliency area, if global mesh subdivision and local mesh subdivision are performed, the order of global mesh subdivision and local mesh subdivision and the number of times of each type of mesh subdivision are not specifically limited.
  • the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.
  • FIG. 6 is a schematic frame diagram of an embodiment of a three-dimensional mesh model reconstruction apparatus 60 of the present disclosure.
  • the three-dimensional mesh model reconstruction device 60 includes a feature extraction module 61 , a saliency region determination module 62 , and a model construction module 63 .
  • the feature extraction module 61 is configured to perform feature extraction on the target image to obtain target feature information, wherein the target image contains the object to be reconstructed;
  • the saliency region determination module 62 is configured to determine the salient region of the object to be reconstructed based on the target feature information;
  • the model The building module 63 is configured to build a final three-dimensional mesh model of the object to be reconstructed according to the saliency region.
  • the feature extraction module 61 performs feature extraction on the target image containing the object to be reconstructed to obtain target feature information, so that the saliency region determination module 62 uses the target feature information to determine the salient region of the object to be reconstructed, and then the model building module 63 After acquiring the saliency region, the final 3D mesh model of the object to be reconstructed can be constructed by using the saliency region.
  • the model construction module 63 may further include an initial three-dimensional mesh model construction module and a model acquisition module, so as to construct a final three-dimensional mesh model of the object to be reconstructed by using salient regions.
  • FIG. 7 is a schematic frame diagram of another embodiment of an apparatus 70 for reconstructing a three-dimensional mesh model of the present disclosure.
  • the three-dimensional mesh model reconstruction device 70 includes a feature extraction module 71, a saliency region determination module 72, an initial three-dimensional mesh model construction module 73, and a model acquisition module 74.
  • the feature extraction module 71 is configured to perform feature extraction on the target image to obtain the target image.
  • the saliency region determination module 72 is configured to determine the saliency region of the object to be reconstructed based on the target feature information;
  • the initial 3D mesh model building module 73 is configured to use the target feature information to construct an initial 3D mesh for the object to be reconstructed a grid model;
  • the model acquisition module 74 is configured to perform grid subdivision on the initial three-dimensional grid model to obtain the final three-dimensional grid model of the object to be reconstructed, wherein the grid subdivision includes local grid subdivision corresponding to the saliency region .
  • the saliency area determination module 72 uses the target feature information of the target image to determine the saliency area of the object to be reconstructed, and the model acquisition module 74 performs grid subdivision on the salient area during reconstruction of the three-dimensional mesh model to reduce the number of meshes.
  • the number of vertices of the grid model, and make the details of the grid model in the salient region have a certain richness. That is, when performing local mesh subdivision corresponding to the saliency area, since the saliency area is an area that reflects more details of the object to be reconstructed, the mesh subdivision for the saliency area not only reduces the number of vertices of the mesh model.
  • the saliency region determination module 62 includes a deformation unit and a determination unit, and the deformation unit is configured to use the target feature information to deform the basic point cloud model into a target point cloud corresponding to the object to be reconstructed Model; the determination unit is configured to determine the saliency region of the target point cloud model. Therefore, the saliency area corresponding to the detail area of the object to be reconstructed can be obtained by using the target feature information, and the saliency area can be obtained by deforming the point cloud model.
  • the deformation unit when the deformation unit is configured to use the target feature information to deform the basic point cloud model into a target point cloud model corresponding to the object to be reconstructed, the deformation unit is further configured to project the basic point cloud model to the plane where the target image is located. , to determine the target feature information corresponding to each point in the basic point cloud model; use the first neural network to process the target feature information corresponding to each point in the basic point cloud model, and obtain the target point cloud model after the basic point cloud model is transformed into the target point cloud model. location information of each point.
  • the determining unit When the determining unit is configured to determine the saliency area of the target point cloud model, it is also configured to obtain the point distribution of the target point cloud model; find out the point cloud area where the point distribution in the target point cloud model meets the requirements of the saliency distribution, as salient area. Therefore, the deformation of the point cloud model is realized through the first neural network, and the saliency region is determined by using the point distribution of the target point cloud model.
  • the deformation unit is further configured to evenly sample points in the unit sphere before projecting the basic point cloud model to the plane where the target image is located, so as to obtain the basic point cloud model; the basic point cloud model is deformed as the target.
  • the position information of each point after the point cloud model is: the position offset of each point after the basic point cloud model is deformed into the target point cloud model; the requirements for the saliency distribution include that the point distribution density is greater than the preset density value, so that the salient area is The dense distribution of points in the interior can better reflect the details of the object to be reconstructed.
  • the saliency region determination module 62 further includes a training unit configured to train to obtain the first neural network.
  • the training unit is configured to obtain a real 3D mesh model of a sample image and a sample object, wherein the sample image contains the sample object; perform feature extraction on the sample image to obtain sample feature information; Projection to the plane where the sample image is located to determine the sample feature information corresponding to each point in the basic point cloud model; use the first neural network to process the sample feature information corresponding to each point in the basic point cloud model, and obtain the basic point cloud model deformation as Predict the position information of each point after the point cloud model; simplify the real 3D mesh model to obtain a simplified 3D mesh model; find out the points in the predicted point cloud model that match the vertices of the simplified 3D mesh model, Several groups of matching point pairs are obtained; the parameters of the first neural network are adjusted by using the position difference of each group of matching point pairs.
  • the real 3D mesh model is monitored to ensure that there are fewer patches in the flat area, and then the vertices of the simplified real 3D mesh model are used as supervision signals for training, and the first neural network obtained by training can output the target point. Position information of each point of the cloud model.
  • the model obtaining module 64 includes a determining unit, an adding unit, and an obtaining unit.
  • the model obtaining module 64 is configured to perform grid subdivision on the initial three-dimensional grid model, and when obtaining the final three-dimensional grid model of the object to be reconstructed, determine that the unit is configured to take the three-dimensional grid model before the current grid subdivision as the first three-dimensional grid model.
  • a three-dimensional mesh model further configured to project the first three-dimensional mesh model to the plane where the target image is located, so as to determine the target feature information corresponding to each vertex in the first three-dimensional mesh model;
  • the additional unit is configured to be in the first three-dimensional mesh model At least one new vertex is added to the target area of the model; wherein, if the current mesh subdivision is subdivided into a local mesh, the target area at least includes a salient area;
  • the acquisition unit is configured to use the original vertex of the first three-dimensional mesh model The target feature information of the new vertex corresponding to the subdivision edge is obtained; the acquisition unit is also configured to obtain the target feature information of the original vertex and the new vertex based on the first three-dimensional mesh model, after this mesh subdivision The second 3D mesh model.
  • the target feature information can be obtained by first projecting the first 3D mesh model, then adding new vertices, and using the original vertices and new vertices of the first 3D mesh model to obtain a subdivided second 3D mesh model to realize the mesh Subdivision, which reflects the details of the object to be reconstructed.
  • the adding unit when the adding unit is configured to add at least one new vertex in the target area of the first three-dimensional mesh model, it is also configured to add at least one new vertex in the first three-dimensional mesh model
  • the mesh model at least one edge located in the salient region is used as a subdivision edge; at least one new vertex is determined on the subdivision edge. Therefore, at least one edge of the saliency region is used as a subdivision edge, and new vertices are obtained on the subdivision edge, so as to determine the new vertex when the local mesh is subdivided.
  • the saliency region includes several salient points; the addition unit is configured to, in the first three-dimensional mesh model, use at least one edge located in the saliency region as a subdivision edge, and is also configured to be in the first three-dimensional mesh model.
  • the edge whose position satisfies the preset position condition is found for each significant point as the edge to be subdivided; the number of times each edge in the first three-dimensional grid model is determined as the edge to be subdivided is counted; The number of subdivision edges meets the preset subdivision conditions as the subdivision edges.
  • the preset position condition is that it is closest to the position of the salient point;
  • the preset subdivision condition is that the number of times of the edge to be subdivided is greater than the preset number of times, or, in the order of the times of all edges of the first three-dimensional mesh model from most to least,
  • the number of edges to be subdivided is within the previous preset number or the previous preset ratio. Therefore, find the edge to be subdivided for each salient point in the saliency area, vote for the edge that satisfies a certain positional relationship with the salient point, and use the edge whose votes meet a certain subdivision condition as the subdivision edge, thereby further reducing the number of new vertices.
  • the mesh subdivision further includes performing global mesh subdivision corresponding to the entire three-dimensional mesh model; if the current mesh subdivision is a global mesh subdivision, the additional unit is further configured to be in the first three-dimensional mesh.
  • the mesh subdivision also includes global mesh subdivision corresponding to the entire 3D mesh model, and the mesh model obtained by the global mesh subdivision is more detailed as a whole.
  • the adding unit is further configured to use the midpoint of the subdivision edge as a new vertex;
  • the acquiring unit is configured to use the target feature information of the original vertex of the first three-dimensional mesh model to obtain the new vertex corresponding to the subdivided edge.
  • the target feature information of the subdivision edge is obtained, the target feature information of the new vertex corresponding to the subdivision edge is obtained by using the target feature information of the two original vertices corresponding to the subdivision edge. Therefore, taking the midpoint of the subdivision edge as the new vertex, it is convenient to obtain the target feature information of the new vertex by using the target feature information of the two original vertices corresponding to the subdivided edge.
  • the acquisition unit is configured to obtain the second three-dimensional mesh model after this mesh subdivision based on the target feature information of the original vertex and the new vertex of the first three-dimensional mesh model, the acquisition unit also The second neural network is configured to process the target feature information of the original vertex and the new vertex of the first three-dimensional mesh model to obtain the position information of each vertex after the first three-dimensional mesh model is deformed into the first three-dimensional mesh model. Therefore, the first three-dimensional mesh model is deformed into a second three-dimensional mesh model using the second neural network.
  • the initial 3D mesh model construction module 63 is configured to use the target feature information to construct an initial 3D mesh model of the object to be reconstructed, and is also configured to project the basic 3D mesh model to the plane where the target image is located, Determine the target feature information corresponding to each vertex in the basic 3D mesh model; use the second neural network to process the target feature information corresponding to each vertex in the basic 3D mesh model to obtain the basic 3D mesh model deformed into an initial 3D mesh The position information of each vertex after the model; wherein, the position information of each vertex is the position offset. Therefore, the basic three-dimensional grid model can be deformed into an initial three-dimensional grid model by using the second neural network, so as to complete the initialization of the object to be reconstructed and reflect the initial shape of the object to be reconstructed.
  • the target image is a two-dimensional image
  • the feature extraction module 61 is configured to perform feature extraction on the target image, and when obtaining target feature information, it is also configured to use a third neural network to perform feature extraction on the target image to obtain several The feature information of the dimension; the feature information of several dimensions is fused to obtain the target feature information, wherein the target feature information is a feature tensor. Therefore, the third neural network is used to perform feature extraction on the two-dimensional target image, and a feature tensor indicating the target feature information is obtained.
  • FIG. 8 is a schematic diagram of a framework of an embodiment of an electronic device 80 of the present disclosure.
  • the electronic device 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is configured to execute the program instructions stored in the memory 81 to implement the steps of any of the foregoing three-dimensional mesh model reconstruction method embodiments.
  • the electronic device 80 may include, but is not limited to, a microcomputer and a server.
  • the electronic device 80 may also include mobile devices such as a notebook computer and a tablet computer, which are not limited herein.
  • the processor 82 is configured to control itself and the memory 81 to implement the steps in any of the foregoing three-dimensional mesh model reconstruction method embodiments, or to implement any of the foregoing image detection method embodiments.
  • the processor 82 may also be referred to as a CPU (Central Processing Unit, central processing unit).
  • the processor 82 may be an integrated circuit chip with signal processing capability.
  • the processor 82 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the processor 82 may be jointly implemented by an integrated circuit chip.
  • the target feature information of the target image is used to determine the salient area of the object to be reconstructed, and when the three-dimensional mesh model is reconstructed, the salient area is meshed to reduce the number of vertices of the mesh model and make the mesh
  • the details of the model in the saliency region have a certain richness.
  • FIG. 9 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 90 of the present disclosure.
  • the computer-readable storage medium 90 stores program instructions 901 that can be executed by the processor, and the program instructions 901 are used to implement the steps of any of the foregoing three-dimensional mesh model reconstruction method embodiments.
  • the target feature information of the target image is used to determine the salient area of the object to be reconstructed, and when the three-dimensional mesh model is reconstructed, the salient area is meshed to reduce the number of vertices of the mesh model and make the mesh
  • the details of the model in the saliency region have a certain richness.
  • the disclosed method and apparatus may be implemented in other manners.
  • the device implementations described above are only illustrative.
  • the division of modules or units is only a logical function division. In actual implementation, there may be other divisions.
  • units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.
  • each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the integrated unit if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium.
  • the technical solutions of the present disclosure can be embodied in the form of software products in essence, or the part that contributes to the prior art, or all or part of the technical solutions, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various embodiments of the present disclosure.
  • the aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .
  • the present disclosure provides a method for reconstructing a three-dimensional mesh model, a device, a device, and a storage medium thereof, wherein the method includes: extracting features from a target image to obtain target feature information; wherein the target image contains information to be reconstructed object; based on the target feature information, determine the saliency area of the object to be reconstructed; based on the saliency area, construct a final three-dimensional mesh model of the object to be reconstructed.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Generation (AREA)
  • Image Analysis (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本实施例公开了一三维网格模型的重建方法及其装置、设备、存储介质，该方法包括：对目标图像进行特征提取，得到目标特征信息；其中，所述目标图像包含待重建对象；基于所述目标特征信息，确定所述待重建对象的显著性区域；基于所述显著性区域，构建所述待重建对象的最终三维网格模型。如此，利用反映待重建对象较多细节的显著性区域，构建待重建对象的最终三维网格模型，可灵活重建到细节明显的三维网格模型。

Description

三维网格模型的重建方法及其装置、设备、存储介质
相关申请的交叉引用
本公开基于申请号为202010699880.1、申请日为2020年7月20日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本公开作为参考。
技术领域
本公开涉及人工智能技术领域，特别是涉及一种三维网格模型的重建方法及其装置、设备、存储介质。
背景技术
随着计算机技术发展，对于物体的三维重建已应用于各领域中。三维重建通常是指将三维空间的物体或场景进行恢复和重构的技术，重建的模型可方便计算机表示、处理和显示。在相关技术中，利用物体的多视角图像恢复出三维物体的三维模型中，所体现的物体细节较为模糊。
发明内容
本公开至少提供一种三维网格模型的重建方法及其装置、设备、存储介质。
本公开第一方面提供了一种三维网格模型的重建方法，包括：对目标图像进行特征提取，得到目标特征信息；其中，所述目标图像包含待重建对象；基于所述目标特征信息，确定所述待重建对象的显著性区域；基于所述显著性区域，构建所述待重建对象的最终三维网格模型。
因此，利用目标图像的目标特征信息确定待重建对象的显著性区域后，利用反映待重建对象较多细节的显著性区域，构建待重建对象的最终三维网格模型，可灵活重建到细节明显的三维网格模型。
其中，所述基于所述目标特征信息，确定所述待重建对象的显著性区域，包括：利用所述目标特征信息，将基础点云模型变形为所述待重建对象对应的目标点云模型；确定所述目标点云模型的显著性区域。
因此，能够利用目标特征信息获取对应待重建对象细节区域的显著性区域，实现利用点云模型变形得到显著性区域。
其中，所述利用所述特征信息，将基础点云模型变形为所述待重建对象对应的目标点云模型，包括：将所述基础点云模型投影至所述目标图像所在平面，以确定所述基础点云模型中各点对应的所述目标特征信息；利用第一神经网络对所述基础点云模型中各点对应的目标特征信息进行处理，得到所述基础点云模型变形为所述目标点云模型后的各点的位置信息；所述确定所述目标点云模型的显著性区域，包括：获取所述目标点云模型的点分布情况；查找出所述目标点云模型中所述点分布情况满足显著性分布要求的点云区域，以作为所述显著性区域。
因此，通过第一神经网络实现点云模型变形，并且利用目标点云模型的点分布情况确定显著性区域。
其中，在所述将所述基础点云模型投影至所述目标图像所在平面之前，所述方法还包括：在单位球内均匀采样点，以得到所述基础点云模型；所述基础点云模型变形为所述目标点云模型后的各点的位置信息为：所述基础点云模型变形为所述目标点云模型后的各点的位置偏移量；所述显著性分布要求包括点分布密度大于预设密度值。
因此，对单位球进行均匀采样点即可得到基础点云模型；第一神经网络输出的点位置信息为偏移量，可利用均匀采样点和位置偏移量得到目标点云模型的各点的位置信息； 且可以但不限于通过点分布密度大于预设密度值确定显著性区域，使得显著性区域内的点分布密集，更能够体现待重建对象的细节。
其中，所述方法还包括以下步骤，以训练得到所述第一神经网络：获取样本图像和样本对象的真实三维网格模型，其中，所述样本图像包含所述样本对象；对所述样本图像进行特征提取，得到样本特征信息；将所述基础点云模型投影至所述样本图像所在平面，以确定所述基础点云模型中各点对应的所述样本特征信息；利用第一神经网络对所述基础点云模型中各点对应的样本特征信息进行处理，得到所述基础点云模型变形为所述预测点云模型后的各点的位置信息；对所述真实三维网格模型进行网格简化，得到简化三维网格模型；查找出所述预测点云模型中与所述简化三维网格模型的各顶点匹配的点，得到若干组匹配点对；利用每组匹配点对的位置差异，调整所述第一神经网络的参数。
因此，将真实三维网格模型进行监护，以保证平坦地方的面片比较少，再利用简化后的真实三维网格模型的顶点作为监督信号进行训练，训练得到的第一神经网络可输出目标点云模型的各点的位置信息。
其中，所述基于所述显著性区域，构建所述待重建对象的最终三维网格模型，包括：利用所述目标特征信息构建得到所述待重建对象的初始三维网格模型；对所述初始三维网格模型进行网格细分，得到所述待重建对象的所述最终三维网格模型，其中，所述网格细分包括对应所述显著性区域进行局部网格细分。
因此，在三维网格模型重建时，对显著性区域进行网格细分，以减少网格模型的顶点数量，且使得网格模型在显著性区域的细节具有一定丰富性。也即，在对应显著性区域进行局部网格细分时，由于显著性区域是反映待重建对象较多细节的区域，针对显著性区域进行网格细分，不仅减少了网格模型的顶点数量，进而可减少三维网格模型的数据所需的存储空间，而且使得重建得到的三维网格模型对应显著性区域不会造成过度平滑，能够较好体现细节，从而能够重建较多细节的三维网格模型，且减少三维网格模型的数据所需的存储空间。
其中，所述对所述初始三维网格模型进行网格细分，得到所述待重建对象的最终三维网格模型，包括：以进行本次网格细分之前的三维网格模型为第一三维网格模型；将所述第一三维网格模型投影至所述目标图像所在平面，以确定所述第一三维网格模型中各顶点对应的所述目标特征信息；在所述第一三维网格模型的目标区域中增加至少一个新顶点；其中，所述目标区域至少包括所述显著性区域；利用所述第一三维网格模型的原顶点的目标特征信息，得到所述细分边对应的新顶点的目标特征信息；基于所述第一三维网格模型的原顶点和新顶点的目标特征信息，得到经本次网格细分后的第二三维网格模型。
因此，可先将第一三维网格模型投影得到目标特征信息，然后增加新顶点，利用第一三维网格模型的原顶点和新顶点得到细分后的第二三维网格模型，实现网格细分，体现待重建对象的细节。
其中，若本次网格细分为所述局部网格细分，则所述在所述第一三维网格模型的目标区域中增加至少一个新顶点，包括：在所述第一三维网格模型中，将位于所述显著性区域的至少一条边作为细分边；在所述细分边上确定至少一个新顶点。
因此，将显著性区域的至少一条边作为细分边，在细分边上得到新顶点，从而在局部网格细分时确定新顶点。
其中，所述显著性区域包括若干显著点；所述在所述第一三维网格模型中，将位于所述显著性区域的至少一条边作为细分边，包括：在所述第一三维网格模型中，为每个所述显著点查找出位置满足预设位置条件的边以作为待细分边；统计所述第一三维网格模型中每条边被确定为所述待细分边的次数；将所述待细分边的次数满足预设细分条件 的边作为所述细分边。
因此，为显著性区域内每个显著点查找出待细分边，为与显著点满足一定位置关系的边投票，将票数满足一定细分条件的边作为细分边，从而进一步减少新顶点的数目，减少网格模型形变所需的内存。
其中，所述预设位置条件为与所述显著点的位置最近；所述预设细分条件为所述待细分边的次数大于预设次数，或者，在所述第一三维网格模型的所有边从多到少的次数排序中，所述待细分边的次数位于前预设数量或前预设比例内。
因此，为与显著点最近的边投票，将票数前一定比例或前一定数量的边作为细分边，从而最终的细分边贴近显著点，则新顶点更接近待重建对象的细节位置。
其中，所述网格细分还包括对应所述整个三维网格模型进行全局网格细分；若本次网格细分为所述全局网格细分，则所述在所述第一三维网格模型的目标区域中增加至少一个新顶点，包括：将所述第一三维网格模型中的每条边分别作为细分边；在所述细分边上确定至少一个新顶点。
因此，网格细分还包括对应整个三维网格模型进行全局网格细分，全局网格细分得到的网格模型在整体上更加细节化。
其中，所述在所述细分边上确定至少一个新顶点，包括：将所述细分边的中点作为所述新顶点；所述利用所述第一三维网格模型的原顶点的目标特征信息，得到所述细分边对应的新顶点的目标特征信息，包括：利用所述细分边对应的两个所述原顶点的目标特征信息，得到所述细分边对应的新顶点的目标特征信息。
因此，将细分边的中点为新顶点，方便利用对应细分边的两个原顶点的目标特征信息得到新顶点的目标特征信息。
其中，所述基于所述第一三维网格模型的原顶点和新顶点的目标特征信息，得到经本次网格细分后的第二三维网格模型，包括：利用第二神经网络对所述第一三维网格模型的原顶点和新顶点的目标特征信息进行处理，得到所述第一三维网格模型变形为所述第一三维网格模型后的各顶点的位置信息。
因此，利用第二神经网络将第一三维网格模型变形为第二三维网格模型。
其中，所述利用所述目标特征信息构建得到所述待重建对象的初始三维网格模型，包括：将基础三维网格模型投影至所述目标图像所在平面，以确定所述基础三维网格模型中各顶点对应的所述目标特征信息；利用第二神经网络对所述基础三维网格模型中各顶点对应的目标特征信息进行处理，得到所述基础三维网格模型变形为所述初始三维网格模型后的各顶点的位置信息；其中，所述各顶点的位置信息为位置偏移量。
因此，可利用第二神经网络将基础三维网格模型变形为初始三维网格模型，完成对待重建对象初始化，体现待重建对象的初始形状。
其中，所述目标图像为二维图像；和/或，所述对目标图像进行特征提取，得到目标特征信息，包括：利用第三神经网络对目标图像进行特征提取，得到若干维度的特征信息；将所述若干维度的特征信息融合得到目标特征信息，其中，所述目标特征信息为特征张量。
因此，利用第三神经网络对二维的目标图像进行特征提取，获取到指示于目标特征信息的特征张量。
本公开第二方面提供了一种三维网格模型的重建装置，包括：特征提取模块，配置为对目标图像进行特征提取，得到目标特征信息；其中，所述目标图像包含待重建对象；显著性区域确定模块，配置为基于所述目标特征信息，确定所述待重建对象的显著性区域；模型构建模块，配置为基于所述显著性区域，构建所述待重建对象的最终三维网格模型。
本公开第三方面提供了一种电子设备，包括相互耦接的存储器和处理器，处理器用 于执行存储器中存储的程序指令，以实现上述第一方面中的三维网格模型的重建方法。
本公开第四方面提供了一种计算机可读存储介质，其上存储有程序指令，程序指令被处理器执行时实现上述第一方面中的三维网格模型的重建方法。
上述方案，利用目标图像的目标特征信息确定待重建对象的显著性区域，在三维网格模型重建时，对显著性区域进行网格细分，以减少网格模型的顶点数量，且使得网格模型在显著性区域的细节具有一定丰富性。也即，在对应显著性区域进行局部网格细分时，由于显著性区域是反映待重建对象较多细节的区域，针对显著性区域进行网格细分，不仅减少了网格模型的顶点数量，进而可减少三维网格模型的数据所需的存储空间，而且使得重建得到的三维网格模型对应显著性区域不会造成过度平滑，能够较好体现细节，从而能够重建较多细节的三维网格模型，且减少三维网格模型的数据所需的存储空间。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，而非限制本公开。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开的技术方案。
图1A是本公开三维网格模型的重建方法一实施例的流程示意图；
图1B是本申请实施例三维网格模型的重建方法的一种网络架构示意图；
图2是本公开三维网格模型的重建方法一实施例步骤S12的流程示意图；
图3是本公开三维网格模型的重建方法另一实施例的流程示意图；
图4是本公开三维网格模型的重建方法另一实施例步骤S24的一流程示意图；
图5是本公开三维网格模型的重建方法另一实施例步骤S24的另一流程示意图；
图6是本公开三维网格模型的重建装置一实施例的框架示意图；
图7是本公开三维网格模型的重建装置另一实施例的框架示意图；
图8是本公开电子设备一实施例的框架示意图；
图9是本公开计算机可读存储介质一实施例的框架示意图。
具体实施方式
下面结合说明书附图，对本公开实施例的方案进行详细说明。
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、接口、技术之类的具体细节，以便透彻理解本公开。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。此外，本文中的“多”表示两个或者多于两个。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
请参阅图1A，图1A是本公开三维网格模型的重建方法一实施例的流程示意图。具体而言，可以包括如下步骤：
步骤S11：对目标图像进行特征提取，得到目标特征信息。
本公开实施例中，目标图像包含待重建对象。三维网格模型的重建方法的执行主体可以是三维网格模型的重建装置，例如，三维网格模型的重建方法可以由终端设备或服务器或其它处理设备执行，其中，终端设备可以为用户设备(User Equipment，UE)、移动设备、用户终端、终端、蜂窝电话、无绳电话、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备等。在一些可能的实现方式中，该 三维网格模型的重建方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。
待重建对象可以是飞机、建筑、人、动物等，本公开实施例对此不作具体限定。目标图像可以为二维图像，例如为单视角拍摄的单视图等，对目标图像的具体形式不作具体限定，能够包含待重建对象的特征信息即可。由于目标图像包含待重建对象，因此对目标图像进行特征提取，得到对应于待重建对象的目标特征信息。目标特征信息指示于待重建对象的特征，也即，目标特征信息可包括待重建对象的整体特征信息和局部特征信息，在一实际应用场景中，可自定义选取部分特征信息作为目标特征信息，且目标特征信息的表现形式不作具体限定，包含待重建对象的特征信息即可。
在一公开实施例中，对目标图像进行特征提取，得到目标特征信息时，利用第三神经网络对目标图像进行特征提取，得到若干维度的特征信息；将若干维度的特征信息融合得到目标特征信息。第三神经网络可以为能够进行图像特征提取的任意神经网络，包括但不限于卷积神经网络。因此，利用第三神经网络对目标图像进行特征提取，获取到体现特征信息的目标特征信息。
在一公开实施例中，第三神经网络为卷积神经网络，卷积神经网络包括若干卷积层和池化层，每预设数量卷积层后使用池化层，以在特征提取时实现特征降维。将目标图像输入卷积神经网络，卷积神经网络的第一层卷积层获取到与目标图像尺寸大小相同的特征信息，第二层卷积层获取到目标图像二分之一尺寸大小的特征信息；以此类推，后一层卷积层所得到的特征信息的尺寸大小是邻近前一层卷积层所得到的特征信息的尺寸大小的一半，从而获取到若干维度的特征信息。目标特征信息可以由所有维度的特征信息融合得到，也可以由部分维度的特征信息融合得到，且特征融合的实现方式不作具体限定。若干维度的特征信息对应于不同尺寸大小的情况下，为方便特征信息的融合，将若干维度的特征信息归一化为与目标图像相同尺寸大小的特征信息，将归一化后的特征信息融合得到目标特征信息。
目标特征信息为特征张量，目标图像为二维图像的情况下，为方便后续利用特征张量实现特征信息进行融合，特征张量与二维图像的尺寸大小相同。同样将若干维度的特征信息归一化为与目标图像相同尺寸大小的特征信息，将归一化后的特征信息融合得到特征张量，从而将不同尺寸的特征信息融合为与二维图像的尺寸大小相同的特征张量。
步骤S12：基于目标特征信息，确定待重建对象的显著性区域。
显著性区域指示于待重建对象的曲率变化大、细节较多的区域，例如，待重建对象为飞机，则显著性区域可以为螺旋桨对应区域、机翼与机身连接区域、弯折曲率变化大的区域等，而机身等平滑区域则为非显著性区域；待重建对象为汽车，则显著性区域可以为轮胎、车灯、后视镜等对应区域。目标特征信息包含待重建对象的特征信息，用于后续实现特征信息的融合。显著性区域可通过二维坐标信息、三维坐标信息和颜色标识信息等体现。例如，显著性区域为坐标信息时，可将坐标密集区域作为显著性区域；显著性区域为颜色标识信息时，可通过颜色深浅指示细节多少，将深色的颜色标识信息所在区域作为显著性区域。
为了能够利用目标特征信息获取对应待重建对象细节区域的显著性区域，实现利用点云模型变形得到显著性区域，在一公开实施例中，显著性区域通过点云表达，利用点云预测待重建对象的细节特征分布，使得点云模型的点聚集在显著性区域。图2是本公开三维网格模型的重建方法一实施例步骤S12的流程示意图。具体而言，步骤S12可以包括如下步骤：
步骤S121：利用目标特征信息，将基础点云模型变形为待重建对象对应的目标点云模型。
基础点云模型为预先获取或设定的模型，且基础点云模型上的点均匀分布，任何目 标图像所对应的基础点云模型是一致的。基础点云模型由单位球、椭球三维网格等均匀采样点所得，由不存在连接关系的一个个点构成。基础点云模型均匀分布的点利用目标特征信息融合特征信息后，点的位置发生偏移，使得基础点云模型变形为目标点云模型。目标点云模型上点的分布情况对应于待重建对象，能够反映待重建对象的特征，从而目标点云模型指示于待重建对象的整体形状和细节。在一些可能的实现方式中，将基础点云模型投影至目标图像所在平面，以确定基础点云模型中各点对应的目标特征信息；利用第一神经网络对基础点云模型中各点对应的目标特征信息进行处理，得到基础点云模型变形为目标点云模型后的各点的位置信息，从而通过第一神经网络实现点云模型变形。其中，第一神经网络为通过深度学习得到目标点云模型的各点的位置信息的任意神经网络，能够实现点云模型变形即可，在本公开实施例中，第一神经网络包括但不限于卷积神经网络。
基础点云模型变形为目标点云模型后的各点的位置信息可以为但不限于各点的位置偏移量、位置偏移路径等用于实现点位置更新的数据。在一公开实施例中，基础点云模型变形为目标点云模型后的各点的位置信息可以为各点的位置偏移量，利用基础点云模型的点的位置信息和位置偏移量，计算得到目标点云模型上点的位置信息；在另一公开实施例中，基础点云模型变形为目标点云模型后的各点的位置信息可以为各点的位置偏移路径，例如，位置偏移路径为但不限于拥有方向和长度的矢量值，基础点云模型的点沿着位置偏移路径至目标点云模型上各点的位置信息。
其中，第一神经网络是利用样本图像和样本对象的真实三维网格模型等数据集训练得到的。在一些可能的实现方式中，获取样本图像和样本对象的真实三维网格模型，其中，样本图像包含样本对象；对样本图像进行特征提取，得到样本特征信息；将基础点云模型投影至样本图像所在平面，以确定基础点云模型中各点对应的样本特征信息；利用第一神经网络对基础点云模型中各点对应的样本特征信息进行处理，得到基础点云模型变形为预测点云模型后的各点的位置信息，上述步骤的相关描述与步骤S11和步骤S12类似，在此不再赘述。获取到预测点云模型的各点的位置信息后，对真实三维网格模型进行网格简化，得到简化三维网格模型；查找出预测点云模型中与简化三维网格模型的各顶点匹配的点，得到若干组匹配点对；利用每组匹配点对的位置差异，调整第一神经网络的参数。
真实三维网格模型指示于样本对象的三维网格模型，实际样本图像作为目标图像，利用本公开三维网格模型的重建方法所获得的待重建对象的最终三维网格模型越接近真实三维网格模型，表明三维网格模型的重建方法的还原度越高。上述第一神经网络的训练过程中，将真实三维网格模型进行监护，以保证平坦地方的面片比较少，再利用简化后的真实三维网格模型的顶点作为监督信号进行训练，训练得到的第一神经网络可输出目标点云模型的各点的位置信息。
步骤S122：确定目标点云模型的显著性区域。
目标点云模型上分布着疏密程度不一的点，根据点的分布情况可以确定显著性区域。在一些可能的实现方式中，获取目标点云模型的点分布情况；查找出目标点云模型中点分布情况满足显著性分布要求的点云区域，以作为显著性区域，从而通过点云表达中的点分布情况，利用点云预测待重建对象的细节特征分布，将满足显著性分布要求的点云区域作为显著性区域即可。显著性分布要求可根据需要予以设定，包括但不限于将点分布密度大于预设密度值的区域作为显著性区域，其中，预设密度值可自定义设置。
在一应用实施例中，在单位球内均匀采样点，以得到基础点云模型后，将基础点云模型投影至目标图像所在平面，以确定基础点云模型中各点对应的目标特征信息；利用卷积神经网络对基础点云模型中各点对应的目标特征信息进行处理，得到基础点云模型变形为目标点云模型后的各点的位置偏移量；利用基础点云模型的点的位置信息和位置 偏移量，计算得到目标点云模型上点的位置信息；获取目标点云模型的点分布情况；查找出目标点云模型中点分布密度大于预设密度值的区域作为显著性区域，从而使得显著性区域内的点分布密集，更能够体现待重建对象的细节。
因此，利用目标特征信息，将基础点云模型变形为待重建对象对应的目标点云模型，然后确定目标点云模型的显著性区域，从而能够利用目标特征信息获取对应待重建对象细节区域的显著性区域，实现利用点云模型变形得到显著性区域。
步骤S13：根据显著性区域，构建待重建对象的最终三维网格模型。
最终三维网格模型可通过网格细分、显著点云等单一或组合的三维网格模型构建方法实现得到，在此不作限定。由于显著性区域是反映待重建对象较多细节的区域，可灵活根据显著性区域进行最终三维网格模型的构建，例如但不限于，仅针对显著性区域进行网格细分；或者，先利用目标特征信息构建得到待重建对象的初始三维网格模型，再对初始三维网格模型进行网格细分。
通过上述方式，对包含待重建对象的目标图像进行特征提取，得到目标特征信息，从而利用目标特征信息，确定待重建对象的显著性区域，进而在获取显著性区域后，即可利用显著性区域构建待重建对象的最终三维网格模型，能够灵活利用显著性区域重建到细节明显的三维网格模型。
在本公开实施例中，可以通过如图1B所示的网络架构，实现对待重建对象的三维重建，图1B是本申请实施例三维网格模型的重建方法的一种网络架构示意图，该网络架构中包括：用户终端201、网络202和三维网格模型的重建终端203。为实现支撑一个示例性应用用户终端201和三维网格模型的重建终端203通过网络202建立有通信连接，用户终端201需要对包括待重建对象的目标图像进行三维重建时，首先，将目标图像通过网络202发送至三维网格模型的重建终端203；然后，三维网格模型的重建终端203通过对该目标图像进行特征提取，得到目标特征信息，并基于该目标特征信息，确定待重建对象的显著性区域；最后，三维网格模型的重建终端203通过该显著性区域，实现对待重建对象的三维重建，得到待重建对象的最终三维网格模型。如此，在目标图像上确定待重建对象的显著性区域后，基于细节较多的显著性区域，构建待重建对象的最终三维网格模型，能够得到细节较为明显的最终三维网格模型。
为了在对显著性区域进行网格细分时，减少网格模型的顶点数量，且使得网格模型在显著性区域的细节具有一定丰富性。根据显著性区域，构建待重建对象的最终三维网格模型时，利用目标特征信息构建得到待重建对象的初始三维网格模型；对初始三维网格模型进行网格细分，得到待重建对象的最终三维网格模型，其中，网格细分包括对应显著性区域进行局部网格细分。请参阅图3，图3是本公开三维网格模型的重建方法另一实施例的流程示意图。具体而言，可以包括如下步骤：
步骤S21：对目标图像进行特征提取，得到目标特征信息。
上述步骤S21中，目标图像包含待重建对象。
步骤S22：基于目标特征信息，确定待重建对象的显著性区域。
上述步骤S21和步骤S22的说明可参阅上图1A所示的步骤S11和步骤S12的具体描述，在此不做赘述。
步骤S23：利用目标特征信息构建得到待重建对象的初始三维网格模型。
初始三维网格模型为进行网格形变之前的简单的三维网格模型，体现待重建对象的初始整体形状和细节。初始三维网格模型均由顶点、边和面构成。可以理解的，步骤S22中基于目标特征信息所确定的显著性区域时，是由若干个无连接关系的点分布反映待重建对象的初始整体形状和细节，而步骤S23中利用目标特征信息构建得到的待重建对象的初始三维网格模型时，是由顶点、边和面共同反映待重建对象的初始整体形状和细节。
为利用深度学习实现网格模型形变，在一公开实施例中，将基础三维网格模型投影 至目标图像所在平面，以确定基础三维网格模型中各顶点对应的目标特征信息；利用第二神经网络对基础三维网格模型中各顶点对应的目标特征信息进行处理，得到基础三维网格模型变形为初始三维网格模型后的各顶点的位置信息；其中，各顶点的位置信息可以为位置偏移量，从而可利用第二神经网络将基础三维网格模型变形为初始三维网格模型，完成对待重建对象初始化，体现待重建对象的初始形状。
基础三维网格模型和初始三维网格模型均由顶点、边和面构成，且基础三维网格模型上顶点的位置经偏移后形成初始三维网格模型上顶点的位置，使得原本均匀分布的顶点偏移到与待重建对象整体形状和细节接近的位置上。第二神经网络为通过深度学习得到三维网格模型的各顶点的位置信息的任意神经网络，能够实现网格模型形变即可，具体的，第二神经网络包括但不限于图卷积神经网络。图卷积神经网络包含的输入层、隐含层、输出层的维度数量可自定义设置，在此不作具体限定。第二神经网络为能够获取各顶点位置信息的神经网络，本步骤中，能够根据目标特征信息对基础三维网格模型进行多次形变，使得初始三维网格模型的顶点位置不断逼近待重建对象真实顶点的位置。
在一个实施场景中，上述步骤S22和步骤S23可以按照先后顺序执行，例如，先执行步骤S22，后执行步骤S23；或者，先执行步骤S23，后执行步骤S22。在另一个实施场景中，上述步骤S22和步骤S23还可以同时执行，具体可以根据实际应用进行设置，在此不做限定。
步骤S24：对初始三维网格模型进行网格细分，得到待重建对象的最终三维网格模型。
上述步骤S24中，网格细分包括对应显著性区域进行局部网格细分。
经过步骤S22和步骤S23后，获取到初始三维网格模型和显著性区域，从而利用显著性区域引导初始三维网格模型的网格细分，由于显著性区域对应于待重建对象的部分区域，因此可以仅将对应显著性区域的区域作为网格细分的对象，可以较好的反应出对应显著性区域的细节信息，而较为平整的区域用较大的网格表示，减少内存消耗；进行局部网格细分时，仅在指示于多细节的显著性区域引入网格细分，更为有效的反应待重建对象固有的特征，不会造成过度平滑。为使得模型展现出较真实的几何细节，在一公开实施例中，显著性区域为利用目标特征信息得到的点云分布，初始三维网格模型为利用目标特征信息得到的网格分布，综合利用显著性区域引导对初始三维网格模型的网格细分结合了点云表达和网格表格两种模型表达方式。
可以理解的，在一公开实施例中，对初始三维网格模型进行网格细分时，可进行全局网格细分和局部网格细分，其中，全局网格细分对应整个三维网格模型，而局部网格细分对应显著性区域，且全局网格细分和局部网格细分的顺序和每种网格细分的次数均不作具体限定。
上述方案，利用目标图像的目标特征信息确定待重建对象的显著性区域，在三维网格模型重建时，对显著性区域进行网格细分，以减少网格模型的顶点数量，且使得网格模型在显著性区域的细节具有一定丰富性。也即，在对应显著性区域进行局部网格细分时，由于显著性区域是反映待重建对象较多细节的区域，针对显著性区域进行网格细分，不仅减少了网格模型的顶点数量，进而可减少三维网格模型的数据所需的存储空间，而且使得重建得到的三维网格模型对应显著性区域不会造成过度平滑，能够较好体现细节，从而能够重建较多细节的三维网格模型，且减少三维网格模型的数据所需的存储空间。
在一公开实施例中，网格细分可全局网格细分和/或局部网格细分，为清楚描述本公开最终三维网格模型的网格细分过程，图4是本公开三维网格模型的重建方法另一实施例步骤S24的一流程示意图，图5是本公开三维网格模型的重建方法另一实施例步骤S24的另一流程示意图，且图4对应局部网格细分，图5对应全局网格细分。
如图4所示，本公开利用目标图像的特征信息确定待重建对象的显著性区域后，在三维网格模型重建时，对显著性区域进行局部网格细分中步骤S24包括以下步骤：
步骤S241a：以进行本次网格细分之前的三维网格模型为第一三维网格模型。
在第一次进行网格细分时，第一三维网格模型为初始三维网格模型，后续每次网格细分时，则以本次网格细分之前的三维网格模型作为第一三维网格模型即可。可以理解的，本次网格细分之前的三维网格模型可以是局部网格细分的结果，也可以是全局网格细分的结果。
步骤S242a：将第一三维网格模型投影至目标图像所在平面，以确定第一三维网格模型中各顶点对应的目标特征信息。
目标图像所在平面配置为将投影的各顶点与对应的目标特征信息进行匹配融合，从而得到各顶点对应的目标特征信息。
步骤S243a：在第一三维网格模型的目标区域中增加至少一个新顶点。
第一三维网格模型由顶点、边和面组成。若本次网格细分为局部网格细分，则目标区域对应于反映待重建对象的细节区域的显著性区域。确定第一三维网格模型的目标区域后，在第一三维网格模型的目标区域中增加至少一个新顶点。新顶点的个数以及新顶点的具体位置不作具体限定，且新顶点配置为连接后形成新的边和面。因此本公开局部网格细分选择性地对待重建对象的细节区域进行网格细分，可减少网格模型形变时的新顶点数量。
在一公开实施例中，在第一三维网格模型的目标区域中增加至少一个新顶点时，在第一三维网格模型中，将位于显著性区域的至少一条边作为细分边；在细分边上确定至少一个新顶点，得到局部网格细分的新顶点。从显著性区域内的若干边中，选中至少一条边作为细分边，每条细分边上确定至少一个新顶点，不同细分边上确定的新顶点数量可相同或不同，最终使得新顶点连接后形成新的边和面并可用于网格细分即可。在一应用场景中，细分边可以为显著性区域的所有细分边，可以为显著性区域内的部分细分边。
为使本次网格细分形成的新顶点更接近待重建对象的实际顶点位置，进而使得最终的三维网格模型细节更好。在一公开实施例中，在第一三维网格模型中，将位于显著性区域的至少一条边作为细分边时，在第一三维网格模型中，为每个显著点查找出位置满足预设位置条件的边以作为待细分边，显著性区域包括若干显著点，从而获取若干待细分边；统计第一三维网格模型中每条边被确定为待细分边的次数；将待细分边的次数满足预设细分条件的边作为细分边，从而为显著性区域内每个显著点查找出待细分边，为与显著点满足一定位置关系的边投票，将票数满足一定细分条件的边作为细分边，从而进一步减少新顶点的数目，减少网格模型形变所需的内存占用。
预设位置条件包括但不限于为与显著点的位置最近，也即将与显著点的位置最近的边作为待细分边；与显著点对应位置重合，也即，显著点对应位置所在的边作为待细分边。因此，为与显著点最近的边投票，将票数前一定比例或前一定数量的边作为细分边，从而最终的细分边贴近显著点，则新顶点更接近待重建对象的细节位置。
由于显著性区域包括若干显著点。每个显著点根据预设位置条件获取待细分边后，不同显著点所确定的待细分边可能相同或不同。确定所有待细分边后，可将所有待细分边作为细分边，也可以选择部分待细分边作为细分边，在此不作限定。在一公开实施例中，预设细分条件为待细分边的次数大于预设次数，或者，在第一三维网格模型的所有边从多到少的次数排序中，待细分边的次数位于前预设数量或前预设比例内。
步骤S244a：利用第一三维网格模型的原顶点的目标特征信息，得到细分边对应的新顶点的目标特征信息。
在确定细分边后可依据预设规则确定新顶点，预设规则包括但不限于可将细分边的中点作为新顶点，距离左侧顶点三分之一位置作为新顶点等，从而利用第一三维网格模型的原顶点的目标特征信息，得到细分边对应的新顶点的目标特征信息时，具体为利用细分边对应的两个原顶点的目标特征信息，得到细分边对应的新顶点的目标特征信息。
在一公开实施例中，将细分边的中点作为新顶点时，将两个原顶点的目标特征信息的平均值作为新顶点的目标特征信息。因此，将细分边的中点为新顶点，方便利用对应细分边的两个原顶点的目标特征信息得到新顶点的目标特征信息。
步骤S245a：基于第一三维网格模型的原顶点和新顶点的目标特征信息，得到经本次网格细分后的第二三维网格模型。
网格细分时，将第一三维网格模型变形为第二三维网格模型，在一些可能的实现方式中，第二三维网格模型的顶点数量大于第一三维网格模型，也即第二三维网格模型包括新顶点以及第一三维网格模型的原顶点，通过更多顶点反映待重建对象的特征，实现由粗到细的形变。可以理解的，网格模型形变可以不断以进行本次网格细分之前的三维网格模型为第一三维网格模型，将第一三维网格模型变形为第二三维网格模型，不断迭代进行网格细分以实现细节体现。
基于第一三维网格模型的原顶点和新顶点的目标特征信息，即可得到经本次网格细分后的第二三维网格模型。在一些可能的实现方式中，可利用第二神经网络对第一三维网格模型的原顶点和新顶点的目标特征信息进行处理，得到第一三维网格模型变形为第一三维网格模型后的各顶点的位置信息，从而利用第二神经网络将第一三维网格模型变形为第二三维网格模型。其中，位置信息包括但不限于位置偏移量。第二神经网络为通过深度学习得到三维网格模型的各顶点的位置信息的任意神经网络，能够实现网格模型形变即可，具体的，第二神经网络包括但不限于图卷积神经网络。图卷积神经网络包含的输入层、隐含层、输出层的维度数量可自定义设置，在此不作具体限定。第二神经网络为能够获取各顶点位置信息的神经网络，在本步骤中，能够根据目标特征信息对第一三维网格模型进行多次形变，使得第二三维网格模型的顶点位置不断逼近待重建对象真实顶点的位置。
因此，可先将第一三维网格模型投影得到目标特征信息，然后增加新顶点，利用第一三维网格模型的原顶点和新顶点得到细分后的第二三维网格模型，实现网格细分，体现待重建对象的细节。
在一公开实施例中，除了局部网格细分，网格细分还包括对应整个三维网格模型进行全局网格细分。如图5所示，本公开利用目标图像的特征信息确定待重建对象的显著性区域后，在三维网格模型重建时，对显著性区域进行全局网格细分中步骤S24包括以下步骤：
步骤S241b：以进行本次网格细分之前的三维网格模型为第一三维网格模型。
步骤S242b：将第一三维网格模型投影至目标图像所在平面，以确定第一三维网格模型中各顶点对应的目标特征信息。
步骤S243b：将第一三维网格模型中的每条边分别作为细分边；在细分边上确定至少一个新顶点。
与对第一三维网格模型进行局部网格细分时目标区域至少包括显著性区域不同，本次网格细分为全局网格细分，目标区域对应于整个第一三维网格模型。因此，将第一三维网格模型中的每条边分别作为细分边；在细分边上确定至少一个新顶点即可。
步骤S244b：利用第一三维网格模型的原顶点的目标特征信息，得到细分边对应的新顶点的目标特征信息。
步骤S245b：基于第一三维网格模型的原顶点和新顶点的目标特征信息，得到经本次网格细分后的第二三维网格模型。
可以理解的，全局网格细分和局部网格细分的目标区域不同，目标区域指示于要进行网格细分的区域。若本次网格细分为全局网格细分，目标区域对应于整个第一三维网格模型；若本次网格细分为局部网格细分，目标区域至少包括显著性区域。除步骤S243b中将第一三维网格模型中的每条边分别作为细分边；在细分边上确定至少一个新顶点外， 全局网格细分其余步骤S241b、步骤S242b、步骤S244b和步骤S245b可参考局部网格细分的相关描述，在此不做赘述。
基于全局网格细分做由粗到细的三维网格形变时，可均匀细分整个三维网格模型，整体上提到三维网格模型的细节精度。基于局部网格细分做三维网格形变时，仅对显著性区域做由粗到细的网格细分，能够减少网格模型形变时所需的内存消耗，较好解决均匀细分导致最终的三维网格模型过度平滑的问题，使得在显著性区域的细节具有一定丰富性。
对初始三维网格模型进行网格细分时，可进行全局网格细分和/或局部网格细分，其中，全局网格细分对应整个三维网格模型，而局部网格细分对应显著性区域，若进行全局网格细分和局部网格细分，则全局网格细分和局部网格细分的顺序和每种网格细分的次数均不作具体限定。
本领域技术人员可以理解，在具体实施方式的上述方法中，各步骤的撰写顺序并不意味着严格的执行顺序而对实施过程构成任何限定，各步骤的具体执行顺序应当以其功能和可能的内在逻辑确定。
请参阅图6，图6是本公开三维网格模型的重建装置60一实施例的框架示意图。三维网格模型的重建装置60包括特征提取模块61、显著性区域确定模块62、模型构建模块63。特征提取模块61配置为对目标图像进行特征提取，得到目标特征信息，其中，目标图像包含待重建对象；显著性区域确定模块62配置为基于目标特征信息，确定待重建对象的显著性区域；模型构建模块63配置为根据显著性区域，构建待重建对象的最终三维网格模型。
上述方案，特征提取模块61对包含待重建对象的目标图像进行特征提取，得到目标特征信息，从而显著性区域确定模块62利用目标特征信息，确定待重建对象的显著性区域，进而模型构建模块63在获取显著性区域后，即可利用显著性区域构建待重建对象的最终三维网格模型。
在一些实施例中，该模型构建模块63可进一步包括初始三维网格模型构建模块和模型获取模块，以实现利用显著性区域构建待重建对象的最终三维网格模型。例如，请参阅图7，图7是本公开三维网格模型的重建装置70另一实施例的框架示意图。三维网格模型的重建装置70包括特征提取模块71、显著性区域确定模块72、初始三维网格模型构建模块73和模型获取模块74，特征提取模块71配置为对目标图像进行特征提取，得到目标特征信息；显著性区域确定模块72，配置为基于目标特征信息，确定待重建对象的显著性区域；初始三维网格模型构建模块73，配置为利用目标特征信息构建得到待重建对象的初始三维网格模型；模型获取模块74，配置为对初始三维网格模型进行网格细分，得到待重建对象的最终三维网格模型，其中，网格细分包括对应显著性区域进行局部网格细分。
上述方案，显著性区域确定模块72利用目标图像的目标特征信息确定待重建对象的显著性区域，模型获取模块74在三维网格模型重建时，对显著性区域进行网格细分，以减少网格模型的顶点数量，且使得网格模型在显著性区域的细节具有一定丰富性。也即，在对应显著性区域进行局部网格细分时，由于显著性区域是反映待重建对象较多细节的区域，针对显著性区域进行网格细分，不仅减少了网格模型的顶点数量，进而可减少三维网格模型的数据所需的存储空间，而且使得重建得到的三维网格模型对应显著性区域不会造成过度平滑，能够较好体现细节，从而能够重建较多细节的三维网格模型。
请继续参阅图6，在一公开实施例中，显著性区域确定模块62包括变形单元和确定单元，变形单元配置为利用目标特征信息，将基础点云模型变形为待重建对象对应的目标点云模型；确定单元配置为确定目标点云模型的显著性区域。因此，能够利用目标特征信息获取对应待重建对象细节区域的显著性区域，实现利用点云模型变形得到显著性区 域。
在一公开实施例中，变形单元配置为利用目标特征信息，将基础点云模型变形为待重建对象对应的目标点云模型时，变形单元还配置为将基础点云模型投影至目标图像所在平面，以确定基础点云模型中各点对应的目标特征信息；利用第一神经网络对基础点云模型中各点对应的目标特征信息进行处理，得到基础点云模型变形为目标点云模型后的各点的位置信息。确定单元配置为确定目标点云模型的显著性区域时，还配置为获取目标点云模型的点分布情况；查找出目标点云模型中点分布情况满足显著性分布要求的点云区域，以作为显著性区域。因此，通过第一神经网络实现点云模型变形，并且利用目标点云模型的点分布情况确定显著性区域。
在一公开实施例中，变形单元还配置为将基础点云模型投影至目标图像所在平面之前，还配置为在单位球内均匀采样点，以得到基础点云模型；基础点云模型变形为目标点云模型后的各点的位置信息为：基础点云模型变形为目标点云模型后的各点的位置偏移量；显著性分布要求包括点分布密度大于预设密度值，使得显著性区域内的点分布密集，更能够体现待重建对象的细节。
在一公开实施例中，显著性区域确定模块62还包括训练单元，配置为训练得到第一神经网络。在一些可能的实现方式中，训练单元配置为获取样本图像和样本对象的真实三维网格模型，其中，样本图像包含样本对象；对样本图像进行特征提取，得到样本特征信息；将基础点云模型投影至样本图像所在平面，以确定基础点云模型中各点对应的样本特征信息；利用第一神经网络对基础点云模型中各点对应的样本特征信息进行处理，得到基础点云模型变形为预测点云模型后的各点的位置信息；对真实三维网格模型进行网格简化，得到简化三维网格模型；查找出预测点云模型中与简化三维网格模型的各顶点匹配的点，得到若干组匹配点对；利用每组匹配点对的位置差异，调整第一神经网络的参数。因此，将真实三维网格模型进行监护，以保证平坦地方的面片比较少，再利用简化后的真实三维网格模型的顶点作为监督信号进行训练，训练得到的第一神经网络可输出目标点云模型的各点的位置信息。
在一公开实施例中，模型获取模块64包括确定单元、增加单元、获取单元。模型获取模块64配置为对初始三维网格模型进行网格细分，得到待重建对象的最终三维网格模型时，确定单元配置为以进行本次网格细分之前的三维网格模型为第一三维网格模型，还配置为将第一三维网格模型投影至目标图像所在平面，以确定第一三维网格模型中各顶点对应的目标特征信息；增加单元配置为在第一三维网格模型的目标区域中增加至少一个新顶点；其中，若本次网格细分为局部网格细分，则目标区域至少包括显著性区域；获取单元配置为利用第一三维网格模型的原顶点的目标特征信息，得到细分边对应的新顶点的目标特征信息；获取单元还配置为基于第一三维网格模型的原顶点和新顶点的目标特征信息，得到经本次网格细分后的第二三维网格模型。因此，可先将第一三维网格模型投影得到目标特征信息，然后增加新顶点，利用第一三维网格模型的原顶点和新顶点得到细分后的第二三维网格模型，实现网格细分，体现待重建对象的细节。
在一公开实施例中，若本次网格细分为局部网格细分，增加单元配置为在第一三维网格模型的目标区域中增加至少一个新顶点时，还配置为在第一三维网格模型中，将位于显著性区域的至少一条边作为细分边；在细分边上确定至少一个新顶点。因此，将显著性区域的至少一条边作为细分边，在细分边上得到新顶点，从而在局部网格细分时确定新顶点。
在一公开实施例中，显著性区域包括若干显著点；增加单元配置为在第一三维网格模型中，将位于显著性区域的至少一条边作为细分边时，还配置为在第一三维网格模型中，为每个显著点查找出位置满足预设位置条件的边以作为待细分边；统计第一三维网格模型中每条边被确定为待细分边的次数；将待细分边的次数满足预设细分条件的边作 为细分边。预设位置条件为与显著点的位置最近；预设细分条件为待细分边的次数大于预设次数，或者，在第一三维网格模型的所有边从多到少的次数排序中，待细分边的次数位于前预设数量或前预设比例内。因此，为显著性区域内每个显著点查找出待细分边，为与显著点满足一定位置关系的边投票，将票数满足一定细分条件的边作为细分边，从而进一步减少新顶点的数目，减少内存；还可为与显著点最近的边投票，将票数前一定比例或前一定数量的边作为细分边，从而最终的细分边贴近显著点，则新顶点更接近待重建对象的细节位置。
在一公开实施例中，网格细分还包括对应整个三维网格模型进行全局网格细分；若本次网格细分为全局网格细分，增加单元还配置为在第一三维网格模型的目标区域中增加至少一个新顶点时，还配置为将第一三维网格模型中的每条边分别作为细分边；在细分边上确定至少一个新顶点。因此，网格细分还包括对应整个三维网格模型进行全局网格细分，全局网格细分得到的网格模型在整体上更加细节化。
在一公开实施例中，增加单元还配置为将细分边的中点作为新顶点；获取单元配置为利用第一三维网格模型的原顶点的目标特征信息，得到细分边对应的新顶点的目标特征信息时，还配置为利用细分边对应的两个原顶点的目标特征信息，得到细分边对应的新顶点的目标特征信息。因此，将细分边的中点为新顶点，方便利用对应细分边的两个原顶点的目标特征信息得到新顶点的目标特征信息。
在一公开实施例中，获取单元配置为基于第一三维网格模型的原顶点和新顶点的目标特征信息，得到经本次网格细分后的第二三维网格模型时，获取单元还配置为利用第二神经网络对第一三维网格模型的原顶点和新顶点的目标特征信息进行处理，得到第一三维网格模型变形为第一三维网格模型后的各顶点的位置信息。因此，利用第二神经网络将第一三维网格模型变形为第二三维网格模型。
在一公开实施例中，初始三维网格模型构建模块63配置为利用目标特征信息构建得到待重建对象的初始三维网格模型时，还配置为将基础三维网格模型投影至目标图像所在平面，以确定基础三维网格模型中各顶点对应的目标特征信息；利用第二神经网络对基础三维网格模型中各顶点对应的目标特征信息进行处理，得到基础三维网格模型变形为初始三维网格模型后的各顶点的位置信息；其中，各顶点的位置信息为位置偏移量。因此，可利用第二神经网络将基础三维网格模型变形为初始三维网格模型，完成对待重建对象初始化，体现待重建对象的初始形状。
在一公开实施例中，目标图像为二维图像，特征提取模块61配置为对目标图像进行特征提取，得到目标特征信息时，还配置为利用第三神经网络对目标图像进行特征提取，得到若干维度的特征信息；将若干维度的特征信息融合得到目标特征信息，其中，目标特征信息为特征张量。因此，利用第三神经网络对二维的目标图像进行特征提取，获取到指示于目标特征信息的特征张量。
请参阅图8，图8是本公开电子设备80一实施例的框架示意图。电子设备80包括相互耦接的存储器81和处理器82，处理器82用于执行存储器81中存储的程序指令，以实现上述任一三维网格模型的重建方法实施例的步骤。在一个具体的实施场景中，电子设备80可以包括但不限于：微型计算机、服务器，此外，电子设备80还可以包括笔记本电脑、平板电脑等移动设备，在此不做限定。
具体而言，处理器82用于控制其自身以及存储器81以实现上述任一三维网格模型的重建方法实施例的步骤，或实现上述任一图像检测方法实施例中的步骤。处理器82还可以称为CPU(Central Processing Unit，中央处理单元)。处理器82可能是一种集成电路芯片，具有信号的处理能力。处理器82还可以是通用处理器、数字信号处理器(Digital Signal Processor,DSP)、专用集成电路(Application Specific Integrated Circuit,ASIC)、现场可编程门阵列(Field-Programmable Gate Array,FPGA)或者其他可编程逻辑器件、分立门或 者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。另外，处理器82可以由集成电路芯片共同实现。
上述方案，利用目标图像的目标特征信息确定待重建对象的显著性区域，在三维网格模型重建时，对显著性区域进行网格细分，以减少网格模型的顶点数量，且使得网格模型在显著性区域的细节具有一定丰富性。
请参阅图9，图9是本公开计算机可读存储介质90一实施例的框架示意图。计算机可读存储介质90存储有能够被处理器运行的程序指令901，程序指令901用于实现上述任一三维网格模型的重建方法实施例的步骤。
上述方案，利用目标图像的目标特征信息确定待重建对象的显著性区域，在三维网格模型重建时，对显著性区域进行网格细分，以减少网格模型的顶点数量，且使得网格模型在显著性区域的细节具有一定丰富性。
可以理解的是，本公开实施例提供的装置和设备所具有的功能或包含的模块可以用于执行上文方法实施例描述的方法，其具体实现可以参照上文方法实施例的描述，为了简洁，这里不再赘述。
上文对各个实施例的描述倾向于强调各个实施例之间的不同之处，其相同或相似之处可以互相参考，为了简洁，本文不再赘述。
在本公开所提供的几个实施例中，应该理解到，所揭露的方法和装置，可以通过其它的方式实现。例如，以上所描述的装置实施方式仅仅是示意性的，例如，模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性、机械或其它的形式。
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本公开各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
工业实用性
本公开提供了一种三维网格模型的重建方法及其装置、设备、存储介质，其中，所述方法包括：对目标图像进行特征提取，得到目标特征信息；其中，所述目标图像包含待重建对象；基于所述目标特征信息，确定所述待重建对象的显著性区域；基于所述显著性区域，构建所述待重建对象的最终三维网格模型。

Claims (18)

  1. 一种三维网格模型的重建方法，包括：
    对目标图像进行特征提取，得到目标特征信息；其中，所述目标图像包含待重建对象；
    基于所述目标特征信息，确定所述待重建对象的显著性区域；
    基于所述显著性区域，构建所述待重建对象的最终三维网格模型。
  2. 根据权利要求1所述的方法，所述基于所述目标特征信息，确定所述待重建对象的显著性区域，包括：
    利用所述目标特征信息，将基础点云模型变形为所述待重建对象对应的目标点云模型；
    确定所述目标点云模型的显著性区域。
  3. 根据权利要求2所述的方法，所述利用所述特征信息，将基础点云模型变形为所述待重建对象对应的目标点云模型，包括：
    将所述基础点云模型投影至所述目标图像所在平面，以确定所述基础点云模型中各点对应的目标特征信息；
    利用第一神经网络对所述基础点云模型中各点对应的目标特征信息进行处理，得到所述基础点云模型变形为所述目标点云模型后的各点的位置信息；
    所述确定所述目标点云模型的显著性区域，包括：
    获取所述目标点云模型的点分布情况；
    查找出所述目标点云模型中所述点分布情况满足显著性分布要求的点云区域，以作为所述显著性区域。
  4. 根据权利要求3所述的方法，在所述将所述基础点云模型投影至所述目标图像所在平面之前，所述方法还包括：
    在单位球内均匀采样点，以得到所述基础点云模型；
    所述基础点云模型变形为所述目标点云模型后的各点的位置信息为：所述基础点云模型变形为所述目标点云模型后的各点的位置偏移量；
    所述显著性分布要求包括点分布密度大于预设密度值。
  5. 根据权利要求3或4所述的方法，所述方法还包括以下步骤，以训练得到所述第一神经网络：
    获取样本图像和样本对象的真实三维网格模型，其中，所述样本图像包含所述样本对象；
    对所述样本图像进行特征提取，得到样本特征信息；
    将所述基础点云模型投影至所述样本图像所在平面，以确定所述基础点云模型中各点对应的所述样本特征信息；
    利用第一神经网络对所述基础点云模型中各点对应的样本特征信息进行处理，得到所述基础点云模型变形为所述预测点云模型后的各点的位置信息；
    对所述真实三维网格模型进行网格简化，得到简化三维网格模型；
    查找出所述预测点云模型中与所述简化三维网格模型的各顶点匹配的点，得到若干组匹配点对；
    利用每组匹配点对的位置差异，调整所述第一神经网络的参数。
  6. 根据权利要求1至5任一项所述的方法，所述基于所述显著性区域，构建所述待重建对象的最终三维网格模型，包括：
    利用所述目标特征信息，构建得到所述待重建对象的初始三维网格模型；
    对所述初始三维网格模型进行网格细分，得到所述待重建对象的所述最终三维网格模型，其中，所述网格细分包括对应所述显著性区域进行局部网格细分。
  7. 根据权利要求6所述的方法，所述对所述初始三维网格模型进行网格细分，得到所述待重建对象的所述最终三维网格模型，包括：
    以进行本次网格细分之前的三维网格模型为第一三维网格模型；
    将所述第一三维网格模型投影至所述目标图像所在平面，以确定所述第一三维网格模型中各顶点对应的所述目标特征信息；
    在所述第一三维网格模型的目标区域中增加至少一个新顶点；其中，所述目标区域至少包括所述显著性区域；
    利用所述第一三维网格模型的原顶点的目标特征信息，得到所述细分边对应的新顶点的目标特征信息；
    基于所述第一三维网格模型的原顶点和新顶点的目标特征信息，得到经本次网格细分后的第二三维网格模型。
  8. 根据权利要求7所述的方法，若本次网格细分为所述局部网格细分，则所述在所述第一三维网格模型的目标区域中增加至少一个新顶点，包括：
    在所述第一三维网格模型中，将位于所述显著性区域的至少一条边作为细分边；
    在所述细分边上确定至少一个新顶点。
  9. 根据权利要求8所述的方法，所述显著性区域包括若干显著点；所述在所述第一三维网格模型中，将位于所述显著性区域的至少一条边作为细分边，包括：
    在所述第一三维网格模型中，为每个所述显著点查找出位置满足预设位置条件的边以作为待细分边；
    统计所述第一三维网格模型中每条边被确定为所述待细分边的次数；
    将所述待细分边的次数满足预设细分条件的边作为所述细分边。
  10. 根据权利要求9所述的方法，所述预设位置条件为与所述显著点的位置最近；
    所述预设细分条件为所述待细分边的次数大于预设次数，或者，在所述第一三维网格模型的所有边从多到少的次数排序中，所述待细分边的次数位于前预设数量或前预设比例内。
  11. 根据权利要求6至10任一项所述的方法，所述网格细分还包括对应所述整个三维网格模型进行全局网格细分；若本次网格细分为所述全局网格细分，则所述在所述第一三维网格模型的目标区域中增加至少一个新顶点，包括：
    将所述第一三维网格模型中的每条边分别作为细分边；
    在所述细分边上确定至少一个新顶点。
  12. 根据权利要求11所述的方法，所述在所述细分边上确定至少一个新顶点，包括：
    将所述细分边的中点作为所述新顶点；
    所述利用所述第一三维网格模型的原顶点的目标特征信息，得到所述细分边对应的新顶点的目标特征信息，包括：
    利用所述细分边对应的两个所述原顶点的目标特征信息，得到所述细分边对应的新顶点的目标特征信息。
  13. 根据权利要求7至12任一项所述的方法，所述基于所述第一三维网格模型的原顶点和新顶点的目标特征信息，得到经本次网格细分后的第二三维网格模型，包括：
    利用第二神经网络对所述第一三维网格模型的原顶点和新顶点的目标特征信息进行处理，得到所述第一三维网格模型变形为所述第一三维网格模型后的各顶点的位置信息。
  14. 根据权利要求6至13任一项所述的方法，所述利用所述目标特征信息构建得到所述待重建对象的初始三维网格模型，包括：
    将基础三维网格模型投影至所述目标图像所在平面，以确定所述基础三维网格模型中各顶点对应的所述目标特征信息；
    利用第二神经网络对所述基础三维网格模型中各顶点对应的目标特征信息进行处理，得到所述基础三维网格模型变形为所述初始三维网格模型后的各顶点的位置信息；
    其中，所述各顶点的位置信息为位置偏移量。
  15. 根据权利要求1至14任一项所述的方法，所述目标图像为二维图像；和/或，
    所述对目标图像进行特征提取，得到目标特征信息，包括：
    利用第三神经网络对目标图像进行特征提取，得到若干维度的特征信息；
    将所述若干维度的特征信息融合得到目标特征信息，其中，所述目标特征信息为特征张量。
  16. 一种三维网格模型的重建装置，包括：
    特征提取模块，配置为对目标图像进行特征提取，得到目标特征信息；其中，所述目标图像包含待重建对象；
    显著性区域确定模块，配置为基于所述目标特征信息，确定所述待重建对象的显著性区域；
    模型构建模块，配置为基于所述显著性区域，构建所述待重建对象的最终三维网格模型。
  17. 一种电子设备，包括相互耦接的存储器和处理器，所述处理器用于执行所述存储器中存储的程序指令，以实现权利要求1至15任一项所述的三维网格模型的重建方法。
  18. 一种计算机可读存储介质，其上存储有程序指令，所述程序指令被处理器执行时实现权利要求1至15任一项所述的三维网格模型的重建方法。
PCT/CN2021/078485 2020-07-20 2021-03-01 三维网格模型的重建方法及其装置、设备、存储介质 WO2022016867A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
KR1020227002922A KR20220028010A (ko) 2020-07-20 2021-03-01 3차원 메쉬 모델의 재구축 방법 및 장치, 기기, 저장 매체
JP2021568963A JP7395617B2 (ja) 2020-07-20 2021-03-01 三次元メッシュモデルの再構築方法およびその装置、機器、並びに記憶媒体

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010699880.1 2020-07-20
CN202010699880.1A CN111882666B (zh) 2020-07-20 2020-07-20 三维网格模型的重建方法及其装置、设备、存储介质

Publications (1)

Publication Number Publication Date
WO2022016867A1 true WO2022016867A1 (zh) 2022-01-27

Family

ID=73154922

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/078485 WO2022016867A1 (zh) 2020-07-20 2021-03-01 三维网格模型的重建方法及其装置、设备、存储介质

Country Status (5)

Country Link
JP (1) JP7395617B2 (zh)
KR (1) KR20220028010A (zh)
CN (1) CN111882666B (zh)
TW (1) TW202205215A (zh)
WO (1) WO2022016867A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114741918A (zh) * 2022-02-24 2022-07-12 西北大学 一种面向遗址劣化有限元分析的并行网格剖分算法
CN114758337A (zh) * 2022-06-16 2022-07-15 山东海量信息技术研究院 一种语义实例重建方法、装置、设备及介质
CN116680778A (zh) * 2023-04-27 2023-09-01 清华大学 建筑结构布置生成方法、系统及生成模型的构建方法

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111882666B (zh) * 2020-07-20 2022-06-21 浙江商汤科技开发有限公司 三维网格模型的重建方法及其装置、设备、存储介质
CN113298948B (zh) * 2021-05-07 2022-08-02 中国科学院深圳先进技术研究院 三维网格重建方法、装置、设备及存储介质
CN113238249B (zh) * 2021-05-17 2021-12-28 湖南力研光电科技有限公司 一种水下实时三维成像方法及雷达装置
CN113706431B (zh) * 2021-08-26 2022-10-21 深圳市慧鲤科技有限公司 模型优化方法及相关装置、电子设备和存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130100128A1 (en) * 2008-03-31 2013-04-25 Microsoft Using photo collections for three dimensional modeling
CN105590335A (zh) * 2014-10-23 2016-05-18 富泰华工业（深圳）有限公司 点云网格精细化系统及方法
CN106355646A (zh) * 2016-09-13 2017-01-25 广州市香港科大霍英东研究院 应用于图像处理的表面细化方法、装置以及系统
CN107924571A (zh) * 2015-08-14 2018-04-17 汤姆逊许可公司 从点云中对人耳进行三维重建
CN110443810A (zh) * 2019-07-17 2019-11-12 北京建筑大学 基于快速邻接体素查询的点云平面分割方法
CN111882666A (zh) * 2020-07-20 2020-11-03 浙江商汤科技开发有限公司 三维网格模型的重建方法及其装置、设备、存储介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108765550B (zh) * 2018-05-09 2021-03-30 华南理工大学 一种基于单张图片的三维人脸重建方法
CN109191584B (zh) * 2018-08-16 2020-09-18 Oppo广东移动通信有限公司 三维模型处理方法、装置、电子设备及可读存储介质
CN109035380B (zh) * 2018-09-11 2023-03-10 北京旷视科技有限公司 基于三维重建的人脸修饰方法、装置、设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130100128A1 (en) * 2008-03-31 2013-04-25 Microsoft Using photo collections for three dimensional modeling
CN105590335A (zh) * 2014-10-23 2016-05-18 富泰华工业（深圳）有限公司 点云网格精细化系统及方法
CN107924571A (zh) * 2015-08-14 2018-04-17 汤姆逊许可公司 从点云中对人耳进行三维重建
CN106355646A (zh) * 2016-09-13 2017-01-25 广州市香港科大霍英东研究院 应用于图像处理的表面细化方法、装置以及系统
CN110443810A (zh) * 2019-07-17 2019-11-12 北京建筑大学 基于快速邻接体素查询的点云平面分割方法
CN111882666A (zh) * 2020-07-20 2020-11-03 浙江商汤科技开发有限公司 三维网格模型的重建方法及其装置、设备、存储介质

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"Authoritative Guide on Finite Element Analysis with ANSYS 14.0", 30 June 2013, MACHINERY INDUSTRY PRESS, CN, ISBN: 978-7-111-42499-4, article WEN, ZHENG ET AL.: "Section 3.5.2. Local Mesh Refinement", pages: 107 - 109, XP009533846 *
CARSTEN STOLL, KARNI ZACHI, RÖSSL CHRISTIAN, YAMAUCHI HITOSHI, SEIDEL HANS-PETER: "Template Deformation for Point Cloud Fitting", EUROGRAPHICS SYMPOSIUM ON POINT-BASED GRAPHICS, THE EUROGRAPHICS ASSOCIATION, 29 July 2006 (2006-07-29), pages 27 - 35, XP055271059, Retrieved from the Internet <URL:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.230.953&rep=rep1&type=pdf> [retrieved on 20160509], DOI: 10.2312/SPBG/SPBG06/027-035 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114741918A (zh) * 2022-02-24 2022-07-12 西北大学 一种面向遗址劣化有限元分析的并行网格剖分算法
CN114741918B (zh) * 2022-02-24 2024-02-23 西北大学 一种面向遗址劣化有限元分析的并行网格剖分方法
CN114758337A (zh) * 2022-06-16 2022-07-15 山东海量信息技术研究院 一种语义实例重建方法、装置、设备及介质
CN114758337B (zh) * 2022-06-16 2022-10-28 山东海量信息技术研究院 一种语义实例重建方法、装置、设备及介质
CN116680778A (zh) * 2023-04-27 2023-09-01 清华大学 建筑结构布置生成方法、系统及生成模型的构建方法
CN116680778B (zh) * 2023-04-27 2024-03-12 清华大学 建筑结构布置生成方法、系统及生成模型的构建方法

Also Published As

Publication number Publication date
JP2022545141A (ja) 2022-10-26
CN111882666A (zh) 2020-11-03
CN111882666B (zh) 2022-06-21
JP7395617B2 (ja) 2023-12-11
KR20220028010A (ko) 2022-03-08
TW202205215A (zh) 2022-02-01

Similar Documents

Publication Publication Date Title
WO2022016867A1 (zh) 三维网格模型的重建方法及其装置、设备、存储介质
WO2020119527A1 (zh) 人体动作识别方法、装置、终端设备及存储介质
CN111784821B (zh) 三维模型生成方法、装置、计算机设备及存储介质
Salzmann et al. Linear local models for monocular reconstruction of deformable surfaces
CN108122277B (zh) 一种建模方法及装置
CN110021069A (zh) 一种基于网格形变的三维模型重建方法
CN109242961A (zh) 一种脸部建模方法、装置、电子设备和计算机可读介质
US11481973B2 (en) Method, device, and storage medium for segmenting three-dimensional object
CN112215050A (zh) 非线性3dmm人脸重建和姿态归一化方法、装置、介质及设备
WO2021253788A1 (zh) 一种人体三维模型构建方法及装置
WO2022001222A1 (zh) 三维模型生成方法、神经网络生成方法及装置
WO2022193335A1 (zh) 点云数据处理方法、装置、计算机设备和存储介质
CN110020600A (zh) 生成用于训练人脸对齐模型的数据集的方法
CN112328715A (zh) 视觉定位方法及相关模型的训练方法及相关装置、设备
CN116958378A (zh) 面部贴图重建方法、装置、计算机可读介质及电子设备
CN111932438A (zh) 图像风格迁移方法、设备及存储装置
CN113298931B (zh) 一种物体模型的重建方法、装置、终端设备和存储介质
Rios et al. Scalability of learning tasks on 3D CAE models using point cloud autoencoders
WO2022213623A1 (zh) 图像生成、三维人脸模型生成的方法、装置、电子设备及存储介质
CN114742956A (zh) 模型处理方法、装置、设备及计算机可读存储介质
CN114677350A (zh) 连接点提取方法、装置、计算机设备及存储介质
CN116824092A (zh) 三维模型生成方法、装置、计算机设备和存储介质
US10861174B2 (en) Selective 3D registration
WO2022236802A1 (zh) 一种物体模型的重建方法、装置、终端设备和存储介质
CN113658320A (zh) 一种三维重建方法、人脸三维重建方法及其相关装置

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2021568963

Country of ref document: JP

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 20227002922

Country of ref document: KR

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21845729

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21845729

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 21845729

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 31.07.2023)