CN111462301A - Method for constructing generation model for multi-view plant three-dimensional reconstruction - Google Patents

Abstract

The invention discloses a method for constructing a generative model for multi-view three-dimensional reconstruction of plants. The method includes: constructing a plant growth model by using a Bayesian probability framework, and constructing a plant skeleton by assigning a prior probability with parameters to the branching growth model Probabilistic representation of growth; by specifying the initial growth factor and bifurcation algebra limit on the root node, the plant growth model generates an instantiated skeleton through random sampling; to obtain the image set of the same plant, the plant growth model uses the vector field method to analyze the image set. A 2D skeleton is calculated for each single image of the 2D skeleton, and the morphological primitives of the image set are extracted by clustering analysis of the 2D skeletons. The model of the morphological primitives is used to fit the skeletons in the training set, and the Gauss-Newton gradient descent method is used to obtain Optimal solutions for plant growth model parameters. This method can reflect the Bayesian probability model of plant growth posture characteristics, which can be used as a general framework for plant representation and can be used for multi-view plant reconstruction.

Description

Construction method of generative model for multi-view 3D reconstruction of plants

technical field

The invention relates to the technical field of three-dimensional reconstruction, in particular to a method for constructing a generative model for multi-view three-dimensional reconstruction of plants.

Background technique

Plants are common elements in urban and natural landscape environments, and plant models are widely used in agriculture, biology, architecture, games, and film industries. However, the creation of plant models is still tedious and expensive work. Existing plant generation systems require fine-tuning of parameters or manual modeling to obtain the desired tree shape, which is inconvenient to obtain a large number of different tree instances. Existing plant reconstruction techniques either use laser-scanned point clouds to reconstruct tree skeletons, or use multiple views to restore a specific tree shape, which cannot learn tree growth patterns and generate new and different tree instances.

SUMMARY OF THE INVENTION

The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

Therefore, an object of the present invention is to propose a method for constructing a generative model for multi-view three-dimensional reconstruction of plants, which can reflect a Bayesian probability model of plant growth posture characteristics, which can be used as a general framework for plant representation and can For multi-view plant reconstruction.

In order to achieve the above object, an embodiment of the present invention provides a method for constructing a generative model for multi-view three-dimensional reconstruction of plants, including:

A plant growth model is constructed by using a Bayesian probability framework, and a probability representation of plant skeleton growth is constructed by assigning a prior probability with parameters to the branching growth model;

The plant growth model generates an instantiated skeleton by random sampling by specifying an initial growth factor and a bifurcation algebra limit on the root node;

Obtain an image set of the same plant, the plant growth model uses the vector field method to calculate a 2D skeleton for each single image in the image set, and extracts the morphological primitives of the image set through cluster analysis of the 2D skeleton , using the model of the morphological primitive to fit the skeleton in the training set, and using the Gauss-Newton gradient descent method to obtain the optimal solution of the parameters of the plant growth model.

The method for constructing a generative model for multi-view plant three-dimensional reconstruction according to the embodiment of the present invention constructs a probability representation of plant skeleton growth by assigning a prior probability with parameters to the model through a parametric fractal model of plant growth. The construction of growth model can reconstruct the morphology of plants at different ages, which constitute the morphological space of the model. Given a single-view or multi-view image of the same plant, the deep learning method can be used to extract the branch probability image of the plant. Using this image, the parameters of the generative model can be optimized, and the posterior of the occluded part of the plant can be inferred and completed. probability. Training the model with pictures of the same plant will specialize the model to that plant, sampling the model will be able to instantiate different random shapes of the plant, interpolating in the shape space will achieve smoothing of the different shapes transition. The Bayesian probability model that can reflect the characteristics of plant growth posture can be used as a general framework for plant representation and can be used for multi-view plant reconstruction.

In addition, the method for constructing a generative model for multi-view plant three-dimensional reconstruction according to the above-mentioned embodiments of the present invention may also have the following additional technical features:

Further, in an embodiment of the present invention, in the plant growth model, each bifurcation of the plant skeleton defines the parent-child relationship between the branches, and each bifurcation can define the number of bifurcations, the bifurcation shape, The parameterized local probability of the growth factor. The growth factor is the growth information transmitted from this bifurcation to the next bifurcation, which will affect the conditional probability of the subsequent bifurcation. The parameters of the local probability of the plant will determine the morphological characteristics of the plant described by the model. .

Further, in an embodiment of the present invention, when the input is a multi-view image, a single plant in the image is segmented, each image is converted into a branch probability image by using the Pix2Pix network, and the branch probability is back projected to the 3D image. In voxel coordinates, a 3D probabilistic branch structure is formed, multiple probabilistic candidate structures are generated using greedy search, and Metropolis Hasting sampling is used for each candidate structure to optimize the optimal candidate to form a discrete approximation of the posterior distribution, and the branch skeleton is reconstructed through sampling. .

Further, in an embodiment of the present invention, by subdividing and clustering the input image, the texture clusters in the image are extracted, and the texture clusters are used to create leaves in the back-projection area of the leaves of the input image. Covered area of the source image, additional leaves will be composited to ensure an evenly distributed leaf density.

Further, in one embodiment of the present invention, by modifying and specifying local parameters of a certain branch in the plant growth model, the subtree involved in the branch will be re-optimized.

Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a flowchart of a method for constructing a generative model for multi-view three-dimensional reconstruction of plants according to an embodiment of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.

The following describes a method for constructing a generative model for multi-view 3D reconstruction of plants according to an embodiment of the present invention with reference to the accompanying drawings.

FIG. 1 is a flowchart of a method for constructing a generative model for multi-view three-dimensional reconstruction of plants according to an embodiment of the present invention.

As shown in Figure 1, the method for constructing a generative model for multi-view three-dimensional reconstruction of plants includes the following steps:

In step S1, a Bayesian probability framework is used to construct a plant growth model, and a prior probability with parameters is assigned to the branched growth model to construct a probability representation of plant skeleton growth.

Specifically, the growth model of plants is synthetic and causal, each instance goes through a bottom-up generation process, and there is a causal connection between the latter process and the former process, and this connection can be described by a Bayesian probability framework. Specifically, each fork of the plant skeleton defines the parent-child relationship between the branches. For each fork, the number of forks, the shape of the fork, and the parameterized local probability of the growth factor can be defined. Forks transmit growth information to the next fork, which will affect the conditional probability of subsequent forks. The parameters of the plant local probability will determine the morphological characteristics of the plant described by the model.

Step S2, by specifying the initial growth factor and the bifurcation algebra limit on the root node, the plant growth model generates an instantiated skeleton through random sampling.

Specifically, by specifying an initial growth factor and a bifurcation algebra limit on the root node, the model can generate an instantiated skeleton through random sampling, which is described by the data structure of the structure tree, which allows fine-tuning and fine-tuning of the structure in subsequent operations. Changes in topology. All structure trees constitute the morphological space of the model, and structure matching can be performed between different structure trees to realize continuous change of structure.

For multi-view image input, first use the Pix2Pix network to convert each image into a branch probability image, and then back-project the branch probabilities into 3D voxel coordinates to form a 3D probabilistic branch structure, and use greedy search to generate multiple probability candidate structures, as The most promising candidates are optimized using Metropolis Hasting sampling for each candidate structure, resulting in a discrete approximation of the posterior distribution, and finally the branch skeleton reconstruction is achieved through sampling. This method has no requirement on the number of input images.

For plant leaves, which are not accurately modeled due to their large number of leaves and their severe mutual occlusion, instead, by subdividing and clustering the image, the texture clusters of the given image can be extracted, using these Textures create leaves in the back-projected areas of the leaves of the input image. For areas not covered by the source image, additional leaves are composited to ensure an evenly distributed leaf density.

In step S3, an image set of the same plant is obtained, the plant growth model uses the vector field method to calculate a 2D skeleton for each single image in the image set, and through clustering analysis of the 2D skeleton, the morphological primitives of the image set are extracted, and the morphological elements are used. The model of the primitive is fitted to the skeleton in the training set, and the Gauss-Newton gradient descent method is used to obtain the optimal solution of the parameters of the plant growth model.

Specifically, the advantage of using a generative model is that its learning process will not be limited by the size of the dataset and can be specialized to varying degrees. Training for the same plant can define basic morphological "primitives" for the growth model, which describe the specific bifurcated growth patterns of that plant. Given a set of images of the same plant (not necessarily of the same plant), a vector field method can be used to build a 2D skeleton for each image, which provides a posterior for the 3D bifurcation of the plant. The cluster analysis can extract the morphological primitives provided by the dataset. By fitting the parent-child relationship and the parameters of primitive instantiation with the skeleton in the training set, the Gauss-Newton gradient descent method is used to obtain the optimal solution of the model parameters.

In conclusion, the plant growth model of the embodiment of the present invention can be used for multi-image three-dimensional reconstruction of a specific plant and multi-instance generation of a certain plant.

First, a Bayesian model of plant growth is established, and the multi-image 3D reconstruction of plants and the multi-instance generation of a certain plant are generated. Analyzing the plant morphology into a time series probability model with a causal structure, each new fork of the plant will be affected by the existing fork morphology. The effects of the natural environment on plants will be modeled as global parameters. In particular, when a model is applied to a particular plant, a set of morphological primitives is defined for the model, which describes the bifurcated growth pattern specific to that plant and serves to mix different probabilistic models.

To apply the above model, first train the model on a set of images of the same plant. For each training image (if the training image has branches and leaves, the branches are extracted), the model will calculate a 2D vector field for this single image, and then use the recursive skeletonization algorithm to successively calculate the 2D projection of the plant. The constraints of first-order, second-order, and higher-order branches, 2D projections give the posterior information on the 3D branches of plants. In order to prevent the angular preference in the training of the model, a projection angle is randomly sampled from 0 to 180° for training. Through cluster analysis of these 2D skeletons, the morphological primitives provided by the dataset can be extracted. The model using these morphological primitives is then fitted to the skeleton in the training set, and the optimal solution of the model parameters is obtained using the Gauss-Newton gradient descent method.

To achieve single-view/multi-view plant reconstruction using the trained specialized model, the Pix2Pix network needs to be trained first. In order for the Pix2Pix network to generate probabilistic images, the network itself preferably has a certain uncertainty, so an additional dropout layer is added to the network to output probabilistic results. Using Speedtree's plant library as the training set, 36 viewpoints (with varying lighting conditions) were rendered for each model, and the images were flipped for training.

For multi-view image input, firstly, a single plant in the image needs to be segmented, and then each image is converted into a branch probability image using the Pix2Pix network, and the branch probability is back projected into 3D voxel coordinates to form a 3D probability branch structure. Greedy search generates multiple probabilistic candidate structures, optimizes the most promising candidate using Metropolis Hasting sampling for each candidate structure, thereby forming a discrete approximation of the posterior distribution, and finally achieves branch skeleton reconstruction through sampling. It should be pointed out that even if there is only one input image, the algorithm can still run, but the reconstruction is more arbitrary. In particular, the generative model is able to self-sample a plant instance even without any input.

For the reconstructed branch skeleton, the user can access the modification and specify the local parameters of a branch on it, and the subtree involved in this branch will be re-optimized for calculation.

For plant leaves, which are not accurately modeled due to their large number of leaves and their severe mutual occlusion, instead, by subdividing and clustering the image, the texture clusters of the given image can be extracted, using these Textures create leaves in the back-projected areas of the leaves of the input image. For areas not covered by the source image, additional leaves are composited to ensure an evenly distributed leaf density.

According to the method for constructing a generative model for multi-view 3D reconstruction of plants proposed in the embodiment of the present invention, a probability representation of plant skeleton growth is constructed by assigning a prior probability with parameters to the model through a parametric fractal model of plant growth. The construction of growth models can reconstruct the morphology of plants at different ages, which constitute the morphological space of the model. Given a single-view or multi-view image of the same plant, the deep learning method can be used to extract the branch probability image of the plant. Using this image, the parameters of the generative model can be optimized, and the posterior of the occluded part of the plant can be inferred and completed. probability. Training the model with pictures of the same plant will specialize the model to that plant, sampling the model will be able to instantiate different random shapes of the plant, interpolating in the shape space will achieve smoothing of the different shapes transition. The Bayesian probability model that can reflect the characteristics of plant growth posture can be used as a general framework for plant representation and can be used for multi-view plant reconstruction.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

Although the embodiments of the present invention have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present invention. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (5)

1. a construction method for the generative model of multi-view plant three-dimensional reconstruction, is characterized in that, comprises the following steps: A plant growth model is constructed by using a Bayesian probability framework, and a probability representation of plant skeleton growth is constructed by assigning a prior probability with parameters to the branching growth model; The plant growth model generates an instantiated skeleton by random sampling by specifying an initial growth factor and a bifurcation algebra limit on the root node; Obtain an image set of the same plant, the plant growth model uses the vector field method to calculate a 2D skeleton for each single image in the image set, and extracts the morphological primitives of the image set through cluster analysis of the 2D skeleton , using the model of the morphological primitive to fit the skeleton in the training set, and using the Gauss-Newton gradient descent method to obtain the optimal solution of the parameters of the plant growth model. 2. The method for constructing a generative model for multi-view three-dimensional reconstruction of plants according to claim 1, wherein, in the plant growth model, each fork of the plant skeleton defines a parent-child relationship between branches and trunks , each bifurcation can define the parameterized local probability of bifurcation number, bifurcation shape, and growth factor. The growth factor is the growth information transmitted from this bifurcation to the next bifurcation, which will affect the conditional probability of subsequent bifurcations, The parameters of the plant local probability will determine the morphological characteristics of the plant described by the model. 3. the construction method of the generative model that is used for multi-view plant three-dimensional reconstruction according to claim 1, is characterized in that, when inputting is multi-view image, segment out the single plant in the image, utilize Pix2Pix network to convert each The image is converted into a branch probability image, and then the branch probability is back projected into the 3D voxel coordinates to form a 3D probability branch structure. A greedy search is used to generate multiple probability candidate structures, and Metropolis Hasting sampling is used for each candidate structure to optimize the optimal candidate. A discrete approximation of the posterior distribution is formed, and the reconstruction of the branch skeleton is achieved by sampling. 4. The method for constructing a generative model for multi-view plant three-dimensional reconstruction according to claim 1, characterized in that, by subdividing and clustering the input image, extracting texture clusters in the image, using the texture The cluster creates leaves in the back-projected area of the leaves in the input image. For areas not covered by the source image, additional leaves are composited to ensure an evenly distributed leaf density. 5. The method for constructing a generative model for multi-view three-dimensional reconstruction of plants according to claim 1, wherein, by accessing, modifying and specifying the local parameters of a certain branch in the plant growth model, the sub-parameters involved in the branch The tree will be re-optimized.

Images