CN111462301A - Method for constructing generation model for multi-view plant three-dimensional reconstruction - Google Patents

Method for constructing generation model for multi-view plant three-dimensional reconstruction Download PDF

Info

Publication number
CN111462301A
CN111462301A CN202010147283.8A CN202010147283A CN111462301A CN 111462301 A CN111462301 A CN 111462301A CN 202010147283 A CN202010147283 A CN 202010147283A CN 111462301 A CN111462301 A CN 111462301A
Authority
CN
China
Prior art keywords
plant
model
probability
image
growth
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010147283.8A
Other languages
Chinese (zh)
Other versions
CN111462301B (en
Inventor
刘烨斌
钟源
王松涛
戴琼海
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Priority to CN202010147283.8A priority Critical patent/CN111462301B/en
Publication of CN111462301A publication Critical patent/CN111462301A/en
Application granted granted Critical
Publication of CN111462301B publication Critical patent/CN111462301B/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00Three dimensional [3D] modelling, e.g. data description of 3D objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/241Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G06F18/24155Bayesian classification
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/08Projecting images onto non-planar surfaces, e.g. geodetic screens

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

The invention discloses a method for constructing a generation model for multi-view plant three-dimensional reconstruction, which comprises the following steps: constructing a plant growth model by using a Bayesian probability framework, and constructing probability representation of plant skeleton growth by giving prior probability with parameters to a branch growth model; by specifying an initial growth factor and a bifurcation algebra limit on a root node, a plant growth model generates an instantiated skeleton through random sampling; the method comprises the steps of obtaining an image set of the same plant, calculating a 2D framework for each single image in the image set by a plant growth model through a vector field method, extracting morphological elements of the image set through clustering analysis of the 2D frameworks, fitting the model of the morphological elements with the frameworks in a training set, and solving the optimal solution of plant growth model parameters through a Gaussian-Newton gradient descent method. The method can reflect the Bayes probability model of the plant growth posture characteristic, can be used as a general frame of plant representation, and can be used for multi-view plant reconstruction.

Description

Method for constructing generation model for multi-view plant three-dimensional reconstruction
Technical Field
The invention relates to the technical field of three-dimensional reconstruction, in particular to a method for constructing a generation model for multi-view plant three-dimensional reconstruction.
Background
Plants are common elements of urban and natural landscape environments, and plant models are widely applied in the agricultural, biological, building, game and film industries. However, the creation of plant models is still a cumbersome and expensive task. Existing plant generation systems require fine-tuning of parameters or manual modeling to obtain the desired tree shape, and are not convenient for obtaining a large number of different tree instances. Existing plant reconstruction techniques either use laser-scanned point clouds to reconstruct the skeleton of the tree or use multiple views to restore the morphology of a particular tree, which cannot learn the growth pattern of the tree and generate new, different tree instances.
Disclosure of Invention
The present invention is directed to solving, at least to some extent, one of the technical problems in the related art.
Therefore, the invention aims to provide a method for constructing a generative model for the three-dimensional reconstruction of a multi-view plant, which can reflect the Bayesian probability model of the plant growth posture characteristic, can be used as a general framework of plant representation, and can be used for the multi-view plant reconstruction.
In order to achieve the above object, an embodiment of an aspect of the present invention provides a method for constructing a generative model for three-dimensional reconstruction of a multi-view plant, including:
constructing a plant growth model by using a Bayesian probability framework, and constructing probability representation of plant skeleton growth by giving prior probability with parameters to the branch growth model;
by specifying initial growth factors and bifurcation algebra limits on root nodes, the plant growth model generates an instantiated skeleton through random sampling;
the method comprises the steps of obtaining an image set of the same kind of plants, calculating a 2D framework for each single image in the image set by the plant growth model through a vector field method, extracting morphological elements of the image set through 2D framework clustering analysis, fitting the model of the morphological elements with the framework in a training set, and solving the optimal solution of parameters of the plant growth model through a Gaussian-Newton gradient descent method.
According to the method for constructing the generation model for the multi-view plant three-dimensional reconstruction, the probability representation of the growth of the plant skeleton is constructed through the parameterized fractal model of the plant growth and the prior probability with parameters given to the model. The construction of the growth model allows to reconstruct the morphology of the plant at different age stages, which constitute the morphological space of the model. The method comprises the steps of giving a single-view image or a multi-view image of the same plant, extracting a branch probability image of the plant by using a deep learning method, optimizing parameters of a generated model by using the image, and reasoning and complementing the posterior probability of the shielded part of the plant. Training the model using pictures of the same plant will specialize the model onto the plant, sampling the model will enable instantiation of different random aspects of the plant, interpolation in aspect space will enable smooth transitions of the different aspects. The Bayes probability model capable of reflecting the plant growth posture characteristics can be used as a general framework of plant representation and can be used for multi-view plant reconstruction.
In addition, the method for constructing the generative model for the three-dimensional reconstruction of the multi-view plant according to the above embodiment of the present invention may further have the following additional technical features:
further, in one embodiment of the present invention, in the plant growth model, each branch of the plant skeleton defines a parent-child relationship between branches, each branch may define a branch number, a branch shape, and a parameterized local probability of a growth factor, the growth factor is growth information transmitted from the branch to the next branch, and affects the conditional probability of the subsequent branch, and the parameter of the plant local probability determines the shape characteristics of the plant described by the model.
Further, in an embodiment of the present invention, when a multi-view image is input, a single plant in the image is segmented, each image is converted into a branch probability image by using a Pix2Pix network, the branch probabilities are back-projected into 3D voxel coordinates to form a 3D probability branch structure, a plurality of probability candidate structures are generated by using greedy search, Metropolis Hasting sampling is used for each candidate structure to optimize an optimal candidate, discrete approximation of posterior distribution is formed, and reconstruction of a branch skeleton is achieved by sampling.
Further, in one embodiment of the present invention, the input image is subdivided and clustered, texture clusters in the image are extracted, leaves are created in back projection areas of the leaves of the input image by using the texture clusters, and the leaves are additionally synthesized for areas not covered by the source image to ensure uniformly distributed leaf density.
Further, in an embodiment of the present invention, by accessing local parameters that modify and specify a branch in the plant growth model, the subtree to which the branch relates is re-optimized.
Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a flowchart of a method for constructing a generative model for three-dimensional reconstruction of a multi-view plant according to an embodiment of the invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative and intended to be illustrative of the invention and are not to be construed as limiting the invention.
The following describes a method for constructing a generative model for three-dimensional reconstruction of a multi-view plant according to an embodiment of the present invention with reference to the accompanying drawings.
Fig. 1 is a flowchart of a method for constructing a generative model for three-dimensional reconstruction of a multi-view plant according to an embodiment of the invention.
As shown in fig. 1, the method for constructing a generative model for the three-dimensional reconstruction of a multi-view plant includes the following steps:
and step S1, constructing a plant growth model by using a Bayesian probability framework, and constructing probability representation of plant skeleton growth by giving prior probability with parameters to the branch growth model.
Specifically, the growth model of the plant is synthetic and causal, each instance is via a bottom-up generation process, with causal links between post and pre processes, which can be characterized by a bayesian probabilistic framework. Specifically, each bifurcation of the plant skeleton defines the parent-child relationship between branches, the number of bifurcations, the form of the bifurcation, and the parameterized local probability of the growth factor can be defined for each bifurcation, and the growth factor can be regarded as growth information transmitted from the bifurcation to the next bifurcation, which affects the conditional probability of the subsequent bifurcation. The parameters of the plant local probability will determine the morphological characteristics of the plant described by the model.
Step S2, the plant growth model generates an instantiated skeleton by random sampling by specifying an initial growth factor and a bifurcation algebra limit on the root node.
In particular, by specifying initial growth factors and bifurcation algebra constraints on the root nodes, the model can generate instantiated skeletons through random sampling, which are described by the data structures of the structure tree, which allows for fine-tuning of the structure and topological changes in subsequent operations. All the structure trees form the form space of the model, and different structure trees can be matched with each other to realize the continuous change of the structure.
For multi-view image input, firstly, a Pix2Pix network is used for converting each image into a branch probability image, then, the branch probability is back projected into a 3D voxel coordinate to form a 3D probability branch structure, a plurality of probability candidate structures are generated by greedy search, Metropolis Hasting sampling is used for optimizing the most promising candidate for each candidate structure, so that discrete approximation of posterior distribution is formed, and finally, reconstruction of a branch framework is achieved through sampling. This method has no requirement on the number of input images.
For plant leaves, because plant leaves are numerous and have severe mutual occlusion, the leaves are not modeled accurately, instead, by subdividing and clustering the images, clusters of textures of a given image can be extracted, and the leaves can be created in the back projection area of the leaves of the input image using these textures. For areas not covered by the source image, leaves will be additionally synthesized to ensure an evenly distributed leaf density.
And step S3, acquiring an image set of the same plant, calculating a 2D skeleton of each single image in the image set by the plant growth model by using a vector field method, extracting morphological elements of the image set by clustering analysis of the 2D skeleton, fitting the model of the morphological elements with the skeleton in a training set, and solving the optimal solution of plant growth model parameters by using a Gaussian-Newton gradient descent method.
In particular, the advantage of using generative models is that their learning process will not be limited by the size of the data set and may be specialized to varying degrees. Training on the same plant, a basic morphological "primitive" may be defined for the growth model, which describes a divergent growth pattern specific to that plant. Given a set of images of a plant of the same kind (not necessarily for the same plant), a 2D skeleton can be built for each image using a vector field method, the 2D skeleton provides a posteriori for 3D bifurcation of the plant, and morphological elements provided by the data set can be extracted by cluster analysis of these 2D skeletons. And fitting the parent-child relationship, the parameter instantiated by the primitive and the skeleton in the training set, and solving the optimal solution of the model parameter by using a Gaussian-Newton gradient descent method.
In summary, the plant growth model of the embodiment of the invention can be used for multi-image three-dimensional reconstruction of a specific plant and multi-instance generation of a plant.
Firstly, establishing a Bayesian model of plant growth, and carrying out three-dimensional reconstruction on multiple images of a plant and generating multiple instances of a certain plant. The plant morphology is analyzed into a time sequence probability model with a causal structure, and the generation of each new bifurcation of the plant is influenced by the existing bifurcation morphology. The effect of the natural environment on the plant will be modeled as a global parameter. In particular, when a model is applied to a particular plant, a set of morphological primitives is defined for the model that describe the specific divergent growth pattern of the plant and that act to mix the different probabilistic models.
To apply the above model, the model is first trained using a set of images of the same plant. For each training picture (if the training picture has the branches and leaves, the branch parts of the training picture are extracted), the model calculates a 2D vector field for the single image, then the first-order, second-order and high-order branches of the 2D projection of the plant are calculated successively by using a skeleton algorithm recursively, and the 3D bifurcation posterior information of the plant is given by the constraint of the 2D projection. In order to prevent the training of the model from generating angular preference, a projection angle is randomly sampled from 0-180 degrees for training. Through cluster analysis of these 2D skeletons, the morphological primitives provided by the dataset can be extracted. And fitting the model using the morphological primitives with a skeleton in a training set, and solving the optimal solution of the model parameters by using a Gaussian-Newton gradient descent method.
To achieve the reconstruction of single-view/multi-view plants using trained specialized models, a Pix2Pix network needs to be trained first. For a Pix2Pix network to generate a probabilistic image, the network itself is preferably provided with some uncertainty, thus adding an extra dropout layer to the network to output the result in the probabilistic sense. Using a plant library of Speedtree as a training set, 36 viewpoints (with different lighting conditions) are rendered for each model, and images are flipped for training.
For multi-view image input, firstly, a single plant in an image needs to be segmented, then, a Pix2Pix network is used for converting each image into a branch probability image, the branch probability is back projected into a 3D voxel coordinate to form a 3D probability branch structure, a plurality of probability candidate structures are generated by greedy search, Metropolis Hasting sampling is used for optimizing the most promising candidate for each candidate structure, and therefore discrete approximation of posterior distribution is formed, and finally reconstruction of a branch framework is achieved through sampling. It should be noted that the algorithm can be run even if there is only one input image, but the reconstruction is of great discretion. In particular, the generative model is able to derive a plant instance from the sample, even without any input.
For a branch skeleton that has been reconstructed, the user has access to the local parameters that modify and specify the branch on which the subtree involved is to be re-optimized.
For plant leaves, because plant leaves are numerous and have severe mutual occlusion, the leaves are not modeled accurately, instead, by subdividing and clustering the images, clusters of textures of a given image can be extracted, and the leaves can be created in the back projection area of the leaves of the input image using these textures. For areas not covered by the source image, leaves will be additionally synthesized to ensure an evenly distributed leaf density.
According to the method for constructing the generation model for the multi-view plant three-dimensional reconstruction, which is provided by the embodiment of the invention, the probability representation of the growth of the plant skeleton is constructed through the parameterized fractal model of the plant growth and the prior probability with parameters given to the model. The construction of the growth model allows to reconstruct the morphology of the plant at different age stages, which constitute the morphological space of the model. The method comprises the steps of giving a single-view image or a multi-view image of the same plant, extracting a branch probability image of the plant by using a deep learning method, optimizing parameters of a generated model by using the image, and reasoning and complementing the posterior probability of the shielded part of the plant. Training the model using pictures of the same plant will specialize the model onto the plant, sampling the model will enable instantiation of different random aspects of the plant, interpolation in aspect space will enable smooth transitions of the different aspects. The Bayes probability model capable of reflecting the plant growth posture characteristics can be used as a general framework of plant representation and can be used for multi-view plant reconstruction.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention, and that variations, modifications, substitutions and alterations can be made to the above embodiments by those of ordinary skill in the art within the scope of the present invention.

Claims (5)

1. A method for constructing a generative model for the three-dimensional reconstruction of a multi-view plant, comprising the steps of:
constructing a plant growth model by using a Bayesian probability framework, and constructing probability representation of plant skeleton growth by giving prior probability with parameters to the branch growth model;
by specifying initial growth factors and bifurcation algebra limits on root nodes, the plant growth model generates an instantiated skeleton through random sampling;
the method comprises the steps of obtaining an image set of the same kind of plants, calculating a 2D framework for each single image in the image set by the plant growth model through a vector field method, extracting morphological elements of the image set through 2D framework clustering analysis, fitting the model of the morphological elements with the framework in a training set, and solving the optimal solution of parameters of the plant growth model through a Gaussian-Newton gradient descent method.
2. The method of claim 1, wherein each bifurcation of the plant skeleton defines a parent-child relationship between branches, each bifurcation can define a bifurcation number, a bifurcation morphology, and a parameterized local probability of a growth factor, wherein the growth factor is growth information transmitted by the bifurcation to the next bifurcation and influences the conditional probability of the subsequent bifurcation, and the parameters of the plant local probability determine the morphological characteristics of the plant described by the model.
3. The method for constructing the generative model for the three-dimensional reconstruction of the multi-view plant as claimed in claim 1, wherein when the multi-view image is inputted, a single plant in the image is segmented, each image is converted into a branch probability image by using a Pix2Pix network, then the branch probability is back-projected into 3D voxel coordinates to form a 3D probability branch structure, a plurality of probability candidate structures are generated by greedy search, Metropolis Hasting sampling is used for each candidate structure to optimize an optimal candidate, discrete approximation of posterior distribution is formed, and reconstruction of a branch skeleton is realized by sampling.
4. The method for constructing the generative model for the three-dimensional reconstruction of multiview plants as claimed in claim 1, wherein the texture clusters in the image are extracted by subdividing and clustering the input image, the leaf is created in the back projection area of the leaf of the input image by using the texture clusters, and the leaf is additionally synthesized for the area not covered by the source image to ensure the uniformly distributed leaf density.
5. Method for constructing a generative model for the three-dimensional reconstruction of multiview plants according to claim 1, characterized in that the subtree involved by a branch in the plant growth model is re-optimized by accessing local parameters modifying and specifying the branch.
CN202010147283.8A 2020-03-05 2020-03-05 Method for constructing generation model for multi-view plant three-dimensional reconstruction Expired - Fee Related CN111462301B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010147283.8A CN111462301B (en) 2020-03-05 2020-03-05 Method for constructing generation model for multi-view plant three-dimensional reconstruction

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010147283.8A CN111462301B (en) 2020-03-05 2020-03-05 Method for constructing generation model for multi-view plant three-dimensional reconstruction

Publications (2)

Publication Number Publication Date
CN111462301A true CN111462301A (en) 2020-07-28
CN111462301B CN111462301B (en) 2022-10-28

Family

ID=71685564

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010147283.8A Expired - Fee Related CN111462301B (en) 2020-03-05 2020-03-05 Method for constructing generation model for multi-view plant three-dimensional reconstruction

Country Status (1)

Country Link
CN (1) CN111462301B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108022278A (en) * 2017-12-29 2018-05-11 清华大学 Character animation method for drafting and system based on motion tracking in video
CN108346142A (en) * 2018-01-16 2018-07-31 中国农业大学 A kind of plant growth state recognition methods based on plant illumination image
CN109285217A (en) * 2018-09-10 2019-01-29 中国科学院自动化研究所 Process type plant model method for reconstructing based on multi-view image
US20190065890A1 (en) * 2017-08-22 2019-02-28 Korea Institute Of Science And Technology System and method for plant leaf identification

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190065890A1 (en) * 2017-08-22 2019-02-28 Korea Institute Of Science And Technology System and method for plant leaf identification
CN108022278A (en) * 2017-12-29 2018-05-11 清华大学 Character animation method for drafting and system based on motion tracking in video
CN108346142A (en) * 2018-01-16 2018-07-31 中国农业大学 A kind of plant growth state recognition methods based on plant illumination image
CN109285217A (en) * 2018-09-10 2019-01-29 中国科学院自动化研究所 Process type plant model method for reconstructing based on multi-view image

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
曾兰玲等: "交互式植物建模系统", 《计算机辅助设计与图形学学报》 *

Also Published As

Publication number Publication date
CN111462301B (en) 2022-10-28

Similar Documents

Publication Publication Date Title
CN109147048B (en) Three-dimensional mesh reconstruction method by utilizing single-sheet colorful image
Fahim et al. Single-View 3D reconstruction: A Survey of deep learning methods
US8698809B2 (en) Creation and rendering of hierarchical digital multimedia data
Li et al. Modeling and generating moving trees from video
Wang et al. Mvpnet: Multi-view point regression networks for 3d object reconstruction from a single image
CN101512633B (en) A hair motion compositor system and optimization techniques for use in a hair/fur pipeline
Gregory et al. Feature-based surface decomposition for correspondence and morphing between polyhedra
Tobor et al. Multi-scale reconstruction of implicit surfaces with attributes from large unorganized point sets
CN111028335B (en) Point cloud data block surface patch reconstruction method based on deep learning
US12118669B2 (en) Subdividing a three-dimensional mesh utilizing a neural network
CN113781667A (en) Three-dimensional structure simplified reconstruction method and device, computer equipment and storage medium
CN110889893B (en) Three-dimensional model representation method and system for expressing geometric details and complex topology
Shabat et al. Design of porous micro-structures using curvature analysis for additive-manufacturing
Rossignac et al. Solid modeling
Ribelles et al. Multiresolution modeling of arbitrary polygonal surfaces: a characterization
Zhang et al. [Retracted] Virtual Reality Design and Realization of Interactive Garden Landscape
Kamra et al. Lightweight reconstruction of urban buildings: Data structures, algorithms, and future directions
Zhou et al. Deeptree: Modeling trees with situated latents
Fischer Multi-level models for reverse engineering and rapid prototyping in remote CAD systems
CN110660132A (en) Three-dimensional model construction method and device
CN108109205B (en) Incomplete model retrieval and reconstruction method based on hole filling
CN111462301B (en) Method for constructing generation model for multi-view plant three-dimensional reconstruction
Boender et al. Finite-element mesh generation from constructive-solid-geometry models
Mirra et al. Exploring a design space of shell and tensile structures generated by AI from historical precedents
Galbraith et al. Implicit visualization and inverse modeling of growing trees

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20221028

CF01 Termination of patent right due to non-payment of annual fee