CN101127075A - Multi-view angle three-dimensional human face scanning data automatic registration method - Google Patents
Multi-view angle three-dimensional human face scanning data automatic registration method Download PDFInfo
- Publication number
- CN101127075A CN101127075A CNA2007100187821A CN200710018782A CN101127075A CN 101127075 A CN101127075 A CN 101127075A CN A2007100187821 A CNA2007100187821 A CN A2007100187821A CN 200710018782 A CN200710018782 A CN 200710018782A CN 101127075 A CN101127075 A CN 101127075A
- Authority
- CN
- China
- Prior art keywords
- registration
- model
- scanning data
- mark region
- eye
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The utility model discloses an automatic registration method for multi-view three-dimensional face scanning data, which is characterized in comprising the following steps: coordinate axis transformation is made to the multi-view three-dimensional face scanning data through the PCA method; the shape index value of a transformed model is calculated, and a plurality of feature regions are screened using the method of threshold segmentation; region screening is restricted by region relative distribution features so as to finally position the symbolic regions such as eyes, outer canthi and nasal tip points; the symbolic regions are matched by the ICP method; and the coordinate translation and rotation transformation are made to the transformation parameters which are obtained from region registration of the whole scanning data so as to complete registration. The coarse registration of the utility model adopts the method of coordinate axis transformation, and the global features are instead of the individual point features, so error caused by symbolic point detection is reduced. The precise registration only makes registration iterative computation to the extracted symbolic regions, so computation complexity is reduced, and the automatic registration of multi-pose three-dimensional scanning data is enabled.
Description
Technical field
The present invention relates to a kind of multi-view angle three-dimensional human face scanning data automatic registration method.
Background technology
The registration of three-dimensional human face scanning data is an important step of setting up the three-dimensional configuration model, registration problems comprises searches the translation and the rotation parameter that can make the correct registration of various visual angles overlapped object part, can rebuild this object by part surface, the full surface that obtains this object is described.For three-dimensional face, obtain the complete description of people's face, need at least the scan-data of (front, left surface, right flank) under three different visual angles is registrated to together.
According to the difference of seeking the strategy of corresponding relation between the model, method for registering can be divided into whole registration and registration two big classes piecewise.Whole registration is to be research object with whole model, and registration is earlier model to be decomposed according to certain rule piecewise, is research object with the subdivision after decomposing then.Be proceed step by step or settle at one go according to registration, method for registering can be divided into thick essence in conjunction with registration and direct registration two big classes.Current most of method for registering all is the mode that adopts thick essence to combine, and at first defines the limited number key point, calculates pairing conversion parameter, as the original state of smart registration.Smart registration generally adopts the method for pointwise correspondence, and block mold is calculated transformational relation.Direct registration is corresponding to the smart registration computing in the thick smart combination.The registration strategies of thick smart combination differs two bigger models for spatial relationship, and the model of this body structure and features of shape more complicated, can significantly reduce the operand of registration work.And directly registration mainly is applicable to the situation that model itself is relatively simple for structure.
With reference to Fig. 3, document " Xiaoguang Lu, Dirk Colbry and Anil K.Jain.Three-Dimensional Model BasedFace Recognition.ICPR, 2004 " discloses a kind of three-dimensional face method for registering of thick smart combination.This method utilizes the ShapeIndex algorithm to find inside and outside canthus point and prenasale in the different visual angles three-dimensional human face scanning data automatically, with Least SquareFitting method detected three unique points is carried out thick registration; Again whole scanning data is carried out accurate registration with a kind of Hybrid ICP method of improving one's methods to ICP (IterativeClosest Point) method.The detection of unique point is under the hypothesis people face data condition vertical with visual plane, is determined by the priori of the value of the Shape Index that calculates and the distribution of people's face face.But there is following problem in this method: at first the feature point detecting method face of will asking for help must be vertical; Secondly there is the situation of flase drop in feature point detection, directly influences the effect of thick registration; This method for registering is the registration between 2.5 dimension scan-datas and the complete three-dimensional face model once more, does not have the situation of data disappearance, therefore can not be used for the registration between the multi-view angle three-dimensional scan-data.
Summary of the invention
Not high in order to overcome prior art to the feature point detection accuracy, can not be used for the multi-view angle three-dimensional scanning data automatic registration deficiency, the invention provides a kind of multi-view angle three-dimensional human face scanning data automatic registration method, wherein thick registration adopts the method that coordinate axis is changed, replace individual some feature with global feature, reduced the error that feature point detection is brought; Smart registration only carries out the registration interative computation to the characteristic area that extracts, and can reduce computational complexity.
The technical solution adopted for the present invention to solve the technical problems: a kind of multi-view angle three-dimensional human face scanning data automatic registration method is characterized in may further comprise the steps:
(a) multi-view angle three-dimensional human face scanning data is carried out coordinate axis conversion with the PCA method, be transformed into the same coordinate system of positive criteria model under;
(b), and adopt the method for Threshold Segmentation to isolate inner eye corner, the tail of the eye, these mark region of nose to its Shape Index value of Model Calculation after the conversion;
(c) set up people's face mark region distributed model, it is as a token of regional to find out the connected region that meets distributed model most, the computation model parameter x
Ij, i=1 ..., m, j=1 ..., 10; After all m model tried to achieve parameter, by
Obtain the average and the variance of these parameters, be unified people's face mark region distributed model; Adopt the distribution screening method, at first analyze the eye socket candidate region in the segmentation result, by comparing the area of all connected domains of eye socket candidate region, two that get the area maximum as the eye socket zone; Secondly, travel through the connected domain in all prenasale candidate regions, calculate the distance of mark region distributed model corresponding parameter, get distance minimum as the nose zone; At last, travel through the connected domain in all tail of the eye candidate regions, calculate the distance of mark region distributed model corresponding parameter, get distance minimum as tail of the eye zone, finally orient mark region: two zones, two zones of the tail of the eye and prenasale zones in the eyes;
(d) adopt the ICP method to carry out registration to the mark region of orienting, obtain corresponding rotation matrix and translation vector;
(e) conversion parameter that whole scanning data is adopted regional registration obtain carries out coordinate translation and rotational transform, finishes registration.
The invention has the beneficial effects as follows: because the thick registration of the present invention adopts the method that coordinate axis is changed, replace individual some feature, reduced the error that feature point detection is brought with global feature; Smart registration only carries out the registration interative computation to the characteristic area that extracts, reduced computational complexity, realized the autoregistration of colourful attitude 3 d scan data, the registration accuracy reaches more than 95.8%, the junction smoothness is better, modal distance average average is 1.67mm behind the registration, and standard deviation mean value is 1.04mm.
Below in conjunction with drawings and Examples the present invention is elaborated.
Description of drawings
Fig. 1 is a multi-view angle three-dimensional human face scanning data automatic registration method process flow diagram of the present invention.
Fig. 2 sets up people's face mark region distributed model synoptic diagram among Fig. 1.
Fig. 3 is a prior art multi-view angle three-dimensional human face scanning data method for registering process flow diagram.
Embodiment
With reference to Fig. 1, Fig. 2.Colourful attitude 3 d scan data autoregistration technology comprises the conversion of PCA coordinate axis; The characteristic area screening; The mark region location; The ICP registration; Whole scanning data is carried out five steps of coordinate conversion.
1. multi-view angle three-dimensional human face scanning data is carried out the coordinate axis conversion with the PCA method:
People's face main shaft coordinate system is three coordinate systems that main shaft constituted by the three-dimensional face data, three main shafts respectively corresponding people's faces towards axle Z axle, the remarkable face symmetrical plane axle Y-axis vertical with the Z axle, and perpendicular to the axle X-axis of symmetrical plane, three axles meet right-handed coordinate system, and the initial point of coordinate system is taken at prenasale.Under this coordinate system, people's face is identical positive attitude.
The concrete steps of PCA (Principal Component Analysis) coordinate axis conversion are:
For point set X (x
1, x
2..., x
N), x
iBe the m dimensional vector, calculate its average o, and then calculate its covariance matrix cov:
Try to achieve the eigenwert (λ of covariance matrix cov
1, λ
2..., λ
m) and characteristic of correspondence vector (u
1, u
2..., u
m).(u
1, u
2..., u
m) be point set X (x
1, x
2..., x
N) main shaft.To the data that strictness is symmetrically distributed, PCA can be in the hope of its axis of symmetry.Suppose the strict symmetry of three-dimensional face data, three proper vector (u that PCA tries to achieve
1, u
2, u
3) will distinguish the XYZ axle of corresponding people's face main shaft coordinate system.The cardinal principle of PCA method is to utilize people's face all to be the characteristics of identical positive attitude under its main shaft coordinate system, to master pattern and model subject to registration, calculate its accurate main shaft coordinate system, and its coordinate all is transformed under the main shaft coordinate system, make master pattern and model subject to registration be positive attitude, thereby realize the thick registration of three-dimensional face model.
2. characteristic area screening:
Calculate the Shape Index value at number of scans strong point, Shape Index value is carried out Region Segmentation, cut zone is carried out mark, obtain the candidate region, concrete grammar is as follows:
Adopt Shape Index feature to represent the concavo-convex degree of every bit in the scan-data.The Shape Index of point p is calculated by its minimax curvature.The definition of Shape Index is as follows:
Wherein, κ
1Be maximum curvature, κ
2Be minimum curvature.
Shape Index feature has reflected the convex and concave feature on people's face surface really, in inner eye corner, the tail of the eye, these mark region of nose, Shape Index feature all presents certain aggregation, be easy to separate, thereby can adopt the method for Threshold Segmentation isolate these mark region Shape Index feature with its peripheral region.
3. mark region is located:
By cutting apart and zone marker, obtain the candidate region of inner eye corner, the tail of the eye and nose, but also had a lot of interference regions in the segmentation result, need from these zones, filter out real mark region.Because the relative distribution of each organ is uniformly stable on people's face surface, the distribution of interference region then is unsettled, therefore can utilize relative distribution characteristics to come the screening of constraint, thus, we adopt a kind of regional selection method based on geometric constraints, the basic ideas of this method are: at first set up unified people's face mark region distributed model, and the three-dimensional face images new to a width of cloth, it is as a token of regional to seek the connected region that meets distributed model most.
Set up distributed model:
Select m representative front three-dimensional face model to set up distributed model as training sample.To each faceform, manual markings goes out two inner eye corners, two tail of the eyes and nose zone, generates distribution plan, and each parameter of calculating chart.
In Fig. 2, D1 is a right eye inner eye corner regional center, and D2 is a left eye inner eye corner regional center, and D3 is right tail of the eye regional center, and D4 is left tail of the eye regional center, and D5 is the nose regional center.Obtained that these regional centers just can be calculated the parameter of each line of centres, the parameter of line angle is set up model.In order to guarantee under the various attitudes accurately witness marker point, selected parameter must have the rotation translation invariance.
In order to express easily, these improve parameter unifications are expressed as x
Ij, i=1 ..., m, j=1 ..., 10.After all m model tried to achieve parameter, just can obtain the average and the variance of these parameters, these mean parameters and variance are unified people's face mark region distributed model.
The mark region screening:
To a new three-dimensional face model, at first passing threshold is cut apart and zone marker, obtains the candidate region of each mark region, therefrom selects the zone that meets the mark region distributed model most then.Adopt the distribution screening method, concrete steps are:
1) the eye socket candidate region in the analysis segmentation result, the connected domain area at two eye socket places is maximum in most cases, therefore at first by comparing the area of all connected domains of eye socket candidate region, two that get the area maximum as the eye socket zone;
2) travel through connected domain in all prenasale candidate regions, calculate the distance of mark region distributed model corresponding parameter, get distance minimum as the nose zone;
3) travel through connected domain in all tail of the eye candidate regions, calculate the distance of mark region distributed model corresponding parameter, get distance minimum as tail of the eye zone;
By the substep screening, can effectively utilize priori, reduce computation complexity.For positive surface model, The selection result is very desirable; For attitude deflection situation, owing to there is the data disappearance, can produce disappearance zone location error situation, but other all accurate positionings of zone.
For getting rid of wrong zone, location, the result after the screening is carried out aftertreatment.To each mark region, the center, zoning is to the distance at other mark region centers, surpasses a certain threshold value if these distances differ with respective distances in the mark region distributed model, thinks that then the zone loses the zone that the location is wrong.Threshold value is taken as 3 σ in the distributed model.
4.ICP registration:
To 5 mark region orienting, carry out further accurately registration by ICP.ICP is a three-dimensional data method for registering the most commonly used, this method is in each iterative process, treat each point on the registration model, in master pattern, seek the most close point, utilize this group corresponding point, calculate corresponding rotation matrix and translation vector, it is acted on the model subject to registration, obtain new model substitution iterative process next time.
Than large-sized model, mark region is 4 for deflection angle.
5. whole scanning data is carried out coordinate conversion:
Characteristic area carries out interative computation by the ICP method, obtains the registration parameter of one group of optimum, and rotation and translation vector adopt the registration parameter that obtains to carry out coordinate transform block mold, finish registration.
ICP exist registration speed slow, may be absorbed in local optimum, can't find shortcoming such as corresponding point.The present invention adopts the point treat in the mark region that registration model orients, and seeks closest approach on master pattern.This method greatly reduces calculated amount, improves registration speed.
Claims (3)
1. multi-view angle three-dimensional human face scanning data automatic registration method is characterized in that may further comprise the steps:
(a) multi-view angle three-dimensional human face scanning data is carried out coordinate axis conversion with the PCA method, be transformed into the same coordinate system of positive criteria model under;
(b), and adopt the method for Threshold Segmentation to isolate inner eye corner, the tail of the eye, these mark region of nose to its Shape Index value of Model Calculation after the conversion;
(c) set up people's face mark region distributed model, it is as a token of regional to find out the connected region that meets distributed model most, the computation model parameter x
Ij, i=1 ..., m, j=1 ..., 10; After all m model tried to achieve parameter, by
Obtain the average and the variance of these parameters, be unified people's face mark region distributed model; Adopt the distribution screening method, at first analyze the eye socket candidate region in the segmentation result, by comparing the area of all connected domains of eye socket candidate region, two that get the area maximum as the eye socket zone; Secondly, travel through the connected domain in all prenasale candidate regions, calculate the distance of mark region distributed model corresponding parameter, get distance minimum as the nose zone; At last, travel through the connected domain in all tail of the eye candidate regions, calculate the distance of mark region distributed model corresponding parameter, get distance minimum as tail of the eye zone, finally orient mark region: two zones, two zones of the tail of the eye and prenasale zones in the eyes;
(d) adopt the ICP method to carry out registration to the mark region of orienting, obtain corresponding rotation matrix and translation vector;
(e) conversion parameter that whole scanning data is adopted regional registration obtain carries out coordinate translation and rotational transform, finishes registration.
2. multi-view angle three-dimensional human face scanning data automatic registration method according to claim 1, it is characterized in that: also comprise each mark region, the center, zoning is to the distance at other mark region centers, when differing, these distances and respective distances in the mark region distributed model surpass a certain threshold value, think that then the zone loses, the zone that the location is wrong.
3. multi-view angle three-dimensional human face scanning data automatic registration method according to claim 1, it is characterized in that: also comprise slow when ICP registration speed, may be absorbed in local optimum, in the time of can't finding corresponding point, the point in the mark region that registration model orients is treated in employing, seeks closest approach and solve on master pattern.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2007100187821A CN101127075A (en) | 2007-09-30 | 2007-09-30 | Multi-view angle three-dimensional human face scanning data automatic registration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2007100187821A CN101127075A (en) | 2007-09-30 | 2007-09-30 | Multi-view angle three-dimensional human face scanning data automatic registration method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101127075A true CN101127075A (en) | 2008-02-20 |
Family
ID=39095103
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2007100187821A Pending CN101127075A (en) | 2007-09-30 | 2007-09-30 | Multi-view angle three-dimensional human face scanning data automatic registration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101127075A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101546376A (en) * | 2009-04-28 | 2009-09-30 | 上海银晨智能识别科技有限公司 | Human biological information acquisition system, human face photo acquisition and quality testing system and method |
| CN102081733A (en) * | 2011-01-13 | 2011-06-01 | 西北工业大学 | Multi-modal information combined pose-varied three-dimensional human face five-sense organ marking point positioning method |
| CN102184410A (en) * | 2011-05-06 | 2011-09-14 | 浙江工业大学 | Three-dimensional recovered cranioface recognition method |
| CN103294894A (en) * | 2013-05-06 | 2013-09-11 | 深圳先进技术研究院 | Optimal computation method of rotating shaft in three-dimensional scanning system |
| CN103486969A (en) * | 2013-09-30 | 2014-01-01 | 上海大学 | Method and device for aligning machine vision |
| CN103714318A (en) * | 2013-12-13 | 2014-04-09 | 谭玉波 | Three-dimension face registration method |
| CN103733225A (en) * | 2011-08-16 | 2014-04-16 | 日本电气株式会社 | Characteristic point coordination system, characteristic point coordination method, and recording medium |
| CN104036546A (en) * | 2014-06-30 | 2014-09-10 | 清华大学 | Method for carrying out face three-dimensional reconstruction at any viewing angle on basis of self-adaptive deformable model |
| CN105740851A (en) * | 2016-03-16 | 2016-07-06 | 中国科学院上海生命科学研究院 | Three-dimensional face automatic positioning method and curved surface registration method and system |
| CN107423773A (en) * | 2016-05-23 | 2017-12-01 | 北京师范大学 | The autoegistration method and device of three-dimensional cranium |
| CN108280388A (en) * | 2017-01-06 | 2018-07-13 | 富士通株式会社 | The method and apparatus and type of face detection method and device of training face detection model |
| CN110933558A (en) * | 2019-12-17 | 2020-03-27 | 京东方科技集团股份有限公司 | Directional sound production method and device, ultrasonic transducer and electronic equipment |
| CN116528065A (en) * | 2023-06-30 | 2023-08-01 | 深圳臻像科技有限公司 | Efficient virtual scene content light field acquisition and generation method |
-
2007
- 2007-09-30 CN CNA2007100187821A patent/CN101127075A/en active Pending
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101546376A (en) * | 2009-04-28 | 2009-09-30 | 上海银晨智能识别科技有限公司 | Human biological information acquisition system, human face photo acquisition and quality testing system and method |
| CN102081733A (en) * | 2011-01-13 | 2011-06-01 | 西北工业大学 | Multi-modal information combined pose-varied three-dimensional human face five-sense organ marking point positioning method |
| CN102184410A (en) * | 2011-05-06 | 2011-09-14 | 浙江工业大学 | Three-dimensional recovered cranioface recognition method |
| CN102184410B (en) * | 2011-05-06 | 2012-12-12 | 浙江工业大学 | Three-dimensional recovered cranioface recognition method |
| US9286543B2 (en) | 2011-08-16 | 2016-03-15 | Nec Corporation | Characteristic point coordination system, characteristic point coordination method, and recording medium |
| CN103733225A (en) * | 2011-08-16 | 2014-04-16 | 日本电气株式会社 | Characteristic point coordination system, characteristic point coordination method, and recording medium |
| CN103733225B (en) * | 2011-08-16 | 2016-08-17 | 日本电气株式会社 | Characteristic point peer system, characteristic point counterpart method and record medium |
| CN103294894A (en) * | 2013-05-06 | 2013-09-11 | 深圳先进技术研究院 | Optimal computation method of rotating shaft in three-dimensional scanning system |
| CN103294894B (en) * | 2013-05-06 | 2016-12-28 | 深圳先进技术研究院 | A kind of optimized calculation method of the rotating shaft in 3 D scanning system |
| CN103486969A (en) * | 2013-09-30 | 2014-01-01 | 上海大学 | Method and device for aligning machine vision |
| CN103486969B (en) * | 2013-09-30 | 2016-02-24 | 上海大学 | Machine vision alignment methods and device thereof |
| CN103714318A (en) * | 2013-12-13 | 2014-04-09 | 谭玉波 | Three-dimension face registration method |
| CN104036546A (en) * | 2014-06-30 | 2014-09-10 | 清华大学 | Method for carrying out face three-dimensional reconstruction at any viewing angle on basis of self-adaptive deformable model |
| CN104036546B (en) * | 2014-06-30 | 2017-01-11 | 清华大学 | Method for carrying out face three-dimensional reconstruction at any viewing angle on basis of self-adaptive deformable model |
| CN105740851A (en) * | 2016-03-16 | 2016-07-06 | 中国科学院上海生命科学研究院 | Three-dimensional face automatic positioning method and curved surface registration method and system |
| CN107423773A (en) * | 2016-05-23 | 2017-12-01 | 北京师范大学 | The autoegistration method and device of three-dimensional cranium |
| CN107423773B (en) * | 2016-05-23 | 2020-02-14 | 北京师范大学 | Automatic registration method and device for three-dimensional skull |
| CN108280388A (en) * | 2017-01-06 | 2018-07-13 | 富士通株式会社 | The method and apparatus and type of face detection method and device of training face detection model |
| CN110933558A (en) * | 2019-12-17 | 2020-03-27 | 京东方科技集团股份有限公司 | Directional sound production method and device, ultrasonic transducer and electronic equipment |
| CN110933558B (en) * | 2019-12-17 | 2023-07-28 | 京东方科技集团股份有限公司 | Directional sounding method and device, ultrasonic transducer and electronic equipment |
| CN116528065A (en) * | 2023-06-30 | 2023-08-01 | 深圳臻像科技有限公司 | Efficient virtual scene content light field acquisition and generation method |
| CN116528065B (en) * | 2023-06-30 | 2023-09-26 | 深圳臻像科技有限公司 | Efficient virtual scene content light field acquisition and generation method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101127075A (en) | Multi-view angle three-dimensional human face scanning data automatic registration method | |
| CN102880866B (en) | Method for extracting face features | |
| Xia et al. | A fast edge extraction method for mobile LiDAR point clouds | |
| CN100382751C (en) | Canthus and pupil location method based on VPP and improved SUSAN | |
| CN103727930B (en) | A kind of laser range finder based on edge matching and camera relative pose scaling method | |
| Guo et al. | Lane detection method based on improved RANSAC algorithm | |
| CN101894254B (en) | Contouring method-based three-dimensional face recognition method | |
| CN106682586A (en) | Method for real-time lane line detection based on vision under complex lighting conditions | |
| CN107909059A (en) | It is a kind of towards cooperateing with complicated City scenarios the traffic mark board of bionical vision to detect and recognition methods | |
| CN103886589A (en) | Goal-oriented automatic high-precision edge extraction method | |
| CN104091155A (en) | Rapid iris positioning method with illumination robustness | |
| CN105740798A (en) | Structure analysis based identification method for object in point cloud scene | |
| CN103489011A (en) | Three-dimensional face identification method with topology robustness | |
| CN102622587A (en) | Hand back vein recognition method based on multi-scale second-order differential structure model and improved watershed algorithm | |
| CN111145228A (en) | Heterogeneous image registration method based on local contour point and shape feature fusion | |
| CN104408462A (en) | Quick positioning method of facial feature points | |
| Mörwald et al. | Geometric data abstraction using b-splines for range image segmentation | |
| Meng et al. | Efficient and reliable LiDAR-based global localization of mobile robots using multiscale/resolution maps | |
| CN114359876B (en) | Vehicle target identification method and storage medium | |
| CN106845482A (en) | A kind of license plate locating method | |
| CN106611158A (en) | Method and equipment for obtaining human body 3D characteristic information | |
| Ding et al. | Efficient vanishing point detection method in complex urban road environments | |
| Moosmann | Interlacing self-localization, moving object tracking and mapping for 3d range sensors | |
| CN110210415A (en) | Vehicle-mounted laser point cloud roadmarking recognition methods based on graph structure | |
| CN106682575A (en) | Human eye point cloud feature location with ELM (Eye Landmark Model) algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |