CN1296148C - Visual data processing system for fruit external appearance quality online detection technology - Google Patents
Visual data processing system for fruit external appearance quality online detection technology Download PDFInfo
- Publication number
- CN1296148C CN1296148C CN 200510038530 CN200510038530A CN1296148C CN 1296148 C CN1296148 C CN 1296148C CN 200510038530 CN200510038530 CN 200510038530 CN 200510038530 A CN200510038530 A CN 200510038530A CN 1296148 C CN1296148 C CN 1296148C
- Authority
- CN
- China
- Prior art keywords
- image
- fruit
- module
- dynamic
- image processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Abstract
The present invention relates to a method for realizing software of a machine vision detection system for real-time and on-line detection and grading, which is composed of a high-quality dynamic image obtaining module, an image dynamic exchange module under external trigger control and an image processing module, wherein the modules are orderly connected through image vision data signals; the high-quality dynamic image obtaining module comprises a module for collecting original images which are shot by a camera in real time, a Bayer conversion module for decoding collected color codes and a Gamma correction module for carrying out linear processing to data; the system continuously detects a triggering interface by the deletion action which is taken in a camera shooting module, an image processing module and a buffer area by the image dynamic exchange module under external trigger control; the image processing module comprises an image quickly dividing module, a sequential image orderly storing module and other image processing modules; the image quickly dividing module carries out background division and single image division in the visual field. Through the intelligent vision identification of a visual system, the grade of each fruit is comprehensively judged, and the position information of each fruit is determined to provide judgement references for the on-line detection of fruit appearance quality.
Description
Affiliated technical field
The present invention relates to the software implementation method of the Machine Vision Detection system of a kind of real-time online detection and classification, refer in particular to a kind of visual data processing system of fruit external appearance quality online detection technology.
Background technology
In recent years, the classification that utilizes machine vision technique to carry out fruit obtains extensive studies, related U.S. patent has, people's such as Yang Tao United States Patent (USP) " Method and apparatus for sorting objects by color (article being carried out the method and apparatus of classification by color) application number: 5339963 ", with a colour imagery shot article on the carrier chain are carried out fast detecting and classification, mainly use the colourity (H) in the HIS color system to carry out classified calculating.People's such as Yang Tao United States Patent (USP) " Methodand apparatus for sorting objects by color including stable color tansformation (method and apparatus that article is carried out classification with color comprises a kind of conversion method of still image) application number: 5533628 " has been described a kind of fruit grading system based on single camera.Domestic, there are people such as Ying Yibin in a plurality of relevant Chinese patent that they applied for (application number: 02136377.3,02266031.3,02160193.302295073.7,02295073.7), also to describe a kind of fruit grading system based on single camera; Number of patent application: in 200410065216.2 by selecting for use at a high speed data signal camera to solve in the past analog signal camera hypograph because fuzzy and many noise problems that motion produces, utilize the online shooting simultaneously of three camera systems simultaneously, can detect the work that single camera systems such as full surface image of fruit and detection fruit defects can't be finished.These systems are made up of fruit conveying, turning system, Computer Vision Recognition system, hierarchy system.Fruit is fed forward with certain speed on conveying device, and fruit is freely rotated around trunnion axis, the image information of camera picked-up fruit surface.The grade of each fruit is comprehensively judged in Visual intelligent identification by computer vision system, and determines the positional information of each fruit, by the control module of computer recognition system instruction is transferred to hierarchy system, finishes the classification of fruit.
But in computer vision system by image pick-up card how to obtain high quality graphic, how to make image in dynamic exchange under the external trigger control, how tentatively to cut apart fast and effectively and how to make sequence image handle in order and store etc. and severally must realize not appearing in the newspapers through the software of step.
Summary of the invention
The invention provides a kind of visual data processing system system of fruit external appearance quality online detection technology, they can be at existing a kind of apparatus and method based on three online fruit quality detection and classifications of camera system, Chinese patent application number: 200410065216.2, on fruit detection line at a high speed, realize that with software external trigger control obtains the image in the high-speed figure camera and sequence image cut apart fast, handles in order and storage.
Technical scheme of the present invention is as follows:
The present invention by the high-quality dynamic image obtain, dynamic exchange and the image of image under external trigger control handle three modules that are connected in sequence through the image vision data-signal and form.
The obtaining of described high-quality dynamic image comprises functional modules such as original image collection, Bayer conversion and Gamma correction, original image is to take extremely by Matrox meteorII series integrated circuit board and Uniq-uc610 camera in real time to online fruit to be detected, and the original image that collects is 10 a monochrome image.The Bayer conversion is exactly a process of decoding according to the coloud coding of original monochrome image self.Thereby the method that Gamma proofreaies and correct is exactly to make the value of frame buffer zone and final display brightness linear by non-linear stretching once.Take out the rgb value of each pixel, select for use correction Gamma=0.45 (0.45=1/2.2) to adjust i.e. R '=255 (R/255)
0.45G '=255 (G/255) 0.45B '=255 (B/255)
0.45Then R ' G ' B ' being composed respectively to obtain effect preferably for original R, G, B value.
The dynamic exchange of described image under external trigger control is meant and is achieved as follows function:
(1) when the software systems of whole detection line bring into operation, deletion action just enters a unlimited wait state in camera shooting, image processing module and the buffering area, also is the uninterrupted detection that system carries out trigger interface.
(2) fruit produces a triggering signal A along with the moving of production line in some positions, and signal A is captured by system and notifies camera to take action simultaneously.
(3) camera shooting end horse back justice is in SBR, and the next one that waiting system sends is taken signal.
(4) image that photographs enters a buffering area subsequently, and sends message B to image processing module.
(5) image processing module is constantly surveyed buffering area, judges whether buffering area sends message B effective.If effectively, just begin image processing operations truly, image is handled the back and is sent message C.
(6) deletion action is judged message B and message C in the buffering area, carries out deletion action when both are effective simultaneously, comes back to after deletion action is finished (1).
Described image processing module comprises the image processing modules such as orderly storage of the cutting apart fast of image in the visual field, sequence image.
Comprise that background segment, monomer function such as cut apart cutting apart fast of image in the described visual field, promptly except will from background, separating object, also need separate three fruit separately (being that monomer is cut apart), and also will be limited to them within the minimum scope in order to reduce the follow-up processing time.
Described background segment adopts can be fit to on-line operation, at the particularity of fruit image, proposes a kind of trapping method and cuts apart background.Described trapping method is realized by following operation: the fruit image on the black background is used as trap, and carpopodium fruit calyx on the fruit or defective are exactly the island in the trap naturally.Record enters the position of trap and climbs out of the position that falls into into during certain delegation of scan image, outside the two positions as a setting.Also have corresponding turnover position when meeting island, can ignore island by setting mark.
Described monomer is cut apart i.e. realization several fruit images in the field-of-view image separately, and single fruit image is limited in its boundary rectangle.Monomer is cut apart by jumping the realization of lattice method, jumps the lattice method and carries out by the following method:
(1) in the entire image zone, begins to carry out single step scanning, up to running into the fruit pixel, thereby obtain leftmost point from high order end.
(2) scan method is just to have run into the fruit pixel just to jump out circulation this time midway, carries out next stringer scanning then.
(3) till having a stringer not scan the fruit pixel, program judgement this moment order longitudinal scanning finishes.
(4) the rightmost point of the not corresponding fruit of last column scan line finds fruit pixel, the end of scan so also need carry out taking turns the vertical single step flyback of backward.
(5) obtain subimage up and down 2 the transversal scanning principle similarly.
(6) based on these 4 points, make the apple boundary rectangle.
The orderly storage of described sequence image is by defining the Three-Dimensional Dynamic array R[3 of a 3*3*4] [3] [4], first dimension expression three number of sub images (also being the image of single fruit) of this array, the image of three different angles of second certain single fruit of dimension expression, the third dimension is represented size under certain angle, shape, color, four characteristic values of defective.The first dimension subscript of array and the relation of triggering times
N=(I-1) %3 wherein I is a triggering times.
The second dimension subscript, 0 corresponding a certain apple first width of cloth image, 1 corresponding a certain apple second width of cloth image, 2 corresponding a certain apple the 3rd width of cloth images.
The element of this dynamic array constantly entered empty, but (the automatic digital on-line test system) technology of using ADO has at last been write into database to the value that dynamic array empties eve.
The present invention is by above technical scheme, the following effect that obtains: by using the Bayer switch technology, the Gamma alignment technique carries out a series of conversion to the second-rate monochrome image of camera output, obtained satisfied image, substantially solved the problem that the image primary signal is obtained, for a series of processing procedures that need carry out this image thereafter lay the foundation.Jump the lattice method and cut apart subimage, not only background is removed fully but also kept fruit original characteristics (as carpopodium, fruit calyx, defective etc.) to greatest extent, speed is very fast simultaneously, handles for whole procedure and has saved many times.The buffered of image and multithreading are intended to improve running efficiency of system.The algorithm of sequence image is that the follow-up mechanically actuated of online detection is laid a good foundation, and all operation results are also real-time has deposited database in, for comparison test provides foundation.
Description of drawings
Annexation schematic diagram between each functional module of Fig. 1;
The color-code figure of Fig. 2 UC-610
The dynamic exchange flow chart of Fig. 3 image under external trigger control;
Fig. 4 jumps the portraitlandscape scanning schematic diagram of lattice method monomer in cutting apart;
The demonstration of Fig. 5 sequence image;
The specific embodiment
Below in conjunction with accompanying drawing three functional modules of the present invention are further described:
As shown in Figure 1, three functional modules are connected in sequence by the image vision data-signal.
1. the acquisition module of high-quality dynamic image: this module comprises functional modules such as original image collection, Bayer conversion and Gamma correction, original image is by Matrox meteorII integrated circuit board and Uniq-uc610 camera online fruit to be detected to be taken in real time to obtain, the original image that collects is 10 a monochrome image, and it is encoded as shown in Figure 2; Other chrominance components that Bayer changes this pixel just can obtain by peripheral pixel.If promptly the source image element is the G value, then the R of target pixel, B value are tried to achieve by the mean value of 2 neighborhoods of source image element; If the source image element is R or B, so two color values of all the other of target pixel then the mean value of 4 neighborhoods by the source image element try to achieve.Very high by the color of image validity after this method conversion.Thereby the method that Gamma proofreaies and correct is exactly to make the value of frame buffer zone and final display brightness linear by non-linear stretching once.Take out the rgb value of each pixel, select for use correction Gamma=0.45 (0.45=1/2.2) to adjust i.e. R '=255 (R/255)
0.45G '=255 (G/255)
0.45B '=255 (B/255)
0.45Then R ' G ' B ' being composed respectively to obtain effect preferably for original R, G, B value.
2. the dynamic exchange module of image under external trigger control:
The flip flop equipment that adopts is the photoelectricity coupling sensor, and transmitting terminal and receiving terminal are arranged.When system brings into operation, trigger and just enter a unlimited wait state, also be the uninterrupted detection that system carries out trigger interface.Fruit is along with production line moves, having cut off transmitting terminal is communicated with the light of receiving terminal, because it is that effectively so just produce a triggering signal from being communicated to blocking-up, this signal is captured by system and notifies camera to take action simultaneously that system is arranged to from high level to low level process.Camera is taken and is finished to be in again SBR, and the next one that waiting system sends is taken signal, so goes round and begins again at once.The image that photographs enters a buffering area subsequently, waits for the operation of image processing module.Image processing module also starts when system brings into operation, has also entered a unlimited wait state, constantly surveys buffering area then, judges whether buffering area has effective information.If find effective information, just begin image processing operations truly.Because the finite capacity of buffering area, so the image that processing finishes need be deleted from buffering area.Deletion action also will enter a unlimited wait state, enter when information is arranged in buffering area, and wait for be image processing module behind EO, send over finish signal.The program flow diagram of this process such as Fig. 3.
3. described image processing module comprises the image processing modules such as orderly storage of the cutting apart fast of image in the visual field, sequence image.
(1) in the visual field image cut apart module fast:
Outside with trapping method object being separated from background, three fruit are separated separately (being that monomer is cut apart) by jumping the lattice scanning method, and single fruit image is limited in its boundary rectangle.Wherein trapping method is: the fruit image on the black background is used as trap, carpopodium fruit calyx on the fruit or defective are exactly the island in the trap naturally, record enters the position of trap and climbs out of the position that falls into into during certain delegation of scan image, outside the two positions as a setting, also have corresponding turnover position when meeting island, can ignore island by setting mark.Be partitioned into image after the background with trapping method, not only background removed fully but also kept fruit original characteristics (as carpopodium, fruit calyx, defective etc.) to greatest extent.Jump the lattice scanning method as shown in Figure 4, at first in the entire image zone, begin to carry out single step scanning, up to running into the fruit pixel, thereby obtain leftmost point from high order end.For reducing sweep time, resemble at fruit and to jump lattice scanning on the rope.Be not to scan a whole stringer shown in the image pattern 4 to jump lattice more midway yet, just jump out circulation this time, carry out next stringer scanning then but just run into the fruit pixel.So go down to have a stringer not scan the fruit pixel, program judgement this moment order longitudinal scanning finishes.The rightmost point of the not corresponding fruit of last column scan line is so also need carry out taking turns the backward longitudinal scanning.The rightmost point of last column scan line and fruit is not far from one another, is not suitable for doing jumping lattice scanning again, but carries out the single step flyback, finds the fruit pixel, the end of scan.Obtain subimage up and down 2 the transversal scanning principle similarly.
(2) the orderly memory module of sequence image:
Patent (application number: 200410065216.2) in order to photograph the fruit whole surface image, used three cameras, gone to take from different angles.Fruit carries out rotation while doing rectilinear motion, and purpose also is an exposure self information as much as possible.Three fruit are arranged like this by turns in the visual field, a fruit has three width of cloth images, and every width of cloth image all has four parameters, so orderly storage that will sequence image.Three number of sub images have been defined, 1,2, No. 3 image (see figure 5) after storage is cut apart respectively for this reason.The Three-Dimensional Dynamic array of a 3*3*4 of definition, first dimension expression three number of sub images (also being the image of single fruit) of this array, the image of three different angles of second certain single fruit of dimension expression, the third dimension is represented size under certain angle, shape, color, four characteristic values of defective.
The rule of sequence image: (the preceding bidimensional of a peek group, that is: R[3 for simplicity ,] [3])
Wherein " I " expression triggering signal amount triggers one-accumulate once.
I=1 R[0] [0] (first first width of cloth)
I=2 R[1] [0] (first width of cloth of second) R[0] [1] (first second width of cloth)
I=3 R[2] [0] (first width of cloth of the 3rd) R[1] [1] (second width of cloth of second) R[0] [2] (first the 3rd width of cloth)
I=4 R[0] [0] (first width of cloth of the 4th) R[2] [1] (second width of cloth of the 3rd) R[1] [2] (the 3rd width of cloth of second)
I=5 R[1] [0] (first width of cloth of the 5th) R[0] [1] (second width of cloth of the 4th) R[2] [2] (the 3rd width of cloth of the 3rd)
I=6 R[2] [0] (sextus first width of cloth) R[1] [1] (second width of cloth of the 5th) R[0] [2] (the 3rd width of cloth of the 4th)
I=7 R[0] [0] (first width of cloth of the 7th) R[2] [1] (sextus second width of cloth) R[1] [2] (the 3rd width of cloth of the 5th)
I＝8.........
Can obtain three conclusions by last surface analysis:
1) in three width of cloth subimage, the left side one width of cloth is represented first width of cloth image of certain fruit forever, and a middle width of cloth is represented second width of cloth image of certain fruit forever, and the right one width of cloth is represented the 3rd width of cloth image of certain fruit forever, this is a constant principle, and variation is that fruit is in continuous turnover.
2) when the 3rd fruit comes into view (triggering signal I=3), triggering signal I=6 when the arrangement of array element and the 6th fruit come into view) just the same, so it is every through three triggerings, the arrangement of array element begins circulation, so get variable X=I%3, from I=1, X equals 1,2,0 respectively; 1,2,0; 1,2,0...
3) I=1 and I=2 are special cases, need separately to consider.Again because the X=1,2,0 of front; 1,2,0..., do not meet order from small to large, thus allow X=(I-1) %3 again, like this, X=0,1,2; 0,1,2...
The element of this dynamic array constantly entered empty, but use the ADO technology that the value that dynamic array empties eve has been write into database at last.
Claims (6)
1. the visual data processing system of a fruit external appearance quality online detection technology, it is characterized in that by the high-quality dynamic image obtain, dynamic exchange and the image of image under external trigger control handle three modules that are connected in sequence through the image vision data-signal and form; The Gamma correction module that the obtaining of wherein said high-quality dynamic image comprises the original image acquisition module that utilizes camera to take in real time, the coloud coding of gathering is decoded the Bayer modular converter and data carried out linear processing; The dynamic exchange of described image under external trigger control by camera shooting, image processing module and buffering area in deletion action, the uninterrupted detection that the realization system carries out trigger interface; Described image processing module comprises in the visual field that the image of cutting apart implementation by background segment, monomer is cut apart fast, the image processing modules such as orderly storage of sequence image.
2. according to the visual data processing system of a kind of fruit external appearance quality online detection technology of claim 1, it is characterized in that original image is by Matrox meteorII series integrated circuit board and Uniq-uc610 camera online fruit to be detected to be taken in real time to obtain in the high-quality dynamic image acquisition module.
3. according to the visual data processing system of a kind of fruit external appearance quality online detection technology of claim 1, it is characterized in that the dynamic exchange process of described image under external trigger control is as follows:
(1) when the software systems of whole detection line bring into operation, deletion action just enters a unlimited wait state in camera shooting, image processing module and the buffering area, also is the uninterrupted detection that system carries out trigger interface;
(2) fruit produces a triggering signal A along with the moving of production line in some positions, and signal A is captured by system and notifies camera to take action simultaneously;
(3) camera is taken and is finished to be in again SBR, and the next one that waiting system sends is taken signal at once;
(4) image that photographs enters a buffering area subsequently, and sends message B to image processing module;
(5) image processing module is constantly surveyed buffering area, judges whether buffering area sends message B effective.If effectively, just begin image processing operations truly, image is handled the back and is sent message C;
(6) deletion action is judged message B and message C in the buffering area, carries out deletion action when both are effective simultaneously, comes back to the first step after deletion action is finished.
4. according to the visual data processing system of a kind of fruit external appearance quality online detection technology of claim 1, it is characterized in that described background segment is that trapping method is cut apart background, it is specially: the fruit image on the black background is used as trap, and carpopodium fruit calyx on the fruit or defective are exactly the island in the trap naturally; Record enters the position of trap and climbs out of the position that falls into into during certain delegation of scan image, outside the two positions as a setting; Also have corresponding turnover position when meeting island, can ignore island by setting mark.
5. according to the visual data processing system of a kind of fruit external appearance quality online detection technology of claim 1, it is characterized in that described monomer cuts apart that the lattice method realizes by jumping, be specially:
(1) in the entire image zone, begins to carry out single step scanning, up to running into the fruit pixel, thereby obtain leftmost point from high order end;
(2) scan method is just to have run into the fruit pixel just to jump out circulation this time midway, carries out next stringer scanning then;
(3) till having a stringer not scan the fruit pixel, program judgement this moment order longitudinal scanning finishes;
(4) the rightmost point of the not corresponding fruit of last column scan line finds fruit pixel, the end of scan so also need carry out taking turns the vertical single step flyback of backward;
(5) obtain subimage up and down 2 the transversal scanning principle similarly;
(6) based on these 4 points, make the apple boundary rectangle.
6. according to the visual data processing system of a kind of fruit external appearance quality online detection technology of claim 1, the orderly storage that it is characterized in that described sequence image is by defining the Three-Dimensional Dynamic array R[3 of a 3*3*4] [3] [4], the first dimension expression, three number of sub images of this array, the image of three different angles of second certain single fruit of dimension expression, the third dimension is represented size under certain angle, shape, color, four characteristic values of defective; The element of this dynamic array constantly entered empty, but (the automatic digital on-line test system) technology of using ADO has at last been write into database to the value that dynamic array empties eve.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200510038530 CN1296148C (en) | 2005-03-23 | 2005-03-23 | Visual data processing system for fruit external appearance quality online detection technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 200510038530 CN1296148C (en) | 2005-03-23 | 2005-03-23 | Visual data processing system for fruit external appearance quality online detection technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN1663697A CN1663697A (en) | 2005-09-07 |
| CN1296148C true CN1296148C (en) | 2007-01-24 |
Family
ID=35035113
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 200510038530 Expired - Fee Related CN1296148C (en) | 2005-03-23 | 2005-03-23 | Visual data processing system for fruit external appearance quality online detection technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN1296148C (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101507962B (en) * | 2009-03-16 | 2012-11-07 | 浙江大学 | Fruit classification method according to passage |
| IT1404310B1 (en) * | 2011-02-24 | 2013-11-22 | Gdm Spa | METHOD OF CORRECTIVE INTERVENTION ON THE FUNCTIONING OF A PRODUCTION LINE OF HYGIENIC ABSORBENT ITEMS, SUCH AS PANELS, BUFFERS AND THE LIKE. |
| CN102750547B (en) * | 2012-06-11 | 2014-07-23 | 陕西科技大学 | Fruit size grading method based on compressed sensing |
| CN105964564A (en) * | 2016-05-04 | 2016-09-28 | 成都贝森伟任科技有限责任公司 | Fruit quality classifying device |
| CN109214372B (en) * | 2018-11-01 | 2021-04-02 | 深圳蓝胖子机器智能有限公司 | Attitude determination method, attitude determination device and computer-readable storage medium |
| CN109482515A (en) * | 2018-11-22 | 2019-03-19 | 珠海格力智能装备有限公司 | The sorting system of the method for sorting and device of screw, screw |
| CN111510698A (en) * | 2020-04-23 | 2020-08-07 | 惠州Tcl移动通信有限公司 | Image processing method, device, storage medium and mobile terminal |
-
2005
- 2005-03-23 CN CN 200510038530 patent/CN1296148C/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN1663697A (en) | 2005-09-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN1296148C (en) | Visual data processing system for fruit external appearance quality online detection technology | |
| CN101512549B (en) | Real-time face tracking in a digital image acquisition device | |
| US7986831B2 (en) | Image processing apparatus, image processing method and computer program | |
| DE4433329C2 (en) | Motion vector detection method and motion vector detection device | |
| CN1184502C (en) | Process for automatic focusing | |
| CN1225889C (en) | Method and circuits for intrinsic processing of image data within image sensing devices | |
| CN1220546A (en) | Method and apparatus for automatic aspect format detection in digital video pictures | |
| CN1711557A (en) | Image segmentation using template prediction | |
| CN112767711B (en) | Multi-class multi-scale multi-target snapshot method and system | |
| CN1750045A (en) | Image processing apparatus and image processing method | |
| CN1825984A (en) | Information processing method and apparatus | |
| CN105100640A (en) | Local registration parallel video stitching method and local registration parallel video stitching system | |
| CN86100610A (en) | Tv system conversion apparatus | |
| CN101047867A (en) | Method for correcting multi-viewpoint vedio color | |
| CN111539888B (en) | Neural network image defogging method based on pyramid channel feature attention | |
| CN112365404A (en) | Contact net panoramic image splicing method, system and equipment based on multiple cameras | |
| RU2325044C1 (en) | Gradient method of object contour extraction at halftone raster image matrix | |
| CN1815489A (en) | Intelligent image counting method | |
| CN1193594C (en) | Signal processing device and signal processing method | |
| CN112104869A (en) | Video big data storage and transcoding optimization system | |
| Lee et al. | Pedestrian detection based on deep fusion network using feature correlation | |
| CN101561316B (en) | On-line test visual data processing system based on region of interest (ROI) | |
| CN113163121A (en) | Video anti-shake method and readable storage medium | |
| CN113163120A (en) | Transformer-based video anti-shake method | |
| CN101040299A (en) | Image motion vector detecting device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C19 | Lapse of patent right due to non-payment of the annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |