CN1782659B - Welding seam tracking sight sensor based on laser structure light - Google Patents

Welding seam tracking sight sensor based on laser structure light Download PDF

Info

Publication number
CN1782659B
CN1782659B CN 200410009931 CN200410009931A CN1782659B CN 1782659 B CN1782659 B CN 1782659B CN 200410009931 CN200410009931 CN 200410009931 CN 200410009931 A CN200410009931 A CN 200410009931A CN 1782659 B CN1782659 B CN 1782659B
Authority
CN
China
Prior art keywords
laser
welding
weld seam
weld
sensing
Prior art date
Application number
CN 200410009931
Other languages
Chinese (zh)
Other versions
CN1782659A (en
Inventor
谭民
徐德
李原
李涛
王麟琨
沈扬
Original Assignee
中国科学院自动化研究所
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院自动化研究所 filed Critical 中国科学院自动化研究所
Priority to CN 200410009931 priority Critical patent/CN1782659B/en
Publication of CN1782659A publication Critical patent/CN1782659A/en
Application granted granted Critical
Publication of CN1782659B publication Critical patent/CN1782659B/en

Links

Abstract

The present invention relates to robot technology, and is weld seam tracking sight sensor based on laser structure light. The welding seam tracking sight sensor has laser as the light source of structure light, and consists of laser, video camera, monitor, treating unit, etc. By means of forming laser plane with laser beam in a cylindrical mirror, projecting the laser plane to a workpiece to generate laser fringes, acquiring the laser image of weld seam with a visual sensor, calculating weld seam positions through image processing and converting into analog signal or radio signal, the present invention controls welding robot for automatic weld seam tracking. Different weld seam processing modules for different kinds of weld seam are developed. The present invention has high weld seam position measuring precision, raised anti-jamming capacity and high versatility and flexibility.

Description

A kind of weld joint tracking vision sensor based on laser structure light

Technical field:

The invention belongs to the vision measurement sensing device among the robot field, specifically a kind ofly can discern position while welding in the welding process and guide the vision sensing equipment of welding gun correction of deviation by visual pattern.

Technical background:

At present, the control method of using in the welding commercial production generally adopts the mode of control manually or robot teaching or off-line programing to carry out path planning and movement programming, just simply repeats predefined action in the welding process.In existing weld joint tracking implementation, the general sensors such as machinery, electromagnetism, vision that adopt provide weld seam information, are handled by system, realize welding torch position control.But such seam tracking system generally seals, and is sensor and Processing Algorithm at particular system exploitation, lacks general, soldered sensor flexibly and independently, for general welding robot or controller provide welding seam position information.

Summary of the invention:

In order to overcome the deficiency of existing seam tracking system, the utility model provides a kind of weld joint tracking vision sensor.This sensor can structure adapts to various weld seam types and weld seam, weld seam different process repeatedly; Gather the laser welded seam image, extract the weld bead feature points position; And analog output interface circuit and network interface be provided, and can network settings working sensor parameter, with standard signal or wireless communication output position while welding.It is general vision sensor towards the weld joint tracking field.

In order to achieve the above object, technical solution of the present invention provides a kind of weld joint tracking vision sensing equipment based on laser structure light, can gather, the processing welding lines characteristic image, and output position while welding signal carries out welding technology with the welding gun of controlling on welding robot or the controller; It a) adopts a word laser that laser instrument is sent out to produce the laser structure optical plane, forms structural light stripes; B) adopt the camera acquisition weld seam feature laser stripe image that installs filter lens additional; C) the laser structure optical plane becomes 30 °~60 ° angles with camera photography head central axis.

Described sensing device, it comprises laser instrument, video camera, welding gun and processing unit, wherein, laser instrument, video camera are packaged together and welding gun is rigidly connected on the framework, the camera photography headband has optical filter, sensor and welding gun are affixed, and both central axial direction are identical, are co-located in same plane, this plane and weld seam formed plane intersect, and with the welded piece Surface Vertical; Video camera is electrically connected with processing unit, and processing unit is electrically connected with welding robot or controller by signal wire.

Described sensing device, its described processing unit, comprise computing machine and parts thereof, control panel, monitor and signal wire, computing machine is electrically connected with control panel, monitor and signal wire respectively, video camera is electrically connected with the image pick-up card of computing machine, and processing unit also is provided with analog output interface circuit and radio network interface.

Described sensing device, its described video camera is industrial camera.

Described sensing device, its laser structure optical plane and video camera be through demarcating, by the structural light stripes characteristic point coordinates as calculated computer go out the side-play amount of weld seam.

Described sensing device, its described analog output interface circuit and network interface are read the position while welding offset information with standard analog signal output or by wireless network.

Described sensing device, it utilizes network or selects by the control panel button, can set, adjust at the weld seam type.

Described sensing device, its described computing machine also is electrically connected with the weld image processing module, and the weld image processing module has the data of multiple weld seam type, is used to extract the weld bead feature points coordinate, and has expanded function.

Described sensing device, its described optical filter can be through the light wave of wavelength 670nm.

Described sensing device, its described signal output has the output of horizontal direction and vertical direction position while welding deviation.

Outstanding feature of the present invention is to adopt a word laser instrument to produce the structured light plane, adopts the ccd video camera that installs filter lens additional to carry out image acquisition, obtains welding seam position information by Flame Image Process.

The technical solution adopted for the present invention to solve the technical problems is: sensor is the center with the PC104 computing machine.A branch of line source that one word laser instrument sends constitutes the structured light plane, projects surface of the work, forms structural light stripes.After ccd video camera receives reflected light, vision signal is sent into image pick-up card, convert digital signal to through A/D.According to weld seam type and the technology selected on the sensor control panel, or the systematic parameter by setting wireless network, call corresponding image processing program (image filtering, enhancing, extraction axis of a weld, extraction weld bead feature points), after obtaining the weld seam deviation signal, through being converted to control signal, through analog output interface circuit and network interface, the position while welding offset information can be with standard analog signal (1~5V, 4~20mA) outputs or wireless network card communication are output as topworks control signal output are provided, to reach the purpose that real-time deviation is adjusted.

The invention has the beneficial effects as follows, can adapt to various weld seam types and technology, the position while welding signal is provided, is general, weld joint tracking vision sensor flexibly and independently.

Description of drawings:

Fig. 1 follows the tracks of the vision sensing equipment synoptic diagram for laser structure flush weld seam of the present invention;

Fig. 2 is the weld joint tracking vision sensor theory diagram that the present invention is based on laser structure light;

The laser image of the dissimilar weld seams that Fig. 3 (a) and (b), (c), (d) arrive for sensor acquisition of the present invention; Wherein: (a) laser image of overlap joint type welding seam, (b) trapezoidal secondary weld seam, weld seam of (c) V font butt joint and (d) directly butt-weld (middle 2~3 milliseconds of gaps).

Embodiment:

See also Fig. 1.In Fig. 1, the laser beam that semiconductor one word laser instrument (LD650-5-3) 1 sends produces the structured light plane, shines the welding work pieces surface, forms weld seam 6 feature laser stripes.Laser linewidth is 0.5mm, and the angle of spread is 20 °, and power is 5mw.WATEC505 black and white industrial camera 3 has optical filter above, can see through the light wave of wavelength 670nm.Spatter shield 5 is equipped with in video camera 3 belows, can stop the spatter thing, the protection video camera.Laser instrument 1, optical filter 2, video camera 3 and spatter shield 5 are encapsulated in the sensor casing, and sensor and welding gun 4 are rigidly connected.The position of one word laser instrument 1, angle can be adjusted, and adjust the angle of laser instrument 1 and video camera 3, make video camera 3 optical axis center lines become 30 °~60 ° angles with laser structure light face, are 45 in this example; And make video camera 3 optical axis center lines and weld seam 6 formed planes and welded piece Surface Vertical, highly be 10~20cm.Industrial camera 3 is gathered the structural light stripes image that is formed by a word laser instrument 1.

The image of weld seam 6 is sent to processing unit 7 with standard P AL standard signal.By image pick-up card (little MOKA-C10 that looks), convert digital signal to through A/D.Processing unit 7 adopts PC104 computing machine and miscellaneous part, comprises image pick-up card, PC104 bus switch amount I/O integrated circuit board PM-582, PC104 simulated-bus amount output card PM-520, wireless network card LucentPC24E-H-FC.PC104 computing machine and parts thereof constitute processing unit with control panel 8 and monitor 9.Video camera 3 collects the laser image that has weld seam 6 features, and processing unit 7 calls weld image processing module storehouse 15 and extracts weld seam streak feature dot image coordinate.

Because laser plane and video camera 3 are through accurately demarcating, can determine the image two-dimensional space to actual three-dimensional transition matrix through calculating, so just can (u v) converses actual welds point three-dimensional coordinate (x by the two-dimensional coordinate of image space weld seam streak feature point, y, z).The method for precisely marking of laser plane and video camera 3 is applied for a patent (application number: 02158341.2 denomination of invention: laser structure light vision device) in addition.Again because when welding gun 4 in the horizontal direction during move left and right, the weld seam stripe pattern is move left and right thereupon also; When under welding gun 4 in the vertical directions when mobile, the weld seam stripe pattern also moves up and down thereupon, so the horizontal-shift of weld seam 6 unique points and vertical shift have been represented the respective offsets of welding gun 4 with respect to weld seam 6 respectively on image.So also can represent the horizontal departure or the vertical missing of welding gun 4 and standard weld position with level, the vertical missing of welding seam image characteristic points coordinate and normal place.So vision sensor can be exported actual solder joint three-dimensional coordinate, perhaps with the level of welding gun 4 and standard weld position, vertical missing amount as output.

The result of the sensor horizontal departure or the vertical missing signal of weld seam 6 (solder joint three-dimensional coordinate or the welding gun 4 with) can be with signal wire 14 outputting standard simulating signals (1~5V, 4~20mA), or to read by wireless network.Two weld seam type B CD scramblers 11, each scrambler can be selected 0~90 kind of state, and totally 100 states that combine can select to be provided with multiple weld seam type; " operation selector switch " 12 selected once/repeatedly weld seams; Pilot lamp 10 and control knob 13 are arranged on the control panel 8; Operation, normal and malfunction indicator lamp can the indication sensor states; START button is determined weld seam 6 reference positions as normal place, and sensor is started working; " stop " button and stop sign as welding job, sensor stops output.The function setting of sensor and the I/O of state can realize in two ways: the parameter (weld seam type, operation) that control knob 13 on the control panel 8 and pilot lamp 10 can setting sensors, the work engineering (begin, stop, resetting) of control sensor, and the duty of indication sensor (work, stop, fault); Also can be by the parameter and the control information of radio communication setting sensor, the state of read sensor.

The course of work of sensor can be consulted the Fundamentals of Sensors block diagram of Fig. 2.The seam laser image that video camera 3 is gathered, the one tunnel delivers to monitor 9 outputs, for operator's Real Time Observation weld image; One the tunnel delivers to PC104 processing unit 7 carries out Flame Image Process.By the select command of control panel 8 or the parameter of setting wireless network, determine the type of weld seam 6, the corresponding program module of calling in the weld image handling procedure storehouse 15 is carried out the weld image processing.In the weld image handling procedure storehouse 15 at the characteristics of image of various weld seam types, developed specific image processing module (as V font, trapezoidal, butt joint, lap weld Flame Image Process etc.), constitute weld image processing capacity module library 15, and can increase new weld seam types of image processing module, have expanded function.Consulting Fig. 3, is respectively the laser image of (a) overlap joint type welding seam, (b) trapezoidal secondary weld seam, weld seam of (c) V font butt joint and (d) directly butt-weld (middle 2~3 milliseconds of gaps) among the figure.By calling weld image processing module 15, extracted the image coordinate of weld bead feature points, export by simulating signal output or by wireless network again, like this, the weld joint tracking vision sensor has just been realized the function of acquisition process seam laser image and outgoing position signal, carries out welding technology with the welding gun of controlling on welding robot or the controller 4.

Claims (8)

1. weld joint tracking vision sensing equipment based on laser structure light can be gathered, the processing welding lines characteristic image, and output position while welding signal, carries out welding technology with the welding gun of controlling on welding robot or the controller; It is characterized in that, a) adopt a word laser that laser instrument is sent out to produce the laser structure optical plane, form structural light stripes; B) adopt the camera acquisition weld seam feature laser stripe image that installs filter lens additional; C) the laser structure optical plane becomes 30 °~60 ° angles with camera photography head central axis;
Described weld joint tracking vision sensing equipment based on laser structure light comprises laser instrument, video camera, welding gun and processing unit, wherein, laser instrument, video camera are packaged together, be rigidly connected on the framework with welding gun, the camera photography headband has optical filter, and sensor and welding gun are affixed, both central axial direction are identical, be co-located in same plane, this plane and weld seam formed plane intersect, and with the welded piece Surface Vertical; Video camera is electrically connected with processing unit, and processing unit is electrically connected with welding robot or controller by signal wire.
2. sensing device according to claim 1, it is characterized in that, described processing unit, comprise computing machine and image pick-up card, control panel, monitor and signal wire, computing machine is electrically connected with control panel, monitor and signal wire respectively, video camera is electrically connected with the image pick-up card of computing machine, and processing unit also is provided with analog output interface circuit and radio network interface.
3. sensing device according to claim 2 is characterized in that, laser structure optical plane and video camera be through demarcating, by the structural light stripes characteristic point coordinates as calculated computer go out the side-play amount of weld seam.
4. sensing device according to claim 2 is characterized in that, described analog output interface circuit and network interface are read the position while welding offset information with standard analog signal output or by wireless network.
5. sensing device according to claim 2 is characterized in that, utilizes network or passes through the selection of control panel button, can set, adjust at the weld seam type.
6. sensing device according to claim 2 is characterized in that described computing machine also is electrically connected with the weld image processing module, and the weld image processing module has the data of multiple weld seam type, is used to extract the weld bead feature points coordinate, and has expanded function.
7. sensing device according to claim 4 is characterized in that, described signal output has the output of horizontal direction and vertical direction position while welding deviation.
8. sensing device according to claim 5 is characterized in that, described signal output has the output of horizontal direction and vertical direction position while welding deviation.
CN 200410009931 2004-12-02 2004-12-02 Welding seam tracking sight sensor based on laser structure light CN1782659B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN 200410009931 CN1782659B (en) 2004-12-02 2004-12-02 Welding seam tracking sight sensor based on laser structure light

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN 200410009931 CN1782659B (en) 2004-12-02 2004-12-02 Welding seam tracking sight sensor based on laser structure light

Publications (2)

Publication Number Publication Date
CN1782659A CN1782659A (en) 2006-06-07
CN1782659B true CN1782659B (en) 2010-04-14

Family

ID=36773110

Family Applications (1)

Application Number Title Priority Date Filing Date
CN 200410009931 CN1782659B (en) 2004-12-02 2004-12-02 Welding seam tracking sight sensor based on laser structure light

Country Status (1)

Country Link
CN (1) CN1782659B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102155920A (en) * 2011-03-22 2011-08-17 华中科技大学 Welding seam measuring method based on micro field depth
CN104625337A (en) * 2015-01-21 2015-05-20 成都焊研威达科技股份有限公司 Adjustable laser visual sensing structure

Families Citing this family (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100460166C (en) * 2007-03-08 2009-02-11 上海交通大学 Welding robot multifunctional double-eye visual sensor and calibration method thereof
CN101961819B (en) * 2009-07-22 2013-10-30 中国科学院沈阳自动化研究所 Device for realizing laser welding and seam tracking and control method thereof
CN101791750B (en) * 2009-12-31 2012-06-06 哈尔滨工业大学 Robot remote control welding system and method used for remote welding
CN101797665B (en) * 2010-01-20 2012-05-16 浙江理工大学 Visual detection sensing unit
CN102455171B (en) * 2010-10-27 2014-01-15 中国科学院沈阳自动化研究所 Method for detecting geometric shape of back of tailor-welding weld and implementing device thereof
CN102284769A (en) * 2011-08-05 2011-12-21 上海交通大学 System and method for initial welding position identification of robot based on monocular vision sensing
JP5494597B2 (en) * 2011-09-16 2014-05-14 株式会社安川電機 Robot system
CN102506711B (en) * 2011-11-01 2013-07-17 江南大学 Line laser vision three-dimensional rotate scanning method
CN102441724A (en) * 2011-11-30 2012-05-09 南昌航空大学 Welding line recognition sensing device based on secondary surface light source
CN103192386B (en) * 2012-01-06 2014-10-22 沈阳新松机器人自动化股份有限公司 Image-vision-based automatic calibration method of clean robot
CN102699534B (en) * 2012-06-26 2014-10-15 哈尔滨工业大学 Scanning type laser vision sensing-based narrow-gap deep-groove automatic laser multilayer welding method for thick plate
CN103252560B (en) * 2013-03-22 2016-01-06 广西机电职业技术学院 Based on the automatic seam tracking method of laser-vision sensing
CN103273233A (en) * 2013-06-09 2013-09-04 东南大学 Object shelf tray self-adaptation welding device based on laser vision
CN104511905A (en) * 2013-09-28 2015-04-15 沈阳新松机器人自动化股份有限公司 Robot visual servo control method based on structured light
CN103600155B (en) * 2013-10-25 2015-04-22 上海工程技术大学 Real-time monitoring device for welding robot welding process
CN103753015B (en) * 2013-12-27 2015-09-23 深圳市光大激光科技股份有限公司 A kind of laser-beam welding machine seam tracking system and welding seam tracking method
CN103934571B (en) * 2014-04-11 2016-03-23 上海交通大学 Slab robot welding system and multi-pass welding seam real-time tracking, planing method
CN103954216B (en) * 2014-05-13 2017-04-12 清华大学 Strong specular reflection workpiece thin and narrow groove detection device and method based on spherical surface light sources
CN104084669B (en) * 2014-06-20 2016-02-03 河北工业大学 A kind of circumferential weld visible detection method detected based on decoupling zero
CN105328699A (en) * 2014-08-05 2016-02-17 罗伯特·博世有限公司 Intelligent robot welding system and method
CN104162752B (en) * 2014-08-12 2016-02-03 河北工业大学 Sheet metal lap weld little characteristic point four head serial vision-based detection tracking means
CN104400265B (en) * 2014-10-08 2017-06-06 吴长兴 A kind of extracting method of the welding robot corner connection characteristics of weld seam of laser vision guiding
CN104741778B (en) * 2015-03-30 2016-08-17 中国石油天然气集团公司 A kind of Combined Welding Weld Seam Tracking Control system
CN104748698B (en) * 2015-04-10 2016-10-26 清华大学 A kind of cap weld using lateral light to build shadow feature detection device and method
CN106141426A (en) * 2015-04-28 2016-11-23 大族激光科技产业集团股份有限公司 A kind of optical imaging system followed the trail of for laser welded seam
CN104964672B (en) * 2015-06-29 2017-05-31 济南大学 A kind of long-distance barrier detecting sensor based on line-structured light
CN105479052B (en) * 2015-12-20 2018-02-27 华南理工大学 A kind of weld seam tracking sensor structure
CN105488514A (en) * 2016-01-07 2016-04-13 张健敏 Welding device with autonomous welding function
CN105627925A (en) * 2016-03-02 2016-06-01 南昌航空大学 Laser vision sensor design method and device thereof
CN105798426B (en) * 2016-04-07 2018-02-02 湘潭大学 A kind of condenser type narrow gap weld seam tracking transducer
CN106017420B (en) * 2016-05-24 2019-03-29 武汉轻工大学 The gesture recognition method and identification device of welding backing piece
CN106003135B (en) * 2016-06-16 2018-07-06 芜湖润众机器人科技有限公司 A kind of manipulator machinery arm configuration
CN106041377A (en) * 2016-08-12 2016-10-26 广东省自动化研究所 Intelligent and compact type welding line visual tracking system
CN106312301A (en) * 2016-09-29 2017-01-11 哈尔滨理工大学 Weld joint tracking system based on image processing
CN106382888B (en) * 2016-11-10 2019-12-10 华南理工大学 Line laser welding seam detection device suitable for six industrial robot of perpendicular articulated type
CN106392267B (en) * 2016-11-28 2018-09-14 华南理工大学 A kind of real-time welding seam tracking method of six degree of freedom welding robot line laser
CN107020468A (en) * 2017-05-31 2017-08-08 石河子大学 A kind of welding control system of shape of a saddle space curve
CN108907455A (en) * 2018-08-04 2018-11-30 苏州佩恩机器人有限公司 A kind of seam tracking system and method merging vision technique based on laser-visible light
CN110681950A (en) * 2019-08-29 2020-01-14 北京航空航天大学 Robot weld joint tracking system based on structured light sensor and control method

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN2522214Y (en) * 2001-11-30 2002-11-27 清华大学 Linear array charge coupler vision sensor for automatic tracing welding seam
CN1511676A (en) * 2002-12-27 2004-07-14 中国科学院自动化研究所 Laser structure light vision sensor

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN2522214Y (en) * 2001-11-30 2002-11-27 清华大学 Linear array charge coupler vision sensor for automatic tracing welding seam
CN1511676A (en) * 2002-12-27 2004-07-14 中国科学院自动化研究所 Laser structure light vision sensor

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
岳宏等.基于结构光焊缝跟踪系统的研究.河北工业大学学报28 4.1999,32(5),全文.
岳宏等.基于结构光焊缝跟踪系统的研究.河北工业大学学报28 4.1999,32(5),全文. *
巴根等.激光视觉焊缝跟踪系统的研究.兵器材料科学与工程27 4.2004,32(5),全文. *
涂志国,徐德,谭民.弧焊机器人视觉测量控制系统.计算机测量与控制 3.2004,(3),第1-3节及其图1-3. *
鞠益兰等.结构光式激光视觉传感器的焊缝跟踪系统.电焊机32 5.2002,32(5),全文. *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102155920A (en) * 2011-03-22 2011-08-17 华中科技大学 Welding seam measuring method based on micro field depth
CN104625337A (en) * 2015-01-21 2015-05-20 成都焊研威达科技股份有限公司 Adjustable laser visual sensing structure

Also Published As

Publication number Publication date
CN1782659A (en) 2006-06-07

Similar Documents

Publication Publication Date Title
US10596650B2 (en) Helmet-integrated weld travel speed sensing system and method
CN107293191B (en) System for characterizing human weld's operation
CN104588838B (en) A kind of automatic tracing control of welding gun on-line amending system and control method thereof
JP6468741B2 (en) Robot system and robot system calibration method
CA2153334C (en) A system and method for tracking a feature on an object using a redundant axis
US4969108A (en) Vision seam tracking method and apparatus for a manipulator
EP0108599B1 (en) Automatic welding
EP1462224B1 (en) Taught position modification device
EP1533671B1 (en) Teaching position correcting device
US9833857B2 (en) Method for teaching/testing a motion sequence of a welding robot, welding robot and control system for same
US8742290B2 (en) Robot system
CN102303190B (en) Method for visually tracking plane abut-jointed weld beam by linear laser
JP5664629B2 (en) Robot system and method of manufacturing processed product
CN101036984B (en) Teaching position correcting apparatus and teaching position correction method
DE4327250C5 (en) Method for measuring coordinates on workpieces
US4427880A (en) Non-contact visual proximity sensing apparatus
CA1206210A (en) Light pattern projector especially for welding
CA2403596C (en) Method of welding three-dimensional structure and apparatus for use in such method
DE102005060967B4 (en) Method and device for setting up a trajectory of a robot device
EP1910009B1 (en) Welding method and welding system with determination of the position of the welding torch
EP1904262B1 (en) Defect detection system for identifying defects in weld seams
US5959425A (en) Vision guided automatic robotic path teaching method
DE3731704C2 (en) Method and arrangement for calibrating a sensor mounted on the hand of an industrial robot
CN101961819B (en) Device for realizing laser welding and seam tracking and control method thereof
CN102211245B (en) Spot welding system and dressing determination method

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
CF01 Termination of patent right due to non-payment of annual fee
CF01 Termination of patent right due to non-payment of annual fee

Granted publication date: 20100414

Termination date: 20171202