CN104400265B - A kind of extracting method of the welding robot corner connection characteristics of weld seam of laser vision guiding - Google Patents
A kind of extracting method of the welding robot corner connection characteristics of weld seam of laser vision guiding Download PDFInfo
- Publication number
- CN104400265B CN104400265B CN201410529039.2A CN201410529039A CN104400265B CN 104400265 B CN104400265 B CN 104400265B CN 201410529039 A CN201410529039 A CN 201410529039A CN 104400265 B CN104400265 B CN 104400265B
- Authority
- CN
- China
- Prior art keywords
- corner connection
- welding
- weld seam
- point
- spot welds
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B23—MACHINE TOOLS; METAL-WORKING NOT OTHERWISE PROVIDED FOR
- B23K—SOLDERING OR UNSOLDERING; WELDING; CLADDING OR PLATING BY SOLDERING OR WELDING; CUTTING BY APPLYING HEAT LOCALLY, e.g. FLAME CUTTING; WORKING BY LASER BEAM
- B23K37/00—Auxiliary devices or processes, not specially adapted to a procedure covered by only one of the preceding main groups
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
Abstract
The invention provides a kind of welding robot corner connection weld seam feature extracting method suitable for laser vision guiding, the method can obtain the corner connection angle and weld seam point coordinates of corner connection weld seam.In the determination part of corner connection angle, by gathering corner connection image with feature striation of the welding robot under two poses, two corner connection face equations of fitting determine corner connection angle;In the extraction part of spot welds, ROI region is intercepted first, then carry out medium filtering, binaryzation, refinement, and then striation straight line is extracted by Hough transform and spot welds are recognized.The determination of corner connection angle is beneficial at the beginning of welding starts, and effective coordination welding robot determines pose, effective recognition methods of spot welds, it is ensured that it is real-time accurate that spot welds are extracted;The method feature extraction is comprehensively, significant to actual welding process with good practicality.
Description
Technical field
Welding robot and its diametrical connection characteristics of weld seam extraction side the present invention relates to a kind of guiding of laser vision sensor
Method, specifically refers to be welded in the diagonal welding seam welding process of welding robot based on the guiding of line laser structured light vision sensor
The determination of angle and the recognition methods of spot welds.
Technical background
In recent years, developing rapidly with national economy, country constantly expands to the demand of metal material, quality requirement not
Disconnected to improve, the important process link that welding is used as metal material, the quality of welding quality is directly connected to metal material
Use.Existing welding method labour intensity is big, poor working environment, complex process, big to injury of human, and welding quality master
The human factors such as individual skill, the experience of welder are leaned on to ensure, it is achieved that Automation of Welding, intellectuality can effectively liberate labor
Power, improvement welding quality, it is significant to China's modernization, industrialized development.
The key issue for realizing Automation of Welding be weld seam from motion tracking, the welding robot of laser vision guiding (shows
It is intended to as shown in Figure 2) welding seam is combined with robot technology, can effectively solve the problem that soldering joint automatic tracking problem.Adopt
Welding robot is guided to have become current Intelligent welding robot system as active light vision sensor with laser structure light
Main flow.
In welding procedure, corner connection weld seam is a kind of important welds types.Corner connection characteristics of weld seam mainly includes fillet welding
The determination of the corner connection angle of seam and spot welds are accurately recognized.Corner connection weld seam feature recognition is to realize the crucial ring of fillet welding seam welding
Section, the precision that its identification is extracted directly influences the precision of tracking and welding in industrial processes.
In corner connection characteristics of weld seam, domestic and foreign scholars bias toward the recognition methods of research spot welds, it is believed that recognize that spot welds are
It is capable of achieving to weld, but only spot welds three-dimensional information, it is impossible to automatically determine robot initial welding pose, it is difficult to real reality
Existing Automation of Welding.Herein in conjunction with practical application request, achievement in research of the forefathers in terms of spot welds identification is used for reference, be given comprehensively
Corner connection weld seam feature extracting method, including the corner connection angle-determining and spot welds of corner connection weld seam accurately recognize, makes welding robot
People efficiently completes weld task.
The content of the invention
To determine the corner connection angle of corner connection weld seam, and realize that the Real time identification of corner connection weld seam is extracted, the present invention is provided and is based on
The corner connection weld seam characteristic detection method of the welding robot platform of laser structure light vision guide.
The key technology scheme that patent of the present invention is used is:On the welding robot system for completing is demarcated, by two
Corner connection image under individual pose determines corner connection angle, and recognizes corner connection weld bead feature points using image processing algorithm.
According to described in the technical scheme that the present invention is provided, laser structure light vision sensor guides welding robot fillet welding
Seam characteristic recognition method is as follows:
The first step, determines the corner connection angle of corner connection weld seam;The given pose T of welding robot two1、T2, in T1Gathered at pose
Corner connection weld image, extracts striation straight line and spot welds P1(identification weld seam point methods such as second step), and it is being belonging respectively to corner connection
Two the two of plane sections of striation straight lines on any selected point P10、P11, in T2Acquisition angle welding seam image at pose, extracts striation
Straight line and spot welds P2, and any selected point P on two the two of plane sections of striation straight lines for be belonging respectively to corner connection20、P21, due to
P1、P10、P2、P20And P1、P11、P2、P21Two corner connection faces of corner connection weld seam are belonging respectively to, two corner connection faces can be fitted by four points
Equation, and then obtain two angles in corner connection face.
Second step, accurately recognizes spot welds;The welding robot system acquisition angle welding seam image guided by laser vision,
Image to gathering carries out grid windowing process and intercepts out ROI region, and the ROI region to intercepting out carries out medium filtering, two-value
Change, refine, striation straight line is extracted using Hough transform on the image after refinement, and then obtain weld seam point coordinates.
Beneficial effect of the present invention:There is provided a kind of feature of the welding robot corner connection weld seam suitable for laser vision guiding
Extracting method, the method can obtain the corner connection angle and weld seam point coordinates of corner connection weld seam.The determination of corner connection angle be beneficial to
At the beginning of welding starts, effective coordination welding robot determines pose, is more efficiently completed weld task, in the extraction unit of spot welds
Point, take on the basis of ROI region is intercepted, calculated using the secure binaryzation of robustness, real-time, refinement is verified
Method, it is ensured that it is real-time accurate that spot welds are extracted.The corner connection weld seam feature extraction comprehensively, has weight to actual welding also termination process
Meaning is wanted, with good practicality.
Brief description of the drawings
Fig. 1 overview flow charts of the present invention.
Fig. 2 laser visions guide welding robot schematic diagram.
Fig. 3 corner connection angle-determining schematic diagrames.
Specific embodiment
To make the object, technical solutions and advantages of the present invention become more apparent, here is specific implementation process, and reference
Accompanying drawing, is described in further detail to the present invention.
Basic ideas of the invention are:The actual application background of the welding robot guided according to laser vision, with reference to angle
The actual welding feature of welding seam, extracts corner connection characteristics of weld seam.Corner connection characteristics of weld seam mainly includes two information content, corner connection angle
And spot welds.The corner connection image with feature striation under corner connection angle-determining part, two poses of collection, solves two planes
Equation so that determine two angles of plane, schematic diagram is as shown in Figure 3;In the extraction part of spot welds, in interception ROI region
On the basis of, it is accurate in real time to recognize spot welds using the secure binaryzation of robustness, real-time, thinning algorithm is verified.
The corner connection weld seam feature extraction is comprehensively, significant to actual welding also termination process, with good practicality.
Fig. 1 is integral calibrating flow chart of the present invention.Figure (a) is determination corner connection angle;Because two planes determine its angle
Determine immediately, by solving two equations of plane respectively and then determining two angles of plane.Two plane equations are by flat
Four points fitting on face;Figure (b) is the identification division of spot welds;Medium filtering is carried out on the basis of ROI region is intercepted, and
After carry out binaryzation, refinement, and then carry out Hough transform identification spot welds.
The first step specific method is as follows:
1.1st, vision guide welding robot parameter and transformation relation
As shown in Fig. 2 the welding robot Mathematical Modeling schematic diagram of laser structure light vision guide, its major parameter includes
Camera intrinsic parameter K, trick relational matrix T, optic plane equationsesIts Mathematical Modeling is:
(1) when system works, vision sensor obtains characteristic point n=(u, v, 1) in structure lightT, in camera coordinate system
Three-dimensional coordinate XC:
Wherein XC=(xc, yc, zc)TIt is spot welds in camera coordinate system three-dimensional coordinate;It is shooting
Machine internal reference matrix, wherein kx、kyReflection image u axles, the proportionality coefficient of v axles, s reflect two angle situations of axle, u0、v0It is light
The intersection point of axle straight line and CCD planes;N=(u, v, 1)TIt is the corresponding image homogeneous coordinates of spot welds;It is structure
Optic plane equationses parameter of the optical plane under camera coordinate system.
(2) characteristic point in the coordinate transform of camera coordinate system to robot basis coordinates system such as following formula:
XW=RHWRXC+RHWt+tHW
Wherein XWIt is coordinate of the spot welds in robot basis coordinates system, RHW、tHWIt is robot basis coordinates system and ending coordinates
The rotation of system and translation relation, R, t are the relations of robot end's coordinate system and camera coordinate system, i.e. trick matrix
1.2nd, the point P on image1、P2、P10、P11、P20And P21Each point transforms to welding robot basis coordinates system
If laser is in pose T1When project laser stripe in two planes of object as shown in figure 3, laser stripe
In a broken line, if P1It is the broken line flex point on two intersections of plane, the broken line flex point is spot welds in welding,
In P1Both sides laser stripe on take respectively a bit, be set to P10And P11, then P10、P11Also it is respectively the point in two planes.Through swashing
Light vision sensor detection obtains P1Point three-dimensional coordinate under camera coordinate system is XC1(xC1, yC1, zC1), it can be tried to achieve in machine
Coordinate X under device people's basis coordinates systemW1(xW1, yW1, zW1)
[xW1, yW1, zW1, 1]T=T1T[xC1, yC1, zC1, 1]T
WhereinIt is robot posture information, T is trick matrix.
Change robot pose and reach T2, can similarly solve P2、P20、P21。
1.3rd, plane included angle solves
By non-colinear coplanar 4 point P1、P2、P10、P20, it is fitted the equation for obtaining plane 1:
a1x+b1y+c1Z+1=0
By non-colinear coplanar 4 point P1、P2、P11、P21Fitting obtains the equation of plane 2:
a2x+b2y+c2Z+1=0
That is the normal vector of two planes is respectivelyWith
Corresponding corner connection angle, θ can be solved according to cos θ values.
The second step specific method is as follows:
2.1st, ROI is intercepted
To the weld seam piece image for gathering, interception ROI region method is as follows, to entire image in horizontally and vertically direction
Sample at certain intervals, carrying out gray value to each pixel of sampling adds up, and the average of sampling result is set to M as background gray scale
Value.
Wherein, W and H are respectively that image is wide and height, I (x, y) is image intensity value.
Laser stripe is general brighter than background, takes M+M0Used as threshold value, interception ROI region is as follows:
2.2nd, Da-Jin algorithm binaryzation
Threshold segmentation Binarization methods are to highlight background and characteristic area optimum method.Adaptive threshold can be with welding
The influence of various enchancement factors in environment, carries out self-adaptative adjustment, is preferably partitioned into striation region.
The basic thought of Da-Jin algorithm is the image inter-class variance and maximum for selecting threshold value to cause segmentation, and threshold value determination process is such as
Under:
(1) gradation of image statistical information is obtained:
Wherein, niFor gray value is the sum of all pixels of i;N is area pixel sum;P (i) is gray value probability;w0It is target
Probability.
(2) initial threshold K is chosen, average, the variance of target and background is calculated:
Whereinw0、w1It is target, the probability of background, μ0, μ1It is target, background
Gray average, σ0, σ1Target, the variance of background.
(3) inter-class variance is calculated, the corresponding K of maximum is selected threshold.
2.3rd, refined based on Medial-Axis Transformation principle
Medial-Axis Transformation basic thought is iteration, the template matches that the target area to obtaining is iterated, and is removed non-central
Point, until the point being only left on center line, specific method is:Note central point is p1, its 8 field point is designated as p in the direction of the clock2,
p3..., p9, wherein p2In p1Top.
Mark meets the point of following 4 conditions simultaneously first:
I:2≤N(p1)≤6;
II:S(p1)=1;
III:p2*p4*p6=0;
IV:p4*p6*p8=0;
Wherein, N (p1)=p2+p3+…+p8+p9;S(p1) it is with p2, p3..., p9, p2When being rotated for sequence, it is worth from 0 → 1
Change total degree.In to region travel through a little on one side after, delete mark point.
Then mark:I and II conditions are constant, and III is changed into p2*p4*p8=0;IV is changed into p2*p6*p8=0 point, deletes mark
Note point;It is to be capable of achieving striation refinement to perform above procedure repeatedly.
2.4th, Hough transform
ρ=x cos θ+y sin θs (0≤θ≤π)
Wherein ρ is the origin of coordinates to air line distance, and θ is the origin of coordinates to the vertical line of straight line and the angle of X-axis positive direction.
2.5th, identification feature point
According to the straight line that Hough transform is obtained, the intersection point of adjacent straight line is taken as three feature angle points.According to three features
The V-type recess region area that angle point is constituted determines welding wire loading in welding, takes groove triangle core as spot welds.Coordinate
For:
Claims (3)
1. a kind of extracting method of the welding robot corner connection characteristics of weld seam of laser vision guiding, the method can obtain corner connection weld seam
Corner connection angle and weld seam point coordinates;The determination of corner connection angle is beneficial at the beginning of welding starts, effective coordination welding robot
Determine pose, be more efficiently completed weld task, in the extraction part of spot welds, take on the basis of ROI region is intercepted, adopt
With being verified the secure binaryzation of robustness, real-time, thinning algorithm, it is ensured that it is real-time accurate that spot welds are extracted;Should
Method includes following module:
Determine the corner connection angle of corner connection weld seam;The given pose T of welding robot two1、T2, in T1Acquisition angle welding seam figure at pose
Picture, extracts striation straight line and spot welds P1, and arbitrarily chosen on two the two of plane sections of striation straight lines for being belonging respectively to corner connection
Point P10、P11, in T2Acquisition angle welding seam image at pose, extracts striation straight line and spot welds P2, and it is being belonging respectively to corner connection
Any selected point P on two the two of plane sections of striation straight lines20、P21, by P on image1、P2、P10、P11、P20And P21Each point is transformed to
Welding robot basis coordinates system, due to P1、P10、P2、P20And P1、P11、P2、P21Two corner connection faces of corner connection weld seam are belonging respectively to,
Two equations in corner connection face can be fitted by four points, and then determine two angles in corner connection face;
Spot welds are extracted in accurate identification;The welding robot system acquisition angle welding seam image guided by laser vision, to collection
Image carry out grid windowing process and intercept out ROI region, the ROI region to intercepting out carries out medium filtering, binaryzation, thin
Change, striation straight line is extracted using Hough transform on the image after refinement, and then obtain weld seam point coordinates.
2. the extracting method of the welding robot corner connection characteristics of weld seam of a kind of laser vision guiding as claimed in claim 1, really
In determining the corner connection angle of weld seam, including by P on image1、P2、P10、P11、P20And P21Each point transforms to welding robot basis coordinates
Two angles in corner connection face of system and determination, with following feature:
(1) by P on image1、P2、P10、P11、P20And P21Each point is transformed in welding robot basis coordinates system, and laser is in pose
T1When project laser stripe in two planes of object in a broken line, if P1It is the broken line on two intersections of plane
Flex point, the broken line flex point is spot welds in welding, in P1Taken respectively a bit on laser stripe on two side planes, be set to
P10And P11;Through the P that laser vision sensor detection is obtained1Point three-dimensional coordinate under camera coordinate system is XC1(xC1, yC1,
zC1), its coordinate X under robot basis coordinates system can be tried to achieveW1(xW1, yW1, zW1):
[xW1, yW1, zW1, 1]T=T1T[xC1, yC1, zC1, 1]T
Wherein T1It is robot pose,It is trick matrix, can similarly solves P10、P11, change robot pose and reach
T2, can similarly solve P2、P20、P21;
(2) in determining two angles in corner connection face, by non-colinear coplanar 4 point P1、P2、P10、P20, it is fitted the side for obtaining plane 1
Journey:
a1x+b1y+c1Z+1=0
By non-colinear coplanar 4 point P1、P2、P11、P21Fitting obtains the equation of plane 2:
a2x+b2y+c2Z+1=0
That is the normal vector of two planes is respectivelyWith
Corresponding corner connection angle, θ can be solved according to cos θ values.
3. a kind of extracting method of the welding robot corner connection characteristics of weld seam of laser vision guiding as claimed in claim 1, smart
Really spot welds are extracted in identification has following feature:Welding image to gathering carries out grid windowing process and intercepts out ROI region, right
The ROI region for intercepting out carries out medium filtering, and binaryzation is carried out using Da-Jin algorithm to filtered image, is then become based on axis
Changing principle is carried out to light thinning, and striation straight line is extracted using Hough transform on the image after refinement, takes the friendship of adjacent straight line
O'clock as three feature angle points, then using three centers of gravity of characteristic point as spot welds.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410529039.2A CN104400265B (en) | 2014-10-08 | 2014-10-08 | A kind of extracting method of the welding robot corner connection characteristics of weld seam of laser vision guiding |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410529039.2A CN104400265B (en) | 2014-10-08 | 2014-10-08 | A kind of extracting method of the welding robot corner connection characteristics of weld seam of laser vision guiding |
Publications (2)
Publication Number | Publication Date |
---|---|
CN104400265A CN104400265A (en) | 2015-03-11 |
CN104400265B true CN104400265B (en) | 2017-06-06 |
Family
ID=52637964
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201410529039.2A Expired - Fee Related CN104400265B (en) | 2014-10-08 | 2014-10-08 | A kind of extracting method of the welding robot corner connection characteristics of weld seam of laser vision guiding |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN104400265B (en) |
Families Citing this family (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106530269A (en) * | 2015-09-15 | 2017-03-22 | 苏州中启维盛机器人科技有限公司 | Weld detection method |
CN105138022A (en) * | 2015-09-16 | 2015-12-09 | 山东中清智能科技有限公司 | Visual tracking decoupling control method and apparatus of narrow welding seam |
CN105397283A (en) * | 2015-12-11 | 2016-03-16 | 中国电子科技集团公司第四十八研究所 | Weld joint track correction method for microwave assembly |
CN105678776B (en) * | 2016-01-11 | 2017-03-22 | 南京工业大学 | Weld image feature point extraction method based on laser vision sensor |
CN106354390B (en) * | 2016-08-30 | 2019-09-17 | 广东小天才科技有限公司 | The method and apparatus of the interception of target image |
US11440119B2 (en) * | 2018-10-12 | 2022-09-13 | Teradyne, Inc. | System and method for weld path generation |
CN109483545B (en) * | 2018-12-04 | 2022-03-29 | 中冶赛迪工程技术股份有限公司 | Weld joint reconstruction method, intelligent robot welding method and system |
CN110497121A (en) * | 2019-08-05 | 2019-11-26 | 苏州热工研究院有限公司 | The image processing mode of welding bead center tracking in a kind of welding process |
CN111390439B (en) * | 2020-03-31 | 2021-11-05 | 北京博清科技有限公司 | Welding seam detection method and device, welding robot and storage medium |
CN113269207B (en) * | 2021-05-28 | 2023-04-18 | 天津理工大学 | Image feature point extraction method for grid structure light vision measurement |
CN113763379B (en) * | 2021-09-23 | 2023-07-18 | 成都唐源电气股份有限公司 | Hanger strand breakage detection method and device, computer equipment and storage medium |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR100500995B1 (en) * | 2003-01-18 | 2005-07-18 | 삼성중공업 주식회사 | Welding system |
CN1782659B (en) * | 2004-12-02 | 2010-04-14 | 中国科学院自动化研究所 | Welding seam tracking sight sensor based on laser structure light |
KR20110022959A (en) * | 2009-08-28 | 2011-03-08 | 현대중공업 주식회사 | System for weld seam tracking |
CN101770710A (en) * | 2009-12-31 | 2010-07-07 | 哈尔滨工业大学 | Laser-vision sensing assisted remote teaching method for remote welding |
CN102794763B (en) * | 2012-08-31 | 2014-09-24 | 江南大学 | Systematic calibration method of welding robot guided by line structured light vision sensor |
CN103500321B (en) * | 2013-07-03 | 2016-12-07 | 无锡信捷电气股份有限公司 | Vision guide welding robot weld seam method for quickly identifying based on double dynamic windows |
CN103759648B (en) * | 2014-01-28 | 2016-05-25 | 华南理工大学 | A kind of complicated angle welding method for detecting position based on Binocular stereo vision with laser |
-
2014
- 2014-10-08 CN CN201410529039.2A patent/CN104400265B/en not_active Expired - Fee Related
Also Published As
Publication number | Publication date |
---|---|
CN104400265A (en) | 2015-03-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN104400265B (en) | A kind of extracting method of the welding robot corner connection characteristics of weld seam of laser vision guiding | |
CN114119464B (en) | Deep learning-based lithium battery cell top cover weld appearance detection method | |
CN107876970A (en) | A kind of robot multi-pass welding welding seam three-dimensional values and weld seam inflection point identification method | |
CN102279190B (en) | Image detection method for weld seam surface defects of laser welded plates of unequal thickness | |
CN102794763B (en) | Systematic calibration method of welding robot guided by line structured light vision sensor | |
CA2950791C (en) | Binocular visual navigation system and method based on power robot | |
CN103411553B (en) | The quick calibrating method of multi-linear structured light vision sensors | |
CN106650701B (en) | Binocular vision-based obstacle detection method and device in indoor shadow environment | |
CN108171796A (en) | A kind of inspection machine human visual system and control method based on three-dimensional point cloud | |
CN104978012B (en) | One kind points to exchange method, apparatus and system | |
CN103500321B (en) | Vision guide welding robot weld seam method for quickly identifying based on double dynamic windows | |
CN109308718B (en) | Space personnel positioning device and method based on multiple depth cameras | |
CN104463108A (en) | Monocular real-time target recognition and pose measurement method | |
CN105913013A (en) | Binocular vision face recognition algorithm | |
CN107595388A (en) | A kind of near infrared binocular visual stereoscopic matching process based on witch ball mark point | |
CN114140439A (en) | Laser welding seam feature point identification method and device based on deep learning | |
CN107220999A (en) | The research of workpiece circular arc Edge Feature Points matching process | |
CN114782626A (en) | Transformer substation scene mapping and positioning optimization method based on laser and vision fusion | |
CN104966302B (en) | A kind of detection localization method of any angle laser cross | |
CN115619738A (en) | Detection method for module side seam welding after welding | |
CN109359537A (en) | Human face posture angle detecting method neural network based and system | |
CN110136186A (en) | A kind of detection target matching method for mobile robot object ranging | |
WO2022036478A1 (en) | Machine vision-based augmented reality blind area assembly guidance method | |
CN108161930A (en) | A kind of robot positioning system of view-based access control model and method | |
CN104992448A (en) | Automatic positioning method for robot picking grapes in a damage-free way |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170606 Termination date: 20181008 |
|
CF01 | Termination of patent right due to non-payment of annual fee |