CN100580704C - Real time self-adapting processing method of image mobile imaging - Google Patents
Real time self-adapting processing method of image mobile imaging Download PDFInfo
- Publication number
- CN100580704C CN100580704C CN200810035329A CN200810035329A CN100580704C CN 100580704 C CN100580704 C CN 100580704C CN 200810035329 A CN200810035329 A CN 200810035329A CN 200810035329 A CN200810035329 A CN 200810035329A CN 100580704 C CN100580704 C CN 100580704C
- Authority
- CN
- China
- Prior art keywords
- target
- imaging
- image
- pixel
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a real time self-adapting processing method of anastigmatic imaging, which is a real time self-adapting processing method of anastigmatic imaging in a high-speed movement small-object imaging system. According to the method, small objects in high-speed movement can be automatically searched and found, FPGA and DSP are used as the real time on-line processing platform, an image reinforcement and inter-frame correlation identification algorism with low complexity is adopted, effect of fake objects are removed to obtain movement characteristics of the objects, so as to drive a high-precision two-dimensional piezoelectricity tilting mirror to complete self-adapting movement compensation, thereby realizing clear imaging of the high-speed movement small-objects.
Description
Technical field
The present invention relates to the Image Acquisition technology, refer to that specifically the little target travel of a kind of high-speed motion compensates the self-adaptation real-time processing method of the image drift imaging that disappears, it is applied to obtaining distant place rapid movement little target high reliability information.
Background technology
The little target of high-speed motion is the important goal that infrared image obtains, and also is applied in visible images in recent years.To such target imaging have target size less (not enough approximately 1m), image-forming range (several kilometers) far away, the imaging background is single relatively and it is slow to change, fast (reaching as high as 1km/s), the shared pixel of target imaging are no less than features such as 4 * 4 to target speed especially.The general spatial resolution of traditional formation method is lower, and is fuzzy insensitive to image drift, only need satisfy the requirement of targeted surveillance basically.And in order to obtain the high reliability information of this moving target, realization is to its blur-free imaging, need guarantee that then internal object motion image drift can not exceed 1 instantaneous field of view in integral time, and the hypothetical target imaging occupies 4 * 4 pixels, target size 0.5m, imaging speed of related movement are 1km/s, and be 5ms integral time, then integral time internal object will move 40 pixels, adopt traditional imaging side rule to have serious image drift fuzzy problem like this.Simultaneously because target speed is very fast, according to above-mentioned target speed, suppose that detector pixel number is 1k * 1k, then target lateral time of passing the visual field only is 125ms, carry out the IMC imaging for the assurance system can capture target, then the real-time processing requirement of system seems particularly important.
Traditional image drift imaging mode that disappears has a lot, control as the exposure that common DC motor pattern adopts, ray machine mode drives camera that aviation measuring camera adopts or detector motion compensation, the electronics motion compensation of TDI mode, the described electronics IMC of patent aviation total-frame transfer type matrix CCD camera image shift compensation method (number of patent application 200710117666.5) that the Beijing Lingyun light-vision the digital image technology Co., Ltd applied in 2007 etc., but these IMC modes are not very high in the face of resolution mainly or the complete known situation of target travel characteristic at present, as aerial survey or synchronous ballistic photography etc., they generally only need the motion compensation direction is adjusted into consistent as direction of motion with target, and configure the image drift motion compensation parameters in advance and open in good time and get final product, view data is directly passback then, do not exist online realtime graphic to handle, for little target and the target trajectory and the self-adaptive processing and the response well still of speed condition of unknown of distant place high-speed motion.The imaging system that has real-time online self-adaptive processing mechanism at present is mainly used in the task of targeted surveillance, its towards object mostly be point target or the slower fixed target of movement velocity greatly, their movement velocitys on imaging plane are not high, do not exist target to flee from out the fuzzy fully situation of image drift in visual field or the imaging easily substantially.In sum, the blur-free imaging for the little target of high-speed motion of speed the unknown does not also have the solution of online in real time self-adaptive processing preferably at present.
Summary of the invention
The object of the present invention is to provide the disappear real-time adaptive disposal route of image drift imaging of the little target of a kind of high-speed motion, adjust IMC mechanism by self-adaptation and eliminate the fuzzy influence of image drift that the relative motion of target high speed brings.
The inventive method is to realize in the little target imaging of high-speed motion system as shown in Figure 3.Whole imaging system is installed in a two-dimensional tracking with degree of precision and points on the turntable 4, this device at first utilizes two-dimensional tracking to point to turntable 4 operation branch line scanning target search pattern, optical signalling enters big face battle array focus planardetector 3 by imaging optical system 1 and realizes Image Acquisition, the image that 5 pairs of detectors 3 of real time processing system obtain is handled and is found target and obtain the image drift imaging moving compensation rate that disappears, and then the control high-precision two-dimensional image drift piezoelectric tilt mirror 2 that disappears carries out IMC, the sensing of controlling two-dimensional tracking sensing turntable 4 simultaneously remains on target in the imaging viewing field, thereby realizes the blur-free imaging to the little target of high-speed motion.
The process flow diagram of the inventive method as shown in Figure 1, its key step of handling in real time comprises:
A. adopt relevant two scale filter that the image that has motion blur that obtains is traveled through and realize the image filtering pre-service, remove gradual background and obtain target enhance image;
B. the algorithm that adopts adjacent picture elements to add up carry out record to possibility target pixel, extracts the possible target pixel that brightness surpasses the certain threshold value of background luminance, suppresses the influence of random noise simultaneously;
C. the position of possibility target pixel is sorted out, noted the number of candidate target and the information such as profile, size, center and brightness of each target;
D. obtain the image that there is the possibility target in 3 frames continuously by A, B, C step, adopt the relevant way of interframe that the pseudo-target of the fixing part of extracting from background of relative position is removed, and pick out the target that to follow the tracks of according to the multiframe motion consistency principle of target;
E. the target that chooses at step D, the characteristics such as installation site according to the frame frequency of imaging, detector pixel dimension, IMC mirror calculate the IMC amount of following the tracks of imaging, send to controller and carry out the IMC imaging.
Above-mentioned steps adopts FPGA+DSP hardware to realize.Real time processing system hardware is realized block diagram as shown in Figure 2, after the system start-up, FPGA produces detector driving sequential and carries out Image Acquisition, and the data that read detector output are carried out the image filtering pre-service and may be extracted by the target pixel, the possible target that the extracts EMIF mouth by DSP is sent to DSP be for further processing promptly above-mentioned steps A and B; DSP sorts out the possible target pixel that receives, the relevant pseudo-target of operation interframe removes algorithm and finds out target and carry out the motion consistency detection after having obtained the image that 3 frames have target continuously, estimate the position and the required motion compensation quantity of imaging of target, and the GPIO mouth by DSP sends to two dimension with these information and points to turntable and high-precision two-dimensional piezoelectric tilt mirror and control it and carry out the IMC imaging, promptly above-mentioned step C, D, E.
Target at steps A: FPGA strengthens the general relevant two scale filter algorithms of Filtering Processing employing, be that the high-pass filtering operator of S * S carries out small scale filtering and finishes protection to the little target with high frequency characteristics to the original images by using yardstick promptly, the span of S is the odd number between 1～5; While original images by using yardstick is that the smothing filtering operator of L * L carries out the estimation that large scale filtering realizes image background, and wherein the range of choice of L is got odd number between 5～9, and L＞S; The image that two scale filters are obtained carries out reducing, has promptly obtained the target enhance image behind the background removal, has also suppressed random noise simultaneously.
At step B:FPGA possible the target pixel that exists is detected and to be actually when image is traveled through filtering realization, it adds up to 4 * 4 neighborhoods of image picture dot, if the value that certain pixel neighborhood brightness adds up has surpassed given threshold value, then think to have target, note its position and brightness value.Choose between 1.2～2 times of the background estimating value that wherein given threshold value obtains for large scale filtering, the selection of concrete numerical value comes calmly with background characteristics according to the target of practical application.Through the aforesaid operations of FPGA, if do not detect possible target pixel, then abandon image and proceed Image Acquisition, otherwise being sent to DSP, the bright spot of noting is for further processing.
At step C: through after the detection of FPGA, DSP needs be for further processing to the possible target pixel few in number that detects.DSP at first may carry out the cycle detection classification by the target pixel to these, calculates possible target numbers total in the image, and even Ji Lu some target pixel position is adjacent, then it is classified as same possibility target.Wait to sort out the number of record possibility target after finishing, and count center, brightness average, the picture dot number that occupies and the X of each possibility target class and the information such as picture dot number that the Y direction is extended respectively.
At step D: (frame number N-1～N+1) contains the image of target bright spot, and then DSP adopts the relevant algorithm of interframe to carry out pseudo-target to remove and the target trajectory consistency detection if obtained 3 frames continuously in system.Be initial point at first, calculate the relative position of other target's center's point and this point with all minimum target of center position X and Y direction in the possibility target after the classification of N-1 and N frame; Compare this two frame then, the relative position error of (be no less than may target numbers half) its X and Y direction is all in the individual pixel of Nf (getting 0～4) if the major part for them may target, then think the position that these may targets be the pseudo-target in the background and note them, otherwise the picture dot that the X position of the initial point of then selecting and setting originally differs minimum is an initial point, proceed aforesaid operations, till can removing pseudo-target over half, if selected all pixels is that initial point still can not remove pseudo-target, then the N-1 frame is abandoned, continue to read the N+2 frame.With above-mentioned method N frame and N+1 frame are carried out pseudo-target query, and the pseudo-target of twice record in three frames is all removed, can remove most pseudo-target through this operation.If still more than 1 of remaining possibility target, then choose the residue of three frames may target in difference in brightness be no more than 20% of average, shared pixel is counted difference, and to be no more than Np (getting 0～8) individual, target is the candidate target group in the target that the difference of the picture dot development length of X and Y direction all is no more than Nxy1 (getting 0～4), judge its motion consistance in three frames, if the movement position difference of certain group candidate target all is no more than the individual pixel of Nxy2 (getting 0～4) at X and Y direction, conclude that then it is the target of systematic search, otherwise, then abandon the N-1 frame and proceed the analysis of N+2 frame till finding target if there is not similar target.Parameter N f, Np, Nxy1, Nxy2 are according to judging strict degree value, and to the pine order, each parameter is carried out value from small to large by as strict as possible.
In the target of coming out, calculate the average movement velocity of target image drift at step e: DSP in X and Y direction at search, promptly N+1 frame and N-1 frame target location poor/(2* frame period), unit is pixel/s.Suppose that image drift speed is respectively Sx and Sy at directions X and Y direction, detector pixel spacing distance is A * Aum
2, the minute surface center of two-dimensional piezoelectric dip sweeping mirror and the distance of detector photosensitive unit are H um, then the motion compensation speed of two-dimensional piezoelectric dip sweeping mirror is Sx * A/2H rad/s at directions X, is Sy * A/2H rad/s in the Y direction.
The advantage of this method is:
1. system adopts the low algorithm of computational complexity can satisfy the requirement of real-time and accuracy, is suitable for handling streaking imaging viewing field at a high speed.The method that wherein relevant two scale filter algorithms and neighborhood add up can remove preferably and change little background and judge possible target picture dot, and adopt the relevant algorithm of multiframe can remove the constant pseudo-target of relative position in the background more exactly, and can detect the little target of high-speed motion exactly according to consistency check.
2. the mode that real-time processing hardware system adopts FPGA to combine with DSP, both can give full play to FPGA hardware handles fireballing characteristic aspect image traversal filtering, can bring into play the high advantage of DSP ordering calculation frequency again, the data processing work such as traversal filtering, target search and track estimation that they are worked in coordination and finish image together, thus satisfy the requirement that system handles in real time well.
Description of drawings
Fig. 1 is the real time processing system workflow diagram.
Fig. 2 is the real time processing system hardware block diagram.
Fig. 3 is the little target imaging of a high-speed motion system;
Among the figure, the 1-imaging optical system;
2-two dimension IMC piezoelectric tilt mirror;
The 3-focus planardetector;
The 4-two-dimensional tracking points to turntable;
5-handles and control gear in real time.
Specific implementation method
According to the real-time adaptive disposal route described in the instructions, the structural representation of its implementation platform as shown in Figure 3, platform is pointed to turntable 4 and is handled with control gear 5 in real time by imaging optical system 1, two-dimentional IMC piezoelectric tilt mirror 2, focus planardetector 3, two-dimensional tracking and constitutes, wherein:
Imaging optical system 1 adopts common optical telescope, and bore is 150mm, and focal length 1800mm, F number are 12, resolution＜1 ", instantaneous field of view of system is 8urad, 0.47 ° of total visual field;
Two dimension IMC piezoelectric tilt mirror 2 adopts four fulcrum XY axle sloping platforms, and its closed loop angle of inclination can reach ± 2mrad, and resolution reaches 0.05urad, the minute surface diameter is 50mm, the closed loop linearity 0.2%, resonant frequency 3.3KHz, the distance H of its minute surface and focal plane is 50mm;
Big face battle array focus planardetector 3 adopts the monochromatic face battle array cmos device with global shutter, and its face battle array size is 1k * 1k, and response wave length is 400～1000nm, and pixel dimension is 14um * 14um, frame frequency 20fps, and integral time, may command reached the us magnitude;
Two-dimensional tracking points to turntable 4 and adopts the ordinary two dimensional turntable with pitching rotation and orientation rotation, and the orientation anglec of rotation is 0 ~ 360 degree, 0.5 °/s of rotating speed ~ 4 °/s, precision＜0.05 °, the pitching anglec of rotation is 0 ° ~ 90 °, 0.5 °/s of rotating speed ~ 4 °/s, precision＜0.1 °, load reaches 200Kg.
The hardware configuration of real time processing system as shown in Figure 2, it is made of the realtime graphic processing of FPGA+DSP and identification module and high-precision two-dimensional controller, wherein the fpga logic unit number is 12060LEs, built-in 2 PLL and 234Kbits RAM, and external clock is 40MHz; The fixed DSP external clock of selecting is 50MHz, and the computing frequency is 600MHz, has GPIO interface and 32 s' EMIF interface, and procedure stores is in Flash.
In the Processing Algorithm, select the filter scale L=7 of large scale smothing filtering in real time, weighted value is 1; The yardstick of small scale filtering is S=3, and the central point weighted value is 4, and the neighborhood weighted value is 1; Possible target picture dot judgment threshold is 1.5 times that background luminance is estimated, when pseudo-target removes, judge whether that the site error Nf with pseudo-target property is taken as 2 pixels, luminance difference is taken as 20% like the judgement target class, the shared picture dot difference of target shape is no more than Np=6 pixel, the target picture dot all is no more than Nxy1=2 pixel in the difference of the extension of X and Y direction etc., and when judging the target travel consistance, the difference of its amount of exercise all is no more than Nxy2=2 pixel at X and Y direction.
Claims (4)
1. the real-time adaptive disposal route of the image drift imaging that disappears, it is characterized in that: this method may further comprise the steps:
A. adopt relevant two scale filter that the image that has motion blur that obtains is traveled through and realize the image filtering pre-service, remove gradual background and obtain target enhance image;
B. the algorithm that adopts adjacent picture elements to add up carry out record to possibility target pixel, extracts the possible target pixel that brightness surpasses the certain threshold value of background luminance, suppresses the influence of random noise simultaneously;
C. the position of possibility target pixel is sorted out, noted the number of candidate target and the information such as profile, size, center and brightness of each target;
D. obtain the image that there is the possibility target in 3 frames continuously by A, B, C step, adopt the relevant way of interframe that the pseudo-target of the fixing part of extracting from background of relative position is removed, and pick out the target that to follow the tracks of according to the multiframe motion consistency principle of target; The concrete grammar that described pseudo-target removes is:
A. be initial point with all minimum target of center position X and Y direction in the possibility target after the classification of N-1 and N frame, calculate the relative position of other target's center's point and this point; Compare this two frame then, if for their major part may target X and the relative position error of Y direction all in the error range Nf that sets, then think the position that these may targets be the pseudo-target in the background and note them, otherwise the picture dot that the X position of the initial point of then selecting and setting originally differs minimum is an initial point, proceed aforesaid operations, till can removing pseudo-target over half, if selected all pixels is that initial point still can not remove pseudo-target, then the N-1 frame is abandoned, continue to read the N+2 frame, with above-mentioned method N frame and N+1 frame are carried out pseudo-target query, and the pseudo-target of twice record in three frames is all removed;
B. operation can remove most pseudo-target through a method, if still more than 1 of remaining possibility target, then choose the residue of three frames may target in difference in brightness be no more than 20% of average, shared pixel is counted difference and is no more than Np pixel, target is the candidate target group in the target that the difference of the picture dot development length of X and Y direction all is no more than Nxy1, judge its motion consistance in three frames, if the motion difference of certain group candidate target all is no more than Nxy2 pixel at X and Y direction, conclude that then it is the target of systematic search, otherwise, then abandon the N-1 frame and proceed the analysis of N+2 frame till finding target if there is not similar target;
Parameter N f, Nxy1, Nxy2 value in 0～4 pixel scope wherein, parameter N p is value in 0～8 pixel scope, and each parameter is according to judging strict degree value, and to the pine order, each parameter is carried out value from small to large by as strict as possible;
E. the target that chooses at step D, the characteristics such as installation site according to the frame frequency of imaging, detector pixel dimension, IMC mirror calculate the IMC amount of following the tracks of imaging, send to controller and carry out the IMC imaging.
2. the real-time adaptive disposal route of a kind of image drift imaging that disappears according to claim 1 is characterized in that: said steps A adopts relevant two scale filter algorithms, is odd number between 1～5 to the span of the S of original image S * S small scale filtering; To the span of the L of the large scale filtering of L * L is odd number between 5～9, and L＞S.
3. the real-time adaptive disposal route of a kind of image drift imaging that disappears according to claim 1 is characterized in that: in said step C, and 1.2～2 times of the background estimating value that the judgment threshold that may have a target obtains for large scale filtering.
4. the real-time adaptive disposal route of a kind of image drift imaging that disappears according to claim 1 is characterized in that: the real-time adaptive disposal route of the said image drift imaging that disappears adopts FPGA+DSP hardware to handle in real time and realizes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200810035329A CN100580704C (en) | 2008-03-28 | 2008-03-28 | Real time self-adapting processing method of image mobile imaging |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200810035329A CN100580704C (en) | 2008-03-28 | 2008-03-28 | Real time self-adapting processing method of image mobile imaging |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101276463A CN101276463A (en) | 2008-10-01 |
| CN100580704C true CN100580704C (en) | 2010-01-13 |
Family
ID=39995878
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200810035329A Expired - Fee Related CN100580704C (en) | 2008-03-28 | 2008-03-28 | Real time self-adapting processing method of image mobile imaging |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100580704C (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102607531B (en) * | 2012-03-19 | 2013-08-14 | 中国科学院上海技术物理研究所 | Spacial low-speed high-accuracy two-dimensional image motion compensation pointing control system |
| CN104199186B (en) * | 2014-09-16 | 2016-09-28 | 中国科学院光电技术研究所 | Piezoelectric tilting mirror high-voltage driver with object frequency characteristic compensation function |
| CN105261036A (en) * | 2015-09-17 | 2016-01-20 | 北京华航无线电测量研究所 | Object tracking method based on matching |
| CN106303223A (en) * | 2016-07-29 | 2017-01-04 | 努比亚技术有限公司 | A kind of image processing method and equipment |
| CN108280845B (en) * | 2017-12-26 | 2022-04-05 | 浙江工业大学 | Scale self-adaptive target tracking method for complex background |
| CN109348126B (en) * | 2018-11-07 | 2020-12-15 | 中国科学院光电研究院 | Area array digital TDI continuous push-broom imaging method for space camera |
| CN110220475B (en) * | 2019-05-30 | 2021-01-26 | 电子科技大学 | Linear CCD two-dimensional variable-speed imaging method based on image segmentation |
| CN112954136B (en) * | 2021-01-29 | 2023-05-19 | 中国科学院长春光学精密机械与物理研究所 | Method and device for inhibiting shot noise of remote sensing image of aviation strabismus remote imaging |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5299577A (en) * | 1989-04-20 | 1994-04-05 | National Fertility Institute | Apparatus and method for image processing including one-dimensional clean approximation |
| US20030012453A1 (en) * | 2001-07-06 | 2003-01-16 | Jasc Software, Inc. | Method for removing defects from images |
| CN101078622A (en) * | 2007-06-21 | 2007-11-28 | 北京凌云光视数字图像技术有限公司 | Aviation total-frame transfer type matrix CCD camera image shift compensation method |
-
2008
- 2008-03-28 CN CN200810035329A patent/CN100580704C/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5299577A (en) * | 1989-04-20 | 1994-04-05 | National Fertility Institute | Apparatus and method for image processing including one-dimensional clean approximation |
| US20030012453A1 (en) * | 2001-07-06 | 2003-01-16 | Jasc Software, Inc. | Method for removing defects from images |
| CN101078622A (en) * | 2007-06-21 | 2007-11-28 | 北京凌云光视数字图像技术有限公司 | Aviation total-frame transfer type matrix CCD camera image shift compensation method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101276463A (en) | 2008-10-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100580704C (en) | Real time self-adapting processing method of image mobile imaging | |
| RU2730687C1 (en) | Stereoscopic pedestrian detection system with two-stream neural network with deep training and methods of application thereof | |
| US20200175330A1 (en) | System and process for detecting, tracking and counting human objects of interest | |
| CN104155006B (en) | A kind of hand-held thermal infrared imager and its method to the range finding of Small object quick lock in | |
| JP3953103B2 (en) | Method and apparatus for quickly detecting the position of a target mark | |
| CN107408303A (en) | System and method for Object tracking | |
| CN107851318A (en) | System and method for Object tracking | |
| CN103959307B (en) | The method of detection and Expressive Features from gray level image | |
| CN103473554B (en) | Artificial abortion's statistical system and method | |
| CN106488115B (en) | For tracking and sensing the control device and method of the image sensing apparatus of tracking target | |
| US20040125207A1 (en) | Robust stereo-driven video-based surveillance | |
| CN101312524B (en) | Moving object detecting apparatus and method using light track analysis | |
| CN104180769A (en) | System and method for efficient surface measurement using a laser displacement sensor | |
| WO2004042673A2 (en) | Automatic, real time and complete identification of vehicles | |
| CN105760846A (en) | Object detection and location method and system based on depth data | |
| Jeong et al. | Real‐time generic target tracking for structural displacement monitoring under environmental uncertainties via deep learning | |
| CN107687818A (en) | Three-dimensional measurement method and three-dimensional measurement device | |
| JP2006012178A (en) | Method and system for detecting parking vehicle | |
| CN109754034A (en) | A kind of terminal device localization method and device based on two dimensional code | |
| KR101032098B1 (en) | Stand-alone environment traffic detecting system using thermal infra-red | |
| CN115690190B (en) | Moving target detection and positioning method based on optical flow image and pinhole imaging | |
| JP2007127478A (en) | Device and method for speed detection of tracking subject | |
| JP2003061075A (en) | Object-tracking device, object-tracking method and intruder monitor system | |
| CN101793530A (en) | Equipment and method for measuring decoupling factor of optical fiber gyro | |
| CN109977796A (en) | Trail current detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C17 | Cessation of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20100113 Termination date: 20130328 |