JPH05296724A - Parts recognition method - Google Patents

Parts recognition method

Info

Publication number
JPH05296724A
JPH05296724A JP4106628A JP10662892A JPH05296724A JP H05296724 A JPH05296724 A JP H05296724A JP 4106628 A JP4106628 A JP 4106628A JP 10662892 A JP10662892 A JP 10662892A JP H05296724 A JPH05296724 A JP H05296724A
Authority
JP
Japan
Prior art keywords
electrode
recognition
detected
recognition method
component
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP4106628A
Other languages
Japanese (ja)
Other versions
JP3098847B2 (en
Inventor
Atsushi Tanabe
敦 田邉
Junichi Hata
純一 秦
Masamichi Morimoto
正通 森本
Masao Iritani
正夫 入谷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Holdings Corp
Original Assignee
Matsushita Electric Industrial Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Matsushita Electric Industrial Co Ltd filed Critical Matsushita Electric Industrial Co Ltd
Priority to JP04106628A priority Critical patent/JP3098847B2/en
Publication of JPH05296724A publication Critical patent/JPH05296724A/en
Application granted granted Critical
Publication of JP3098847B2 publication Critical patent/JP3098847B2/en
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Landscapes

  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Image Processing (AREA)
  • Supply And Installment Of Electrical Components (AREA)

Abstract

PURPOSE:To provide a method for detecting the position of an electrode as a recognition object at high speed and with accuracy, regarding image recognition and processing in an electronic part packaging facility or the like. CONSTITUTION:With both lights for transmission and reflection turned on, an electronic part is photographed and an image pattern is obtained at the first process. Then, a boundary between an electronic part pattern in the image pattern and a background is obtained, and the outline of the part is thereby recognized. The rough inclination and center of the part is obtained on the basis of the outline so obtained. A processing area for detecting an electrode is set, according to the rough center and inclination, and the position of the electrode is detected by finding the extreme value of brightness projection data in the processing area.

Description

【発明の詳細な説明】Detailed Description of the Invention

【0001】[0001]

【産業上の利用分野】本発明は、電子部品実装設備等に
おいて、電子部品を高速かつ高精度に実装するために必
要とされる視覚認識装置の画像認識方法で、処理対象物
である電子部品の電極位置を画像パターンによって正確
に計測する方法に関するものである。
BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image recognition method of a visual recognition device required for mounting electronic parts at high speed and with high accuracy in electronic part mounting equipment and the like. The present invention relates to a method of accurately measuring the electrode position of the electrode according to an image pattern.

【0002】[0002]

【従来の技術】近年、電子部品実装分野では電子部品を
高速かつ高精度に回路基板に実装する技術が必要とされ
ている。このため、電子部品を撮像して得られる画像パ
ターンを高速に処理して、電子部品の位置・傾きの検出
を高速かつ正確に行う画像認識技術が活用される傾向に
ある。また、電子部品の細密化が進むにつれ電極の多ピ
ン化がなされている。そこで、多数の電極を有する電子
部品に関しては、個々の電極の位置を検出することが望
まれている。
2. Description of the Related Art In recent years, in the field of mounting electronic components, a technique for mounting electronic components on a circuit board at high speed and with high precision is required. Therefore, there is a tendency to utilize an image recognition technique that processes an image pattern obtained by capturing an image of an electronic component at high speed to detect the position and inclination of the electronic component at high speed and accurately. In addition, as electronic components are becoming finer, the number of pins of electrodes is increasing. Therefore, for electronic components having a large number of electrodes, it is desired to detect the position of each electrode.

【0003】従来、画像認識技術を活用した電子部品の
認識方法としては、対象電子部品の後方から照明を当て
そのシルエットを処理して、中心・傾き・電極位置など
を検出する透過認識方法や、前方から照明を当て電極よ
り反射された光を処理し中心・傾き・電極位置などを検
出する反射認識方法が提案されている。
Conventionally, as a method of recognizing an electronic component utilizing image recognition technology, a transparent recognition method of illuminating the target electronic component from behind and processing its silhouette to detect the center, inclination, electrode position, etc., A reflection recognition method has been proposed in which light is applied from the front and the light reflected from the electrodes is processed to detect the center, inclination, electrode position, and the like.

【0004】以下図を参照しながら従来の部品認識方法
に付いて説明する。
A conventional component recognition method will be described below with reference to the drawings.

【0005】図4は、透過認識方法の一例で、リード付
き電子部品42の後方より照明を当て(以下透過照明と
記す)、リードの先端位置を検出する例を示したもので
ある。図4の(a)に示すように、リード付き電子部品
42に透過照明41を当て、ビデオカメラ43で撮像す
ると、図4(b)に示すような画像パターン44を得
る。リード付き電子部品42の画像パターン45は透過
照明により、そのシルエットが映し出される。リード付
き電子部品42の画像パターン45と背景の境界を図4
の(c)に示すように検出し、境界追跡の軌跡を求める
と、Uターンを行う位置46が検出される。このUター
ンを行う位置46を検出すべき電極位置として検出す
る。この透過認識方法は、シルエットを処理するため、
論理が単純であり処理も高速である。
FIG. 4 is an example of a transmission recognition method and shows an example of detecting the tip position of the lead by illuminating the leaded electronic component 42 from behind (hereinafter referred to as transmission illumination). As shown in FIG. 4A, when the transillumination 41 is applied to the leaded electronic component 42 and an image is taken by the video camera 43, an image pattern 44 as shown in FIG. 4B is obtained. The silhouette of the image pattern 45 of the leaded electronic component 42 is displayed by transmitted illumination. The boundary between the image pattern 45 of the leaded electronic component 42 and the background is shown in FIG.
When the locus of the boundary tracking is obtained by the detection as shown in (c) of FIG. 3, the position 46 at which the U-turn is made is detected. The position 46 at which this U-turn is made is detected as the electrode position to be detected. Since this transparent recognition method processes silhouettes,
The logic is simple and the processing is fast.

【0006】しかし、透過認識方法では、電極が部品の
外郭形状より内側に曲げられている電子部品(Jリード
電子部品と記す)に関しては、電極位置を検出すること
が困難である。そこで電極を直接とらえることの出来る
反射認識方法が提案された。
However, with the transmission recognition method, it is difficult to detect the electrode position of an electronic component (referred to as a J-lead electronic component) in which the electrode is bent inward from the outer shape of the component. Therefore, a reflection recognition method has been proposed that can directly capture the electrodes.

【0007】図5は、反射認識方法の一例で、Jリード
電子部品51のリード位置を検出する例を示したもので
ある。図5の(a)に示すように、Jリード電子部品5
1の前方より照明(以下反射照明と記す)52を当て、
ビデオカメラ43で撮像すると、図5の(b)に示すよ
うな画像パターン53を得る。Jリード電子部品51の
画像パターン54は、反射照明52の反射光が映し出さ
れるため、電極部分が明るくなっている。図5の(c)
に示すように、明るく光る電極部分55は、Sのように
走査することで大まかな位置が検出される。検出された
大まかな位置に基き処理ウィンドウ56を設け、処理ウ
ィンドウ56内の輝度分布を57のように投影し、極値
58を検出する。電極部分は、高輝度であるので、検出
された極値58を検出すべき電極位置として検出する。
FIG. 5 shows an example of the reflection recognition method, in which the lead position of the J-lead electronic component 51 is detected. As shown in FIG. 5A, the J lead electronic component 5
Illuminate (hereinafter referred to as reflective illumination) 52 from the front of 1
When picked up by the video camera 43, an image pattern 53 as shown in FIG. In the image pattern 54 of the J-lead electronic component 51, the reflected light of the reflective illumination 52 is projected, so that the electrode portion is bright. FIG. 5 (c)
As shown in FIG. 6, the electrode portion 55 that shines brightly is scanned in the manner of S, and the rough position is detected. The processing window 56 is provided on the basis of the detected rough position, the luminance distribution in the processing window 56 is projected as 57, and the extreme value 58 is detected. Since the electrode portion has high brightness, the detected extreme value 58 is detected as the electrode position to be detected.

【0008】[0008]

【発明が解決しようとする課題】前述のように電子部品
の多ピン化が進むにつれ、電極を部品側面に配置するこ
とに限界をむかえつつある。そうした背景のもと、図6
に示すように電極を部品の外郭形状より内部に配した部
品が登場している。
As described above, as the number of pins of electronic parts increases, the limitation of arranging electrodes on the side surface of parts is being reached. Against this background, Figure 6
As shown in Fig. 3, some parts have been introduced in which the electrodes are arranged inside the outer shape of the part.

【0009】前述したように透過認識方法では、外形よ
り内部に配された電極を認識することは不可能である。
As described above, in the transmission recognition method, it is impossible to recognize the electrodes arranged inside the outer shape.

【0010】また反射認識方法においても、図6の
(a)のように部品61が傾いている場合、電極が部品
の外郭形状の内側にあるため、電極のみを検出するよう
に走査Sすると、間違った処理ウィンドウ62を設定し
てしまう。また外形を走査Sすると図6の(b)に示す
ような形状の部品63では、図示するような間違った処
理ウィンドウ64を設定する場合を生じる。いずれの場
合も走査間隔を密にし、範囲を広げることである程度解
決できるが、処理速度がかなり遅くなるため良い解決策
とは言えない。
Also in the reflection recognition method, when the component 61 is inclined as shown in FIG. 6A, since the electrodes are inside the outer shape of the component, if the scanning S is performed to detect only the electrodes, The wrong processing window 62 is set. Further, if the outer shape is scanned S, in the case of the component 63 having the shape as shown in FIG. 6B, a wrong processing window 64 as shown may be set. In either case, it is possible to solve the problem to some extent by making the scanning interval dense and widening the range, but it cannot be said to be a good solution because the processing speed becomes considerably slow.

【0011】本発明は、上述した従来技術の欠点を克服
し、高速かつ信頼性の高い部品認識方法を提供すること
を目的とする。
An object of the present invention is to overcome the above-mentioned drawbacks of the prior art and to provide a high-speed and highly reliable component recognition method.

【0012】[0012]

【課題を解決するための手段】前記問題点を解決するた
めに本発明は下記の手段を用いている。
In order to solve the above problems, the present invention uses the following means.

【0013】まず透過照明、反射照明を同時に当てなが
ら対象電子部品を撮像し画像パターンを得る(第1工
程)。透過照明により背景は光輝度となっているので、
部品ボディと背景の境界を検出する(第2工程)。得ら
れた境界より、部品の大まかな中心と傾きを検出する
(第3工程)。次に検出された中心と傾きを利用し電極
検出用の処理エリアを設定し、処理エリア内の輝度分布
を解析し、電極の位置を検出する(第4工程)。
First, an image pattern is obtained by picking up an image of a target electronic component while simultaneously applying transmitted illumination and reflected illumination (first step). Since the background has a light intensity due to transmitted illumination,
The boundary between the component body and the background is detected (second step). From the obtained boundary, the approximate center and inclination of the component are detected (third step). Next, a processing area for electrode detection is set using the detected center and inclination, the luminance distribution in the processing area is analyzed, and the position of the electrode is detected (fourth step).

【0014】[0014]

【作用】本発明は上記した構成により、透過認識方法と
反射認識方法の両方の長所を活かすことが可能である。
すなわち、透過認識方法における高速な傾き・中心検出
を用い、高精度な認識を可能とする反射認識方法の処理
エリアを、簡単かつ正確に設定することが可能となる。
The present invention can make use of the advantages of both the transmission recognition method and the reflection recognition method by the above-mentioned configuration.
That is, it is possible to easily and accurately set the processing area of the reflection recognition method that enables highly accurate recognition by using the high-speed tilt / center detection in the transmission recognition method.

【0015】[0015]

【実施例】本発明の一実施例として、電極が部品外郭形
状の内側に配されたPGA(PadGrid Array)部品の電
極位置を検出する方法について図1、図2、図3を用い
て説明する。
DESCRIPTION OF THE PREFERRED EMBODIMENTS As an embodiment of the present invention, a method for detecting the electrode position of a PGA (Pad Grid Array) component in which electrodes are arranged inside the outer shape of the component will be described with reference to FIGS. 1, 2 and 3. ..

【0016】図1において第1工程は、画像パターン入
力工程S1である。図2の(a)に示すように、認識対
象物であるPGA部品21に透過照明22と反射照明2
3を当て、ビデオカメラ24などの撮像手段で撮像する
ことにより映像信号を得る。
In FIG. 1, the first step is an image pattern input step S1. As shown in FIG. 2A, the PGA component 21, which is the recognition target, has a transmission illumination 22 and a reflection illumination 2.
3 is applied, and an image is obtained by an image pickup means such as the video camera 24 to obtain a video signal.

【0017】得られた映像信号をデジタル化し、図2の
(b)に示す画像パターン25とする。
The obtained video signal is digitized to form an image pattern 25 shown in FIG. 2 (b).

【0018】第2工程は、部品外形検出工程S2であ
る。画像パターン25において、背景部分は透過照明に
より高輝度となっている。そのため部品ボディと背景の
境界では、著しい輝度レベルの変化がみられる。そこで
第1工程により得られた画像パターン25に対し、図3
の(a)に示すように、輝度変化の著しい境界点の追跡
Pを行う。こうして得られた境界がPGA部品21の外
形として検出されることになる。
The second step is a component outline detection step S2. In the image pattern 25, the background portion has a high brightness due to transmitted illumination. Therefore, a significant change in the brightness level is seen at the boundary between the component body and the background. Therefore, for the image pattern 25 obtained in the first step, as shown in FIG.
As shown in (a) of (1), the tracking P of the boundary point where the luminance change is remarkable is performed. The boundary thus obtained is detected as the outer shape of the PGA component 21.

【0019】第3工程は、中心・傾き粗検出S3であ
る。前記第2工程で検出された部品外形から従来用いら
れてきた方法を用い中心・傾き32を決定する。外形か
ら中心・傾きを算出する方法としてはこれまでにいろい
ろな方法が提案されているが、例えば部品外形の外接長
方形を求めその中心をもって部品の中心とする方法や、
ある任意の一点と外形を構成する境界点群との2次モー
メントを算出することにより傾きを検出する方法などが
ある。
The third step is center / tilt coarse detection S3. The center / inclination 32 is determined from the external shape of the component detected in the second step by using a conventionally used method. Various methods have been proposed as a method of calculating the center / inclination from the outer shape, but for example, a method of obtaining a circumscribed rectangle of the outer shape of the component and setting the center as the center of the component,
There is a method of detecting the inclination by calculating the second moment between a certain arbitrary point and the boundary point group forming the outer shape.

【0020】第4工程は、電極位置検出工程S4であ
る。前記第3工程によって決定された大まかな中心と傾
き、及び予め与えられている部品の外形寸法や電極間寸
法を利用し、図3の(b)に示すような処理ウィンドウ
33を設定する。設定した処理ウィンドウ33内の輝度
分布を各々α・β方向に投影し、投影データ34,34
を得る。得られた夫々の投影データ34,34の極値を
なす位置の交点35を各電極の位置として検出する。
The fourth step is an electrode position detecting step S4. A processing window 33 as shown in FIG. 3B is set by utilizing the rough center and inclination determined in the third step, and the external dimensions and inter-electrode dimensions given in advance. The brightness distributions in the set processing window 33 are projected in the α and β directions, and the projection data 34, 34
To get The intersection 35 of the extreme positions of the obtained projection data 34, 34 is detected as the position of each electrode.

【0021】この第4工程での処理を繰り返し行い、全
ての電極位置を検出し処理を終了する。
The process in the fourth step is repeated to detect all the electrode positions, and the process is completed.

【0022】[0022]

【発明の効果】本発明によれば、透過認識方法の長所を
活かした中心と傾きの粗検出を利用して処理エリアを設
定することができるため、反射認識方法の短所を是正す
ることができ、処理エリアの設定を簡単でかつ正確に行
うことが可能である。そして本発明によれば、透過認識
方法と反射認識方法の夫々の長所を生かすことによっ
て、処理対象となる電子部品の形状や姿勢に影響される
ことなく、より信頼性の高い対象物の電極位置を検出す
ることが可能となる。
According to the present invention, since the processing area can be set by utilizing the center and inclination coarse detection, which makes use of the advantages of the transmission recognition method, the disadvantages of the reflection recognition method can be corrected. The processing area can be set easily and accurately. According to the present invention, by utilizing the advantages of each of the transmission recognition method and the reflection recognition method, the electrode position of a more reliable target can be obtained without being affected by the shape and orientation of the electronic component to be processed. Can be detected.

【図面の簡単な説明】[Brief description of drawings]

【図1】本発明の一実施例の概要を示すフローチャート
である。
FIG. 1 is a flowchart showing an outline of an embodiment of the present invention.

【図2】本発明の一実施例における撮像方法と画像パタ
ーンを示す説明図である。
FIG. 2 is an explanatory diagram showing an imaging method and an image pattern according to an embodiment of the present invention.

【図3】本発明の一実施例における電極位置の検出方法
を示す説明図である。
FIG. 3 is an explanatory diagram showing a method for detecting an electrode position according to an embodiment of the present invention.

【図4】従来例の電子部品の電極位置の検出方法を示す
説明図である。
FIG. 4 is an explanatory diagram showing a method of detecting electrode positions of an electronic component of a conventional example.

【図5】他の従来例の電極位置の検出方法を示す説明図
である。
FIG. 5 is an explanatory diagram showing another conventional electrode position detection method.

【図6】従来例の問題点を示す説明図である。FIG. 6 is an explanatory diagram showing a problem of the conventional example.

【符号の説明】[Explanation of symbols]

21 PGA部品(認識対象物) 22 透過照明 23 反射照明 24 ビデオカメラ 32 部品の中心・傾き 33 処理ウィンドウ(処理エリア) 35 電極位置 21 PGA component (recognition target) 22 Transmitted illumination 23 Reflective illumination 24 Video camera 32 Center / tilt of component 33 Processing window (processing area) 35 Electrode position

───────────────────────────────────────────────────── フロントページの続き (51)Int.Cl.5 識別記号 庁内整理番号 FI 技術表示箇所 H05K 13/08 B 8315−4E (72)発明者 入谷 正夫 大阪府門真市大字門真1006番地 松下電器 産業株式会社内─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. 5 Identification number Reference number within the agency FI technical display location H05K 13/08 B 8315-4E (72) Inventor Masao Iriya 1006 Kadoma, Kadoma, Osaka Prefecture Matsushita Electric Sangyo Co., Ltd.

Claims (1)

【特許請求の範囲】[Claims] 【請求項1】 認識対象物に対し前後両面より照明を当
てた状態で撮像し画像パターンを得る第1工程と、認識
対象物の画像パターンと背景との境界から認識対象物の
外形を検出する第2工程と、前記第2工程で検出された
認識対象物の外形より認識対象物の大まかな中心と傾き
を検出する第3工程と、前記第3工程により検出された
認識対象物の大まかな中心と傾きを利用し処理エリアを
設定し、処理エリア内の輝度分布に基き認識対象物の電
極位置を検出する第4工程を備えた部品認識方法。
1. A first step of capturing an image pattern of a recognition target object while illuminating the recognition target object from both front and rear surfaces, and detecting the outer shape of the recognition target object from the boundary between the image pattern of the recognition target object and the background. A second step; a third step of detecting a rough center and an inclination of the recognition object from the outer shape of the recognition object detected in the second step; and a rough recognition object of the recognition object detected in the third step. A component recognition method comprising a fourth step of setting a processing area using a center and an inclination, and detecting an electrode position of a recognition target object based on a luminance distribution in the processing area.
JP04106628A 1992-04-24 1992-04-24 Component recognition method Expired - Lifetime JP3098847B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP04106628A JP3098847B2 (en) 1992-04-24 1992-04-24 Component recognition method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP04106628A JP3098847B2 (en) 1992-04-24 1992-04-24 Component recognition method

Publications (2)

Publication Number Publication Date
JPH05296724A true JPH05296724A (en) 1993-11-09
JP3098847B2 JP3098847B2 (en) 2000-10-16

Family

ID=14438381

Family Applications (1)

Application Number Title Priority Date Filing Date
JP04106628A Expired - Lifetime JP3098847B2 (en) 1992-04-24 1992-04-24 Component recognition method

Country Status (1)

Country Link
JP (1) JP3098847B2 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006226748A (en) * 2005-02-16 2006-08-31 Juki Corp Imaging recognition device of transparent body
JP2009294123A (en) * 2008-06-06 2009-12-17 Advanced Mask Inspection Technology Kk Pattern discriminator, pattern discriminating method and inspection device of sample
JP2013089803A (en) * 2011-10-19 2013-05-13 Disco Abrasive Syst Ltd Work-piece imaging device and work-piece imaging method
CN114788436A (en) * 2019-12-16 2022-07-22 株式会社富士 Component mounting machine

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006226748A (en) * 2005-02-16 2006-08-31 Juki Corp Imaging recognition device of transparent body
JP2009294123A (en) * 2008-06-06 2009-12-17 Advanced Mask Inspection Technology Kk Pattern discriminator, pattern discriminating method and inspection device of sample
JP2013089803A (en) * 2011-10-19 2013-05-13 Disco Abrasive Syst Ltd Work-piece imaging device and work-piece imaging method
CN114788436A (en) * 2019-12-16 2022-07-22 株式会社富士 Component mounting machine
CN114788436B (en) * 2019-12-16 2023-11-10 株式会社富士 Component mounting machine

Also Published As

Publication number Publication date
JP3098847B2 (en) 2000-10-16

Similar Documents

Publication Publication Date Title
EP0501683B1 (en) Technique for enhanced two-dimensional imaging
JP3269170B2 (en) Electronic component position detection method
JP3098847B2 (en) Component recognition method
JPH05231837A (en) Method and device for measuring shape
JPS59107202A (en) Checking device of fitting position of minute parts
JPH05296725A (en) Position detecting method
JPH0831497B2 (en) Spherical object position detection method and apparatus
JPH0989525A (en) Position detecting method for lead shoulder and for lead end
JPH04329344A (en) Mounted board inspecting device
JPH11132737A (en) Image processing method
JPH07119705B2 (en) Electronic component inspection device
JP3000615B2 (en) Pattern recognition method
JP3160838B2 (en) Surface defect inspection equipment
JPH095022A (en) Pattern position detecting method and apparatus by image recognition
JPH0553320B2 (en)
JP3038718B2 (en) Method and apparatus for inspecting solder bridge of lead component
JP3006437B2 (en) Electronic component observation device and electronic component observation method
JPH05275900A (en) Method for inspection of state of mounted part lead
JP3031955B2 (en) Soldering inspection method
JPH10170224A (en) Visual checking method for soldering joint and its device
JP2819696B2 (en) Soldering inspection equipment
JPS63179245A (en) Inspection system for deviation in chip position on printed circuit board
JPS6280504A (en) Pin position measurement system for article with plural pins
JPS63190354A (en) Visual inspection device
JPS59102106A (en) Inspecting system

Legal Events

Date Code Title Description
FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20070811

Year of fee payment: 7

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20080811

Year of fee payment: 8

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20080811

Year of fee payment: 8

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20090811

Year of fee payment: 9

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20090811

Year of fee payment: 9

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20100811

Year of fee payment: 10

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20110811

Year of fee payment: 11

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20110811

Year of fee payment: 11

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20120811

Year of fee payment: 12

EXPY Cancellation because of completion of term
FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20120811

Year of fee payment: 12