JP4369187B2 - Personal identification device - Google Patents

Personal identification device Download PDF

Info

Publication number
JP4369187B2
JP4369187B2 JP2003326347A JP2003326347A JP4369187B2 JP 4369187 B2 JP4369187 B2 JP 4369187B2 JP 2003326347 A JP2003326347 A JP 2003326347A JP 2003326347 A JP2003326347 A JP 2003326347A JP 4369187 B2 JP4369187 B2 JP 4369187B2
Authority
JP
Japan
Prior art keywords
face
user
fingerprint
camera
finger
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP2003326347A
Other languages
Japanese (ja)
Other versions
JP2005092611A5 (en
JP2005092611A (en
Inventor
幸喜 奥西
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Priority to JP2003326347A priority Critical patent/JP4369187B2/en
Publication of JP2005092611A publication Critical patent/JP2005092611A/en
Publication of JP2005092611A5 publication Critical patent/JP2005092611A5/ja
Application granted granted Critical
Publication of JP4369187B2 publication Critical patent/JP4369187B2/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Description

この発明は、顔と指紋という二つの生体要素を用いて利用者を識別する個人識別装置に関するものである。   The present invention relates to a personal identification device that identifies a user using two biological elements, a face and a fingerprint.

顔の形を識別する顔識別装置を利用した個人識別装置の構成は、顔画像を撮影するカメラと画像処理および個人識別を行う制御部とからなり、制御部は、以下の手順で個人識別を行う。
(1)カメラで捕らえた画像から人の顔を識別し切り取る。
(2)複数の顔画像のうち、識別に適している画像を選別する。
(3)顔画像の大きさを調整する。
(4)あらかじめ記憶しておいた顔データと照合を行い、個人情報を得る。
しかし、顔の画像だけでは個人識別が困難なため、顔以外の生体情報を同時に利用する発明が例えば特許文献1で提案されている。
The configuration of the personal identification device using the face identification device for identifying the shape of the face includes a camera that captures a face image and a control unit that performs image processing and personal identification. The control unit performs personal identification according to the following procedure. Do.
(1) A human face is identified and cut out from an image captured by a camera.
(2) An image suitable for identification is selected from a plurality of face images.
(3) Adjust the size of the face image.
(4) The personal data is obtained by collating with the face data stored in advance.
However, since it is difficult to identify individuals by using only a face image, Patent Document 1 proposes an invention that uses biological information other than a face at the same time.

特開2002−230553号公報JP 2002-230553 A

顔の識別の難しさは識別する相手(利用者)の顔が常に変化することにある。同じ人間の画像であっても、基本(例えば証明写真等に要求される表情)での映像に対して、まばたきをした時の映像、横を向いたときの映像、カメラから少し離れたときの映像などがあり、個人識別の支障となる。顔識別装置の照合時間は2秒程度が実用に耐える時間とされる。常に変化する顔の画像から最適な顔画像を得るためには、2秒間に何枚もの顔画像を取り込む必要がある。すなわち、上記(1)の画像切り取る処理と(2)の選別処理を超高速に行う必要がある。よって、超高速な画像処理装置を使用しないと実用に耐える顔識別装置が提供できず、現在市販されている顔識別装置は非常に高価なものとなっている。   The difficulty in identifying a face is that the face of the other party (user) to be identified always changes. Even if it is the same human image, the image when blinking, the image when looking sideways, the image when facing away from the basic (for example, the facial expression required for ID photos, etc.) There are images, etc., which hinders personal identification. The collation time of the face identification device is about 2 seconds to be practically used. In order to obtain an optimal face image from a constantly changing face image, it is necessary to capture several face images in 2 seconds. That is, it is necessary to perform the image cutting process (1) and the selection process (2) at an extremely high speed. Therefore, a face recognition device that can withstand practical use cannot be provided unless an ultra-high-speed image processing device is used, and currently commercially available face recognition devices are very expensive.

顔の影も問題となる。顔にあたる光の角度により影の発生場所が異なるためである。顔識別を行うときには、影をなくすように光源で顔を照らすのが望ましい。しかし、光源を常に点灯したままにしておくと、省エネやランプの寿命など環境上の問題が発生する。   The shadow of the face is also a problem. This is because the location where a shadow is generated varies depending on the angle of light hitting the face. When performing face identification, it is desirable to illuminate the face with a light source so as to eliminate shadows. However, if the light source is always turned on, environmental problems such as energy saving and lamp life will occur.

また、顔識別装置の誤識別について説明する。世の中には例えば双子のように非常によく似た顔がある。顔識別装置では、非常によく似た顔を識別することはできない。よって、高度な識別性能を要求するような用途においては、顔識別装置だけでは性能が不十分である。   Further, misidentification of the face identification device will be described. There are very similar faces in the world, such as twins. A face identification device cannot identify very similar faces. Therefore, in applications that require a high level of discrimination performance, the face discrimination device alone is insufficient in performance.

この発明は、上記のような問題を解消するためになされたもので、顔と指紋を利用して、上記顔識別装置の欠点を解消した個人識別装置を得ることを目的とする。   The present invention has been made to solve the above-described problems, and an object of the present invention is to obtain a personal identification device that eliminates the drawbacks of the face identification device using a face and a fingerprint.

この発明にかかる個人識別装置は、カメラにより利用者の顔の情報を取り込み登録データと照合して本人判定を行う制御部と、利用者の指紋を取り込み登録データと照合して本人判定を行う指紋照合部と、適切な顔画像が撮影できる位置に利用者が上記カメラと対面するとき、利用者が自分の顔を確認できる位置に設置された縦方向に湾曲する縦長の鏡とを備え、上記指紋照合部が指紋照合した動作情報を、利用者の顔情報を取り込むタイミングの生成に利用するようにしたものである。
A personal identification device according to the present invention includes a control unit that captures a user's face information by a camera and compares it with registered data to determine the identity, and a fingerprint that captures a user's fingerprint and matches the registered data to determine the identity. A collation unit, and a vertically long mirror that is installed in a position where the user can confirm his / her face when the user faces the camera at a position where an appropriate face image can be captured , and The operation information collated with the fingerprint by the fingerprint collation unit is used to generate timing for capturing user face information.

この発明によれば、指紋照合部の指置き部に指が置かれたことを検出する指紋照合部が指紋照合した動作情報を、カメラで顔画像を取り込むタイミングとして利用することにより、指紋照合部の指置き部に指が置かれたときに顔画像を取り込むことができる。指置き部に指を置くとき、利用者はちょうど腕の長さだけカメラから離れていることになり、常に安定した大きさの顔画像を得ることができる。また、利用者は指置き部に指を置くときは、目を閉じたりよそ見をしたりしていることは少なく、指置き部を見つめていることが多い。よって、顔画像の識別に適した顔画像を得ることができる。その結果、顔画像の切り取り処理および顔画像の選別処理を大幅に軽減できるので、制御部を安価に構成することができると同時に、顔の識別結果と指紋照合の結果の両方を使用して個人識別を行うため、識別性能を高めることができる。
According to this invention, the fingerprint collation unit uses the operation information collated by the fingerprint collation unit that detects that the finger is placed on the finger placement unit of the fingerprint collation unit as the timing to capture the face image with the camera. When a finger is placed on the finger placement part, a face image can be captured. When placing a finger on the finger placement unit, the user is away from the camera by the length of his arm, and a face image having a stable size can be obtained at all times. Further, when placing a finger on the finger placement unit, the user rarely closes his eyes or looks away, and often looks at the finger placement unit. Therefore, a face image suitable for identification of the face image can be obtained. As a result, the face image clipping process and the face image selection process can be greatly reduced, so that the control unit can be configured at a low cost, and at the same time, an individual using both the face identification result and the fingerprint matching result. Since identification is performed, identification performance can be improved.

実施の形態1.
実施の形態1.
以下、この発明の実施の形態1に係る個人識別装置について、図1の装置外観図、図2のテーブル、および図3のフローチャートを使用して説明する。図1において、本発明に係る個人識別装置1は、顔画像を撮影するカメラ2、指紋照合部の指置き部4、利用者の顔の明るさを一定以上に照らす照明装置5、自分の顔を映すことができる縦に長い凸形状の鏡6を納めた箱体3からなる。箱体3の前面に、カメラ2が配置され、カメラ2に近接して照明装置5が配置されている。さらにカメラ2の10cm以内の下方に指置き部4が設けられ、この指置き部4に指を置いて指紋データを取り込むようになされている。
Embodiment 1 FIG.
Embodiment 1 FIG.
Hereinafter, the personal identification device according to Embodiment 1 of the present invention will be described with reference to the external view of the device in FIG. 1, the table in FIG. 2, and the flowchart in FIG. In FIG. 1, a personal identification device 1 according to the present invention includes a camera 2 that captures a face image, a finger placement unit 4 of a fingerprint collation unit, a lighting device 5 that illuminates the brightness of a user's face to a certain level, and his / her face. Is a box 3 in which a vertically long mirror 6 is housed. A camera 2 is disposed on the front surface of the box 3, and an illumination device 5 is disposed in the vicinity of the camera 2. Further, a finger rest portion 4 is provided below the camera 2 within 10 cm, and a finger is placed on the finger rest portion 4 to capture fingerprint data.

これらカメラ2および指置き部4の側方に上下方向に湾曲する縦に長い凸面鏡6が箱体3の前面に沿って貼り付けられている。鏡6を凸形状にすることにより、鏡6の縦寸法を小さくしても背の低い利用者から背の高い利用者まで対応できるようにしている。また、この鏡6は、利用者がカメラ2の正面の撮影最適な位置に立ったときに自分の顔が映るような位置に設置されている。   A vertically long convex mirror 6 that is curved in the vertical direction is attached to the side of the camera 2 and the finger holder 4 along the front surface of the box 3. By making the mirror 6 into a convex shape, even if the vertical dimension of the mirror 6 is reduced, it is possible to cope with a short user to a tall user. In addition, the mirror 6 is installed at a position where the user's face is reflected when the user stands at the optimal shooting position in front of the camera 2.

この個人識別装置1には、図示していないが、カメラ2と、カメラ2の画像を処理し照合を行う制御部とからなる顔識別部、および指置き部4で得た指紋を照合する指紋照合部、照明5を制御・処理する制御部を備えている。   Although not shown, the personal identification device 1 includes a face identification unit including a camera 2 and a control unit that processes and collates images of the camera 2, and a fingerprint that collates fingerprints obtained by the finger placement unit 4. The collation part and the control part which controls and processes the illumination 5 are provided.

次に動作を説明する。利用者は、個人識別装置1のカメラ2に向かって自分の顔が鏡6に写るように立つと、指置き部4に指を置くことが要求される。以下、顔識別、指紋照合、照明制御の動作を、図3のフローチャートを用いて説明する。ステップS1で、指置き部4に指が置かれるのを待つ。指が置かれなければスタートに戻る。指が置かれると、S2で、登録データに基づいて指紋照合部が指紋の照合を行い、個人を特定し、個人番号を得る。S3で、図2のテーブルを利用して個人番号に対応した顔の高さ情報を、また、S4で、上記個人番号に対応して登録している顔データを得る。上記指紋照合部が指紋照合した動作情報を検出し、これをトリガとして、S6で、カメラ2を動作させ顔の画像を取り込む。このとき、S5で、照明5を点灯し利用者の顔を照らすようにする。S7で、S3で得た顔の高さ情報を利用し、顔の部分を切り出す。
Next, the operation will be described. When the user stands so that his / her face is reflected in the mirror 6 toward the camera 2 of the personal identification device 1, the user is required to place his / her finger on the finger placement unit 4. Hereinafter, operations of face identification, fingerprint collation, and illumination control will be described with reference to the flowchart of FIG. In step S1, it waits for a finger to be placed on the finger placement unit 4. If no finger is placed, return to start. When the finger is placed, in S2, the fingerprint collation unit collates the fingerprint based on the registered data, identifies the individual, and obtains a personal number. In S3, the height information of the face corresponding to the personal number is obtained using the table of FIG. 2, and the face data registered corresponding to the personal number is obtained in S4. Operation information collated by the fingerprint collation unit is detected, and using this as a trigger, the camera 2 is operated to capture a face image in S6. At this time, in S5, the illumination 5 is turned on to illuminate the user's face. In S7, the face height information obtained in S3 is used to cut out the face portion.

S8で、切り出した顔画像の良否を判断する。顔画像が良好であればS9へ進んで照明5を消灯し、S10で、S4で得た顔データを利用して顔画像の判別を行う。S11では、指紋の照合結果と顔画像の判別結果から総合判別結果を得る。そして、S12で総合判別結果を出力し、スタートに戻る。   In S8, the quality of the cut face image is determined. If the face image is good, the process proceeds to S9 and the illumination 5 is turned off. In S10, the face image is determined using the face data obtained in S4. In S11, an overall discrimination result is obtained from the fingerprint collation result and the face image discrimination result. In S12, the comprehensive discrimination result is output, and the process returns to the start.

上記動作において、この発明の第1の特徴は、指置き部4に指が置かれると、指紋照合部は指が置かれたことを検出して指紋照合を行うが、その指紋照合した動作情報を、カメラ2で顔画像を取り込むタイミングとして利用することである。このため、指紋照合部の指置き部4に指が置かれたときに顔画像を取り込むことができる。指置き部4に指を置くとき、利用者はちょうど腕の長さだけカメラ2から離れていることになり、常に安定した大きさの顔画像を得ることができ、利用者の前後方向のずれを少なくする効果がある。また、利用者は指置き部4に指を置くときは、目を閉じたりよそ見をしたりしていることは少なく、指置き部4を見つめていることが多い。よって、顔画像識別に適した顔画像を得ることができ、顔の向き、目の向きの補正効果が期待できる。また、常に映像を取り込み利用者の有無を判別し続ける必要が無くなるので、消費電力を削減することができる。 In the above operation, the first feature of the invention, the finger on the finger rest portion 4 instructions are built, the operation fingerprint verification unit performs the fingerprint verification by detecting that the finger is placed, which is the fingerprint matching The information is used as timing when the face image is captured by the camera 2. For this reason, a face image can be captured when a finger is placed on the finger placement unit 4 of the fingerprint collation unit. When placing a finger on the finger placement unit 4, the user is separated from the camera 2 by the length of the arm, and a face image having a stable size can be obtained at all times. Has the effect of reducing Further, when placing a finger on the finger placement unit 4, the user rarely closes his eyes or looks away, and often looks at the finger placement unit 4. Therefore, a face image suitable for face image identification can be obtained, and a face orientation and eye orientation correction effect can be expected. In addition, since it is not necessary to always capture video and continue to determine the presence or absence of a user, power consumption can be reduced.

また、この発明の第2の特徴は、指紋照合部が動作し、顔画像を取り込むときのみ照明装置5を点灯する制御を行うことである。このため、照明用のランプの点灯時間を大幅に減らすことができ、省エネおよびランプの寿命を伸ばす効果がある。また、利用者に顔画像を取り込んでいることを知らせる効果もある。   A second feature of the present invention is that the illumination device 5 is controlled to be turned on only when the fingerprint collation unit operates and a face image is captured. For this reason, the lighting time of the lamp for illumination can be significantly reduced, and there is an effect of energy saving and extending the life of the lamp. In addition, there is also an effect of notifying the user that the face image is captured.

また、この発明の第3の特徴は、カメラ2を指置き部4の10cm以内に設置したことである。10cmとしているのは、カメラの視野、人間の顔の大きさから考えての許容値である。これによれば、指置き部4を見つめる利用者の顔をより正面から撮影することができ、安定した顔画像を得ることができる。   The third feature of the present invention is that the camera 2 is installed within 10 cm of the finger rest 4. 10 cm is an allowable value considering the field of view of the camera and the size of the human face. According to this, the user's face staring at the finger placement unit 4 can be photographed from the front, and a stable face image can be obtained.

また、この発明の第4の特徴は、利用者がカメラの正面に立つたときに自分の顔が映るように調整された鏡6を設置したことである。利用者が自分の顔が鏡6に映るようにした状態で指置き部4に指を置けば、横方向のずれの少ない顔画像を得ることができ、顔画像の切り出し時間を短くすることができる。   The fourth feature of the present invention is that a mirror 6 adjusted to reflect his / her face when the user stands in front of the camera is installed. If the user places his / her finger on the finger rest 4 in a state where his / her face is reflected in the mirror 6, a face image with little lateral displacement can be obtained, and the cut-out time of the face image can be shortened. it can.

また、この発明の第5の特徴は、鏡6を縦方向に凸の縦長にしたことである。これにより利用者の身長に関わらず利用が可能である。   The fifth feature of the present invention is that the mirror 6 has a vertically long shape that is convex in the vertical direction. Thereby, it can be used regardless of the height of the user.

また、この発明の第6の特徴は、利用者の顔の高さ情報を顔の切り出しに利用するようにしたことである。その結果、短時間で顔の切り取り処理を行うことができる。つまり、高さ方向の補正効果がある。   A sixth feature of the present invention is that the height information of the user's face is used for face extraction. As a result, the face cut-out process can be performed in a short time. That is, there is a correction effect in the height direction.

このように、この発明によれば、顔画像の切り取り処理および顔画像の選別処理を大幅に軽減することができるので、顔識別部を安価に構成することができる。また、顔の識別結果と指紋照合の結果の両方を使用して個人識別を行うため、識別性能を高めることができる。   As described above, according to the present invention, the face image cut-out process and the face image selection process can be greatly reduced, so that the face identification unit can be configured at low cost. Further, since the personal identification is performed using both the face identification result and the fingerprint collation result, the identification performance can be improved.

本発明は入退室管理やパソコンへのアクセス管理等に適用できる。   The present invention can be applied to entrance / exit management, access management to personal computers, and the like.

この発明の実施の形態1に係る個人識別装置の外観を示す図で、(a)は平面図、(b)は正面図、(c)は側面図である。It is a figure which shows the external appearance of the personal identification device which concerns on Embodiment 1 of this invention, (a) is a top view, (b) is a front view, (c) is a side view. 実施の形態1で使用されるテーブルを示す図である。FIG. 3 is a diagram showing a table used in the first embodiment. 実施の形態1の動作を説明するフローチャートである。3 is a flowchart for explaining the operation of the first embodiment.

符号の説明Explanation of symbols

1 個人識別装置、 2 カメラ、
3 箱体、 4 指置き部、
5 照明装置、 6 鏡。
1 personal identification device, 2 camera,
3 boxes, 4 finger rests,
5 lighting devices, 6 mirrors.

Claims (2)

カメラにより利用者の顔の情報を取り込み登録データと照合して本人判定を行う制御部と、利用者の指紋を取り込み登録データと照合して本人判定を行う指紋照合部と、適切な顔画像が撮影できる位置に利用者が上記カメラと対面するとき、利用者が自分の顔を確認できる位置に設置された縦方向に湾曲する縦長の鏡とを備え、上記指紋照合部が指紋照合した動作情報を、利用者の顔情報を取り込むタイミングの生成に利用するようにしたことを特徴とする個人識別装置。 A control unit that captures the user's face information by camera and compares it with registered data to determine the identity, a fingerprint verification unit that captures the user's fingerprint and matches the registered data to determine the identity, and an appropriate face image When the user faces the camera at a position where the image can be taken, a vertically long mirror that is installed in a position where the user can check his / her face and which is curved in the vertical direction, the operation information collated by the fingerprint collation unit Is used for generating timing for capturing user face information. 上記登録データとして、利用者毎に顔の高さ方向の位置情報を記憶しておき、指紋照合の結果得た個人情報から利用者の顔の高さ方向の位置情報を得て、顔情報の識別に利用することを特徴とする請求項1に記載の個人識別装置。 As the registration data, the position information in the height direction of the face is stored for each user, the position information in the height direction of the user's face is obtained from the personal information obtained as a result of fingerprint matching, and the face information The personal identification device according to claim 1, wherein the personal identification device is used for identification.
JP2003326347A 2003-09-18 2003-09-18 Personal identification device Expired - Fee Related JP4369187B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2003326347A JP4369187B2 (en) 2003-09-18 2003-09-18 Personal identification device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2003326347A JP4369187B2 (en) 2003-09-18 2003-09-18 Personal identification device

Publications (3)

Publication Number Publication Date
JP2005092611A JP2005092611A (en) 2005-04-07
JP2005092611A5 JP2005092611A5 (en) 2006-08-10
JP4369187B2 true JP4369187B2 (en) 2009-11-18

Family

ID=34456561

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2003326347A Expired - Fee Related JP4369187B2 (en) 2003-09-18 2003-09-18 Personal identification device

Country Status (1)

Country Link
JP (1) JP4369187B2 (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007004613A (en) * 2005-06-24 2007-01-11 Glory Ltd Face authentication device
JP6787391B2 (en) 2016-02-26 2020-11-18 日本電気株式会社 Face matching system, face matching method, and program
JP6621383B2 (en) * 2016-07-12 2019-12-18 バイオニクス株式会社 Personal authentication device
JP6895802B2 (en) * 2017-05-19 2021-06-30 新明和工業株式会社 Garbage truck and person recognition device equipped on it
JP6583460B2 (en) * 2018-03-23 2019-10-02 株式会社セガゲームス Authentication system
CN108846924A (en) * 2018-05-31 2018-11-20 上海商汤智能科技有限公司 Vehicle and car door solution lock control method, device and car door system for unlocking
US11383676B2 (en) 2018-05-31 2022-07-12 Shanghai Sensetime Intelligent Technology Co., Ltd Vehicles, vehicle door unlocking control methods and apparatuses, and vehicle door unlocking systems
JP7238617B2 (en) * 2019-06-17 2023-03-14 大日本印刷株式会社 Service providing device, service providing system, program, and service providing method

Also Published As

Publication number Publication date
JP2005092611A (en) 2005-04-07

Similar Documents

Publication Publication Date Title
US8494286B2 (en) Face detection in mid-shot digital images
US6882741B2 (en) Facial image recognition apparatus
US8831295B2 (en) Electronic device configured to apply facial recognition based upon reflected infrared illumination and related methods
KR101005974B1 (en) Photographed body authenticating device, face authenticating device, portable telephone, photographed body authenticating unit, photographed body authenticating method and photographed body authenticating program
JP2012113687A (en) Method of authenticating driver's face in vehicle
US20160026781A1 (en) Ear biometric capture, authentication, and identification method and system
EP1628239A3 (en) Display control by means of fingerprint recognition and tracking
EP1605412A2 (en) Multi-identification method and multi-identification apparatus
US20080212849A1 (en) Method and Apparatus For Facial Image Acquisition and Recognition
JP2002083298A5 (en)
JP2004118627A (en) Figure identification device and method
WO2006088042A1 (en) Biometric discrimination device, authentication device, and biometric discrimination method
JP2001338296A (en) Face image recognizing device and passing through controller
JP2000083930A (en) Personal identifying device using iris
JP4369187B2 (en) Personal identification device
JP2000237167A (en) Iris recognizing device
JP2002153445A (en) Iris recognition device
US20080080751A1 (en) Method of capturing fingerprint image
JP2004078791A5 (en)
KR20200080533A (en) Apparatus and Method for Recognizing Fake Face By Using Minutia Data Variation
KR100847142B1 (en) Preprocessing method for face recognition, face recognition method and apparatus using the same
JP2004280277A (en) Image pickup device and face authenticating device
JP2004157771A (en) Person identification device, person identification method, and passage controller
Schneider et al. Feature based face localization and recognition on mobile devices
KR102176882B1 (en) Apparatus and Method for Recognizing Fake Iris By Using Minutia Data Variation

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20060623

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20060623

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20090428

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20090601

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20090818

A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20090827

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20120904

Year of fee payment: 3

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20130904

Year of fee payment: 4

LAPS Cancellation because of no payment of annual fees