JP4580122B2 - Detecting foreign matter in liquid - Google Patents

Detecting foreign matter in liquid Download PDF

Info

Publication number
JP4580122B2
JP4580122B2 JP2001151267A JP2001151267A JP4580122B2 JP 4580122 B2 JP4580122 B2 JP 4580122B2 JP 2001151267 A JP2001151267 A JP 2001151267A JP 2001151267 A JP2001151267 A JP 2001151267A JP 4580122 B2 JP4580122 B2 JP 4580122B2
Authority
JP
Japan
Prior art keywords
liquid
image data
light
foreign matter
bubbles
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP2001151267A
Other languages
Japanese (ja)
Other versions
JP2002340808A (en
Inventor
篤 尾原
喜文 須田
政克 鎌田
純 市川
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Daiichi Sankyo Co Ltd
Original Assignee
Daiichi Sankyo Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Daiichi Sankyo Co Ltd filed Critical Daiichi Sankyo Co Ltd
Priority to JP2001151267A priority Critical patent/JP4580122B2/en
Publication of JP2002340808A publication Critical patent/JP2002340808A/en
Application granted granted Critical
Publication of JP4580122B2 publication Critical patent/JP4580122B2/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G01MEASURING; TESTING
    • G01NINVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84Systems specially adapted for particular applications
    • G01N21/88Investigating the presence of flaws or contamination
    • G01N21/90Investigating the presence of flaws or contamination in a container or its contents
    • G01N21/9018Dirt detection in containers
    • G01N21/9027Dirt detection in containers in containers after filling

Landscapes

  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Biochemistry (AREA)
  • General Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Immunology (AREA)
  • Pathology (AREA)
  • Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Description

【0001】
【発明の属する技術分野】
本発明は、液中に混入した異物を検出する方法に関する。
【0002】
【従来の技術】
液状の医薬品が注入されたガラス瓶やプラスチック容器等の製品は、クリーンルームで製造される。したがって、製品中に異物が混入することは殆どないが、ごく稀に作業員の毛髪、製造装置等の金属片や繊維が混入する場合がある。そして、これらの異物は製品品質上問題を生じるので、異物が発見されたときには、これを排除する処理が行われており、近年では画像処理装置を使用した自動検査が導入されている。
【0003】
そして、画像処理装置を使用した異物の検出は、一般に、液体の撮像データを2値化して連結成分を抽出し、所定サイズ以上の連結成分が存在するか否かを判定することによって行われる。そのため、液中に気泡が存在し、2値化したときに該気泡に応じて前記所定サイズ以上の連結成分が抽出されたときには、該気泡を異物と誤認して、良品を不良品と判定するいわゆる良品錯誤が生じてしまう。
【0004】
そこで、異物の検査を行なう前に、液体が注入された容器を回転させて気泡を除去する処理等が行なわれているが、この処理では大きな気泡は除去されるが異物と同程度の小さな気泡を除去することが難しかった。
【0005】
そのため、検査工程において、液中の気泡を異物と誤認する良品錯誤が生じ、生産効率が低下する場合があった。
【0006】
【発明が解決しようとする課題】
本発明は、上記背景を鑑みてなされたものであり、液中の気泡と異物を区別して、異物のみを検出することができる液中異物の検出方法を提供することを目的とする。
【0007】
【課題を解決するための手段】
本発明は上記目的を達成するためになされたものであり、照明手段により光透過性の容器に注入された光透過性の液体に光を照射して、該照明手段と該容器を挟んで対向する位置に設けられた撮像手段により前記液体を撮像し、画像処理手段により、前記撮像手段から出力される前記液体の原画像データを輝度が所定の閾値以上である第1レベルと該閾値未満である第2レベルに2値化して2値画像データを作成して、該2値画像データに所定サイズ以上の該第2レベルの連結成分が存在するか否かを確認することによって、前記液体中の異物を検出する液中異物の検出方法の改良に関する。
【0008】
本願発明者らは、上記目的を達成するために各種検討を重ねた結果、詳細は後述するが、前記光透過性の液体中に外周がほぼ等しい異物と気泡とが存在する場合に、前記照明手段の照度又は前記撮像手段への入光量を調節することによって、該気泡に応じて前記2値画像データに現れる前記第2レベルの連結成分のサイズを前記所定サイズよりも小さくし、且つ、該異物に応じて前記2値画像データに現れる前記第2レベルの連結成分のサイズを前記所定サイズ以上とすることができることを知見した。
【0009】
そこで、本発明の第1の態様は、前記容器中の液体への光の照射を、前記容器を挟んで前記撮像手段と対向した前記照明手段のみによって行い、前記照明手段の照度を調整することによって、前記照明手段から前記液体中の気泡を介して前記撮像手段に入光される光量を、前記照明手段から照射された光が該気泡の周囲から該気泡の中心方向に回り込んで該気泡の外周部分の輝度が前記閾値以上となり、前記2値画像データに該気泡に応じ前記所定サイズ以上の前記第2レベルの連結成分が現れなくなるように設定したことを特徴とする。
【0010】
かかる本発明によれば、前記照明手段の照度を、前記気泡に対して前記2値画像データに前記所定サイズ以上の前記第2レベルの連結成分が現れないように設定したときに、前記異物に対しては前記2値画像データに前記所定サイズ以上の前記第2レベルの連結成分が現れるようにすることができる。これにより、前記気泡が前記異物と誤認されることを防止することができる。
【0011】
また、本発明の第2の態様は、前記容器中の液体への光の照射を、前記容器を挟んで前記撮像手段と対向した前記照明手段のみによって行い、前記撮像手段の絞りを調節することによって、前記照明手段から前記液体中の気泡を介して前記撮像手段に入光される光量を、前記照明手段から照射された光が該気泡の周囲から該気泡の中心方向に回り込んで該気泡の外周部分の輝度が前記閾値以上となり、前記2値画像データに該気泡に応じ前記所定サイズ以上の前記第2レベルの連結成分が現れなくなるように設定したことを特徴とする。
【0012】
かかる本発明によれば、前記入光量を、前記気泡に対して前記2値画像データに前記所定サイズ以上の前記第2レベルの連結成分が現れないように設定したときに、前記第1の態様と同様に、前記異物に対しては前記2値画像データに前記所定サイズ以上の前記第2レベルの連結成分が現れるようにすることができる。
【0013】
また、本発明の第3の態様は、前記容器中の液体への光の照射を、前記容器を挟んで前記撮像手段と対向した前記照明手段のみによって行い、前記照明手段の照度と前記撮像手段の絞りを調節することによって、前記照明手段から前記液体中の気泡を介して前記撮像手段に入光される光量を、前記照明手段から照射された光が該気泡の周囲から該気泡の中心方向に回り込んで該気泡の外周部分の輝度が前記閾値以上となり、前記2値画像データに該気泡に応じ前記所定サイズ以上の前記第2レベルの連結成分が現れなくなるように設定したことを特徴とする。
【0014】
かかる本発明によれば、前記入光量と前記照明手段の照度とを、前記気泡に対して前記2値画像データに前記所定サイズ以上の前記第2レベルの連結成分が現れないように設定したときに、前記第1の態様及び前記第2の態様と同様に、前記異物に対しては前記2値画像データに前記所定サイズ以上の前記第2レベルの連結成分が現れるようにすることができる。
【0015】
また、上述した前記照明手段の照度の設定や前記撮像手段の入光量の設定を行なった場合、詳細は後述するが、前記2値画像データにおける気泡に応じた前記第2レベルの連結成分のサイズが減少すると共に、異物に応じた前記第2レベルの連結成分のサイズも減少する。
【0016】
そのため、前記液中に前記所定サイズよりも若干大きい異物が存在していたときに、このように該異物に応じた前記第2レベルの連結成分のサイズが減少して前記所定値未満となると、本来異物として検出されるべきものが検出されなくなってしまうという不都合がある。
【0017】
そして、かかる不都合を解消するべく検討を重ねた結果、本願発明者らは、前記照明手段が発光する光の波長を600nm以上とすることにより、前記原画像データにおける異物のコントラストが上がることを知見した。
【0018】
そこで、前記第1の態様から前記第3の態様において、前記照明手段から発光される光の波長を600nm以上として、前記原画像データにおける異物のコントラストを上げることによって、前記2値画像データにおける該異物に応じた前記第2レベルの連結成分のサイズの減少を抑制することができる。
【0019】
【発明の実施の形態】
本発明の実施の形態の一例を図1〜図6を参照して説明する。図1は本発明の液中異物の検出方法を使用する検査装置の全体構成図、図2は図1に示した検査装置による画像処理の説明図、図3は図1に示した液体中に存在する気泡の影響を排除する処理の説明図、図4は図1に示した液体中に存在する異物のコントラストを上げる処理の説明図、図5〜図6は図1に示した検査装置による液中異物の検査の実行フローチャートである。
【0020】
図1を参照して、本実施の形態の検査装置は、光透過性のシリンジ1(本発明の容器に相当する)に注入された光透過性の液体2内に混入した異物3の有無を検査するためのものであり、液体2に光を照射する照明器4(本発明の照明手段に相当する)、照明器4とシリンジ1を挟んで対向する位置に設けられた4台のカメラ5(5a〜5d,本発明の撮像手段に相当する)、カメラ5から出力される液体2の原画像データを入力して異物3の検出処理を行なう画像処理装置6(本発明の画像処理手段に相当する)を備える。
【0021】
また、シリンジ1は台7にセットされ、照明器4には波長600nm以上の光のみを透過させる赤色フィルタ8が取付けられ(この場合、照明器4と赤色フィルタ8とにより本発明の照明手段が構成される)、画像処理装置6にはモニタ9が接続されている。
【0022】
図2を参照して、画像処理装置6は、カメラ5から出力されるアナログの原画像データ20を入力して8ビットのA/D変換を行い、各画素の輝度が0〜255段階で表される多値画像データ21を得る。そして、多値画像データ21の各画素の輝度(V_sig)を所定の閾値(Th)と比較し、輝度(V_sig)が閾値(Th)以上であったときは該画素のデータを「1」(本発明の第1レベルに相当する)とし、輝度(V_sig)が閾値(Th)未満であったときには該画素のデータを「0」(本発明の第2レベルに相当する)とするいわゆる2値化処理を行なって(図中22)、2値画像データ23を作成する。
【0023】
ここで、シリンジ1内の液中に異物3が存在する場合、該異物3は照明器4からの光を透過しないので輝度(V_sig)が低くなるため、上述した2値化処理を行なうと該異物3に応じた画素は「0」となる。そのため、前記「1」を「白」、前記「0」を「黒」として表示すると、異物3に応じた画素部分は「黒」で表示される(図中25)。
【0024】
そして、画像処理装置6は、2値画像データ23から、「0」の画素が互いに連結した一つの塊である連結成分(以下、単に「連結成分」という)を、2値画像データ23から抽出し、抽出した各連結成分にラベリングを行なう。本実施の形態では、ラベリングを行なった連結成分のサイズ(水平方向の長さ、垂直成分の長さ、面積)を配列L[i](i=1,2,・・・)に収容する。
【0025】
このように、2値化処理を行なうことによって、液中の異物3に応じた連結成分25を抽出することができる。そして、画像処理装置6は、異物3に応じた連結成分25のサイズが、異物を検知するための所定の基準値(本発明の所定サイズに相当する)を超えていたときに、液中に異物が混入していると検知することができる。
【0026】
しかし、液中に気泡が存在する場合には、気泡の外周部分は光を透過せずに輝度が低くなるため、2値化処理を行なったときに、該気泡に応じた連結成分も抽出される。
【0027】
そして、気泡に応じた連結成分のサイズが前記基準値を超えると、画像処理装置6は該気泡を異物と誤検知してしまう。そこで、本実施の形態では、このような誤検知がなされることを防止するため、照明器4の照度と、カメラ5の絞りを調整(本発明の入光量の調節に相当する)して、気泡に応じた前記基準値以下のサイズを有する連結成分が抽出されないようにしている。以下、図3〜図4を参照して、この照明器4の照度とカメラ5の絞りの調整について説明する。
【0028】
図3(a)は、液中に気泡31と異物32が存在する場合の多値画像データ30のモニタ9(図1参照)への表示画面を示している。この場合、多値画像データ30において気泡31を消して異物32だけを残すようにすれば、2値化処理を行なったときに異物32に応じた連結成分のみを抽出することができる。
【0029】
そこで、本実施の形態では、照明器4の照度とカメラ5の絞りを調整して、照明器4から気泡31を介してカメラ5に入光される光量を、図3(b)に示したように、照明器4から照射された光35が気泡31の周囲から該気泡31の中心方向に回り込んで気泡31の外周部分の輝度が増加し、2値化処理を行なったときに気泡31の外周部分が「1」レベルとなるように設定する。
【0030】
このように、気泡31の外周部分が「1」となるようにカメラ5に入光される光量を設定した場合、気泡の内側部分は光を透過しているため輝度が高く「1」レベルであるので、気泡31に応じた「0」レベルの連結成分は抽出されないことになる。一方、異物32の周囲から該異物32の中心方向にも照明器4から照射された光36が回り込むが、異物32の場合は内側部分が光を遮断していて輝度が低いので、2値化処理を行なったときに内側部分は「0」となる。そのため、異物32に応じた連結成分のサイズは多少減少するが、完全に消失することはない。
【0031】
そのため、画像処理装置6は、気泡31の影響を排除して、2値画像データから異物32に応じた連結成分のみを抽出することができる。なお、気泡31に応じた「0」レベルの連結成分が完全に消えなかった場合であっても、2値化処理を行なって抽出された気泡31の残存部分に応じた連結成分のサイズが前記基準値未満であれば、異物32と区別して排除することができる。
【0032】
このように、照明器4の照度とカメラ5の絞りを調節してカメラ5への入光量を適切に設定する(以下、この設定処理を「光量設定処理」という)ことにより、気泡31の影響を排除して、異物32の有無を検知することができる。しかし、異物32の大きさが前記基準値よりも若干大きい程度のときには、光量設定処理により該異物の外周部の輝度が高くなって、2値化処理を行なったときに抽出される該異物に応じた「0」レベルの連結成分のサイズが前記基準値よりも小さくなってしまうことがあり得る。そしてこの場合には、該異物32が検出されなくなってしまう。
【0033】
そこで、本実施の形態では、照明器4の前面に波長600nm以上の光のみを透過させる赤色フィルタ8を設けている。このように、赤色フィルタ8を設けて液体2に照射する光の波長を制限することにより、図4(a)に示したように、上述した光量設定処理により周囲が減少した異物32のコントラストを上げ、図4(b)に示したように、異物32の外周部分の減少を抑制することができる。
そして、これにより、2値化処理により抽出される異物32に応じた連結成分のサイズが減少することを抑制することができる。
【0034】
次に、図5〜図6に示したフローチャートに従って、本実施の形態における画像処理装置6による液中異物の有無の検査の実行手順について説明する。
【0035】
画像処理装置6は、図5のSTEP1で4台のカメラ5(5a〜5d,図1参照)から出力される液体2の画像を取り込み、STEP2で、上述したA/D変換により生成された多値画像データ21(図2参照)に対して輪郭強調、ノイズ除去等の前処理を行なう。
【0036】
続くSTEP3で、画像処理装置6は、多値画像データ21(図2参照)に2値化処理を施して2値画像データ23(図2参照)を生成し、STEP4で、2値画像データ23に存在する連結成分を抽出してラベリングを行なう。
【0037】
STEP5以降は、抽出された連結成分の中に異物に該当するものがあるか否かを検出するための処理であり、画像処理装置6は、先ず、抽出された連結成分の個数をSTEP5で変数nにセットし、STEP6でカウンタ変数CNTをクリアし、STEP7でループ変数iに1をセットする。
【0038】
そして、画像処理装置6は、続く図6のSTEP8〜STEP10のループにより、ループ変数iをカウントアップしながら(STEP30)、各連結成分L[i](i=1,2,・・・,n)に対して、STEP8で連結成分のサイズ(幅、高さ、面積)を測定し、STEP9で測定したサイズが基準値(例えば、幅2以下 or 高さ2画素以下or 面積3画素以下)を満たしているか否かを判定する。
【0039】
連結成分L[i]のサイズが前記基準値を満たしていないとき(NG)は、STEP9からSTEP20に分岐してカウンタ変数CNTがカウントアップされる。そして、STEP10でループ変数iの値が連結成分の個数nに達したときに、ループを抜けてSTEP11に進む。
【0040】
画像処理装置6は、STEP11でカウンタ変数CNTの値が所定の異物の許容個数を超えていたときはSTEP40に分岐してモニタ9(図1参照)に「検査NG」を表示し、カウンタ変数CNTの値が該許容個数以下であったときにはSTEP12に進んでモニタ9に「検査OK」を表示して検査を終了する。
【0041】
なお、本実施の形態では、照明器4の照度とカメラ5の絞りを調節することによって前記光量設定処理を行なったが、いずれか一方のみを調節することにより前記光量設定処理を行なってもよい。
【図面の簡単な説明】
【図1】本発明の液中異物の検出方法を使用する検査装置の全体構成図。
【図2】図1に示した画像処理装置による画像処理の説明図。
【図3】気泡が消える様子を示した説明図。
【図4】異物のコントラストが上がる様子を示した説明図。
【図5】液中異物の検査の実行フローチャート。
【図6】液中異物の検査の実行フローチャート。
【符号の説明】
1…シリンジ、2…液体、3…異物、4…照明器、5…カメラ、6…画像処理装置、7…台、8…赤色フィルタ、9…モニタ
[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method for detecting foreign matter mixed in a liquid.
[0002]
[Prior art]
Products such as glass bottles and plastic containers filled with liquid pharmaceuticals are manufactured in a clean room. Therefore, foreign matters are hardly mixed in the product, but very rarely metal pieces and fibers such as a worker's hair and a manufacturing apparatus may be mixed. Since these foreign matters cause problems in product quality, when a foreign matter is found, a process for eliminating the foreign matter is performed. In recent years, automatic inspection using an image processing apparatus has been introduced.
[0003]
In general, detection of a foreign object using an image processing apparatus is performed by binarizing liquid imaging data to extract a connected component, and determining whether a connected component having a predetermined size or larger is present. For this reason, when bubbles are present in the liquid and when a binarized binarized component having a size larger than the predetermined size is extracted, the bubbles are misidentified as foreign matters and a non-defective product is determined as a defective product. A so-called non-defective product error will occur.
[0004]
Therefore, before inspecting the foreign matter, a process of removing the bubbles by rotating the container into which the liquid has been injected is performed. In this process, the large bubbles are removed, but the small bubbles are as small as the foreign matters. It was difficult to remove.
[0005]
For this reason, in the inspection process, a non-defective product error that misrecognizes bubbles in the liquid as a foreign substance may occur, resulting in a decrease in production efficiency.
[0006]
[Problems to be solved by the invention]
The present invention has been made in view of the above background, and an object of the present invention is to provide a detection method of foreign matter in liquid that can detect only foreign matters by distinguishing bubbles and foreign matters in the liquid.
[0007]
[Means for Solving the Problems]
The present invention has been made to achieve the above object, and irradiates a light-transmitting liquid injected into a light-transmitting container by an illuminating means so as to face the illuminating means with the container interposed therebetween. The liquid is imaged by an imaging means provided at a position where the original image data of the liquid output from the imaging means is output by the image processing means at a first level where the luminance is equal to or higher than a predetermined threshold and below the threshold. A binary image data is generated by binarizing to a certain second level, and whether or not the second level connected component having a predetermined size or more exists in the binary image data is determined. It is related with the improvement of the detection method of the foreign material in a liquid which detects this foreign material.
[0008]
As a result of various investigations to achieve the above object, the inventors of the present application will be described in detail later. However, when the foreign substance and bubbles having substantially the same outer periphery exist in the light-transmitting liquid, the illumination is performed. By adjusting the illuminance of the means or the amount of incident light to the imaging means, the size of the second level connected component appearing in the binary image data in accordance with the bubble is made smaller than the predetermined size, and the It has been found that the size of the connected component of the second level appearing in the binary image data in accordance with a foreign object can be equal to or larger than the predetermined size.
[0009]
Therefore, in the first aspect of the present invention, light irradiation to the liquid in the container is performed only by the illumination unit facing the imaging unit with the container interposed therebetween, and the illuminance of the illumination unit is adjusted. Therefore, the amount of light incident on the imaging unit from the illumination unit via the bubbles in the liquid is changed so that the light irradiated from the illumination unit circulates from the periphery of the bubble toward the center of the bubble. brightness of the peripheral portion becomes equal to or larger than the threshold value, characterized in that the coupling component of the predetermined size or more of the second level corresponding to the bubbles in the binary image data is set to appear such Kunar so.
[0010]
According to the present invention, when the illuminance of the illuminating unit is set so that the second level connected component of the predetermined size or more does not appear in the binary image data with respect to the bubble, On the other hand, the second-level connected component having the predetermined size or more may appear in the binary image data. Thereby, it can prevent that the said bubble is misidentified as the said foreign material.
[0011]
In the second aspect of the present invention, the liquid in the container is irradiated with light only by the illumination unit facing the imaging unit with the container interposed therebetween, and the diaphragm of the imaging unit is adjusted. Therefore, the amount of light incident on the imaging unit from the illumination unit via the bubbles in the liquid is changed so that the light irradiated from the illumination unit circulates from the periphery of the bubble toward the center of the bubble. brightness of the peripheral portion becomes equal to or larger than the threshold value, characterized in that the coupling component of the predetermined size or more of the second level corresponding to the bubbles in the binary image data is set to appear such Kunar so.
[0012]
According to the present invention, when the incident light amount is set so that the second-level connected component of the predetermined size or more does not appear in the binary image data with respect to the bubbles, the first aspect. Similarly, the second level connected component of the predetermined size or more can appear in the binary image data for the foreign matter.
[0013]
In the third aspect of the present invention, the liquid in the container is irradiated with light only by the illumination means facing the imaging means with the container interposed therebetween, and the illumination intensity of the illumination means and the imaging means By adjusting the diaphragm, the amount of light incident on the imaging means from the illumination means via the bubbles in the liquid is changed so that the light emitted from the illumination means extends from the periphery of the bubbles to the center of the bubbles. wraps around the brightness of the peripheral portion of the bubble becomes equal to or larger than the threshold value is, the connected component of the predetermined size or more of the second level corresponding to the bubbles in the binary image data is set to appear such Kunar so on It is characterized by.
[0014]
According to the present invention, when the incident light amount and the illuminance of the illuminating unit are set so that the second level connected component of the predetermined size or more does not appear in the binary image data with respect to the bubbles. In addition, similarly to the first aspect and the second aspect, the second level connected component of the predetermined size or more may appear in the binary image data with respect to the foreign matter.
[0015]
Further, when the above-described illuminance setting of the illuminating unit and the incident light amount of the imaging unit are set, the size of the second-level connected component corresponding to the bubble in the binary image data will be described in detail later. And the size of the second level connected component corresponding to the foreign object is also reduced.
[0016]
Therefore, when a foreign substance slightly larger than the predetermined size is present in the liquid, the size of the second level connected component corresponding to the foreign substance is reduced and becomes less than the predetermined value. There is an inconvenience that an object that should originally be detected as a foreign object is not detected.
[0017]
As a result of repeated studies to eliminate such inconvenience, the present inventors have found that the contrast of foreign matter in the original image data is increased by setting the wavelength of light emitted by the illumination means to 600 nm or more. did.
[0018]
Therefore, in the first aspect to the third aspect, the wavelength of the light emitted from the illumination unit is set to 600 nm or more, and the contrast of the foreign matter in the original image data is increased, whereby the binary image data It is possible to suppress a decrease in the size of the second level connected component according to the foreign matter.
[0019]
DETAILED DESCRIPTION OF THE INVENTION
An example of an embodiment of the present invention will be described with reference to FIGS. FIG. 1 is an overall configuration diagram of an inspection apparatus that uses the foreign substance detection method of the present invention, FIG. 2 is an explanatory diagram of image processing by the inspection apparatus shown in FIG. 1, and FIG. 3 is in the liquid shown in FIG. FIG. 4 is an explanatory view of a process for eliminating the influence of bubbles present, FIG. 4 is an explanatory view of a process for increasing the contrast of a foreign substance present in the liquid shown in FIG. 1, and FIGS. 5 to 6 are based on the inspection apparatus shown in FIG. It is an execution flowchart of a test | inspection of the foreign material in a liquid.
[0020]
Referring to FIG. 1, the inspection apparatus according to the present embodiment determines whether or not there is a foreign substance 3 mixed in a light transmissive liquid 2 injected into a light transmissive syringe 1 (corresponding to the container of the present invention). An illuminator 4 for irradiating the liquid 2 with light (corresponding to the illuminating means of the present invention) for inspection, and four cameras 5 provided at positions facing the illuminator 4 and the syringe 1. (5a to 5d, corresponding to the image pickup means of the present invention), an image processing apparatus 6 for inputting the original image data of the liquid 2 output from the camera 5 and detecting the foreign matter 3 (in the image processing means of the present invention) Equivalent).
[0021]
Further, the syringe 1 is set on a table 7 and a red filter 8 that transmits only light having a wavelength of 600 nm or more is attached to the illuminator 4 (in this case, the illuminator 4 and the red filter 8 allow the illumination means of the present invention to be used). A monitor 9 is connected to the image processing apparatus 6.
[0022]
Referring to FIG. 2, image processing apparatus 6 receives analog original image data 20 output from camera 5 and performs 8-bit A / D conversion, and the brightness of each pixel is expressed in 0 to 255 levels. Multivalued image data 21 is obtained. Then, the luminance (V_sig) of each pixel of the multi-valued image data 21 is compared with a predetermined threshold (Th). If the luminance (V_sig) is equal to or higher than the threshold (Th), the data of the pixel is set to “1” ( A so-called binary value in which the pixel data is “0” (corresponding to the second level of the present invention) when the luminance (V_sig) is less than the threshold (Th). Binary processing is performed (22 in the figure) to create binary image data 23.
[0023]
Here, when the foreign matter 3 is present in the liquid in the syringe 1, since the foreign matter 3 does not transmit light from the illuminator 4, the luminance (V_sig) is lowered. The pixel corresponding to the foreign object 3 is “0”. Therefore, when “1” is displayed as “white” and “0” is displayed as “black”, the pixel portion corresponding to the foreign material 3 is displayed as “black” (25 in the figure).
[0024]
Then, the image processing apparatus 6 extracts from the binary image data 23 a connected component (hereinafter simply referred to as “connected component”) that is a single block in which “0” pixels are connected to each other from the binary image data 23. Then, the extracted connected components are labeled. In the present embodiment, the size (horizontal direction length, vertical component length, area) of the connected components subjected to labeling is accommodated in the array L [i] (i = 1, 2,...).
[0025]
Thus, by performing the binarization process, the connected component 25 corresponding to the foreign matter 3 in the liquid can be extracted. Then, when the size of the connected component 25 corresponding to the foreign material 3 exceeds a predetermined reference value (corresponding to the predetermined size of the present invention) for detecting the foreign material, the image processing device 6 It can be detected that foreign matter is mixed.
[0026]
However, when bubbles are present in the liquid, the outer peripheral portion of the bubbles does not transmit light and the brightness is lowered. Therefore, when binarization processing is performed, a connected component corresponding to the bubbles is also extracted. The
[0027]
When the size of the connected component corresponding to the bubble exceeds the reference value, the image processing apparatus 6 erroneously detects the bubble as a foreign object. Therefore, in this embodiment, in order to prevent such erroneous detection, the illuminance of the illuminator 4 and the aperture of the camera 5 are adjusted (corresponding to the adjustment of the incident light amount of the present invention), A connected component having a size equal to or smaller than the reference value corresponding to the bubbles is prevented from being extracted. Hereinafter, adjustment of the illuminance of the illuminator 4 and the diaphragm of the camera 5 will be described with reference to FIGS.
[0028]
FIG. 3A shows a display screen of the multi-valued image data 30 on the monitor 9 (see FIG. 1) when bubbles 31 and foreign matter 32 are present in the liquid. In this case, if the bubbles 31 are deleted in the multi-value image data 30 to leave only the foreign matter 32, only the connected component corresponding to the foreign matter 32 can be extracted when the binarization process is performed.
[0029]
Therefore, in the present embodiment, the illuminance of the illuminator 4 and the diaphragm of the camera 5 are adjusted, and the amount of light incident on the camera 5 from the illuminator 4 via the bubbles 31 is shown in FIG. As described above, the light 35 irradiated from the illuminator 4 circulates from the periphery of the bubble 31 toward the center of the bubble 31 to increase the luminance of the outer peripheral portion of the bubble 31, and the bubble 31 is subjected to the binarization process. Is set to be “1” level.
[0030]
In this way, when the amount of light incident on the camera 5 is set so that the outer peripheral portion of the bubble 31 is “1”, the inner portion of the bubble transmits light and the luminance is high and the “1” level. Therefore, the connected component of “0” level corresponding to the bubble 31 is not extracted. On the other hand, the light 36 irradiated from the illuminator 4 also travels around the foreign object 32 in the center direction of the foreign object 32. However, in the case of the foreign object 32, the inner part blocks light and the luminance is low, so binarization is performed. When the processing is performed, the inner portion becomes “0”. For this reason, the size of the connected component corresponding to the foreign matter 32 is somewhat reduced, but is not completely lost.
[0031]
Therefore, the image processing apparatus 6 can extract only the connected component corresponding to the foreign matter 32 from the binary image data by eliminating the influence of the bubbles 31. Even when the connected component of “0” level corresponding to the bubble 31 has not completely disappeared, the size of the connected component corresponding to the remaining portion of the bubble 31 extracted by performing the binarization process is If it is less than the reference value, it can be distinguished from the foreign matter 32 and excluded.
[0032]
In this way, by adjusting the illuminance of the illuminator 4 and the aperture of the camera 5 to appropriately set the amount of light incident on the camera 5 (hereinafter, this setting process is referred to as “light quantity setting process”), the influence of the bubbles 31 is affected. The presence or absence of the foreign matter 32 can be detected. However, when the size of the foreign matter 32 is slightly larger than the reference value, the brightness of the outer peripheral portion of the foreign matter is increased by the light amount setting process, and the foreign matter extracted when the binarization process is performed. The size of the corresponding connected component of “0” level may be smaller than the reference value. In this case, the foreign matter 32 is not detected.
[0033]
Therefore, in the present embodiment, a red filter 8 that transmits only light having a wavelength of 600 nm or more is provided on the front surface of the illuminator 4. In this way, by providing the red filter 8 and limiting the wavelength of the light irradiated to the liquid 2, as shown in FIG. 4A, the contrast of the foreign matter 32 whose periphery has been reduced by the light amount setting process described above can be obtained. As shown in FIG. 4B, the decrease in the outer peripheral portion of the foreign matter 32 can be suppressed.
And thereby, it can suppress that the size of the connection component according to the foreign material 32 extracted by a binarization process reduces.
[0034]
Next, according to the flowcharts shown in FIGS. 5 to 6, the execution procedure of the presence / absence of foreign matter in the liquid by the image processing apparatus 6 in the present embodiment will be described.
[0035]
The image processing apparatus 6 captures the images of the liquid 2 output from the four cameras 5 (5a to 5d, see FIG. 1) in STEP1 of FIG. 5, and in STEP2, the multiple images generated by the A / D conversion described above. Preprocessing such as edge enhancement and noise removal is performed on the value image data 21 (see FIG. 2).
[0036]
In subsequent STEP 3, the image processing device 6 performs binarization processing on the multi-valued image data 21 (see FIG. 2) to generate binary image data 23 (see FIG. 2). In STEP 4, the binary image data 23 The connected components existing in are extracted and labeled.
[0037]
Step 5 and subsequent steps are processing for detecting whether or not there is any extracted connected component corresponding to a foreign substance. First, the image processing apparatus 6 sets the number of extracted connected components as a variable in STEP 5. n is set, the counter variable CNT is cleared in STEP6, and the loop variable i is set to 1 in STEP7.
[0038]
Then, the image processing apparatus 6 counts each connected component L [i] (i = 1, 2,..., N) while counting up the loop variable i by the loop of STEP 8 to STEP 10 in FIG. ), The size (width, height, area) of the connected component is measured in STEP 8, and the size measured in STEP 9 is a reference value (for example, width 2 or less or height 2 pixels or less or area 3 pixels or less). It is determined whether it is satisfied.
[0039]
When the size of the connected component L [i] does not satisfy the reference value (NG), the process branches from STEP 9 to STEP 20, and the counter variable CNT is counted up. When the value of the loop variable i reaches the number n of connected components in STEP 10, the process exits the loop and proceeds to STEP 11.
[0040]
When the value of the counter variable CNT exceeds the predetermined allowable number of foreign substances in STEP 11, the image processing apparatus 6 branches to STEP 40 and displays “inspection NG” on the monitor 9 (see FIG. 1). If the value is equal to or less than the permissible number, the process proceeds to STEP 12 to display “inspection OK” on the monitor 9 and terminate the inspection.
[0041]
In the present embodiment, the light amount setting process is performed by adjusting the illuminance of the illuminator 4 and the diaphragm of the camera 5, but the light amount setting process may be performed by adjusting only one of them. .
[Brief description of the drawings]
FIG. 1 is an overall configuration diagram of an inspection apparatus that uses a method for detecting foreign matter in liquid according to the present invention.
FIG. 2 is an explanatory diagram of image processing by the image processing apparatus shown in FIG.
FIG. 3 is an explanatory diagram showing how bubbles disappear.
FIG. 4 is an explanatory diagram showing how the contrast of a foreign object increases.
FIG. 5 is an execution flowchart of inspection of foreign matter in liquid.
FIG. 6 is an execution flowchart of inspection of foreign matter in liquid.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Syringe, 2 ... Liquid, 3 ... Foreign material, 4 ... Illuminator, 5 ... Camera, 6 ... Image processing apparatus, 7 ... Stand, 8 ... Red filter, 9 ... Monitor

Claims (4)

照明手段により光透過性の容器に注入された光透過性の液体に光を照射して、該照明手段と該容器を挟んで対向する位置に設けられた撮像手段により前記液体を撮像し、
画像処理手段により、前記撮像手段から出力される前記液体の原画像データを輝度が所定の閾値以上である第1レベルと該閾値未満である第2レベルに2値化して2値画像データを作成して、該2値画像データに所定サイズ以上の該第2レベルの連結成分が存在するか否かを確認することによって、前記液体中の異物を検出する液中異物の検出方法において、
前記容器中の液体への光の照射を、前記容器を挟んで前記撮像手段と対向した前記照明手段のみによって行い、
前記照明手段の照度を調整することによって、前記照明手段から前記液体中の気泡を介して前記撮像手段に入光される光量を、前記照明手段から照射された光が該気泡の周囲から該気泡の中心方向に回り込んで該気泡の外周部分の輝度が前記閾値以上となり、前記2値画像データに該気泡に応じ前記所定サイズ以上の前記第2レベルの連結成分が現れなくなるように設定したことを特徴とする液中異物の検出方法。
Light is applied to the light transmissive liquid injected into the light transmissive container by the illuminating means, and the liquid is imaged by an imaging means provided at a position facing the illuminating means and the container,
The image processing means binarizes the original image data of the liquid output from the imaging means into a first level where the luminance is equal to or higher than a predetermined threshold value and a second level where the luminance is lower than the threshold value, thereby generating binary image data Then, in the liquid foreign matter detection method for detecting the foreign matter in the liquid by checking whether or not the second level connected component having a predetermined size or more exists in the binary image data,
Irradiation of light to the liquid in the container is performed only by the illumination unit facing the imaging unit with the container interposed therebetween,
By adjusting the illuminance of the illuminating means, the amount of light incident on the imaging means from the illuminating means via the bubbles in the liquid is changed so that the light emitted from the illuminating means is emitted from the periphery of the bubbles to the bubbles. in the wraps around toward the center luminance of the peripheral portion of the bubble becomes equal to or larger than the threshold value, the binary image data into connected components of said predetermined size or more of the second level corresponding to the bubbles appear such Kunar so A method for detecting foreign matter in liquid, characterized by being set.
照明手段により光透過性の容器に注入された光透過性の液体に光を照射して、該照明手段と該容器を挟んで対向する位置に設けられた撮像手段により前記液体を撮像し、
画像処理手段により、前記撮像手段から出力される前記液体の原画像データを輝度が所定の閾値以上である第1レベルと該閾値未満である第2レベルに2値化して2値画像データを作成して、該2値画像データに所定サイズ以上の該第2レベルの連結成分が存在するか否かを確認することによって、前記液体中の異物を検出する液中異物の検出方法において、
前記容器中の液体への光の照射を、前記容器を挟んで前記撮像手段と対向した前記照明手段のみによって行い、
前記撮像手段の絞りを調節することによって、前記照明手段から前記液体中の気泡を介して前記撮像手段に入光される光量を、前記照明手段から照射された光が該気泡の周囲から該気泡の中心方向に回り込んで該気泡の外周部分の輝度が前記閾値以上となり、前記2値画像データに該気泡に応じ前記所定サイズ以上の前記第2レベルの連結成分が現れなくなるように設定したことを特徴とする液中異物の検出方法。
Light is applied to the light transmissive liquid injected into the light transmissive container by the illuminating means, and the liquid is imaged by an imaging means provided at a position facing the illuminating means and the container,
The image processing means binarizes the original image data of the liquid output from the imaging means into a first level where the luminance is equal to or higher than a predetermined threshold value and a second level where the luminance is lower than the threshold value, thereby generating binary image data Then, in the liquid foreign matter detection method for detecting the foreign matter in the liquid by checking whether or not the second level connected component having a predetermined size or more exists in the binary image data,
Irradiation of light to the liquid in the container is performed only by the illumination unit facing the imaging unit with the container interposed therebetween,
By adjusting the diaphragm of the imaging means, the amount of light incident on the imaging means from the illumination means via the bubbles in the liquid is changed so that the light emitted from the illumination means is emitted from the periphery of the bubbles to the bubbles. in the wraps around toward the center luminance of the peripheral portion of the bubble becomes equal to or larger than the threshold value, the binary image data into connected components of said predetermined size or more of the second level corresponding to the bubbles appear such Kunar so A method for detecting foreign matter in liquid, characterized by being set.
照明手段により光透過性の容器に注入された光透過性の液体に光を照射して、該照明手段と該容器を挟んで対向する位置に設けられた撮像手段により前記液体を撮像し、
画像処理手段により、前記撮像手段から出力される前記液体の原画像データを輝度が所定の閾値以上である第1レベルと該閾値未満である第2レベルに2値化して2値画像データを作成して、該2値画像データに所定サイズ以上の該第2レベルの連結成分が存在するか否かを確認することによって、前記液体中の異物を検出する液中異物の検出方法において、
前記容器中の液体への光の照射を、前記容器を挟んで前記撮像手段と対向した前記照明手段のみによって行い、
前記照明手段の照度と前記撮像手段の絞りを調節することによって、前記照明手段から前記液体中の気泡を介して前記撮像手段に入光される光量を、前記照明手段から照射された光が該気泡の周囲から該気泡の中心方向に回り込んで該気泡の外周部分の輝度が前記閾値以上となり、前記2値画像データに該気泡に応じ前記所定サイズ以上の前記第2レベルの連結成分が現れなくなるように設定したことを特徴とする液中異物の検出方法。
Light is applied to the light transmissive liquid injected into the light transmissive container by the illuminating means, and the liquid is imaged by an imaging means provided at a position facing the illuminating means and the container,
The image processing means binarizes the original image data of the liquid output from the imaging means into a first level where the luminance is equal to or higher than a predetermined threshold value and a second level where the luminance is lower than the threshold value, thereby generating binary image data Then, in the liquid foreign matter detection method for detecting the foreign matter in the liquid by checking whether or not the second level connected component having a predetermined size or more exists in the binary image data,
Irradiation of light to the liquid in the container is performed only by the illumination unit facing the imaging unit with the container interposed therebetween,
By adjusting the illuminance of the illuminating means and the diaphragm of the imaging means, the amount of light incident on the imaging means from the illuminating means via the bubbles in the liquid is changed by the light emitted from the illuminating means. The brightness of the outer periphery of the bubble that goes around from the periphery of the bubble toward the center of the bubble becomes equal to or greater than the threshold value, and the second level connected component of the predetermined size or more corresponding to the bubble is included in the binary image data. detection method of the foreign matter in the liquid, characterized in that appeared a set in Kunar so.
前記照明手段は、600nm以上の波長の光を発光することを特徴とする請求項1から請求項3のうちいずれか1項記載の液中異物の検出方法。  The method for detecting foreign matter in liquid according to any one of claims 1 to 3, wherein the illuminating means emits light having a wavelength of 600 nm or more.
JP2001151267A 2001-05-21 2001-05-21 Detecting foreign matter in liquid Expired - Fee Related JP4580122B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2001151267A JP4580122B2 (en) 2001-05-21 2001-05-21 Detecting foreign matter in liquid

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2001151267A JP4580122B2 (en) 2001-05-21 2001-05-21 Detecting foreign matter in liquid

Publications (2)

Publication Number Publication Date
JP2002340808A JP2002340808A (en) 2002-11-27
JP4580122B2 true JP4580122B2 (en) 2010-11-10

Family

ID=18996148

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2001151267A Expired - Fee Related JP4580122B2 (en) 2001-05-21 2001-05-21 Detecting foreign matter in liquid

Country Status (1)

Country Link
JP (1) JP4580122B2 (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4163039B2 (en) * 2003-04-21 2008-10-08 株式会社エム・アイ・エル In-container bubble determination method and apparatus
US7163035B2 (en) * 2004-05-13 2007-01-16 Forhealth Technologies, Inc. Automated use of a vision system to detect foreign matter in reconstituted drugs before transfer to a syringe
US7343943B2 (en) 2004-05-13 2008-03-18 Forhealth Technologies, Inc. Medication dose underfill detection system and application in an automated syringe preparing system
JP4536614B2 (en) 2005-07-19 2010-09-01 本田技研工業株式会社 Electronic key position information display device
WO2008051576A2 (en) 2006-10-24 2008-05-02 Mallinckrodt Inc. Optical inclusion sensor
TWI840888B (en) 2011-08-29 2024-05-01 美商安美基公司 Methods and apparati for nondestructive detection of undissolved particles in a fluid
JP6628185B2 (en) * 2016-03-11 2020-01-08 パナソニックIpマネジメント株式会社 Inspection method for transparent objects
US10088660B2 (en) 2017-02-10 2018-10-02 Amgen Inc. Imaging system for counting and sizing particles in fluid-filled vessels

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS6098341A (en) * 1983-11-04 1985-06-01 Eisai Co Ltd Method and device for detecting photodecomposable solid matter in solution
JPH01109485A (en) * 1987-10-22 1989-04-26 Fujisawa Pharmaceut Co Ltd Device for checking liquid mixing foreign matter
JPH01141342A (en) * 1987-11-27 1989-06-02 Hajime Sangyo Kk Bottle bottom inspection instrument
JPH0448251A (en) * 1990-06-18 1992-02-18 Asahi Chem Ind Co Ltd Bottle inspecting device
JPH0634575A (en) * 1992-07-21 1994-02-08 Asahi Chem Ind Co Ltd Bottle inspection method
JPH06505333A (en) * 1991-02-01 1994-06-16 ノボ ノルディスク アクティーゼルスカブ Method and apparatus for inspecting liquid-filled containers
JPH0821798A (en) * 1994-07-06 1996-01-23 Kanegafuchi Chem Ind Co Ltd Method and apparatus for detecting foreign matter
JPH08285789A (en) * 1995-04-18 1996-11-01 Nippon Sheet Glass Co Ltd Method for determining binarization level in defect detection method
JPH1019799A (en) * 1996-07-09 1998-01-23 Mutual Corp Inspection method of mixed foreign matter in vessel and inspection equipment
JPH11125604A (en) * 1997-10-22 1999-05-11 Ritsumeikan Method and device for detecting foreign matter
JP2001116703A (en) * 1999-10-21 2001-04-27 M I L:Kk Method and apparatus for discriminating flotage in container

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS6098341A (en) * 1983-11-04 1985-06-01 Eisai Co Ltd Method and device for detecting photodecomposable solid matter in solution
JPH01109485A (en) * 1987-10-22 1989-04-26 Fujisawa Pharmaceut Co Ltd Device for checking liquid mixing foreign matter
JPH01141342A (en) * 1987-11-27 1989-06-02 Hajime Sangyo Kk Bottle bottom inspection instrument
JPH0448251A (en) * 1990-06-18 1992-02-18 Asahi Chem Ind Co Ltd Bottle inspecting device
JPH06505333A (en) * 1991-02-01 1994-06-16 ノボ ノルディスク アクティーゼルスカブ Method and apparatus for inspecting liquid-filled containers
JPH0634575A (en) * 1992-07-21 1994-02-08 Asahi Chem Ind Co Ltd Bottle inspection method
JPH0821798A (en) * 1994-07-06 1996-01-23 Kanegafuchi Chem Ind Co Ltd Method and apparatus for detecting foreign matter
JPH08285789A (en) * 1995-04-18 1996-11-01 Nippon Sheet Glass Co Ltd Method for determining binarization level in defect detection method
JPH1019799A (en) * 1996-07-09 1998-01-23 Mutual Corp Inspection method of mixed foreign matter in vessel and inspection equipment
JPH11125604A (en) * 1997-10-22 1999-05-11 Ritsumeikan Method and device for detecting foreign matter
JP2001116703A (en) * 1999-10-21 2001-04-27 M I L:Kk Method and apparatus for discriminating flotage in container

Also Published As

Publication number Publication date
JP2002340808A (en) 2002-11-27

Similar Documents

Publication Publication Date Title
US5444480A (en) Method of inspecting solid body for foreign matter
JPH04220551A (en) Method and apparatus for inspecting flaw of transparent body
JP4048848B2 (en) Foreign object detection device in container and foreign object detection method in container
JP4580122B2 (en) Detecting foreign matter in liquid
JP3767695B2 (en) Empty bottle inspection system
KR20180114826A (en) Sheet inspection device
JP2010091530A (en) Method and apparatus for inspecting foreign substance
KR102632169B1 (en) Apparatus and method for inspecting glass substrate
JP6628185B2 (en) Inspection method for transparent objects
JP2004257937A (en) Foreign matter inspection device and inspection method
JP2008039444A (en) Method and apparatus for inspecting foreign matter
JP4254185B2 (en) Method and apparatus for manufacturing hollow fiber membrane module
CN114820622A (en) Interlayer foreign matter detection method and device
JP3047168B2 (en) Inspection method of chicken eggs
JP5679564B2 (en) Surface foreign matter inspection device
JPH0448251A (en) Bottle inspecting device
JPH0658733A (en) Inspecting method of nonuniform section of glass bottle
JPH10282014A (en) Surface defect detector with polarizing plate
JP2010008125A (en) Bubble sorting method in glass substrate
JP3149336B2 (en) Optical member inspection device
JPH0933342A (en) Method for controlling lighting luminance, and optical member inspecting device
JPH04270951A (en) Method for inspecting bottle
EP0610956A2 (en) Container inner surface tester
JP2002055053A (en) Visual inspection apparatus for tablet and ptp packaging machine
JP2006189294A (en) Inspection method and device of irregular defect

Legal Events

Date Code Title Description
A711 Notification of change in applicant

Free format text: JAPANESE INTERMEDIATE CODE: A712

Effective date: 20080117

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20080118

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A821

Effective date: 20080117

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20100604

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20100615

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20100806

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20100824

A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20100827

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20130903

Year of fee payment: 3

R150 Certificate of patent or registration of utility model

Free format text: JAPANESE INTERMEDIATE CODE: R150

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20130903

Year of fee payment: 3

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

RD02 Notification of acceptance of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: R3D02

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

LAPS Cancellation because of no payment of annual fees