JP6314798B2 - Surface defect detection method and surface defect detection apparatus - Google Patents
Surface defect detection method and surface defect detection apparatus Download PDFInfo
- Publication number
- JP6314798B2 JP6314798B2 JP2014229868A JP2014229868A JP6314798B2 JP 6314798 B2 JP6314798 B2 JP 6314798B2 JP 2014229868 A JP2014229868 A JP 2014229868A JP 2014229868 A JP2014229868 A JP 2014229868A JP 6314798 B2 JP6314798 B2 JP 6314798B2
- Authority
- JP
- Japan
- Prior art keywords
- image
- inspection object
- surface defect
- irradiation pattern
- defect detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000007547 defect Effects 0.000 title claims description 58
- 238000001514 detection method Methods 0.000 title claims description 38
- 238000007689 inspection Methods 0.000 claims description 34
- 238000000034 method Methods 0.000 claims description 31
- 238000012545 processing Methods 0.000 claims description 25
- 238000003384 imaging method Methods 0.000 claims description 13
- 239000000284 extract Substances 0.000 claims description 5
- 230000001678 irradiating effect Effects 0.000 claims description 2
- 229910000831 Steel Inorganic materials 0.000 description 44
- 239000010959 steel Substances 0.000 description 44
- 239000000463 material Substances 0.000 description 30
- 238000010606 normalization Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 17
- 230000008859 change Effects 0.000 description 16
- 230000008569 process Effects 0.000 description 16
- 230000001788 irregular Effects 0.000 description 4
- 239000000047 product Substances 0.000 description 4
- 230000003595 spectral effect Effects 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000010365 information processing Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- NAWXUBYGYWOOIX-SFHVURJKSA-N (2s)-2-[[4-[2-(2,4-diaminoquinazolin-6-yl)ethyl]benzoyl]amino]-4-methylidenepentanedioic acid Chemical compound C1=CC2=NC(N)=NC(N)=C2C=C1CCC1=CC=C(C(=O)N[C@@H](CC(=C)C(O)=O)C(O)=O)C=C1 NAWXUBYGYWOOIX-SFHVURJKSA-N 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 239000010960 cold rolled steel Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000000275 quality assurance Methods 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 239000011265 semifinished product Substances 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Description
本発明は、鋼材等の表面の色合いが均一な検査対象物の表面に形成された凹凸形状の表面欠陥を検出する表面欠陥検出方法及び表面欠陥検出装置に関する。 The present invention relates to a surface defect detection method and a surface defect detection device for detecting uneven surface defects formed on the surface of an inspection object having a uniform surface color such as steel.
近年、鉄鋼製品の製造工程では、品質保証及び歩留まり向上の観点から熱間又は冷間での搬送中における鋼材の表面に形成された凹凸形状の表面欠陥を検出することが求められている。なお、本明細書中において、鋼材とは、継目無鋼管、溶接鋼管、熱延鋼板、冷延鋼板、及び厚板等の鋼板や形鋼をはじめとした鉄鋼製品及びこれらの鉄鋼製品が製造される過程で発生するスラブ等の半製品のことを意味する。 In recent years, in the manufacturing process of steel products, it has been required to detect uneven surface defects formed on the surface of steel during hot or cold conveyance from the viewpoint of quality assurance and yield improvement. In the present specification, the term “steel material” refers to steel products such as seamless steel pipes, welded steel pipes, hot-rolled steel sheets, cold-rolled steel sheets, and thick steel plates, and steel products, and these steel products. This means semi-finished products such as slabs generated in the process.
一般に、鋼材の形状は柱状であり、鋼材はその長手方向に平行移動させて製造ラインを搬送されことが多く、また鋼材表面の色合いも均一であることが多い。このような鋼材の表面に形成された凹凸形状の表面欠陥を検出する場合、鋼材表面の形状や深さ位置に関する情報は、凹凸形状の表面欠陥の有害度を判定する上で非常に重要な情報であり、凹凸形状の表面欠陥の検出に必要不可欠な情報である。 In general, the shape of a steel material is a columnar shape, and the steel material is often moved in the production line by being translated in the longitudinal direction, and the color of the steel material surface is often uniform. When detecting uneven surface defects formed on the surface of such steel materials, information on the surface shape and depth position of the steel material is very important information for determining the degree of harm of uneven surface defects. This information is indispensable for detecting irregular surface defects.
このため、特許文献１には、凹凸形状の表面欠陥を検出することを目的として検査対象物の３次元形状を計測する方法が提案されている。具体的には、特許文献１には、長手方向に移動する検査対象物にスリット光を照射し、撮像装置を利用してスリット光の照射位置を撮影し、撮影されたスリット光の照射位置の変動から検査対象物の表面の凹凸情報を取得して凹凸形状の表面欠陥の有無を判別する方法が記載されている。 For this reason, Patent Document 1 proposes a method for measuring the three-dimensional shape of an inspection object for the purpose of detecting a surface defect having an uneven shape. Specifically, Patent Document 1 irradiates an inspection target moving in the longitudinal direction with slit light, images the slit light irradiation position using an imaging device, and records the slit light irradiation position. A method is described in which unevenness information on the surface of an inspection object is acquired from fluctuations and the presence or absence of uneven surface defects is determined.
しかしながら、特許文献１記載の方法によれば、検査対象物が高速で搬送されている場合、検査エリアに対して高いフレームレートで画像を撮影する必要があり、撮影画像データの伝送速度や処理速度の不足のために表面欠陥を精度よく検出できないことがある。なお、このような問題を解決するために、特許文献２に記載されているようなスペクトルパターンを投影することによって検査対象物表面の３次元形状を広範囲に検出する方法を用いることが考えられる。 However, according to the method described in Patent Document 1, when the inspection object is conveyed at high speed, it is necessary to capture an image at a high frame rate with respect to the inspection area, and the transmission speed and processing speed of the captured image data are required. In some cases, surface defects cannot be detected accurately due to the lack of. In order to solve such a problem, it is conceivable to use a method for detecting the three-dimensional shape of the surface of the inspection object in a wide range by projecting a spectrum pattern as described in Patent Document 2.
ところが、スペクトルパターンを投影する場合には、スペクトルパターンが全検査領域において可能な限り単色であることが必要である。なお、ここで述べる単色とは、光線に含まれる波長が狭帯域となる現象を意味し、そのようなスペクトルパターンを生成する光学系を設計することは現実的には非常に困難であるために実現することは難しい。 However, when projecting a spectral pattern, it is necessary that the spectral pattern be as monochromatic as possible in the entire inspection region. The single color described here means a phenomenon in which the wavelength contained in the light beam becomes a narrow band, and it is actually very difficult to design an optical system that generates such a spectral pattern. It is difficult to realize.
本発明は、上記課題に鑑みてなされたものであって、その目的は、色合いが均一な検査対象物の表面に形成された凹凸形状の表面欠陥を精度よく検出可能な表面欠陥検出方法及び表面欠陥検出装置を提供することにある。 SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and its object is to provide a surface defect detection method and a surface capable of accurately detecting a surface defect having an uneven shape formed on the surface of an inspection object having a uniform hue. It is to provide a defect detection apparatus.
本発明に係る表面欠陥検出方法は、搬送される検査対象物の表面に所定方向に沿って色合いが変化する照射パターンを照射する照射ステップと、前記照射パターンが照射された検査対象物の表面の画像を複数撮影する撮影ステップと、前記撮影ステップにおいて撮影された各画像と基準画像とを比較することによって前記検査対象物の表面に形成された凹凸形状の表面欠陥を検出する検出ステップと、を含むことを特徴とする。 The surface defect detection method according to the present invention includes an irradiation step of irradiating a surface of an inspection object to be conveyed with an irradiation pattern whose hue changes along a predetermined direction, and a surface of the inspection object irradiated with the irradiation pattern. A photographing step for photographing a plurality of images, and a detecting step for detecting a surface defect having an uneven shape formed on the surface of the inspection object by comparing each image photographed in the photographing step with a reference image. It is characterized by including.
本発明に係る表面欠陥検出方法は、上記発明において、前記検出ステップは、前記撮影ステップにおいて撮影された各画像について、輝度を正規化することによって各画像の色合い情報を抽出し、各画像の色合い情報と基準画像の色合い情報とを比較することによって前記検査対象物の表面に形成された凹凸形状の表面欠陥を検出するステップを含むことを特徴とする。 In the surface defect detection method according to the present invention, in the above invention, the detection step extracts the hue information of each image by normalizing the luminance for each image photographed in the photographing step, and the hue of each image. The method includes a step of detecting a surface defect having a concavo-convex shape formed on the surface of the inspection object by comparing the information with the color tone information of the reference image.
本発明に係る表面欠陥検出方法は、上記発明において、前記検出ステップは、前記撮影ステップにおいて撮影された各画像について、色空間を変換することによって各画像の色合い情報を抽出し、各画像の色合い情報と基準画像の色合い情報とを比較することによって前記検査対象物の表面に形成された凹凸形状の表面欠陥を検出するステップを含むことを特徴とする。 In the surface defect detection method according to the present invention, in the above invention, the detection step extracts the hue information of each image by converting a color space for each image photographed in the photographing step, and the hue of each image. The method includes a step of detecting a surface defect having a concavo-convex shape formed on the surface of the inspection object by comparing the information with the color tone information of the reference image.
本発明に係る表面欠陥検出方法は、上記発明において、前記検出ステップは、前記撮影ステップにおいて複数の画像を撮影する際に前記照射パターンを照射する光源、前記画像を撮影する撮像装置、及び前記検査対象物の位置関係が変化した場合、画像間で画像中における照射パターンの位置を合わせ込むステップを含むことを特徴とする。 In the surface defect detection method according to the present invention, in the above invention, the detection step includes a light source that irradiates the irradiation pattern when imaging a plurality of images in the imaging step, an imaging device that images the image, and the inspection. When the positional relationship of the target object changes, the method includes a step of aligning the position of the irradiation pattern in the image between the images.
本発明に係る表面欠陥検出装置は、搬送される検査対象物の表面に対して所定方向に沿って色合いが変化する照射パターンを照射する光源と、前記照射パターンが照射された検査対象物の表面の画像を複数撮影する撮像装置と、前記撮像装置によって撮影された各画像と基準画像とを比較することによって前記検査対象物の表面に形成された凹凸形状の表面欠陥を検出する画像処理装置と、を備えることを特徴とする。 The surface defect detection device according to the present invention includes a light source that irradiates an irradiation pattern whose hue changes along a predetermined direction with respect to a surface of an inspection object to be conveyed, and a surface of the inspection object that is irradiated with the irradiation pattern. An imaging device that captures a plurality of images of the image, and an image processing device that detects surface defects of irregularities formed on the surface of the inspection object by comparing each image captured by the imaging device with a reference image; It is characterized by providing.
本発明に係る表面欠陥検出方法及び表面欠陥検出装置によれば、色合いが均一な検査対象物の表面に形成された凹凸形状の表面欠陥を精度よく検出することができる。 According to the surface defect detection method and the surface defect detection apparatus according to the present invention, it is possible to accurately detect uneven surface defects formed on the surface of an inspection object having a uniform hue.
以下、図面を参照して、本発明の一実施形態である表面欠陥検出装置について説明する。 Hereinafter, a surface defect detection device according to an embodiment of the present invention will be described with reference to the drawings.
〔表面欠陥検出装置の構成〕
始めに、図１を参照して、本発明の一実施形態である表面欠陥検出装置の構成について説明する。図１は、本発明の一実施形態である表面欠陥検出装置の構成を示す模式図である。
[Configuration of surface defect detection device]
First, with reference to FIG. 1, the structure of the surface defect detection apparatus which is one Embodiment of this invention is demonstrated. FIG. 1 is a schematic diagram showing a configuration of a surface defect detection apparatus according to an embodiment of the present invention.
図１に示すように、本発明の一実施形態である表面欠陥検出装置１は、矢印Ｄ方向（＝鋼材Ｓの長手方向）に搬送される色合いが均一な柱状の鋼材Ｓの表面に形成された凹凸形状の表面欠陥を検出する装置であり、光源２、カラーカメラ３、及び画像処理装置４を主な構成要素として備えている。なお、表面の色合いが均一な鋼材とは、表面全体においては反射強度にバラつきがあっても、波長毎の分光反射率比がほぼ同じになる表面を有する鋼材のことを意味する。 As shown in FIG. 1, the surface defect detection apparatus 1 according to an embodiment of the present invention is formed on the surface of a columnar steel material S having a uniform hue conveyed in the direction of arrow D (= longitudinal direction of the steel material S). The apparatus detects an uneven surface defect having a light source 2, a color camera 3, and an image processing apparatus 4 as main components. The steel material having a uniform surface color means a steel material having a surface with the same spectral reflectance ratio for each wavelength even if the reflection intensity varies over the entire surface.
光源２は、スリットレーザ光源２ａ、レンズ２ｂ、及びプリズム２ｃを備えている。スリットレーザ光源２ａは、平行スリットレーザ光を射出する。レンズ２ｂは、スリットレーザ光源２ａから射出された平行スリットレーザ光をプリズム２ｃ方向に集光する。プリズム２ｃは、レンズ２ｂからの平行スリットレーザ光を所定の照射パターン５にして鋼材Ｓの表面に照射する。 The light source 2 includes a slit laser light source 2a, a lens 2b, and a prism 2c. The slit laser light source 2a emits parallel slit laser light. The lens 2b condenses the parallel slit laser light emitted from the slit laser light source 2a in the prism 2c direction. The prism 2c irradiates the surface of the steel material S with the parallel slit laser beam from the lens 2b in a predetermined irradiation pattern 5.
なお、本実施形態では、所定の照射パターンとは、所定方向に沿って色合いが連続的又は段階的に変化する光のパターン、換言すれば、複数の波長選択フィルタを用いて光を受光した時に各波長選択フィルタの受光量比が所定方向に沿って連続的又は段階的に変化する光のパターンのことを意味する。 In the present embodiment, the predetermined irradiation pattern is a light pattern whose hue changes continuously or stepwise along a predetermined direction, in other words, when light is received using a plurality of wavelength selection filters. It means a light pattern in which the received light amount ratio of each wavelength selection filter changes continuously or stepwise along a predetermined direction.
本実施形態では、プリズムを利用して所定の照射パターン５を生成したが、回折格子等の分光素子を利用して所定の照射パターン５を生成してもよい。また、所定の照射パターン５の色合いが変化する所定方向はどの方向であっても問題ないが、後述する復元処理を容易にするために鋼材Ｓの長手方向（搬送方向）に対して平行な向きであることが望ましい。 In the present embodiment, the predetermined irradiation pattern 5 is generated using a prism, but the predetermined irradiation pattern 5 may be generated using a spectroscopic element such as a diffraction grating. Further, the predetermined direction in which the color of the predetermined irradiation pattern 5 changes may be any direction, but the direction parallel to the longitudinal direction (conveying direction) of the steel material S in order to facilitate the restoration process described later. It is desirable that
カラーカメラ３は、所定の照射パターン５が照射された鋼材Ｓの表面の画像を所定の制御周期毎に撮影し、撮影された画像のデータを画像処理装置４に出力する。なお、通常の光切断法と同様、画像の撮影範囲及び分解能は、鋼材表面の法線ベクトルに対する光源２やカラーカメラ３の位置関係及びカラーカメラ３の撮像素子面の大きさや解像度に応じて決まる。本実施形態では、鋼材表面の撮影位置に対する光源２及びカラーカメラ３の位置関係が同じになるようにして鋼材Ｓの表面の画像を撮影する。 The color camera 3 captures an image of the surface of the steel material S irradiated with the predetermined irradiation pattern 5 for each predetermined control period, and outputs the captured image data to the image processing device 4. Note that, as in the normal light cutting method, the image capturing range and resolution are determined according to the positional relationship of the light source 2 and the color camera 3 with respect to the normal vector of the steel material surface and the size and resolution of the image sensor surface of the color camera 3. . In the present embodiment, an image of the surface of the steel material S is photographed so that the positional relationship of the light source 2 and the color camera 3 with respect to the photographing position of the steel material surface is the same.
画像処理装置４は、パーソナルコンピュータ等の情報処理装置によって構成され、情報処理装置内部の演算処理装置がコンピュータプログラムを実行することによって後述する表面欠陥検出処理を実行する。 The image processing device 4 is configured by an information processing device such as a personal computer, and an arithmetic processing device inside the information processing device executes a surface defect detection process described later by executing a computer program.
〔表面欠陥検出処理〕
次に、図２〜図５を参照して、画像処理装置４による表面欠陥検出処理の流れについて説明する。
[Surface defect detection processing]
Next, the flow of surface defect detection processing by the image processing device 4 will be described with reference to FIGS.
いまカラーカメラ３を利用して搬送中の鋼材Ｓ表面を鋼材Ｓの長手方向全領域にわたって撮像した２次元画像をＩｍ（Ｉｒ，Ｉｇ，Ｉｂ）と定義する。ここで、ｍは画像番号を示し（１≦ｍ≦Ｍ、Ｍは全画像枚数）、Ｉｒは２次元画像の赤色チャンネル（Ｒ）の輝度を示し、Ｉｇは２次元画像の緑色チャンネル（Ｇ）の輝度を示し、Ｉｂは２次元画像の青色チャンネル（Ｂ）の輝度を示している。 A two-dimensional image obtained by imaging the surface of the steel material S being conveyed using the color camera 3 over the entire longitudinal direction of the steel material S is defined as Im (Ir, Ig, Ib). Here, m represents an image number (1 ≦ m ≦ M, M is the total number of images), Ir represents the luminance of the red channel (R) of the two-dimensional image, and Ig represents the green channel (G) of the two-dimensional image. Ib represents the luminance of the blue channel (B) of the two-dimensional image.
また、２次元画像の大きさを（Ｘ，Ｙ）として、２次元画像中の位置(ｘ，ｙ)（１≦ｘ≦Ｘ、１≦ｙ≦Ｙ）における赤色チャンネル、緑色チャンネル、及び青色チャンネルの輝度をＩｒ(ｘ，ｙ)、Ｉｇ(ｘ，ｙ)、Ｉｂ(ｘ，ｙ)と定義する。また、ｘ方向は鋼材Ｓの搬送方向に平行な方向と定義し、ｘ方向は撮像面内においてｘ方向に直交する方向と定義する。 Also, assuming that the size of the two-dimensional image is (X, Y), the red channel, the green channel, and the blue channel at the position (x, y) (1 ≦ x ≦ X, 1 ≦ y ≦ Y) in the two-dimensional image Are defined as Ir (x, y), Ig (x, y), and Ib (x, y). Further, the x direction is defined as a direction parallel to the conveying direction of the steel material S, and the x direction is defined as a direction orthogonal to the x direction in the imaging surface.
なお、鋼材Ｓの長手方向端部が撮影されている２次元画像を除き、２次元画像の各色チャンネルの輝度はサチレーションしていないものとする。また、各色チャンネルの輝度比が連続的、且つ、なだらかに変化するように、照射パターン５及びカラーカメラ３の受光波長を調整することが望ましい。例えば連続的に光量が変化するＮＤ（減光）フィルタと赤色光源及び青色光源とを用いて、なだらかに赤色から青色まで変化する照射パターン５とそれに対応するカラーカメラ３の受光波長を用いても良い。また、本実施形態では赤色、緑色、及び青色の３チャンネルを用いたが、別途波長選択フィルタを用いて２以上のチャンネル数の装置構成であってもよい。 Note that the luminance of each color channel of the two-dimensional image is not saturated except for the two-dimensional image in which the longitudinal end of the steel material S is photographed. Further, it is desirable to adjust the light receiving wavelengths of the irradiation pattern 5 and the color camera 3 so that the luminance ratio of each color channel changes continuously and gently. For example, by using an ND (attenuation) filter whose light quantity changes continuously, a red light source and a blue light source, the irradiation pattern 5 that gradually changes from red to blue and the light receiving wavelength of the color camera 3 corresponding thereto can be used. good. In the present embodiment, three channels of red, green, and blue are used. However, a device configuration with two or more channels may be used by using a separate wavelength selection filter.
本実施形態の表面欠陥検出処理では、始めに、画像処理装置４が、撮影したＭ枚の２次元画像についてそれぞれ反射強度のばらつきの影響を除去し、色合い情報のみを取り出すために２次元画像内の各色チャンネルの輝度に対して以下の数式（１）〜（３）で示す正規化処理を実行する。図２（ａ），（ｂ）はそれぞれ、正規化処理前及び正規化処理後の２次元画像の一例を示す図である。また、図２（ｂ）の下部には、図２（ｂ）に示す線分部分における赤色、緑色、及び青色チャンネルの輝度プロファイルを示す。 In the surface defect detection processing according to the present embodiment, first, the image processing apparatus 4 removes the influence of the variation in reflection intensity for each of the captured M two-dimensional images, and extracts only the hue information. The normalization process shown by the following mathematical formulas (1) to (3) is performed on the luminance of each color channel. FIGS. 2A and 2B are diagrams illustrating examples of two-dimensional images before and after normalization processing, respectively. Further, the lower part of FIG. 2B shows the luminance profiles of the red, green, and blue channels in the line segment shown in FIG.
なお、本実施形態では、反射強度のばらつきの影響を除去し、色合い情報のみを取り出すために正規化処理を行ったが、ＨＳＬ（Hue, Saturation, Lightness）やＨＬＳ（Hue, Lightness, Saturation）等の別色空間に変換する手法を用いて色合い情報を得る方法や、ＲＧＢの輝度情報から統計的な手法を用いて鋼材表面性状に影響される成分を除外してもよい。 In this embodiment, the normalization process is performed to remove the influence of the variation in the reflection intensity and to extract only the hue information. The method of obtaining the hue information using the method of converting to another color space, or the component affected by the surface property of the steel material using the statistical method may be excluded from the RGB luminance information.
次に、画像処理装置４は、以下に示す数式（４）を用いて正規化処理後の全２次元画像の平均画像を生成する。なお、数式（４）に示すＩ’ｍ（Ｉｒ’，Ｉｇ’，Ｉｂ’）は、２次元画像Ｉｍ（Ｉｒ，Ｉｇ，Ｉｂ）に対して正規化処理を行った画像を表している。図３は、正規化処理後の全２次元画像の平均画像の一例を示す図である。また、図３の下部には、線分部分における赤色、緑色、及び青色チャンネルの輝度プロファイルを示す。 Next, the image processing device 4 generates an average image of all the two-dimensional images after the normalization process by using the following formula (4). Note that I ′m (Ir ′, Ig ′, Ib ′) shown in Expression (4) represents an image obtained by normalizing the two-dimensional image Im (Ir, Ig, Ib). FIG. 3 is a diagram illustrating an example of an average image of all two-dimensional images after the normalization process. Also, the lower part of FIG. 3 shows the luminance profiles of the red, green, and blue channels in the line segment.
数式（４）に示す正規化処理後の全２次元画像の平均画像における表面形状は、実際には鋼材Ｓ表面に局所的に凹凸形状の表面欠陥があり、表面形状が変形していたとしても、平均化処理によって均一、且つ、滑らかになっている。そこで、次に、画像処理装置４は、数式（４）に示す平均画像とＭ枚の正規化処理後の２次元画像Ｉ’ｍ（Ｉｒ’，Ｉｇ’，Ｉｂ’）とをそれぞれ比較することによって、表面形状の変化に起因して照射パターン５の色合いの変化方向であるｘ方向に沿って照射パターン５がどれだけ変化しているかを計算する。 Even if the surface shape in the average image of all the two-dimensional images after the normalization process shown in Formula (4) actually has surface irregularities on the surface of the steel S, the surface shape is deformed. It is uniform and smooth by the averaging process. Therefore, next, the image processing device 4 compares the average image shown in Equation (4) with the M two-dimensional images I ′m (Ir ′, Ig ′, Ib ′) after normalization. Thus, how much the irradiation pattern 5 changes along the x direction, which is the direction of change in the hue of the irradiation pattern 5, due to the change in the surface shape is calculated.
具体的には、画像処理装置４は、以下に示す数式（５）を用いて、平均画像からのｘ方向の変化量を表す画像Ｉｄ（ｘ，ｙ，ｍ）をＭ枚の正規化処理後の２次元画像Ｉ’ｍ（ｘ，ｙ）毎に算出する。なお、数式（５）に示すｘ１は、凹凸形状の表面欠陥が形成されていない場合に２次元画像Ｉ’ｍ（ｘ，ｙ）内の位置ｘで受光される光が受光された２次元画像Ｉ’ｍ（ｘ，ｙ）中のｘ座標を示している。図４は、平均画像からのｘ方向の変化量を表す画像の一例を示す図である。また、図４の下部には、線分部分における赤色、緑色、及び青色チャンネルの輝度プロファイルを示す。 Specifically, the image processing device 4 uses the following formula (5) to normalize the image Id (x, y, m) representing the amount of change in the x direction from the average image by M sheets. For each two-dimensional image I ′m (x, y). Incidentally, x 1 shown in Equation (5) is two-dimensional case in the two-dimensional image I'm (x, y) of surface defects of the uneven shape is not formed light received by the position x within the received The x coordinate in the image I ′m (x, y) is shown. FIG. 4 is a diagram illustrating an example of an image representing the amount of change in the x direction from the average image. Also, the lower part of FIG. 4 shows the luminance profiles of the red, green, and blue channels in the line segment.
次に、画像処理装置４は、三角測量の原理に基づき数式（５）に示す画像Ｉｄ（ｘ，ｙ，ｍ）を利用して、表面形状の変化に起因する平均画像の表面（基準面）からの深さ方向の変化量をＭ枚の正規化処理後の２次元画像Ｉ’ｍ（ｘ，ｙ）毎に算出する。ここで、図５を参照して、一般的な三角測量の原理について説明する。なお、三角測量の詳細については、例えば参考文献（吉澤徹編「光三次元計測」、新技術コミュニケーションズ）を参照のこと。 Next, the image processing device 4 uses the image Id (x, y, m) shown in Formula (5) based on the principle of triangulation, and the surface (reference plane) of the average image resulting from the change in the surface shape. The amount of change in the depth direction is calculated for each of the M two-dimensional images I ′m (x, y) after normalization. Here, the principle of general triangulation will be described with reference to FIG. For details on triangulation, refer to, for example, the reference (Toru Yoshizawa, “Optical 3D Measurement”, New Technology Communications).
図５に示すように、スリットレーザ光源２ａの設置角度及びカラーカメラ３の受光角度をそれぞれθ，φ、スリットレーザ光源２ａとカラーカメラ３との間の距離をＬと表すと、鋼材Ｓの基準面の位置とスリットレーザ光源２ａ及びカラーカメラ３との間の距離Ｚは以下に示す数式（６）で表される。 As shown in FIG. 5, when the installation angle of the slit laser light source 2a and the light receiving angle of the color camera 3 are represented by θ and φ, respectively, and the distance between the slit laser light source 2a and the color camera 3 is represented by L, the reference of the steel S A distance Z between the position of the surface and the slit laser light source 2a and the color camera 3 is expressed by the following formula (6).
また、カラーカメラ３の画素分解能及び画素ズレから、カラーカメラ３のレンズ３ａと画像素子面３ｂとの間の距離ｌと画像素子面３ｂにおける凹凸形状の表面欠陥に起因するレーザ反射光の受光位置ズレ量Δｘとの比Δｘ／ｌを計算することができる。 Further, from the pixel resolution and pixel shift of the color camera 3, the distance l between the lens 3a and the image element surface 3b of the color camera 3 and the light receiving position of the laser reflected light caused by the surface defect of the irregular shape on the image element surface 3b. A ratio Δx / l with the deviation amount Δx can be calculated.
従って、以下に示す数式（７）にこの比Δｘ／ｌの値を代入することによって、表面形状の変化に起因するレーザ反射光の受光角度の変化量Δφが算出され、検査位置Ｏにおける基準面Ｐからの深さ方向の移動距離、すなわち鋼材Ｓの凹凸形状を算出することができる。 Therefore, by substituting the value of this ratio Δx / l into the following formula (7), the amount of change Δφ in the light receiving angle of the laser reflected light caused by the change in the surface shape is calculated, and the reference surface at the inspection position O is calculated. The moving distance in the depth direction from P, that is, the uneven shape of the steel material S can be calculated.
なお、本実施形態によれば、搬送時に鋼材にばたつきがあり、鋼材の位置や向きが搬送中に変化したとしても画像処理によって後から鋼材の位置や向きを合わせこむことができる。すなわち、正規化処理後の２次元画像Ｉ’ｍにおける照射パターンの位置及び形状を最初に撮影された２次元画像Ｉ’１における照射パターンの位置及び形状に合わせこむためには、必要に応じて回転・平行移動・拡大縮小の線形変換のパラメータαをそれぞれ２次元画像Ｉ’ｍ毎に最適化すればよい。パラメータαは必要に応じて回転・平行移動・拡大縮小等の変換とする。 In addition, according to this embodiment, even if the steel material fluctuates at the time of conveyance and the position and orientation of the steel material change during conveyance, the position and orientation of the steel material can be adjusted later by image processing. That is, in order to match the position and shape of the irradiation pattern in the two-dimensional image I ′ m after the normalization processing with the position and shape of the irradiation pattern in the first two-dimensional image I ′ 1 photographed, it is rotated as necessary. It is only necessary to optimize the linear conversion parameter α for translation / enlargement / reduction for each two-dimensional image I ′ m . The parameter α is a conversion such as rotation, parallel movement, and enlargement / reduction as required.
具体的には、合わせ込み後の２次元画像をＩ” ｍとすると、以下に示す数式（８）を用いてｍ枚目の２次元画像Ｉ’ｍのパラメータαｍを算出し、数式（９）を用いてｍ枚目の２次元画像Ｉ’ｍの合わせ込みを実施する。本実施形態では、線形変換を用いて照射パターンの位置合わせを行ったが、ＲＡＮＳＡＣ（Random Sample Consensus）アルゴリズム等の非線形な変形手法を用いて照射パターンの位置合わせを行ってもよい。また、最初に撮影された２次元画像における照射パターンの形状に合わせこむとしたが、合わせ込む２次元画像は何番目の２次元画像でもよく、最も平均画像に近い２次元画像であってもよい。 Specifically, assuming that the combined two-dimensional image is I ″ m , the parameter α m of the m-th two-dimensional image I ′ m is calculated using the following equation (8), and the equation (9 ) Is used to align the m-th two-dimensional image I ′ m.In this embodiment, the alignment of the irradiation pattern is performed using linear transformation, but a RANSAC (Random Sample Consensus) algorithm or the like is used. The irradiation pattern may be aligned by using a non-linear deformation method, and the irradiation pattern shape in the two-dimensional image that was first photographed is adjusted, but the two-dimensional image to be aligned is the second number. It may be a two-dimensional image or a two-dimensional image closest to the average image.
以上の説明から明らかなように、本発明の一実施形態である表面欠陥検出装置１は、搬送される鋼材Ｓの表面に対して鋼材Ｓの搬送方向に沿って色合いが変化する照射パターン５を照射する光源２と、照射パターン５が照射された鋼材Ｓの表面の画像を複数撮影するカラーカメラ３と、カラーカメラ３によって撮影された各画像と基準画像とを比較することによって鋼材Ｓの表面に形成された凹凸形状の表面欠陥を検出する画像処理装置４と、を備えるので、色合いが均一な検査対象物の表面に形成された凹凸形状の表面欠陥を精度よく検出することができる。 As is clear from the above description, the surface defect detection apparatus 1 according to an embodiment of the present invention has the irradiation pattern 5 whose color changes along the transport direction of the steel material S with respect to the surface of the steel material S to be transported. The light source 2 to be irradiated, the color camera 3 that captures a plurality of images of the surface of the steel material S irradiated with the irradiation pattern 5, and the surface of the steel material S by comparing each image captured by the color camera 3 with a reference image. And the image processing device 4 for detecting the irregular surface defect formed on the surface, it is possible to accurately detect the irregular surface defect formed on the surface of the inspection object having a uniform hue.
本実施例では、図６に示すようなプリズム２ｃ及びカラーカメラ３の配置位置で図７に示す凹凸形状の表面欠陥を有する鋳片サンプルの表面を検査した。図７（ａ）に示す表面欠陥画像は、レーザ距離計を走査することによって得られた凹凸形状の表面欠陥の３次元画像を示し、図７（ｂ）は図７（ａ）の線分Ｌにおける表面の凹凸プロファイルを示す。 In this example, the surface of the slab sample having the surface defects of the uneven shape shown in FIG. 7 was inspected at the position where the prism 2c and the color camera 3 as shown in FIG. The surface defect image shown in FIG. 7A shows a three-dimensional image of the uneven surface defect obtained by scanning the laser distance meter, and FIG. 7B shows a line segment L in FIG. The uneven | corrugated profile of the surface in is shown.
表面欠陥検出処理では、平行移動させた鋳片サンプルに対して光源２から照射パターン５を照射し、カラーカメラ３により４５枚の２次元画像を連続的に撮影し、その平均画像からのｘ方向の変化量を表す画像Ｉｄ（ｘ，ｙ，ｍ）を用いて鋳片サンプル表面の形状復元を行った。また、ＲＧＢの各色チャンネルの輝度がサチレーションせずに同程度となるようホワイトバランスを調整し、解像度：２０４６×１５３６、レンズの焦点距離：１２．５ｍｍ、分解能０．２ｍｍ／ピクセルとした。 In the surface defect detection processing, the slab sample moved in parallel is irradiated with the irradiation pattern 5 from the light source 2, and 45 color two-dimensional images are continuously photographed by the color camera 3, and the x direction from the average image is obtained. The shape of the slab sample surface was restored using an image Id (x, y, m) representing the amount of change in the slab. Further, the white balance was adjusted so that the luminance of the RGB color channels would be the same without saturation, and the resolution was 2046 × 1536, the focal length of the lens was 12.5 mm, and the resolution was 0.2 mm / pixel.
具体的には、始めに、２次元画像に対して正規化処理を行い、正規化処理後の２次元画像の平均画像を算出する。正規化処理前後の２次元画像及びその輝度プロファイルをそれぞれ図８，９に示す。図８，９に示すように、正規化処理によって表面の反射強度のばらつきの影響が抑えられ、表面の色合いが滑らかに変動している様子が見て取れる。次に、正規化処理後の２次元画像の平均画像を用いて各２次元画像中における照射パターンの変化量を算出する。なお、鋳片の表面に何らかの凹凸形状が形成されている場合には照射パターンの変化量を算出しなくても、輝度の絶対値の差分を取るだけである程度凹凸形状に由来する輝度信号を検出できる。また、必要に応じてローパスフィルターにより２次元画像から高周波ノイズを除去してもよい。 Specifically, first, a normalization process is performed on the two-dimensional image, and an average image of the two-dimensional image after the normalization process is calculated. The two-dimensional image before and after the normalization process and its luminance profile are shown in FIGS. As shown in FIGS. 8 and 9, it can be seen that the normalization process suppresses the influence of the variation in the reflection intensity of the surface, and the hue of the surface changes smoothly. Next, the amount of change of the irradiation pattern in each two-dimensional image is calculated using the average image of the two-dimensional images after the normalization process. If some uneven shape is formed on the surface of the slab, the luminance signal derived from the uneven shape can be detected to some extent by simply taking the difference in absolute value of the brightness without calculating the amount of change in the irradiation pattern. it can. Moreover, you may remove a high frequency noise from a two-dimensional image with a low-pass filter as needed.
図１０は、照射パターンの変化量の算出結果を示す図である。図１０に示すように、単純に平均画像と２次元画像との絶対値差分を取るだけで凹凸形状に由来する輝度信号が検出できている様子が見て取れる。簡易化のため、照射パターンの照射角をθとし、カラーカメラ３を鋼片サンプルの移動方向に対して垂直に撮像し、１画素の分解能がdであることから、ｎ画素パターンが移動した場合の奥行きΔＺは以下に示す数式（１０）で表される。 FIG. 10 is a diagram illustrating a calculation result of the change amount of the irradiation pattern. As shown in FIG. 10, it can be seen that the luminance signal derived from the concavo-convex shape can be detected simply by taking the absolute value difference between the average image and the two-dimensional image. For simplification, when the irradiation angle of the irradiation pattern is θ, the color camera 3 is imaged perpendicularly to the moving direction of the steel slab sample, and the resolution of one pixel is d. The depth ΔZ is expressed by the following formula (10).
上記数式（１０）を用いて、照射パターンの照射角を３０度、１画素の分解能を０．２ｍｍ／ピクセルとして表面形状を復元した結果を図１１に示す。本実施例では、高周波ノイズが乗っていたため、ローパスフィルターをかけた。また、レーザ距離計を利用した表面形状の実測結果を図１２に示す。図１１と図１２との比較から明らかなように、本実施例によれば、鋳片サンプル表面のおおよその深さ及び形状が復元できている様子が見て取れる。なお、本実施例ではカラーカメラを用いたが２種類以上の波長フィルタであれば問題ない。また、反射率補正のために正規化処理を行ったが、色相情報や色空間を用いてもよい。 FIG. 11 shows the result of restoring the surface shape by using the above equation (10) and setting the irradiation angle of the irradiation pattern to 30 degrees and the resolution of one pixel to 0.2 mm / pixel. In this embodiment, since high-frequency noise was present, a low-pass filter was applied. Moreover, the actual measurement result of the surface shape using a laser distance meter is shown in FIG. As is clear from the comparison between FIG. 11 and FIG. 12, according to this example, it can be seen that the approximate depth and shape of the slab sample surface can be restored. In this embodiment, a color camera is used, but there is no problem if two or more types of wavelength filters are used. Further, although normalization processing is performed for reflectance correction, hue information or a color space may be used.
また、本実施例では鋼片の欠陥サンプルに対して表面欠陥検出処理を実施したが、柱状で平行搬送される鋼片の製造ラインであれば適用可能である。さらに熱間における赤熱した検査対象物に対しても熱ガラスやホットミラー等で自発光成分を除去すれば適用可能である。また、検査対象物にばたつきや振動があるような搬送ラインに対しても画像毎に照射パターンの位置補正を行うことによって容易に適用可能である。 In the present embodiment, the surface defect detection processing is performed on the defective sample of the steel slab. However, the present invention is applicable to any steel slab production line that is conveyed in parallel in a column shape. Furthermore, it can be applied to a hot red inspection object by removing the self-luminous component with a hot glass or a hot mirror. In addition, the present invention can be easily applied to a conveyance line in which the inspection target has fluttering or vibration by correcting the position of the irradiation pattern for each image.
以上、本発明者らによってなされた発明を適用した実施形態について説明したが、本実施形態による本発明の開示の一部をなす記述及び図面により本発明は限定されることはなく、上述した各構成要素を適宜組み合わせて構成したものも本発明に含まれる。すなわち、本実施形態に基づいて当業者等によりなされる他の実施形態、実施例及び運用技術等は全て本発明の範疇に含まれる。 As mentioned above, although the embodiment to which the invention made by the present inventors was applied has been described, the present invention is not limited by the description and drawings which form part of the disclosure of the present invention according to this embodiment. What was comprised combining the component suitably is also contained in this invention. That is, other embodiments, examples, operation techniques, and the like made by those skilled in the art based on this embodiment are all included in the scope of the present invention.
１ 表面欠陥検出装置
２ 光源
２ａ スリットレーザ光源
２ｂ レンズ
２ｃ プリズム
３ カラーカメラ
４ 画像処理装置
５ 照射パターン
Ｓ 鋼材
DESCRIPTION OF SYMBOLS 1 Surface defect detection apparatus 2 Light source 2a Slit laser light source 2b Lens 2c Prism 3 Color camera 4 Image processing apparatus 5 Irradiation pattern S Steel material
Claims (5)
前記照射パターンが照射された検査対象物の表面の画像を複数撮影する撮影ステップと、
前記撮影ステップにおいて撮影された各画像と基準画像とを比較することによって、各色の光線の照射位置から表面形状を復元することにより、前記検査対象物の表面に形成された凹凸形状の表面欠陥を検出する検出ステップと、
を含むことを特徴とする表面欠陥検出方法。 Irradiation step of irradiating light of each color so that an irradiation pattern whose hue changes along a predetermined direction of the inspection object is formed on the surface of the inspection object to be conveyed;
An imaging step of capturing a plurality of images of the surface of the inspection object irradiated with the irradiation pattern;
By comparing each image photographed in the photographing step with a reference image, and restoring the surface shape from the irradiation position of each color beam , surface irregularities formed on the surface of the inspection object A detection step to detect;
A method for detecting surface defects, comprising:
前記照射パターンが照射された検査対象物の表面の画像を複数撮影する撮像装置と、
前記撮像装置によって撮影された各画像と基準画像とを比較することによって、各色の光線の照射位置から表面形状を復元することにより、前記検査対象物の表面に形成された凹凸形状の表面欠陥を検出する画像処理装置と、
を備えることを特徴とする表面欠陥検出装置。 A light source that emits light of each color so that an irradiation pattern whose hue changes along a predetermined direction of the inspection object is formed on the surface of the inspection object to be conveyed;
An imaging device that captures a plurality of images of the surface of the inspection object irradiated with the irradiation pattern;
By comparing each image photographed by the imaging device with a reference image, and restoring the surface shape from the irradiation position of the light beam of each color , surface irregularities formed on the surface of the inspection object An image processing device to detect;
A surface defect detection apparatus comprising:
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2014229868A JP6314798B2 (en) | 2014-11-12 | 2014-11-12 | Surface defect detection method and surface defect detection apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2014229868A JP6314798B2 (en) | 2014-11-12 | 2014-11-12 | Surface defect detection method and surface defect detection apparatus |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| JP2016095160A JP2016095160A (en) | 2016-05-26 |
| JP6314798B2 true JP6314798B2 (en) | 2018-04-25 |
Family
ID=56071746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| JP2014229868A Active JP6314798B2 (en) | 2014-11-12 | 2014-11-12 | Surface defect detection method and surface defect detection apparatus |
Country Status (1)
| Country | Link |
|---|---|
| JP (1) | JP6314798B2 (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101908808B1 (en) | 2016-12-22 | 2018-12-10 | 주식회사 포스코 | Apparatus and method for detecting strip wave |
| JP7039299B2 (en) * | 2017-03-03 | 2022-03-22 | 株式会社神戸製鋼所 | Residual grain boundary oxide layer inspection method and residual grain boundary oxide layer inspection device |
| JP6938093B2 (en) * | 2017-11-21 | 2021-09-22 | 株式会社マクシスエンジニアリング | A method for inspecting the boundary of an object to be inspected and its inspection device |
| JP7187330B2 (en) * | 2019-01-15 | 2022-12-12 | 株式会社小松製作所 | Shape measuring device, shape measuring system, and shape measuring method |
| JP7136064B2 (en) * | 2019-10-25 | 2022-09-13 | Ｊｆｅスチール株式会社 | Apparatus for inspecting surface of object to be inspected and method for inspecting surface of object to be inspected |
| CN111915704A (en) * | 2020-06-13 | 2020-11-10 | 东北林业大学 | Apple hierarchical identification method based on deep learning |
| CN113077414B (en) * | 2021-03-01 | 2023-05-16 | 湖北工程学院 | Steel plate surface defect detection method and system |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH04106461A (en) * | 1990-08-28 | 1992-04-08 | Mazda Motor Corp | Inspecting apparatus of surface defect |
| DE19509345A1 (en) * | 1995-03-15 | 1996-09-19 | Ver Glaswerke Gmbh | Methods for recognizing and evaluating defects in partially reflecting surface layers |
| US20050068544A1 (en) * | 2003-09-25 | 2005-03-31 | Gunter Doemens | Panoramic scanner |
| JP2008190872A (en) * | 2007-01-31 | 2008-08-21 | Toyota Motor Corp | Surface defectiveness detector, surface defectiveness detecting method and program |
| JP5297930B2 (en) * | 2009-07-29 | 2013-09-25 | 株式会社日立ハイテクノロジーズ | Defect inspection apparatus and method |
| JP5684505B2 (en) * | 2010-07-16 | 2015-03-11 | 株式会社キーエンス | Image processing apparatus, image processing method, and computer program |
| JP6115012B2 (en) * | 2011-08-30 | 2017-04-19 | 富士通株式会社 | Inspection device, inspection method, and inspection program |
| TWI529385B (en) * | 2011-09-26 | 2016-04-11 | 三菱麗陽股份有限公司 | Inspection device and inspection method for member having fine uneven structure on surface, method for manufacturing member having anodized alumina layer on surface and method for manufacturing optical film |
| JP5857675B2 (en) * | 2011-11-24 | 2016-02-10 | 富士通株式会社 | Surface defect inspection apparatus and surface defect inspection method |
| JP6049475B2 (en) * | 2013-01-25 | 2016-12-21 | リコーエレメックス株式会社 | Inspection device |
-
2014
- 2014-11-12 JP JP2014229868A patent/JP6314798B2/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| JP2016095160A (en) | 2016-05-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6314798B2 (en) | Surface defect detection method and surface defect detection apparatus | |
| KR102549058B1 (en) | Optical measurement apparatus and optical measurement method | |
| US9842430B2 (en) | Method and device for automatically identifying a point of interest on a viewed object | |
| JP5488953B2 (en) | Method and apparatus for inspection of uneven surface | |
| JP5418378B2 (en) | Surface defect inspection apparatus, surface defect inspection method, and program | |
| US10345248B2 (en) | Optical system and method for inspecting a transparent plate | |
| JP2010190886A (en) | Pantograph height measuring device and calibration method therefor | |
| JP6123135B2 (en) | Thickness inspection method and thickness inspection apparatus | |
| JP5413271B2 (en) | Surface inspection apparatus and surface inspection method | |
| JP5347661B2 (en) | Belt surface inspection apparatus, surface inspection method, and program | |
| WO2014181625A1 (en) | Surface inspection method and surface inspection device | |
| JP5992315B2 (en) | Surface defect detection device and surface defect detection method | |
| JP5710314B2 (en) | Mask inspection method and apparatus | |
| US9230337B2 (en) | Analysis of the digital image of the internal surface of a tyre and processing of false measurement points | |
| JP2016138761A (en) | Three-dimensional measurement method by optical cutting method and three-dimensional measuring instrument | |
| JP7136064B2 (en) | Apparatus for inspecting surface of object to be inspected and method for inspecting surface of object to be inspected | |
| JP2015200544A (en) | Surface irregularity inspection device and surface irregularity inspection method | |
| JP2008281493A (en) | Surface defect inspection system, method, and program | |
| KR102402386B1 (en) | Method, apparatus and the system for detecting thickness of an object | |
| JP5768349B2 (en) | Slit light intensity distribution design method and light cutting uneven surface wrinkle detecting device | |
| JP6146389B2 (en) | Steel surface inspection apparatus and method | |
| JP2006349598A (en) | Application irregularity detection method and apparatus | |
| JP5087165B1 (en) | Adjustment device for outputting data for adjusting surface inspection device, adjustment data output method and program | |
| JP2019078625A (en) | Inspection system and inspection method | |
| WO2023047866A1 (en) | Sheet-like material unevenness measuring device, and sheet-like material unevenness measuring method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| A621 | Written request for application examination |
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20160622 |
|
| A977 | Report on retrieval |
Free format text: JAPANESE INTERMEDIATE CODE: A971007 Effective date: 20170323 |
|
| A131 | Notification of reasons for refusal |
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20170404 |
|
| A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20170601 |
|
| A02 | Decision of refusal |
Free format text: JAPANESE INTERMEDIATE CODE: A02 Effective date: 20171017 |
|
| A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20180105 |
|
| A911 | Transfer to examiner for re-examination before appeal (zenchi) |
Free format text: JAPANESE INTERMEDIATE CODE: A911 Effective date: 20180115 |
|
| TRDD | Decision of grant or rejection written | ||
| A01 | Written decision to grant a patent or to grant a registration (utility model) |
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20180227 |
|
| A61 | First payment of annual fees (during grant procedure) |
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20180312 |
|
| R150 | Certificate of patent or registration of utility model |
Ref document number: 6314798 Country of ref document: JP Free format text: JAPANESE INTERMEDIATE CODE: R150 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |