JP2013092465A - Three-dimensional surface inspection device and three-dimensional surface inspection method - Google Patents

Three-dimensional surface inspection device and three-dimensional surface inspection method Download PDF

Info

Publication number
JP2013092465A
JP2013092465A JP2011235267A JP2011235267A JP2013092465A JP 2013092465 A JP2013092465 A JP 2013092465A JP 2011235267 A JP2011235267 A JP 2011235267A JP 2011235267 A JP2011235267 A JP 2011235267A JP 2013092465 A JP2013092465 A JP 2013092465A
Authority
JP
Japan
Prior art keywords
intensity distribution
measurement object
curved surface
pattern light
dimensional
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP2011235267A
Other languages
Japanese (ja)
Other versions
JP6099115B2 (en
Inventor
Soni Ro
存偉 盧
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NGP KK
Fukuoka Institute of Technology
Original Assignee
NGP KK
Fukuoka Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NGP KK, Fukuoka Institute of Technology filed Critical NGP KK
Priority to JP2011235267A priority Critical patent/JP6099115B2/en
Priority to CN201210417628.2A priority patent/CN103075979B/en
Publication of JP2013092465A publication Critical patent/JP2013092465A/en
Application granted granted Critical
Publication of JP6099115B2 publication Critical patent/JP6099115B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Length Measuring Devices By Optical Means (AREA)
  • Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional surface inspection device which allows a three-dimensional defect such as a recess and a large flaw on the surface to be accurately and rapidly inspected with a simple structure at low cost, even in the case of having acute mirror reflection from a measurement object.SOLUTION: A three-dimensional surface inspection device includes a camera device 3 which takes a picture of a measurement object X, a filter membrane which is arranged in a curve form so as to cover the measurement object, curve pattern light projection means 5 which projects pattern light having a curved intensity distribution with a periodic intensity distribution along the curve form of the filter membrane to the measurement object through the filter membrane, and defect detection means 13 which detects a three-dimensional defect in the surface of the measurement object through a decoding process to convert the curved intensity distribution to a linear intensity distribution for the picture taken by the camera device 3.

Description

本発明は、自動車やその部品などの計測対象物の表面の凹みや大きな疵などの三次元的欠陥を非接触で検査する三次元表面検査装置および三次元表面検査方法に関する。   The present invention relates to a three-dimensional surface inspection apparatus and a three-dimensional surface inspection method for inspecting non-contact three-dimensional defects such as dents and large wrinkles on the surface of a measurement object such as an automobile and its parts.

自動車のボディーの表面の疵、凹みや塗装の剥がれなどの検査は、自動車の生産やリサイクルにおいて非常に重要である。しかしながら、現状ではこれらの品質検査は、熟練工の目視に頼っており、作業員によりばらつきがあるだけでなく、作業員に過大な負担を掛けることになる。そのため、デジタルカメラやデジタルビデオカメラ等のデジタル式撮影装置を用い、非接触で高速かつ高精度に計測対象物の表面検査を行うことが可能な表面検査装置が望まれている。   Inspections such as wrinkles, dents and paint peeling on the surface of automobile bodies are very important in automobile production and recycling. However, at present, these quality inspections rely on the visual inspection of skilled workers, which not only varies depending on the worker, but also places an excessive burden on the worker. Therefore, there is a demand for a surface inspection apparatus capable of performing a surface inspection of a measurement object at high speed and with high accuracy in a non-contact manner using a digital photographing apparatus such as a digital camera or a digital video camera.

自動車のボディーのような三次元的形状を有する計測対象物を計測する三次元画像計測は、受動法と能動法に大別できる。典型的な受動計測法は、2台のカメラによるステレオ計測法が挙げられる。ステレオ計測法は、計測物体の各部分の画像特徴を利用して2台のカメラにより撮影した2枚の画像の対応点関係を明らかにし、その三次元座標を計算する手法であるので、物体の輪郭などの計測に好適であるが、緩やかな面などの特徴の少ない部分の計測には不向きである。   Three-dimensional image measurement for measuring a measurement object having a three-dimensional shape such as an automobile body can be broadly divided into a passive method and an active method. A typical passive measurement method includes a stereo measurement method using two cameras. The stereo measurement method is a technique for clarifying the corresponding point relationship between two images taken by two cameras using the image features of each part of the measurement object, and calculating the three-dimensional coordinates. Although it is suitable for measuring contours and the like, it is not suitable for measuring parts with few features such as gentle surfaces.

一方、典型的な能動計測法としては、パターン光投影計測法が挙げられる。パターン光投影計測法は、パターン光を計測対象物に投影することにより、能動的に計測対象物に特徴を付けるので、緩やかな面などの特徴の少ない部分の計測にも適応できる。   On the other hand, as a typical active measurement method, there is a pattern light projection measurement method. The pattern light projection measurement method actively applies a feature to the measurement object by projecting the pattern light onto the measurement object, so that it can be applied to measurement of a part having a small feature such as a gentle surface.

また、正弦波のようなパターン光を計測対象物に投影し、その複数枚の反射パターン画像に基づく位相シフト解析方法が知られている。例えば、パターン光投影に基づく三次元画像計測の計測時間の短縮と計測精度の向上のため、特許文献1,2に記載の手法が提案されている。特許文献1の手法では、最適強度変調パターン光の使用により、1回のパターン光の投影で三次元画像計測を実現でき、計測の時間短縮と精度向上が可能であるが、表面反射の強い自動車のボディーの表面計測には不向きである。   A phase shift analysis method is also known in which pattern light such as a sine wave is projected onto a measurement object and a plurality of reflection pattern images are used. For example, methods described in Patent Documents 1 and 2 have been proposed in order to shorten the measurement time of three-dimensional image measurement based on pattern light projection and improve measurement accuracy. In the method of Patent Document 1, the use of the optimum intensity modulation pattern light can realize three-dimensional image measurement by one-time projection of pattern light, and the measurement time can be shortened and the accuracy can be improved. It is not suitable for measuring the surface of the body.

また、特許文献2の手法では、より高い密度を持つ計測線パターンと基準線パターンの使用により、より高精度の三次元画像計測が実現できるが、ハイライト等の表面反射の強い計測対象物への応用が困難である。すなわち、一般のパターン光投影技術に基づく三次元画像計測は、自動車のボディーのような艶のある物体、すなわち鏡面反射の強い物体には適応困難である。   Further, in the method of Patent Document 2, more accurate three-dimensional image measurement can be realized by using a measurement line pattern having a higher density and a reference line pattern, but to a measurement object with strong surface reflection such as highlight. Is difficult to apply. That is, three-dimensional image measurement based on a general pattern light projection technique is difficult to apply to a glossy object such as an automobile body, that is, an object having a strong specular reflection.

そこで、鏡面反射の強い物体に対しては、以下の技術が提案されている。例えば、特許文献3の手法では、計測対象物に対して光源光が直接当たらないようにすることにより、鏡面反射が酷い計測対象物であってもその表面検査を可能としている。また、特許文献4に記載の手法では、直線状の明暗パターンを計測対象物に投影し、その反射パターンの境界点の解析により計測対象物の表面の凹凸の欠陥を検出するものであり、自動車表面の凹みの計測に対応可能となっている。   Therefore, the following techniques have been proposed for objects with strong specular reflection. For example, in the method of Patent Document 3, the surface inspection can be performed even on a measurement object with severe specular reflection by preventing the light source light from directly hitting the measurement object. In the method described in Patent Document 4, a linear light / dark pattern is projected onto a measurement object, and a defect on the surface of the measurement object is detected by analyzing a boundary point of the reflection pattern. It can be used to measure surface dents.

特開2006−145405号公報JP 2006-145405 A 特開2011−185872号公報JP 2011-185872 A 特開2010−185820号公報JP 2010-185820 A 特開2010−85165号公報JP 2010-85165 A

ところが、特許文献3の手法では、二次元式の画像計測技術を用いるので、自動車の表面疵等の二次元的な計測は可能であるものの、凹みの深さなどの三次元的な検査を行うことができない。また、特許文献4の手法は、パターンの境界点を主に用いた解析手法であり、二値化画像解析に似た解析手法であるため、凹凸の深さを精密に計測することが困難である。また、境界線のないところの凹凸の計測は困難である。   However, since the method of Patent Document 3 uses a two-dimensional image measurement technique, it can perform two-dimensional measurement such as surface flaws of automobiles, but performs three-dimensional inspections such as the depth of a dent. I can't. The method of Patent Document 4 is an analysis method mainly using pattern boundary points, and is an analysis method similar to binarized image analysis. Therefore, it is difficult to accurately measure the depth of unevenness. is there. In addition, it is difficult to measure unevenness where there is no boundary line.

そこで、本発明においては、鏡面反射が酷い計測対象物であっても、簡単な構造により低コストで高精度かつ高速に表面の凹みや大きな疵などの三次元的欠陥を検査することが可能な三次元表面検査装置および三次元表面検査方法を提供することを目的とする。   Therefore, in the present invention, it is possible to inspect three-dimensional defects such as surface dents and large wrinkles with high accuracy and high speed at a low cost with a simple structure even for a measurement object with severe specular reflection. It is an object to provide a three-dimensional surface inspection apparatus and a three-dimensional surface inspection method.

本発明の三次元表面検査装置は、計測対象物を撮影する撮影装置と、曲面状に配設されて計測対象物を覆うフィルタ膜と、フィルタ膜の曲面状に沿って周期的な強度分布を持つ曲面状強度分布のパターン光を投影し、フィルタ膜を介して計測対象物に投影する曲面パターン光投影手段と、撮影装置により撮影された画像に対し、曲面状強度分布を直線状強度分布に変換するデコード処理を行うことで、計測対象物の表面の三次元的欠陥を検出する欠陥検出手段とを含むものである。   The three-dimensional surface inspection apparatus of the present invention has an imaging device for photographing a measurement object, a filter film disposed in a curved surface and covering the measurement object, and a periodic intensity distribution along the curved surface of the filter film. Curved pattern light projection means for projecting the pattern light of the curved surface intensity distribution and projecting onto the measurement object through the filter film, and the curved surface intensity distribution to the linear intensity distribution for the image photographed by the photographing apparatus It includes a defect detection means for detecting a three-dimensional defect on the surface of the measurement object by performing a decoding process for conversion.

また、本発明の三次元表面検査方法は、曲面状に配設されて計測対象物を覆うフィルタ膜に対し、フィルタ膜の曲面状に沿って周期的な強度分布を持つ曲面状強度分布のパターン光を投影し、フィルタ膜を介して計測対象物に投影すること、計測対象物を撮影装置により撮影し、撮影された画像に対し、曲面状強度分布を直線状強度分布に変換するデコード処理を行うことで、計測対象物の表面の三次元的欠陥を検出することを含むことを特徴とする。   Further, the three-dimensional surface inspection method of the present invention is a curved surface intensity distribution pattern having a periodic intensity distribution along the curved surface shape of the filter film with respect to the filter film disposed in a curved surface and covering the measurement object. A process of projecting light and projecting it onto a measurement object through a filter film, photographing the measurement object with a photographing device, and converting a curved intensity distribution into a linear intensity distribution for the photographed image This includes detecting a three-dimensional defect on the surface of the measurement object.

これらの発明によれば、計測対象物に対してパターン光は直接投影されず、フィルタ膜を介して投影されるので、鏡面反射が酷い計測対象物であっても、その表面にパターン光の光源が直接映り込まないため、この計測対象物を普通に撮影することで、この撮影された画像から容易に計測対象物の表面欠陥を検出することができる。このとき、投影されるパターン光は、フィルタ膜の曲面状に沿って周期的な強度分布を持つが、この撮影された画像に対し、曲面状強度分布を直線状強度分布にデコード処理により変換すると、計測対象物の表面の三次元的欠陥部分は直線状強度分布から外れるので、この三次元的欠陥部分を検出することが可能となる。   According to these inventions, the pattern light is not directly projected onto the measurement object, but is projected through the filter film. Therefore, even if the measurement object has severe specular reflection, the light source of the pattern light on the surface Therefore, the surface defect of the measurement object can be easily detected from the photographed image by normally photographing the measurement object. At this time, the projected pattern light has a periodic intensity distribution along the curved surface shape of the filter film. However, if the curved surface intensity distribution is converted into a linear intensity distribution by decoding processing on the photographed image. Since the three-dimensional defect portion on the surface of the measurement object deviates from the linear intensity distribution, this three-dimensional defect portion can be detected.

本発明によれば、鏡面反射が酷い計測対象物であっても、その表面にパターン光の光源が直接映り込まないため、この計測対象物を普通に撮影することで、この撮影された画像から容易に計測対象物の表面欠陥を検出することができ、計測対象物表面の凹みや大きな疵などの三次元的欠陥部分を低コストで高精度かつ高速に検出することが可能となる。   According to the present invention, even if the object to be measured has a severe specular reflection, the light source of the pattern light is not directly reflected on the surface thereof. The surface defect of the measurement object can be easily detected, and a three-dimensional defect portion such as a dent or a large wrinkle on the measurement object surface can be detected with high accuracy and high speed at low cost.

本発明の実施の形態における三次元表面検査装置の全体構成図である。1 is an overall configuration diagram of a three-dimensional surface inspection apparatus in an embodiment of the present invention. 図1の三次元表面検査装置の詳細な構成を示すブロック図である。It is a block diagram which shows the detailed structure of the three-dimensional surface inspection apparatus of FIG. 本実施形態における三次元表面検査装置による検査処理のフロー図である。It is a flowchart of the inspection process by the three-dimensional surface inspection apparatus in this embodiment. 本発明の実施形態における表面検査装置の曲面状空間強度分布パターン光生成のイメージ図である。It is an image figure of curved-surface space intensity distribution pattern light generation of the surface inspection apparatus in the embodiment of the present invention. 本発明の実施形態における表面検査装置の投影パターンのその他の実施例である。It is another Example of the projection pattern of the surface inspection apparatus in embodiment of this invention. (a)は計測対象物の表面に凹みなどの形状変化がない場合の画像の強度分布を示す図、(b)は(a)のデコード解析結果の強度分布を示す図である。(A) is a figure which shows intensity distribution of the image when there is no shape change, such as a dent, on the surface of a measurement object, (b) is a figure which shows intensity distribution of the decoding analysis result of (a). (a)は計測対象物の表面に凹みなどの形状変化がある場合の画像の強度分布を示す図、(b)は(a)のデコード解析結果の強度分布を示す図である。(A) is a figure which shows intensity distribution of an image in case there exists shape changes, such as a dent, on the surface of a measurement target object, (b) is a figure which shows intensity distribution of the decoding analysis result of (a).

以下、本発明の実施の形態における三次元表面検査装置について、図面を用いて説明する。図1は本発明の実施の形態における三次元表面検査装置の全体構成図、図2は図1の三次元表面検査装置の詳細な構成を示すブロック図である。   Hereinafter, a three-dimensional surface inspection apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is an overall configuration diagram of a three-dimensional surface inspection apparatus according to an embodiment of the present invention, and FIG. 2 is a block diagram showing a detailed configuration of the three-dimensional surface inspection apparatus of FIG.

図1に示すように、本発明の実施の形態における三次元表面検査装置は、自動車のボディーやその部品などの計測対象物Xの表面の疵や凹みなどの三次元的欠陥を非接触で検査する装置であり、線状光源1と、線状光源1と計測対象物Xとの間に配置され、線状光源1の光が計測対象物Xに直接当たらないように前記線状光源1の光を均一に減光するフィルタ膜2と、計測対象物Xを撮影する撮影装置としてのカメラ装置3と、線状光源1およびカメラ装置3が接続されるコンピュータ4とから構成される。   As shown in FIG. 1, the three-dimensional surface inspection apparatus in the embodiment of the present invention inspects three-dimensional defects such as wrinkles and dents on the surface of a measurement object X such as an automobile body and its parts in a non-contact manner. The linear light source 1 is disposed between the linear light source 1 and the measurement object X so that the light from the linear light source 1 does not directly hit the measurement object X. It comprises a filter film 2 that uniformly attenuates light, a camera device 3 as a photographing device for photographing the measurement object X, and a computer 4 to which the linear light source 1 and the camera device 3 are connected.

線状光源1は、普通の市販されている例えば、蛍光灯やLEDなどの線状の照明機器である。フィルタ膜2は、半透明な材料によりドーム状やトンネル状などの曲面状に形成されたものであり、線状光源1からの光を減光および拡散するものである。線状光源1は、このフィルタ膜2の曲面状に沿って複数本配置されている。   The linear light source 1 is an ordinary commercially available linear illumination device such as a fluorescent lamp or LED. The filter film 2 is formed in a curved shape such as a dome shape or a tunnel shape using a translucent material, and dimmes and diffuses the light from the linear light source 1. A plurality of linear light sources 1 are arranged along the curved surface of the filter film 2.

フィルタ膜2の透過率は、線状光源1の光量および計測対象物Xの表面反射率に応じて適切に選ばれるが、一般的には5%から95%までの範囲である。カメラ装置3はデジタル式カメラである。カメラ装置3は画像のデジタルデータを取得可能な撮影装置であればよく、8ビット、10ビット、12ビットや16ビット等のもの、3CCD、1CCDやCMOS等のイメージセンサを用いたもの、静止画カメラ、動画カメラやビデオカメラ等のどのようなものでもよい。   The transmittance of the filter film 2 is appropriately selected according to the light amount of the linear light source 1 and the surface reflectance of the measurement object X, but is generally in the range of 5% to 95%. The camera device 3 is a digital camera. The camera device 3 may be any photographing device capable of acquiring digital image data, such as an 8-bit, 10-bit, 12-bit, or 16-bit device, an image sensor such as a 3CCD, 1CCD, or CMOS, or a still image. Any device such as a camera, a video camera, and a video camera may be used.

コンピュータ4は、線状光源1およびカメラ装置3を接続するためのインターフェース(図示せず。)を備えており、図示しない検査プログラムの実行により、図2に示す記憶手段10、投影パターン光制御手段11、写真撮影手段12、欠陥検出手段13、寸法推定手段14、検査結果表現手段15および出力手段16として機能する。   The computer 4 includes an interface (not shown) for connecting the linear light source 1 and the camera device 3, and the storage means 10 and the projection pattern light control means shown in FIG. 2 are executed by executing an inspection program (not shown). 11, functions as a photographing unit 12, a defect detection unit 13, a size estimation unit 14, an inspection result expression unit 15 and an output unit 16.

投影パターン光制御手段11は、複数本の線状光源1を制御、例えばオン−オフ制御することにより、フィルタ膜2の曲面状に沿って周期的な強度分布を持つ曲面状強度分布のパターン光を投影するものである。この投影パターン光制御手段11による線状光源1の制御データは、記憶手段10に記憶されている。   The projection pattern light control means 11 controls a plurality of linear light sources 1, for example, on / off control, thereby causing a curved intensity distribution pattern light having a periodic intensity distribution along the curved surface of the filter film 2. Is projected. Control data of the linear light source 1 by the projection pattern light control means 11 is stored in the storage means 10.

写真撮影手段12は、検査に適用できる画像を撮影するために必要な最適なパラメータ(以下、「撮影パラメータ」と称す。)でカメラ装置3を制御して、計測対象物Xの表面の撮影を行い、撮影された画像データを記憶装置10に記憶させるものである。このカメラ装置3の制御に必要な撮影パラメータ等は、記憶手段10に記憶されている。また、記憶手段10には、後述する各手段により算出された結果のデータ等も記憶される。   The photographic means 12 controls the camera device 3 with the optimum parameters (hereinafter referred to as “imaging parameters”) necessary for photographing an image applicable to the examination, and photographs the surface of the measurement object X. The image data taken is stored in the storage device 10. Shooting parameters and the like necessary for controlling the camera device 3 are stored in the storage unit 10. Further, the storage means 10 also stores data of results calculated by each means described later.

欠陥検出手段13は、写真撮影手段12により入力された画像データの解析により、計測対象物Xの表面にある凹みや大きな疵等の三次元的欠陥を検出するものである。寸法推定手段14は、検出された三次元的欠陥の寸法(径や深さ等)を画像解析により推定するものである。   The defect detection means 13 detects three-dimensional defects such as dents and large wrinkles on the surface of the measurement object X by analyzing the image data input by the photography means 12. The size estimation means 14 estimates the size (diameter, depth, etc.) of the detected three-dimensional defect by image analysis.

検査結果表現手段15は、三次元的欠陥の位置、形状やサイズ等の検査結果を、ユーザが分かりやすい方法でコンピュータ4のモニタ上で表現するものである。また、検査結果表現手段15は、必要に応じてモニタ上で検査結果を拡大表示、縮小表示、全体表示や部分表示等させる機能を備えている。出力手段16は、写真撮影した画像や検査結果等を、画像、テキストファイル、表、グラフや数値などの方式で、各種のメディアに保存したり、出力したりするものである。   The inspection result expression means 15 expresses inspection results such as the position, shape and size of the three-dimensional defect on the monitor of the computer 4 in a manner that is easy for the user to understand. Further, the inspection result expression means 15 has a function of enlarging display, reduction display, whole display, partial display, etc. on the monitor as necessary. The output unit 16 stores or outputs a photographed image, an inspection result, and the like in various media using an image, a text file, a table, a graph, a numerical value, and the like.

なお、図2に示すように、曲面パターン光投影手段5は、線状光源1と投影パターン光制御手段11により構成される。なお、線状光源1は、必要とする投影パターン光を生成するために、予め、その総合的な位置関係と、互いの間隔と、フィルタ膜2との間の距離を調整しておく。   As shown in FIG. 2, the curved pattern light projection unit 5 includes a linear light source 1 and a projection pattern light control unit 11. The linear light source 1 adjusts the overall positional relationship, the distance between each other, and the distance between the filter films 2 in advance in order to generate the necessary projection pattern light.

次に、本実施形態における三次元表面検査装置による計測対象物Xの表面の凹み等の三次元的欠陥部分の検査処理について説明する。図3は本実施形態における三次元表面検査装置による検査処理のフロー図である。   Next, an inspection process for a three-dimensional defect portion such as a dent on the surface of the measurement object X by the three-dimensional surface inspection apparatus according to the present embodiment will be described. FIG. 3 is a flowchart of inspection processing by the three-dimensional surface inspection apparatus in the present embodiment.

本表面検査装置を使用する前に、線状光源1を設定する。設定は、まず線状光源1の位置、各線状光源1の間隔、すなわち各線状光源1間の距離を調節する(S101)。続いて、線状光源1とフィルタ膜2との間の距離を調節し、曲面状強度分布パターン光の生成のための環境を備える(S102)。上記の設定は、毎回計測の前に行う必要はなく、表面検査装置の構築の際に一回だけ行えば良い。   Before using the surface inspection apparatus, the linear light source 1 is set. In the setting, first, the position of the linear light source 1 and the interval between the linear light sources 1, that is, the distance between the linear light sources 1 is adjusted (S 101). Subsequently, the distance between the linear light source 1 and the filter film 2 is adjusted to provide an environment for generating curved-surface intensity distribution pattern light (S102). The above setting does not need to be performed before each measurement, and may be performed only once when the surface inspection apparatus is constructed.

続いて、コンピュータ4の投影パターン光制御手段11の指令により、各線状光源1のオン−オフを制御し、計測のための曲面状空間強度分布パターン光を生成する(S103)。図4は曲面状強度分布パターン光の生成のイメージ図であり、(a)は線状光源1の配置と、フィルタ膜2の表面の曲面状に沿って周辺に生成したパターン光の強度分布のイメージ図、(b)は(a)のフィルタ膜2を平面上に展開(円弧Rを直線化)した際のパターン光の強度分布のイメージ図である。図4(b)に示すように、同図(a)の曲面状強度分布のパターン光の強度分布は、平面上に展開すると強度分布が正弦波となるパターン光である。   Subsequently, on / off of each linear light source 1 is controlled by a command from the projection pattern light control means 11 of the computer 4 to generate curved spatial intensity distribution pattern light for measurement (S103). FIG. 4 is an image diagram of the generation of curved-surface intensity distribution pattern light. FIG. 4A is an image diagram of the arrangement of the linear light sources 1 and the intensity distribution of pattern light generated around the curved surface of the surface of the filter film 2. (B) is an image figure of intensity distribution of pattern light when filter film 2 of (a) is developed on a plane (arc R is straightened). As shown in FIG. 4B, the intensity distribution of the pattern light having the curved surface intensity distribution in FIG. 4A is a pattern light whose intensity distribution becomes a sine wave when developed on a plane.

次に、コンピュータ4の写真撮影手段12がカメラ装置3の撮影パラメータを調節し、計測対象物Xから反射された曲面状強度分布パターン光をカメラ装置3により撮影し、画像をコンピュータ4に入力する(S104)。写真撮影手段12は入力した画像の強度分布を解析し、欠陥検出に理想的な画像かどうかを判断する。理想的な画像であれば次の処理に進むが、理想的な画像でなければ、理想的な画像が撮れるまで繰り返しカメラ装置3の撮影パラメータを調節し、撮影する(S105)。   Next, the photographic means 12 of the computer 4 adjusts the photographic parameters of the camera device 3, the curved intensity distribution pattern light reflected from the measurement object X is photographed by the camera device 3, and the image is input to the computer 4. (S104). The photography unit 12 analyzes the intensity distribution of the input image and determines whether the image is ideal for defect detection. If the image is an ideal image, the process proceeds to the next process. If the image is not an ideal image, the image capturing parameter of the camera device 3 is adjusted repeatedly until an ideal image is captured, and the image is captured (S105).

理想的な画像が撮影されたら、欠陥検出手段13により曲面状強度分布のパターン光である正弦波を直線状強度分布に変換するデコード処理を行う(S106)。このとき、計測対象物Xの表面に凹みなどの形状変化がなければ、撮影された画像の強度分布は図6(a)に示すような綺麗な正弦波分布なので、デコードの結果は同図(b)に示すような直線状の強度分布になる。一方、計測対象物Xの表面に凹みなどの形状変化があれば、図7(a)に示すように凹みなどの欠陥部分の画像の強度の位相分布に変化が生じ、綺麗な正弦波分布は得られなくなる。そのため、デコードすると、同図(b)に示すように、この欠陥部分は直線から外れるので、この直線から外れた部分に凹みなどの欠陥が存在すると判定できる(S107)。   When an ideal image is taken, the defect detection means 13 performs a decoding process for converting a sine wave, which is a pattern light of a curved intensity distribution, into a linear intensity distribution (S106). At this time, if there is no shape change such as a dent on the surface of the measurement object X, the intensity distribution of the photographed image is a clean sine wave distribution as shown in FIG. It becomes a linear intensity distribution as shown in b). On the other hand, if there is a shape change such as a dent on the surface of the measurement object X, the intensity phase distribution of the image of the defective part such as the dent changes as shown in FIG. It can no longer be obtained. Therefore, when the decoding is performed, as shown in FIG. 5B, the defective portion deviates from the straight line, so that it can be determined that there is a defect such as a dent in the portion deviating from the straight line (S107).

また、寸法推定手段14により凹みの寸法および深さを推定する。図7(b)に示す直線から外れた範囲は凹みの寸法と比例し、直線から外れた程度は凹みの深さと比例するため、寸法推定手段14は、直線から外れた範囲と程度より、凹みの寸法と深さを推定できる(S108)。この検査結果は、検査結果表現手段15によりコンピュータ4のモニタ上でコンピュータグラフィックスによる表現ができ(S109)、出力手段16により各種の記憶メディアに保存したり、出力したりすることができる(S110)。   Further, the size and depth of the dent are estimated by the size estimation means 14. The range deviating from the straight line shown in FIG. 7B is proportional to the size of the dent, and the extent deviating from the straight line is proportional to the depth of the dent. Can be estimated (S108). The inspection result can be expressed by computer graphics on the monitor of the computer 4 by the inspection result expression means 15 (S109), and can be stored in various storage media or output by the output means 16 (S110). ).

なお、上記実施形態では、曲面状強度分布のパターン光として正弦波状の強度分布を持つパターン光を用いたが、その他のパターン光として、図5に示すような台形波状や矩形波状等の周期的な強度分布を持つパターン光を使用することも可能である。   In the above embodiment, pattern light having a sinusoidal intensity distribution is used as the pattern light of the curved surface intensity distribution. However, as other pattern light, a periodic light such as a trapezoidal wave or a rectangular wave as shown in FIG. It is also possible to use pattern light having a strong intensity distribution.

本実施形態における三次元表面検査装置では、計測対象物Xに対してパターン光は直接投影されず、フィルタ膜2を介して投影され、フィルタ膜2により減光および拡散されるので、鏡面反射が酷い計測対象物Xであっても、その表面にパターン光の光源が直接映り込まないため、この計測対象物Xを普通に撮影することで、この撮影された画像から容易に計測対象物Xの表面欠陥を検出することが可能となっており、数秒から数十秒の短時間で処理することが可能である。   In the three-dimensional surface inspection apparatus according to the present embodiment, the pattern light is not directly projected onto the measurement object X, but is projected through the filter film 2 and is attenuated and diffused by the filter film 2. Even if it is a terrible measurement object X, the light source of the pattern light is not directly reflected on its surface. Therefore, by photographing the measurement object X normally, the measurement object X can be easily obtained from the captured image. Surface defects can be detected, and processing can be performed in a short time of several seconds to several tens of seconds.

また、計測対象物Xに投影されるパターン光は、フィルタ膜2の曲面状に沿って周期的な強度分布を持ち、撮影された画像に対し、曲面状強度分布を直線状強度分布にデコード処理により変換すると、計測対象物Xの表面の三次元的欠陥部分は直線状強度分布から外れるので、この三次元的欠陥部分を容易に検出することが可能となっており、数秒から数十秒で、凹み等の三次元的欠陥の検査を行うことが可能である。   Further, the pattern light projected onto the measurement object X has a periodic intensity distribution along the curved surface shape of the filter film 2, and the curved surface intensity distribution is decoded into a linear intensity distribution for the photographed image. Since the three-dimensional defect portion on the surface of the measurement object X deviates from the linear intensity distribution, it is possible to easily detect this three-dimensional defect portion in a few seconds to a few tens of seconds. It is possible to inspect three-dimensional defects such as dents.

本発明の三次元表面検査装置および三次元表面検査方法は、自動車やその部品などの計測対象物の表面の凹みや大きな疵などの三次元的欠陥を非接触で検査する装置および方法として有用であり、特に鏡面反射の酷い計測対象物の検査に好適である。   INDUSTRIAL APPLICABILITY The three-dimensional surface inspection apparatus and the three-dimensional surface inspection method of the present invention are useful as an apparatus and method for inspecting three-dimensional defects such as dents and large wrinkles on the surface of measurement objects such as automobiles and parts thereof in a non-contact manner. In particular, it is suitable for inspection of a measurement object having a severe specular reflection.

X 計測対象物
1 線状光源
2 フィルタ膜
3 カメラ装置
4 コンピュータ
5 曲面パターン光投影手段
10 記憶手段
11 投影パターン光制御手段
12 写真撮影手段
13 欠陥検出手段
14 寸法推定手段
15 検査結果表現手段
16 出力手段
X Measurement object 1 Linear light source 2 Filter film 3 Camera device 4 Computer 5 Curved pattern light projection means 10 Storage means 11 Projection pattern light control means 12 Photographing means 13 Defect detection means 14 Dimension estimation means 15 Inspection result expression means 16 Output means

Claims (4)

計測対象物を撮影する撮影装置と、
曲面状に配設されて前記計測対象物を覆うフィルタ膜と、
前記フィルタ膜の曲面状に沿って周期的な強度分布を持つ曲面状強度分布のパターン光を投影し、前記フィルタ膜を介して前記計測対象物に投影する曲面パターン光投影手段と、
前記撮影装置により撮影された画像に対し、前記曲面状強度分布を直線状強度分布に変換するデコード処理を行うことで、前記計測対象物の表面の三次元的欠陥を検出する欠陥検出手段と
を含む三次元表面検査装置。
A photographing device for photographing a measurement object;
A filter film disposed in a curved surface and covering the measurement object;
A curved surface pattern light projection means for projecting a pattern light of a curved surface intensity distribution having a periodic intensity distribution along the curved surface of the filter film, and projecting it onto the measurement object via the filter film;
Defect detection means for detecting a three-dimensional defect on the surface of the measurement object by performing a decoding process for converting the curved surface intensity distribution into a linear intensity distribution on the image photographed by the photographing apparatus. Including 3D surface inspection equipment.
前記曲面パターン光投影手段は、
前記フィルタ膜の曲面状に沿って複数本配置された線状光源と、
前記複数本の線状光源を制御して前記フィルタ膜の表面形状に沿って周期的な強度分布を持つ曲面状強度分布のパターン光を構成する投影パターン光制御手段と
を含むものである請求項1記載の三次元表面検査装置。
The curved surface pattern light projection means includes:
A plurality of linear light sources arranged along the curved surface of the filter film;
2. A projection pattern light control means for controlling the plurality of linear light sources to form a curved light intensity distribution pattern light having a periodic intensity distribution along the surface shape of the filter film. 3D surface inspection equipment.
前記曲面状強度分布のパターン光は、正弦波状、矩形波状または台形波状の周期的な強度分布である請求項1または2に記載の三次元表面検査装置。   The three-dimensional surface inspection apparatus according to claim 1, wherein the pattern light of the curved surface intensity distribution is a periodic intensity distribution having a sine wave shape, a rectangular wave shape, or a trapezoidal wave shape. 曲面状に配設されて計測対象物を覆うフィルタ膜に対し、前記フィルタ膜の曲面状に沿って周期的な強度分布を持つ曲面状強度分布のパターン光を投影し、前記フィルタ膜を介して前記計測対象物に投影すること、
前記計測対象物を撮影装置により撮影し、撮影された画像に対し、前記曲面状強度分布を直線状強度分布に変換するデコード処理を行うことで、前記計測対象物の表面の三次元的欠陥を検出すること
を含む三次元表面検査方法。
A pattern light of a curved surface intensity distribution having a periodic intensity distribution is projected along the curved surface shape of the filter film on a filter film that is arranged in a curved surface and covers the measurement object, and passes through the filter film. Projecting onto the measurement object;
The measurement object is imaged by an imaging device, and a three-dimensional defect on the surface of the measurement object is detected by performing a decoding process on the captured image to convert the curved surface intensity distribution into a linear intensity distribution. A three-dimensional surface inspection method comprising detecting.
JP2011235267A 2011-10-26 2011-10-26 Three-dimensional surface inspection apparatus and three-dimensional surface inspection method Active JP6099115B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2011235267A JP6099115B2 (en) 2011-10-26 2011-10-26 Three-dimensional surface inspection apparatus and three-dimensional surface inspection method
CN201210417628.2A CN103075979B (en) 2011-10-26 2012-10-26 Three-dimensional surface detecting device and three-dimensional surface detecting method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2011235267A JP6099115B2 (en) 2011-10-26 2011-10-26 Three-dimensional surface inspection apparatus and three-dimensional surface inspection method

Publications (2)

Publication Number Publication Date
JP2013092465A true JP2013092465A (en) 2013-05-16
JP6099115B2 JP6099115B2 (en) 2017-03-22

Family

ID=48152593

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2011235267A Active JP6099115B2 (en) 2011-10-26 2011-10-26 Three-dimensional surface inspection apparatus and three-dimensional surface inspection method

Country Status (2)

Country Link
JP (1) JP6099115B2 (en)
CN (1) CN103075979B (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017015647A (en) * 2015-07-06 2017-01-19 株式会社豊田中央研究所 Surface inspection device
CN111260611A (en) * 2020-01-09 2020-06-09 五邑大学 Surface defect detection method and system for reflecting curved surface
CN111650215A (en) * 2020-07-07 2020-09-11 福建泉州源始物语信息科技有限公司 Cosmetic bottle cap flaw detection system

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104574408A (en) * 2015-01-16 2015-04-29 东华大学 Industry transparent film package detecting method and device based on shape feature extraction
CN105627942A (en) * 2015-12-25 2016-06-01 华南理工大学 Imaging apparatus and method for detecting object surface micro deformation through machine vision
JP7056131B2 (en) * 2017-12-15 2022-04-19 オムロン株式会社 Image processing system, image processing program, and image processing method
CN109975324A (en) * 2019-05-05 2019-07-05 苏州天准科技股份有限公司 A kind of Structured Illumination flaw detection method based on square wave configuration
CN113030101A (en) * 2021-03-02 2021-06-25 成都小淞科技有限公司 Stripe light coating flaw detection device

Citations (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0850103A (en) * 1994-08-06 1996-02-20 Mazda Motor Corp Coating surface check device
JPH08184567A (en) * 1994-10-05 1996-07-16 Musco Corp Device and method for inspecting mirror reflection surface or half-mirror reflection surface
JPH09318337A (en) * 1996-05-31 1997-12-12 Nissan Motor Co Ltd Surface defect inspecting device
JPH1096615A (en) * 1996-01-18 1998-04-14 Nissan Motor Co Ltd Painting defect inspection device
JPH10318938A (en) * 1997-05-20 1998-12-04 Nissan Motor Co Ltd Surface-inspection apparatus
JPH1163959A (en) * 1997-08-25 1999-03-05 Nissan Motor Co Ltd Surface-inspecting device
JPH11118731A (en) * 1997-10-20 1999-04-30 Nissan Motor Co Ltd Method and apparatus for inspecting defect on surface to be inspected
JPH11118443A (en) * 1997-10-15 1999-04-30 Ckd Corp Shape measuring device
JPH11271038A (en) * 1998-03-24 1999-10-05 Nissan Motor Co Ltd Painting defect inspection device
JP2000136917A (en) * 1998-10-30 2000-05-16 Moritex Corp Observation method for surface of molding and illumination device used for it
JP2000321039A (en) * 1999-05-14 2000-11-24 Nissan Motor Co Ltd Apparatus and method for inspecting coating fault
JP2002286656A (en) * 2001-01-22 2002-10-03 Hitachi Electronics Eng Co Ltd Substrate inspection apparatus
JP2004109106A (en) * 2002-07-22 2004-04-08 Fujitsu Ltd Method and apparatus for inspecting surface defect
JP2006145405A (en) * 2004-11-19 2006-06-08 Fukuoka Institute Of Technology Three-dimensional measuring instrument, method, and program
JP2006293124A (en) * 2005-04-13 2006-10-26 Seiko Epson Corp Projector
JP2006329898A (en) * 2005-05-30 2006-12-07 Toyota Motor Corp Measuring method and device of surface distortion
JP2007183225A (en) * 2006-01-10 2007-07-19 Toyota Motor Corp Light radiation device, surface shape inspection system, and surface shape inspection method
JP2008292365A (en) * 2007-05-25 2008-12-04 Toyota Motor Corp Shape evaluation method, shape evaluation device, and three-dimensional inspection device
JP2010085165A (en) * 2008-09-30 2010-04-15 Toray Ind Inc Surface inspection device and surface inspection method
JP2010185820A (en) * 2009-02-13 2010-08-26 Fukuoka Institute Of Technology Device and method for surface inspection
JP2010532870A (en) * 2007-07-12 2010-10-14 カールツァイス アーゲー Optical inspection method and inspection apparatus for object surface
JP2010281665A (en) * 2009-06-04 2010-12-16 Yamaha Motor Co Ltd Phase shift imaging device, component transfer device, and phase shift imaging method
JP2011179982A (en) * 2010-03-01 2011-09-15 Fukuoka Institute Of Technology Surface inspection apparatus, method of inspecting surface, and surface inspection program
JP2011185872A (en) * 2010-03-10 2011-09-22 Canon Inc Information processor, and processing method and program of the same

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101694375B (en) * 2009-10-23 2011-06-22 北京航空航天大学 Stereoscopic vision detecting method for measuring three-dimensional morphology on strong reflection surface
CN101813462A (en) * 2010-04-16 2010-08-25 天津理工大学 Three-dimensional feature optical measuring system controlled by uniprocessor and measuring method
CN101881605B (en) * 2010-06-02 2011-11-30 南京航空航天大学 Optical three-dimensional measuring method based on phase coding technology

Patent Citations (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0850103A (en) * 1994-08-06 1996-02-20 Mazda Motor Corp Coating surface check device
JPH08184567A (en) * 1994-10-05 1996-07-16 Musco Corp Device and method for inspecting mirror reflection surface or half-mirror reflection surface
US5636024A (en) * 1994-10-05 1997-06-03 Musco Corporation Apparatus and method of inspection of specular and semi-specular surfaces
JPH1096615A (en) * 1996-01-18 1998-04-14 Nissan Motor Co Ltd Painting defect inspection device
JPH09318337A (en) * 1996-05-31 1997-12-12 Nissan Motor Co Ltd Surface defect inspecting device
JPH10318938A (en) * 1997-05-20 1998-12-04 Nissan Motor Co Ltd Surface-inspection apparatus
JPH1163959A (en) * 1997-08-25 1999-03-05 Nissan Motor Co Ltd Surface-inspecting device
JPH11118443A (en) * 1997-10-15 1999-04-30 Ckd Corp Shape measuring device
JPH11118731A (en) * 1997-10-20 1999-04-30 Nissan Motor Co Ltd Method and apparatus for inspecting defect on surface to be inspected
JPH11271038A (en) * 1998-03-24 1999-10-05 Nissan Motor Co Ltd Painting defect inspection device
JP2000136917A (en) * 1998-10-30 2000-05-16 Moritex Corp Observation method for surface of molding and illumination device used for it
JP2000321039A (en) * 1999-05-14 2000-11-24 Nissan Motor Co Ltd Apparatus and method for inspecting coating fault
JP2002286656A (en) * 2001-01-22 2002-10-03 Hitachi Electronics Eng Co Ltd Substrate inspection apparatus
JP2004109106A (en) * 2002-07-22 2004-04-08 Fujitsu Ltd Method and apparatus for inspecting surface defect
JP2006145405A (en) * 2004-11-19 2006-06-08 Fukuoka Institute Of Technology Three-dimensional measuring instrument, method, and program
JP2006293124A (en) * 2005-04-13 2006-10-26 Seiko Epson Corp Projector
JP2006329898A (en) * 2005-05-30 2006-12-07 Toyota Motor Corp Measuring method and device of surface distortion
JP2007183225A (en) * 2006-01-10 2007-07-19 Toyota Motor Corp Light radiation device, surface shape inspection system, and surface shape inspection method
JP2008292365A (en) * 2007-05-25 2008-12-04 Toyota Motor Corp Shape evaluation method, shape evaluation device, and three-dimensional inspection device
JP2010532871A (en) * 2007-07-12 2010-10-14 カールツァイス アーゲー Method and apparatus for optical inspection of the surface of an object
JP2010532870A (en) * 2007-07-12 2010-10-14 カールツァイス アーゲー Optical inspection method and inspection apparatus for object surface
JP2010085165A (en) * 2008-09-30 2010-04-15 Toray Ind Inc Surface inspection device and surface inspection method
JP2010185820A (en) * 2009-02-13 2010-08-26 Fukuoka Institute Of Technology Device and method for surface inspection
JP2010281665A (en) * 2009-06-04 2010-12-16 Yamaha Motor Co Ltd Phase shift imaging device, component transfer device, and phase shift imaging method
JP2011179982A (en) * 2010-03-01 2011-09-15 Fukuoka Institute Of Technology Surface inspection apparatus, method of inspecting surface, and surface inspection program
JP2011185872A (en) * 2010-03-10 2011-09-22 Canon Inc Information processor, and processing method and program of the same

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017015647A (en) * 2015-07-06 2017-01-19 株式会社豊田中央研究所 Surface inspection device
CN111260611A (en) * 2020-01-09 2020-06-09 五邑大学 Surface defect detection method and system for reflecting curved surface
CN111260611B (en) * 2020-01-09 2023-03-21 深圳市瑞天激光有限公司 Surface defect detection method and system for reflecting curved surface
CN111650215A (en) * 2020-07-07 2020-09-11 福建泉州源始物语信息科技有限公司 Cosmetic bottle cap flaw detection system
CN111650215B (en) * 2020-07-07 2023-02-14 福建泉州源始物语信息科技有限公司 Cosmetic bottle cap flaw detection system

Also Published As

Publication number Publication date
JP6099115B2 (en) 2017-03-22
CN103075979B (en) 2017-02-01
CN103075979A (en) 2013-05-01

Similar Documents

Publication Publication Date Title
JP6099115B2 (en) Three-dimensional surface inspection apparatus and three-dimensional surface inspection method
JP4784555B2 (en) Shape evaluation method, shape evaluation apparatus, and three-dimensional inspection apparatus
JP5728699B2 (en) Surface inspection apparatus, surface inspection method, and surface inspection program
JP6937642B2 (en) Surface evaluation method and surface evaluation device
JP5014003B2 (en) Inspection apparatus and method
KR20150128300A (en) method of making three dimension model and defect analysis using camera and laser scanning
WO2018163530A1 (en) Three-dimensional shape measurement device, three-dimensional shape measurement method, and program
US9990724B2 (en) Image recording simulation in a coordinate measuring machine
JP2007322162A (en) Three-dimensional shape measuring apparatus and three-dimensional shape measuring method
Pösch et al. Rigid and flexible endoscopes for three dimensional measurement of inside machine parts using fringe projection
JPH1038533A (en) Instrument and method for measuring shape of tire
JP2016224707A (en) Inspection system
JP5224288B2 (en) Surface inspection apparatus and surface inspection method
JP2011064617A (en) Three-dimensional information measuring device and three-dimensional information measuring method
JP6276092B2 (en) Inspection system and inspection method
JP7159624B2 (en) Surface texture inspection method and surface texture inspection device
JP6420131B2 (en) Inspection system and inspection method
KR101555027B1 (en) Appapatus for three-dimensional shape measurment and method the same
JP5895733B2 (en) Surface defect inspection apparatus and surface defect inspection method
TWM345227U (en) An optical fringe projection measuring system that can measure the surface roughness and the profile of sheet materials at the same time
JP2007315946A (en) Three-dimensional shape measurement method and three-dimensional shape measuring device using this
JP2017016169A (en) Inspection method, inspection apparatus, image processing apparatus, program, and recording medium
JP5214323B2 (en) Visual inspection apparatus and method
Huang A novel direct structured-light inspection technique for contaminant and defect detection
JP2009069063A (en) Measurement method, shape measurement method, measuring device, and shape measuring apparatus

Legal Events

Date Code Title Description
A711 Notification of change in applicant

Free format text: JAPANESE INTERMEDIATE CODE: A711

Effective date: 20140512

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20140515

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A821

Effective date: 20140514

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20150218

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20150303

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20150421

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20150818

A02 Decision of refusal

Free format text: JAPANESE INTERMEDIATE CODE: A02

Effective date: 20160216

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20160428

A911 Transfer to examiner for re-examination before appeal (zenchi)

Free format text: JAPANESE INTERMEDIATE CODE: A911

Effective date: 20160511

A912 Re-examination (zenchi) completed and case transferred to appeal board

Free format text: JAPANESE INTERMEDIATE CODE: A912

Effective date: 20160701

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20170216

R150 Certificate of patent or registration of utility model

Ref document number: 6099115

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250