JP2009210409A - Method and device for image area division - Google Patents

Method and device for image area division Download PDF

Info

Publication number
JP2009210409A
JP2009210409A JP2008053500A JP2008053500A JP2009210409A JP 2009210409 A JP2009210409 A JP 2009210409A JP 2008053500 A JP2008053500 A JP 2008053500A JP 2008053500 A JP2008053500 A JP 2008053500A JP 2009210409 A JP2009210409 A JP 2009210409A
Authority
JP
Japan
Prior art keywords
region
image
area
biopsy
color space
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
JP2008053500A
Other languages
Japanese (ja)
Inventor
Tomohito Ueno
智史 上野
Masayuki Hashimoto
真幸 橋本
Atsushi Koike
淳 小池
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
KDDI Corp
Original Assignee
KDDI Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by KDDI Corp filed Critical KDDI Corp
Priority to JP2008053500A priority Critical patent/JP2009210409A/en
Publication of JP2009210409A publication Critical patent/JP2009210409A/en
Pending legal-status Critical Current

Links

Landscapes

  • Image Analysis (AREA)
  • Investigating Or Analysing Biological Materials (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)
  • Image Processing (AREA)

Abstract

<P>PROBLEM TO BE SOLVED: To automatically perform an area division based on color information and relative positional relationship of each area without depending on the shape of the area. <P>SOLUTION: Biopsy image data D is input into an image input section 101. A color space converting section 103 includes a plurality of color space conversion scripts and performs mutual conversion between each color space. A threshold determining section 104 automatically determines a threshold k for performing a binary coding of the biopsy image data D. A binary coding section 105 performs binary coding of the biopsy image data D based on the threshold k. An area dividing section 106 divides the biopsy image data D into two areas based on binary coded image D2. A division controlling section 108 controls a pretreating section 102, the color space converting section 103, the threshold determining section 104, the binary coding section 105, and the area dividing section 106 configured such that an area division for biopsy image data D is repeated to sequentially extract image data Dp (Dp1, Dp2, Dp3, ...) of a plurality of areas. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

本発明は、画像の領域分割方法および装置に係り、特に、胃生検画像において細胞核などの診断に利用される領域分割に好適な画像領域分割方法および装置に関する。   The present invention relates to an image area dividing method and apparatus, and more particularly, to an image area dividing method and apparatus suitable for area division used for diagnosis of cell nuclei in stomach biopsy images.

生検画像の領域分割は、病理専門医の診断を支援する自動診断装置を作成するにあたり、医師の診断の観点を反映させて自動診断装置の精度を向上させるために必要な技術である。特に、胃粘膜上皮細胞の胃生検画像に対する知見から、腺細胞の並びである腺管の領域、およびその細胞核の領域を抽出することが重要である。   The area division of the biopsy image is a technique necessary for improving the accuracy of the automatic diagnosis apparatus by reflecting the viewpoint of the diagnosis of the doctor in creating an automatic diagnosis apparatus that supports the diagnosis of a pathologist. In particular, it is important to extract a region of a gland duct that is an array of glandular cells and a region of its cell nucleus from knowledge of gastric biopsy images of gastric mucosal epithelial cells.

非特許文献１に開示されている領域分割手法では、初めに、胃生検画像中で腺管が存在する領域が手動で矩形に分割され、この分割領域の画像を対象に、RGB色空間でGおよびB成分を用いて固定閾値で背景領域の抽出が行われる。次いで、RおよびB成分を用いて細胞核領域が抽出され、最後に、腺管の細胞核の連結性と背景領域の関係とに基づいて腺管領域候補が結合されて腺管領域が抽出される。
勝倉、“胃組織画像の腺腔抽出法の改良、”電子情報通信学会論文誌、Vol.J71-D, No.1 pp.176-181, 1988. In the region segmentation method disclosed in Non-Patent Document 1, first, a region where a gland duct is present in a stomach biopsy image is manually segmented into rectangles, and the image of this segmented region is processed in RGB color space. Background regions are extracted with a fixed threshold using the G and B components. Next, the cell nucleus region is extracted using the R and B components, and finally, the duct region candidates are combined based on the connectivity of the nucleus of the duct and the relationship between the background regions, and the duct region is extracted.
Katsukura, “Improvement of Glandular Extraction Method for Gastric Tissue Images,” IEICE Transactions, Vol. J71-D, No.1 pp.176-181, 1988.

生検画像の領域分割は、最終的に自動で病理専門医の注目する特徴を抽出し診断の支援をする前処理として必須であるが、上記した先行技術では、まず取得した生検画像から腺管領域を含む矩形に病理専門医が手動で分割する必要があり、計算機により自動抽出する方法ではなかった。   The segmentation of a biopsy image is indispensable as a pre-processing that automatically extracts features noticed by a pathologist and automatically supports diagnosis. However, in the above-described prior art, a gland duct is first obtained from an acquired biopsy image. A pathologist needs to manually divide the rectangle including the area, and it is not a method of automatic extraction by a computer.

また、背景領域を抽出する際に固定閾値が利用されているが、撮影環境が異なれば取得される画像の色特徴が変化するため、ある画像用に用意した閾値が他の画像で有効に働くとは限らず、撮影環境にロバストな領域分割が難しかった。   Also, a fixed threshold is used when extracting the background area, but the color characteristics of the acquired image change if the shooting environment is different, so the threshold prepared for one image works effectively for other images. However, it is difficult to divide the area robust to the shooting environment.

さらに、間質領域、腺管領域、細胞核領域を含む全体画像から細胞核領域を抽出する際、細胞核領域は腺管領域などの細胞質領域中に存在するにもかかわらず、このような各領域の位置的な関係が考慮されていなかった。   Furthermore, when extracting the cell nucleus region from the entire image including the stroma region, the ductal region, and the cell nucleus region, the position of each region is determined even though the cell nucleus region exists in the cytoplasmic region such as the gland duct region. Relationship was not considered.

さらに、腺管領域の領域分割は、腺管に存在する細胞核の並びが正常に近い形状であることを前提とした処理であったため、細胞核の並びに個体差がある一般的な胃生検画像の腺管領域の領域分割には不向きであった。   Furthermore, the segmentation of the gland duct region was a process based on the premise that the arrangement of cell nuclei existing in the gland duct was a normal shape. It was not suitable for segmentation of the ductal region.

本発明は、上記した従来技術の課題を解決し、生検画像から間質領域、腺管領域、細胞核領域などを分割する際に、各領域の形状に依存することなく、各領域の色情報および相対的な位置関係に基づいて、領域分割を自動的に行えるようにした画像領域分割方法および装置を提供することにある。   The present invention solves the above-described problems of the prior art, and when dividing a stroma region, a gland duct region, a cell nucleus region, etc. from a biopsy image, the color information of each region without depending on the shape of each region Another object of the present invention is to provide an image region dividing method and apparatus that can automatically perform region division based on relative positional relationships.

上記した目的を達成するために、本発明は、カラーの生検画像を複数の領域に分割する画像領域分割装置において、生検画像の色空間を変換する色空間変換手段と、生検画像を所定の色空間で二値化するための閾値を決定する閾値決定手段と、生検画像を前記閾値に基づいて二値化する二値化手段と、前記二値画像に基づいて前記生検画像を２つの領域に分割する領域分割手段と、前記分割された一方の領域を対象に、前記色空間変換手段により色空間を変換しながら領域分割が繰り返させるように、前記色空間変換手段、閾値決定手段、二値化手段および領域分割手段を制御する分割制御手段とを含むことを特徴とを含むことを特徴とする。   In order to achieve the above-described object, the present invention provides an image area dividing device that divides a color biopsy image into a plurality of areas, color space conversion means for converting the color space of the biopsy image, and a biopsy image. Threshold determining means for determining a threshold for binarization in a predetermined color space; binarizing means for binarizing a biopsy image based on the threshold; and the biopsy image based on the binary image The color space conversion means, the threshold value, and the threshold value division means for repeating the area division while converting the color space by the color space conversion means for the one divided area. Including a determining unit, a binarizing unit, and a division control unit for controlling the region dividing unit.

本発明によれば、生検画像を診断支援装置に利用するために重要な領域である腺管領域や細胞核領域を、各領域の形状を考慮することなく、その色彩的な特徴量および相対的な位置関係のみに基づいて分割できるので、対象画像領域を予め指定することなく、自動的かつロバストが領域分割が可能になる。   According to the present invention, a gland duct region and a cell nucleus region, which are important regions for using a biopsy image in a diagnosis support apparatus, can be obtained without regard to the shape of each region and its color feature amount and relative Since the image can be divided based only on the positional relationship, the area can be automatically and robustly divided without specifying the target image area in advance.

以下、図面を参照して本発明の最良の実施形態について詳細に説明する。図２は、本実施形態において領域分割の対象となる生検画像の一例を示した図であり、ここでは、生検画像を間質領域A1、腺管領域A2、細胞核領域A3およびその他の背景領域(A4)に４分割する場合を例にして説明する。   Hereinafter, the best embodiment of the present invention will be described in detail with reference to the drawings. FIG. 2 is a diagram showing an example of a biopsy image that is an object of region division in the present embodiment. Here, the biopsy image is divided into stroma region A1, gland duct region A2, cell nucleus region A3, and other backgrounds. A case where the area (A4) is divided into four will be described as an example.

生検画像は間質領域A1と背景領域A4とに排他的に領域分割され、この間質領域A1中に腺管領域A2が存在し、さらに腺管領域A2中に細胞核領域A3が存在する。すなわち、各領域の相対的な位置関係および大きさは、間質領域A1＞腺管領域A2＞細胞核領域A3となる。   The biopsy image is divided into an interstitial region A1 and a background region A4 exclusively. A gland duct region A2 exists in the interstitial region A1, and a cell nucleus region A3 exists in the gland duct region A2. That is, the relative positional relationship and size of each region are stromal region A1> gland duct region A2> cell nucleus region A3.

図３は、染色液で染色された各領域の色分布が色空間に応じて変化する様子を示した図であり、横軸は彩度や明度などのカラー画像に固有の色彩的な特徴量を示し、縦軸は度数を示している。   FIG. 3 is a diagram showing how the color distribution of each region stained with a staining liquid changes according to the color space, and the horizontal axis indicates the color feature amount unique to the color image such as saturation and brightness. The vertical axis represents the frequency.

同図(a)に示した第１の色空間では、背景領域A4と、それ以外の間質領域A1、腺管領域A2および細胞核領域A3とは特徴量の差が大きく、両者の選択性が高くなる一方、間質領域A1、腺管領域A2および細胞核領域A3には特徴量に大きな差がなく、それぞれの選択性が低くなることが判る。   In the first color space shown in Fig. 11 (a), the background region A4 and the other interstitial regions A1, gland duct region A2, and cell nucleus region A3 have a large difference in feature amount, and the selectivity between them is large. On the other hand, the interstitial region A1, gland duct region A2 and cell nucleus region A3 are not significantly different in feature quantity, and it can be seen that the respective selectivity decreases.

これに対して、同図(b)に示した第２の色空間では、背景領域A4および間質領域A1と、それ以外の腺管領域A2および細胞核領域A3とは特徴量の差が大きく、両者の選択性が高くなる一方、背景領域A4と間質領域A1、および腺管領域A2と細胞核領域A3とには特徴量に大きな差がなく、両者の選択制が低くなることが判る。   On the other hand, in the second color space shown in FIG. 4B, the background region A4 and the interstitial region A1, and the other gland duct region A2 and the cell nucleus region A3 have a large difference in feature amount. It can be seen that, while the selectivity between the two is high, there is no large difference in the feature amount between the background region A4 and the interstitial region A1, and between the gland duct region A2 and the cell nucleus region A3, and the selectivity between the two is low.

本発明は、このような考察に基づいてなされたものであり、各領域を、その相対的な位置関係および大小関係、換言すれば包含関係に基づいて、大きな領域から小さな領域を抽出する分割処理を繰り返すことで生検画像を複数の領域に分割すると共に、その際、分割したい領域の選択性が高くなるように、分割対象に応じて生検画像の色空間を変換するようにした点に特徴がある。   The present invention has been made on the basis of such consideration, and a division process for extracting a small region from a large region based on the relative positional relationship and the size relationship, in other words, the inclusion relationship, of each region. The biopsy image is divided into a plurality of regions by repeating the above, and at that time, the color space of the biopsy image is converted according to the division target so that the selectivity of the region to be divided becomes high. There are features.

図４は、本発明における領域分割の手順を模式的に示した図であり、初めは同図(a)に示したように、背景領域A4から、その一部分を占める間質領域A1を分割すべく、特徴量に基づく両者の選択性が高くなるように、生検画像の色空間を第１の色空間に設定し、この第１の色空間における特徴量に基づいて背景領域A4から間質領域A1を分割する。   FIG. 4 is a diagram schematically showing the procedure of area division in the present invention. At first, as shown in FIG. 4A, the interstitial area A1 occupying a part thereof is divided from the background area A4. Therefore, the color space of the biopsy image is set to the first color space so that the selectivity between the two based on the feature amount is high, and the interstitial region from the background region A4 is based on the feature amount in the first color space. Divide area A1.

次いで同図(b)に示したように、前記間質領域A1から、その一部分を占める腺管領域A2を分割すべく、特徴量に基づく両者の選択性が高くなるように、生検画像の色空間を第２の色空間に設定し、この第２の色空間における特徴量に基づいて間質領域A1から腺管領域A2を分割する。   Next, as shown in FIG. 6 (b), in order to divide the gland duct region A2 occupying a part thereof from the interstitial region A1, the selectivity of both based on the feature amount is increased, The color space is set to the second color space, and the gland duct region A2 is divided from the interstitial region A1 based on the feature quantity in the second color space.

最後は同図(c)に示したように、前記腺管領域A2から、その一部分を占める細胞核領域A3を分割すべく、特徴量に基づく両者の選択性が高くなるように、生検画像の色空間を第３の色空間に設定し、この第３の色空間における特徴量に基づいて腺管領域A2から細胞核領域A3を分割する。   Finally, as shown in FIG. 7 (c), in order to divide the cell nucleus region A3 occupying a part thereof from the gland duct region A2, the biopsy image of the biopsy image is increased so that the selectivity of both is increased based on the feature amount. The color space is set to the third color space, and the cell nucleus region A3 is divided from the gland duct region A2 based on the feature amount in the third color space.

図１は、本発明に係る画像領域分割装置の主要部の構成を示したブロック図であり、画像入力部１０１には、領域分割の対象となる生検画像のデータDが入力される。この生検画像データDは、適宜の染色手法で染色された組織片を撮影したカラー画像である。前処理部１０２は、この生検画像データDに対して、コントラスト強調や平滑化などの前処理を適宜に行う。   FIG. 1 is a block diagram showing a configuration of a main part of an image region dividing apparatus according to the present invention, and data D of a biopsy image to be subjected to region division is input to an image input unit 101. The biopsy image data D is a color image obtained by photographing a tissue piece stained by an appropriate staining method. The preprocessing unit 102 appropriately performs preprocessing such as contrast enhancement and smoothing on the biopsy image data D.

色空間変換部１０３は、複数の色空間変換スクリプトを備え、RGB，HSL，RGBA, YCbCr，CMY，CMYK，L*a*b*等の各色空間の相互変換を行う。閾値決定部１０４は、生検画像データDを二値化するための閾値kを自動的に決定する。二値化部１０５は、前記生検画像データDを前記閾値kに基づいて画素単位で二値化して二値画像D2を生成する。   The color space conversion unit 103 includes a plurality of color space conversion scripts, and performs mutual conversion between color spaces such as RGB, HSL, RGBA, YCbCr, CMY, CMYK, and L * a * b *. The threshold determination unit 104 automatically determines a threshold k for binarizing the biopsy image data D. The binarization unit 105 binarizes the biopsy image data D on a pixel basis based on the threshold value k to generate a binary image D2.

領域分割部１０６は、生検画像データDを前記二値画像D2に基づいて２つの領域に分割する。分割された一方の領域の生検画像データDpは、後処理部１０７で平滑化や膨張・縮退などの後処理を適宜に施された後に前記前処理部１０２へ戻され、この一方領域の生検画像データDpに対して更なる領域分割が繰り返される。   The area dividing unit 106 divides the biopsy image data D into two areas based on the binary image D2. The biopsy image data Dp of one of the divided areas is appropriately subjected to post-processing such as smoothing, expansion / reduction by the post-processing unit 107, and then returned to the pre-processing unit 102. Further area division is repeated for the inspection image data Dp.

分割制御部１０８は、前記生検画像データDに対する領域分割が繰り返されて複数の領域の画像データDp(Dp1，Dp2，Dp3…)が順次に抽出されるように、前処理部１０２、色空間変換部１０３、閾値決定部１０４、二値化部１０５および領域分割部１０６を制御する。前記各分割領域の生検画像データDp1，Dp2，Dp3…は記憶部１０９に記憶される。   The division control unit 108 repeats the region division on the biopsy image data D, and sequentially extracts the image data Dp (Dp1, Dp2, Dp3...) Of a plurality of regions, It controls the conversion unit 103, the threshold determination unit 104, the binarization unit 105, and the region division unit 106. The biopsy image data Dp1, Dp2, Dp3... Of each divided area is stored in the storage unit 109.

次いで、本実施形態の動作をフローチャートに沿って説明する。図５は、本発明に係る領域分割の手順を示したフローチャートであり、図６は、領域分割の過程で得られる二値画像D2の一例を示した図である。   Next, the operation of the present embodiment will be described with reference to a flowchart. FIG. 5 is a flowchart showing a procedure of region division according to the present invention, and FIG. 6 is a diagram showing an example of a binary image D2 obtained in the region division process.

ステップＳ１では、図６(a)に一例を示した生検画像の画像データDが画像入力部１０１に取り込まれて一時記憶される。本実施形態では、領域分割が色彩的な特徴量に基づいて行われるので、この画像データは適宜の染色手法により染色された組織片を撮影したカラー画像である。ここでは、染色液としてHE(ヘマトキシリン・エオジン)を使用した場合を例にして説明する。   In step S1, image data D of a biopsy image shown as an example in FIG. 6A is taken into the image input unit 101 and temporarily stored. In the present embodiment, since the region division is performed based on the color feature amount, the image data is a color image obtained by photographing a tissue piece stained by an appropriate staining method. Here, a case where HE (hematoxylin / eosin) is used as a staining solution will be described as an example.

ステップＳ２では、前記前処理部１０２において、前記生検画像データDに対して、画素単位でコントラスト強調および平滑化が実行される。本実施形態では、コントラスト強調がガンマ補正により行われる。ガンマ補正は、入力画素濃度をBin、出力画素濃度をBoutとすれば次式(1)で表される。本実施形態では、RGB色空間のそれぞれの空間においてガンマ補正が施される。このガンマ補正により、後述する領域分割処理の閾値決定が容易になる。   In step S <b> 2, the preprocessing unit 102 executes contrast enhancement and smoothing on the biopsy image data D in pixel units. In this embodiment, contrast enhancement is performed by gamma correction. The gamma correction is expressed by the following equation (1) when the input pixel density is Bin and the output pixel density is Bout. In this embodiment, gamma correction is performed in each of the RGB color spaces. This gamma correction facilitates determination of a threshold value for area division processing described later.

また、平滑化に関して、本実施形態ではメディアンフィルタが利用される。メディアンフィルタは、対象画素付近のm×mピクセルの領域で中央値を取る画素値に対象画素を変換する手法である。この平滑化によりノイズの除去が可能になる。   Further, regarding smoothing, a median filter is used in the present embodiment. The median filter is a technique for converting a target pixel into a pixel value that takes a median value in an m × m pixel region near the target pixel. This smoothing makes it possible to remove noise.

ステップＳ３では、前記色空間変換部１０３において、生検画像データDの色空間を変換する処理が行われる。本発明者等の実験結果によれば、染色液としてHEを使用した場合、HSV色空間において、そのS（彩度）成分に着目すれば、間質領域A1と背景領域A4との選択性が高くなることが実験的に確認されているので、ここでは、生検画像データDの色空間をRGBからHSVへ変換する色変換が行われる。ステップＳ４では、前記閾値決定部１０４において、前記HSV色空間に変換された生検画像データDのS成分に着目して、当該画像データDを二値化するための閾値kが決定される。   In step S3, the color space conversion unit 103 performs a process of converting the color space of the biopsy image data D. According to the results of experiments by the present inventors, when HE is used as a staining solution, the selectivity between the interstitial region A1 and the background region A4 can be obtained by paying attention to the S (saturation) component in the HSV color space. Since it has been experimentally confirmed that it becomes higher, color conversion for converting the color space of the biopsy image data D from RGB to HSV is performed here. In step S4, the threshold determination unit 104 determines a threshold k for binarizing the image data D by paying attention to the S component of the biopsy image data D converted into the HSV color space.

ここでは、分割対象の間質領域A1が、それ以外の背景領域A4に比較して大きな領域を占めるため、Kittlerの判別分析法(J. Kittler, and J. Illingworth, “Minimum Error Thresholding,” Pattern Recognition, vol.19, no.1,pp.41-47, 1986.)を利用する。この手法は、対象画像を二値化する際に各領域の画素値が共に正規分布に従うという仮定のもとで、平均誤識別率に関する基準を最小とする閾値選定法である。   Here, since the stromal area A1 to be divided occupies a larger area than the other background area A4, Kittler's discriminant analysis method (J. Kittler, and J. Illingworth, “Minimum Error Thresholding,” Pattern Recognition, vol.19, no.1, pp.41-47, 1986.). This method is a threshold selection method that minimizes the criterion regarding the average misclassification rate under the assumption that the pixel values of each region follow a normal distribution when binarizing the target image.

閾値kによって各画素を二つのクラスC1，C2に分類する際、全画素数で正規化した画素値のヒストグラムを利用して、閾値k以下の画素数の割合をy1(k)，閾値k+1以上の画素数の割合をy2(k)、各クラスの分散をs12, s22としたとき、次式(2)で定義されるj(k)を最小値とするkが閾値と定義される。なお、背景領域が存在しない場合は閾値kが正しく計算されず、この場合には背景領域が存在しない旨の判定がなされる。 When classifying each pixel into two classes C1 and C2 based on the threshold value k, using the histogram of pixel values normalized by the total number of pixels, the ratio of the number of pixels below the threshold value k is y1 (k), the threshold value k + When the ratio of the number of pixels equal to or greater than 1 is y2 (k) and the variance of each class is s1 2 , s2 2 , k is defined as the threshold value with k (k) defined as Is done. Note that when the background area does not exist, the threshold value k is not correctly calculated, and in this case, it is determined that the background area does not exist.

ステップＳ５では、前記二値化部１０５において、生検画像データDが前記閾値kに基づいて二値化される。図６(b)は、二値画像D2の一例を示した図であり、ここでは、間質領域A1の画素値が「２５５（白）」に変換され、背景領域A4の画素値が「０（黒色）」に変換される。ステップＳ６では、前記領域分割部１０６により、二値画像D2と生検画像データDとの論理積を取ることで生検画像データDから間質領域A1が分割される。ステップＳ７では、分割された間質領域A1の画像データDp1に対して平滑化や膨張・縮退などの後処理が施される。   In step S5, the binarization unit 105 binarizes the biopsy image data D based on the threshold value k. FIG. 6B is a diagram illustrating an example of the binary image D2, in which the pixel value of the interstitial region A1 is converted to “255 (white)” and the pixel value of the background region A4 is “0”. (Black) ". In step S6, the interstitial region A1 is divided from the biopsy image data D by taking the logical product of the binary image D2 and the biopsy image data D by the region dividing unit 106. In step S7, post-processing such as smoothing, expansion / reduction is performed on the image data Dp1 of the divided interstitial region A1.

ステップＳ８では、腺管領域A2と細胞核領域A3との分割が完了したか否かが判定され、ここでは未だ完了していないと判定されるので、前記分割された間質領域の生検画像データDp1が、次の分割対象として前処理部１０２へ入力され、この領域分割された間質領域A1の生検画像データDp1から腺管領域A2を更に分割する処理が同様に繰り返される。   In step S8, it is determined whether or not the division between the gland duct region A2 and the cell nucleus region A3 has been completed. Here, since it is determined that the division has not yet been completed, the biopsy image data of the divided stromal region is determined. Dp1 is input to the preprocessing unit 102 as the next division target, and the process of further dividing the gland duct area A2 from the biopsy image data Dp1 of the interstitial area A1 divided in this area is similarly repeated.

このとき、本発明等の実験結果によれば、間質領域A1と腺管領域A2とは赤色の度合いが異なり、YCbCr色空間のCb成分を利用すれば両者の選択性が高くなることが実験的に確認されている。したがって、ステップＳ３では、前記色変換部１０２において、生検画像データDp1の色空間がHSVからYCbCrへ変換される。   At this time, according to the experimental results of the present invention and the like, the stroma region A1 and the ductal region A2 have different degrees of red, and if the Cb component of the YCbCr color space is used, the selectivity between the two is increased. Has been confirmed. Therefore, in step S3, the color conversion unit 102 converts the color space of the biopsy image data Dp1 from HSV to YCbCr.

ステップＳ４の閾値決定処理では、YCbCr色空間の生検画像データDp1を二値化するための閾値kが決定される。なお、前記間質領域A1と背景領域A4との二値化では、その面積の違いに着目して閾値kを設定したが、間質領域A1と腺管領域A2との間には、一般的な領域面積の差の傾向がないため、ここでは大津の判別分析法(大津展之，“判別および最小２乗基準に基づく自動しきい値選定法，” 電子通信学会論文誌，vol.63-D，no.4，pp.349-356，1980.)を利用する。   In the threshold value determination process in step S4, a threshold value k for binarizing the biopsy image data Dp1 in the YCbCr color space is determined. In the binarization of the interstitial region A1 and the background region A4, the threshold value k is set by paying attention to the difference in area, but the interstitial region A1 and the gland duct region A2 are commonly used. Because there is no tendency for the difference in the area of the region, here is the Otsu's discriminant analysis method (Otsunoyuki Nobuyuki, “Automatic threshold selection method based on discriminant and least square criterion,” IEICE Transactions, vol.63- D, no.4, pp.349-356, 1980.).

この手法は、閾値kによって各画素を二つのクラスC1，C2に分類する際、全画素数で正規化した画素値のヒストグラムを利用して、閾値k以下の画素数の割合をy1(k)，閾値kレベルまでの画素値の分布の１次モーメントの累積量をl1(k)、画像全体の画素値の平均をlTとしたときに、次式(3)で与えられるクラス間分散sE2 を最大値とするkを閾値とする手法である。 This method uses a histogram of pixel values normalized by the total number of pixels to classify each pixel into two classes C1 and C2 based on the threshold k, and the ratio of the number of pixels below the threshold k is y1 (k) , The interclass variance sE 2 given by the following equation (3), where l1 (k) is the cumulative first moment of the distribution of pixel values up to the threshold k level and lT is the average of the pixel values of the entire image In this method, k is a maximum value and the threshold value is k.

ステップＳ５では、間質領域A1の生検画像データDp1が前記閾値kに基づいて二値化される。図６(c)は、二値画像D2の一例を示した図であり、ここでは、腺管領域A2の画素値が「２５５（白色）」に変換され、間質領域A1を含む他の領域の画素値が「０（黒色）」に変換される。ステップＳ６では、二値画像D2と生検画像データDとの論理積を取ることで生検画像データDから腺管領域A2が分割される。ステップＳ７では、分割された腺管領域A2の画像データDp2に対して平滑化や膨張・縮退などの後処理が施される。   In step S5, the biopsy image data Dp1 of the interstitial region A1 is binarized based on the threshold value k. FIG. 6C is a diagram showing an example of the binary image D2, in which the pixel value of the gland duct region A2 is converted to “255 (white)” and other regions including the stroma region A1 Is converted to “0 (black)”. In step S6, the duct region A2 is divided from the biopsy image data D by taking the logical product of the binary image D2 and the biopsy image data D. In step S7, post-processing such as smoothing, expansion / reduction is performed on the image data Dp2 of the divided gland duct region A2.

ステップＳ８では、未だ腺管領域A2と細胞核領域A3との分割が完了していないと判定されるので、前記分割された腺管領域A2の生検画像データDp2が、次の分割対象として前処理部１０２へ入力され、この腺管領域A2に関する生検画像データDp2から細胞核領域A3を分割する処理が繰り返される。   In step S8, since it is determined that the division of the ductal region A2 and the cell nucleus region A3 has not been completed, the biopsy image data Dp2 of the divided ductal region A2 is preprocessed as the next division target. The process of dividing the cell nucleus region A3 from the biopsy image data Dp2 related to the gland duct region A2 is repeated.

このとき、本発明等の実験結果によれば、腺管領域A2と細胞核領域A3とは、L*a*b*色空間のb*成分を利用すれば両者の選択性が高くなることが実験的に確認されているので、ステップＳ３では、前記色変換部１０２において、生検画像データDp2の色空間がYCbCrからL*a*b*へ変換される。ステップＳ４では、前記大津の判別分析法を適用して閾値kが決定される。   At this time, according to the experimental results of the present invention, the ductal region A2 and the cell nucleus region A3 are experimentally shown to be highly selective when using the b * component of the L * a * b * color space. In step S3, the color conversion unit 102 converts the color space of the biopsy image data Dp2 from YCbCr to L * a * b *. In step S4, the threshold k is determined by applying the Otsu discriminant analysis method.

ステップＳ５では、腺管領域A2の生検画像データDp2が前記閾値kに基づいて二値化される。図６(d)は、二値画像D2の一例を示した図であり、ここでは、細胞核領域A3の画素値が「２５５（白色）」に変換され、腺管領域A2を含む他の領域の画素値が「０（白色）」に変換される。ステップＳ６では、二値画像D2と腺管領域A2の生検画像データDp2との論理積を取ることで、腺管領域A2から細胞核領域A3が分割される。ステップＳ７では、分割された細胞核領域A3の画像データDp3に対して平滑化や膨張・縮退などの後処理が施される。
In step S5, the biopsy image data Dp2 of the ductal region A2 is binarized based on the threshold value k. FIG. 6D is a diagram showing an example of the binary image D2, in which the pixel value of the cell nucleus region A3 is converted to “255 (white)” and other regions including the gland duct region A2 are displayed. The pixel value is converted to “0 (white)”. In step S6, the cell nucleus region A3 is divided from the gland duct region A2 by taking the logical product of the binary image D2 and the biopsy image data Dp2 of the gland duct region A2. In step S7, post-processing such as smoothing, expansion / reduction is performed on the image data Dp3 of the divided cell nucleus region A3.

図７は、上記した領域分割処理による領域分割の結果を示した図、図８は、その部分拡大図であり、いずれにおいても、生検画像D[各図(a)]から間質領域A1、腺管領域A2および細胞核領域A3のいずれもが正確に分割されている[各図(b)]ことがわかる。   FIG. 7 is a diagram showing the result of region segmentation by the region segmentation process described above, and FIG. 8 is a partial enlarged view thereof. In any case, from biopsy image D [each diagram (a)] to stromal region A1 It can be seen that both the ductal region A2 and the cell nucleus region A3 are accurately divided [each figure (b)].

本発明に係る画像領域分割装置の構成を示したブロック図である。It is the block diagram which showed the structure of the image area division | segmentation apparatus based on this invention. 領域分割対象の生検画像の一例を示した図である。It is the figure which showed an example of the biopsy image of area | region division | segmentation object. 各領域の選択性が色空間に応じて変化する様子を示した図である。It is the figure which showed a mode that the selectivity of each area | region changed according to color space. 本発明における領域分割の手順を模式的に示した図である。It is the figure which showed typically the procedure of the area | region division in this invention. 本発明に係る領域分割の手順を示したフローチャートである。It is the flowchart which showed the procedure of the area | region division based on this invention. 領域分割の過程で得られる二値画像の一例を示した図である。It is the figure which showed an example of the binary image obtained in the process of area | region division. 領域分割結果の一例を示した図である。It is the figure which showed an example of the area | region division result. 領域分割結果の一例を示した拡大図である。It is the enlarged view which showed an example of the area | region division result.

符号の説明Explanation of symbols

１０１…入力部、１０２…前処理部、１０３…色空間変換、１０４…閾値決定部、１０５…二値化部、１０６…領域分割部、１０７…後処理部、１０８…分割制御部、１０９…記憶部   DESCRIPTION OF SYMBOLS 101 ... Input part, 102 ... Pre-processing part, 103 ... Color space conversion, 104 ... Threshold determination part, 105 ... Binarization part, 106 ... Area division part, 107 ... Post-processing part, 108 ... Division control part, 109 ... Memory

Claims (10)

カラーの生検画像を複数の領域に分割する画像領域分割装置において、
生検画像の色空間を変換する色空間変換手段と、
生検画像を所定の色空間で二値化するための閾値を決定する閾値決定手段と、
生検画像を前記閾値で二値化して二値画像を生成する二値化手段と、
前記二値画像に基づいて前記生検画像を２つの領域に分割する領域分割手段と、
前記分割された一方の領域を対象に、前記色空間変換手段により色空間を変換しながら領域分割が繰り返させるように、前記色空間変換手段、閾値決定手段、二値化手段および領域分割手段を制御する分割制御手段とを含むことを特徴とする画像領域分割装置。 In an image region dividing device for dividing a color biopsy image into a plurality of regions,
Color space conversion means for converting the color space of the biopsy image;
Threshold determination means for determining a threshold for binarizing the biopsy image in a predetermined color space;
Binarization means for binarizing a biopsy image with the threshold value to generate a binary image;
Area dividing means for dividing the biopsy image into two areas based on the binary image;
The color space conversion means, the threshold value determination means, the binarization means, and the area division means are arranged to repeat the area division while converting the color space by the color space conversion means for the one divided area. An image area dividing device comprising: a division control means for controlling. 前記色空間変換手段は、前記生検画像を、その一部分を占める第１領域と他の領域との選択性が高い第１の色空間に色変換し、
前記閾値決定手段は、前記生検画像を前記第１領域と他の領域とに二値化する第１閾値を決定し、
前記二値化手段は、前記生検画像を前記第１閾値に基づいて二値化し、
前記領域分割手段は、前記生検画像から第１領域を分割することを特徴とする請求項１に記載の画像領域分割装置。 The color space conversion means performs color conversion of the biopsy image into a first color space having high selectivity between a first region and another region occupying a part thereof,
The threshold value determining means determines a first threshold value for binarizing the biopsy image into the first region and another region,
The binarization means binarizes the biopsy image based on the first threshold value,
The image area dividing device according to claim 1, wherein the area dividing unit divides the first area from the biopsy image. 前記色空間変換手段はさらに、前記第１領域の生検画像を、その一部分を占める第２領域と他の領域との選択性が高い第２の色空間に色変換し、
前記閾値決定手段はさらに、前記第１領域の生検画像を前記第２領域と他の領域とに二値化する第２閾値を決定し、
前記二値化手段は、前記第１領域の生検画像を前記第２閾値に基づいて二値化し、
前記領域分割手段はさらに、前記第１領域の生検画像から第２領域を分割することを特徴とする請求項２に記載の画像領域分割装置。 The color space converting means further converts the biopsy image of the first region into a second color space having high selectivity between the second region and another region occupying a part thereof,
The threshold value determining means further determines a second threshold value for binarizing the biopsy image of the first region into the second region and another region,
The binarization means binarizes the biopsy image of the first region based on the second threshold,
The image region dividing device according to claim 2, wherein the region dividing unit further divides the second region from the biopsy image of the first region. 前記色空間変換手段はさらに、前記第２領域の生検画像を、その一部分を占める第３領域と他の領域との選択性が高い第３の色空間に色変換し、
前記閾値決定手段はさらに、前記第２領域の生検画像を前記第３領域と他の領域とに二値化する第３閾値を決定し、
前記二値化手段は、前記第２領域の生検画像を前記第３閾値に基づいて二値化し、
前記領域分割手段はさらに、前記第２領域の生検画像から第３領域を分割することを特徴とする請求項３に記載の画像領域分割装置。 The color space conversion means further color-converts the biopsy image of the second region into a third color space having a high selectivity between the third region occupying a part thereof and another region,
The threshold value determining means further determines a third threshold value for binarizing the biopsy image of the second region into the third region and another region,
The binarization means binarizes the biopsy image of the second region based on the third threshold value,
4. The image area dividing apparatus according to claim 3, wherein the area dividing means further divides a third area from the biopsy image of the second area. 前記第１領域が間質領域であることを特徴とする請求項２に記載の画像領域分割装置。   The image area dividing apparatus according to claim 2, wherein the first area is an interstitial area. 前記第１領域が間質領域であり、前記第２領域が腺管領域であることを特徴とする請求項３に記載の画像領域分割装置。   The image area dividing device according to claim 3, wherein the first area is an interstitial area and the second area is a gland duct area. 前記第１領域が間質領域であり、前記第２領域が腺管領域であり、前記第３領域が細胞核領域であることを特徴とする請求項４に記載の画像領域分割装置。   5. The image region dividing device according to claim 4, wherein the first region is a stroma region, the second region is a gland duct region, and the third region is a cell nucleus region. 分割対象の画像に前処理を施す前処理手段を含み、
前記前処理手段が、コントラスト強調および平滑化の少なくとも一方を実行することを特徴とする請求項１ないし７のいずれかに記載の画像領域分割装置。 Including preprocessing means for performing preprocessing on the image to be divided;
8. The image area dividing apparatus according to claim 1, wherein the preprocessing means executes at least one of contrast enhancement and smoothing. 分割された画像に後処理を施す後処理手段を含み、
前記後処理手段が、平滑化および膨張・縮退の少なくとも一方を実行することを特徴とする請求項１ないし８のいずれかに記載の画像領域分割装置。 Including post-processing means for performing post-processing on the divided images;
9. The image area dividing apparatus according to claim 1, wherein the post-processing means executes at least one of smoothing and expansion / reduction. カラーの生検画像を複数の領域に分割する画像領域分割装置において、
生検画像の色空間を変換する第１手順と、
生検画像を所定の色空間で二値化するための閾値を決定する第２手順と、
生検画像を前記閾値で二値化して二値画像を生成する第３手順と、
前記二値画像に基づいて前記生検画像を２つの領域に分割する第４手順と、
前記分割された一方の領域を対象に、前記第１手順で色空間を変換しながら領域分割が繰り返させるように、前記第１ないし第４手順を繰り返す第５手順とを含むことを特徴とする画像領域分割方法。 In an image region dividing device for dividing a color biopsy image into a plurality of regions,
A first procedure for converting the color space of the biopsy image;
A second procedure for determining a threshold for binarizing the biopsy image in a predetermined color space;
A third procedure for binarizing a biopsy image with the threshold value to generate a binary image;
A fourth procedure for dividing the biopsy image into two regions based on the binary image;
A fifth procedure that repeats the first to fourth procedures so that the region division is repeated while converting the color space in the first procedure for the one divided region. Image segmentation method.
JP2008053500A 2008-03-04 2008-03-04 Method and device for image area division Pending JP2009210409A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2008053500A JP2009210409A (en) 2008-03-04 2008-03-04 Method and device for image area division

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2008053500A JP2009210409A (en) 2008-03-04 2008-03-04 Method and device for image area division

Publications (1)

Publication Number Publication Date
JP2009210409A true JP2009210409A (en) 2009-09-17

Family

ID=41183723

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2008053500A Pending JP2009210409A (en) 2008-03-04 2008-03-04 Method and device for image area division

Country Status (1)

Country Link
JP (1) JP2009210409A (en)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2010140654A1 (en) * 2009-06-03 2010-12-09 日本電気株式会社 Pathologic tissue image analyzing apparatus, pathologic tissue image analyzing method, and pathologic tissue image analyzing program
WO2012043498A1 (en) * 2010-09-30 2012-04-05 日本電気株式会社 Information processing device, information processing system, information processing method, program, and recording medium
JP2013238459A (en) * 2012-05-14 2013-11-28 Tokyo Univ Of Science Cancer cell region extraction device, method and program
US8847011B2 (en) 2007-12-05 2014-09-30 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US8847012B2 (en) 2007-12-05 2014-09-30 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US9045786B2 (en) 2008-03-04 2015-06-02 Toyota Jidosha Kabushiki Kaisha Gene that increases production of plant fat-and-oil and method for using the same
US9169488B2 (en) 2009-06-04 2015-10-27 Toyota Jidosha Kabushiki Kaisha Gene capable of improving material productivity in seed and method for use thereof
WO2016027542A1 (en) * 2014-08-18 2016-02-25 株式会社Ｓｃｒｅｅｎホールディングス Threshold value determination method, image processing method, and image processing device
US9303265B2 (en) 2009-06-04 2016-04-05 Toyota Jidosha Kabushiki Kaisha Gene for increasing plant weight and method for using the same
US9309531B2 (en) 2009-06-04 2016-04-12 Toyota Jidosha Kabushiki Kaisha Plant with reduced protein productivity in seeds, and method for producing same
JP2018185265A (en) * 2017-04-27 2018-11-22 日本電気株式会社 Information processor, method for control, and program

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0235358A (en) * 1988-07-25 1990-02-05 Toa Medical Electronics Co Ltd Image processing method for taking out glandular cavity of stomack tissue
JP2001059842A (en) * 1999-08-25 2001-03-06 Nec Corp Pathological diagnostic apparatus

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0235358A (en) * 1988-07-25 1990-02-05 Toa Medical Electronics Co Ltd Image processing method for taking out glandular cavity of stomack tissue
JP2001059842A (en) * 1999-08-25 2001-03-06 Nec Corp Pathological diagnostic apparatus

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9062318B2 (en) 2007-12-05 2015-06-23 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US9068193B2 (en) 2007-12-05 2015-06-30 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US8847011B2 (en) 2007-12-05 2014-09-30 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US8847012B2 (en) 2007-12-05 2014-09-30 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US9012726B2 (en) 2007-12-05 2015-04-21 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US9012727B2 (en) 2007-12-05 2015-04-21 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US9018446B2 (en) 2007-12-05 2015-04-28 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US9018450B2 (en) 2007-12-05 2015-04-28 Toyota Jidosha Kabushiki Kaisha Genes that increase plant oil and method for using the same
US9045786B2 (en) 2008-03-04 2015-06-02 Toyota Jidosha Kabushiki Kaisha Gene that increases production of plant fat-and-oil and method for using the same
JP2010281636A (en) * 2009-06-03 2010-12-16 Nec Corp Pathologic tissue image analyzing apparatus, pathologic tissue image analyzing method, pathologic tissue image analyzing program
WO2010140654A1 (en) * 2009-06-03 2010-12-09 日本電気株式会社 Pathologic tissue image analyzing apparatus, pathologic tissue image analyzing method, and pathologic tissue image analyzing program
US8594411B2 (en) 2009-06-03 2013-11-26 Nec Corporation Pathologic tissue image analyzing apparatus, pathologic tissue image analyzing method, and pathologic tissue image analyzing program
US9309530B2 (en) 2009-06-04 2016-04-12 Toyota Jidosha Kabushiki Kaisha Gene capable of improving material productivity in seed and method for use thereof
US9309529B2 (en) 2009-06-04 2016-04-12 Toyota Jidosha Kabushiki Kaisha Gene capable of improving material productivity in seed and method for use thereof
US10000764B2 (en) 2009-06-04 2018-06-19 Toyota Jidosha Kabushiki Kaisha Gene for increasing plant weight and method for using the same
US9169488B2 (en) 2009-06-04 2015-10-27 Toyota Jidosha Kabushiki Kaisha Gene capable of improving material productivity in seed and method for use thereof
US9970020B2 (en) 2009-06-04 2018-05-15 Toyota Jidosha Kabushiki Kaisha Plant with reduced protein productivity in seeds and method for producing same
US9856488B2 (en) 2009-06-04 2018-01-02 Toyota Jidosha Kabushiki Kaisha Plant with reduced protein productivity in seeds and method for producing same
US9840717B2 (en) 2009-06-04 2017-12-12 Toyota Jidosha Kabushiki Kaisha Plant with reduced protein productivity in seeds and method for producing same
US9303265B2 (en) 2009-06-04 2016-04-05 Toyota Jidosha Kabushiki Kaisha Gene for increasing plant weight and method for using the same
US9309531B2 (en) 2009-06-04 2016-04-12 Toyota Jidosha Kabushiki Kaisha Plant with reduced protein productivity in seeds, and method for producing same
US9816099B2 (en) 2009-06-04 2017-11-14 Toyota Jidosha Kabushiki Kaisha Gene for increasing plant weight and method for using the same
JP5871325B2 (en) * 2010-09-30 2016-03-01 日本電気株式会社 Information processing apparatus, information processing system, information processing method, program, and recording medium
WO2012043498A1 (en) * 2010-09-30 2012-04-05 日本電気株式会社 Information processing device, information processing system, information processing method, program, and recording medium
US9076198B2 (en) 2010-09-30 2015-07-07 Nec Corporation Information processing apparatus, information processing system, information processing method, program and recording medium
US10115191B2 (en) 2010-09-30 2018-10-30 Nec Corporation Information processing apparatus, information processing system, information processing method, program, and recording medium
JP2013238459A (en) * 2012-05-14 2013-11-28 Tokyo Univ Of Science Cancer cell region extraction device, method and program
JP2016041032A (en) * 2014-08-18 2016-03-31 株式会社Ｓｃｒｅｅｎホールディングス Threshold value determination method, image processing method, and image processing device
WO2016027542A1 (en) * 2014-08-18 2016-02-25 株式会社Ｓｃｒｅｅｎホールディングス Threshold value determination method, image processing method, and image processing device
JP2018185265A (en) * 2017-04-27 2018-11-22 日本電気株式会社 Information processor, method for control, and program

Similar Documents

Publication Publication Date Title
JP2009210409A (en) Method and device for image area division
CN110678901B (en) Information processing apparatus, information processing method, and computer-readable storage medium
US9807280B2 (en) Image processing apparatus, method, and storage medium
Roth et al. Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation
JP6100744B2 (en) Color document image segmentation and binarization using automatic restoration
JP6547386B2 (en) Image processing apparatus and method
US8503781B2 (en) Finding text regions from coloured image independent of colours
JP2007504719A (en) System and method for detecting and correcting red eyes in an image suitable for embedded applications
JP2010525486A (en) Image segmentation and image enhancement
CN110866932A (en) Multi-channel tongue edge detection device and method and storage medium
CN107038704B (en) Retina image exudation area segmentation method and device and computing equipment
CN107659799B (en) Image pickup apparatus, image processing method, and storage medium
CN107174232B (en) Electrocardiogram waveform extraction method
CN114820594B (en) Method for detecting edge sealing defect of plate based on image, related equipment and storage medium
KR101549495B1 (en) An apparatus for extracting characters and the method thereof
US9064179B2 (en) Region extraction apparatus, region extraction method, and computer program product
JP4599110B2 (en) Image processing apparatus and method, imaging apparatus, and program
US8810877B2 (en) Image processing apparatus, image processing method, and storage medium for eliminating blurring of scanned image
JP2010186246A (en) Image processing apparatus, method, and program
JP2005275854A (en) Image processor, image processing method, image processing program and recording medium with this program stored thereon
JP5979008B2 (en) Image processing apparatus, image processing method, and program
WO2022056875A1 (en) Method and apparatus for segmenting nameplate image, and computer-readable storage medium
CN114913099A (en) Method and system for processing video file
JP6317413B1 (en) Image processing apparatus, image processing method, and image processing program
TWI442329B (en) Image processing method for deriving text characteristic image from input image and related apparatus thereof

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20100715

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20120314

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20120314

A02 Decision of refusal

Free format text: JAPANESE INTERMEDIATE CODE: A02

Effective date: 20120704