JP4200887B2 - Color conversion device - Google Patents

Color conversion device Download PDF

Info

Publication number
JP4200887B2
JP4200887B2 JP2003402649A JP2003402649A JP4200887B2 JP 4200887 B2 JP4200887 B2 JP 4200887B2 JP 2003402649 A JP2003402649 A JP 2003402649A JP 2003402649 A JP2003402649 A JP 2003402649A JP 4200887 B2 JP4200887 B2 JP 4200887B2
Authority
JP
Japan
Prior art keywords
color
conversion
degree
image
blind
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related
Application number
JP2003402649A
Other languages
Japanese (ja)
Other versions
JP2005173634A (en
Inventor
茂樹 中内
仁貴 中野
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Artience Co Ltd
Original Assignee
Toyo Ink Mfg Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Toyo Ink Mfg Co Ltd filed Critical Toyo Ink Mfg Co Ltd
Priority to JP2003402649A priority Critical patent/JP4200887B2/en
Priority to US10/991,428 priority patent/US20050111019A1/en
Publication of JP2005173634A publication Critical patent/JP2005173634A/en
Application granted granted Critical
Publication of JP4200887B2 publication Critical patent/JP4200887B2/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Images

Landscapes

  • Image Processing (AREA)
  • Processing Of Color Television Signals (AREA)
  • Facsimile Image Signal Circuits (AREA)
  • Color Image Communication Systems (AREA)

Description

本発明は、色覚異常者にとって混同しやすい配色を変換し、色覚異常者にとっても見やすい画像を生成することを可能としたカラー信号変換装置に関する。   The present invention relates to a color signal conversion apparatus capable of converting a color scheme that is easily confused for a color blind person and generating an image that is easy to see for a color blind person.

我々の身の回りにある様々な印刷物、表示物は、カラーであふれ、文字や画像を色で表現し、多くの情報を提供している。また、近年の様々なデジタルカラーデバイスの普及は、フルカラーの世界をより身近にさせるもので、例えば、インターネットなどによる情報提供は、最もポピューラーな情報提供手段の1つとなっており、それら情報の中でも色彩情報の重要性はますます高くなって来ている。   Various prints and displays around us are overflowing in color, and text and images are expressed in color to provide a great deal of information. In addition, the spread of various digital color devices in recent years brings the full-color world closer, and for example, the provision of information via the Internet has become one of the most popular information provision means. The importance of color information is increasing.

一方、色覚障害や色覚異常の人々は、日本人男性の20人に1人、白人の12人に1人はいるとされている。色覚異常は、青・赤・緑の3つある視物質遺伝子のうち、いずれかの変異により残る2つの視物質を使って見ることにより起こることが多くの報告で示されている。これらの人は、かなり広範囲の色が見分けられるが、一部非常に見分けにくい色もあり、前記提供される様々な色情報を把握できなかったり、間違って認識することもある。   On the other hand, it is said that there are 1 person in 20 Japanese men and 1 in 12 white people with color blindness and color blindness. Many reports indicate that color blindness is caused by viewing using two visual substances remaining due to any one of the three visual substance genes of blue, red, and green. These people can recognize a wide range of colors, but there are some colors that are very difficult to distinguish, and may not be able to grasp the various color information provided or may recognize them incorrectly.

これに対して、色覚異常者にも識別が容易となるように、印刷デザインや画面設計において、色覚正常者と同程度に色覚異常者にも理解を得る色彩設計を行うために、実験、心理的に証明された結果に即して、トライアルアンドエラーで修正を施しているのが現状である。また、特別の対策をとるための費用の増大を伴うことから、色覚異常者に対する十分な配慮が行われないことが多いのも現状である。また、従来の色覚異常者向けの画像色変換技術に関しては、カラー映像信号変換装置(特許文献1参照)、色覚変換装置(特許文献2参照)、画像処理装置(特許文献3参照)、色覚障害者用表示システム(特許文献4参照)等が提案されているが、色変換する手段としては、近接する色の境界部分のみを縁どりなどの処理をするものであったり、色変換テーブルを用いる、あるいは、輝度または彩度を変更するという記載のみで具体的に色変換する方法を開示するものではなかった。また、これらの技術では、画像のデザインコンセプトの保持を意識していないので、色相の変換まで言及されていなかった。
特開2002−44678号公報 特公平6−82385号公報 特開平8−16129号公報 特開平11−175050号公報
On the other hand, in order to make it easier for color-blind people to identify, in print design and screen design, in order to perform color design that can be understood by color-blind people as well as color-blind people, The current situation is that trial and error corrections are made in accordance with the results that have been proved. In addition, since there is an increase in costs for taking special measures, it is often the case that sufficient consideration is not given to people with color blindness. As for conventional image color conversion technology for color blind persons, a color video signal converter (see Patent Document 1), a color vision converter (see Patent Document 2), an image processing apparatus (see Patent Document 3), a color blindness A display system for people (see Patent Document 4) has been proposed, but as a means for color conversion, only a border portion of adjacent colors is processed such as edging, or a color conversion table is used. Alternatively, a specific color conversion method is not disclosed only by the description of changing luminance or saturation. In addition, since these technologies are not conscious of maintaining the design concept of an image, mention has not been made of hue conversion.
JP 2002-44678 A Japanese Patent Publication No. 6-82385 JP-A-8-16129 JP 11-175050 A

上述の点に鑑み、本発明は、カラーデザインやカラー信号の中にある色覚異常者が識別しにくい配色を検出し、それらを色覚正常者と同程度に、色覚異常者にも区別できるように色を変換する装置を提案しようとするものである。   In view of the above, the present invention detects color schemes that are difficult for color blind people in color designs and color signals to be identified, and can distinguish them from color blind people as well as color blind people. An attempt is made to propose a device for converting colors.

本発明は、入力したカラー画像の色変換装置であって、
前記入力したカラー画像の各画素の色情報の配色組み合わせを検索し
当該配色組み合わせが、色覚異常者にとって混同しやすいか否かを決定する手段と、
前記決定する手段において、混同しやすいと決定された配色組み合わせについては
色覚異常者の認識色空間内での色の差で表す混同度と、色変換後と変換前の色の差であらわす変更度との和により構成される評価関数最小となるように当該配色組み合わせの画素の色情報を変換する手段とを有することを特徴とする色変換装置である。
The present invention is a color conversion device for an input color image,
The color combination of the color information of each pixel of the input color image is searched.
Means for determining whether the color combination is likely to be confused for color blind persons;
In the means for determining the color difference between before conversion and confused degree, and after the color conversion expressed by the color difference in the recognition color space of <br/> color defectives for color combinations that are determined to confuse And a means for converting the color information of the pixels of the color combination so that the evaluation function constituted by the sum of the degree of change and the degree of change is minimized.

本発明によれば、カラーデザインやカラー信号の中にある色覚異常者が識別しにくい配色を検出し、それらを色覚正常者と同程度に、色覚異常者にも区別できるように色を変換することが可能となる効果がある。   According to the present invention, color schemes or color signals in color designs that are difficult for color blind persons to detect are detected, and colors are converted so that they can be distinguished by color blind persons as well as normal color blind persons. There is an effect that becomes possible.

本発明で解決する課題の色覚異常は、眼の錐体細胞が1種類しかない1色型色覚(人口の0.01%以下)、2種類の錐体がある2色型色覚(男性の約1.5%)、3種類の錐体があるがそのうち1つの波長特性がずれている異常3色型色覚(男性の約3.5%)の3種類に大別される。2色型や異常3色型色覚には、どの錐体に変化が出ているかによって第1色覚異常、第2色覚異常、第3色覚異常の3つの種類に分けられる。
第1色覚異常と第2色覚異常は色の見え方が比較的近く、総称して「赤緑色覚異常」と呼ばれる。第3色覚異常は稀である。
(池田光男,中嶋芳雄「色覚のメカニズムと色覚異常」眼科Mook No.16 1982)
The color vision abnormalities to be solved by the present invention are one-color type color vision with only one type of eye cone cells (0.01% or less of the population), two-color type color vision with two types of cones (about 1.5% of men) ), There are three types of cones, one of which is roughly divided into three types: abnormal three-color type color vision (about 3.5% of men). The two-color type and abnormal three-color type color vision can be divided into three types: first color vision abnormality, second color vision abnormality, and third color vision abnormality, depending on which cone changes.
The first color vision anomaly and the second color vision anomaly are relatively similar in color appearance and are collectively referred to as “red-green color vision anomaly”. Third color blindness is rare.
(Mitsuo Ikeda, Yoshio Nakajima “Color Perception Mechanism and Color Defects” Ophthalmology Mook No.16 1982)

本発明の基本的な考え方は、正常色覚者の色空間内での色差を色覚異常者の色空間に移した場合の、色の差をもとに、認識の際に間違いを起こすと考えられる色群を危険色とし明示するとともに、危険色群の色のみを評価関数に従い変換することで、危険度を下げるように色を変化させるようにするものである。   The basic concept of the present invention is considered to cause an error in recognition based on the color difference when the color difference in the color space of the normal color vision person is transferred to the color space of the color vision abnormality person. The color group is specified as a dangerous color, and only the color of the dangerous color group is converted according to the evaluation function so that the color is changed so as to reduce the degree of danger.

本発明で使用される前記危険色の検出手段は、色覚正常者が判断する色差を色覚異常者の判断する色差に変換し、あらかじめ設定された閾値に従い判断する。   The dangerous color detection means used in the present invention converts a color difference determined by a person with normal color vision into a color difference determined by a person with abnormal color vision, and makes a determination according to a preset threshold.

また、本発明で使用される評価関数は、色覚異常者の認識色空間内での色の差で表す混同度と、色変換後と変換前の色の差であらわす変更度との和により規定され、該評価関数を最小とするように変換対象色を変換する。   The evaluation function used in the present invention is defined by the sum of the degree of confusion expressed by the color difference in the color space recognized by the color blind person and the degree of change expressed by the difference between the color after conversion and the color before conversion. Then, the conversion target color is converted so as to minimize the evaluation function.

以下に、図を参照して、本発明の色変換装置の例を説明する。
図1は色変換装置の処理を示すフローチャートである。まず、変換対象となる色彩値あるいはRGB値若しくはCMYK値等からなる画像データを該変換装置に入力する(ステップ1)とともに、モニタなどで表示を行う。次に、必要に応じた調整用の各パラメータを入力する(ステップ2)。当該パラメータは、色覚異常者にとっても正常者とほぼ近い色差として認識させるように変換する調整方法のパラメータで、例えば、色覚異常者と正常者の色差の比率、色覚正常者にも区別できない色差、色覚異常者にも十分に区別できる色差、色覚異常者に識別できない色を変換する際のその変換更新量と割合からなる。これらのパラメータは、将来、色覚異常者の見え方について測定データが得られ次第、置換できるものである。ステップ3では、該入力された画像に色覚異常者が混同しやすい色の組み合わせがあるか走査検出を行う。該検出の方法は、まず、色覚正常者にも区別できない色、色覚異常者にも十分に区別できる色を検出、除外し、さらに色覚異常者と正常者の色差の比率を前記パラメータで制御することで混同しやすい色を検出する。該手法の概念図を図2に示す。
Hereinafter, an example of the color conversion apparatus of the present invention will be described with reference to the drawings.
FIG. 1 is a flowchart showing processing of the color conversion apparatus. First, image data consisting of color values or RGB values or CMYK values to be converted is input to the conversion device (step 1) and displayed on a monitor or the like. Next, parameters for adjustment as required are input (step 2). The parameter is a parameter of an adjustment method for converting the color difference so that it can be recognized as a color difference that is almost similar to that of a normal person, for example, the ratio of the color difference between a color blind person and a normal person, It consists of a color difference that can be sufficiently distinguished even by a color blind person, and a conversion update amount and rate when converting a color that cannot be distinguished by a color blind person. These parameters can be replaced as soon as measurement data is obtained regarding the appearance of color blind persons. In step 3, scanning detection is performed to determine whether there is a color combination that is likely to be confused by a color blind person in the input image. The detection method first detects and excludes a color that cannot be distinguished by a color blind person and a color that can be sufficiently distinguished by a color blind person, and further controls the ratio of the color difference between the color blind person and the normal person using the above parameters. In this way, colors that are easily confused are detected. A conceptual diagram of this technique is shown in FIG.

次にステップ4で、入力元画像に用いられている色を複数の色組に分けるクラスタ化を実施する。ステップ4、5は、本発明において、必須ではないが、高速に処理し、自然な変換画像を得るために推奨されるステップである。該クラスタ化の概念を示したものが図3である。この処理により前記混同しやすい色に近似する色も合わせて検出することが可能となる。次に前記ステップ3で検出された危険色を含有する前記ステップ4の色組(クラスタ)を選別する。クラスタは、例えば、クラスタに変換対象となる危険色が半数以上占める場合、該クラスタを変換対象とするなどの基準で選別される。   Next, in step 4, clustering is performed to divide the colors used in the input source image into a plurality of color sets. Steps 4 and 5 are not essential in the present invention, but are recommended for processing at a high speed and obtaining a natural converted image. FIG. 3 shows the concept of the clustering. By this processing, it is possible to detect a color that approximates the color that is easily confused. Next, the color group (cluster) in Step 4 containing the dangerous color detected in Step 3 is selected. For example, when more than half of the dangerous colors to be converted occupy a cluster, the clusters are selected based on a criterion such that the cluster is to be converted.

図4に示すように、クラスタに含まれる色の中で変換対象色が一定の割合以上(例えば、半数以上)を占める場合はそのクラスタを変換対象クラスタとする(ステップ5)。そして、元画像の変換対象となる該当クラスタのみを識別可能とする画像を生成する(ステップ6)。ステップ4,5を省略した場合は、ステップ6では、ステップ3の結果が表示される。ステップ6の結果は、前記変換対象クラスタを例えば黒く塗りつぶして表示できるようにすれば、変換前に変換対象クラスタを確認することができる。   As shown in FIG. 4, when the conversion target color occupies a certain ratio or more (for example, more than half) among the colors included in the cluster, the cluster is set as the conversion target cluster (step 5). Then, an image that can identify only the corresponding cluster to be converted of the original image is generated (step 6). When steps 4 and 5 are omitted, in step 6, the result of step 3 is displayed. As a result of step 6, if the conversion target cluster can be displayed in black, for example, the conversion target cluster can be confirmed before conversion.

次に、ステップ6で得られた変換対象色に対して、色変換を行う。色変換は、色覚異常者の色の混同度評価する手段と色の変更の割合を評価する手段とからなり該混同度と該変更度の和が最小となるように、前記危険色を繰り返し変換する(ステップ7)。そして該変換された画像を表示若しくは印字(ステップ8)し、確認評価を行う(ステップ9)。評価の結果、パラメータを再度制御して、繰り返し調整を行うことも可能となる。   Next, color conversion is performed on the conversion target color obtained in step 6. The color conversion comprises means for evaluating the degree of color confusion of a person with color blindness and means for evaluating the rate of color change. (Step 7). Then, the converted image is displayed or printed (step 8), and confirmation evaluation is performed (step 9). As a result of the evaluation, it is possible to repeatedly adjust the parameters by controlling them again.

次に前記色の混同度評価と変更度について例を挙げて示す。図5は、色に対する人間の視細胞の応答を表すLMS空間であり、色覚正常者の認識する2色の組み合わせの色の差と、色覚異常者の認識する色の該2色の色の差を示したものである。   Next, an example is given and shown about the said color confusion evaluation and change degree. Fig. 5 is an LMS space representing the response of the human photoreceptor to the color. The color difference between the two colors recognized by a normal color blind person and the color difference between the two colors recognized by a color blind person Is shown.

まず、色Ciの混同度は、CIELUV均等色空間などの色空間内で他の色Ckとの距離に基づいて例えば、式1のように定義する。混同度は、混同の度合いが大きいほど大きな値を示す。   First, the degree of confusion of the color Ci is defined as, for example, Equation 1 based on the distance from the other color Ck in a color space such as the CIELUV uniform color space. The degree of confusion shows a larger value as the degree of confusion increases.

Figure 0004200887
Figure 0004200887

次に、色Ciの変更度をCIELUV均等色空間などの色空間内での変更前の色Coとの距離に基づいて例えば、式2のように定義する。変更度は、変更の度合いが大きいほど大きな値を示す。 Next, the degree of change of the color Ci is defined as, for example, Expression 2 based on the distance from the color Co before the change in a color space such as the CIELUV uniform color space. The degree of change shows a larger value as the degree of change is larger.

Figure 0004200887
Figure 0004200887

変更対象色の変換は、式1と式2の和を最小とするように、色Ciの色を変更する。 In the conversion of the change target color, the color Ci is changed so that the sum of Expression 1 and Expression 2 is minimized.

Figure 0004200887
Figure 0004200887

式3が最小となるように変換対象色を変更更新する際には、最急降下法やランダムサーチ法など様々な既知の算出法を使うことが出来る。   When the conversion target color is changed and updated so that Equation 3 is minimized, various known calculation methods such as the steepest descent method and the random search method can be used.

図6は、本発明の装置の構成の例を示した図である。本装置は画像を入力する外部入力制御と、入力された画像が色彩値でない場合、該データを色彩値に変換する色データ制御と、色彩値からなるデータを色彩異常者にとっても認識できる色に変換する該変換制御と、変換された画像を外部出力装置に出力するためのデータに変換する外部出力制御とからなる。外部出力装置は、モニタやプリンタなど色を確認するための手段であり、変換された画像の色が忠実に再現されることが望ましい。   FIG. 6 is a diagram showing an example of the configuration of the apparatus of the present invention. This device has an external input control for inputting an image, a color data control for converting the input image into a color value when the input image is not a color value, and a color that can be recognized by color-impaired persons. The conversion control includes conversion and external output control for converting the converted image into data for output to an external output device. The external output device is a means for confirming the color, such as a monitor or a printer, and it is desirable that the color of the converted image be faithfully reproduced.

カラー変換装置の処理を示すフローチャートFlow chart showing processing of color conversion device 色覚異常者にとって区別しにくい色使いの検出の概念図Conceptual diagram of color detection that is difficult to distinguish for people with color blindness 画像の色クラスタ化の概念図Conceptual image color clustering 警告対象クラスタの決定概念図Conceptual diagram for determining the alert target cluster 色覚異常者にとって区別しにくい色使いの検出の概念図Conceptual diagram of color detection that is difficult to distinguish for people with color blindness 装置の構成の例Example of device configuration

Claims (1)

入力したカラー画像の色変換装置であって、
前記入力したカラー画像の各画素の色情報の配色組み合わせを検索し
当該配色組み合わせが、色覚異常者にとって混同しやすいか否かを決定する手段と、
前記決定する手段において、混同しやすいと決定された配色組み合わせについては
色覚異常者の認識色空間内での色の差で表す混同度と、色変換後と変換前の色の差であらわす変更度との和により構成される評価関数最小となるように当該配色組み合わせの画素の色情報を変換する手段とを有することを特徴とする色変換装置。
A color conversion device for an input color image,
The color combination of the color information of each pixel of the input color image is searched.
Means for determining whether the color combination is likely to be confused for color blind persons;
In the means for determining the color difference between before conversion and confused degree, and after the color conversion expressed by the color difference in the recognition color space of <br/> color defectives for color combinations that are determined to confuse And a means for converting the color information of the pixels of the color combination so that the evaluation function constituted by the sum of the degree of change and the degree of change is minimized.
JP2003402649A 2003-11-21 2003-12-02 Color conversion device Expired - Fee Related JP4200887B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2003402649A JP4200887B2 (en) 2003-11-21 2003-12-02 Color conversion device
US10/991,428 US20050111019A1 (en) 2003-11-21 2004-11-19 Color image processing method and color image processing system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2003392021 2003-11-21
JP2003402649A JP4200887B2 (en) 2003-11-21 2003-12-02 Color conversion device

Publications (2)

Publication Number Publication Date
JP2005173634A JP2005173634A (en) 2005-06-30
JP4200887B2 true JP4200887B2 (en) 2008-12-24

Family

ID=34742020

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2003402649A Expired - Fee Related JP4200887B2 (en) 2003-11-21 2003-12-02 Color conversion device

Country Status (1)

Country Link
JP (1) JP4200887B2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CZ306549B6 (en) * 2005-01-27 2017-03-08 Samick Precision Ind. Co., Ltd. A method of producing linear bushing

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008129162A (en) * 2006-11-17 2008-06-05 National Institute Of Advanced Industrial & Technology Video conversion processing method and video conversion system
JP2008237406A (en) * 2007-03-26 2008-10-09 Samii Kk Image display control device and method, and game machine
JP4981531B2 (en) * 2007-06-19 2012-07-25 キヤノン株式会社 Image processing apparatus, image processing method, and program

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CZ306549B6 (en) * 2005-01-27 2017-03-08 Samick Precision Ind. Co., Ltd. A method of producing linear bushing

Also Published As

Publication number Publication date
JP2005173634A (en) 2005-06-30

Similar Documents

Publication Publication Date Title
EP1909484B1 (en) Image processing device, image processing program, and image processing method
US7873213B2 (en) Systems and methods for color-deficient image enhancement
JP5179829B2 (en) Gray data correction apparatus and method
JP2004266821A (en) Converted digital color image with improved color distinction for color blindness
JP2009055465A (en) Image processing device and method
US20100321400A1 (en) Image processing apparatus, image processing method, and computer program product
US20050156942A1 (en) System and method for identifying at least one color for a user
JP2008129162A (en) Video conversion processing method and video conversion system
JP4764781B2 (en) Color vision conversion system and color vision conversion network system
JP4200887B2 (en) Color conversion device
JP2006238339A (en) Digital camera
JP2011182006A (en) Color conversion device and image forming apparatus
KR20050011115A (en) Method and apparatus for car equipped color compensation for color blindness
JP4200888B2 (en) Color correction device
EP1706081B1 (en) System and method for identifying at least one color for a user
JP4859927B2 (en) Method, system, and device for color quality control
JP2017167291A (en) Image processing apparatus, image processing method, and image processing program
JP2008275776A (en) Color display device
US20050111019A1 (en) Color image processing method and color image processing system
JP2005182432A (en) Color conversion apparatus
JP2005185588A (en) Color warning device
JP2005346273A (en) Method and program for teaching color existence for color-sense abnormal person
JP4272972B2 (en) Color chart for color adjustment and gradation adjustment of color printers and color monitors in endoscopes
JP2000251076A (en) Image evaluating device
JP3795422B2 (en) Visual load degree measuring method, apparatus, program, and recording medium recording the program

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20060811

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20080507

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20080704

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20080916

A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20080929

R150 Certificate of patent or registration of utility model

Ref document number: 4200887

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150

Free format text: JAPANESE INTERMEDIATE CODE: R150

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20111017

Year of fee payment: 3

S533 Written request for registration of change of name

Free format text: JAPANESE INTERMEDIATE CODE: R313533

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20111017

Year of fee payment: 3

R350 Written notification of registration of transfer

Free format text: JAPANESE INTERMEDIATE CODE: R350

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20111017

Year of fee payment: 3

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20121017

Year of fee payment: 4

FPAY Renewal fee payment (event date is renewal date of database)

Free format text: PAYMENT UNTIL: 20131017

Year of fee payment: 5

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

R250 Receipt of annual fees

Free format text: JAPANESE INTERMEDIATE CODE: R250

LAPS Cancellation because of no payment of annual fees