JP3829506B2 - Document classification method, document classification apparatus, and recording medium on which document classification processing program is recorded - Google Patents

Description

【００６４】
で表される。なお、ハニング関数を示す図４は横軸に文字位置ｉをとり、縦軸に重みをとったもので、この図４の場合は、重み付けをする範囲Ｗは、ハニング窓の中心位置ｌに対し左右両側に１５文字分をとった場合が示されている。
【００６５】
このように、処理対象文書を１次元文字列に展開して、キーワードの出現位置からキーワード出現密度を求めるのは、前述した第１の文献（黒橋、白木、長尾：出現密度分布を用いた語の重要説明箇所の特定、情報処理学会論文誌、Vol.38，No.4，pp.845-854 (1997)）に述べられており、これは公知の技術である。ただし、この第１の文献では、キーワード位置としては、それぞれのキーワードの先頭位置としているが、本発明では、それぞれのキーワードの中心位置としている点は異なる。
【００６６】
なお、重み関数ｈｌ（ｉ）としてはここではハニング関数を用いたが、これは、中心から周辺になだらかに変化する関数ならば色々使える。たとえば、ガウス関数もその１つであるが、ここでは、重み付け範囲Wでゼロになるハニング窓関数を用いる。ガウス関数は無限遠方でもゼロにならないため、キーワード近傍の影響だけ調べるにはハニング窓関数が向いていると考えられる。しかし、応用の仕方によって関数を適宜変えることは可能である。なお、ガウス関数でも重み付け範囲W以外は強制的にゼロとするような設定を行えば使える。
【００６７】
図５は上述した計算方法で文書密度を計算した例である。この文書はある商品についてのレポートであるが、その商品の評判に関するキーワードを適切に設定しておくと、文書中で商品の感想、批評の記述に関する部分の密度が高くなっていることがわかる。
【００６８】
この図５は、処理対象文書が商品についてのレポートであって、このレポートに対し、前述の（１）式を用いてキーワード出現密度を求めることによって得られたキーワード出現密度分布を示すものである。図５において、横軸は処理対象文書の文字位置、縦軸は上述の（１）式で求められたキーワード出現密度ｄ（ｌ）を示している。
【００６９】
なお、横軸の文字位置は、レポートの文書を１次元文字列に展開し、その文書の先頭から文書末までを文書の先頭からのバイト数で表している。また、縦軸のキーワード出現密度は、（１）式で求められたキーワード出現密度ｄ（ｌ）であるが、その処理対象文書について求められたキーワード出現密度の中で最大値を1.0にした正規化を行って表わされている。このように、処理対象文書ごとに正規化を行うことで、他の処理対象文書との比較が容易となる。
【００７０】
この図５に示される処理対象文書は、具体的にはあるメーカのディジタルカメラについての評価をまとめたレポートであり、設定されたキーワードとしては、たとえば、「評価」、「性能」、「価格」、「使い勝手」などであるとし、これらのキーワードは、デザインについて触れた部分や、画質について触れた部分でのキーワード出現密度が特に高く、その他、パノラマ機能などについてやメモリなどについて触れた部分においてもキーワード出現密度が比較的高いことがわかる。
【００７１】
ところで、上述の密度計算を行う際、重み付け範囲Wの設定の仕方により密度の出方が変わる。一般には重み付け範囲Wを広くとると、キーワード出現位置近傍のキーワード出現密度が求められることになり、逆に重み付け範囲Ｗを広くとると、広い範囲に存在するキーワードの影響まで計算されることになる。しかし、重み付け範囲Wが広すぎると重要箇所が不明瞭になり、狭すぎると単なるキーワード頻度と変わりなくなるので、重み付け範囲の設定の仕方は重要である。
【００７２】
そこで、それぞれの処理対象文書ごとに１つの処理対象文書の文書長により適切な重み付け範囲W を設定する必要がある。図６は重み付け範囲Ｗの取り方によって、前述の（１）式により求められるキーワード出現密度ｄ（ｌ）分布曲線がどのようになるかを示したものである。図６において、分布曲線Ｕ１は重み付け範囲Ｗを４０９６バイト、分布曲線Ｕ２は重み付け範囲Ｗを２０４８バイト、分布曲線Ｕ３は重み付け範囲Ｗを１０２４バイトとして、それぞれ（１）式を用いて求められたキーワード出現密度ｄ（ｌ）の分布曲線を示すものである。なお、分布曲線Ｕ３は図５の分布曲線とほぼ同じである。つまり、図５は重み付け範囲Ｗを１０２４バイトとした場合であるといえる。
【００７３】
この図６からわかるように、重み付け範囲Ｗを広くとりすぎると、密度の変化に明確性を欠き（範囲Ｗを２０４８バイトや４０９６バイトとした場合）、重要箇所が不明瞭となる。この３つの範囲の中では、重み付け範囲ｗを１０２４バイトとした場合が最も適切であるといえる。この場合、ここで用いられた処理対象文書の文書長が、1.2×１０の４乗バイトの文書長を有する場合であるので、おおよそ、１つの処理対象文書の文書長の１／１０を目安にするのがよいことがわかる。特に、技術系の文書においてはその傾向が強いことがわかった。
【００７４】
次に、このようにして求められたキーワード出現密度を用いてシステム上で文書を分類する方法について説明する。
【００７５】
前述した説明では、処理対象文書はプレインテキストであることを前提にしているが、実際に我々が扱う文書は色々な形式が存在する。したがって、キーワード位置検出を行う場合、文書形式によって検出方法が異なる。一般に前述したキーワード出現位置というのは、人間が視覚的に見たレイアウト上の位置である。処理対象文書が蓄積されている文書ファイルの内容自体には、文字自身の情報の他に制御に関する情報も多いので、これらを適宜排除した上でないと視覚的に見たレイアウトに一致するような位置を見つけることは難しい。たとえば、HTML文書では表示に使うタグが多数あるので、このタグは読み飛ばして文字の位置を計算する必要がある。
【００７６】
すなわち、本発明を実施するには、文書を制御記号などの存在しない文字だけの一次元文字列に変換して行うが、たとえば、文書中に様々な制御記号などが入った文書にあっては、それらの制御記号を省いて文字だけで１次元文字列を構成する必要がある。
【００７７】
したがって、処理対象文書を文字だけによる１次元文字列に展開し、キーワード位置検出を行う処理は、処理対象文書ごとに、その文書がどのような形式の文書であるかを判定し、その判定結果に基づいて、処理対象文書の文書形式ごとにアルゴリズムを設定して行う必要がある。しかし、これを実際に行うのは処理負担が大きく実用的ではない。
【００７８】
したがって、ここでは、どのような文書でも全て一旦プレインテキストに変換し、その上でキーワード出現位置の検出を行ってキーワード出現密度計算を行う方法をとる。この方法をとれば、文書形式ごとにキーワード位置検出アルゴリズムを開発する必要がない。また、プレインテキスト変換は殆どの文書作成ソフトで備えられている機能であるので、システムに組み込むことが容易である。
【００７９】
また、全ての文書形式をプレインテキストに変換してキーワード位置検出を行う場合は、キーワード位置検出は、単純なキーワードマッチングで可能となる。しかし、プレインテキストに変換せず、各文書形式ごとにキーワード位置を見る場合は、それぞれの文書形式ごとに、キーワード位置検出アルゴリズムが必要となる。具体的には、タグや特殊記号、ヘッダなどの知識と、それらがどのような場合は何文字飛ばすなどの指示が必要である。
【００８０】
次にこれまで説明した方法により、多数の文書について図５に示すようなキーワード出現密度分布が求められたとして、それぞれの文書の中から最も適当と思われる文書を抽出する処理（文書分類処理）について説明する。
【００８１】
この文書分類の仕方としては、まず、図５に示すように求められたキーワード出現密度の総和を求めて（積分する）、その中から最も大きい値の文書を出力したり、あるいは、値の高い順から幾つかの文書を出力する方法が考えられる。
【００８２】
これとは別の方法としてエントロピを計算して分類する方法もある。この方法は、上述の総和だけでは判定が微妙となるような場合に有効なものとなる。
【００８３】
たとえば、図１に示すような３つの文書Ａ〜Ｃについて考えると、この３つの文書Ａ〜Ｃは、キーワード（×印）の数はどれも１０個で同じであり、また、その密度を積分した値もほぼ同じであるとする。しかし、これらの文書のキーワード分布はそれぞれ大きく異なっている。つまり、文書Ａはページのある一部に詳しい情報がまとまって記載されており、文書Ｂはページ全体に関連する記事があるが、その内容は薄いと思われる。また、文書Ｃは文書Ａと文書Ｂの中間程度の詳しい情報が２箇所あるというように、それぞれの文書の分布形態は異なる。
【００８４】
これらの点から、抽出されるべき文書が、文書Ａあるいは文書Ｃであることが望ましいとして、これら文書Ａあるいは文書Ｃを選択するような処理がなされるような処理を行う。これを、キーワード出現密度以外の値（キーワードのヒット数やキーワードが存在した場所の数など）で総合的に判定するのは容易ではない。つまり、キーワード出現密度以外の値で総合的に判定しようとすると、判定の条件分岐が複雑になり、微妙な判定が難しくなる。そこで、エントロピを使う。求めるべきエントロピをＥとすると、
【００８５】
【数２】

【００８６】
で表される。
【００８７】
エントロピＥはそれぞれの処理対象文書におけるキーワード出現密度ｄ（ｌ）の分布が平坦であると小さく、先鋭な部分が多い分布では大きくなる傾向をもっている。したがって、それぞれの処理対象文書ごとのキーワード出現密度全体の状況を一意に表現する尺度として有用だと考えられ、図１に示すようなキーワード出現密度の総和が似ている場合の微妙な判定に役立つ。
【００８８】
図７は処理対象文書として９０個の雑誌（ある商品の評判が書かれている）の内容を主観評価で３段階に分類した結果と、文書密度のエントロピＥの相関をとったグラフである。ここで、主観評価で３段階の分類というのは、たとえば、Ｈ３は商品の評判について非常によく書かれている文書群、Ｈ２は商品の評判について比較的よく書かれている文書群、Ｈ１は商品の評判について一応書かれている文書群というように、ここでは主観的な評価を３つの段階Ｈ１，Ｈ２，Ｈ３に分けて、９０個の雑誌をそれぞれの段階に分類している。
【００８９】
このように粗い分類をしたのち、それぞれの処理対象文書（それぞれの雑誌）についてエントロピを計算する。エントロピは先鋭な部分の多いキーワード出現密度分布ほど大きな値となるので、エントロピの値の大きいものほどその内容についてよく書かれているものであるといえる。
【００９０】
上述したような３つの段階Ｈ１，Ｈ２，Ｈ３の分類は、かなり粗い分類であるが、それぞれの分類の中で、内容が濃くなるほどエントロピの値も高いものとなるので、エントロピを指標にして文書内容の程度を定量的に評価できる。なお、図７において、それぞれの文書に対して求められたエントロピを１つ１つの四角形で表している（たとえば、段階Ｈ１の文書群については、求められたエントロピをＥ１１，Ｅ１２，・・・， Ｅ１ｎで表し、Ｈ２の文書群については、求められたエントロピをＥ２１．Ｅ２２，・・・，Ｅ２ｎで表し、 Ｈ３の文書群については、求められたエントロピをＥ３１，Ｅ３２，・・・，Ｅ３ｎで表している）が、それぞれの文書群において求められたエントロピは似た値になる場合もあり、その場合は、エントロピの値を示す四角形が重なるので、四角形は９０個全ては図示されてはいない。
【００９１】
このようにして幾つかの文書がユーザの所望とする文書候補として抽出されるが、これら抽出された複数の文書が一定以上存在する場合は、有用度の高い順から並べて表示する。このような表示を行う際、有用度の高い順から並べただけでは内容が解りにくいので、その他の情報も合わせて表示することが好ましい。
【００９２】
図８はその表示例を示すものである。図８（ａ）において、＃１，＃２，＃３，・・・は有用度の高い順番を示すもので、その順番に対するそれぞれの文書の重要度ｚ１としてキーワード出現密度、求められたエントロピ、キーワードヒット数なども表示する。さらに、文書の概要（見出しや文書の冒頭部分を抽出）ｚ２、その文書の重要部分（これについては後述する）ｚ３などの併せて表示する。さらに、図８（ｂ）に示すように、それぞれの文書ごとに文書全文の特にキーワード出現密度の高い部分を色やブリンクでの表示を可能とする。すなわち、それぞれの文書対応にアイコンのようなマークＭ１，Ｍ２，Ｍ３などを付して、たとえば、マークＭ１をクリックすると、＃１の文書全体を表示し、かつ、その文書の中で特に密度の高い部分ｚ０を他の部分とは異なった色やブリンクで表示する。
【００９３】
本発明は複数の文書を分類することが主な目的としているが、分類した文書のどのが重要だったか示すことは、その文書が選ばれた理由を知ったり、分類された文書の概要を把握する上で有用である。以下にその有用部分の切り出しについて説明する。
【００９４】
まず、文書の有用部分を特定する方法を述べる。基本的には図１のように、局所的にキーワード出現密度の高い部分（山の部分）を有用部分とし、あるしきい値を設定して、そのしきい値以上の密度を有する部分を有用部分として抽出することができる。
【００９５】
なお、これまでの説明では、文書分類を主としているので、キーワード出現密度は文書の全ての文字位置について計算をするが、有用部分切り出しを主とし、それとともに文書分類も行いたいというような場合は、キーワードが出現した位置のみのキーワード出現密度から文書全体の様子を推定することも可能である。
【００９６】
図９はキーワード出現位置のみについて、キーワード出現密度と全ての文字位置について計算した例を比較して示すもので、図５で説明した文書と同じ文書を用いた場合である。この図９からもわかるように、キーワード出現位置のみにおけるキーワード出現密度のピークｐ１，ｐ２，ｐ３，・・・を包絡線でつないで得られたキーワード出現密度分布曲線Ｌは、文書の全ての文字位置についてのキーワード出現密度分布曲線（図５参照）とほぼ同じになる。
【００９７】
このように、キーワード出現位置のみについてキーワード出現密度を求める方法は、計算時間が速いので、多少精度が悪くても、速くおおよその判定をしたい場合などに有効である。
【００９８】
次に有用部分の表示方法について述べる。抽出するかしないかの基準は上述したように、適当なしきい値以上のキーワード出現密度部分を単純に切り出せばよいが、そのまま単純に切り出すと、不具合が生じ、それに対処する必要がある。
【００９９】
これを示したものが図１０である。図１０（ａ）で示すような元文書（その一部のみが図示されている）が存在したとし、アンダーライン部分が、あるしきい値以上のキーワード出現密度を有する有用部分として抽出された区間であるとする。
【０１００】
しかし、この抽出された区間の内容は、この場合、「い表示を後処理と」であり、これでは、何が書かれているのか意味がわからないことになる。そこで、図１０（ｂ）に示すような形態素解析を行い、語句の切れ目が自然なものとなる境界を有用部分として抽出する。つまり、この場合は、形態素解析結果は、「無理」・「が」・「ない」・「表示」・「を」・「後処理」・「として」・「行う」・「こと」であり、このような形態素解析結果において、「表示」・「を」・「後処理」・「として」を抽出し、図１１（ｃ）に示すように、「表示を後処理として」という内容を抽出する。
【０１０１】
この例では、有用部分の先頭に助詞（上述の例では「い」）が単独で出現する場合はそれを無視し、語尾の一部（上述の例では「と」）が存在しているときは語尾として意味をなすような処理を行い、有用部分が意味のある文章になるようにしている。このように、形態素解析を行ってその結果に基づき、有用部分として抽出された部分の文が不自然なものとならないようにすることができる。
【０１０２】
以上は文書が文字だけのテキストの場合について述べたが、実際にはHTML文書などのレイアウトや画像などの複数の要素が組み合わされた文書の場合、密度の高い場所だけ抽出して表示を行うと不自然なものとなる。このような場合、次のような方法をとることで、抽出した有用部分を自然な内容として表示できる。
【０１０３】
すなわち、一般文書からテキスト文書に変換する際に、テキスト文書のどの文字はテキスト変換前の元の文書のどの位置かを記録しておく。そして、テキスト文書状態で有用部分を決定し、その有用部分の文字の位置に相当する元の文書の位置を特定する。そして、特定した元文書の位置を中心にレイアウトが不自然でない区切りを判定する。たとえば、その領域を含む最低のかたまり（段落や章）で区切る。ただし、有用部分の中に段落が存在しているような場合、その段落部分で区切ると不自然となるので、前後の段落を含めた少し広い範囲で区切るようにしたり、あるいは、有用部分の中に異なった章が存在しているような場合、その章の変わるで区切ると不自然となるので、前後の章を含めた少し広い範囲で区切るようにしたりする。
【０１０４】
このように、文書密度計算のためのテキスト変換を行った際に、元文書とテキスト文書の位置の対応づけを行っておけば、レイアウトに無理がない表示を後処理として行うことが可能である。
【０１０５】
次に本発明を実現する際のシステム構成を説明する。図１１は、本発明を実現するためのシステム構成図を示すもので、検索要求入力部１、知識データベース部２、キーワード群決定部３、検索対象文書データベース部４、文書タイプ判定部５、テキスト変換部６、キーワード出現密度計算部７、有用文書判定部８、表示部９、有用部分抽出部１０、表示レイアウト生成部１１などから構成されている。なお、知識データベース部２とキーワード群決定部３によってキーワード群設定部２０が構成され、表示部９、有用部分抽出部１０、表示レイアウト生成部１１によって文書出力部３０が構成される。
【０１０６】
検索要求入力部１は、ユーザが何らかの情報を収集しようとしたとき、ユーザの知りたい内容について入力可能となっており、これは、自然言語で入力するようにしてもよく、あるいは、予め多数の項目を用意しておき、ユーザがその中から所定の項目を指示するようにしてもよい。
【０１０７】
知識データベース部２は、ユーザの入力した検索要求内容に基づいてその内容に対してはどのような用語（単語など）がキーワードとして適切であるかを判断して、ユーザの検索入力内容に対する適切な用語を複数個選択し、ここで選択された複数の用語はキーワード群決定部３によって、キーワード群として決定される。ここで設定されるキーワードは、ユーザの入力した検索内容によっては、数十あるいは１００個以上ということもある。
【０１０８】
ところで、ユーザの検索要求に対しそれに適合したキーワードを決定する処理は次のようにして行われる。
【０１０９】
たとえば、検索要求入力部１から、ユーザがある製品について知りたい旨を入力すると、知識データベース部２では、そのユーザの入力に対して、「性能」、「価格」、「評価」、「使い勝手」などその製品を表す上で必要な用語を選び、これらが、キーワードとして決定される。つまり、ユーザ検索要求に対して有用な文書を抽出する際文書の分類が必要となるが、このとき、分類したい分野によって、その分野に関する一般的な知識や連想される用語をキーワードとする。また、連想される用語の類似語や類義語も利用できる。これらは個人の直感または辞書、国語辞典などの事例などから予め作成して知識データベース部２に持たせることができる。また、分類したい分野に該当する文書を複数収集して、これら収集された文書に数多く目にする用語を探し出し、その頻度が高いものをキーワードとする方法も可能である。
【０１１０】
さらに、１つのキーワードについてそのキーワードに対する類義語や類似語などの関連語も適宜組み合わせて使用したり、そのキーワードに一緒に使われる形容詞、副詞、感嘆詞などを含んだ表現とすることで、検索精度がより一層向上する。たとえば、前者の場合、キーワードが「評価」であれば、その類義語や関連後「批判」、「評判」、「好評」などというように「評価」から連想される類義語や類似語を適宜組み合わせて用いる。また、後者の場合は、キーワードが「評価」であれば、「素晴らしい」、「品質のよい」、「非常に」などを適宜組み合わせて用いる。
【０１１１】
文書タイプ判定部５は、検索対象文書データベース部４に存在する文書がどのような形式で書かれた文書であるかを判定するものである。また、テキスト変換部６は、文書タイプ判定部５による判定結果に基づいて、その文書をテキスト文書に変換する。なお、検索対象文書データベース４は、たとえば、インターネット上のサーバ側に存在するデータベースを考えているが、これに限らず、ユーザ個人のパーソナルコンピュータなどの情報処理機器に保存されている文書であってもよい。
【０１１２】
キーワード出現密度計算部７は、キーワード群決定部３で決定された複数のキーワードに基づき、テキスト変換された文書に対し前述したような方法でキーワード出現密度を計算する。このキーワード出現密度計算については、すでに詳細に説明したので、ここではその説明は省略する。
【０１１３】
このキーワード出現密度計算部４によってそれぞれの文書について、キーワード出現密度が計算されると、有用文書判定部８は、それぞれの文書ごとに、たとえば図５に示すようなキーワード出現密度分布を作成して、その結果に基づいて、有用度（ユーザの所望とする文書としての有用度）を判定し、その判定結果を用いて表示すべき文書を決定する。なお、重要度の判定は、前述したように、たとえば、図５に示すようなグラフを積分してその結果よって判定する。
【０１１４】
一方、有用部分抽出部１０は、キーワード出現密度計算部４によって計算されたそれぞれの文書ごとに計算されたキーワード出現文書密度を用いて、前述したような方法により有用部分の抽出を行う。そして表示レイアウト生成部１１によって、図８に示すような表示レイアウトを生成し、それを表示部９で表示する。その表示結果の一例が図８である。
【０１１５】
以上のような手順によって、ユーザの検索要求に対し、それに適合した幾つかの文書が表示されることになる。
【０１１６】
なお、本発明は以上説明した実施の形態に限定されるものではなく、本発明の要旨を逸脱しない範囲で種々変形実施可能となるものである。たとえば、前述の実施の形態では、表示部９での表示例は図８のような表示の仕方であるが、この表示の仕方は種々設定可能である。
【０１１７】
また、以上説明した本発明の文書分類処理を行う処理プログラムは、フロッピィディスク、光ディスク、ハードディスクなどの記録媒体に記録させておくことができ、本発明はその記録媒体をも含むものである。また、ネットワークから処理プログラムを得るようにしてもよい。
【０１１８】
【発明の効果】
本発明によれば、処理対象文書ごとに、設定されたキーワードが当該処理対象文書内にどのような密度で存在するか（キーワード出現密度）を判断し、このキーワード出現密度に基づいてユーザの検索要求に対する文書の有用度について判定するようにしているので、膨大な文書の中からユーザの所望とする文書を取り出す際の前処理としての文書分類を、精度よく効率的に、しかも、少ない計算量で可能となる。
【０１１９】
このように、計算量を少なくできる１つの例として、本発明では、キーワード出現位置はそれぞれのキーワードを構成する文字の中心の文字またはその中心付近のある１つの文字としていることが挙げられる。すなわち、ある１つのキーワードを構成する文字数に関係なく、キーワード出現位置を示すデータは、１つのキーワードについて中心またはその付近の１つの文字に対応した位置のデータのみを持っていればよいので、キーワードを構成する文字全てについてキーワード出現位置を示すデータを持つ場合に比べて大幅にデータ量を少なくすることができ、またそれに伴う計算量も大幅に少なくすることができる。
【０１２０】
また、本発明では、処理対象文書を構成する全ての文字位置についてキーワード出現密度を計算することによって、より高精度なキーワード出現密度分布を得ることも可能であるが、それぞれの処理対象文書内の前記キーワードの出現位置についてのみキーワード出現密度を計算することも可能であり、これによれば、多少おおまかなキーワード出現密度分布となるが、計算量が少なく高速な処理が可能となる。
【０１２１】
また、処理対象文書をテキスト変換した後にキーワード出現密度計算を行うことで、どのような形式の文書に対してもキーワード出現密度計算以降の処理を共通化することができる。しかも、テキスト変換を行う際、テキスト変換前の元文書とテキスト変換後の文書との位置の対応付けを行って、その対応付けした内容を保存しておくことで、有用部分を表示する場合、無理のない自然なレイアウトでの表示が行える。
【０１２２】
また、それぞれの処理対象文書が有用であるか否かの判定は、算出されたキーワード出現密度の総和を求めて、その総和の値によって判断することができ、このように、キーワード出現密度の総和によって有用度を判断する方法は、それぞれの文書ごとにキーワード出現密度の総和を求めるだけでよいので、少ない計算量で容易に有用度の判定を行うことができる。
【０１２３】
また、それぞれの処理対象文書が有用であるか否かの判定を行う際、それぞれの処理対象文書において求められたキーワード出現密度のエントロピを計算してそのエントロピから判断することもできる。これによれば、キーワード出現密度の総和だけでは、判定が微妙となるような場合でも、高精度な有用度の判定が行える。
【０１２４】
また、分類された結果の出力は、分類を行うために求められた有用度の高い文書から順番に出力し、その出力内容は、有用度やどの部分が有用であるかを示す有用部分さらにそれぞれの文書の要約など（全てでなくともよい）を表示可能とするので、ユーザは、出力された文書内容を即座に把握することができる。
【０１２５】
そして、有用部分を表示する場合、有用部分としては、キーワード出現密度の特に高い部分の区間を設定しその区間を有用部分として抽出し、少なくともその抽出された区間を含む所定範囲を形態素解析して、その形態素解析結果を用いて意味のある内容として抽出するようにしたので、有用部分を１つのまとまった内容の文として出力することができる。
【０１２６】
また、上述の出力内容についてさらに詳細な内容を出力可能としておき、その詳細な文書内容において、重要部分は色を変えたりブリンクさせるようにすることによって、ユーザは自分の知りたい情報を一目で見ることができる。
【０１２７】
また、ユーザの検索要求入力によって、システム側がその検索要求に基づいて適正な用語をキーワードとして選択するようにしているので、ユーザ自身がキーワードの設定を行う必要がない。したがって、ユーザは何を知りたいかを検索要求として入力するだけでよいので、ユーザの行う入力操作をきわめて簡単なものとすることができる。また、キーワードとして設定された用語に関連する類義語や類似語さらにはキーワードに付加される形容詞、副詞、感嘆詞などをもキーワードとすることができるので、これによって、より一層、検索精度が向上し、ユーザの所望とする情報を適切に取り出すことができるようになる。
【０１２８】
このように、本発明は、計算量が少なく高精度な文書分類が可能となりる。しかも、どこがどの程度有用なのかを適切に表示することができる。また、本発明を適用すれば、作成した文書にキーワードやインデックスをつけて分類できるように準備しておく必要がなく、作成された文書に対し後に行われる分類処理について何等意識する必要がない。つまり、どのような文書であっても、キーワードさえ設定されれば適切な分類がなされる。
【図面の簡単な説明】
【図１】本発明の実施の形態を説明するためにキーワード出現密度の概念を説明する図である。
【図２】処理対象文書を１次元文字列に展開してキーワード出現位置を求める例を説明する図である。
【図３】個々のキーワードの出現位置を決定する処理を説明する図である。
【図４】ハニング窓関数の一例を示す図である。
【図５】処理対象文書の１次元文字列上におけるキーワード出現密度分布を示す図である。
【図６】キーワード出現密度を計算する際にハニング窓の幅（重み付け範囲）を変えて計算して得られたキーワード出現密度分布を示す図である。
【図７】複数の処理対象文書を３段階の文書群に分けてそれぞれの文書ごとに得られたキーワード出現密度からエントロピを求めてそのエントロピの分布を示す図である。
【図８】本発明の実施の形態の表示例を示す図であり、（ａ）は有用度の大きい順に必要な表示項目を併せて表示する表示例を示す図、（ｂ）はその中から指定された文書内容を表示する例を示す図である。
【図９】キーワード出現位置のみでキーワード出現密度を計算した場合の１次元文字列上におけるキーワード出現密度分布を示す図である。
【図１０】有用部分の抽出処理法法を説明する図である。
【図１１】本発明の実施の形態のシステム構成を示す図である。
【符号の説明】
１ 検索要求入力部
２ 知識データベース部
３ キーワード群決定部
４ 検索対象文書データベース部
５ 文書タイプ検出部
６ テキスト変換部
７ キーワード出現密度計算部
８ 有用文書判定部
９ 表示部
１０ 有用部分抽出部
１１ 表示レイアウト生成部
２０ キーワード設定部
３０ 文書出力部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a document classification method, a document classification device, and a recording medium on which a document classification processing program is recorded for determining whether a document is related to a certain field from a large number of documents.
[0002]
[Prior art]
It has been conventionally performed to determine whether a document is related to a certain field from among a large number of documents. In particular, recently, with the spread of the Internet, it has become possible to retrieve information desired by users relatively easily. However, an enormous amount of documents exist on the Internet, and it is not easy to appropriately determine and take out a document desired by the user from among them. As described above, in order to extract a document desired by a user from an enormous amount of documents, when a user's search request is received, a large number of documents are classified by field based on the search request. Processing such as extracting a document desired by the user is required. Various techniques for enabling such document classification have been proposed.
[0003]
To date, the most commonly used method for classifying multiple documents is to set some keyword and find out how many such keywords exist in each document in order to search for the document desired by the user. . A document having many keywords can be said to be a document having a high degree of usefulness desired by the user.
[0004]
However, since this method only looks at whether or not a keyword exists, it is effective to some extent when narrowing down candidates from a large number of documents. Is difficult. However, since this is a simple keyword matching, it is also used in applications that require real-time processing to some extent, such as the web.
[0005]
On the other hand, in addition to simply determining whether or not a keyword exists, in each document, effective terms in the document are selected from the appearance frequency and ratio of each term in the entire document. A method of improving the accuracy of document classification by matching keywords input at the time of search with those terms is also used. As a typical method, the TF / IDF method is known. This method is more accurate than the keyword matching method in a simple document.
[0006]
[Problems to be solved by the invention]
However, the TF / IDF method described above is quite effective when, for example, a document for a certain product is accurately searched from among many documents, but the reputation of the product is determined from the document for the certain product. It is difficult to find a document with detailed information. Moreover, since it is necessary to calculate the word frequency for all documents to be searched in advance, there is a problem that the calculation amount is large.
[0007]
In addition, it is necessary to divide sentences into words by morphological analysis, and when the number of search target documents increases, the amount of calculation becomes enormous. Therefore, it is difficult to find a corresponding document immediately by simply giving a keyword.
[0008]
Thus, the main document classification method used so far uses the frequency of appearance of keywords. On the other hand, we are not simply looking at the frequency, but at what density the related keywords are present in the document. In fact, it has been reported that the keyword appearance density considering the appearance position of the keyword accurately indicates the important place in the document (Kurohashi, Shiraki, Nagao: Important explanation part of the word using the appearance density distribution) Identification, IPSJ Journal, Vol.38, No.4, pp.845-854 (1997)). This document is hereinafter referred to as the first document.
[0009]
According to the method described in the first document, the keyword appearance density can be easily calculated from the position on the document of the phrase that matches the set keyword. Therefore, it is not necessary to perform frequency and morphological analysis in advance, and it is only necessary to collate with a set keyword. In addition, since useful parts in the document are known, it is considered that document classification with high accuracy becomes possible if a document having a relatively large number of such useful parts is found.
[0010]
In addition, as a method of using the position of the keyword, a method of calculating and classifying the degree of coincidence of the vocabulary chain in the whole document is also being studied (Mochizuki, Iwayama, Okumura: Passage search based on lexical chain, IPSJ research Meeting report, 98-NL-127, pp.39-46, 1998). This document is hereinafter referred to as the second document.
[0011]
The method described in the second document is a kind of pattern matching in which a part that applies to a query (such as a word that is first input by a search engine) input during keyword search is found in a document. However, since this method only checks the matching with a given query, it looks for a document that is difficult to express with a simple query, such as searching for a document that details the reputation of a product as described above. Not suitable for.
[0012]
Therefore, the present invention obtains a keyword appearance density in consideration of the keyword appearance position in the document, and classifies the document by using the keyword appearance density, so that the document classification can be performed with very simple calculation and high accuracy. The purpose is to.
[0013]
[Means for Solving the Problems]
In order to achieve the above object, a document classification method of the present invention classifies a document of a large number of documents based on a search request from a user, and outputs a classification result. Expanding into a one-dimensional character string, specifying the appearance position of the keyword based on the search request of the user on the one-dimensional character string, and setting a predetermined range as a weighting range on the one-dimensional character string, The keyword appearance density with respect to the central character position of the set weighting range is set using a weighting function that increases the weight for the appearance of the keyword near the center position of the weighting range and reduces the weight as the distance from the center increases. By sequentially calculating while shifting the center character position, a keyword appearance density distribution on the one-dimensional character string is obtained. From keyword occurrence density distribution sought usefulness of the processing target document to the search request of the user, and to perform the classification of a plurality of the target document by its usefulness.
[0014]
The document classification device according to the present invention classifies documents of a large number of documents based on a user's search request, and outputs the classification results for each document to be processed. Expanding to a one-dimensional character string, specifying the appearance position of the keyword based on the user's search request on the one-dimensional character string, and setting a predetermined range on the one-dimensional character string as a weighted range, The keyword appearance density with respect to the central character position of the set weighting range is set by using a weighting function that increases the weight for the appearance of the keyword near the central character position of the weighting range and reduces the weight as the distance from the center increases. The keyword appearance density calculating means for sequentially calculating the central character position while shifting the central character position, and the keyword density calculating means Useful document determination means for obtaining a keyword appearance density distribution on the one-dimensional character string of the processing target document based on the keyword appearance density and obtaining the usefulness of the document with respect to the user search request from the keyword appearance density distribution It is configured to include.
[0015]
In these document classification methods and document classification devices, the weighting function is a Hanning window function or a Gaussian function. When a Gaussian function is used, the weight value is set to zero outside the set weighting range. Used as a setting.
[0016]
Further, the appearance position of the keyword is a character position at the center of the character string constituting the detected keyword, and when the number of characters constituting the keyword is an even number of characters, a value obtained by dividing the number of characters by two is obtained. The character is positioned next or next.
[0017]
Further, the keyword appearance density distribution on the one-dimensional character string is obtained for each processing target document by performing normalization with the maximum value of the keyword appearance density obtained for each processing target document being 1.0. To do.
[0018]
In addition, it is preferable that the weighting range when performing the keyword appearance density calculation is set to approximately 1/10 of the length of each processing target document for each processing target document.
[0019]
In addition, the keyword appearance density may be calculated for all character positions constituting each processing target document for each processing target document, and the keyword appearance density may be calculated for each processing target document. The keyword appearance density is calculated for all character positions constituting the processing target document by calculating the appearance position of the keyword in each processing target document and connecting the calculated keyword appearance density peak values with an envelope. Alternatively, the keyword appearance density distribution may be estimated.
[0020]
Further, the keyword appearance density calculation is performed after text-converting each processing target document for each processing target document, and associating positions of the original document before text conversion and the document after text conversion, The correspondence contents are saved.
[0021]
In addition, the usefulness of the user search request obtained for each processing target document is obtained by obtaining the sum of keyword appearance densities calculated in the processing target document.
[0022]
Furthermore, the usefulness of the user search request obtained for each processing target document may be obtained by calculating the entropy based on the keyword appearance density calculated in the processing target document.
[0023]
When outputting the classified results, if there are multiple documents to be output, the documents are output in order from the highest usefulness required for the classification, and the output contents are the user in each document. At least one of a usefulness level for each search request, a useful portion for a user search request in each document, and a portion representing a summary of each document is displayed.
[0024]
The useful part is extracted as a useful part by setting a section having a particularly high keyword appearance density and extracting the section, and a morphological analysis is performed on a predetermined range including at least the extracted section. To extract as meaningful content.
[0025]
Further, a detailed document content corresponding to the output content is provided by linking to the output content, and when the predetermined part is instructed by the user, the detailed document content can be output, and a keyword appears in the detailed document content. For a portion having a particularly high density, the display method of the portion is displayed differently from the other portions.
[0026]
Further, the keyword is set by the system side selecting an appropriate term based on the search request by the user's search request input, and the synonym or similar term related to the selected term and the selection are also selected. Adjectives, adverbs, and exclamations that are added to the selected terms can also be set as keywords.
[0027]
In the document classification device of the present invention, a text conversion unit that converts the processing target document into a text document can be provided before the keyword appearance density calculation unit. As described above, this text conversion means associates the positions of the original document before text conversion and the document after text conversion when converting the text to be processed, and saves the correspondence contents. Is possible. Further, a search request input means for receiving a user search request and a keyword setting means are provided. This keyword setting means has a knowledge database of which terms are selected as keywords in response to a user search request input to the search request input means, and synonyms and similar terms related to the selected terms, Adjectives, adverbs, and exclamations added to the selected term can also be set as keywords.
[0028]
Further, the document classification apparatus of the present invention can be provided with a document output means for generating a display layout of a document to be output and outputting the document, and the contents output from this document output means are as described above. , At least one of a usefulness level for a user search request in each document, a useful portion for a user search request in each document, and a portion representing a summary of each document, and a predetermined display corresponding to the document that outputs these Set the layout and output.
[0029]
Furthermore, it is possible to provide useful part extraction means for extracting useful parts. As described above, this useful part extracting means sets a section having a particularly high keyword appearance density, extracts the section as a useful part, and extracts at least a predetermined range including the extracted section by morphological analysis. Then, it extracts as meaningful contents using the morphological analysis result.
[0030]
The recording medium on which the document classification processing program of the present invention is recorded is a recording medium on which a document classification processing program for classifying documents of a large number of documents based on a user's search request and outputting the classification result is recorded. In the document classification processing program, a procedure for expanding a plurality of processing target documents into a one-dimensional character string for each processing target document, and an appearance position of a keyword based on the search request of the user are determined in the one-dimensional character string. In addition to specifying above, a predetermined range is set as a weighting range on the one-dimensional character string, and the weight is increased for the appearance of a keyword near the center character position of the weighting range, and the weight is decreased as the distance from the center increases. The keyword appearance density with respect to the central character position of the set weighting range using the weighting function is Obtaining a keyword appearance density distribution on the one-dimensional character string of the document to be processed based on a procedure for sequentially calculating while shifting the position and a keyword appearance density obtained thereby, and the user search from the keyword appearance density distribution And a procedure for determining the usefulness of the document for the request
[0031]
The present invention is an invention on how to efficiently perform document classification as pre-processing when a document desired by a user is extracted from an enormous amount of documents. In order to realize this, first, it is determined at what density the set keyword exists in the document. In the present invention, the density at which the keyword is present in the document is referred to as a keyword appearance density. The keyword appearance density is examined, and the usefulness of the document is determined based on the result.
[0032]
This keyword appearance density is determined by a known method, that is, the above-mentioned first document (Kurohashi, Shiraki, Nagao: identification of important explanation parts of words using the appearance density distribution, Information Processing Society of Japan Journal, Vol. , No. 4, pp. 845-854 (1997)), but the present invention is not intended to obtain the keyword appearance density itself. The main object of the present invention is how to classify documents efficiently and with high accuracy using this keyword appearance density.
[0033]
In addition, as a function used when obtaining the keyword appearance density, a function that gradually decreases from the center to both sides, such as a Hanning window function or a Gaussian function, can be used.
[0034]
Further, in the present invention, the keyword appearance position can be accurately expressed by setting the keyword appearance position to the character at the center of the characters constituting each keyword or the character near the center. In other words, in the above-mentioned first document, the keyword appearance position is set as the head of the keyword. Thus, when the character position at the head of the characters constituting the keyword is set as the keyword position, the keyword is an idiom having a large number of characters. , Problems such as inability to accurately represent the keyword position.
[0035]
Further, for each processing target document, normalization with a maximum keyword appearance density of 1.0 is performed to obtain a keyword appearance density distribution, which makes it easy to compare with other processing target documents.
[0036]
The keyword appearance density distribution obtained by setting the weighting range set when performing the keyword appearance density calculation to about 1/10 of the length of each processing target document for each processing target document is useful. The location is appropriately represented, and the classification process as the subsequent process can be appropriately performed.
[0037]
Further, by calculating the keyword appearance density for all the character positions constituting each processing target document for each processing target document, the keyword appearance density distribution can be obtained with high accuracy.
[0038]
On the other hand, it is also possible to calculate the keyword appearance density only for the appearance position of the keyword in each processing target document. According to this, the keyword appearance density distribution is somewhat approximate, but the calculation amount is small. Less and faster processing is possible. It is also possible to estimate the keyword appearance density distribution when the calculated keyword appearance density is calculated for all the character positions constituting the processing target document by connecting the calculated peak values of the keyword appearance density with an envelope.
[0039]
In addition, by performing keyword appearance density calculation after text conversion of each processing target document for each processing target document, the processing after the keyword appearance density calculation can be made common to any type of document. Can do. In addition, when performing text conversion, if you want to display the useful part by associating the position of the original document before text conversion and the document after text conversion, and storing the associated contents, A natural layout can be displayed without difficulty.
[0040]
Further, whether or not each processing target document is useful can be determined based on the sum of the calculated keyword appearance densities and the value of the sum. Since the usefulness is calculated based on this summation, it is only necessary to obtain the sum of the keyword appearance densities for each document, so that the usefulness can be easily determined with a small amount of calculation.
[0041]
Further, when determining whether or not each processing target document is useful, the entropy of the keyword appearance density obtained in each processing target document can be calculated and determined from the entropy. According to this, even when the determination is delicate only by the sum of the keyword appearance densities, it is possible to determine the usefulness with high accuracy.
[0042]
In addition, the output of the classified results is output in order from the documents with the highest usefulness obtained for classification, and the output contents include useful parts indicating usefulness and which parts are useful, Therefore, the user can immediately grasp the contents of the output document.
[0043]
And when displaying a useful part, as a useful part, the section of a part with a particularly high keyword appearance density is set, the section is extracted as a useful part, and at least a predetermined range including the extracted section is subjected to morphological analysis. Since the morpheme analysis result is used to extract the contents as meaningful, the useful part can be output as a sentence having one set of contents.
[0044]
In addition, more detailed contents can be output for the above-mentioned output contents, and in the detailed document contents, the important part is changed in color or blinking, so the user can see detailed information that he / she wants to know at a glance. Can see.
[0045]
In addition, since the keyword is automatically selected by the user selecting an appropriate term based on the search request by the user's search request input, the user does not need to set the keyword, and the user Since it is only necessary to input whether the user wants to know as a search request, the input operation is simplified. In addition, synonyms and similar words related to terms set as keywords, as well as their adjectives, adverbs, exclamations, etc. can be set as keywords. Information can be extracted appropriately.
[0046]
DETAILED DESCRIPTION OF THE INVENTION
The present invention is a method for determining whether or not a document is related to a certain field from a plurality of documents, and classifying the field as a plurality of keywords (not only words set as keywords but also synonyms and Similar words may also be included, such as adjectives, adverbs, exclamations, etc.) for each search target document, and based on the obtained keyword appearance density The usefulness of each document is obtained, and the documents are classified according to the usefulness.
[0047]
Hereinafter, a method for classifying a document from the keyword position information in the document using the keyword appearance density distribution in the document will be described. The content described below is also an explanation of the content of the document classification processing program of the present invention.
[0048]
First, here is a definition of document classification. A method for determining whether or not contents related to a certain field from a plurality of documents exist in the document is as follows.
[0049]
First, it is considered that there is some knowledge that can be described by a plurality of keywords and expressions about whether or not it is “a field”. For example, assuming content about the reputation of a product, it can be said that content that includes many keywords such as “critic”, “popular”, and “impression” has content related to the field of product reputation.
[0050]
The present invention calculates for each document the density at which such keywords appear in each document, and selects and classifies high-density documents as documents that often contain the contents of the field. It is.
[0051]
Next, a specific document density calculation method will be described. Here, a simple text having no tags or special symbols is considered as a document to be processed. 1A, 1B, and 1C show the concept of keyword appearance density of a certain keyword in a certain three documents (referred to as document A, document B, and document C). In the figure, a cross indicates a keyword existing in the document. The number of keywords when these documents are viewed from the left to the right in the figure is represented by the height of the mountain indicated by the curve R. In the example of FIG. 1, the document A in FIG. 1A shows that a high mountain is formed in a narrow range and the keywords are concentrated in that portion, and the document B in FIG. Several low peaks are formed, indicating that the keywords are evenly distributed throughout the document. Further, the document C in FIG. 5C shows that high peaks are formed in two narrow ranges, and keywords are concentrated in the two ranges.
[0052]
Such keyword appearance density is calculated as follows. First, a document to be processed as shown in FIG. 2A is expanded into a simple one-dimensional character string as shown in FIG. That is, the document shown in FIG. 2 (a) is a document written in horizontal writing, and the horizontal writing document has S at the beginning and E at the end of the document, and expands from the beginning S of the document to the end of the line in the left direction in the figure. After that, the entire processing target document is expanded into a one-dimensional character string by going back to the beginning of the next line and expanding it again to the left from there.
[0053]
Then, the position where the keyword exists is recorded on the document expanded into such a one-dimensional character string. Keywords are prepared in advance for each field or item to be classified (this will be described later). 2A and 2B, it is assumed that a certain keyword k1 exists in a portion indicated by an ellipse F1 and another keyword k2 exists in a portion indicated by a triangle F2. In this way, in this case, there are two types of keywords k1 and k2, but here there is no distinction according to the type of keyword, and only the presence or absence of a predetermined keyword is considered and the keyword is determined. If there is a, record all occurrences.
[0054]
Here, the keyword appearance position refers to the center of the found keyword, and only the center position of the keyword is recorded. For example, a part of the document content converted into a one-dimensional character string as shown in FIG. 2B is the document content as shown in FIG. 3A, and a predetermined keyword k1 is “People's government”. If the “army” and the keyword k2 are “re-registration” (respectively underlined in FIG. 3A), as shown in FIG. The position of the character that becomes the center in the column direction of each character that constitutes, and the position of the character that becomes the center in the row direction of each character that constitutes the "re-registration" that is the keyword k2, respectively. Record as a position.
[0055]
In FIG. 3B, each circle mark (white circle and black circle) corresponds to each character of the document content, and the black circle indicates the character that is the center of the keywords k1 and k2, respectively. In other words, in the “people's government army” that is the keyword k1, “government” is the central character of the keyword, and in “re-registration”, “to” is the central character of the keyword.
[0056]
The keywords k1 and k2 shown in FIG. 3 are five characters for the keyword k1 (people's government army) and three characters for the keyword k2 (re-registration), and both have an odd number of characters. If there is an even number of characters in the keyword, the central character can be determined based on the value obtained by dividing it by 2. For example, if there are 6 characters, based on the value 3 obtained by dividing it by 2, the third character is regarded as the center, or the fourth character that is added to 3 is regarded as the center. This can be addressed by previously decide in advance how to.
[0057]
As described above, in the present invention, the keyword appearance position is the character at the center of the characters constituting each keyword or one character position near the center.
[0058]
Thus, regardless of the number of characters constituting one keyword, it is only necessary to have the position corresponding to one character at or near the center of one keyword as the appearance position information of the keyword. When determining the appearance density, it is possible to determine the exact appearance density that is not affected by the number of characters that make up the keyword, and to significantly reduce the amount of data compared to the case of having keyword position information for all the characters that make up the keyword. And the amount of calculation associated therewith can be greatly reduced.
[0059]
In addition, in the present invention, one character at or near the center of the character string constituting the keyword is used as the keyword appearance position. Therefore, even if the keyword is an idiom having a large number of characters, the appearance position is accurately represented. be able to.
[0060]
When the center position of each keyword is detected and recorded in this manner, the keyword appearance density at a certain position is obtained using a Hanning window function as shown in FIG.
[0061]
Now, a (i) is a position where a keyword (regardless of type) is found in the document content. However, i represents a character position (a character position when the processing target document is expanded into a one-dimensional character string), and if there is a keyword at that position, a (i) = 1, and if there is no keyword, a (i) = 0. And
[0062]
Then, starting from the top of the document (i = 0), each character position is sequentially set as the center position l of the Hanning window, and the keyword appearance density d (l) for the center position l is obtained. When the weighting function hl (i) at the character position i and the weighting range (hanning window width) are W,
[0063]
[Expression 1]

[0064]
It is represented by In FIG. 4 showing the Hanning function, the horizontal axis represents the character position i and the vertical axis represents the weight. In this FIG. 4, the weighting range W is relative to the center position l of the Hanning window. The case where 15 characters are taken on both the left and right sides is shown.
[0065]
In this way, the processing target document is expanded into a one-dimensional character string, and the keyword appearance density is obtained from the keyword appearance position using the above-mentioned first document (Kurohashi, Shiraki, Nagao: appearance density distribution). This is a well-known technique, as described in the identification of important explanation parts of words, Journal of Information Processing Society of Japan, Vol.38, No.4, pp.845-854 (1997)). However, in this first document, the keyword position is the head position of each keyword. However, in the present invention, the keyword position is the center position of each keyword.
[0066]
Although the Hanning function is used here as the weighting function hl (i), various functions can be used as long as the function changes smoothly from the center to the periphery. For example, a Gaussian function is one of them, but here, a Hanning window function that becomes zero in the weighting range W is used. Since the Gaussian function does not become zero even at infinity, it is considered that the Hanning window function is suitable for examining only the influence of the keyword neighborhood. However, it is possible to appropriately change the function depending on the application method. Note that the Gaussian function can be used if it is set to force zero except for the weighting range W.
[0067]
FIG. 5 shows an example in which the document density is calculated by the calculation method described above. This document is a report on a certain product, but if keywords related to the reputation of the product are set appropriately, it can be seen that the density of the part related to the impression of the product and the description of criticism is high.
[0068]
FIG. 5 shows a keyword appearance density distribution obtained by obtaining a keyword appearance density using the above-described equation (1) for a report on a product whose processing target document is a product. . In FIG. 5, the horizontal axis indicates the character position of the document to be processed, and the vertical axis indicates the keyword appearance density d (l) obtained by the above equation (1).
[0069]
Note that the character position on the horizontal axis represents the report document as a one-dimensional character string, and represents the number of bytes from the beginning of the document from the beginning to the end of the document. Further, the keyword appearance density on the vertical axis is the keyword appearance density d (l) obtained by the equation (1), but the normal value in which the maximum value is 1.0 among the keyword appearance densities obtained for the processing target document. It is expressed by making. In this way, by performing normalization for each processing target document, comparison with other processing target documents becomes easy.
[0070]
The processing target document shown in FIG. 5 is specifically a report that summarizes evaluations of a digital camera of a certain manufacturer. Examples of set keywords include “evaluation”, “performance”, and “price”. These keywords are particularly high in terms of the keyword appearance density in the parts touched on the design and the parts touched on the image quality, and in the parts touched on the panorama function and the memory etc. It can be seen that the keyword appearance density is relatively high.
[0071]
By the way, when performing the above-described density calculation, the density output varies depending on how the weighting range W is set. In general, when the weighting range W is wide, the keyword appearance density near the keyword appearance position is obtained. Conversely, when the weighting range W is wide, the influence of keywords existing in a wide range is calculated. . However, if the weighting range W is too wide, the important part becomes unclear, and if it is too narrow, it is not different from the simple keyword frequency, so the method of setting the weighting range is important.
[0072]
Therefore, it is necessary to set an appropriate weighting range W according to the document length of one processing target document for each processing target document. FIG. 6 shows how the keyword appearance density d (l) distribution curve obtained by the above-described equation (1) depends on the weighting range W. In FIG. 6, the distribution curve U1 has a weighting range W of 4096 bytes, the distribution curve U2 has a weighting range W of 2048 bytes, and the distribution curve U3 has a weighting range W of 1024 bytes. It shows a distribution curve of the appearance density d (l). The distribution curve U3 is almost the same as the distribution curve in FIG. That is, FIG. 5 can be said to be a case where the weighting range W is 1024 bytes.
[0073]
As can be seen from FIG. 6, when the weighting range W is too wide, the density change lacks clarity (when the range W is set to 2048 bytes or 4096 bytes), and the important part becomes unclear. Among these three ranges, the case where the weighting range w is 1024 bytes is the most appropriate. In this case, since the document length of the processing target document used here has a document length of 1.2 × 10 4 bytes, it is about 1/10 of the document length of one processing target document. It turns out that it is good to do. In particular, it was found that this tendency is strong in technical documents.
[0074]
Next, a method for classifying documents on the system using the keyword appearance density thus obtained will be described.
[0075]
In the above description, it is assumed that the processing target document is plain text, but there are various types of documents that we actually handle. Therefore, when performing keyword position detection, the detection method differs depending on the document format. In general, the keyword appearance position described above is a position on the layout visually viewed by a human. Since the content of the document file in which the document to be processed is stored contains a lot of information related to the control in addition to the information of the character itself, a position that matches the layout visually seen unless these are appropriately excluded. Hard to find. For example, since there are many tags used for display in an HTML document, it is necessary to skip this tag and calculate the position of characters.
[0076]
That is, in order to implement the present invention, the document is converted into a one-dimensional character string including only non-existing characters such as control symbols. For example, in a document in which various control symbols are included in the document, Therefore, it is necessary to construct a one-dimensional character string only with characters by omitting those control symbols.
[0077]
Therefore, the process of expanding the processing target document into a one-dimensional character string using only characters and performing keyword position detection determines, for each processing target document, what type of document the document is, and the determination result Therefore, it is necessary to set an algorithm for each document format of the processing target document. However, this is actually impractical because of the processing burden.
[0078]
Therefore, here, a method is adopted in which any document is once converted into plain text, and then the keyword appearance density is calculated by detecting the keyword appearance position. If this method is adopted, it is not necessary to develop a keyword position detection algorithm for each document format. In addition, plain text conversion is a function provided in most document creation software, so it can be easily incorporated into the system.
[0079]
When keyword position detection is performed by converting all document formats into plain text, keyword position detection can be performed by simple keyword matching. However, when the keyword position is viewed for each document format without being converted to plain text, a keyword position detection algorithm is required for each document format. Specifically, it is necessary to have knowledge of tags, special symbols, headers, etc., and instructions such as how many characters to skip.
[0080]
Next, assuming that the keyword appearance density distribution as shown in FIG. 5 is obtained for a large number of documents by the method described so far, a process for extracting the most suitable document from each document (document classification process) Will be described.
[0081]
As a document classification method, first, as shown in FIG. 5, a total sum of keyword appearance densities obtained is obtained (integrated), and a document having the largest value is output from among them, or a high value is obtained. A method of outputting several documents in order can be considered.
[0082]
Another method is to calculate and classify entropy. This method is effective when the determination is delicate only by the above-mentioned summation.
[0083]
For example, when considering three documents A to C as shown in FIG. 1, these three documents A to C have the same number of keywords (x marks), and the density is integrated. The values obtained are almost the same. However, the keyword distributions of these documents are greatly different. That is, the detailed information is described in a part of the page of the document A, and there is an article related to the entire page of the document B, but the content seems to be thin. Further, the distribution form of each document is different such that the document C has two pieces of detailed information that is intermediate between the documents A and B.
[0084]
From these points, it is desirable that the document to be extracted is the document A or the document C, and the process for selecting the document A or the document C is performed. It is not easy to comprehensively determine this based on values other than the keyword appearance density (such as the number of keyword hits and the number of locations where the keyword exists). In other words, when trying to make a comprehensive determination with a value other than the keyword appearance density, the conditional branching of the determination becomes complicated, making it difficult to make a delicate determination. So we use entropy. If the entropy to be calculated is E,
[0085]
[Expression 2]

[0086]
It is represented by
[0087]
Entropy E tends to be small when the distribution of keyword appearance density d (l) in each document to be processed is flat and large in a distribution with many sharp parts. Therefore, it is considered useful as a scale for uniquely expressing the overall status of keyword appearance density for each processing target document, and is useful for delicate determination when the sum of keyword appearance densities is similar as shown in FIG. .
[0088]
FIG. 7 is a graph showing the correlation between the result of classifying the contents of 90 magazines (in which the reputation of a certain product is written) as processing target documents into three stages by subjective evaluation and the entropy E of the document density. Here, the three-level classification in the subjective evaluation is, for example, that H3 is a document group that is very well written about the reputation of the product, H2 is a document group that is relatively well written about the reputation of the product, and H1 is Here, the subjective evaluation is divided into three stages H1, H2, and H3, and 90 magazines are classified into the respective stages, such as a document group in which the reputation of the product is written once.
[0089]
After such rough classification, entropy is calculated for each processing target document (each magazine). Since entropy has a larger value as the keyword appearance density distribution with more sharp parts, it can be said that the larger the entropy value, the better the contents are written.
[0090]
The classification of the three stages H1, H2, and H3 as described above is a fairly coarse classification. In each classification, the darker the content, the higher the entropy value. Therefore, the entropy is used as an index. The degree of content can be evaluated quantitatively. In FIG. 7, the entropy obtained for each document is represented by a single rectangle (for example, for the document group at the stage H1, the obtained entropy is E11, E12,... E1n, for the H2 document group, the obtained entropy is represented by E21.E22,..., E2n, and for the H3 document group, the obtained entropy is represented by E31, E32,. However, the entropy obtained in each document group may have a similar value. In this case, since the rectangles indicating the entropy values overlap, not all 90 rectangles are shown in the figure. .
[0091]
In this way, several documents are extracted as document candidates desired by the user. When there are a plurality of extracted documents that are more than a certain level, they are displayed side by side in descending order of usefulness. When such display is performed, it is difficult to understand the contents simply by arranging them in order of high usefulness. Therefore, it is preferable to display other information together.
[0092]
FIG. 8 shows an example of the display. In FIG. 8A, # 1, # 2, # 3,... Indicate the order of high usefulness, and the keyword appearance density, the obtained entropy as the importance z1 of each document with respect to the order, The number of keyword hits is also displayed. Further, a summary of the document (extracting the headline and the beginning of the document) z2, an important part of the document (which will be described later) z3, and the like are also displayed. Further, as shown in FIG. 8 (b), it is possible to display, in color and blink, a portion having a particularly high keyword appearance density in the entire document for each document. That is, marks M1, M2, M3 such as icons are attached to each document, and for example, when the mark M1 is clicked, the entire document # 1 is displayed, and the density of the document is particularly high. The high part z0 is displayed in a color or blink different from the other parts.
[0093]
The main purpose of the present invention is to classify a plurality of documents, but to indicate which of the classified documents is important, it is possible to know the reason why the document was selected or to understand the summary of the classified documents. It is useful to do. Hereinafter, extraction of the useful portion will be described.
[0094]
First, a method for identifying useful parts of a document will be described. Basically, as shown in Fig. 1, a portion with high keyword appearance density (mountain portion) is regarded as a useful portion, and a threshold value is set, and a portion having a density higher than the threshold value is useful. It can be extracted as a part.
[0095]
In the explanation so far, since the document classification is mainly used, the keyword appearance density is calculated for all the character positions in the document. It is also possible to estimate the state of the entire document from the keyword appearance density only at the position where the keyword appears.
[0096]
FIG. 9 shows a comparison example of the keyword appearance density and all the character positions calculated only for the keyword appearance position, and is a case where the same document as the document described in FIG. 5 is used. As can be seen from FIG. 9, the keyword appearance density distribution curve L obtained by connecting the keyword appearance density peaks p1, p2, p3,... This is almost the same as the keyword appearance density distribution curve (see FIG. 5) for the position.
[0097]
As described above, the method of obtaining the keyword appearance density only for the keyword appearance position is effective when the approximate determination is desired quickly even if the accuracy is somewhat low because the calculation time is fast.
[0098]
Next, a method for displaying useful parts will be described. As described above, the criteria for determining whether or not to extract may be simply extracting a keyword appearance density portion equal to or higher than an appropriate threshold value. However, if it is simply extracted as it is, a problem occurs, and it is necessary to deal with it.
[0099]
This is shown in FIG. Assume that an original document (only a part of which is shown in FIG. 10A) exists, and an underline portion is extracted as a useful portion having a keyword appearance density equal to or higher than a certain threshold. Suppose that
[0100]
However, in this case, the content of the extracted section is “I display is post-processed”, and it is impossible to understand what is written. Therefore, a morphological analysis as shown in FIG. 10B is performed, and a boundary at which a break between words is natural is extracted as a useful portion. In other words, in this case, the morphological analysis results are “Unreasonable”, “Ga”, “None”, “Display”, “Do”, “Post-processing”, “As”, “Do”, “Koto”, In such a morphological analysis result, “display”, “to”, “post-processing”, and “as” are extracted, and the contents “display as post-processing” are extracted as shown in FIG. .
[0101]
In this example, if a particle ("i" in the above example) appears alone at the beginning of the useful part, it is ignored, and a part of the ending ("to" in the above example) is present Performs processing that makes sense as a ending, so that useful parts become meaningful sentences. Thus, based on the result of the morphological analysis, the sentence of the part extracted as the useful part can be prevented from becoming unnatural.
[0102]
The above describes the case where the document is text only. However, in the case of a document in which a plurality of elements such as a layout such as an HTML document or an image are combined, only a high-density area is extracted and displayed. It becomes unnatural. In such a case, the extracted useful part can be displayed as a natural content by taking the following method.
[0103]
That is, when converting from a general document to a text document, which character in the text document is recorded in which position in the original document before text conversion. Then, the useful part is determined in the text document state, and the position of the original document corresponding to the character position of the useful part is specified. Then, a break whose layout is not unnatural is determined around the position of the specified original document. For example, it is separated by the lowest block (paragraph or chapter) that includes the area. However, if there is a paragraph in the useful part, it will be unnatural if it is separated by the paragraph part, so it may be separated by a little wider range including the preceding and following paragraphs, or in the useful part If there are different chapters, it will be unnatural if they are separated by changing the chapters.
[0104]
In this way, when text conversion for document density calculation is performed, if the positions of the original document and the text document are associated with each other, it is possible to perform a display that is reasonable in the layout as post-processing. .
[0105]
Next, a system configuration for realizing the present invention will be described. FIG. 11 shows a system configuration diagram for realizing the present invention. The search request input unit 1, knowledge database unit 2, keyword group determination unit 3, search target document database unit 4, document type determination unit 5, text It comprises a conversion unit 6, a keyword appearance density calculation unit 7, a useful document determination unit 8, a display unit 9, a useful part extraction unit 10, a display layout generation unit 11, and the like. The knowledge database unit 2 and the keyword group determination unit 3 constitute a keyword group setting unit 20, and the display unit 9, useful part extraction unit 10, and display layout generation unit 11 constitute a document output unit 30.
[0106]
When the user tries to collect some information, the search request input unit 1 can input the content that the user wants to know. This may be input in a natural language, or in advance, a large number of information may be input. An item may be prepared, and the user may instruct a predetermined item from the item.
[0107]
The knowledge database unit 2 determines what terms (such as words) are appropriate as keywords for the contents based on the contents of the search request input by the user, A plurality of terms are selected, and the plurality of terms selected here are determined as keyword groups by the keyword group determination unit 3. The keywords set here may be several tens or 100 or more depending on the search contents input by the user.
[0108]
By the way, a process for determining a keyword suitable for a user search request is performed as follows.
[0109]
For example, when the user inputs information that he / she wants to know about a product from the search request input unit 1, the knowledge database unit 2 performs “performance”, “price”, “evaluation”, “usability” in response to the user input. The term necessary for expressing the product is selected, and these are determined as keywords. That is, when a useful document is extracted in response to a user search request, it is necessary to classify the document. At this time, depending on the field to be classified, general knowledge about the field or an associated term is used as a keyword. Also, similar terms and synonyms of associated terms can be used. These can be created in advance from personal intuition or cases such as dictionaries, Japanese dictionaries, and the like, and can be stored in the knowledge database unit 2. It is also possible to collect a plurality of documents corresponding to the field to be classified, search for terms that are frequently seen in the collected documents, and use keywords that are frequently used as keywords.
[0110]
In addition, search accuracy can be improved by using an appropriate combination of related terms such as synonyms and similar words for a keyword, or using adjectives, adverbs, and exclamations used together with the keyword. Is further improved. For example, in the former case, if the keyword is “evaluation”, synonyms and similar terms associated with “evaluation” such as “criticism”, “reputation”, “popularity”, etc. Use. In the latter case, if the keyword is “evaluation”, “excellent”, “good quality”, “very”, etc. are used in appropriate combination.
[0111]
The document type determination unit 5 determines in what format the document existing in the search target document database unit 4 is written. The text conversion unit 6 converts the document into a text document based on the determination result by the document type determination unit 5. The search target document database 4 is, for example, a database existing on the server side on the Internet. However, the search target document database 4 is not limited to this, and is a document stored in an information processing device such as a personal computer of a user. Also good.
[0112]
The keyword appearance density calculation unit 7 calculates the keyword appearance density for the text-converted document based on the plurality of keywords determined by the keyword group determination unit 3 by the method described above. Since the keyword appearance density calculation has already been described in detail, the description thereof is omitted here.
[0113]
When the keyword appearance density is calculated for each document by the keyword appearance density calculation unit 4, the useful document determination unit 8 creates, for example, a keyword appearance density distribution as shown in FIG. 5 for each document. The usefulness (usefulness as a document desired by the user) is determined based on the result, and the document to be displayed is determined using the determination result. As described above, the importance level is determined based on the result obtained by integrating a graph as shown in FIG. 5, for example.
[0114]
On the other hand, the useful part extraction unit 10 uses the keyword appearance document density calculated for each document calculated by the keyword appearance density calculation unit 4 to extract the useful part by the method described above. Then, the display layout generation unit 11 generates a display layout as shown in FIG. 8 and displays it on the display unit 9. An example of the display result is shown in FIG.
[0115]
With the above procedure, several documents conforming to the search request of the user are displayed.
[0116]
The present invention is not limited to the embodiment described above, and various modifications can be made without departing from the gist of the present invention. For example, in the above-described embodiment, the display example on the display unit 9 is the display method as shown in FIG. 8, but this display method can be variously set.
[0117]
The processing program for performing the document classification processing of the present invention described above can be recorded on a recording medium such as a floppy disk, an optical disk, or a hard disk, and the present invention includes the recording medium. Further, the processing program may be obtained from a network.
[0118]
【The invention's effect】
According to the present invention, for each processing target document, it is determined at what density the set keyword exists in the processing target document (keyword appearance density), and a user search is performed based on the keyword appearance density. Since the usefulness of a document for a request is determined, document classification as a pre-processing when a user-desired document is extracted from a large number of documents can be accurately and efficiently performed with a small amount of calculation. Is possible.
[0119]
As described above, as an example in which the amount of calculation can be reduced, in the present invention, the keyword appearance position is the character at the center of the characters constituting each keyword or one character near the center. That is, regardless of the number of characters constituting a certain keyword, the data indicating the keyword appearance position only needs to have data at a position corresponding to one character at or near the center of one keyword. The amount of data can be significantly reduced as compared to the case of having data indicating the keyword appearance position for all the characters constituting, and the amount of calculation associated therewith can be greatly reduced.
[0120]
Further, in the present invention, it is possible to obtain a more accurate keyword appearance density distribution by calculating the keyword appearance density for all the character positions constituting the processing target document. It is also possible to calculate the keyword appearance density only for the appearance position of the keyword. According to this, the keyword appearance density distribution is somewhat rough, but the calculation amount is small and high-speed processing is possible.
[0121]
Further, by performing keyword appearance density calculation after text conversion of the processing target document, it is possible to share the processing after the keyword appearance density calculation for any type of document. In addition, when performing text conversion, if you want to display the useful part by associating the position of the original document before text conversion and the document after text conversion, and storing the associated contents, A natural layout can be displayed without difficulty.
[0122]
Further, the determination as to whether or not each processing target document is useful can be made by determining the sum of the calculated keyword appearance densities and determining the sum of the keyword appearance densities. The method for determining the usefulness based on the above method is that it is only necessary to obtain the sum of the keyword appearance densities for each document, so that the usefulness can be easily determined with a small amount of calculation.
[0123]
Further, when determining whether or not each processing target document is useful, the entropy of the keyword appearance density obtained in each processing target document can be calculated and determined from the entropy. According to this, it is possible to determine the usefulness with high accuracy even when the determination is delicate only by the sum of the keyword appearance densities.
[0124]
In addition, the output of the classified results is output in order from the documents with the highest usefulness obtained for classification, and the output contents include useful parts indicating usefulness and which parts are useful, Therefore, the user can immediately grasp the contents of the output document.
[0125]
And when displaying a useful part, as a useful part, the section of a part with a particularly high keyword appearance density is set, the section is extracted as a useful part, and at least a predetermined range including the extracted section is subjected to morphological analysis. Since the morpheme analysis result is used to extract the contents as meaningful, the useful part can be output as a sentence having a single content.
[0126]
In addition, by making it possible to output more detailed contents of the above-mentioned output contents and changing the color or blinking of important parts in the detailed document contents, the user can see the information he / she wants to know at a glance. be able to.
[0127]
Further, since the system side selects an appropriate term as a keyword based on the search request by the user's search request input, it is not necessary for the user himself to set the keyword. Therefore, since the user only has to input what he wants to know as a search request, the input operation performed by the user can be made extremely simple. In addition, synonyms and similar terms related to terms set as keywords, as well as adjectives, adverbs, and exclamations added to keywords can be used as keywords, which further improves search accuracy. The information desired by the user can be appropriately extracted.
[0128]
As described above, according to the present invention, it is possible to classify documents with a small amount of calculation and high accuracy. In addition, it is possible to appropriately display where and how useful it is. In addition, if the present invention is applied, it is not necessary to prepare the created document so that it can be classified by adding keywords or indexes, and there is no need to be aware of the classification processing to be performed on the created document later. In other words, any document can be properly classified as long as keywords are set.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a concept of keyword appearance density in order to describe an embodiment of the present invention.
FIG. 2 is a diagram for explaining an example of obtaining a keyword appearance position by expanding a processing target document into a one-dimensional character string;
FIG. 3 is a diagram illustrating processing for determining the appearance position of each keyword.
FIG. 4 is a diagram illustrating an example of a Hanning window function.
FIG. 5 is a diagram showing a keyword appearance density distribution on a one-dimensional character string of a processing target document.
FIG. 6 is a diagram showing a keyword appearance density distribution obtained by changing the width (weighting range) of the Hanning window when calculating the keyword appearance density.
FIG. 7 is a diagram showing entropy distribution by dividing entropy from keyword appearance density obtained for each document by dividing a plurality of processing target documents into three stages of document groups.
FIGS. 8A and 8B are diagrams showing display examples according to the embodiment of the present invention. FIG. 8A is a diagram showing a display example in which necessary display items are displayed in descending order of usefulness, and FIG. It is a figure which shows the example which displays the designated document content.
FIG. 9 is a diagram illustrating a keyword appearance density distribution on a one-dimensional character string when a keyword appearance density is calculated only from the keyword appearance position.
FIG. 10 is a diagram for explaining a useful part extraction processing method;
FIG. 11 is a diagram showing a system configuration according to the embodiment of the present invention.
[Explanation of symbols]
1 Search request input part
2 Knowledge Database Department
3 keyword group decision part
4 Search target document database section
5 Document type detector
6 Text conversion part
7 Keyword appearance density calculator
8 Useful document judgment part
9 Display section
10 Useful part extractor
11 Display layout generator
20 Keyword setting section
30 Document output section

Claims (25)

In a document classification apparatus constructed using a computer that classifies a document of a large number of document groups based on a user's search request and outputs the classification result,
The keyword appearance density calculation means included in the computer expands the processing target document into a one-dimensional character string, specifies the keyword appearance position based on the search request of the user on the one-dimensional character string, and A predetermined range is set as a weighting range on the character string, and the weighting function is used to increase the weight for the appearance of a keyword near the center position of the weighting range, and to decrease the weight as the distance from the center increases. Sequentially calculating a keyword appearance density with respect to a central character position of a set weighting range while shifting the central character position;
The useful document determination means provided in the computer obtains a keyword appearance density distribution on the one-dimensional character string of the processing target document based on the keyword appearance density obtained by the keyword appearance density calculation means, and this keyword appearance density Obtaining the usefulness of the processing target document with respect to the user search request from the distribution, and
The document classification method characterized in that the usefulness of the user search request obtained for each processing target document is obtained by calculating its entropy based on the keyword appearance density calculated in the processing target document.   2. The document classification according to claim 1, wherein the weighting function is a Hanning window function or a Gaussian function, and when the Gaussian function is used, the weight value is set to zero outside the set weighting range. Method.   The appearance position of the keyword is the character position at the center of the character string constituting the detected keyword. If the number of characters constituting the keyword is an even number of characters, the keyword is located at a value obtained by dividing the number of characters by two. 3. The document classification method according to claim 1, wherein the document is a character or a character positioned next thereto.   The keyword appearance density distribution on the one-dimensional character string is obtained for each processing target document by performing normalization with a maximum value of the keyword appearance density obtained for each processing target document being 1.0. The document classification method according to any one of claims 1 to 3.   5. The weighting range for performing the keyword appearance density calculation is set for each processing target document by using about 1/10 of each processing target document length as a guide. The document classification method according to item 1.   The document classification method according to any one of claims 1 to 5, wherein the keyword appearance density is calculated for all character positions constituting each processing target document for each processing target document.   The keyword appearance density is calculated for each processing target document for the appearance position of the keyword in each processing target document, and the peak value of the calculated keyword appearance density is connected by an envelope, The document classification method according to claim 1, wherein a keyword appearance density distribution is estimated when the keyword appearance density is calculated for all the character positions constituting the document.   The keyword appearance density calculation is performed after text conversion of each processing target document for each processing target document, and the correspondence between the position of the original document before text conversion and the document after text conversion is performed. The document classification method according to claim 1, wherein the attached contents are stored.   When outputting classified results, if there are multiple documents to be output, the documents are output in order from the most useful ones required for classification, and the output contents are searched by users in each document. 9. The display device according to claim 1, wherein at least one of a usefulness level for a request, a useful portion for a user search request in each document, and a portion representing a summary of each document is displayed. Document classification method.   The useful part is extracted as a useful part by setting a section having a particularly high keyword appearance density and extracting the section, and a morphological analysis is performed on a predetermined range including at least the extracted section. The document classification method according to claim 9, wherein the content is extracted as a meaningful content using.   A detailed document content corresponding to the output content is linked to the output content, and a predetermined part is instructed by a user, so that the detailed document content can be output. The document classification method according to claim 9 or 10, wherein a particularly high portion is displayed differently from other portions.   The setting of the keyword is performed by the system side selecting an appropriate term based on the search request by a user's search request input, and the synonym or similar term related to the selected term and the selected term are also selected. 12. The document classification method according to claim 1, wherein adjectives, adverbs, and exclamations added to the terms can be set as keywords. In a document classification device that classifies documents of a large number of documents based on a user's search request and outputs the classification result,
The processing target document is expanded into a one-dimensional character string for each processing target document, and the appearance position of the keyword based on the search request of the user is specified on the one-dimensional character string, and on the one-dimensional character string, The predetermined weighting is set as a weighting range, and the weighting function is set using a weighting function that increases the weight for the appearance of a keyword near the center character position in the weighting range and reduces the weight as the distance from the center increases. A keyword appearance density calculating means for sequentially calculating a keyword appearance density with respect to a central character position of a range while shifting the central character position;
The keyword appearance density distribution on the one-dimensional character string of the processing target document is obtained based on the keyword appearance density obtained by the keyword density calculating means, and the usefulness of the document for the user search request is obtained from the keyword appearance density distribution. Useful document determination means for obtaining the degree,
The document classification apparatus characterized in that the usefulness for the user search request obtained for each processing target document is obtained by calculating the entropy based on the keyword appearance density calculated in the processing target document.   The weighting function is a Hanning window function or a Gaussian function, and when a Gaussian function is used, the weight value is set to zero outside the set weighting range. Document classification device.   The appearance position of the keyword is the character position at the center of the character string constituting the detected keyword. If the number of characters constituting the keyword is an even number of characters, the keyword is located at a value obtained by dividing the number of characters by two. 15. The document classification device according to claim 13, wherein the document classification device is a character or a character positioned next thereto.   The keyword appearance density distribution on the one-dimensional character string is obtained for each processing target document by performing normalization with a maximum value of the keyword appearance density obtained for each processing target document being 1.0. The document classification device according to any one of claims 13 to 15.   17. The weighting range for performing the keyword appearance density calculation is set for each processing target document with about 1/10 of each processing target document length as a guide. The document classification device according to item 1.   18. The document classification device according to claim 13, wherein the keyword appearance density is calculated for all character positions constituting each processing target document for each processing target document. 18.   The keyword appearance density is calculated for each processing target document for the appearance position of the keyword in each processing target document, and the peak value of the calculated keyword appearance density is connected by an envelope, 18. The document classification device according to claim 13, wherein a keyword appearance density distribution is estimated when the keyword appearance density is calculated for all the character positions constituting the document.   A text conversion unit that converts the processing target document into a text document is provided before the keyword appearance density calculation unit, and when the processing target document is converted into text, the original document before text conversion and the document after text conversion The document classification apparatus according to any one of claims 13 to 19, wherein the positions are associated and the contents of the association are stored.   Before the keyword appearance density calculating means, there is a search request input means for receiving a user search request, and a knowledge database of which terms are selected as keywords for the user search request input to the search request input means. And a keyword setting means for setting synonyms and similar words related to the selected term, and adjectives, adverbs, and exclamations added to the selected term as keywords. The document classification device according to any one of claims 13 to 19.   Document output means for generating a display layout of the document to be output and outputting the document is provided, and the output contents are the usefulness for the user search request in each document, and the useful part for the user search request in each document 23. The apparatus according to claim 13, wherein at least one part representing a summary of each document is output by performing a predetermined display layout setting corresponding to the document to which these are output. Document classification device.   The useful part extracting means for extracting the useful part is provided, and the useful part extracting means extracts a useful part by setting a section having a particularly high keyword appearance density and extracting the section, and at least the extracted part is extracted. 23. The document classification apparatus according to claim 22, wherein a predetermined range including the section is subjected to morphological analysis and extracted as meaningful contents using a result of the morphological analysis.   A detailed document content corresponding to the output content is provided by linking to the output content, and the detailed document content can be output when a predetermined part is instructed by the user. The document classification apparatus according to claim 22 or 23, wherein a particularly high portion is displayed differently from the other portions. A computer that categorizes a large number of documents based on a user's search request and outputs the classification results.
A procedure for expanding a plurality of processing target documents into a one-dimensional character string for each processing target document;
The appearance position of the keyword based on the search request of the user is specified on the one-dimensional character string, and a predetermined range is set as a weighting range on the one-dimensional character string, and the keyword near the central character position of the weighting range The keyword appearance density for the central character position in the set weighting range is sequentially calculated while shifting the central character position using a weighting function that increases the weight for the occurrence of, and decreases the weight as the distance from the center increases. procedure,
A procedure for obtaining a keyword appearance density distribution on the one-dimensional character string of the processing target document based on the keyword appearance density obtained thereby, and obtaining the usefulness of the document with respect to the user search request from the keyword appearance density distribution ,
A procedure for calculating and determining the entropy based on the keyword appearance density calculated in the processing target document as the usefulness for the user search request obtained for each processing target document;
The computer-readable recording medium which recorded the program for performing this.

Images