JP4356347B2 - Document extraction system - Google Patents

Description

【００７１】
図８は、文書データ抽出処理の一部を示すフローチャートである。
文書データ抽出処理は、図８に示すように、まず、ステップＳ１００に移行するようになっている。なお、図８のフローチャートに示す処理は、文書データ抽出処理のうち類似度算出部１４として実現される処理である。
ステップＳ１００では、文書データ登録ＤＢ１２に登録された文書データが一つずつ選択され、ステップＳ１０２に移行して、選択された各文書データの文書が文字列ごとに分割され、ステップＳ１０４に移行する。
【００７２】
ステップＳ１０４では、文書と文字列との対応関係を示す文字列−文書行列に文字列の頻度情報が記憶され、ステップＳ１０６に移行して、文書データ登録ＤＢ１２に未処理の文書データが残っているか否かが判定され、未処理の文書データが残っていると判定されたとき(Yes)は、その文書データが選択されて同様な処理が行われ、すべての文書データがなくなるまでこれが繰り返される。
【００７３】
一方、ステップＳ１０６で、文書データ登録ＤＢ１２に未処理の文書データが残っていないと判定されたとき(No)は、ステップＳ１０８に移行して、完成した文字列−文書行列の頻度情報を基にＴＦＩＤＦによって重み付けし直した文字列−文書行列が生成される。これにより、すべての文書データは、それらに出現するユニークな文字列の数と同じ次元（数千〜数十万）の文書ベクトルとして表現できることになる。
【００７４】
このように、すべての文書データがベクトル化されたならば、ベクトル演算類似度算出部３４によって各文書データ相互間の類似度が算出される。具体的には、ベクトル演算類似度算出部３４は、公知のベクトル空間法を採用するものであり、ＴＦＩＤＦによって求められた各文書ベクトルは、ベクトル空間法によって相互の類似度が定義されることになる。すなわち、対比する２つの文書ベクトルの類似度は、図９に示すように、２つのベクトルのなす角θの余弦値（０〜１）として定義することができることから、文書データ同士の類似度は、図１０に示すような対称行列で表現できる。
【００７５】
図９は、文書ベクトルおよびその相関関係を示す図である。
図１０および図１１は、文書−文書間の対称行列を示す図である。
その後、その対称行列をもとに類似情報のグルーピングやカットを行うことで類似文書を除外した文書抽出が実現可能となる。例えば、図１０のような対称行列では、図１１に示すように、文書Ｄ_１と文書Ｄ_２の類似度が「０．９」、文書Ｄ_１と文書Ｄ_３の類似度が「０．３」というように各文書データ相互間の類似度が定量的に示される。
【００７６】
このように、類似度算出部１４によって各文書データ相互間の類似度が定量的に求められたならば、文書データ分類部１６により、類似度算出部１４で算出された類似度に基づいてクラスタリング手法により文書データ登録ＤＢ１２の文書データが階層的に分類される。
クラスタリング手法は、多変量を統計的に分析する一手法である。文書ベクトルＤ_１〜Ｄ_ｎは、多次元空間（ユニークな文字列の数と同じ次元数であり、一般には１万〜十万次元）空間内での一点を指しているとも考えられるので、それぞれの文書ベクトル同士には距離を定義することができる。その距離が、文書Ｄ_１〜Ｄ_ｎを分類したときのクラスタ間距離である。クラスタ間距離の定義方法として、例えば、従来の最短距離法、最長距離法、群平均法、重心法、メジアン法、ウォード法を採用することができるが、本実施の形態では、重心法を用いて文書−文書間の対称行列からクラスタ分析を行う。なお、クラスタ間距離は、クラスタ間の類似度ではなく距離なので、図１２に示すように、０〜１に正規化されている類似度に対して「１−類似度」の関係となる。
【００７７】
図１２は、図１１の文書−文書間の類似度の対称行列をクラスタ間距離に変換したものである。
次に、クラスタ間距離に変換した図１２の場合を例にとって文書Ｄ_１〜Ｄ_４を分類する場合を説明する。
最も類似しているクラスタｐ，ｑを統合して新たなクラスタｔを構成した場合、クラスタｔと他のクラスタｒとのクラスタ間距離Ｓｔｒは、下式（２）により算出することができる。
【００７８】
【数２】

【００７９】
上式（２）において、Ｓｐｒはクラスタｐ，ｒのクラスタ間距離、Ｓｑｒはクラスタｑ，ｒのクラスタ間距離、Ｓｐｑはクラスタｐ，ｑのクラスタ間距離である。また、ｎｐはクラスタｐに含まれる文書数、ｎｑはクラスタｑに含まれる文書数、ｎｒはクラスタｒに含まれる文書数である。
図１３および図１４は、文書Ｄ_１〜Ｄ_４が属するクラスタのクラスタ間距離を算出する場合を説明するための図である。
【００８０】
まず、図１２に示すように、文書Ｄ_１〜Ｄ_４のうち文書Ｄ_１，Ｄ_２が距離「０．１」と最も近いので、その２つを統合して新たなクラスタＤ_１’を構成する。クラスタＤ_１’と他のクラスタとのクラスタ間距離は、ｐ＝Ｄ_１、ｑ＝Ｄ_２、ｔ＝Ｄ_１’として上式（２）により算出される。
クラスタＤ_１’と他のクラスタ（文書Ｄ_３）とのクラスタ間距離Ｓｔ３は、図１３に示すように、上式（２）により「０．８」となる。また、クラスタＤ_１’と他のクラスタ（文書Ｄ_４）とのクラスタ間距離Ｓｔ４は、図１３に示すように、上式（２）により「０．３７５」となる。
【００８１】
次に、図１３に示すように、クラスタＤ_１’とクラスタＤ_４が距離「０．３」と最も近いので、その２つのクラスタを統合して新たなクラスタＤ_１’’を構成する。クラスタＤ_１’’と他のクラスタとのクラスタ間距離は、ｐ＝Ｄ_１’、ｑ＝Ｄ_４、ｔ＝Ｄ_１’’として上式（２）により算出される。
クラスタＤ_１’’と他のクラスタ（文書Ｄ_３）とのクラスタ間距離Ｓｔ３は、図１４に示すように、上式（２）により「０．８５」となる。
【００８２】
図１５は、クラスタ分析によって構成された文書Ｄ_１〜Ｄ_４の階層構造を示す図である。
これにより、文書Ｄ_１〜Ｄ_４は、図１５に示すように、階層的に分類されることになる。
このように、文書データ分類部１６によって文書Ｄ_１〜Ｄ_４が階層的に分類されたならば、文書データ抽出部２０により、文書データ分類部１６で分類された文書データ群のなかからユーザ情報登録ＤＢ１８のユーザ情報に基づいて文書データが抽出される。
【００８３】
図１６は、文書データ分類部１６で分類した文書データ群のなかから文書データを抽出する場合を説明するための図である。
ここで、ユーザＡ，Ｂが設定した冗長度は、ユーザＡが「０．７」、ユーザＢが「０．４」であるとする。つまり、ユーザＡは、ある程度重複は許容してもよいと考えているが、ユーザＢは、あまり重複は許容しないと考えている。
【００８４】
ユーザＢの許容する冗長度は「０．４」なので、ユーザＢに対しては、図１６の階層構造との関係から、文書Ｄ_１，Ｄ_２，Ｄ_４のうちいずれかと、文書Ｄ_３との２つの文書を提供すればよいことが分かる。つまり、文書Ｄ_１，Ｄ_２，Ｄ_４のクラスタと、文書Ｄ_３のクラスタとはそれだけ十分に離れている（内容的に類似してない）ということになる。
【００８５】
具体的に、文書Ｄ_１〜Ｄ_４のなかからユーザＢに提供する文書を選択する方法を図１７を参照しながら詳細に説明する。
図１７は、文書Ｄ_１〜Ｄ_４のなかからユーザＢに提供する文書を選択する場合を説明するための図である。
まず、ユーザＢが設定している冗長度「０．４」に対して、図１７の階層構造との交点を求めると、文書Ｄ_３のクラスタから伸びている線と、文書Ｄ_１，Ｄ_２，Ｄ_４のクラスタから伸びている線の２つに交わることが分かる。この時点で、文書Ｄ_３が確定となり、残りは文書Ｄ_１，Ｄ_２，Ｄ_４から１つ選択するということにする。
【００８６】
既に文書Ｄ_３が確定しているので、類似度算出部１４の結果である図１１の文書−文書間の対称行列から、文書Ｄ_３と、残る文書Ｄ_１，Ｄ_２，Ｄ_４との類似度との関係で判断する。基本的に、文書Ｄ_１，Ｄ_２，Ｄ_４からどれでも１つ選べば冗長度「０．４」以下という大枠の要求はクリアしていることになる。仮に、ユーザＢが冗長度「０．４」以下で、さらにできるだけ冗長度を低くということを希望している属性が設定可能で、それを設定していたとすれば、文書Ｄ_３と最も類似度の低い文書Ｄ_２を選択する。
【００８７】
また、場合によっては、必ずしも冗長度の低いものばかりを選択するとは限らない。ある程度の冗長度を排除しながらも、ある程度の関連性をもった文書を選択することも可能である。図１７の例では、まず、文書Ｄ_３を選択することで冗長度「０．４」以下という条件を満たしているため、そのなかで可能な限りる類似している文書（Ｄ_１，Ｄ_４）を選択すれば、可能な限り冗長性を排除することができるとともに、可能な限り設定した冗長度に近い文書選択が可能になる。図１７の例では、文書Ｄ_３の類似度と、文書Ｄ_１，Ｄ_４の類似度が等しいため、どれか１つに絞るための明確な例とはならないが、実際には、何百〜何十万超の次元の文書ベクトルを扱うので、同値になる可能性はほとんどなく、多くの場合、数値的な大小で上述の２つの方法（冗長度を小さくする選択方法および設定した冗長度に近くする選択方法）で文書選択が可能となる。
【００８８】
一方、ユーザＡの許容する冗長度は「０．７」なので、ユーザＡに対しては、図１６の階層構造との関係から、文書Ｄ_１，Ｄ_２のうちいずれかと、文書Ｄ_３，Ｄ_４との３つの文書を提供すればよいことが分かる。つまり、文書Ｄ_１，Ｄ_２のクラスタと、文書Ｄ_３のクラスタと、文書Ｄ_４のクラスタとはそれだけ十分に離れている（内容的に類似してない）ということになる。文書Ｄ_１，Ｄ_２のクラスタからどれを選択するかについては、例えば、ランダムでもよいし、より重複の少ない組み合わせにするのであれば、図１１の文書−文書間の対称行列から、文書Ｄ_３，Ｄ_４との類似度が小さいものを選択すればよい。
【００８９】
この場合、文書Ｄ_１を選択すると、文書Ｄ_３とは「０．３」、文書Ｄ_４とは「０．５」となり、文書Ｄ_２を選択すると、文書Ｄ_３とは「０．２」、文書Ｄ_４とは「０．８」となるので、単純に類似度の総和で考えるなら文書Ｄ_１を選択する。文書Ｄ_１を選択すると、既に選択されている文書Ｄ_３，Ｄ_４との類似度の総和が小さくなる。また、図１６の例では、結果が変わらない場合もあるが、図１１に示すように、既に選択されている文書Ｄ_４との関係で類似度が最大値となる文書Ｄ_２を避けて文書Ｄ_１を選択するとか、逆に、既に選択されている文書Ｄ_３との関係で類似度が最小値となる文書Ｄ_２を選択するとか、様々に考えられる。
【００９０】
具体的に、文書Ｄ_１〜Ｄ_４のなかからユーザＡに提供する文書を選択する方法を図１８を参照しながら詳細に説明する。
図１８は、文書Ｄ_１〜Ｄ_４のなかからユーザＡに提供する文書を選択する場合を説明するための図である。
まず、ユーザＡが設定している冗長度「０．７」に対して、図１８の階層構造との交点を求めると、文書Ｄ_３のクラスタから伸びている線と、文書Ｄ_４のクラスタから伸びている線と、文書Ｄ_１，Ｄ_２のクラスタから伸びている線の３つに交わることが分かる。この時点で、文書Ｄ_３，Ｄ_４が確定となり、残りは文書Ｄ_１，Ｄ_２から１つ選択するということになる。
【００９１】
文書Ｄ_３，Ｄ_４が確定しているので、文書Ｄ_１，Ｄ_２のうちいずれかおよび文書Ｄ_３，Ｄ_４を含む３つの文書の組み合わせについて冗長度を算出し、取り得るすべての組み合わせのなかから、算出した冗長度が所定値以下となる組み合わせを決定し、決定した組み合わせとなるように残りの１つを選択する。
文書Ｄ_１，Ｄ_２のいずれかおよび文書Ｄ_３，Ｄ_４の組み合わせは、（Ｄ_１，Ｄ_３，Ｄ_４）および（Ｄ_２，Ｄ_３，Ｄ_４）の２通りであるが、冗長度は、各組み合わせにごとに、その組み合わせの文書群のなかから２個の文書を選択した場合のすべての組み合わせについての類似度を総和し、その値を平均したものを冗長度として算出する。
【００９２】
第１に、（Ｄ_１，Ｄ_３，Ｄ_４）の組み合わせについてそのなかから２個の文書を選択する場合は、（Ｄ_１，Ｄ_３）、（Ｄ_１，Ｄ_４）および（Ｄ_３，Ｄ_４）の３通りである。そして、それぞれの類似度は、図１１に示すように、「０．３」、「０．５」および「０．３」であるので、それらの総和は、「１．１」となる。したがって、その平均値は、「０．３７」となり、この値が（Ｄ_１，Ｄ_３，Ｄ_４）の組み合わせについての冗長度として算出される。
【００９３】
第２に、（Ｄ_２，Ｄ_３，Ｄ_４）の組み合わせについてそのなかから２個の文書を選択する場合は、（Ｄ_２，Ｄ_３）、（Ｄ_２，Ｄ_４）および（Ｄ_３，Ｄ_４）の３通りである。そして、それぞれの類似度は、図１１に示すように、「０．２」、「０．８」および「０．３」であるので、それらの総和は、「１．３」となる。したがって、その平均値は、「０．４３」となり、この値が（Ｄ_２，Ｄ_３，Ｄ_４）の組み合わせについての冗長度として算出される。
【００９４】
ここで、ユーザＡが設定した冗長度は、「０．７」であるので、（Ｄ_１，Ｄ_３，Ｄ_４）および（Ｄ_２，Ｄ_３，Ｄ_４）の組み合わせはユーザＡにとって許容範囲内である。したがって、ユーザＡには、文書Ｄ_１〜Ｄ_４のうちそれらの組み合わせのものを提供するのが望ましい。この場合、冗長度が低いものを選択するのであれば、文書Ｄ_１を選択し、設定された冗長度に近いものを選択するのであれば、文書Ｄ_２を選択すればよい。
【００９５】
次に、ユーザが冗長度を設定する場合を説明する。
ユーザが冗長度を設定する場合、設定に慣れたユーザにとっては、自己の要望に添った適切な冗長度を設定することができるが、設定に不慣れなユーザにとっては、自己の設定内容によって抽出文書量がどのように影響を受けるかを把握できていないので、自己の要望に添った適切な冗長度を設定することが困難である。そこで、本実施の形態では、ユーザの要求に応じて支援案内を表示することによりユーザの設定を支援する。
【００９６】
図１９は、冗長度の設定画面を示す図である。
ユーザが冗長度の設定を要求すると、図１９に示すような設定画面が表示される。図１９の設定画面には、冗長度を入力するテキストボックス３００と、支援案内の表示を要求するボタン３０２と、冗長度の登録を要求するボタン３０４とが配置されている。ユーザは、まず、冗長度の設定に先立ってボタン３０２をマウス等でクリックすると、設定支援案内表示部２８により、設定支援情報登録ＤＢ２６の設定支援情報に基づいて、冗長度設定部２２の設定可能範囲内で設定値を変化させた場合にその設定値の変化に対する抽出文書量の変化が表示される。設定支援情報登録ＤＢ２６には、設定支援情報が登録されているが、これは、設定支援情報生成部２４により、他のユーザが設定した冗長度と、その冗長度に基づいて文書データ抽出部２０で抽出された抽出文書量との関係が設定支援情報として設定支援情報登録ＤＢ２６に登録される。
【００９７】
ユーザは、この表示を参考にすれば、自己の設定内容によって抽出文書量がどのように影響を受けるかを把握することができるので、設定に不慣れなユーザであっても、自己の要望に添った適切な冗長度を設定することができる。
図２０は、文書データ分類部１６で文書データを階層的に分類した分類結果の一例を示す図である。
【００９８】
例えば、図２０の分類結果においては、文書Ｄ１と文書Ｄ２でクラスタＤ１２を構成し、文書Ｄ３と文書Ｄ４でクラスタＤ３４を構成し、クラスタＤ１２とクラスタＤ３４でクラスタＤ１４を構成し、クラスタＤ１４と文書Ｄ５でクラスタＤ１５を構成している。また、文書Ｄ７と文書Ｄ８でクラスタＤ７８を構成し、クラスタＤ７８と文書Ｄ６でクラスタＤ６８を構成している。さらに、クラスタＤ１５とクラスタＤ６８でクラスタＤ１８を構成している。
【００９９】
図２１は、設定値と文書数との関係をスライドバー形式により表示した図である。
図２０の場合に、ユーザは、図１９の設定画面においてボタン３０２をクリックすると、図２１に示すように、冗長度設定部２２の設定可能範囲内で設定値を変化させた場合にその設定値の変化に対する抽出文書量の変化がスライドバー形式により表示される。図２１の例では、冗長度を「０」、「０．１」、「０．２」、「０．３」、「０．４」、「０．５」、「０．６」、「０．７」、「０．８」、「０．９」および「１．０」に設定した場合は、０個、０個、２個、２個、２個、４個、４個、５個、６個、８個および８個の文書がそれぞれ抽出されることを示している。
【０１００】
ユーザは、スライドバーを操作しながら所望の冗長度にスライドさせ、ボタン３０６をクリックすると、図１９の設定画面においてスライドバーによる設定値がテキストボックス３００に入力される。そして、ボタン３０４をクリックすることにより冗長度を設定する。
図２２は、設定値と文書数との関係をグラフ形式により表示した図である。
【０１０１】
また、図２１に限らず、図２２に示すように、冗長度設定部２２の設定可能範囲内で設定値を変化させた場合にその設定値の変化に対する抽出文書量の変化をグラフにより表示することも可能である。図２２の場合の方が設定値と文書量との関係をより視覚的に把握することができる。
同様に、ユーザは、スライドバーを操作しながら所望の冗長度にスライドさせ、ボタン３０６をクリックすると、図１９の設定画面においてスライドバーによる設定値がテキストボックス３００に入力される。そして、ボタン３０４をクリックすることにより冗長度を設定する。
【０１０２】
このようにして、本実施の形態では、複数の文書データを登録した文書データ登録ＤＢ１２と、冗長度を設定する冗長度設定部２２と、抽出される文書データの冗長度が冗長度設定部２２で設定した冗長度以下となるように文書データ登録ＤＢ１２のなかから文書データを抽出する文書データ抽出部２０と、冗長度設定部２２で設定されるであろう設定値と文書データ抽出部２０で抽出される抽出文書量との関係を表示する設定支援案内表示部２８とを備える。
【０１０３】
これにより、ユーザは、自己の設定内容によって抽出文書量がどのように影響を受けるかを把握しやすくなるので、自己の要望に添った比較的適切な冗長度を設定することができる。また、類似度が大きい文書同士は選択されにくくなるので、内容が類似するような文書を重複して抽出する可能性を低減することができる。したがって、従来に比して、内容の重複を排除することについてユーザの希望に比較的適合した文書抽出を行うことができる。
【０１０４】
さらに、本実施の形態では、設定支援案内表示部２８は、冗長度設定部２２の設定可能範囲内で設定値を変化させた場合にその設定値の変化に対する抽出文書量の変化を表示するようになっている。
これにより、ユーザは、自己の設定内容によって抽出文書量がどのように影響を受けるかをさらに把握しやすくなる。したがって、内容の重複を排除することについてユーザの希望にさらに適合した文書抽出を行うことができる。
【０１０５】
さらに、本実施の形態では、設定値と抽出文書量との関係を示す設定支援情報を登録する設定支援情報登録ＤＢ２６を備え、設定支援案内表示部２８は、設定支援情報登録ＤＢ２６の設定支援情報に基づいて設定値と抽出文書量との関係を表示するようになっている。
これにより、設定値に比較的適合した抽出文書量が表示されるので、ユーザは、自己の設定内容によって抽出文書量がどのように影響を受けるかをさらに把握しやすくなる。したがって、内容の重複を排除することについてユーザの希望にさらに適合した文書抽出を行うことができる。
【０１０６】
さらに、本実施の形態では、文書データ抽出部２０の抽出結果に基づいて設定値と抽出文書量との関係を示す設定支援情報を生成する設定支援情報生成部２４を備え、設定支援案内表示部２８は、設定支援情報生成部２４で生成した設定支援情報に基づいて設定値と抽出文書量との関係を表示するようになっている。
これにより、設定値に比較的適合した抽出文書量が表示されるので、ユーザは、自己の設定内容によって抽出文書量がどのように影響を受けるかをさらに把握しやすくなる。したがって、内容の重複を排除することについてユーザの希望にさらに適合した文書抽出を行うことができる。
【０１０７】
さらに、本実施の形態では、複数の文書データを登録した文書データ登録ＤＢ１２と、文書データ登録ＤＢ１２の文書データについてそれら文書データ相互間の類似度を算出する類似度算出部１４と、類似度算出部１４で算出した類似度に基づいて文書データ登録ＤＢ１２の文書データを階層的に分類する文書データ分類部１６と、文書データ分類部１６で分類した文書データ群のなかから所定の分類規則に基づいて文書データを抽出する文書データ抽出部２０とを備える。
【０１０８】
これにより、類似度が大きい文書同士はさらに選択されにくくなるので、内容が類似するような文書を重複して抽出する可能性をさらに低減することができる。したがって、内容の重複を排除することについてユーザの希望にさらに適合した文書抽出を行うことができる。また、文書の抽出に際しては各文書にキーワードを付与する等の作業を要しないので、従来に比して、その作業に要するコストを低減することができる。
【０１０９】
さらに、本実施の形態では、文書データ分類部１６は、類似度算出部１４で算出した類似度に基づいて、文書データを分類したときのクラスタ間距離を算出し、算出したクラスタ間距離に基づいて、類似度が大きい文書データ同士ほど近い階層となるように、文書データ登録ＤＢ１２の文書データを階層的に分類するようになっており、文書データ抽出部２０は、文書データ分類部１６で分類した階層構造のうち階層下のクラスタ間距離に相当する冗長度が所定値以下となる点を特定し、特定した点の階層下の文書データを文書データ群のなかから抽出するようになっている。
【０１１０】
これにより、各文書データをその類似度に応じて比較的適切に分類・抽出することができるので、内容が類似するような文書を重複して抽出する可能性をさらに低減することができる。したがって、内容の重複を排除することについてユーザの希望にさらに適合した文書抽出を行うことができる。
さらに、本実施の形態では、文書データ分類部１６は、クラスタリング手法またはグルーピング手法により文書データ登録ＤＢ１２の文書データを階層的に分類するようになっている。
【０１１１】
これにより、各文書データをその類似度に応じて比較的適切に分類することができるので、内容が類似するような文書を重複して抽出する可能性をさらに低減することができる。したがって、内容の重複を排除することについてユーザの希望にさらに適合した文書抽出を行うことができる。
さらに、本実施の形態では、類似度算出部１４は、文書データ登録ＤＢ１２の各文書データの文書を所定の文字列に分割する文書分割部３０と、文書分割部３０で分割した文字列の出現頻度に基づいて文書データ登録ＤＢ１２の各文書データについて文書ベクトルを生成する文書ベクトル生成部３２と、文書ベクトル生成部３２で生成した文書ベクトルに基づいて文書データ登録ＤＢ１２の文書データ相互間の類似度を算出するベクトル演算類似度算出部３４とを有する。
【０１１２】
これにより、各文書データ相互間の類似度を比較的適切に算出することができるので、内容が類似するような文書を重複して抽出する可能性をさらに低減することができる。したがって、内容の重複を排除することについてユーザの希望にさらに適合した文書抽出を行うことができる。
さらに、本実施の形態では、文書分割部３０は、形態素解析、ｎ−ｇｒａｍおよびストップワードのいずれかの文字列分割方式を用いて文書データ登録ＤＢ１２の各文書データの文書を所定の文字列に分割するようになっている。
【０１１３】
形態素解析、ｎ−ｇｒａｍおよびストップワードといった文字列分割方式は、従来から多用されている信頼性に優れた方式であり、これらを本発明の文書分割部３０として用いることにより、各文書を比較的適切に文字列に分割できることは勿論、これらいずれかの方式を用いることにより様々な形態の文書にも対応することができる。
【０１１４】
さらに、本実施の形態では、文書ベクトル生成部３２は、文字列が出現する文書におけるその文字列の出現頻度、および複数の文書における文字列の出現頻度に基づいて、文書データ登録ＤＢ１２の各文書データについてＴＦＩＤＦで重み付けした文書ベクトルを生成するようになっている。
各文書の文書ベクトルを生成するに際し、分割された文字列の出現頻度をそのまま用いてもよいが、ＴＦＩＤＦという文字列の重要度を反映した公知の重み付け方法を用いると各文書の特徴を反映した文書ベクトルを生成することができる。そのため、各文書データ相互間の類似度をさらに適切に算出することができるので、内容が類似するような文書を重複して抽出する可能性をさらに低減することができる。したがって、内容の重複を排除することについてユーザの希望にさらに適合した文書抽出を行うことができる。
【０１１５】
さらに、本実施の形態では、ベクトル演算類似度算出部３４は、文書ベクトル生成部３２で生成した文書ベクトルに基づいてベクトル空間法により文書データ相互間の類似度を算出するようになっている。
各文書データ相互間の類似度を算出する方式としてベクトル空間法を用いれば、２つの文書ベクトルの類似度は２つのベクトルのなす角の余弦値（０〜１）として定量的に表現することが可能となり、文書抽出をより的確に行うことが可能となる。
【０１１６】
上記実施の形態において、文書データ登録ＤＢ１２は、発明１、２、８ないし１０の文書データ記憶手段に対応し、類似度算出部１４は、発明１、２または８の類似度算出手段に対応し、文書データ分類部１６は、発明１または２の文書データ分類手段に対応している。また、文書データ抽出部２０は、発明１、２または５の文書データ抽出手段に対応している。
【０１１７】
また、上記実施の形態において、冗長度設定部２２は、発明１、２または３の冗長度設定手段に対応し、設定支援情報生成部２４は、発明５の設定支援情報生成手段に対応している。また、設定支援情報登録ＤＢ２６は、発明４の設定支援情報記憶手段に対応し、設定支援案内表示部２８は、発明１ないし５の設定支援案内出力手段に対応している。
【０１１８】
また、上記実施の形態において、文書分割部３０は、発明８または９の文書分割手段に対応し、文書ベクトル生成部３２は、発明８、１０または１１の文書ベクトル生成手段に対応し、ベクトル演算類似度算出部３４は、発明８または１１のベクトル演算類似度算出手段に対応している。
なお、上記実施の形態においては、文書の重複度合いについてのみユーザの好みを反映させるように構成したが、これに限らず、文書の内容についてもユーザの好みを反映させるように構成することもできる。具体的には、例えば、次の２つの構成を提案することができる。
【０１１９】
図２３は、本発明に係る文書抽出装置１０の他の実施の形態を示したものである。
文書抽出装置１０は、図２３に示すように、文書データ登録ＤＢ１２と、ユーザ情報登録ＤＢ１８と、ユーザ情報登録ＤＢ１８のユーザ情報に基づいて文書データ登録ＤＢ１２のなかから文書データを選択する文書データ選択部７０と、文書データ選択部７０で選択した文書データについてそれら文書データ相互間の類似度を算出する類似度算出部１４と、類似度算出部１４で算出した類似度に基づいて文書データ選択部７０で選択した文書データを階層的に分類する文書データ分類部１６と、ユーザ情報を登録したユーザ情報登録ＤＢ１８と、文書データ分類部１６で分類した文書データ群のなかからユーザ情報登録ＤＢ１８のユーザ情報に基づいて文書データを抽出する文書データ抽出部２０とで構成されている。
【０１２０】
これにより、ユーザの希望に適合した内容の文書データが選択されたのちに、そのなかから類似度が大きい文書データが除外されるので、ユーザの好みによる偏りを大きくし、情報の均一性よりもユーザの好みを重視した文書抽出を行うことができる。例えば、幅広い情報よりも自己の好みに適合した情報を求めているユーザに対しては、好適な情報提供を行うことができる。
【０１２１】
この場合において、文書データ登録ＤＢ１２は、発明１１の文書データ記憶手段に対応し、ユーザ情報登録ＤＢ１８は、発明６のユーザ情報記憶手段に対応し、類似度算出部１４は、発明６の類似度算出手段に対応し、文書データ分類部１６は、発明６の文書データ分類手段に対応している。また、文書データ選択部７０は、発明６の文書データ選択手段に対応している。
【０１２２】
図２４は、本発明に係る文書抽出装置１０の他の実施の形態を示したものである。
文書抽出装置１０は、図２４に示すように、文書データ登録ＤＢ１２と、ユーザ情報登録ＤＢ１８と、文書データ登録ＤＢ１２の文書データについてそれら文書データ相互間の類似度を算出する類似度算出部１４と、類似度算出部１４で算出した類似度に基づいて文書データ登録ＤＢ１２の文書データを階層的に分類する文書データ分類部１６と、ユーザ情報を登録したユーザ情報登録ＤＢ１８と、文書データ分類部１６で分類した文書データ群のなかからユーザ情報登録ＤＢ１８のユーザ情報に基づいて文書データを抽出する文書データ抽出部２０と、文書データ抽出部２０で抽出した文書データ群のなかからユーザ情報登録ＤＢ１８のユーザ情報に基づいて文書データを選択する文書データ選択部７０とで構成されている。
【０１２３】
これにより、類似度が大きい文書データが除外されたのちに、そのなかからユーザの希望に適合した内容の文書データが選択されるので、ユーザの好みによる偏りを小さくし、ユーザの好みよりも情報の均一性を重視した文書抽出を行うことができる。例えば、自己の好みに適合した情報よりも幅広い情報を求めているユーザに対しては、好適な情報提供を行うことができる。
【０１２４】
この場合において、ユーザ情報登録ＤＢ１８は、発明１２のユーザ情報記憶手段に対応し、文書データ抽出部２０は、発明１２の文書データ抽出手段に対応し、文書データ選択部７０は、発明１２の文書データ選択手段に対応している。
図２３の構成を構成Ａとし、図２４の構成を構成Ｂとして、構成Ａ，Ｂの違いについて詳細に説明する。
【０１２５】
ユーザＡの文書データ選択のための属性として、ニュースカテゴリが「大リーグ」関係、それが少なければ上位の概念である「スポーツ」に選択範囲を広げる。そして、文書データ選択部７０で選択されるニュースは５個と指定されているものとする。「大リーグ」や「スポーツ」は、キーワードが直接指定されていることをも含めて、キーワード群や、関連文章や、その他何らかの公知技術を使って情報カテゴリが指定されているものとする。冗長性排除の属性は、別途同様に設定されているものとするが、簡略化のため、以下の例で「重複」の欄に記事番号が書かれているものが、その記事と重複しており冗長性排除によって排除される候補であることを示している。
【０１２６】
図２５および図２６は、図２３の構成と図２４の構成との違いを説明するための図である。
図２５および図２６において、「適合度」とは、ユーザＡが設定した情報カテゴリに、各記事がどれぐらい適合するかを文書データ選択部７０によって計算した結果得られる数値である。「重複」の欄には、別途指定されている冗長性排除のための属性に従って計算した結果、その欄に書かれてある記事と重複しているため、排除される候補になることを示している。図２５および図２６の例では、記事１は記事３と重複しており、記事１の方が排除候補になるということを示している。
【０１２７】
この例では、構成Ａ，Ｂの違いが、記事１０があるかないかに現れている。これは、図２５の例（構成Ａ）では、よりユーザの嗜好を重視しよりその嗜好に偏った記事選択が行われるのに対して、図２６の例（構成Ｂ）では、ユーザの嗜好を重視しながらも、それを逸脱しない範囲で冗長性が排除された記事選択が行われているといえる。
【０１２８】
この結果からは、通常は、冗長性を排除しながらも、情報量が指定した５つになる図２６の例の方が好ましい構成と考えられる。図２５の例は、絶対記事数が５以下になる可能性が大きい（数が保証される可能性が小さい）のに対して、図２６の例は数が保証される可能性が大きい。ただし、適合度の判定、重複の判定に左右される可能性が大いにあり、一概にどちらがよいとはいえない。
【０１２９】
また、上記実施の形態において、文書Ｄ_１，Ｄ_２のうちいずれかおよび文書Ｄ_３，Ｄ_４を含む３つの文書の組み合わせについて冗長度を算出にあたって相加平均を用いているが、これに限らず、積をとって平方根等を求めその値を冗長度とする方法、各文書データ相互間の類似度のうち最大値または最小値を冗長度とする方法を採用することもできる。
【０１３０】
長方形の面積で、「縦＋横」の長さが一定ならば正方形に近い方が面積が大きくなる原理と同じく、相乗平均は、相加平均と比較して以下のような特徴が挙げられる。
（ａ）文書データ相互間の類似度のばらつきが大きい組み合わせは、算出結果として小さくなる傾向がある。
（ｂ）文書データ相互間の類似度のばらつきが小さい組み合わせは、算出結果として大きくなる（小さくならない）傾向がある。
例えば、上記実施の形態において３個の文書を選択する組み合わせについて、相加平均を用いて冗長度を算出した場合は、次のようになる。ここで、（１）は（Ｄ_１，Ｄ_３，Ｄ_４）についての計算結果であり、（２）は（Ｄ_２，Ｄ_３，Ｄ_４）についての計算結果である。
（１）(0.3+0.5+0.3)/3 = 0.37
（２）(0.2+0.8+0.3)/3 = 0.43
また、上記実施の形態において３個の文書を選択する組み合わせについて、相乗平均を用いて冗長度を算出した場合は、次のようになる。ここで、（１）は（Ｄ_１，Ｄ_３，Ｄ_４）についての計算結果であり、（２）は（Ｄ_２，Ｄ_３，Ｄ_４）についての計算結果である。
（１）3√(0.3*0.5*0.3) = 0.356
（２）3√(0.2*0.8*0.3) = 0.363
相加平均の（１）（２）と、相乗平均の（１）（２）の結果を比較すると（２）の結果がほぼ同じなのに対して、相乗平均の（１）の結果は、相加平均の（１）に比べて小さくなっている。これは、（２）が文書データ相互間の類似度にばらつきのある組み合わせであるためであって、相加平均では、１つの高い数値に平均が押し上げられる場合であっても、相乗平均の方法を用いれば、部分的に類似している程度の冗長性であれば影響を受けない結果となる。したがって、相加平均の場合は、一部にでも文書データ相互間の類似度の大きい組み合わせが混じっているとリニアに影響を受ける文書群の冗長性算出方法といえる。この方法を用いれば、大部分の文書データ相互間の類似度が大きい場合は、文書群の冗長性が大きいと判定するが、一部の文書データ相互間の類似度が大きいだけでは（つまり、類似していない文書も多数含まれている文書群の場合は）、冗長度が低いと判断する特徴を持つことになる。相加平均および相乗平均の場合はいずれも、冗長度が低いものを選択するのであれば、文書Ｄ_１を選択し、設定された冗長度に近いものを選択するのであれば、文書Ｄ_２を選択すればよい。
【０１３１】
また、上記実施の形態において３個の文書を選択する組み合わせについて、各文書データ相互間の類似度のうち最小値を用いて冗長度を算出した場合は、次のようになる。ここで、（１）は（Ｄ_１，Ｄ_３，Ｄ_４）についての計算結果であり、（２）は（Ｄ_２，Ｄ_３，Ｄ_４）についての計算結果である。
（１）min(0.3,0.5,0.3) = 0.3
（２）min(0.2,0.8,0.3) = 0.2
文書データ相互間の類似度の組み合わせのなかから、最小値をその文書群の冗長度とする方法である。ある文書群のなかに少なくとも冗長度の小さい組み合わせが存在する場合に小さくなる。言い換えれば、全部類似しているわけではなく、少なくとも何らかの特徴的な文書が含まれている場合に小さくなる指標といえる。これは、文書群の冗長性を全体的に表現する指標としては最適でないかもしれない。しかし、例えば、通常は、冗長度が大きいものばかりで、時々文書群のなかに混じってくる特徴的な文書を見逃さないようにするような局所的な指標が必要とされる場合に有効である。この場合、冗長度が低いものを選択するのであれば、文書Ｄ_２を選択し、設定された冗長度に近いものを選択するのであれば、文書Ｄ_１を選択すればよい。
【０１３２】
また、上記実施の形態において３個の文書を選択する組み合わせについて、各文書データ相互間の類似度のうち最大値を用いて冗長度を算出した場合は、次のようになる。ここで、（１）は（Ｄ_１，Ｄ_３，Ｄ_４）についての計算結果であり、（２）は（Ｄ_２，Ｄ_３，Ｄ_４）についての計算結果である。
（１）max(0.3,0.5,0.3) = 0.5
（２）max(0.2,0.8,0.3) = 0.8
文書データ相互間の類似度の組み合わせのなかから、最大値をその文書群の冗長度とする方法である。ある文書群のなかに少なくとも冗長度の大きい組み合わせが存在する場合に大きくなる。言い換えれば、少しでも類似している文書がある場合に大きくなる指標といえる。これは、文書群の冗長性を全体的に表現する指標としては最適でないかもしれない。しかし、例えば、少しでも冗長な文書の組み合わせは選択したくないときや、文書の冗長性はある閾値以下に完全に排除したいときなど、局所的な指標が必要とされる場合に有効である。この場合、冗長度が低いものを選択するのであれば、文書Ｄ_１を選択し、設定された冗長度に近いものを選択するのであれば、文書Ｄ_２を選択すればよい。
【０１３３】
また、上記実施の形態においては、文書ベクトル同士の距離を利用するように構成したが、これに限らず、文書ベクトル同士の角度（余弦値）を利用するように構成することもできる。
また、上記実施の形態において、図８のフローチャートに示す処理を実行するにあたっては、ＲＯＭ５２にあらかじめ格納されている制御プログラムを実行する場合について説明したが、これに限らず、これらの手順を示したプログラムが記憶された記憶媒体から、そのプログラムをＲＡＭ５４に読み込んで実行するようにしてもよい。
【０１３４】
ここで、記憶媒体とは、ＲＡＭ、ＲＯＭ等の半導体記憶媒体、ＦＤ、ＨＤ等の磁気記憶型記憶媒体、ＣＤ、ＣＤＶ、ＬＤ、ＤＶＤ等の光学的読取方式記憶媒体、ＭＯ等の磁気記憶型／光学的読取方式記憶媒体であって、電子的、磁気的、光学的等の読み取り方法のいかんにかかわらず、コンピュータで読み取り可能な記憶媒体であれば、あらゆる記憶媒体を含むものである。
【０１３５】
また、上記実施の形態においては、本発明に係る情報抽出システム、文書抽出システム、情報抽出プログラムおよび文書抽出プログラム、並びに情報抽出方法および文書抽出方法を、図１に示すように、内容の重複が少ない複数の文書をユーザに提供する場合について適用したが、これに限らず、本発明の主旨を逸脱しない範囲で他の場合にも適用可能である。
【図面の簡単な説明】
【図１】 本発明に係る文書抽出装置１０の実施の一形態を示したものである。
【図２】 コンピュータ１００の構成を示すブロック図である。
【図３】 形態素解析による文字列分割の一例を示す図である。
【図４】 ｎ−ｇｒａｍによる文字列分割の一例を示す図である。
【図５】 ストップワードによる文字列分割の一例を示す図である。
【図６】 文字列−文書行列を示す図である。
【図７】 形態素解析による文字列分割結果を示す図である。
【図８】 文書データ抽出処理の一部を示すフローチャートである。
【図９】 文書ベクトルおよびその相関関係を示す図である。
【図１０】 文書−文書間の対称行列を示す図である。
【図１１】 文書−文書間の対称行列を示す図である。
【図１２】 図１１の文書−文書間の類似度の対称行列をクラスタ間距離に変換したものである。
【図１３】 文書Ｄ_１〜Ｄ_４が属するクラスタのクラスタ間距離を算出する場合を説明するための図である。
【図１４】 文書Ｄ_１〜Ｄ_４が属するクラスタのクラスタ間距離を算出する場合を説明するための図である。
【図１５】 クラスタ分析によって構成された文書Ｄ_１〜Ｄ_４の階層構造を示す図である。
【図１６】 文書データ分類部１６で分類した文書データ群のなかから文書データを抽出する場合を説明するための図である。
【図１７】 文書Ｄ_１〜Ｄ_４のなかからユーザＢに提供する文書を選択する場合を説明するための図である。
【図１８】 文書Ｄ_１〜Ｄ_４のなかからユーザＡに提供する文書を選択する場合を説明するための図である。
【図１９】 冗長度の設定画面を示す図である。
【図２０】 文書データ分類部１６で文書データを階層的に分類した分類結果の一例を示す図である。
【図２１】 設定値と文書数との関係をスライドバー形式により表示した図である。
【図２２】 設定値と文書数との関係をグラフ形式により表示した図である。
【図２３】 本発明に係る文書抽出装置１０の他の実施の形態を示したものである。
【図２４】 本発明に係る文書抽出装置１０の他の実施の形態を示したものである。
【図２５】 図２３の構成と図２４の構成との違いを説明するための図である。
【図２６】 図２３の構成と図２４の構成との違いを説明するための図である。
【符号の説明】
１０…文書抽出装置，１２…文書データ登録ＤＢ，１４…類似度算出部，１６…文書データ分類部，１８…ユーザ情報登録ＤＢ，２０…文書データ抽出部，２２…冗長度設定部，２４…設定支援情報生成部，２６…設定支援情報登録ＤＢ，２８…設定支援案内表示部，３０…文書分割部，３２…文書ベクトル生成部，３４…ベクトル演算類似度算出部，５０…ＣＰＵ，５２…ＲＯＭ，５４…ＲＡＭ，５８…Ｉ／Ｆ，６０…入力装置，６４…表示装置，７０…文書データ選択部，１００…コンピュータ，Ｄ_１〜Ｄ_ｍ…文書，Ｓ…情報供給源，３００…テキストボックス，３０２〜３０６…ボタン[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to a document distribution system that automatically distributes a document such as news according to a user's preference, and in particular, can perform information extraction that meets a user's desire for eliminating duplication of contents. Suitable for reducing work costsDocument extraction systemAbout.
[0002]
[Prior art]
In the information distribution system that can be customized for each user, the user automatically sets filtering conditions, and the computer automatically selects from various information such as news (hereinafter referred to as text information) that is sent in real time. In general, only a document that meets the set filtering condition is extracted and distributed to the user.
[0003]
In the case of such a form of document distribution system, depending on the filtering conditions, there are problems that the distributed documents are too biased, and documents with similar contents are repeatedly sent. In particular, with regard to the latter problem, due to duplication of document contents, information that is distributed becomes wasteful or other important documents are cut when the document placement space is limited. Inconveniences such as the above, and the convenience and reliability of the document distribution system itself are greatly impaired.
[0004]
For this reason, a filtering or classification technique for efficiently extracting only necessary documents to prevent such a duplicate delivery of documents is extremely important. As conventional techniques related to these, for example, techniques shown in the following Patent Documents 1 to 3 have been proposed.
In Patent Document 1 (Japanese Patent Laid-Open No. 10-275160), a keyword is assigned to all documents, the documents are vectorized from the keywords, and a maximum value is obtained when a certain document A is included in another document B. In other words, a technique for recognizing representative documents, subordinate documents, independent documents, etc. and collecting related documents as appropriate is introduced.
[0005]
Patent Document 2 (Japanese Patent Laid-Open No. 9-101990) discloses a technique for calculating a similarity between a search condition designated by a user and information, and extracting a certain number of articles or articles above a certain threshold value in the order of similarity. It is disclosed.
In Patent Document 3 (Japanese Patent Application Laid-Open No. 2000-148770), after calculating feature quantities of documents to be classified, obtaining similarity between these feature quantities, the documents are analyzed by mathematical and statistical cluster analysis. Techniques such as classification are disclosed.
[0006]
[Patent Document 1]
JP-A-10-275160
[Patent Document 2]
Japanese Patent Laid-Open No. 9-101990
[Patent Document 3]
JP 2000-148770 A
[0007]
[Problems to be solved by the invention]
As described above, in any of the techniques described in Patent Documents 1 to 3, highly related information is grouped by a clustering method or a grouping method in order to associate and present information to the user. .
By the way, in an information distribution system that can be customized for each user, it is not enough to group highly relevant articles, and the articles to be distributed are extracted from the grouped articles according to the user's wishes. There must be. For example, when distributing articles within the range of duplication that users can tolerate, set the duplication tolerance range for each user, and after grouping highly related articles, the duplication of article content is It is necessary to extract articles to be distributed from articles grouped so as to be within the allowable range.
[0008]
When the user sets the allowable range of duplication, for example, an index such as redundancy is set, and the redundancy of articles as distribution candidates is quantitatively calculated, and the redundancy set by the user is less than or equal to A configuration is possible in which articles to be distributed are extracted from grouped articles. However, if the amount of distribution candidate articles is constant, the amount of articles extracted (article data amount or the number of articles) will decrease when trying to eliminate redundancy, and conversely, the amount of articles extracted will increase. Attempting to do so increases the redundancy. Therefore, when setting the redundancy, if the user can grasp at the time of setting how the amount of articles extracted by his / her setting contents is affected, it is convenient for the user. In other words, there is a desire to eliminate redundancy as much as possible even if the amount of articles is somewhat reduced (redundancy priority type), and conversely, a request to increase the amount of articles as much as possible even if the redundancy is somewhat high (article amount priority type). ).
[0009]
However, all of the techniques described in Patent Documents 1 to 3 merely group information that is highly relevant by a clustering method or a grouping method, and set an allowable range of duplication according to such a user's request. It cannot be set.
In addition, all of the techniques described in Patent Documents 1 to 3 disclose a method for grouping, but do not disclose a method for extracting information after grouping. Therefore, there is a problem that information extraction suitable for the user's wishes cannot be performed for eliminating duplication of contents.
[0010]
  Further, in the technique described in Patent Document 1, it is necessary to add a feature such as a keyword to all documents, but it takes a lot of cost to assign a keyword to all documents.
  Therefore, the present invention has been made paying attention to such an unsolved problem of the conventional technology, and can perform information extraction suitable for the user's desire for eliminating duplication of contents. Suitable for reducing work costsDocument extraction systemThe purpose is to provide.
[0023]
[Invention 1]
  To achieve the above objectiveThe document extraction system of invention 1 is
  A system for extracting documents from a plurality of documents to be extracted,
  Document data storage means for storing a plurality of document data, redundancy setting means for setting the redundancy, and redundancy of the extracted document data is less than or equal to the redundancy set by the redundancy setting means Document data extraction means for extracting document data from the document data storage means, and a relationship between a set value that will be set by the redundancy setting means and the amount of extracted document that is extracted by the document data extraction means Setting support guidance output means for outputting, similarity calculation means for calculating the similarity between the document data of the document data storage means, and the document based on the similarity calculated by the similarity calculation means Document data classification means for hierarchically classifying the document data in the data storage means,
  The document data classifying unit is a distance between categories when the document data is classified based on the similarity calculated by the similarity calculating unit, and a cluster having a smaller distance as the similarity between the categories is larger. The inter-cluster distance is calculated, and based on the calculated inter-cluster distance, the document data of the document data storage means is classified hierarchically so that the document data having a higher similarity are closer to each other.
  In the dendrogram showing the hierarchical classification result with the inter-cluster distance as the vertical axis and the position of the classified document data as the horizontal axis, the document data extracting means And the inter-cluster distance corresponding to the set redundancy is identified. First, among the document data groups belonging to the cluster connected to the branch extending from each identified intersection to the lower layer side, the identification The document data belonging to the cluster connected to any one of the intersections is extracted, and then, from among the identified intersections, the lower side from each intersection other than the intersection connected to the cluster to which the extracted document data belongs Among the document data group belonging to the cluster connected to the branch extending toBetweenSimilarity ofButmosthighThe document data is extracted for each intersection.
[0024]
  With such a configuration, the setting support guidance output means outputs the relationship between the redundancy setting value and the extracted document amount. In response to this output, the user can easily understand how the extracted document amount is affected by his / her setting contents, so that the user's request (for example, redundancy priority type or document amount priority type) is met. A relatively appropriate redundancy can be set by the redundancy setting means. When the redundancy is set by the redundancy setting means, the document data extraction means extracts the document data from the document data storage means so that the redundancy of the extracted document data is equal to or less than the set redundancy. The
  Further, the similarity between the document data of the document data storage means is calculated by the similarity calculation means, and the document data is classified based on the calculated similarity by the document data classification means. The inter-cluster distance between the classifications is calculated, and the document data in the document data storage unit is hierarchically classified based on the calculated inter-cluster distance so that the document data having higher similarity are closer to each other. Then, each cluster to which each document data belongs in the dendrogram showing the result of hierarchical classification with the inter-cluster distance as the vertical axis and the position of each classified document data as the horizontal axis by the document data extraction means An intersection point between the branch connecting them and the inter-cluster distance corresponding to the set redundancy is specified. Further, the document data extracting means first extracts the document data belonging to the cluster connected to any one of the specified intersections from the document data group belonging to the cluster connected to the branch extending from the specified intersection to the lower layer side. Extracted. Next, among the specified intersections, the extracted document data out of the document data group belonging to the cluster connected to the branch extending to the lower layer side from each intersection other than the intersection connected to the cluster to which the extracted document data belongsBetweenSimilarity ofButmosthighDocument data is extracted for each intersection.
[0025]
  As a result, the user can easily understand how the extracted document amount is affected by his / her setting contents, and can therefore set a relatively appropriate degree of redundancy in accordance with his / her request. In addition, since documents having a high degree of similarity are difficult to select, it is possible to reduce the possibility of extracting documents having similar contents in duplicate. Therefore, as compared with the prior art, it is possible to extract a document that is relatively suitable for the user's desire to eliminate duplication of contents.
Furthermore, when extracting a document, an operation such as assigning a keyword to each document is not required, so that the cost required for the operation can be reduced as compared with the conventional case.
Furthermore, since some users do not always select only those with low redundancy, in such a case, it is possible to extract a document having a certain degree of relevance while eliminating a certain degree of redundancy. The effect that it is possible is also acquired.
[0026]
  Here, the extracted document amount includes the data amount of the document to be extracted or the number of documents. The same applies to the document extraction system according to the second aspect.
  The setting support guidance output means may have any configuration as long as it outputs the relationship between the redundancy setting value and the extracted document amount. For example, the setting support guidance output unit outputs the setting value and the extracted document amount. One set or a plurality of sets corresponding to each other may be output, or a change in the extracted document amount with respect to a change in the set value may be output in a graph format or the like. The same applies to the document extraction system according to the second aspect.
  Further, the setting support guidance output means may have any configuration as long as it outputs the relationship between the redundancy setting value and the extracted document amount. For example, the setting value of the redundancy setting value and the extracted document The relationship with the amount may be visually output by display or printing, or the relationship between the redundancy setting value and the extracted document amount may be output audibly by a predetermined voice pattern or the like. It may be. Any other method that can be perceived by the user may be used. The same applies to the document extraction system according to the second aspect.
  The document data storage means stores the document data at any time and at any time. The document data storage means may store the document data in advance, or store the document data in advance without storing the document data. The document data may be stored by external input or the like when the system is operating. The same applies to the document extraction system according to the second aspect.
  Further, the present system may be realized as a single device, terminal, or other device, or may be realized as a network system in which a plurality of devices, terminals, or other devices are communicably connected. In the latter case, each component may belong to any one of a plurality of devices and the like as long as they are connected so as to communicate with each other. The same applies to the document extraction system according to the second aspect.
  [Invention 2]In order to achieve the above purposeThe document extraction system of invention 2 is
  A system for extracting documents from a plurality of documents to be extracted,
  Document data storage means for storing a plurality of document data, redundancy setting means for setting the redundancy, and redundancy of the extracted document data is less than or equal to the redundancy set by the redundancy setting means Document data extraction means for extracting document data from the document data storage means, and a relationship between a set value that will be set by the redundancy setting means and the amount of extracted document that is extracted by the document data extraction means Setting support guidance output means for outputting, similarity calculation means for calculating the similarity between the document data of the document data storage means, and the document based on the similarity calculated by the similarity calculation means Document data classification means for hierarchically classifying the document data in the data storage means,
  The document data classifying unit is a distance between categories when the document data is classified based on the similarity calculated by the similarity calculating unit, and a cluster having a smaller distance as the similarity between the categories is larger. The inter-cluster distance is calculated, and based on the calculated inter-cluster distance, the document data of the document data storage means is classified hierarchically so that the document data having a higher similarity are closer to each other.
  In the dendrogram showing the hierarchical classification result with the inter-cluster distance as the vertical axis and the position of the classified document data as the horizontal axis, the document data extracting means The intersection of the branch connecting the two and the inter-cluster distance corresponding to the set redundancy is specified, and the document extracted from the document data group belonging to the cluster connected to the branch extending from the specified intersection to the lower layer side is identified. dataBetween each otherThe document data is extracted one by one for each intersection so that the total sum of the similarities is minimized.
  With such a configuration, the setting support guidance output means outputs the relationship between the redundancy setting value and the extracted document amount. In response to this output, the user can easily understand how the extracted document amount is affected by his / her setting contents, so that the user's request (for example, redundancy priority type or document amount priority type) is met. A relatively appropriate redundancy can be set by the redundancy setting means. When the redundancy is set by the redundancy setting means, the document data extraction means extracts the document data from the document data storage means so that the redundancy of the extracted document data is equal to or less than the set redundancy. The
  Further, the similarity between the document data of the document data storage means is calculated by the similarity calculation means, and the document data is classified based on the calculated similarity by the document data classification means. The inter-cluster distance between the classifications is calculated, and the document data in the document data storage unit is hierarchically classified based on the calculated inter-cluster distance so that the document data having higher similarity are closer to each other. Then, the document data extraction means connect the clusters to which each document data belongs in the dendrogram showing the hierarchical classification result with the distance between the clusters as the vertical axis and the position of each document data as the horizontal axis. The intersection point between the branch and the intercluster distance corresponding to the set redundancy is specified. Furthermore, the extracted document data from the document data group belonging to the cluster connected to the branch extending from the specified intersection to the lower layer side.Between each otherDocument data is extracted one by one for each intersection so that the total sum of the similarities is minimized.
  As a result, the user can easily understand how the extracted document amount is affected by his / her setting contents, and can therefore set a relatively appropriate degree of redundancy in accordance with his / her request. In addition, since documents having a high degree of similarity are difficult to select, it is possible to reduce the possibility of extracting documents having similar contents in duplicate. Therefore, as compared with the prior art, it is possible to extract a document that is relatively suitable for the user's desire to eliminate duplication of contents.
  Furthermore, when extracting a document, an operation such as assigning a keyword to each document is not required, so that the cost required for the operation can be reduced as compared with the conventional case.
  Furthermore, each document data can be extracted so that the sum of the similarities is minimized, so that it is possible to further reduce the possibility of extracting duplicate documents having similar contents. .
[Invention 3]
  Furthermore, the document extraction system of the invention 3 is the document extraction system of any one of the inventions 1 and 2,
  The setting support guidance output means outputs a change in the extracted document amount with respect to a change in the setting value when the setting value is changed within a settable range of the redundancy setting means. Features.
[0027]
  With such a configuration, when the setting value is changed within the settable range of the redundancy setting means by the setting support guidance output means, a change in the extracted document amount with respect to the change in the setting value is output.
  This makes it easier for the user to understand how the extracted document amount is affected by his / her settings. Therefore, it is possible to obtain a document extraction that further suits the user's desire for eliminating duplication of contents.
〔invention4]
  In addition, the invention4The document extraction system of the invention3In the document extraction system of
  And further comprising setting support information storage means for storing setting support information indicating a relationship between the setting value and the extracted document amount,
  The setting support guidance output means outputs a relationship between the setting value and the extracted document amount based on setting support information in the setting support information storage means.
[0028]
With such a configuration, the setting support guidance output means outputs the relationship between the setting value and the extracted document amount based on the setting support information in the setting support information storage means.
As a result, the extracted document amount that is relatively suitable for the set value is output, so that it becomes easier for the user to understand how the extracted document amount is affected by his / her setting contents. Therefore, it is possible to obtain a document extraction that further suits the user's desire for eliminating duplication of contents.
[0029]
  Here, the setting support information storage means stores the setting support information by any means and at any time. The setting support information may be stored in advance, or the setting support information may be stored in advance. Instead, the setting support information may be stored by external input or the like during operation of the system.
〔invention5]
  In addition, the invention5The document extraction system of the invention3In the document extraction system of
  Furthermore, setting support information generating means for generating setting support information indicating a relationship between the setting value and the extracted document amount based on the extraction result of the document data extracting means,
  The setting support guidance output means outputs the relationship between the setting value and the extracted document amount based on the setting support information generated by the setting support information generation means.
[0030]
  With this configuration, the setting support information generation unit generates setting support information indicating the relationship between the setting value and the extracted document amount based on the extraction result of the document data extraction unit, and the setting support guide output unit Based on the generated setting support information, the relationship between the setting value and the extracted document amount is output.
  As a result, the extracted document amount that is relatively suitable for the set value is output, so that it becomes easier for the user to understand how the extracted document amount is affected by his / her setting contents. Therefore, it is possible to perform document extraction that further suits the user's desire for eliminating duplicate contents..
[0035]
[invention6]
  In addition, the invention6The document extraction system of the invention1Or5One of1In the document extraction system of
  Furthermore, user information storage means for storing user information about the user, and document data selection means for selecting a plurality of document data from the document data storage means based on the user information of the user information storage means. ,
  The similarity calculation means and the document data classification means are configured to calculate the similarity and classify the document data for the document data selected by the document data selection means.
[0036]
With such a configuration, the document data selection means selects a plurality of document data from the document data storage means based on the user information in the user information storage means. Next, the similarity calculation unit calculates the similarity between the selected document data, and the document data classification unit selects the selected document data hierarchically based on the calculated similarity. being classified. Then, the document data extracting means extracts the document data from the classified document data group based on a predetermined classification rule.
[0037]
As a result, after selecting document data having contents suitable for the user's wishes, document data having a high degree of similarity is excluded from the selected document data. There is also an effect that document extraction can be performed with emphasis on user preference. For example, it is possible to provide suitable information to users who are seeking information that suits their preferences rather than a wide range of information.
[0038]
  Here, the user information storage means stores user information at any time and at any time, and may store user information in advance, or without storing user information in advance, User information may be stored by external input or the like during operation of the system. Hereinafter, the invention7This is the same in all document extraction systems.
〔invention7]
  In addition, the invention7The document extraction system of the invention1Or5One of1In the document extraction system of
  Further, a user information storage unit for storing user information regarding the user, and a document for selecting document data based on the user information of the user information storage unit from among a plurality of document data groups extracted by the document data extraction unit Data selection means.
[0039]
  With such a configuration, the document data extraction unit extracts the document data from the classified document data group based on a predetermined classification rule, and the document data selection unit extracts the user data in the user information storage unit. Based on this, document data is selected from the extracted document data group.
  Thus, after document data having a high degree of similarity is excluded, document data having contents that match the user's wishes are selected from the data, so that bias due to user preferences is reduced and information is more than user preferences. It is also possible to perform document extraction that emphasizes the uniformity of the image. For example, it is possible to provide suitable information to a user who is seeking a wider range of information than information suitable for his / her preference.
〔invention8]
  In addition, the invention8The document extraction system of the invention1Or7One of1In the document extraction system of
  The similarity calculation means includes a document dividing means for dividing a document of each document data in the document data storage means into predetermined character elements, and the document data storage based on the appearance frequency of the character elements divided by the document dividing means. A vector calculation means for calculating a similarity between the document data of the document data storage means based on the document vector generated by the document vector generation means and a document vector generation means for generating a document vector for each document data of the means And a calculating means.
[0040]
With such a configuration, the document dividing unit divides the document data in the document data storage unit into predetermined character elements, and the document vector generation unit generates a document based on the appearance frequency of the divided character elements. A document vector is generated for each document data in the data storage means. Then, the similarity between the document data in the document data storage unit is calculated by the vector calculation similarity calculation unit based on the generated document vector.
[0041]
Thereby, since the similarity between each document data can be calculated comparatively appropriately, the possibility that duplicate documents having similar contents can be further reduced. Therefore, it is possible to obtain a document extraction that further suits the user's desire for eliminating duplication of contents.
Here, the character element includes a character string and also includes a single character. Examples of single characters include particles (“no”, “ha”, “ga”, “ni”, “wo”, “ya”).
[0042]
  The document vector generation means may have any configuration as long as the document vector is generated based on the appearance frequency of the character element. For example, the document vector generation unit directly generates the document vector from the appearance frequency of the character element. The intermediate product (for example, other vectors) may be generated from the appearance frequency of the character elements, and the document vector may be generated from the generated intermediate product.
〔invention9]
  In addition, the invention9The document extraction system of the invention8In the document extraction system of
  The document dividing unit divides a document of each document data in the document data storage unit into predetermined character elements by using any one of character element dividing methods of morphological analysis, n-gram and stop word. It is characterized by that.
[0043]
  With such a configuration, the document dividing unit divides each document data document of the document data storage unit into predetermined character elements using any one of morphological analysis, n-gram, and stop word character element division methods. Is done.
  Character string segmentation methods such as morphological analysis, n-gram, and stop word are methods with excellent reliability that have been widely used in the past. By using these as document segmentation means of the present invention, each document is relatively appropriate. In addition to being able to be divided into character elements, it is possible to obtain an effect that it is possible to deal with various forms of documents by using any of these methods.
〔invention10]
  In addition, the invention10The document extraction system of the invention8and9One of1In the document extraction system of
  The document vector generation unit is configured to use TFIDF for each document data of the document data storage unit based on the appearance frequency of the character element in the document in which the character element appears and the appearance frequency of the character element in the plurality of documents. A weighted document vector is generated.
[0044]
  With such a configuration, each document data in the document data storage unit is generated by the document vector generation unit based on the appearance frequency of the character element in the document in which the character element appears and the appearance frequency of the character element in a plurality of documents. A document vector weighted by TFIDF is generated.
  When generating the document vector of each document, the appearance frequency of the divided character element may be used as it is. However, if a known weighting method reflecting the importance of the character element called TFIDF is used, the characteristics of each document are reflected. A document vector can be generated. Therefore, the degree of similarity between the respective document data can be calculated more appropriately, so that the possibility of extracting duplicate documents having similar contents can be further reduced. Therefore, it is possible to obtain a document extraction that further suits the user's desire for eliminating duplication of contents.
〔invention11]
  In addition, the invention11The document extraction system of the invention8Or10One of1In the document extraction system of
  The vector calculation similarity calculation means calculates the similarity between document data by a vector space method based on the document vector generated by the document vector generation means.
[0045]
  With such a configuration, the similarity between the document data is calculated by the vector space similarity calculation means based on the generated document vector by the vector space method.
  If the vector space method is used as a method for calculating the similarity between each document data, the similarity between two document vectors can be quantitatively expressed as a cosine value (0 to 1) of an angle formed by the two vectors. It is possible to obtain the effect that document extraction can be performed more accurately..
[0052]
DETAILED DESCRIPTION OF THE INVENTION
  Embodiments of the present invention will be described below with reference to the drawings. 1 to 22 relate to the present invention.Document extraction systemIt is a figure which shows this embodiment.
[0053]
  This embodiment relates to the present invention.Document extraction systemAs shown in FIG. 1, this is applied to a case where a plurality of documents with little content overlap is provided to the user.
[0054]
FIG. 1 shows an embodiment of a document extraction apparatus 10 according to the present invention. As shown in FIG. 1, the document extraction apparatus 10 includes a document data registration database (hereinafter simply referred to as “DB”) 12 for registering a plurality of document data, and document data in the document data registration DB 12. A similarity calculation unit 14 that calculates the similarity between the document data, a document data classification unit 16 that hierarchically classifies the document data in the document data registration DB 12 based on the similarity calculated by the similarity calculation unit 14, User information registration DB 18 that has registered user information, document data extraction unit 20 that extracts document data from the document data group classified by the document data classification unit 16 based on user information in the user information registration DB 18, and redundancy Redundancy setting unit 22 for setting, setting support information generating unit 24 for generating setting support information, and setting support for registering setting support information An information registration DB 26, is composed of a setting support guide display unit 28 for displaying help guidance that can be used as a guide in setting the redundancy in redundancy setting unit 22.
[0055]
The document data registration DB 12 registers some information supplied from an information supply source S in an information communication network such as the Internet as document data.
Based on the similarity calculated by the similarity calculation unit 14, the document data classification unit 16 is a distance between classifications (clusters) when document data is classified. The inter-cluster distance is calculated, and based on the calculated inter-cluster distance, the document data in the document data registration DB 12 is hierarchically classified so that the document data with higher similarity are closer to each other. As the hierarchical classification method, for example, a conventional clustering method or a grouping method can be adopted. In this embodiment, the document data is classified by the clustering method.
[0056]
Based on the user information in the user information registration DB 18, the document data extraction unit 20 has a redundancy corresponding to the inter-cluster distance below the hierarchy in the hierarchical structure classified by the document data classification unit 16 (user-set redundancy). ) The following points are specified, and the document data under the specified point is extracted from the document data group.
As shown in FIG. 1, the similarity calculation unit 14 divides a document of each document data in the document data registration DB 12 into a predetermined character string, and the appearance frequency of the character string divided by the document dividing unit 30. The document vector generation unit 32 that generates a document vector for each document data in the document data registration DB 12 based on the document data, and the similarity between the document data in the document data registration DB 12 based on the document vector generated by the document vector generation unit 32 It comprises a vector calculation similarity calculation unit 34 for calculating.
[0057]
The similarity calculation unit 14 first divides the document of each document data selected from the document data registration DB 12 into character strings by the document dividing unit 30, and then uses the document vector generation unit 32 to determine the appearance frequency of each divided character string. The document vector of each document data is generated by calculation. Thereafter, the similarity between the document vectors obtained by the document vector generation unit 32 is calculated by the vector calculation similarity calculation unit 34.
[0058]
The redundancy setting unit 22 sets an upper limit value (redundancy) of an overlapping range that can be allowed by the user based on an input from the user, and registers user information including the set redundancy in the user information registration DB 18. ing.
Based on the user information in the user information registration DB 18 and the extraction result of the document data extraction unit 20, the setting support information generation unit 24 extracts the setting value that will be set by the redundancy setting unit 22 and the document data extraction unit 20. Setting support information indicating the relationship with the extracted document amount is generated, and the generated setting support information is registered in the setting support information registration DB 26.
[0059]
When the setting support guide display unit 28 changes the set value within the settable range of the redundancy setting unit 22 based on the setting support information in the setting support information registration DB 26 in response to a request from the user, the setting support guide display unit 28 The change of the extracted document amount with respect to the change is displayed.
The document extraction apparatus 10 is realized by a computer 100 configured as shown in FIG.
[0060]
FIG. 2 is a block diagram illustrating a configuration of the computer 100.
As shown in FIG. 2, the computer 100 includes a CPU 50 that controls operations and the entire apparatus based on a control program, a ROM 52 that stores a control program of the CPU 50 in a predetermined area, data read from the ROM 52, etc. A RAM 54 for storing calculation results necessary in the calculation process of the CPU 50 and an I / F 58 for mediating input / output of data to / from an external device are configured. These are signal lines for transferring data. Are connected to each other and to be able to exchange data.
[0061]
The I / F 58 includes, as external devices, an input device 60 such as a keyboard or mouse capable of data input, a display device 64 that displays a screen based on an image signal, a document data registration DB 12, and a user information registration DB 18. And are connected.
The document data registration DB 12 is an external storage device such as a hard disk, for example, and is supplied with predetermined information from an information supply source S such as the Internet periodically or as needed.
[0062]
The CPU 50 includes a microprocessing unit (MPU) and the like, starts a document extraction program stored in a predetermined area of the ROM 52, and according to the program, the similarity calculation unit 14, the document data classification unit 16, and the document data extraction unit 20 As shown in FIG. 8, the document data extraction process (FIG. 8) is executed.
[0063]
Next, the operation of the present embodiment will be described.
As shown in FIG. 1, first, in the document data registration DB 12, document data having a content corresponding to the user's preference is supplied from the information supply source S at regular intervals or irregularly, and temporarily stored. When the number of document data reaches a predetermined number, or when the storage time has passed for a certain time, all of the stored document data is once sent to the similarity calculation unit 14, where each document data Similarity is calculated.
[0064]
First, each document data sent to the similarity calculation unit 14 is divided into character strings by the document dividing unit 30.
FIG. 3 is a diagram illustrating an example of character string division by morphological analysis.
The character string dividing method (method) is not particularly limited. For example, as shown in FIG.₁~ D_mCan be divided into character strings (words) by grammatical division with reference to the morphological analysis dictionary. Here, there are various methods for morphological analysis, and the result differs depending on whether the dictionary is good or bad. For example, “wireless /// security /// topic /////” in FIG. And the like, such as nouns, verbs, adjective particles and auxiliary verbs. In addition, morphological analysis has good accuracy of segmentation, but previously there was a drawback that it took cost to create and maintain the dictionary to maintain accuracy, but recently, dictionaries that have been well built for many years Since it can be used as an asset, the problem of cost has gradually been solved, and it is the most commonly used character string dividing method. However, the morphological analysis is disadvantageous in that a Japanese dictionary can be used only in Japanese, and other languages such as English and Chinese require a dictionary for that language.
[0065]
Each document D₁~ D_mIt is also possible to use a character string dividing method called n-gram that cuts a character string at regular intervals instead of morphological analysis when dividing the character string into character strings.
FIG. 4 is a diagram illustrating an example of character string division by n-gram.
When the n-gram method is used, the document is divided as shown in FIG. In other words, “n” in n-gram is a number indicating how many bytes (or every character), and in the case of FIG. 4, since it is every two characters, it can be written as 2-gram. However, in the case of a 2-byte character such as Japanese, since 2 characters = 4 bytes, 4-gram may be written, but here, the accuracy of the number is not a problem. In n-gram, it is difficult to cut out meaningful words as a lump, but if only the divided words are processed statistically as they are, it is not always necessary that the meaningful words are lump. is there. In addition, n-gram has an advantage that the algorithm is simpler than morphological analysis and can be used for any language.
[0066]
Each document D₁~ D_mInstead of morphological analysis and n-gram, it is also possible to use a character string dividing method called a stop word.
FIG. 5 is a diagram illustrating an example of character string division by stop words.
The stop word method is a method of registering characters and rules to be cut in a document and dividing them according to them. For example, in the example shown in FIG. 5, (1) “no”, “ha”, “ga”, “ni”, “ya”, “2”, punctuation marks “,” “.”, “3” The division is made when any of the three rules such as kanji, katakana, and the change of the character type such as alphabet is established. Although it is possible to extract words that are meaningful to some extent, there is a problem that long words such as “information communication technology” and long katakana compound words such as “Internet technology” cannot be divided. In English, it is possible to use words based on rules such as (1) space, (2) comma, period, colon, semicolon, other symbols, (3) alphabet, numbers, symbols, etc. By using a technique called stemming that drops the utilization form, it is possible to perform character string division to some extent.
[0067]
In this way, all the documents D are processed by the document dividing unit 30.₁~ D_mThen, the document vector generation unit 32 calculates the character string frequency and creates a character string-document matrix as shown in FIG.
FIG. 6 is a diagram showing a character string-document matrix.
FIG. 7 is a diagram illustrating a result of character string division by morphological analysis.
[0068]
As shown in FIG. 6, the character string-document matrix represents each document D.₁~ D_mAnd a unique string T₁~ T_nEach character string T₁~ T_nIs each document D₁~ D_mThis is a count of how many times it appears in the. For example, in the case of a division result using morphological analysis as a character string division method, as shown in FIG.₁The character string “T” (shaded character) (T₁) Has appeared three times and its W₁₁The element of the matrix corresponding to is “3” when the number of appearances is used as it is.
[0069]
Where W_mnThe number of occurrences of the character string may be used as it is for each element of the matrix corresponding to, but if a weighting method that reflects the importance of the character string called TFIDF is used, a document vector that well expresses the characteristics of each document is generated. It is known that it can be used, and can be used in later calculation of mutual similarity.
That is, as shown in the following formula (1), TFIDF indicates the appearance frequency (TF: Term Frequency) of a character string T in a document D and the frequency of the number of documents in which the character string T appears in the entire document set. It is obtained by the product of an inverse number (IDF: Inverse Document Frequency). The larger the numerical value, the more important the character string T is. TF is an index that a character string that appears frequently is important, and has a property of increasing as the frequency of occurrence of a character string in a document increases. IDF is an index that character strings appearing in many documents are not important, that is, character strings appearing in a specific document are important, and increase as the number of documents in which a certain character string is used decreases. Have nature. Therefore, the value of TFIDF has a property of increasing for a character string that frequently appears in a specific document. Conversely, a character string that frequently appears but appears in many documents (conjunctions, particles, etc.). ), Or a character string that appears only in a specific document and appears infrequently in that document. The character string in the document is digitized by TFIDF, and the document can be vectorized using the numeric value taking the importance of the character string into account as an element.
[0070]
[Expression 1]

[0071]
FIG. 8 is a flowchart showing a part of the document data extraction process.
In the document data extraction process, as shown in FIG. 8, first, the process proceeds to step S100. Note that the process shown in the flowchart of FIG. 8 is a process realized as the similarity calculation unit 14 in the document data extraction process.
In step S100, the document data registered in the document data registration DB 12 is selected one by one, the process proceeds to step S102, the document of each selected document data is divided for each character string, and the process proceeds to step S104.
[0072]
In step S104, the frequency information of the character string is stored in the character string-document matrix indicating the correspondence between the document and the character string, and the process proceeds to step S106 to check whether unprocessed document data remains in the document data registration DB 12 or not. When it is determined whether or not unprocessed document data remains (Yes), the document data is selected, the same processing is performed, and this is repeated until there is no more document data.
[0073]
On the other hand, if it is determined in step S106 that there is no unprocessed document data remaining in the document data registration DB 12 (No), the process proceeds to step S108, based on the frequency information of the completed character string-document matrix. A character string-document matrix re-weighted by TFIDF is generated. As a result, all the document data can be expressed as a document vector having the same dimension (several thousands to hundreds of thousands) as the number of unique character strings appearing therein.
[0074]
As described above, when all the document data is vectorized, the similarity between the respective document data is calculated by the vector calculation similarity calculation unit 34. Specifically, the vector calculation similarity calculation unit 34 employs a well-known vector space method, and each document vector obtained by TFIDF has a mutual similarity defined by the vector space method. Become. That is, the similarity between the two document vectors to be compared can be defined as a cosine value (0 to 1) of the angle θ formed by the two vectors as shown in FIG. 10 can be expressed by a symmetric matrix as shown in FIG.
[0075]
FIG. 9 is a diagram showing document vectors and their correlation.
10 and 11 are diagrams showing a symmetric matrix between documents.
Thereafter, by extracting and grouping similar information based on the symmetric matrix, document extraction excluding similar documents can be realized. For example, in the symmetric matrix as shown in FIG. 10, as shown in FIG.₁And document D₂Has a similarity of "0.9" and document D₁And document D₃The degree of similarity between the document data is quantitatively shown such that the degree of similarity is “0.3”.
[0076]
As described above, when the similarity between the respective document data is quantitatively obtained by the similarity calculation unit 14, the document data classification unit 16 performs clustering based on the similarity calculated by the similarity calculation unit 14. The document data in the document data registration DB 12 is classified hierarchically by the technique.
The clustering method is a method for statistically analyzing multivariate. Document vector D₁~ D_nCan be thought of as pointing to a point in a multidimensional space (the number of dimensions is the same as the number of unique character strings, generally 10,000 to 100,000). Can be defined. The distance is document D₁~ D_nIs the distance between clusters. For example, the conventional shortest distance method, longest distance method, group average method, centroid method, median method, and Ward method can be adopted as the method of defining the intercluster distance. In this embodiment, the centroid method is used. Cluster analysis from the document-document symmetric matrix. Note that the inter-cluster distance is not a similarity between clusters but a distance, and therefore, as shown in FIG. 12, a relationship of “1-similarity” is obtained with respect to the similarity normalized to 0 to 1.
[0077]
FIG. 12 is obtained by converting the symmetric matrix of the similarity between documents in FIG. 11 into the distance between clusters.
Next, taking the case of FIG.₁~ D₄The case of classifying will be described.
When the most similar clusters p and q are integrated to form a new cluster t, the intercluster distance Str between the cluster t and another cluster r can be calculated by the following equation (2).
[0078]
[Expression 2]

[0079]
In the above equation (2), Spr is the intercluster distance between the clusters p and r, Sqr is the intercluster distance between the clusters q and r, and Spq is the intercluster distance between the clusters p and q. Np is the number of documents included in cluster p, nq is the number of documents included in cluster q, and nr is the number of documents included in cluster r.
13 and 14 show the document D₁~ D₄It is a figure for demonstrating the case where the distance between clusters of the cluster which belongs to is calculated.
[0080]
First, as shown in FIG.₁~ D₄Document D₁, D₂Is the closest to the distance “0.1”, and the two are combined to form a new cluster D₁'. Cluster D₁The intercluster distance between ′ and other clusters is p = D₁, Q = D₂, T = D₁'Is calculated by the above equation (2).
Cluster D₁'And other clusters (Document D₃As shown in FIG. 13, the inter-cluster distance St3 with “)” is “0.8” according to the above equation (2). Cluster D₁'And other clusters (Document D₄As shown in FIG. 13, the inter-cluster distance St4 with “)” is “0.375” according to the above equation (2).
[0081]
Next, as shown in FIG.₁'And cluster D₄Is the closest to the distance “0.3”, and the two clusters are integrated into a new cluster D₁"". Cluster D₁The inter-cluster distance between ″ and other clusters is p = D₁′, Q = D₄, T = D₁″ Is calculated by the above equation (2).
Cluster D₁‘’ And other clusters (Document D₃), The inter-cluster distance St3 is “0.85” according to the above equation (2) as shown in FIG.
[0082]
FIG. 15 shows a document D constructed by cluster analysis.₁~ D₄It is a figure which shows the hierarchical structure of.
As a result, document D₁~ D₄Are classified hierarchically as shown in FIG.
In this way, the document data classification unit 16 performs document D.₁~ D₄Are hierarchically classified, the document data extraction unit 20 extracts the document data from the document data group classified by the document data classification unit 16 based on the user information in the user information registration DB 18.
[0083]
FIG. 16 is a diagram for explaining a case where document data is extracted from the document data group classified by the document data classification unit 16.
Here, it is assumed that the redundancy set by the users A and B is “0.7” for the user A and “0.4” for the user B. That is, the user A thinks that some overlap may be allowed, but the user B thinks that the overlap is not allowed much.
[0084]
Since the redundancy level permitted by user B is “0.4”, document D is assigned to user B from the relationship with the hierarchical structure of FIG.₁, D₂, D₄Document D₃It can be seen that it is sufficient to provide two documents. That is, document D₁, D₂, D₄Cluster and document D₃This means that the cluster is sufficiently distant (not similar in content).
[0085]
Specifically, Document D₁~ D₄A method of selecting a document to be provided to the user B from among the above will be described in detail with reference to FIG.
FIG. 17 shows document D₁~ D₄It is a figure for demonstrating the case where the document provided to the user B is selected from among these.
First, for the redundancy “0.4” set by the user B, the intersection with the hierarchical structure in FIG.₃A line extending from the cluster and document D₁, D₂, D₄It can be seen that it intersects two of the lines extending from the cluster. At this point, Document D₃Is finalized and the rest is document D₁, D₂, D₄Let's select one from
[0086]
Document D already₃Since the symmetric matrix between the document and the document shown in FIG.₃And the remaining document D₁, D₂, D₄Judgment based on the relationship with the similarity. Basically, Document D₁, D₂, D₄If any one is selected from the above, the general requirement of redundancy “0.4” or less is cleared. If it is possible to set an attribute that the user B desires to have a redundancy level of “0.4” or less and to reduce the redundancy level as much as possible, and the attribute is set, the document D₃Document D with the lowest similarity₂Select.
[0087]
In some cases, only those with low redundancy are not necessarily selected. It is possible to select a document having a certain degree of relevance while eliminating a certain degree of redundancy. In the example of FIG.₃Since the condition that the redundancy is “0.4” or less is satisfied by selecting, a similar document (D₁, D₄), It is possible to eliminate redundancy as much as possible and to select a document that is as close to the set redundancy as possible. In the example of FIG.₃And the similarity of document D₁, D₄It is not a clear example to narrow down to any one because the similarities of are equal, but in reality it deals with document vectors with hundreds to hundreds of thousands of dimensions, so there is almost no possibility of equivalence In many cases, it is possible to select a document by the above-described two methods (a selection method for reducing the redundancy and a selection method for approaching the set redundancy) in numerical values.
[0088]
On the other hand, since the degree of redundancy permitted by user A is “0.7”, document D is informed to user A from the relationship with the hierarchical structure of FIG.₁, D₂Document D₃, D₄It can be seen that it is sufficient to provide the following three documents. That is, document D₁, D₂Cluster and document D₃Cluster and document D₄This means that the cluster is sufficiently distant (not similar in content). Document D₁, D₂As to which of the clusters to be selected, for example, random or a combination with less overlap, the document D can be obtained from the document-document symmetric matrix of FIG.₃, D₄It is sufficient to select the one having a low similarity to.
[0089]
In this case, document D₁Will select Document D₃Is "0.3", Document D₄Becomes "0.5" and document D₂Will select Document D₃Is "0.2", Document D₄Is "0.8", so if you simply consider the sum of similarities, document D₁Select. Document D₁Is selected, the already selected document D₃, D₄The sum of the similarities with is small. In the example of FIG. 16, the result may not change, but as shown in FIG.₄Document D with maximum similarity in relation to₂Avoid Document D₁Or vice versa, already selected document D₃Document D with minimum similarity in relation to₂There are various ways to choose.
[0090]
Specifically, Document D₁~ D₄A method for selecting a document to be provided to the user A from among them will be described in detail with reference to FIG.
FIG. 18 shows document D₁~ D₄It is a figure for demonstrating the case where the document provided to the user A is selected from among.
First, for the redundancy “0.7” set by the user A, the intersection with the hierarchical structure of FIG.₃A line extending from the cluster and document D₄A line extending from the cluster and document D₁, D₂It can be seen that it intersects three of the lines extending from the cluster. At this point, Document D₃, D₄Is finalized and the rest is document D₁, D₂One of them will be selected.
[0091]
Document D₃, D₄Is confirmed, so document D₁, D₂And document D₃, D₄Redundancy is calculated for a combination of three documents including, and from among all possible combinations, a combination in which the calculated redundancy is equal to or less than a predetermined value is determined, and the remaining one is selected so as to be the determined combination. select.
Document D₁, D₂And document D₃, D₄The combination of (D₁, D₃, D₄) And (D₂, D₃, D₄However, for each combination, the redundancy is the sum of the similarities of all combinations when two documents are selected from the document group of the combination, and the values are averaged. This is calculated as the redundancy.
[0092]
First, (D₁, D₃, D₄) To select two documents from among them, (D₁, D₃), (D₁, D₄) And (D₃, D₄). Since the respective similarities are “0.3”, “0.5”, and “0.3” as shown in FIG. 11, the total sum thereof is “1.1”. Therefore, the average value is “0.37”, and this value is (D₁, D₃, D₄) Is calculated as a redundancy degree.
[0093]
Second, (D₂, D₃, D₄) To select two documents from among them, (D₂, D₃), (D₂, D₄) And (D₃, D₄). Since the respective similarities are “0.2”, “0.8”, and “0.3” as shown in FIG. 11, the total sum thereof is “1.3”. Therefore, the average value is “0.43”, and this value is (D₂, D₃, D₄) Is calculated as a redundancy degree.
[0094]
Here, since the redundancy set by the user A is “0.7”, (D₁, D₃, D₄) And (D₂, D₃, D₄) Is within an acceptable range for user A. Therefore, user A receives document D₁~ D₄It is desirable to provide a combination of these. In this case, if a document with low redundancy is selected, the document D₁And select a document that is close to the set redundancy.₂Should be selected.
[0095]
Next, a case where the user sets the redundancy will be described.
When the user sets the redundancy, the user who is accustomed to the setting can set an appropriate redundancy according to his / her request. Since it is not possible to grasp how the quantity is affected, it is difficult to set an appropriate redundancy according to one's request. Therefore, in the present embodiment, the user setting is supported by displaying support guidance in response to a user request.
[0096]
FIG. 19 is a diagram illustrating a redundancy setting screen.
When the user requests redundancy setting, a setting screen as shown in FIG. 19 is displayed. On the setting screen of FIG. 19, a text box 300 for inputting redundancy, a button 302 for requesting display of support guidance, and a button 304 for requesting registration of redundancy are arranged. When the user first clicks the button 302 with a mouse or the like prior to setting the redundancy, the setting support guidance display unit 28 can set the redundancy setting unit 22 based on the setting support information in the setting support information registration DB 26. When the set value is changed within the range, the change of the extracted document amount with respect to the change of the set value is displayed. Setting support information is registered in the setting support information registration DB 26. This is based on the redundancy set by another user by the setting support information generation unit 24 and the document data extraction unit 20 based on the redundancy. The relationship with the extracted document amount extracted in step (b) is registered in the setting support information registration DB 26 as setting support information.
[0097]
By referring to this display, the user can understand how the amount of extracted documents is affected by his / her settings, so even users who are unfamiliar with the settings can meet his / her needs. The appropriate redundancy can be set.
FIG. 20 is a diagram illustrating an example of a classification result obtained by hierarchically classifying document data by the document data classification unit 16.
[0098]
For example, in the classification result of FIG. 20, the document D1 and the document D2 form a cluster D12, the document D3 and the document D4 form a cluster D34, the cluster D12 and the cluster D34 form a cluster D14, and the cluster D14 and the document D5 constitutes a cluster D15. The document D7 and the document D8 constitute a cluster D78, and the cluster D78 and the document D6 constitute a cluster D68. In addition, cluster D15 and cluster D68 constitute cluster D18.
[0099]
FIG. 21 is a diagram showing the relationship between the set value and the number of documents in a slide bar format.
In the case of FIG. 20, when the user clicks the button 302 on the setting screen of FIG. 19, the setting value is changed when the setting value is changed within the settable range of the redundancy setting unit 22 as shown in FIG. The change in the amount of extracted document with respect to the change in is displayed in a slide bar format. In the example of FIG. 21, the redundancy is “0”, “0.1”, “0.2”, “0.3”, “0.4”, “0.5”, “0.6”, “ When set to “0.7”, “0.8”, “0.9” and “1.0”, 0, 0, 2, 2, 2, 4, 4, 5 This shows that pieces of documents, 6, 8, and 8 documents are extracted, respectively.
[0100]
When the user slides to a desired redundancy while operating the slide bar and clicks the button 306, the setting value by the slide bar is input to the text box 300 on the setting screen of FIG. 19. Then, by clicking the button 304, the redundancy is set.
FIG. 22 is a diagram showing the relationship between the set value and the number of documents in a graph format.
[0101]
In addition to FIG. 21, as shown in FIG. 22, when the set value is changed within the settable range of the redundancy setting unit 22, the change in the extracted document amount with respect to the change in the set value is displayed in a graph. It is also possible. In the case of FIG. 22, the relationship between the setting value and the document amount can be more visually grasped.
Similarly, when the user slides the slide bar to a desired redundancy while operating the slide bar and clicks the button 306, the setting value by the slide bar is input to the text box 300 on the setting screen of FIG. Then, by clicking the button 304, the redundancy is set.
[0102]
In this way, in the present embodiment, the document data registration DB 12 in which a plurality of document data are registered, the redundancy setting unit 22 for setting the redundancy, and the redundancy of the extracted document data is the redundancy setting unit 22. The document data extraction unit 20 that extracts document data from the document data registration DB 12 so as to be equal to or less than the redundancy set in the above, and the setting value and document data extraction unit 20 that will be set by the redundancy setting unit 22 A setting support guidance display unit 28 that displays a relationship with the extracted document amount to be extracted.
[0103]
As a result, the user can easily understand how the extracted document amount is affected by his / her setting contents, and can therefore set a relatively appropriate degree of redundancy in accordance with his / her request. In addition, since documents having a high degree of similarity are difficult to select, it is possible to reduce the possibility of extracting documents having similar contents in duplicate. Therefore, as compared with the prior art, it is possible to perform document extraction that is relatively suitable for the user's desire to eliminate duplication of contents.
[0104]
Furthermore, in the present embodiment, the setting support guidance display unit 28 displays the change in the extracted document amount with respect to the change in the set value when the set value is changed within the settable range of the redundancy setting unit 22. It has become.
This makes it easier for the user to understand how the extracted document amount is affected by his / her settings. Therefore, it is possible to perform document extraction that further suits the user's desire for eliminating duplication of contents.
[0105]
Further, in the present embodiment, a setting support information registration DB 26 for registering setting support information indicating the relationship between the setting value and the extracted document amount is provided, and the setting support guidance display unit 28 sets the setting support information in the setting support information registration DB 26. Based on this, the relationship between the set value and the extracted document amount is displayed.
As a result, the extracted document amount that is relatively suitable for the set value is displayed, so that the user can more easily understand how the extracted document amount is affected by the setting content of the user. Therefore, it is possible to perform document extraction that further suits the user's desire for eliminating duplication of contents.
[0106]
Further, in the present embodiment, a setting support information generation unit 24 that generates setting support information indicating the relationship between the setting value and the extracted document amount based on the extraction result of the document data extraction unit 20 is provided, and the setting support guidance display unit 28 displays the relationship between the setting value and the extracted document amount based on the setting support information generated by the setting support information generating unit 24.
As a result, the extracted document amount that is relatively suitable for the set value is displayed, so that the user can more easily understand how the extracted document amount is affected by the setting content of the user. Therefore, it is possible to perform document extraction that further suits the user's desire for eliminating duplication of contents.
[0107]
Furthermore, in the present embodiment, the document data registration DB 12 in which a plurality of document data is registered, the similarity calculation unit 14 that calculates the similarity between the document data of the document data registration DB 12, and the similarity calculation The document data classification unit 16 that hierarchically classifies the document data in the document data registration DB 12 based on the similarity calculated by the unit 14 and the document data group classified by the document data classification unit 16 based on a predetermined classification rule. And a document data extraction unit 20 for extracting document data.
[0108]
As a result, documents having a high degree of similarity are more difficult to be selected, so that the possibility of extracting documents with similar contents in duplicate can be further reduced. Therefore, it is possible to perform document extraction that further suits the user's desire for eliminating duplication of contents. In addition, when extracting a document, since a task such as adding a keyword to each document is not required, the cost required for the task can be reduced as compared with the prior art.
[0109]
Further, in the present embodiment, the document data classification unit 16 calculates the intercluster distance when the document data is classified based on the similarity calculated by the similarity calculation unit 14, and based on the calculated intercluster distance. Thus, the document data in the document data registration DB 12 is hierarchically classified so that the document data having a higher degree of similarity are closer to each other. The document data extraction unit 20 classifies the document data by the document data classification unit 16. In the hierarchical structure, a point at which the redundancy corresponding to the inter-cluster distance below the hierarchy is a predetermined value or less is specified, and document data below the specified point in the hierarchy is extracted from the document data group.
[0110]
Thereby, since each document data can be classified and extracted relatively appropriately according to the degree of similarity, it is possible to further reduce the possibility of redundantly extracting documents having similar contents. Therefore, it is possible to perform document extraction that further suits the user's desire for eliminating duplication of contents.
Further, in the present embodiment, the document data classification unit 16 classifies the document data in the document data registration DB 12 hierarchically by a clustering method or a grouping method.
[0111]
Thereby, since each document data can be classified comparatively appropriately according to the degree of similarity, it is possible to further reduce the possibility of extracting duplicate documents having similar contents. Therefore, it is possible to perform document extraction that further suits the user's desire for eliminating duplication of contents.
Further, in the present embodiment, the similarity calculation unit 14 divides a document of each document data in the document data registration DB 12 into a predetermined character string, and the appearance of a character string divided by the document dividing unit 30. The document vector generation unit 32 that generates a document vector for each document data in the document data registration DB 12 based on the frequency, and the similarity between the document data in the document data registration DB 12 based on the document vector generated by the document vector generation unit 32 And a vector calculation similarity calculation unit 34.
[0112]
Thereby, since the similarity between each document data can be calculated comparatively appropriately, the possibility that duplicate documents having similar contents can be further reduced. Therefore, it is possible to perform document extraction that further suits the user's desire for eliminating duplication of contents.
Further, in the present embodiment, the document dividing unit 30 converts each document data document of the document data registration DB 12 into a predetermined character string using any one of the character string dividing methods of morphological analysis, n-gram, and stop word. It comes to divide.
[0113]
Character string division methods such as morphological analysis, n-gram, and stop word are methods with excellent reliability that have been widely used in the past. By using these as the document dividing unit 30 of the present invention, each document is relatively Of course, it can be appropriately divided into character strings, and any one of these methods can be used to deal with various forms of documents.
[0114]
Furthermore, in the present embodiment, the document vector generation unit 32 determines each document in the document data registration DB 12 based on the appearance frequency of the character string in the document in which the character string appears and the appearance frequency of the character string in a plurality of documents. A document vector weighted by TFIDF is generated for the data.
When generating the document vector of each document, the appearance frequency of the divided character string may be used as it is, but if a known weighting method that reflects the importance of the character string called TFIDF is used, the characteristics of each document are reflected. A document vector can be generated. Therefore, the degree of similarity between the respective document data can be calculated more appropriately, so that the possibility of extracting duplicate documents having similar contents can be further reduced. Therefore, it is possible to perform document extraction that further suits the user's desire for eliminating duplication of contents.
[0115]
Furthermore, in the present embodiment, the vector operation similarity calculation unit 34 calculates the similarity between document data based on the document vector generated by the document vector generation unit 32 by the vector space method.
If the vector space method is used as a method for calculating the similarity between each document data, the similarity between two document vectors can be quantitatively expressed as a cosine value (0 to 1) of an angle formed by the two vectors. This makes it possible to extract documents more accurately.
[0116]
  In the above embodiment, the document data registration DB 12 is an invention.1, 2, 8 to 10The similarity calculation unit 14 corresponds to the document data storage means of the invention.1, 2 or 8The document data classification unit 16 corresponds to the1 or 2It corresponds to the document data classification means. In addition, the document data extraction unit 20 is the invention 12 or 5Compatible with document data extraction meansdo itYes.
[0117]
  Moreover, in the said embodiment, the redundancy setting part 22 is invention.1, 2 or 3Redundancy setting handIn stepsCorrespondingly, SettingThe fixed support information generation unit 245Corresponds to the setting support information generating means. The setting support information registration DB 264The setting support guidance display unit 28 corresponds to the setting support information storage means of the first to third inventions.5Compatible with setting support guidance output meansdo itYes.
[0118]
  In the above embodiment, the document dividing unit 30 is an invention.8Or9The document vector generation unit 32 corresponds to the document dividing unit of the invention.8, 10 or 11The vector calculation similarity calculation unit 34 corresponds to the document vector generation means of8 or 11This corresponds to the vector calculation similarity calculation means.
  In the above-described embodiment, the user preference is reflected only on the degree of document overlap. However, the present invention is not limited to this, and the user preference may be reflected on the content of the document. . Specifically, for example, the following two configurations can be proposed.
[0119]
FIG. 23 shows another embodiment of the document extracting apparatus 10 according to the present invention.
As shown in FIG. 23, the document extracting apparatus 10 selects document data from the document data registration DB 12 based on the user data registration DB 12, the user information registration DB 18, and the user information of the user information registration DB 18. Unit 70, similarity calculation unit 14 for calculating the similarity between the document data selected by document data selection unit 70, and document data selection unit based on the similarity calculated by similarity calculation unit 14 The document data classification unit 16 that classifies the document data selected in 70 hierarchically, the user information registration DB 18 in which user information is registered, and the user in the user information registration DB 18 among the document data group classified in the document data classification unit 16 The document data extraction unit 20 extracts document data based on the information.
[0120]
As a result, after selecting document data having contents suitable for the user's wishes, document data having a high degree of similarity is excluded from the selected document data. Document extraction that emphasizes user preferences can be performed. For example, it is possible to provide suitable information to users who are seeking information that suits their preferences rather than a wide range of information.
[0121]
  In this case, the document data registration DB 12 corresponds to the document data storage means of the invention 11, and the user information registration DB 186The similarity calculation unit 14 corresponds to the user information storage means of the invention.6The document data classification unit 16 corresponds to the6It corresponds to the document data classification means. In addition, the document data selection unit 706Corresponds to the document data selection means.
[0122]
FIG. 24 shows another embodiment of the document extracting apparatus 10 according to the present invention.
As shown in FIG. 24, the document extraction apparatus 10 includes a document data registration DB 12, a user information registration DB 18, and a similarity calculation unit 14 that calculates the similarity between the document data in the document data registration DB 12. The document data classification unit 16 for hierarchically classifying the document data in the document data registration DB 12 based on the similarity calculated by the similarity calculation unit 14, the user information registration DB 18 for registering user information, and the document data classification unit 16 The document data extraction unit 20 that extracts document data based on the user information in the user information registration DB 18 from the document data group classified in the above, and the user information registration DB 18 from the document data group extracted by the document data extraction unit 20. The document data selection unit 70 selects document data based on user information.
[0123]
Thus, after document data having a high degree of similarity is excluded, document data having contents that match the user's wishes are selected from the data, so that bias due to user preferences is reduced and information is more than user preferences. Can be extracted with emphasis on uniformity of the document. For example, it is possible to provide suitable information to a user who is seeking a wider range of information than information suitable for his / her preference.
[0124]
In this case, the user information registration DB 18 corresponds to the user information storage unit of the twelfth aspect, the document data extraction unit 20 corresponds to the document data extraction unit of the twelfth aspect, and the document data selection unit 70 corresponds to the document of the twelfth aspect. It corresponds to the data selection means.
The difference between the configurations A and B will be described in detail with the configuration in FIG. 23 as configuration A and the configuration in FIG. 24 as configuration B.
[0125]
As an attribute for selecting document data of the user A, the selection range is expanded to the “major league” news category and, if there are few, the “sport” which is a higher concept. It is assumed that five news items are selected by the document data selection unit 70. For “major league” and “sports”, it is assumed that an information category is specified using a keyword group, related sentences, and some other known technique, including that the keyword is directly specified. The redundancy exclusion attribute is assumed to be set separately in the same way. However, for the sake of simplicity, the article number in the “Duplicate” column in the following example is duplicated with that article. This indicates that the candidate is excluded by redundancy elimination.
[0126]
25 and 26 are diagrams for explaining the difference between the configuration of FIG. 23 and the configuration of FIG.
In FIG. 25 and FIG. 26, “fitness” is a numerical value obtained as a result of calculating how much each article fits into the information category set by the user A by the document data selection unit 70. In the "Duplicate" column, the result of calculation according to the attribute for redundancy removal specified separately indicates that it is a candidate to be excluded because it overlaps with the article written in that column. Yes. 25 and 26, article 1 overlaps with article 3, indicating that article 1 is a candidate for exclusion.
[0127]
In this example, the difference between the configurations A and B appears whether or not the article 10 exists. This is because, in the example of FIG. 25 (configuration A), the user's preference is emphasized, and article selection more biased to the preference is performed, whereas in the example of FIG. 26 (configuration B), the user preference is selected. It can be said that the selection of articles from which redundancy has been eliminated is made within a range that does not deviate from that while emphasizing.
[0128]
From this result, it can be considered that the example of FIG. 26 in which the amount of information is designated as five is usually preferable while redundancy is eliminated. In the example of FIG. 25, the absolute number of articles is likely to be 5 or less (the possibility that the number is guaranteed is small), whereas in the example of FIG. 26, the number is likely to be guaranteed. However, there is a great possibility that it depends on the determination of the degree of fitness and the determination of duplication.
[0129]
In the above embodiment, the document D₁, D₂And document D₃, D₄The arithmetic mean is used to calculate the redundancy for the combination of three documents including, but this is not restrictive, and the method of calculating the square root by taking the product and setting the value as the redundancy, between each document data It is also possible to adopt a method in which the maximum value or the minimum value among the similarities is used as the redundancy.
[0130]
Similar to the principle that the area becomes larger when the length of “rectangle + width” is constant in the rectangular area, the geometrical average has the following characteristics compared to the arithmetic average.
(A) A combination having a large variation in similarity between document data tends to be small as a calculation result.
(B) A combination having a small variation in similarity between document data tends to be large (not small) as a calculation result.
For example, when the redundancy is calculated using the arithmetic mean for the combination of selecting three documents in the above embodiment, the following is obtained. Where (1) is (D₁, D₃, D₄), And (2) is (D₂, D₃, D₄).
(1) (0.3 + 0.5 + 0.3) / 3 = 0.37
(2) (0.2 + 0.8 + 0.3) / 3 = 0.43
Further, when the redundancy is calculated using the geometric mean for the combination of selecting three documents in the above embodiment, the following is obtained. Where (1) is (D₁, D₃, D₄), And (2) is (D₂, D₃, D₄).
(1) 3√ (0.3 * 0.5 * 0.3) = 0.356
(2) 3√ (0.2 * 0.8 * 0.3) = 0.363
Comparing the results of arithmetic mean (1) (2) and geometric mean (1) (2), the result of (2) is almost the same, whereas the result of geometric mean (1) is It is smaller than the average (1). This is because (2) is a combination in which the degree of similarity between document data varies, and in the arithmetic average, even when the average is pushed up to one high numerical value, the geometric average method is used. If it is used, if the redundancy is partially similar, the result is not affected. Therefore, the arithmetic mean can be said to be a redundancy calculation method for a document group that is linearly affected when a combination having a high degree of similarity between document data is mixed even in part. If this method is used, if the degree of similarity between most of the document data is large, it is determined that the redundancy of the document group is large, but only if the degree of similarity between some of the document data is large (that is, In the case of a document group including a lot of dissimilar documents), it has a feature of determining that the redundancy is low. For both arithmetic and geometric means, if you choose one with low redundancy, then document D₁And select a document that is close to the set redundancy.₂Should be selected.
[0131]
Further, in the above embodiment, when the redundancy is calculated using the minimum value among the similarities between the document data for the combination of selecting three documents, it is as follows. Where (1) is (D₁, D₃, D₄), And (2) is (D₂, D₃, D₄).
(1) min (0.3,0.5,0.3) = 0.3
(2) min (0.2,0.8,0.3) = 0.2
This is a method in which the minimum value is used as the redundancy of the document group among combinations of similarities between document data. It becomes small when there is a combination with at least a small redundancy in a document group. In other words, they are not all similar, but can be said to be an index that decreases when at least some characteristic document is included. This may not be optimal as an index that represents the overall redundancy of documents. However, for example, it is usually useful when only a large amount of redundancy is required and a local index is required so as not to miss a characteristic document that is sometimes mixed in the document group. . In this case, if a document with low redundancy is selected, the document D₂And select a document that is close to the set redundancy.₁Should be selected.
[0132]
In the above embodiment, when the redundancy is calculated using the maximum value among the similarities between the document data for the combination of selecting three documents, the following is performed. Where (1) is (D₁, D₃, D₄), And (2) is (D₂, D₃, D₄).
(1) max (0.3,0.5,0.3) = 0.5
(2) max (0.2,0.8,0.3) = 0.8
This is a method in which the maximum value is set as the redundancy of the document group among combinations of similarities between document data. This becomes large when a combination having at least a high degree of redundancy exists in a document group. In other words, it can be said that the index increases when there is a document that is a little similar. This may not be optimal as an index that represents the overall redundancy of documents. However, this is effective when a local index is required, for example, when it is not desired to select a combination of documents that is at least redundant, or when it is desired to eliminate document redundancy completely below a certain threshold. In this case, if a document with low redundancy is selected, the document D₁And select a document that is close to the set redundancy.₂Should be selected.
[0133]
In the above embodiment, the distance between the document vectors is used. However, the present invention is not limited to this, and the angle (cosine value) between the document vectors may be used.
Further, in the above embodiment, the case where the control program stored in advance in the ROM 52 is executed when executing the processing shown in the flowchart of FIG. 8 is described, but the procedure is not limited to this. The program may be read from the storage medium storing the program into the RAM 54 and executed.
[0134]
Here, the storage medium is a semiconductor storage medium such as RAM or ROM, a magnetic storage type storage medium such as FD or HD, an optical reading type storage medium such as CD, CDV, LD, or DVD, or a magnetic storage type such as MO. / Optical reading type storage media, including any storage media that can be read by a computer regardless of electronic, magnetic, optical, or other reading methods.
[0135]
In the above embodiment, the information extraction system, the document extraction system, the information extraction program and the document extraction program, the information extraction method and the document extraction method according to the present invention are duplicated as shown in FIG. The present invention is applied to a case where a small number of documents are provided to a user. However, the present invention is not limited to this, and can be applied to other cases without departing from the gist of the present invention.
[Brief description of the drawings]
FIG. 1 shows an embodiment of a document extraction apparatus 10 according to the present invention.
2 is a block diagram showing a configuration of a computer 100. FIG.
FIG. 3 is a diagram illustrating an example of character string division by morphological analysis.
FIG. 4 is a diagram illustrating an example of character string division by n-gram.
FIG. 5 is a diagram illustrating an example of character string division by stop words.
FIG. 6 is a diagram showing a character string-document matrix.
FIG. 7 is a diagram illustrating a result of character string division by morphological analysis.
FIG. 8 is a flowchart showing a part of document data extraction processing;
FIG. 9 is a diagram showing document vectors and their correlation.
FIG. 10 is a diagram showing a symmetric matrix between a document and a document.
FIG. 11 is a diagram showing a symmetric matrix between a document and a document.
12 is a graph obtained by converting a symmetric matrix of similarity between documents in FIG. 11 into a distance between clusters.
FIG. 13 Document D₁~ D₄It is a figure for demonstrating the case where the distance between clusters of the cluster which belongs to is calculated.
FIG. 14 Document D₁~ D₄It is a figure for demonstrating the case where the distance between clusters of the cluster which belongs to is calculated.
FIG. 15 is a document D constructed by cluster analysis.₁~ D₄It is a figure which shows the hierarchical structure of.
FIG. 16 is a diagram for explaining a case where document data is extracted from a document data group classified by the document data classification unit 16;
FIG. 17 Document D₁~ D₄It is a figure for demonstrating the case where the document provided to the user B is selected from among these.
FIG. 18 Document D₁~ D₄It is a figure for demonstrating the case where the document provided to the user A is selected from among.
FIG. 19 is a diagram showing a redundancy setting screen.
FIG. 20 is a diagram illustrating an example of a classification result obtained by hierarchically classifying document data by a document data classification unit.
FIG. 21 is a diagram showing a relationship between a setting value and the number of documents in a slide bar format.
FIG. 22 is a diagram showing a relationship between a setting value and the number of documents in a graph format.
FIG. 23 shows another embodiment of the document extracting apparatus 10 according to the present invention.
FIG. 24 shows another embodiment of the document extraction apparatus 10 according to the present invention.
25 is a diagram for explaining a difference between the configuration of FIG. 23 and the configuration of FIG. 24;
26 is a diagram for explaining a difference between the configuration of FIG. 23 and the configuration of FIG. 24;
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 10 ... Document extraction apparatus, 12 ... Document data registration DB, 14 ... Similarity calculation part, 16 ... Document data classification part, 18 ... User information registration DB, 20 ... Document data extraction part, 22 ... Redundancy setting part, 24 ... Setting support information generation unit 26 ... Setting support information registration DB 28 ... Setting support information display unit 30 ... Document division unit 32 ... Document vector generation unit 34 ... Vector calculation similarity calculation unit 50 ... CPU 52 ... ROM, 54 ... RAM, 58 ... I / F, 60 ... input device, 64 ... display device, 70 ... document data selection unit, 100 ... computer, D₁~ D_m... Document, S ... Information supply source, 300 ... Text box, 302 to 306 ... Button

Claims (11)

A system for extracting documents from a plurality of documents to be extracted,
Document data storage means for storing a plurality of document data, redundancy setting means for setting redundancy, and redundancy of the extracted document data is less than or equal to the redundancy set by the redundancy setting means Document data extraction means for extracting document data from the document data storage means, and a relationship between a set value that will be set by the redundancy setting means and the amount of extracted document that is extracted by the document data extraction means Setting support guidance output means for outputting, similarity calculation means for calculating the similarity between the document data of the document data storage means, and the document based on the similarity calculated by the similarity calculation means Document data classification means for hierarchically classifying the document data in the data storage means,
The document data classifying unit is a distance between categories when the document data is classified based on the similarity calculated by the similarity calculating unit, and a cluster having a smaller distance as the similarity between the categories is larger. The inter-cluster distance is calculated, and based on the calculated inter-cluster distance, the document data of the document data storage means is classified hierarchically so that the document data having a higher similarity are closer to each other.
In the dendrogram showing the hierarchical classification result with the inter-cluster distance as the vertical axis and the position of each classified document data as the horizontal axis, the document data extracting means And the inter-cluster distance corresponding to the set redundancy is identified. First, among the document data groups belonging to the cluster connected to the branch extending from each identified intersection to the lower layer side, the identification The document data belonging to the cluster connected to any one of the intersections is extracted, and then, among the specified intersections, the lower side from each intersection other than the intersection connected to the cluster to which the extracted document data belongs among the document data belonging to the cluster to be connected to a branch extending to extract the highest document data similarity between the document data the extracted every intersection Document retrieval system characterized in that it has become. A system for extracting documents from a plurality of documents to be extracted,
Document data storage means for storing a plurality of document data, redundancy setting means for setting redundancy, and redundancy of the extracted document data is less than or equal to the redundancy set by the redundancy setting means Document data extraction means for extracting document data from the document data storage means, and a relationship between a set value that will be set by the redundancy setting means and the amount of extracted document that is extracted by the document data extraction means Setting support guidance output means for outputting, similarity calculation means for calculating the similarity between the document data of the document data storage means, and the document based on the similarity calculated by the similarity calculation means Document data classification means for hierarchically classifying the document data in the data storage means,
The document data classifying unit is a distance between categories when the document data is classified based on the similarity calculated by the similarity calculating unit, and a cluster having a smaller distance as the similarity between the categories is larger. The inter-cluster distance is calculated, and based on the calculated inter-cluster distance, the document data of the document data storage means is classified hierarchically so that the document data having a higher similarity are closer to each other.
In the dendrogram showing the hierarchical classification result with the inter-cluster distance as the vertical axis and the position of each classified document data as the horizontal axis, the document data extracting means The intersection of the branch connecting the two and the inter-cluster distance corresponding to the set redundancy is specified, and the document extracted from the document data group belonging to the cluster connected to the branch extending from the specified intersection to the lower layer side is identified. A document extraction system, wherein document data is extracted one by one at each intersection so that the sum of similarities between data is minimized. In any one of Claim 1 and 2,
The setting support guidance output means outputs a change in the extracted document amount with respect to a change in the setting value when the setting value is changed within a settable range of the redundancy setting means. Feature document extraction system. In claim 3,
And further comprising setting support information storage means for storing setting support information indicating a relationship between the setting value and the extracted document amount,
The setting support guidance output means outputs a relationship between the set value and the extracted document amount based on setting support information in the setting support information storage means. In claim 3,
Furthermore, setting support information generating means for generating setting support information indicating a relationship between the setting value and the extracted document amount based on an extraction result of the document data extracting means,
The setting support guidance output unit outputs a relationship between the setting value and the extracted document amount based on the setting support information generated by the setting support information generation unit. . In any one of Claims 1 thru | or 5,
Furthermore, user information storage means for storing user information about the user, and document data selection means for selecting a plurality of document data from the document data storage means based on the user information of the user information storage means. ,
The similarity calculating unit and the document data classifying unit calculate the similarity and classify the document data for the document data selected by the document data selecting unit. Extraction system. In any one of Claims 1 thru | or 5,
Further, a user information storage unit for storing user information regarding the user, and a document for selecting document data based on the user information of the user information storage unit from among a plurality of document data groups extracted by the document data extraction unit A document extraction system comprising data selection means. In any one of Claims 1 thru | or 7,
The similarity calculation means includes a document dividing means for dividing a document of each document data in the document data storage means into predetermined character elements, and the document data storage based on the appearance frequency of the character elements divided by the document dividing means. A vector calculation means for calculating a similarity between the document data of the document data storage means based on the document vector generated by the document vector generation means and a document vector generation means for generating a document vector for each document data of the means And a document extraction system. In claim 8,
The document dividing unit divides a document of each document data of the document data storage unit into predetermined character elements using any one of morphological analysis, n-gram, and stop word character element division methods. A document extraction system characterized by that. In any one of Claims 8 and 9,
The document vector generation unit is configured to perform TFIDF (for each document data in the document data storage unit based on the appearance frequency of the character element in the document in which the character element appears and the appearance frequency of the character element in the plurality of documents. A document extraction system that generates a document vector weighted by Term Frequency & Inverse Document Frequency. In any one of Claims 8 thru | or 10,
The document extraction system, wherein the vector operation similarity calculation means calculates similarity between document data by a vector space method based on the document vector generated by the document vector generation means.

Images