JP2013222418A - Passage division method, device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve such a problem that in the conventional method, it is difficult to correctly divide a passage when a plurality of passages containing sentences with kindred meaning and similar feature quantity are included in one document.SOLUTION: A passage division device 100, under control of a control unit 101, divides a document input from an input unit 102 into sentence units at a sentence division unit 103. A feature quantity calculation unit 104, with the divided sentence as a query, performs associative retrieval of a document which is stored beforehand in a corpus unit 111 and acquires a document vector. A similarity calculation unit 105 retrieves two document vectors whose similarity becomes maximum, and when the similarity is equal to or larger than a prescribed threshold, a retrieval query generation unit 106 consolidates the two sentences to generate a query as a common element. The feature quantity calculation unit 104 regenerates a document vector by using this query. A feature quantity update unit 107 updates the feature quantity on the basis of its reliability, and connects corresponding sentences sequentially to make a passage while updating the feature quantity.

Description

  The present invention relates to processing of an electronic document, and more particularly to a passage dividing technique for an electronic document.

  In recent years, with the progress of computerization of documents and creation of databases, natural language processing technology has greatly developed. For example, much research has been conducted on automatic summarization of documents and automatic keyword extraction for document retrieval. However, it is often assumed that the documents covered by these technologies are passages, that is, documents that are divided into topics, that is, divided into topical or content and semantic units, or that contain only a single passage. . Therefore, for a document including a plurality of passages, dividing the passages in advance is effective. Conventionally, as such a passage division method, a text segmentation method described in Patent Literature 1 and Patent Literature 2 is known.

JP 2009-15595 A JP 2004-145790 A

  However, in the conventional methods for dividing passages and text segmentation, it is difficult to correctly divide passages when a plurality of passages including sentences having similar meanings, that is, sentences having similar features, are included in one document. As a result, automatic summarization of documents and automatic keyword extraction for document retrieval cannot be efficiently advanced.

  An object of the present invention is to provide a passage dividing method, apparatus, and program for effectively dividing a document including a plurality of passages.

  In order to achieve the above object, the present invention provides a passage dividing method in which a processing unit divides a document into passages. The processing unit divides the document into sentence units, and uses the divided sentences as queries. A related document is extracted from a plurality of stored documents to create a feature quantity, and the two feature quantities in the created feature quantities have a similarity equal to or greater than a predetermined threshold. There is provided a passage dividing method for updating feature amounts using the common elements.

  In order to achieve the above object, according to the present invention, there is provided a passage dividing apparatus for dividing an input document into passages, comprising a processing unit and a storage unit, and the processing unit divides the document into sentence units. Then, using the divided and stored sentences as queries, extracting related documents from a plurality of documents stored in the storage unit in advance, creating feature amounts, and the two similarities of the created feature amounts are Provided is a passage dividing device configured to update a feature amount using a common element of the feature amount that is equal to or greater than a predetermined threshold.

  Furthermore, in order to achieve the above object, in the present invention, there is provided a passage dividing program executed by a processing unit of a passage dividing apparatus that includes a processing unit and a storage unit and divides an input document into passages, The processing unit divides the document into sentence units, uses the divided sentence as a query, extracts related documents from a plurality of documents stored in the storage unit in advance, and uses the extracted related documents to determine the feature amount. Provided is a passage dividing program that is operated so as to update a feature quantity using a common element of the feature quantities, in which two similarities of the created feature quantities are equal to or greater than a predetermined threshold.

  According to the present invention, even when a plurality of passages including sentences having similar meanings, that is, sentences having similar feature quantities, are included in one document, the passages can be correctly divided.

It is a figure which shows one function structure of the passage division | segmentation apparatus of 1st Example. It is a figure which shows one hardware constitutions of the passage division | segmentation apparatus of a 1st Example. It is a figure which shows an example of operation | movement of the passage division | segmentation program based on 1st Example. It is a figure which shows a mode that a sentence is connected according to the similarity of a document vector based on a 1st Example. It is a figure which shows one function structure of the passage division | segmentation apparatus of a 2nd Example. It is a figure which shows an example of operation | movement of the passage division | segmentation program based on 2nd Example. It is a figure for demonstrating an example of a document vector based on each Example. It is a figure for demonstrating an example of a word vector based on each Example.

  Examples of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the examples described below. In this specification, “document” and “document” are synonymous. The “passage” means a unit having a topic or content and semantic unit. Further, a document vector means a vector whose dimension is an accumulated document, and a word vector means a vector whose dimension is all words appearing in all documents. In this specification, the “feature amount” of a sentence quantitatively indicates the meaning of the sentence. For example, a document vector or a word vector will be described as an example.

  The first embodiment is an embodiment of a passage dividing method, apparatus, and program that uses a document vector for similarity calculation and a word vector for similar document search. In this embodiment, the document vector is a vector whose dimensions are all documents included in the corpus unit of the dividing device.

Prior to describing the details of this embodiment, an example of a document vector and a word vector will be described.
FIG. 6 shows an example of a document vector. In FIG. 6, the total number of documents included in the corpus is illustrated as 10. When the documents obtained as a result of the search are 1, 3, 4, and 8, the document vector can be expressed as a document vector 601 shown in FIG. Similarly, when a search score is obtained as a result of the search, the obtained search score can be used to represent a document vector 602 as shown in FIG.

  FIG. 7 shows an example of a word vector. The word vector is a vector whose dimensions are all words appearing in all documents. In the word vector of FIG. 7, the types of words appearing in all documents are exemplified as 10. Then, if the words included in a document are 3, 6, 7, and 8 and the application frequencies are 1, 5, 3, and 9, respectively, the appearance frequency is substituted into the corresponding element, so that A word vector 701 shown in FIG.

  FIG. 1A is a diagram illustrating an example of functional blocks of the passage dividing apparatus according to the first embodiment. FIG. 1B is a diagram illustrating an example of a hardware configuration that implements the passage dividing apparatus according to the first embodiment. 1B includes a central processing unit (CPU) 11 that is a normal processing unit, a memory, a RAM, a ROM, a hard disk drive (HDD), a storage unit 12 such as a storage device, and an input / output unit 13. The communication unit 14 is a network interface, and each of these blocks represents a computer connected to each other by an internal bus 15.

1A, a passage dividing apparatus 100 includes a control unit 101, an input unit 102, a sentence dividing unit 103, a feature amount calculating unit 104, a similarity calculating unit 105, a search query generating unit 106, and a feature amount updating. Unit 107, passage update unit 108, output unit 109, sentence storage unit 110, corpus unit 111, feature amount storage unit 112, passage storage unit 113, and morpheme analysis unit 114. As a premise, it is assumed that the corpus unit 111 stores, for example, a document such as a newspaper article and _SD documents.

  Among them, the input unit 102 and the output unit 109 correspond to the input / output unit 13 and the communication unit 14, and the sentence storage unit 110, the corpus unit 111, the feature amount storage unit 112, and the passage storage unit 113 are stored in the storage unit 12. Supports memory and storage devices. The remaining control unit 101, sentence division unit 103, feature amount calculation unit 104, similarity calculation unit 105, search query generation unit 106, feature amount update unit 107, passage update unit 108, and morpheme analysis unit 114 can be realized by processing of various programs stored in a storage unit such as an operating system (OS) or a ROM in the CPU 11.

The movement of each functional block of the passage dividing apparatus according to the first embodiment shown in FIG.
First, a document to be subjected to passage division is input from the input unit 102 to the apparatus. The sentence dividing unit 103 divides the input document into sentence units by executing a predetermined program of the CPU 11 which is a processing unit, and stores a plurality of sentences as division results in the sentence storage unit 110.

  Similarly, the feature amount calculation unit 104 acquires a related document from the corpus unit 111 using each of the sentences read from the sentence storage unit 110, converts the obtained plurality of related documents into a document vector, and stores the feature amount. Store in the unit 112. That is, the feature amount calculation unit 104 generates a document vector as illustrated in FIG. 6 by substituting a value into a dimension corresponding to the acquired related document.

  The search query generation unit 106 has a function of generating a search query and sending it to the control unit 101.

  When a search query is given via the control unit 101, the feature amount calculation unit 104 acquires a document related to the search query from the sentence storage unit 110, converts the obtained plurality of related documents into a document vector, As a feature value, the feature value is stored in the feature value storage unit 112 and output to the feature value update unit 107 via the control unit 101.

  The similarity calculation unit 105 has a function of reading two document vectors from the feature amount storage unit 112 based on the designation of the control unit 101 and calculating the similarity of the two document vectors. A method for calculating similarity in this embodiment will be described later. Further, the similarity calculation unit 105 determines whether or not the similarity obtained by the calculation is equal to or greater than a predetermined threshold value.

  The search query generation unit 106 reads two document vectors from the feature amount storage unit 112 based on the designation of the control unit 101, and extracts a document group common to the two document vectors from the corpus unit 111. A search query is generated from the extracted common document group and output to the control unit 101. A method for generating this search query will be described later.

The feature amount update unit 107 reads two document vectors V _i and V _j from the feature amount storage unit 112 based on the designation of the control unit 101. Also, one document vector V _k is input from the control unit 101 to the feature amount update unit 107. The reliability is calculated from the three input document vectors V _k , V _i and V _j , and V _k is corrected based on the reliability. This reliability will be described later. Thereafter, V _i and V _j are deleted from the feature amount storage unit 112, and V _k is stored in the feature amount storage unit 112.

  The passage update unit 108 reads out two sentences or passage candidates from the sentence storage unit 110 or the passage storage unit 113 based on the designation of the control unit 101. The read sentence or passage candidate is deleted from the sentence storage unit 110 or the passage storage unit 113, the read sentence or passage candidate is connected, and the connection result is stored in the passage storage unit 113 as a passage candidate. Remember.

  The output unit 109 reads a sentence and a passage candidate from the sentence storage unit 110 and the passage storage unit 113, determines whether the passage is an unknown passage, and outputs a label with a label based on the determination result. Here, the unknown passage refers to a sentence or a passage candidate that cannot be determined as to which passage to connect. A method for determining the unknown passage will be described later.

FIG. 2 is a flowchart showing the operation of the passage dividing program executed by the passage dividing apparatus according to this embodiment. Hereinafter, an example of the operation of the passage dividing program will be described with reference to FIG.
Here, as an example, the case where a document including two passages is input will be described. However, the number of passages in the input document may be two or more, and the subsequent processing is the same. An example of a document including

_{A 1,} a 2 statements contained in the first passage, ..., _{a N,} the statements contained in the second passage _{b 1, b 2, ...,} is defined as _{b M.} Here, N is the number of sentences (natural number) included in the first passage, and M is the number of sentences (natural number) included in the second passage.

First, in step 201, a document is input from the input unit 102.
In step 202, the input document is divided into sentence units by the sentence dividing unit 103 and stored in the sentence storage unit 110.

In step 203, all the sentences a ₁ , a ₂ ,..., A _N , b ₁ , b ₂ ,..., B _M stored in the sentence storage unit 110 are input to the feature quantity calculation unit 104, as described above. Get the document vector. As a method for calculating the document vector, for example, a method using a cosine scale can be cited. The cosine scale is used as one of methods for measuring the similarity between two vectors. The cosine measure of the two vectors Q and P is calculated by Equation 1 below.

本実施例においては、上述の通り、類似するドキュメントの検索に単語ベクトルを用いる。そこで、例えば、コーパス部１１１に記憶された各ドキュメントに対して、含まれる単語の出現頻度を要素とする単語ベクトルＷ_ｉ（０≦ｉ＜Ｓ_Ｄ）を作成しておく。入力された文についても同様に単語ベクトル化し、Ｗ_currentとする。単語ベクトルＷ_currentと、単語ベクトルＷ_ｉ（０≦ｉ＜Ｓ_Ｄ）のコサイン尺度を計算し、得られた類似度が高いドキュメントからＬ番目（Ｌは所定の自然数）までのドキュメントを得て、ドキュメントベクトル化し、特徴量記憶部１１２に蓄積する。

In this embodiment, as described above, word vectors are used for searching for similar documents. Therefore, for example, for each document stored in the corpus unit 111, a word vector W _i (0 ≦ i <S _D ) having the appearance frequency of the included word as an element is created. Similarly, the input sentence is converted into a word vector, and is set as W _current . The cosine measure of the word vector W _current and the word vector W _i (0 ≦ i <S _D ) is calculated, and the obtained documents from the high similarity to the Lth (L is a predetermined natural number) are obtained. The document is vectorized and stored in the feature amount storage unit 112.

  Although the cosine scale is used here as an example of similarity calculation, the similarity may be calculated using other scales. As described in FIGS. 6A and 6B, the value of each element of the document vector may be 1 for the selected document and 0 for the other documents, or the calculated similarity may be used. For example, some weighting may be performed.

Next, in step 204, two document vectors stored in the feature amount storage unit 112 are read out, and the similarity calculation unit 105 is used to find a set of document vectors V _i and V _j having the highest similarity. As a method for calculating the similarity in this case, the above-described cosine scale or the like may be used, or the elements existing in both of the two document vectors, that is, the number of common elements may be used.

  In step 205, the similarity calculation unit 105 determines whether or not the maximum similarity calculated in step 204 is greater than or equal to a preset threshold value. The threshold value may be a fixed value set in advance, or when the similarity is calculated in step 204, the average or variance of the calculated similarity may be calculated and used.

Step 206 and step 207 are performed by the search query generation unit 106. In step 206, if the maximum similarity calculated in step 204 is greater than or equal to the threshold value, the common elements of the document vector sets V _i and V _j are extracted and set as the common elements V _ij of the document vector.

In step 207, a search query is generated from the common element V _ij obtained in step 206. An example of a search query generation method is a method using TFIDF. TFIDF is a kind of weight related to words. TF (Term Frequency) and IDF (Inverse Document Frequency) are respectively expressed by the following equations, and TFIDF is obtained by the product of TF and IDF.

ここでｎ_ｉはドキュメントｄにおける単語ｉの出現回数、｜Ｄ｜は総ドキュメント数、｜｛ｄ：ｔ_ｉ∈ｄ｝｜は単語ｔ_ｉを含むドキュメント数である。本実施例においては、総ドキュメント数Ｄはコーパス部１１１に記憶されている全ドキュメント数に相当する。

Here, n _i is the number of occurrences of the word i in the document d, | D | is the total number of documents, and | {d: t _i εd} | is the number of documents including the word t _i . In the present embodiment, the total document number D corresponds to the total number of documents stored in the corpus unit 111.

The document d, performs morphological analysis by using the morphological analysis unit 114 extracts S _W number of words in order TFIDF is large, a search query for this. In addition to TFIDF, for example, the importance may be determined by the frequency of appearance, the document title may be used as a query, or a search query may be generated by other methods.

In step 208, the search query generated in step 207 is input to the feature amount calculation unit 104 via the control unit 101, and the feature amount calculation unit 104 obtains a new document vector _V′ij .

Subsequently, Step 209 and Step 210 for executing the calculation of the reliability of the newly obtained document vector V ′ _{ij and} the like are executed. These step 209 and step 210 are executed by the feature amount updating unit 107 shown in FIG. First, in step 209, the reliability of the document vector V ′ _ij obtained in step 208 is calculated, and the vector size of the document vector is corrected according to the result.

In the present embodiment, the reliability is an index obtained by quantifying how many elements of the common element V _ij are included in the document vector V ′ _ij . The calculation of reliability, for example, the document vector V _'ij is counting whether include any number of elements of the common elements V _ij pair V _i, V _j of document vectors, and a method of dividing the number of elements common elements V _ij It is done. In addition, when the elements of the common element V _ij are weighted by importance, the reliability may be calculated according to the weighted importance. In any case, when the reliability is lower than a predetermined value, feedback of reliability such as increasing or decreasing the vector size of the obtained document vector V ′ _ij is performed.

In step 210, the document vectors V _i and V _j when the common element V _ij is generated are deleted from the feature amount storage unit 112, and the newly obtained document vector V ′ _ij is stored in the feature amount storage unit 112. .

In step 211, for the passage dividing method of this embodiment, the passage update unit 108 concatenates two sentences or passage candidates corresponding to V _i and V _j . A sentence that has never been linked is stored in the sentence storage unit 110. When the sentence is connected, the sentence before connection is deleted from the sentence storage unit 110. When the passage candidate and the sentence are connected, or when the passage candidates are connected to each other, not only the sentence before connection but also the passage candidate before connection is deleted from the passage storage unit 113. The connected sentence or passage candidate is stored in the passage storage unit 113 as a new passage candidate.

  In the passage dividing method and apparatus of the present embodiment, the target passage is created by repeating step 204 to step 211 in the flow of FIG. In step 205, if the maximum similarity between the two document vectors is less than the predetermined threshold, step 212 is executed to end the creation of the passage.

  Step 212 is a step that is executed by the output unit 109 to determine the unknown passage and output the passage. As an example of a method for determining an unknown passage, there is a method for examining the number of morphemes included in a sentence or a passage candidate. When the number of morphemes contained in a sentence or passage candidate is small, a document vector may not be created properly and connection may be difficult. Therefore, when the number of morphemes included in the remaining sentence or passage candidate is equal to or smaller than a certain threshold in step 21, the output unit 409 outputs the unknown passage with a label, and ends the processing flow.

FIG. 3 is an example schematically showing how sentences are connected according to the similarity of document vectors in this embodiment. The threshold value in step 205 in FIG.
The first similarity calculation result is 301. The highest similarity in the result 301 is the similarity 40 of the set of a ₂ and a ₃ .

Therefore, the process from step 205 to step 211 in FIG. The ligated Results are expressed as _{a 23.} Similarly, b ₁ and b ₂ are selected in the result 302, and a ₁ and a ₂₃ are selected as the set having the highest similarity in the result 303, and the processing from step 205 in FIG. 2 to step 211 in FIG. 2 is performed. Since the threshold is set to 10, there is no set selected in the result 304, and the creation of the passage is completed.

  According to the first embodiment described above in detail, even when a plurality of passages including sentences having similar meanings, that is, sentences having similar feature values, are included in one document, the plurality of passages can be correctly divided. Furthermore, automatic summarization of documents and automatic keyword extraction for document retrieval.

The second embodiment is an embodiment of a passage dividing method, apparatus, and program using a word vector for similarity calculation and a word vector for similar document search.
FIG. 4 is a functional block diagram of the passage dividing apparatus according to the second embodiment. It goes without saying that the hardware configuration of the passage dividing apparatus shown in the figure can also be realized by the computer shown in FIG. 1B as in the apparatus shown in FIG. 1A of the first embodiment.

  The input unit 402, the sentence division unit 403, the passage update unit 408, the output unit 409, the sentence storage unit 410, the feature amount storage unit 412, the passage storage unit 413, and the morpheme analysis unit 414 are described in the first embodiment. Therefore, only the corpus unit 411, the feature amount calculation unit 404, the similarity calculation unit 405, the search query generation unit 406, and the feature amount update unit 407, which are different from the first embodiment, are described. To do. Note that the morpheme analysis unit 414 is connected to the feature amount calculation unit 404.

  For the corpus 411, for example, a collection of documents such as newspaper articles, a thesaurus, or both are used.

  The feature quantity calculation unit 404 performs morpheme analysis on the sentence read from the sentence storage unit 410 using the morpheme analysis unit 414, and converts the sentence into a word vector. When the number of elements of the word vector is not sufficient, a method of increasing the number of elements using the corpus unit 411 is effective. For example, when a thesaurus is used as a corpus, synonyms are searched by using each word obtained from the input sentence as a query, and the resulting synonyms are added to the word vector. When a set of documents is used as the corpus, a word vector extracted from each document in the corpus can be added to the word vector obtained from the input sentence.

  Another example of the method of adding word vector elements is a method of extracting important words from the top several documents using TFIDF or the like and adding them to the word vector. However, the number of elements of the word vector may be sufficient by obtaining and adding words related to the sentence by other methods. Then, the obtained word vector is stored in the feature amount storage unit 412. Also, when a word vector is given from the search query generation unit 406 to the feature amount calculation unit 404 via the control unit 401, the number of elements of the word vector is expanded by the same method and stored in the feature amount storage unit 112. At the same time, the word vector is output to the feature amount updating unit 407 via the control unit 401.

  The similarity calculation unit 405 according to the present embodiment reads two word vectors from the feature amount storage unit 412 based on the designation of the control unit 401, and calculates the similarity between the two word vectors. Examples of the similarity calculation method include the cosine scale described above.

  The search query generation unit 406 of this embodiment reads two word vectors from the feature amount storage unit 412 based on the designation of the control unit 401 and extracts a word group common to the two word vectors from the corpus 411. A word vector is created from the extracted common word group, and is output to the feature amount calculation unit 404 via the control unit 401.

The feature amount update unit 407 reads two word vectors V _i and V _j from the feature amount storage unit 412 based on the designation of the control unit 401. Also, one word vector V _k is input from the control unit 401. The reliability is calculated from the three input word vectors V _k , V _i , and V _j, and the vector size of V _k is corrected based on the reliability. Thereafter, V _i and V _j are deleted from the feature amount storage unit 412, and V _k is stored in the feature amount storage unit 412.

  FIG. 5 is a processing flowchart illustrating the operation of the program according to the second embodiment. In the first embodiment, a document vector is used for similarity calculation. In the second embodiment, as described above, a word vector is used, which differs from that in the first embodiment, but the other operations are the same as in the first embodiment. It is the same.

  According to the second embodiment, even when a plurality of passages including sentences having similar meanings, that is, sentences having similar feature amounts, are included in one document, the passages can be correctly divided.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Moreover, it is possible to add the structure of another Example to the structure of a certain Example. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. In addition, each configuration, function, and the like have been described by exemplifying a case where they are realized by software by executing a program that realizes each function. However, information on programs, tables, files, and the like that realize each function It can be stored not only in memory but also in storage devices such as hard disks and SSDs (Solid State DriVe), or recording media such as IC cards, SD cards, and DVDs, and can be downloaded and installed via a network or the like as necessary. It is also possible to do.

11 CPU
12 storage unit 13 input / output unit 14 communication unit 100, 400 passage division device 101, 401 control unit 102, 402 input unit 103, 403 sentence division unit 104, 404 feature amount calculation unit 105, 405 similarity calculation unit 106, 406 search Query generation unit 107, 407 Feature amount update unit 108, 408 Passage update unit 109, 409 Output unit 110, 410 Sentence storage unit 111, 411 Corpus unit 112, 412 Feature amount storage unit 113, 413 Passage storage unit 114, 414 Morphological analysis Part

Claims (15)

A passage dividing method for dividing a document into passages by a processing unit,
The processor is
Dividing the document into sentence units,
Using the divided sentence as a query, extracting a related document from a plurality of previously stored documents, creating a feature amount,
Updating the feature amount using a common element of the two feature amounts, wherein the similarity between the two feature amounts of the created feature amounts is equal to or greater than a predetermined threshold;
A passage dividing method characterized by the above. The passage dividing method according to claim 1,
The processor is
A document vector is used as the feature amount.
A passage dividing method characterized by the above. The passage dividing method according to claim 2,
The processor is
When the similarity between two document vectors V _i and V _j which are the two feature quantities is equal to or greater than a predetermined threshold, a common element V _ij of the two document vectors V _i and V _j is extracted and a search query is obtained. Generate,
A passage dividing method characterized by the above. The passage dividing method according to claim 3,
The processor is
A new document vector V ′ _ij is obtained using the generated search query.
A passage dividing method characterized by the above. It is the passage division | segmentation method of Claim 4, Comprising:
The processor is
The new document vector V _'ij is the corresponding to the degree that contains the elements of the common elements V _ij, the new document vector V' to modify the vector size _ij,
A passage dividing method characterized by the above. It is the passage division | segmentation method of Claim 4, Comprising:
The processor is
Concatenating the sentence or passage candidate corresponding to the new document vector V ′ _ij into a new passage candidate;
A passage dividing method characterized by the above. The passage dividing method according to claim 1,
The processor is
A word vector is used as the feature amount.
A passage dividing method characterized by the above. It is the passage division | segmentation method of Claim 7, Comprising:
When the similarity between the two word vectors V _i and V _j that are the two feature quantities is equal to or greater than a predetermined threshold, a common element V _ij of the two word vectors V _i and V _j is extracted, and a search query is obtained. Generate
A new word vector V ′ _ij is obtained using the generated search query.
A passage dividing method characterized by the above. The passage dividing method according to claim 8,
The processor is
The new word vector V _'ij is, in response to the degree that contains the elements of the common element V _ij, the new word vector V' to modify the vector size _ij,
A passage dividing method characterized by the above. It is the passage division | segmentation method of Claim 9, Comprising:
The processor is
Concatenating the sentence or passage candidate corresponding to the new word vector V ′ _ij into a new passage candidate;
A passage dividing method characterized by the above. A passage dividing device for dividing an input document into passages,
A processing unit and a storage unit;
The processor is
Dividing the document into sentence units,
Using the divided sentence as a query, extracting a related document from a plurality of documents stored in the storage unit in advance, creating a feature amount,
Updating the feature value using a common element of the feature value, wherein two similarities of the created feature values are equal to or greater than a predetermined threshold;
Passage dividing apparatus characterized by the above. The passage dividing apparatus according to claim 11,
The processor is
As the feature amount, a document vector or a word vector based on the related document is used.
Passage dividing apparatus characterized by the above. The passage dividing apparatus according to claim 12, wherein
The processor is
When the similarity between two document vectors or word vectors V _i and V _j that are the two feature quantities is equal to or greater than a predetermined threshold, the common elements of the two document vectors or word vectors V _i and V _{j Extract} V _ij , generate a search query,
A new document vector or word vector V ′ _ij is obtained using the generated search query,
The new document vector, or word vector V 'is _ij, the common element in response to the degree that contains the elements of V _ij, the new document vector, or word vector V' to modify the vector size _ij,
Passage dividing apparatus characterized by the above. The passage dividing apparatus according to claim 13,
The processor is
Concatenating the sentence or passage candidate corresponding to the new document vector V ′ _ij and storing the newly connected passage candidate in the storage unit;
Passage dividing apparatus characterized by the above. A passage dividing program that includes a processing unit and a storage unit, and that is executed by a processing unit of a passage dividing device that divides a passage of an input document,
The processing unit is
Dividing the document into sentence units,
Using the divided sentences as queries, extracting related documents from a plurality of documents stored in the storage unit in advance,
Create a feature using the extracted related document,
Updating the feature value using a common element of the feature value, wherein two similarities of the created feature values are equal to or greater than a predetermined threshold;
Make it work,
A passage dividing program characterized by that.

Images