JP2015203961A - document extraction system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable extraction of a document depending on how many descriptions related to a concept that a conditional sentence owns are included in the document.SOLUTION: A document extraction method includes, in sequence, a partial score calculation procedure (S113) in which each document is divided into a plurality of predetermined segments when a conditional sentence is acquired and similarity to the conditional sentence is quantified for each segment to be a score of the segment, and an extraction procedure (S115) in which a document score is calculated representing similarity between the relevant document and the conditional sentence using the score of each segment constituting one sentence, and a document having a close concept to the conditional sentence is selected from among a plurality of documents using the document score.

Description

  The present invention relates to a document extraction system for extracting a document including a large number of description parts similar to a search condition from a plurality of documents.

  There has been proposed a system for searching a document having a content close to a conditional sentence from a large number of documents (see, for example, Patent Document 1). As the method, a method using a keyword and a method using a feature vector based on the appearance frequency of words are representative. In both cases, the degree of similarity with a conditional sentence is generally converted into a numerical value by some algorithm and extracted in descending order of score value.

JP 2013-30089 A

  However, no matter which method is used, if the entire document is targeted, if the content close to the conditional statement is described in only a part of the document, even if the content is very close to the conditional statement, it will not be searched There is a case. This is because the size of the document ranges from a wide and large document such as a book to a small document with a short and straightforward description such as a dictionary item. For example, when the score is simply calculated, in the case of a document such as a book, even if the description is significantly different from the conditional sentence, the score value becomes high due to the large amount of sentences. In order to prevent this, there is a method of dividing the score by the sentence amount. As a result, it is possible to prevent a document having a large amount of text from the conditional sentence and from being extracted. However, there arises a new problem that a document having a large amount of text and partly including a content very close to a conditional sentence cannot be extracted.

  In order to solve the above problems and extract documents that have a description close to a conditional sentence, the inventor divides the document into formal parts (hereinafter referred to as document segments) and calculates a score for each segment. Then, it has been found that it is effective to extract a valid document as a score value of the entire document by giving a constant weight to each segment.

Specifically, the document extraction method according to the present invention includes:
A document extraction method for extracting a document close to the concept of a conditional sentence from a plurality of documents,
When the partial score calculation unit obtains the conditional sentence, the partial score is divided into a plurality of predetermined segments, and the degree of similarity with the conditional sentence is quantified for each segment to obtain the score of the segment Calculation procedure,
The extraction unit calculates the document score representing the similarity between the document and the conditional sentence after weighting based on a certain algorithm using the score of each segment constituting one document, An extraction procedure for selecting a document having a concept similar to the conditional sentence from the plurality of documents using a document score;
In order.

  In the document extraction method according to the present invention, the mapping unit generates a document feature vector of the document by combining the feature vectors of the segments using the weights of the segments of the document extracted in the extraction procedure, and the document A map that calculates the similarity between documents in the document group extracted in the extraction procedure using a feature vector, and arranges the documents extracted in the extraction procedure on the map according to the similarity between the documents A conversion procedure may be further provided after the extraction procedure.

  In the document extraction method according to the present invention, in the extraction procedure, the maximum value of the score included in one document is calculated for each document, and a predetermined range having a high maximum value is selected from the plurality of documents. A document may be extracted.

  In the document extraction method according to the present invention, in the extraction procedure, a comparison value is calculated for each document using a score in a high range of a predetermined score among the scores included in one document, A document in a predetermined range having a high comparison value may be extracted from the documents.

Specifically, the document extraction system according to the present invention is:
A document extraction system that extracts a document close to the concept of a conditional sentence from a plurality of documents,
When the conditional sentence is acquired, each document is divided into a plurality of predetermined segments, and a partial score calculation unit that quantifies the similarity with the conditional sentence for each segment and sets the score of the segment;
A document score representing the similarity between the document and the conditional sentence is calculated using the score of each segment constituting one document, and the conditional sentence and the concept are calculated from the plurality of documents using the document score. An extractor that selects documents close to each other,
Is provided.

  In the document extraction system according to the present invention, a feature vector of each segment is generated by using the weight of each segment of the document extracted by the extraction unit to generate a document feature vector of the document, and the document feature vector is used. A mapping unit for calculating a similarity between documents in the document group extracted by the extraction unit and arranging the documents extracted by the extraction unit on a map according to the similarity between the documents; It may be.

  In the document extraction system according to the present invention, the extraction unit calculates a maximum value of the score included in one document for each document, and has a predetermined range in which the maximum value is high among the plurality of documents. A document may be extracted.

  In the document extraction system according to the present invention, the extraction unit calculates a comparison value for each document using a score in a high range of a predetermined score among the scores included in one document, and A document in a predetermined range having a high comparison value may be extracted from the documents.

  The above inventions can be combined as much as possible.

  According to the present invention, it is possible to extract a document depending on how much description related to the concept of the conditional sentence is included in the document.

The structural example of the partial information extraction system which concerns on Embodiment 1 is shown. The sequence of the partial information extraction system which concerns on Embodiment 1 is shown. An example of a method for calculating a score P _k using a vector space model will be described. An example of a method for calculating the document score X _i will be described. An example of a method for calculating the score P _k using the appearance frequency of the keyword will be shown. The structural example of the partial information extraction system which concerns on Embodiment 2 is shown. The sequence of the partial information extraction system which concerns on Embodiment 2 is shown. An example of a map is shown.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, this invention is not limited to embodiment shown below. These embodiments are merely examples, and the present invention can be implemented in various modifications and improvements based on the knowledge of those skilled in the art. In the present specification and drawings, the same reference numerals denote the same components.

(Embodiment 1)
FIG. 1 shows a configuration example of a document extraction system according to the present embodiment. The document extraction system according to this embodiment includes a server 10, a storage 20, and a user terminal 30. The storage 20 is an arbitrary storage medium accessible from the server 10. The server 10 and the user terminal 30 are computers having computer resources such as a CPU (Central Processing Unit) and a storage medium, and a program is installed in the storage medium. Any number of servers 10, storages 20, and user terminals 30 may be employed. In the present embodiment, a case where there is one server 10, two storages 20, and one user terminal 30 will be described.

  The storage 20 holds a document. The document includes arbitrary data transmitted / received via a communication network, and includes, for example, text, numerical data, log data, and customer information. Examples of sentences include patents, papers, books, reports, and homepages. Examples of the numerical data include sensor data, measurement data, and POS (Point Of Sales) data. Examples of the log data include online access data and status data of various devices. In the present embodiment, a case where a document is a sentence will be described as an example.

  FIG. 2 shows a sequence of the partial information extraction system according to this embodiment. The server 10 generates a feature vector for each segment before step S101 for acquiring a conditional statement from the user terminal 30. For example, the server 10 acquires a document from the storage 20, divides the acquired document into a plurality of predetermined segments, and generates a feature vector based on the vector space model based on the index for each segment. The feature vector of each segment is preferably stored in the storage 20 and used for the subsequent calculation of similarity. At this time, the feature vector of each segment is preferably stored in the secondary storage 20 separately from the original information group. The original information group is not used at all in the calculation stage, and is used only when displaying the original information in the final stage.

  The user terminal 30 transmits a conditional statement via the communication network (S101). Upon receiving the conditional statement from the user terminal 30, the server 10 acquires the feature vector of each segment from the storage 20 (S102), calculates the score of the acquired segment (S113), and uses the segment score to determine the document and the condition. A document score representing the degree of similarity with the sentence is calculated, a document having a concept similar to the conditional sentence is extracted using the document score (S115), and the extraction result is transmitted to the user terminal 30 (S116). The user terminal 30 displays the extraction result received from the server 10 (S117).

  The server 10 includes a communication function unit (not shown) that transmits and receives information to and from the user terminal 30 and the storage 20 via a communication network, and a configuration for extracting a document. The configuration for extracting a document includes, for example, a conditional sentence acquisition unit 11, a partial score calculation unit 15, and an extraction unit 17. The server 10 may be realized by causing a computer to function as the conditional sentence acquisition unit 11, the partial score calculation unit 15, and the extraction unit 17. In this case, each configuration is realized by the CPU in the server 10 executing a computer program stored in a storage unit (not shown).

  The server 10 executes the document extraction method according to the present embodiment when extracting a document. The document extraction method according to this embodiment includes a partial score calculation procedure (S113) and an extraction procedure (S115) in order.

  In the partial score calculation procedure (S113), when the partial score calculation unit 15 acquires the conditional sentence, the similarity with the conditional sentence is digitized for each segment constituting each part of the document to obtain the score of the segment. When the document includes a sentence, the segment is, for example, a paragraph or a sentence. In the case of paragraphs, for example, paragraph units are identified by detecting line breaks. In the case of a sentence, a unit sentence is identified by detecting a punctuation mark “.” Or “.”, A question mark “?”, And an exclamation mark “!”. In the present embodiment, as an example, a case where a segment is a paragraph will be described.

In the partial score calculation procedure (S113), the partial score calculation unit 15 calculates a feature vector P _i representing the concept of the segment for each segment based on the vector space model. For example, the vector determination unit 12 vectorizes the conditional statement d _k and the segment d _i based on the vector space model, and calculates the conditional statement vector and the feature vector. Here, the feature vector P _i representing the concept of the segment may be calculated in advance before acquiring the conditional sentence. Thus, by making the segment feature vector P _i independent of the conditional statement, the load on the server 10 can be reduced and the extraction result can be promptly provided to the user terminal 30.

When the information d _i can be expressed in matrix with respect to the element t _j , the information d _i can be described by a vector space model d _i = (t ₁ , t ₂ , t ₃ ,...). For this reason, the conditional sentence can be described by a conditional sentence vector whose elements are words included in the conditional sentence. A segment can also be described by a feature vector whose elements are words included in the segment.

When the frequency of occurrence of elements _{t j} appearing in segment _{d i} and _{n ij,} segment _{d i} concept vector _{d i = (n i1, n} i2, n i3, ......) can be represented by. For example, the word _t 1 in the segment _{d 1,} application number of t 2, _{t 3} are 0,1,0 respectively, the words _t 1 in the segment _{d 2, t 2,} t ₃ of the application number, respectively 2,1, 0, if the applicant number of words _{t 1,} t 2, _{t 3} in the segment _{d 3} is 1, 2, and 3, respectively, the matrix M of the segment is expressed as follows.

In the partial score calculation procedure (S113), the partial score calculation unit 15 quantifies the similarity between the feature vector and the conditional sentence vector representing the concept of the conditional sentence for each segment to obtain the score of the segment. For example, the closeness of the contents of the segment d _i and the conditional statement d _k can be quantified by calculating the feature vector d _i and the conditional statement vector d _k . The calculation used for digitization may be a distance between vectors, or an arbitrary calculation such as inner product or outer product.

  Here, words commonly used for any segment do not affect the closeness of the content of the document. Therefore, in calculating the vector, it is preferable to provide a difference in the contribution of each word characteristic to each document to the vector of the other words. For example, weighting is performed using a tfidf (Term Frequency Inverse Document Frequency) method. Thereby, the precision of the closeness of the content of a segment can be improved. The word weight tfidf used in the same manner in any document is small, and the tfidf having a large frequency used by the document has a large tfidf.

In the extraction procedure (S115), extraction unit 17 calculates the document score X _m representing the similarity between the document and the conditional statement using the scores P _i constituting one document m. Calculation of document score X _m is arbitrary, it is preferable the user terminal 30 can be set. For example, when calculated based the similarity between conditional sentence and each paragraph of the document vector space model, the distribution of scores P _i in the document m, as shown in FIG. 3, the distribution function for the exact match and 1. Therefore, the score the highest density score P _i may be used as the document score X _m of P _i, to the average value of the score P _i may document score X _m, the maximum value document score X _m of the score P _i Alternatively, a comparison value using a score P _i in a range having a predetermined high score among the scores P _i may be used as the document score X _m . The comparison value is, for example, is the sum of the scores of the top 3 paragraphs of high score P _i paragraph. The upper three upper paragraphs are preferably settable by the user terminal 30.

The score value P _m of the document m is preferably calculated after weighting based on a certain algorithm. For example, let S _i be the score value for the conditional sentence of segment i of document m. The score value P _m of the document m is P _m = ΣA _i * S _i (1)
It is expressed. A _i is a weighting coefficient of segment i.

There are various ways of thinking about A _i in equation (1). As specific examples, four cases will be described here.
(1) Use peak value Set A _i = 1 only for the segment with the highest score value, and A _i = 0 for the other segments.
(2) Select the top three segments The scores of the three segments are totaled in descending order of score values to obtain the document score.
(3) Select segments with a certain score value or higher Total the scores of the target segments to obtain the document score.
(4) Weighting is halved in order from the segment with the highest score value.
A _i k ₁ = 1, A _i k ₂ = 1/2, A _i k ₃ = 1/4,. .

In order to avoid the influence of the segment text amount, the total text amount of the extracted segments may be normalized. That is, the total score may be divided and compared by the total sentence amount of the target segment.
By the above method, it is possible to extract a document including contents close to a conditional sentence regardless of the document amount.

4 shows an example of the distribution D (X) of the document score _{X m.} Since the document score P _m is a distribution function with a perfect match of 1, the distribution D (X) is also a distribution function of 0 to 1. Extracting unit 17, using the document score X _m, selects the document close to that of conditional statements and concepts from a plurality of documents. The server 10 transmits the selected document to the user terminal 30. At this time, only the selected segment may be transmitted to the user terminal 30. As a result, the user terminal 30 can provide the user terminal 30 with a document that includes many parts that are similar in concept to the conditional sentence.

The document selection method is arbitrary, and it is preferable that the user terminal 30 can be set. For example, a predetermined number and ratio of documents are extracted. When extracting the top 10% documents having a high degree of similarity, a document m that satisfies the integral value of D (X> X ₀ ) = 0.1 as indicated by the oblique lines is extracted. It is preferable that the user terminal 30 can set the number and ratio to extract.

In the partial score calculation procedure (S113), the determination of the closeness of the contents may be performed based on, for example, the presence or absence of a word included in the conditional sentence. When a single word is included in the conditional sentence, a determination is made based on a binary value indicating whether a word is included in each segment. For example, consider the case of two words “rare earth” and “magnet” as evaluation conditions. In the case of the description relating to the rare earth magnet, as shown in FIG. 5, the score P _i of the paragraph including both words is set to 1, and the other scores P _i are set to 0. A total score of scores of all paragraphs in the document m is set as a document score X _i .

  In the present embodiment, an example in which the document is a sentence has been described, but the document in the present invention is not limited to this. When the document includes numerical data or log data, the segment is, for example, time or time, region or place, or attribution. If the document includes customer data, the segment is, for example, time or time, region or location, attribution, or age. The unit of time is arbitrary, for example, it may be a second unit or a year unit.

When the document includes numerical data or log data, vectorization based on the vector space model is performed as follows.
When the document is the access log data of the user in the online service, the number of accesses of the user t _j between the times d _{i to} d _i + T (time interval T) is n _ij . The time d _i can be expressed as a vector d _i = (n _i1 , n _i2 , n _i3 ,...).
If the document is sensor data, between time _d i _~d i + T (time interval T), the output value of the sensor _{t j} and _{n ij.} The time d _i can be expressed as a vector d _i = (n _i1 , n _i2 , n _i3 ,...).
When the document is image data, the image d _i is subjected to frequency conversion, and the numerical value of the component t _j of each frequency after conversion is set to n _ij . The time d _i can be expressed as a vector d _i = (n _i1 , n _i2 , n _i3 ,...).

When the document includes numerical data or log data, the weighting tfidf is performed as follows.
When the document is the access log data of the user in the online service, the weight tfidf of the user who accesses on average is small, and the weight tfidf of the user having large access unevenness is large.
When the document is sensor data, the sensor weight tfidf for which the output numerical value does not change much decreases, and the sensor weight tfidf for which the output numerical value changes greatly increases.
When the document is image data, the frequency weighting tfidf with a small component value variation between images is small, and the frequency weighting tfidf with a large component value variation between images is large.

(Embodiment 2)
FIG. 6 shows a configuration example of the partial information extraction system according to this embodiment. The partial information extraction system according to the present embodiment further includes a mapping unit 14 in addition to the configuration of the first embodiment.

  FIG. 7 shows a sequence of the partial information extraction system according to this embodiment. The partial information extraction method according to the present embodiment further includes a mapping procedure (S126) after the extraction procedure (S115) described in the first embodiment. The server 10 transmits the map created by the mapping procedure to the user terminal 30 (S127). The user terminal 30 displays the map received from the server 10 (S128).

In the mapping procedure (S126), the mapping unit 14 places the points represented by the feature vector P _i and the conditional statement vector P _k on the map according to the closeness of the concept for the document extracted by the extraction unit 17. Deploy.

Here, the feature vector P _m of the document m is obtained by synthesizing the segment feature vectors P _i . In the synthesis, the segment weighting factor A _i is taken into account. For example, the feature vector P _m of the document m is P _m = ΣA _i * P _i (2)
It is expressed. To calculate the similarity between documents mutually in document set extracted in the extraction procedure using the document feature vector P _m.

When the content of the segment is close to the conditional sentence, the types of words used are similar, and the points indicated by the vectors are arranged close to each other. Therefore, the closeness between the feature vectors and the conditional sentence vectors is calculated, and mapping based on the closeness of the contents between information, that is, the “semantic distance” is performed based on the closeness between the vectors. The calculation may be a distance between vectors, or an arbitrary calculation such as inner product or outer product. Based on the proximity of the contents between the information obtained d _i each other can be created map as shown in FIG. 8 using the mapping algorithm.

  The system according to the present embodiment can extract segments using concept search, and map the distribution of the contents of each segment using a vector calculated using concept search. For this reason, it is possible to display in a state in which the word in the conditional sentence is close to the segment.

  The present invention can be applied to the information communication industry.

10: server 11: conditional sentence acquisition unit 15: partial score calculation unit 17: extraction unit 14: mapping unit 20: storage 30: user terminal

Claims (8)

A document extraction method for extracting a document close to the concept of a conditional sentence from a plurality of documents,
When the partial score calculation unit obtains the conditional sentence, a partial score calculation procedure that quantifies the similarity with the conditional sentence for each segment constituting each part of the document and sets the score of the segment;
The extraction unit calculates the document score representing the similarity between the document and the conditional sentence after weighting based on a certain algorithm using the score of each segment constituting one document, An extraction procedure for selecting a document having a concept similar to the conditional sentence from the plurality of documents using a document score;
Document extraction method having in order.   The mapping unit synthesizes the feature vectors of the segments using the weights of the segments of the document extracted in the extraction procedure to generate the document feature vectors of the document, and uses the document feature vectors in the extraction procedure. A mapping procedure for calculating the similarity between the documents in the extracted document group and arranging the document extracted in the extraction procedure on the map according to the similarity between the documents is performed after the extraction procedure. The document extracting method according to claim 1, further comprising: In the extraction procedure, the highest value of the score included in one document is calculated for each document, and a document in a predetermined range having the highest highest value is extracted from the plurality of documents.
The document extraction method according to claim 1 or 2. In the extraction procedure, a comparison value is calculated for each document using a score in a range of a predetermined score among the scores included in one document, and the comparison value is high among the plurality of documents. Extract documents in a predetermined range,
The document extraction method according to claim 1 or 2. A document extraction system that extracts a document close to the concept of a conditional sentence from a plurality of documents,
When the conditional sentence is acquired, a similarity score with the conditional sentence is converted into a numerical value for each segment constituting each part of the document, and a partial score calculation unit that sets the score of the segment;
A document score representing the similarity between the document and the conditional sentence is calculated using the score of each segment constituting one document, and the conditional sentence and the concept are calculated from the plurality of documents using the document score. An extractor that selects documents close to each other,
A document extraction system comprising: A feature vector of each segment is generated by using the weight of each segment of the document extracted by the extraction unit to generate a document feature vector of the document, and the document feature vector is used to generate a document feature vector of the document. A mapping unit that calculates the degree of similarity between the documents, and places the extracted document on the map according to the degree of similarity between the documents,
The document extraction system according to claim 5, further comprising: The extraction unit calculates the highest value of the score included in one document for each document, and extracts a document in a predetermined range having the highest highest value from the plurality of documents.
The document extraction system according to claim 5 or 6. The extraction unit calculates a comparison value for each document using a score in a range of a predetermined high score among the scores included in one document, and the comparison value is high among the plurality of documents. Extract documents in a predetermined range,
The document extraction system according to claim 5 or 6.

Images