JPH09114850A - Document processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically divide an input document corresponding to the change of a contex or a topic by providing a word life computing element obtaining information on a word life in which the word operates an asserting function in a document based on the appearing position of an optical word in the input document so as to divide the input document by referring to life information on this word. SOLUTION: When an optical input document 1 is inputted to the life computing element 2, the input document 1 is divided into sentences being computing units. After being divided, the respective sentences are sent to a computing object recognizing means 21 in order to recognize the computing objects. Next, after obtaining the potential of the word through the use of a potential computing means 22, the life of the word is estimated through the use of a life estimating means 23. Namely the potential distribution of the word, the life of the word, etc., are calculated from the potential of each word in a life table 3 and the result is stored in the slot of the life table 3. After then the life computing element 2 starts a document dividing filter 4 and a feature word extraction filter 5.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document processing apparatus for processing a document described in any natural language, and more particularly,
It can function as a document classification device and a characteristic word extraction device.

[0002]

2. Description of the Related Art In recent years, computer networks and C
Documents described in a large-scale computerized natural language have come to be provided at low cost via a computerized medium represented by a D-ROM or the like. Along with this, the opportunities for organizations and individuals to handle electronic documents are increasing, and how to effectively utilize these electronic documents is becoming a serious issue.

In order to handle a large amount of digitized documents, the categories of the documents are identified and categorized, which is necessary in consideration of retrieval of desired documents. An example of a device having such a function is a document processing device (automatic document adding device) disclosed in Japanese Patent Laid-Open No. 6-75995.

This document processing apparatus, as shown in FIG.
It is realized by an information processing device such as a workstation including a processing device 100, an input device 101, an output device 102, an internal memory 103, an external memory 104, and the like. Basic data for classification is created based on the data about the classification and the keyword, and the basic data is used to add the classification suitable for the description content to the unclassified document.

Here, as the basic data for the classification giving process, the classification distance table shown in FIG. 3 showing the strength of the relationship between the two classifications, and the classification deeply related to each keyword and the classification distance table shown in FIG. The keyword / classification table shown in FIG. 4 in which the degree of the depth of the classification is stored is used.

When a plurality of keywords included in the uncategorized document are input to this document processing device, the keywords / keyword /
With reference to the classification table, calculate the total value of the degree of association of the classifications related to the input keyword for each classification, determine the classification candidates to be assigned according to the order of the size of this total value, and then classify By referring to the distance table, it is checked whether the determined distances between the plurality of candidate classifications are within a proper range, and the final classification (not necessarily one) of the unclassified document is determined. .

[0007]

However, the conventional document processing apparatus has the following problems.

(1) It is necessary to prepare in advance a "keyword / classification table" for matching keywords and classifications, and a "classification distance table" indicating the strength of the relationship between classifications. However, if the data covering most of the classification of the target document is not stored, the classification accuracy cannot be obtained to some extent. In particular, unless the number of keywords in the keyword / classification table is large, it is not possible to secure a certain level of classification accuracy. In order to secure a certain degree of classification accuracy in this way, the amount of data increases and a large-capacity external memory is required.

(2) The document to be classified is "multi-discipline"
Documents (eg, economics, politics, life,
In the case of one-day newspaper articles including texts in fields such as current affairs), it is difficult to set consistent keywords for the entire target uncategorized document. In such a situation where it is difficult to set a keyword, if the keyword is manually set, there is a possibility that individual differences may occur in the setting due to knowledge, experience, etc., and the accuracy of classification may be adversely affected. .

(3) Even if the final classification of the compound document across the composite fields can be determined, it is expected that the result is merely a list of many classifications. Therefore, even if the classification is added to the document, the effect of using the added classification becomes small.

[0011]

The document processing apparatus according to the first aspect of the present invention includes (1) an input unit for inputting a document described in natural language, and (2) an input position of an arbitrary word in the input document. Based on the word lifespan calculator, which obtains the lifespan information of the word on which the word acts the assertion function, and (3) the word lifespan information that stores the lifespan information of the word obtained by this word lifespan calculator It is characterized by comprising storage means and (4) document dividing means for dividing the input document by referring to the life information of words stored in the word life information storage means.

In the document processing apparatus of the first invention,
By dividing the input document by referring to the word life information, it is possible to divide the input document according to the change of context or topic.

The document processing apparatus according to the second aspect of the present invention comprises: (1) input means for inputting a document described in natural language; and (2) the word based on the appearance position of an arbitrary word in the input document. A word lifespan calculator that obtains the lifespan information of a word that has an asserting function in the document, and (3) a word lifespan information storage unit that stores the lifespan information of the word obtained by this word lifespan calculator,
(4) Characteristic word extracting means for extracting a characteristic word serving as an index of the input document by referring to the word life information stored in the word life information storage means.

In the document processing apparatus of the second invention,
By extracting the characteristic word in the input document by referring to the life information of the word, it may be unnecessary for the user to specify the characteristic word of the input document.

According to a third aspect of the document processing apparatus of the present invention, (1) input means for inputting a document described in natural language, and (2) the word based on the appearance position of an arbitrary word in the input document. A word lifespan calculator that obtains the lifespan information of a word that has an asserting function in the document, and (3) a word lifespan information storage unit that stores the lifespan information of the word obtained by this word lifespan calculator,
(4) Document division means for dividing the input document by referring to the word lifespan information stored in the word lifespan information storage means, and (5) Word lifespan information stored in the word lifespan information storage means. With reference to the above-mentioned input document and / or the divided document, a characteristic word extracting means for extracting a characteristic word serving as an index is provided.

In the document processing apparatus of the third invention,
By dividing the input document by referring to the word life information, it is possible to divide the input document according to the change of context or topic, and also by extracting the characteristic word in the input document by referring to the word life information. This can eliminate the need for the user to specify the characteristic word of the input document.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION Next, an embodiment of a document processing apparatus according to the present invention will be described with reference to the drawings.

First, before specifically describing this embodiment, the concept of basic document processing (document classification) followed by this embodiment will be described, and main terms will be defined through such description.

In order to accurately classify the input document, it is necessary to accurately define the classification characteristics of the document. One way to define the classification feature is to represent the purpose of the document by some character strings in the sentence (hereinafter referred to as words) and use such words as an index of the document classification. Also in this embodiment, the method of defining the classification feature is adopted. Below, the word used as the index of document classification is called a characteristic word. Since the selection of the characteristic word greatly affects the accuracy of the classification result, the method of selecting and extracting the characteristic word from the input document becomes very important.

In this embodiment, the following concept
According to (1) to (7), characteristic words are extracted and the documents are classified.

(1) Words having the same character string that appear multiple times in an arbitrary document (hereinafter referred to as the same word) represent the concept of maintaining consistency in the document. Therefore, it may be a characteristic word.

(2) The appearance of the same word multiple times in a document means that the word (arguing something) is intended from the first appearance position to the last appearance position of the word. Can be seen as working.

(3) If the intention of the same word exceeds the range in which the intention works, the same word can naturally be considered to have a life.

(4) One identical word has its appearance position and
From the relationship with the appearance position of the same word adjacent to each other, it is possible to determine the influence range (hereinafter, referred to as word potential) due to the current appearance of the word. Since the arrangement (appearance) of the same word in the document faithfully reflects the context expressed by the document, the potential of the word is also influenced by the context.

(5) Considering in this way, generally speaking, it can be said that a word having a larger distance from the same word adjacent thereto has a larger potential. However, in a compound document, there is a context change (hereinafter referred to as a gap), and the same word may appear in different contexts. Therefore, a large distance from the adjacent same word does not necessarily mean that the word is adjacent to the same word. It does not mean that the potential is large.

(6) Therefore, the word life is defined as the sum of the potentials of a plurality of effective words (words that are considered to belong to the same context) in the same word that appear multiple times.

(7) A word having a long life is extracted as a characteristic word representing the characteristic of the document, and the document is classified based on the extracted word. Also, by observing the distribution of the lifespan of multiple types of words, we determined the gaps in the compound document and made it possible to assign classifications to each divided document part.

The document processing apparatus of the embodiment according to the above concept is the processing apparatus 10 as shown in FIG.
0, input device 101, output device 102, internal memory 10
Although it is realized by an information processing device such as a workstation including the external memory 3 and the external memory 104, it functionally has the configuration shown in FIG.

The configuration of the information processing apparatus will be briefly described below in relation to the embodiment. The input device 101 is a keyboard, a mouse, a CD-ROM reading device or the like for inputting an uncategorized document. Output device 102
Mainly outputs the classified results, and is a CRT, printer, or the like. The internal memory 103 stores a processing program in the processing device 100 as a processing unit and data therefor, and the external memory 15 stores input document data. The processing device 100 mainly executes the document classification operation while appropriately accessing the peripheral devices 101, 102 and 104.

In FIG. 1, the document processing apparatus according to this embodiment is functionally provided with an input / output interface 7, an input document (storage unit) 1, a life calculator 2, a life table 3, a document division filter 4, and features. It is composed of a word extraction filter 5, a document segment (storage unit) 61, and a document index (storage unit) 62. The document segment (storage unit) 61 and the document index (storage unit) 62 form an output result (storage unit) 6.

The input / output interface 7 takes in the input document 1 described in natural language from an input device such as a keyboard, and outputs the classification result and processing status to an output device such as a display.

If the text format of the input document 1 is a so-called natural language format, the content of the document may be a compound document that covers a complicated field. Also,
It is not restricted by the type of language such as English or Japanese.

The life calculator 2 calculates the life etc. of an arbitrary word in the input document 1. Specifically, the life calculator 2 is composed of a calculation target recognition means 21, a potential calculation means 22 and a life estimation means 23. . The calculation target recognition means 21 is
It recognizes the position of the word that is the calculation target. The potential calculator 22 calculates the potential of the word to be calculated. The life estimation means 23 estimates the life of the word to be calculated from the calculated potential.

The life table (word life information storage means) 3 stores the estimated life etc. of a word having an arbitrary character string in the input document 1, and has a structure as shown in FIG. 5, for example. Unit (hereinafter referred to as Life Table Unit) 20
It is equipped with multiple 0s. In FIG. 5, one unit 200 of the life table 3 is for any one word (word having a certain heading) of the input document 1.

Each life table unit 200 has heading slots 211-213 for storing the headings of words and physical position slots 211-2 for storing the physical positions of the words.
23, logical position slots 231 to 233 for storing normalized logical positions in consideration of the size of the input document 1 (for example, total number of words, number of text data, etc.), and bibliographical information such as part of speech of words. Information slot 241 for storing various information
˜243 and potential slots 251 to 253 for storing potential values of words. For each occurrence of the same word, a set 21i of different types of slots on the same line
Information is set to 25i (i is 1 to 3). That is, the slot groups 211 to 251 in the first row are set with information relating to the first occurrence of the same word, and the slot groups 212 to 252 in the second row are associated with the second occurrence of the same word. Information is set, and the slot group 213 to the last row
Information related to the last appearance of the same word is set in 253.

Further, each life table unit 200 has a heading slot 261 for storing a heading of a word, a frequency slot 262 for storing the appearance frequency of the same word (equal to the number of lines), and the potential slots 251 to 253. A potential distribution slot 263 that stores the average value (average potential) and standard deviation value of the stored potentials, and a life slot 26 that stores the life value of a word
4 and a potential distribution (263) and a gap slot 265 that stores the position of the gap in the context estimated from each potential (251 to 253).

Information of the same heading is set in the heading slots 211 to 213 and 261 in the same unit 200. For the life table unit, the word heading information may be stored in only one slot.

A document division filter 4 which is a document division means.
Is to divide the input document 1 according to the context using the information stored in the life table 3. This division may be done hierarchically. Hereinafter, the part thus divided will be referred to as a document segment.

The characteristic word extraction filter 4, which is a characteristic word extracting means, extracts one or a plurality of words (characteristic words) representing the characteristics of the input document 1 by using the information stored in the life table 3. .

The document element storage unit 61 stores the document elements divided by the document division filter 4.
On the other hand, the document index storage unit 62 stores index information (hereinafter referred to as a document index) of each document segment based on the division information from the document division filter 4 and the characteristic word information from the characteristic word extraction filter 4. It is something to store.

The document segment storage unit 61 and the document index storage unit 62 constitute the output result storage unit 6,
The information stored in the output result storage unit 6 is output to the user via the input / output interface 7.

FIG. 6 is an explanatory diagram showing an example of the storage format of the document index storage unit 62, that is, the storage configuration of the document index.

The document index storage unit 62 grows as a hierarchical structure while storing the index information of the divided document pieces 61 as the input document 1 is divided stepwise. On the contrary, it is also possible to first obtain the document pieces 61 that have been subdivided and integrate them to form larger document pieces 61 to realize a hierarchical structure. Since each document segment 61 has a hierarchical structure, the document index 62 for each document segment 61 will be called a node in the description of FIG.

In FIG. 6, the document segment defined by the node 300 is divided into document segments related to a plurality of nodes 310, 320, ... At the next level.
20 shows that the document segment defined by 20 ... Is divided into document segments related to a plurality of nodes 311, 312, ... At the next level, and the document segment corresponding to each node at each level is shown. A piece is defined, and a document index is stored corresponding to each of the document pieces. The highest-level node 300 represents a node corresponding to the input document 1 itself.

The storage structure of each node, that is, the index information is the same regardless of the node level, and the label portion 301, the identifier slot 302, and the start position slot 3 are included.
03, end position slot 304 and characteristic word slot 3
It consists of 05.

The label portion 301 has numbers (for example, 0, 1, 1.1, 1.2, ...
2, ...) is stored. The identifier slot 302 is an area for storing an identifier for identifying the corresponding document element, and it becomes possible to access the corresponding document element by the stored identifier. The start position slot 303 is an area for storing the start position information of the document segment related to the node, and is stored as the logical position information for the document segment one level higher. The end position slot 304 is an area for storing the end position information of the document segment related to the node, and is stored at the logical position for the document segment one level higher. In addition,
In the case of the top node 300 having no upper node, the start position slot 303 and the end position slot 30
Each of 4 stores the minimum value and the maximum value of the logical position with respect to the entire input document 1. Feature word slot 305
A characteristic word serving as an index of a document segment corresponding to the node is stored in the.

Each node has the same format as described above, and continues to grow while making its level deeper or shallower each time a document fragment is divided or integrated.

Next, the operation of the document processing apparatus having the above-described functional configuration will be described with reference to the flowchart.

First, the overall operation of the life calculator 2 will be described with reference to the flowchart of FIG.

When an arbitrary input document 1 is input to the life calculator 2 (step 401), the input document 1 is divided into sentences which are units of calculation (step 402). The unit of calculation is not limited to “sentence”, and even a character example (text) physically limited to an arbitrary size can be used for the subsequent life calculation.

After the input document 1 is divided into sentences which are arithmetic units, the respective sentences are sequentially sent to the arithmetic object recognizing means 21 and the arithmetic objects are recognized (step 403). The calculation object recognition means 21 mainly recognizes a word of a sent sentence as a calculation object, calculates the position of the input document 1 of the word, and calculates the result or the like of the corresponding slot (21 1 to 21 1) of the life table 3. 213, 221-223, 2
31-233, 241-243). Details of the operation of the calculation target recognition means 21 will be described later with reference to FIG.

After recognizing the calculation target using the calculation target recognition means 21, the potential calculation means 22 is activated (step 404). The potential calculation means 22 is
The potential of the word is calculated for each of the same words, and the result is calculated as the potential slot (25
1 to 253). Details of the operation of the potential calculation means 22 will be described later with reference to FIG.

After the potential of the word is obtained by using the potential calculating means 22, the life of the word is estimated by using the life estimating means 23 (step 405). The life estimation means 23 uses the potential (25
1-253), the potential distribution of the word, the life of the word, etc. are calculated, and the result is stored in the potential distribution slot (263) and the life slot (264) of the life table 3. For details of the operation of the life estimation means 23, see FIG.
It will be described later using 0.

After the information is stored in each slot of the life table 3 as described above, the life calculator 2 activates the document division filter 4 and the characteristic word extraction filter 5 (step 406), and a series of processes. To finish.

FIG. 8 is a flow chart showing the details of the operation of the above-mentioned arithmetic object recognizing means 21, and the arithmetic object recognizing means 2 will be described below with reference to this flowchart.
The operation of No. 1 will be described in detail.

First, when a sentence group divided from the input document 1 is input, each sentence is divided into words (step 501). The operation of dividing into words is performed using, for example, a known morphological analysis technique.

Next, one of the divided words is set as a processing target, and if this word has a new heading, a new life table unit (200) for that word is generated and then that heading is held. The life table unit (200) related to the heading if the word becomes the second or subsequent processing target
After recognizing, the heading slot (21
i) is generated and the headline of the specific word is stored therein, and the bibliographic information slot (24i) is generated to play the part of the specific word or the role of the sentence (“occurred in bullets”, “ Bibliographic information indicating "the words forming the title" and the like) is stored (step 502).

After that, the physical position and the logical position of the word to be processed are sequentially calculated (steps 503 and 504), and the physical position slot (23i) and the logical position slot (24) are calculated.
i) is generated and the physical position and the logical position obtained by the calculation are respectively stored (step 505).

Here, the calculation of the physical position and the logical position of a word is performed by a calculation unit (described as "document" in this embodiment) set when the input document 1 is first input to the life calculator 2. Calculated as a unit.

As an example of the method of calculating the physical position, the 15th sentence of the input document 1 is "/ special search section 1/2/3/4 so far.
/ 5 / share 6/7 / transfer 8 / receive 9/10 / civilian 11/12/13 / almost 14 / to 15 / interview 16/17 / perform 18/19 /. 20 / ”
It is assumed that the sentence is (for convenience of description, a word delimiter is represented by “/” and the appearance order of each word is indicated by numbers “1” to “20”).

Assuming that the word to be processed is "stock" in this sentence, its physical position is the position "15" of the sentence in the input document 1 and the order of appearance "6" of that word in that sentence. It is calculated as a set [15, 6].

This physical position is used in the subsequent calculation of the logical position. When the physical position of the word to be processed is confirmed,
The logical position of the word is then calculated (step 504 described above). As an example, if the logical position is defined as a word position in which the maximum distance in the input document is normalized to 100 (%), the physical position is [m, n] (the specific word is the m-th sentence of the input document). logical position (m, n) of (nth occurrence)
Can be calculated according to the following equation (1).

[0063]

(Equation 1) For example, assuming that the number of sentences in the input document 1 is “250”, the physical position of “stock” in the sentence illustrated above is [1].
5, 6], its logical position (15, 6)
Is calculated as 5.72% by the formula (2) in which a specific numerical value is substituted into the formula (1). That is, the word "stock" appears at the logical position of 5.72% from the beginning of the input document 1.

Logical position (15,6) = {(15-1) +6/20} × 100/250 = 5.72 (%) (2) Here, in the input document 1 for the convenience of the following calculation. The maximum distance of is normalized to 100 (%), but it goes without saying that the normalization index does not need to be determined with particular restrictions.

The physical position and logical position of the word to be processed obtained as described above are stored in the respective slots (step 505 described above).

As described above, when the heading, the physical position, the logical position, and the bibliographical information of the word to be processed now are stored in the life table unit 200 for the word,
Whether or not there is an unprocessed word that has not been processed is checked (step 506), and if there is an unprocessed word, the word to be processed is set to one of the unprocessed words and the above-mentioned processing from step 502 onward is performed. On the other hand, when there is no unprocessed word (when all the words have been processed), the operation of the calculation target recognition means 21 ends.

FIG. 9 is a flow chart showing the operation of the potential calculating means 22. As described above, the operation is started when the operation recognizing means 21 ends.

When the potential calculation means 22 starts the calculation process, first, an arbitrary life table unit (2
00) and select the logical position slot (23
The logical position is extracted from 1) (step 601).

Next, the logical position of the same word in the line immediately before the line to be processed (the line selected in step 601 above, or the line selected in step 606 or step 609 described later) of the life table unit to be processed is determined. It is obtained from the corresponding logical position slot (step 602). If the line to be processed is the first line of the same word in the life table unit (the line selected by the operation of step 601 or step 609), the logical position of the immediately preceding line does not exist, so the logical position is set to 0. % (That is, a value indicating the start position of the input document 1).

Next, the logical position of the same word in the line immediately after the line to be processed in the life table unit to be processed is obtained from the corresponding logical position slot (step 603). Note that the line to be processed is the last line in which the logical positions of the same words in the life table unit are stored (when the life table is finally completed, it is the line immediately before the last line in the life table unit; In the case of the slot “213”), since the logical position of the line immediately after does not exist, the logical position is set to 100%.
(Value indicating the last position of the input document 1).

The potential of the word of the processing target line is calculated from the respective logical positions of the "processing target line", "previous line" and "immediately subsequent line" obtained as described above, and the potential slot is set in the processing target line. Generate (25i),
The potential which is the calculation result is stored (step 604).

Here, in this embodiment, the word potential is defined as the distance between the processing target word and the immediately preceding same word (hereinafter referred to as the left side distance) and the same word immediately after the processing target word. Is defined as a moving average of distances (hereinafter, referred to as right-side distances) (average of distances of two sections formed by three consecutive points, that is, 1/2 of distance of a route of three points).

That is, the sequence of logical positions of the same word having the same heading as the word to be processed {D1, D2, ..., DF
} (F is the frequency of appearance of the same word), the potential (i) of the word at the i-th logical position in this series can be defined as in equation (3). Note that i is 1
To F.

Potential (i) = {left side distance (i) + right side distance (i)} / 2 left side distance (i) = Di-Di-1 (however, D0 = 0) right side distance (i) = Di + 1 -Di (however, DF + 1 = 100) (3) After the potential of the word of the processing target line is determined as described above, it is determined whether or not the same uncalculated word exists (step 605). ). Then, when the same unprocessed word is present, the "next row" of the processing target row of the life table unit is newly set as the processing target row, and the logical position is taken out from the logical position slot (step 606). Return to 602.

Therefore, by repeatedly executing the processing loop consisting of steps 602 to 606, potentials are obtained and stored for all the same words of the same heading related to the life table unit currently being processed, and in step 605, they are not yet stored. The result is that there is no processed word.

At this time, that is, when the same uncalculated word no longer exists, a new row is provided at the end of the life table unit to be processed, and a heading slot, a frequency slot, a potential distribution slot, and a life slot are placed there. , Gap slots (261 to 265) are generated, and the numbers (frequency) of the same words as the headline are stored in the corresponding slots (261, 262) (step 607). The storage of the predetermined information in the potential distribution slot (263), the life slot (264) and the gap slot (265) generated at this time is as follows.
As will be described later, it is performed by the life estimation means 23.

When the processing of the potential calculating means 22 for a certain life table unit is completed as described above, it is judged whether or not there is a life table unit for which the potential calculation processing has not been executed in the life table 3 (step). 6
08).

If an unprocessed life table unit exists, an unprocessed life table unit is selected as the life table unit to be processed, the logical position is extracted from the logical position slot (231) in the first row, and the above-mentioned is performed. Step 6
02 (step 609). Thereby, the potential for each appearance position of the same word having the heading related to the unit is calculated and stored in the same manner as described above.

When the potential calculation processing for all life table units (200) in the life table 3 is completed and the result of the determination in step 608 is that there is no unprocessed life table unit, the potential calculation means 22 A series of operations ends.

FIG. 10 is a flowchart showing the operation of the life estimating means 23. As described above, when the operation of the potential calculating means 22 is completed, the life estimating means 2
3 is activated.

When the life estimation means 23 starts its operation,
First, an arbitrary life table unit (200) of life table 3 is selected (step 701).

Next, the potentials are taken out from the potential slots (251 to 253) of each row of the life table unit currently being processed, and the total sum of the potentials of the same word and the sum of squares of the potentials are obtained (step 70).
2). Also, the frequency slot (26
The frequency is extracted from 2) (step 703). Next, the average potential of the same word related to the life table unit is calculated by dividing the total of the obtained potentials by the frequency of extraction (step 704). Further, the standard deviation of the potential of the processing target word is calculated from the sum of squares of the potential, the frequency, and the average potential (step 705).

Now, a series of potentials {P (M) 1, P (M) 2, ..., P of the same word having the heading M
Considering (M) F (M)}, the arithmetic expression of standard deviation (M) is
It can be expressed by equation (4). Note that F (M) is the appearance frequency of the same headword.

[0084]

(Equation 2) As described above, when the average potential and standard deviation of the same word related to the life table unit to be processed are obtained, the potential distribution slot (263) of the life table unit is obtained.
The paired information of the calculated average potential and standard deviation is stored in (step 706).

Next, the life of the word having the heading is estimated from the potential of each identical word, the frequency of the word having the heading, the average potential and the standard deviation, and stored in the life slot (264) of the life table unit. (Step 707).

Lifespan (M) for words with heading M
The estimation of is performed by the calculation of the following equation (5).

[0087]

(Equation 3) That is, if the potential of each same word is smaller than the value deviated from the average potential by a larger standard deviation, it is determined that the same word appears in the same context and topic, and a potential with a small deviation from the average potential is selected. The lifetime is estimated by the sum of the potentials of the same words each has. Since the potential distribution has a smaller value, the threshold value deviated from the average potential by a smaller standard deviation is not provided.

Although the standard deviation of 1 is used as the characteristic value for discriminating whether or not the user is in the same context and topic, this characteristic value may be different. For example, 1.5 times or 2 times the standard deviation may be used, and the sum of potentials equal to or lower than the third quartile of the quartile deviation may be used as the “lifetime”.

Next, the gap of the document 1 is derived from the logical position of the same word whose potential does not fall within the range of the sum of the average potential and the standard deviation and is stored in the gap slot (265) of the life table unit (step). 70
8). This completes the life table unit.

As mentioned above, the potential of a word is
It is the average of the left side distance and the right side distance. If the potential does not fall within the range less than the sum of the average potential and the standard deviation, either (the influence of) the left side distance or the right side distance
Is checked, and it is considered that there is a gap as a turning point in the context of the document in the direction in which the influence is large. That is, the potential is greatly deviated from the average potential (potential does not fall within the range less than the sum of the average potential and standard deviation). The left side distance and the right side distance of the appearing word are compared, and the direction having the large distance and the logic of the word are compared. A set of positions is a “gap”. Since the number of appearing words whose potential is greatly deviated from the average potential is not limited to one, the number of gaps stored in the gap slot may be plural.

When the calculation and storage of the average potential, standard deviation, lifespan, and gap for the headword related to the life table unit to be processed are completed as described above, there is a life table unit for another unprocessed headword. It is determined whether or not (step 709). When there is an unprocessed life table unit, a certain unprocessed life table unit is selected (step 710) and the above step 70 is performed.
The processing of 2 to step 709 is performed.

By repeating the processing loop of steps 702 to 710, the calculation and storage of the average potential, standard deviation, lifespan and gap for all life table units (headwords) are completed, and in step 709. A negative result is obtained, and the operation of the life estimation means 23 is ended.

FIG. 11 is a flowchart showing the operation of the document division filter 4, which is started after the processing of the life estimation means 23 is completed.

When the document division filter 4 starts its operation, first, the input document 1 to be divided is subjected to the life calculator 2
(Step 801) and the accuracy of document division is set (step 802).

The accuracy of the division of this document is given by the user through the input / output interface 7, for example. When no information is given from the outside, a predetermined accuracy (default accuracy) is set. For example, the precision of division is set such that even the most finely divided portion has one paragraph or more, or has a predetermined number of sentences or more. In addition, when the division has a hierarchy, the division accuracy is set so as to differ depending on the hierarchy.

Next, from the unprocessed life table unit of life table 3, the gap of the headword related to the unit is extracted (step 803), and the gap is classified by the logical position as a key, and the type of the logical position is determined. Separately, the number is integrated (step 804). For example, each sentence of the input document 1 is set as the type of the logical position, and the type (sentence) of the logical position to which the logical position in the gap information belongs is integrated for each direction.

This processing is performed while checking whether or not there is an unprocessed life table unit (step 80).
5) Repeat until there are no outstanding life table units.
Therefore, at the stage where there is no unprocessed life table unit, the type of logical position having a large integrated value (for example, a sentence) gives position information capable of dividing the document 1.

Therefore, when there is no unprocessed life table unit, the calculation result (integrated value) of the gap is limited by the division accuracy, and the input document 1 is divided into document pieces 61 and output (step 806).

Then, a node (see FIG. 6) corresponding to the document segment 61 is generated in the document index 62, and the identifier, start position and end position of the document segment 61 are assigned to the node identifier slot (302), Start position slot (30
3) and the end position slot (304) (step 807), and the operation of the document division filter 4 is ended.

FIG. 12 is a flowchart showing the operation of the characteristic word extraction filter 5. Feature word extraction filter 5
Is started after the processing of the life estimation means 23 is completed,
It is possible to perform parallel processing except for the processing of the document division filter 4 and the processing of step 905.

When the operation of the characteristic word extraction filter 5 is started, first, the number of characteristic word candidates to be output as an output result is set (step 901). The number of characteristic word candidates is
For example, it is given from the user through the input / output interface 7. When no information is input by the user, a predetermined number of candidates (default number of candidates) is set.

Next, from the unprocessed life table unit of life table 3, the lifespan of the headword related to that unit is taken out from the lifespan slot (step 902), and the headwords of this time whose lifes have been taken out are included. All the headwords are sorted in order of the longest life of the word, with the life of the heading as a key (step 903). Such processing is repeated while checking whether or not there is an unprocessed life table unit (step 805) until there are no unprocessed life table units.

When there is no unprocessed life table unit due to such repetitive processing, the result of rearranging the headwords is limited by the set number of candidates, and the headwords of that number of candidates are set as the characteristic words, and each node of the document index 62. (Step 905), and the characteristic word extraction filter 5 ends the operation.

For example, a gap is extracted from the life table unit related to the headword selected as the characteristic word, and the position of the gap is compared with the start position and the end position of the node related to each document segment 61, and the characteristic word is compared. The document segment 61 including the is recognized, and the information of the characteristic word is stored in the characteristic word slot (305) of the node related to the recognized document segment 61.

As described above, according to the above embodiment, the means for obtaining the life of each word in the document is provided, and the distribution of the life of the word is observed to divide the document. It is possible to easily identify the place where the topic change occurs, and automatically and properly divide the document. Therefore, according to the above embodiment, even if the input document is a compound document, the document can be divided according to the context or topic of the document.

Further, according to the above-mentioned embodiment, when classifying a document, it is possible to classify the document with high accuracy without preparing a huge amount of data for a person to collate the classification category of the document in advance. .

Further, according to the above-described embodiment, since the means for obtaining the life of each word in the document is provided and the characteristic word is extracted based on the life of the word, the intention and the context of the input document are faithful. It is possible to accurately and automatically extract the characteristic words reflected in. Therefore, it is not necessary to manually add a keyword to the document.

In the above embodiment, the document processing apparatus of the present invention is applied to an apparatus intended for document classification, but the technical idea may be applied to other document processing apparatuses. good. For example, the document dividing device may be configured by applying only the technical idea for dividing the document, or the characteristic word extracting device may be configured by applying only the technical idea for extracting the characteristic word from the document. good. Further, in such a feature word extraction device configuration, a context vector having feature words as vector elements may be defined, and a functional unit for searching for similar sentences may be provided.

In the above embodiment, the life table unit (life span, etc.) is formed for all headwords included in the document. However, life table units (only for independent words such as nouns and verbs) are formed. May be formed. Furthermore, although the characteristic word has been described as a so-called word such as a noun, the characteristic word can be applied to, for example, a device for summarizing a document by forming the characteristic word in the form of “phrase” or “section”.

Further, it is needless to say that the defining equations for the potential and the life are not limited to those in the above embodiment. For example, for the potential of a word in which “used for title” is stored as bibliographic information, a predetermined offset value may be added to the above calculation formula. Further, only the words whose appearance frequency is a predetermined number of times or more may be the target of life calculation.

Furthermore, in the above-described embodiment, the characteristic word is commonly found in the input documents and shown in correspondence with each divided document. However, the lifespans of the constituent words are sorted for each divided document. You may make it extract.

[0112]

According to the document processing apparatus of the first aspect of the present invention,
Based on the appearance position of an arbitrary word in the input document, the word life calculator that obtains the life information of the word in which the word has an asserting function in the document, and the life of the word obtained by this word life calculator A word lifespan information storage means for storing information, and a document division means for dividing the input document by referring to the lifespan information of the words stored in the word lifespan information storage means,
Since the input document is divided by referring to the word life information, it is possible to automatically divide the input document according to the change of context or topic in the situation where the information prepared in advance is almost unnecessary. You can

According to the document processing apparatus of the second aspect of the present invention, based on the appearance position of an arbitrary word in the input document, the life of the word for which the asserting function is applied in the document is obtained. An index of the input document with reference to the calculator, the word life information storage means for storing the life information of the word obtained by the word life calculator, and the life information of the word stored in the word life information storage means. Since the feature word extracting means for extracting the feature word to extract the feature word from the input document is extracted by referring to the life information of the word, the user can specify the feature word of the input document. It is possible to provide a highly convenient device that can automatically extract a characteristic word without the need.

According to the document processing apparatus of the third aspect of the present invention, based on the appearance position of an arbitrary word in the input document, the life of the word for which the asserting function of the word is obtained is obtained. The input document is divided by referring to an arithmetic unit, a word lifetime information storage unit for storing the lifetime information of the word obtained by this word lifetime arithmetic unit, and the lifetime information of the word stored in this word lifetime information storage unit. Since the document dividing means and the characteristic word extracting means for extracting the characteristic word serving as the index of the input document and / or the divided document by referring to the life information of the word stored in the word life information storing means are provided, Curing with the device of the second invention can be obtained together.

[Brief description of the drawings]

FIG. 1 is a functional block diagram of a document processing apparatus according to an embodiment.

FIG. 2 is a block diagram showing a system configuration example of a document processing apparatus.

FIG. 3 is an example of a memory table used in a conventional document processing apparatus.

FIG. 4 is an example of a memory table used in a conventional document processing apparatus.

FIG. 5 is an explanatory diagram showing a configuration example of a life table unit of the embodiment.

FIG. 6 is an explanatory diagram showing a storage format of a document index according to the embodiment.

FIG. 7 is an operation flowchart of the life calculator of the embodiment.

FIG. 8 is an operation flowchart of a calculation target recognition unit of the embodiment.

FIG. 9 is an operation flowchart of the potential calculation means of the embodiment.

FIG. 10 is an operation flowchart of a life estimation unit of the embodiment.

FIG. 11 is an operation flowchart of the document division filter according to the embodiment.

FIG. 12 is an operation flowchart of the characteristic word extraction filter of the embodiment.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Input document, 2 ... Lifetime calculator, 3 ... Life table, 4 ... Document division filter, 5 ... Characteristic word extraction filter, 6 ... Output result (storage part), 21 ... Calculation object recognition means, 22 ... Potential calculation means , 23 ... Life estimation means, 61 ... Document segment (storage unit), 62 ... Document index (storage unit).

Claims (4)

[Claims] 1. Input means for inputting a document described in natural language, and lifetime information of a word which causes the asserting function in the document based on the appearance position of an arbitrary word in the input document. The word life calculator for obtaining, the word life information storage means for storing the life information of the word obtained by the word life calculator, and the above-mentioned input with reference to the life information of the word stored in the word life information storage means A document processing apparatus comprising: a document dividing unit that divides a document. 2. Input means for inputting a document described in natural language, and lifespan information of a word which causes the asserting function in the document based on the appearance position of an arbitrary word in the input document. With reference to the word lifespan calculator for obtaining, the word lifespan information storage means for storing the lifespan information of the word obtained by this word lifespan calculator, and the lifespan information of the words stored in this word lifespan information storage means, A document processing apparatus comprising: a characteristic word extracting unit that extracts a characteristic word that serves as an index of an input document. 3. Input means for inputting a document described in natural language, and lifetime information of a word which causes the asserting function in the document based on the appearance position of an arbitrary word in the input document. The word life calculator for obtaining, the word life information storage means for storing the life information of the word obtained by the word life calculator, and the above-mentioned input with reference to the life information of the word stored in the word life information storage means Document dividing means for dividing a document and characteristic word extracting means for extracting characteristic words serving as indexes of the input document and / or divided documents by referring to the life information of words stored in the word life information storage means. A document processing device comprising: 4. The word life calculator calculates the word position recognition means for recognizing the appearance position of a target word, and the distance between the appearance positions of the word to be processed and the adjacent same word. A potential calculation means for obtaining a potential representing a range affected by the word to be processed, and a life estimation means for obtaining life information of the word based on the potential distribution information of the same word obtained by the potential calculation means. The document processing apparatus according to claim 1, further comprising: