JP2005141476A - Document management device, program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a document management device for efficiently registering a large quantity of documents belonging to a plurality of categories with no distinction between documents relating to a plurality of topics and documents with a plurality of topics not relating to one another included therein and displaying only parts belonging to the categories when displaying a retrieval result. <P>SOLUTION: The management device performing segmentation of documents to determine whether the plurality of the categories are applied resulting from collecting the topics not relating with one another or the plurality of the categories are applied resulting from collecting the topics relating to one another when applying the categories to the document to register it into a database. The management device determines that document contents include the plurality of topics relating to one another and applies the plurality of categories to the documents, as long as the documents have a single segment even if the plurality of categories are applied thereto. The management device determines that the documents include the plurality of topics not relating to one another when the documents are segmented into two or more segments and re-applies the categories at each divided segment without applying the categories to the documents. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a document management apparatus, a program, and a recording medium, and more specifically to a technique for classifying document data into a plurality of categories.

  In recent years, a large amount of electronic documents can be exchanged through a computer network such as the Internet. For this reason, a large amount of information has been acquired, making it difficult to extract features of individual information. Therefore, a technique for classifying and organizing the acquired information is required.

  Typical methods for automatically classifying document data include known categories such as decimal classification of books, and a method for classifying new document data into categories that are considered appropriate, There is a method in which categories to be classified are unknown, and similar documents are collected from a set of documents to create and assign classification categories. With these technologies, it is possible to sort and organize a large number of documents.

  When the category such as the former is known, a category for pre-classification, a sample document belonging to the category, and a characteristic word are given, and the importance of the word is calculated from the sample document, A feature quantity vector is generated with a pair of importance levels for the category. Similarly, with respect to the document data to be classified, the importance level for the word and the document data is calculated to generate a feature vector.

  Next, the distance between the feature quantity vector of the category and the feature quantity vector of the document is defined, and each document is assigned to a similar category using the value. If it is determined that the distance is very far, that is, it is not similar to any category, it is not assigned to any category.

In the technique of Patent Document 1 using this method, a category to be classified as document data and its features are input, a feature vector is calculated and stored from the input category and features, and a classification criterion is input. Using the input category and its features and classification criteria, a new intermediate category is created, and similar input categories are assigned to the same intermediate category. And the inputted category are used to classify the inputted document data.
As a result, the conventional method avoids the problem that as the total number of categories to be classified and the number of words appearing increase, the calculation time increases in proportion to the number of information. Can be efficiently classified into a plurality of categories in a short time.

  By the way, a document including a related topic, for example, a document written about “sports and its economic ripple effect” is classified into “sports” and “economy” categories by the classification method as described above. Also, newspaper articles with various topics mixed on a single page, for example, articles written about “Sports” and articles written about “Economy” in one, are the same. It will be classified into such a category.

  As a technique for classifying a document such as a newspaper article in which various topics of the latter are mixed on a single page, Patent Document 2 classifies and classifies the class information that is a concept of context and the class. The semantic structured result of the document with segment information added is input, and for each attribute in the attribute set that is the structured result, the document unique ID, class information, and the division result according to the class By assigning certain segment information and storing it in the database, the viewpoint is not constant, and flexible attribute-designated search according to the context of each searcher can be realized.

Here, as a technique for dividing a document in which a plurality of articles and the like are mixed into segments for each meaning, there is Patent Document 3. The technique of Patent Document 3 divides an electronic document into paragraphs, calculates the degree of association between paragraphs based on keywords extracted from the paragraphs, and creates a square matrix with the paragraphs as dimensions. The degree of relevance is put in each component of one side area with the diagonal component of this square matrix as a boundary, and in the one side area where the degree of relevance is put, an arbitrary first row (or column) and an arbitrary first column (or row) Then, the total value of the relevance in the triangular area surrounded by the diagonal components is obtained, and the document is divided based on the total value of the relevance.
JP 2000-112971 A JP 2001-195426 A JP 2000-235574 A

As described above, when a document related to a plurality of topics and a document containing a plurality of unrelated topics are classified by a general method, the same category is assigned and it is not possible to distinguish between the documents. There is a case.
In the above example, two categories of “sports” and “economy” are assigned to the categories of documents written on “sports and their economic ripple effects”. Similarly, categories assigned to documents that combine articles written about “sports” and articles written about “economics” into two categories are also assigned to “sports” and “economics”. It will be.

This problem is that before assigning a category to a document in which a plurality of articles are grouped into one, separate the documents into articles (segments) and assign a category to each article as in Patent Document 2. The problem can be solved.
However, when registering a large number of documents as a document database, it is necessary to check each type of document, and then to divide into segments and assign categories to the segments for registration. It is unrealistic.

  The present invention has been made in consideration of the above-described circumstances, and a large number of documents belonging to a plurality of categories are not distinguished from a document containing a plurality of topics not related to a document related to a plurality of topics. An object of the present invention is to provide a document management apparatus, a program, and a recording medium that can be efficiently registered in a database and that display only the segment when the search results are displayed that match the segment category.

  In order to solve the above-mentioned problems, the invention of claim 1 categorizes a document content into a predetermined category, associates the category classified by the classification unit with the document content, and registers them in the document database. A document segmentation unit that divides the document content into segments composed of individual topics, and when the document is classified into a plurality of categories by the classification unit, the document segmentation unit includes the document segmentation unit. The content is divided into segments, and when the number of divided segments is one, a plurality of categories assigned by the classification unit are assigned to the document content, and when the number of divided segments is two or more, the classification unit Re-categorize each segment divided in (1) and assign categories to each segment and register them in the document database. Characterized in that the so that.

  According to a second aspect of the present invention, in the document management apparatus according to the first aspect, the registration unit associates a single category flag and the category with the document when one category is given to the document content. When a plurality of categories are assigned and the document content is not divided into a plurality of segments, a plurality of category flags and a plurality of categories are set in the document attribute record in association with the document. If the segment is divided into segments, a plurality of category flags are set in the document attribute record in association with the document, and the category assigned to each segment and the segment position in the document are set in the segment attribute record in association with the document. Then, these document attribute records and segment attribute records are registered in the document database. Characterized by being adapted to.

  According to a third aspect of the present invention, in the document management apparatus according to the second aspect of the present invention, the document management apparatus includes a search unit that searches the document database by specifying a category and displays the searched document content. When there is a category matching the designated category in the segment attribute table, only the document content of the matched segment is displayed.

The invention according to claim 4 is a program for causing a computer to execute the function of the document management apparatus according to claim 1, 2 or 3.
The invention of claim 5 is a computer-readable recording medium on which the program according to claim 4 is recorded.

According to the present invention, when a category is assigned to a document and registered in the database, the document is divided into segments for each topic, and the divided segments are categorized, so a large number of documents belonging to a plurality of categories. Can be efficiently registered without distinction from a document containing a plurality of topics not related to a document related to a plurality of topics.
Further, when displaying search results, if the search results are divided into segments, only the parts belonging to the category are displayed, so that a document composed of a plurality of topics can be accurately displayed and managed. .

A preferred embodiment according to the document management apparatus of the present invention will be described below with reference to the drawings.
In the present embodiment, description will be made assuming that a classifier is configured using a support vector machine (SVM). However, classification may be performed by other Bayes method, discriminant analysis using Fisher discriminant, or the like.

  SVM is a classification method that classifies an object represented by a multidimensional vector into two classes, and performs classification by dividing a feature space into two subspaces on a hyperplane. And whether the object belongs to the class is determined by the expression (2).

g (x) = w · x + b (1)
f (x) = sgn (g (x)) (2)

Here, “·” indicates the inner product of the vector w and the object x, w is a weight vector that defines the normal of the hyperplane, and b is a scalar quantity that represents a threshold value.
The function sgn (y) takes a value of “+1” when the value of the argument y (scalar amount) is positive, and takes a value of “−1” when the value is 0 or less.

Here, the above classification function is applied as follows in document classification.
n categories C = (C ₁ , C ₂ ,..., C _n ) and d feature words T = (t ₁ , t ₂ ,..., t _d ) for categorization are determined in advance. deep. Each element v _i of the feature vector V representing the feature of the document data is obtained from the frequency at which the feature word t _i corresponding to the element v _i appears in the document, the number of documents in which the feature word t _i appears, and the like. To do. In addition, in category C _i, the elements w _ij of the weight vector is, in category C _i when the probability and the feature word t _j belonging to the category C _i when the feature word t _j appeared in the document did not appear in the document Assume that the calculation is performed in combination with the probability of belonging.

Next, a plurality of learning documents belonging to a certain category C _i are prepared, a feature quantity vector V of the learning document is calculated, the value of f (V) becomes “+1”, and learning documents belonging to other categories are calculated. calculates a weight vector W _i and the threshold b _i the value of f (V) is "-1" in the document, for all categories, calculate a weight vector W _i and the threshold value b _i.
Hereinafter, a classifier in which the value of f (V) is “+1” for a certain category C _i will be written as f _i (V).

Therefore, when the input document is classified into n categories, the feature quantity vector X of the input document is calculated, and n classifiers f ₁ (X), f ₂ (X) _,. (X)
Calculate
If f _i (X)> 0, it belongs to category C _i , and if f _i (x) ≦ 0, it is determined not to belong to category C _i . By repeating this determination n times for each category classifier, the category of the given document is determined.

Next, a document management apparatus that performs document classification and search using the above-described classifier will be described.
FIG. 1 is a block diagram showing a functional configuration of a document management apparatus according to the present invention. In the figure, the document management apparatus includes a document database 10, a classification unit 20, a document segmentation unit 30, a registration unit 40, a search unit 50, The learning unit 60 is included.

The document DB 10 includes a classification parameter, a document attribute table, a segment attribute table, and a plurality of documents, and is managed by a relational database.
The plurality of documents includes at least a document ID (identifier), a document name, a bibliographic item (creator name, creation date, issuing place, etc.) and a column of document contents.
The classification parameter is made up of a weight vector W _i and a threshold value b _i for each category C _i corresponding to the feature word T.

The document attribute table includes at least columns of document ID (identifier), document name, category name, and multi-category flag.
The multi-category flag has a value of “0” when the document belongs to a single category, and has a value of “1” when the document belongs to a plurality of categories.
For example, as shown in FIG. 2, documents Doc1 and Doc5 are classified into a single category “Economy”, Doc2 is classified into a single category “Politics”, and Doc4 is classified into a single category “Sports”.

  Documents Doc3 and Doc6 belong to a plurality of categories. Among these, the document Doc6 is a case where the contents are related to a plurality of categories ("politics" and "economy"). In this case, the multi-category flag is set to “1”, and the number of categories is listed as a single category record.

  Further, the document Doc3 is a case in which documents of different topics (“politics” and “economy”) are collected. In this case, the multi-category flag is set to “1”, and the document is divided into segments of each category, and it is recorded in the segment attribute table that a category is given to the segment.

The segment attribute table includes at least columns of document ID (identifier), category name, segment start position, and segment end position (see FIG. 2).
That is, the category name assigned to this segment is recorded using the partial character string indicated by the segment start position to the segment end position in the document character string indicated by the document ID.

The classification unit 20 reads the classification parameters (the feature word set T, the weight vector W for each category, and the threshold value b) of the document DB 10 into the memory, and uses the classifier represented by the above-described equations (1) and (2). Classify actual data of documents to be classified.
The feature quantity vector is calculated from the appearance frequency of the feature word in the document to be classified and the ratio of the document in which the feature word appears, and n classifiers are applied.
As a result, classifier f _i be a positive value if m pieces, so that the m categories is assigned to that document.

If m = 1, the determined category is set as the document category. If m> 1, the actual data of the document to be classified is passed to the document segmentation unit 30 for classification processing.
The document segmentation unit 30 performs segmentation of the delivered document using a known technique (for example, see Japanese Patent Application Laid-Open No. 2000-235574), and determines whether this document is a document composed of a plurality of topics. At this time, each segment is indicated by the start position and end position of the segment in the document.

Next, as a result of segmentation, when the image is divided into a plurality of segments, a category is assigned to each segment using the classification unit 20. In this case, it is determined that a plurality of documents that are to be processed as separate documents are connected and input.
Further, when the segmentation result indicates that the document is not divided into a plurality of segments, the document is composed of a plurality of related topics (a plurality of categories).

  After classification as above, for the following three cases,

Case 1: Single segment, single category.
Case 2: Multiple segments, multiple categories.
Case 3: Single segment, multiple categories.

  Generate data consisting of the following items:

  (Document name, category name, multi-category flag, segment start position, segment end position)

For example, in case 1, the following five sets are generated.
(Document name, category name, 0,0,0)

In case 2, the first line indicates that the document represented by “document name” consists of a plurality of categories, and the second and subsequent lines are repeated by the number of segments.
(Document name, 0,1,0,0)
(Document name, category name 1,0, start position and end position of segment 1)
(Document name, category name 2,0, start position and end position of segment 2)
. . .

In case 3, the first line indicates that the document represented by “document name” consists of a plurality of categories, and the second and subsequent lines are repeated by the number of categories.
(Document name, 0,1,0,0)
(Document name, Category name 1,0,0,0)
(Document name, Category name 2,0,0,0)
(Document name, Category name 3,0,0,0)
. . .

FIG. 3 is a flowchart showing the processing procedure of the classification unit 20 and the document segmentation unit 30.
The classification parameter of the document DB 10 is read, and the feature quantity vector X for the feature word of the document to be classified is calculated (step S10).
The weight vector W, the threshold value b, and the document feature vector X of each category are applied to each classifier f _i (X), i = 1 to n, and the value of each f _i (X) is calculated (step S11). .
When the number of categories for which the value of f _i (X) is positive is 1 (NO in step S12), this category is assigned to the document as being a single category, and the five groups of the above case 1 (document name) , Category name, 0,0,0) (step S19), and the classification process for this document is completed.

On the other hand, when the number of categories is 2 or more (YES in step S12), the passed document is segmented by a known technique (for example, Japanese Patent Laid-Open No. 2000-235574). At this time, each segment is indicated by the start position and end position of the segment in the document (step S13).
If the number of divided segments is 1 (NO in step S14), the set of case 3 as a single segment and multiple categories in step S19
(Document name, 0,1,0,0)
(Document name, Category name 1,0,0,0)
(Document name, Category name 2,0,0,0)
(Document name, Category name 3,0,0,0)
. . .
Is generated (step S19), and the classification process for this document is terminated.

  On the other hand, when the number of segment divisions is 2 or more (YES in step S14), a quintuple (document name, 0,1,0,0) of the first line of case 2 is generated and the segment is divided. A feature vector S is calculated for one segment from (Step S15).

The weight vector W of each category, the threshold value b, and the feature vector S of the segment are applied to each classifier f _i (S), i = 1 to n, and the value of each f _i (S) is calculated (step S16). .
A category in which the value of f _i (S) is positive is assigned as the category of this segment, and the quintuple (document name, category name, 0, segment start position, end position) of the above case 2 is generated ( Step S17).
If the categories are assigned to the remaining segments in the same manner and all the segments have been assigned, the classification process for this document is terminated (steps S5 to S18).

  The registration unit 40 receives the document data to be classified, assigns an identifier (document ID) for specifying the document data, and, for example, the document attribute table and the segment attribute table of the document DB 10 using the following SQL sentence: Register with.

insert into document attribute table
values (doc_id, doc_name, category_name, multi_category_flag);
insert into segment attribute table
values (doc_id, category_name, seg_start, seg_end);

  The search unit 50 searches the document DB 10 in response to a search request from the user, for example, a search sentence of an SQL sentence, and displays the document content of the search result on a display device or the like.

(1) When searching for documents by category name:
For example, it can be searched with the following SQL sentence.
select Document name from Document attribute table where Category name = “Politics”

(2) When searching for documents belonging to multiple categories and their categories:
For example, it can be searched with the following SQL sentence.
select Document name, category name from document attribute table
where multi-category flag = 1

(3) When searching for documents and constituent segments belonging to multiple categories and their categories:
For example, it can be searched with the following SQL sentence.
select t1.document name, t2.category name, t2.segment start position,
t2. Segment end position
from document attribute table t1, segment attribute table t2
where t1. Multi-category flag = 1 and t1. Document ID = t2. Document ID

(4) When searching for a document consisting of a single segment belonging to multiple categories:
For example, it can be searched with the following SQL sentence.
select Document name, category name from document attribute table
where document attribute table, multi-category flag = 1 and not exists
(select * from document attribute table
where Document attribute table. Document ID = Segment attribute table. Document ID)

Of the contents of the document searched in this way, when the entire document is targeted, the display is as before.
However, in the display of the document content in which the search result is divided into segments, portions other than the corresponding segment of the document are masked, and only the segment portion to which the corresponding category is assigned is displayed. As a result, the searcher need not refer to unintended data.

For example, in the case where there is a GUI system, as shown in FIG. 4, other than the corresponding segment is masked with gray so as not to be displayed.
If there is no GUI system, characters in segments other than the corresponding segment are replaced with symbols such as periods and midpoints as shown in FIG.

  As described above, the learning unit 60 uses the document data classified for each category as learning document data, and uses the classification parameters (weight vectors for each category) of the n classifiers defined in Expressions (1) and (2). And the threshold value), and this classification parameter is stored in the classification parameter of the document DB 10.

As described above, by configuring the document management device, when a category is assigned to a document and registered in the database, the document is divided into segments for each topic, and the divided segments are categorized. Thus, a large number of documents belonging to a plurality of categories can be efficiently registered without being distinguished from a document containing a plurality of topics not related to a document related to a plurality of topics.
Further, when displaying search results, if the search results are divided into segments, only the parts belonging to the category are displayed, so that a document composed of a plurality of topics can be accurately displayed and managed. .

Further, the present invention is not limited only to the above-described embodiment, and the functions of the document management apparatus of the above-described embodiment are each programmed, written in a recording medium such as a CD-ROM in advance, and installed in a computer. It goes without saying that the object of the present invention can also be achieved by installing the CD-ROM or the like in a medium driving device such as a CD-ROM drive, and installing and executing these programs.
In this case, the program read from the recording medium itself realizes the above-described embodiment, and the program and the recording medium on which the program is recorded also constitute the present invention.

As a recording medium, a semiconductor medium (for example, ROM, nonvolatile memory card, etc.), an optical medium (for example, DVD, MO, MD, CD-R, etc.), a magnetic medium (for example, magnetic tape, flexible disk, etc.) Either may be sufficient.
Alternatively, the program stored in the storage device may be directly supplied from the server computer via a communication network such as the Internet. In this case, the storage device of this server computer is also included in the recording medium of the present invention.

  Further, not only the above-described embodiment is realized by executing the loaded program, but the operating system or the like performs part or all of the actual processing based on the instruction of the program, and the above-described embodiment is performed by the processing. The case where the form is realized is also included.

  Accordingly, by distributing a program for executing the functions of the above-described embodiment and a recording medium storing the program, installing the program in an internal storage device or an external storage device of the computer, and executing the installed program Since the functions of the above-described embodiment are realized, cost, portability, and versatility can be improved.

It is a block diagram which shows the function structure of the document management apparatus of this invention. It is a figure which shows the example of a data structure of the document attribute table of a document database, and a segment attribute table. It is a flowchart which shows the procedure which categorizes the document used as a classification | category object. FIG. 6 is a diagram illustrating an example of a display screen of document content divided into segments when a GUI system is provided. FIG. 6 is a diagram illustrating an example of a display screen of document content divided into segments when a GUI system is not provided.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Document database (DB), 20 ... Classification part, 30 ... Document segmentation part, 40 ... Registration part, 50 ... Search part, 60 ... Learning part.

Claims (5)

  In a document management apparatus having a classification unit that classifies document contents into predetermined categories, and a registration unit that associates the categories classified by the classification unit with the document contents and registers them in the document database. A document segmentation unit that divides the segment into topics, and the document segmentation unit divides the document content into segments when the classification unit categorizes the segment, and the number of divided segments is one. Sometimes, a plurality of categories assigned by the classification unit are assigned to the document content, and when the number of divided segments is two or more, the segments divided by the classification unit are reclassified for each segment. A document management apparatus characterized in that a category is assigned and registered in the document database.   2. The document management apparatus according to claim 1, wherein when a category is assigned to the document content, the registration unit sets a single category flag and the category in the document attribute record in association with the document, If the document category is assigned and the document content is not divided into a plurality of segments, a plurality of category flags and the plurality of categories are set in the document attribute record in association with the document, and the document content is divided into a plurality of segments. In association with the document, a plurality of category flags are set in the document attribute record, and the category assigned to each segment and the segment position in the document are set in the segment attribute record in association with the document. Records and segment attribute records are registered in the document database. Document management apparatus according to claim.   The document management apparatus according to claim 2, further comprising: a search unit that searches the document database by specifying a category and displays the searched document content, and the search unit matches the specified category. A document management apparatus characterized by displaying only the document contents of a matched segment when there is a category to be matched in the segment attribute table.   A program for causing a computer to execute the function of the document management apparatus according to claim 1, 2 or 3.   The computer-readable recording medium which recorded the program of Claim 4.

Images