JP2006330995A - Document processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technlogy for classifying documents into categories and assigning a proper name to each category. <P>SOLUTION: In the document processor 20, a control part 21 analyzes, upon receipt of document data, the document data to extract character string data. Whether each of a plurality of words predetermined based on the extracted character string data appears in the character string data or not is determined, and a characteristic vector is generated based on the determination result. The document data are classified to a category number designated by a user using SVM algorithm based on the generated characteristic vector. Appearance frequencies of the words are calculated at each of the classified categories, and a word having a high appearance frequency is displayed on a display part 23. If a word stored in association with the word displayed is contained in a relevant word table stored in a nonvolatile storage part 25b, this word is also displayed in parallel. The user selects a category name based on the displayed word. The control part 21 determines the selected word as the category name. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for storing electronic documents, and more particularly to a technique for classifying and storing electronic documents based on the contents of the electronic documents.

  When transmitting and recording information, a paper document is used. If an attempt is made to save the paper document in a cabinet or the like, a storage space is required. As the number of paper documents increases, the space to be stored must be expanded accordingly, and thus the storage of information by paper documents lacks effective use of the location. Therefore, a paper document is read and digitized by a scanner or the like, and the digitized image data is filed for each paper document and stored in a hard disk or the like.

  By the way, when saving a filed document, it is convenient to classify and save it in a directory or the like according to the type of the document. As such a technique, Non-Patent Documents 1 and 2 describe a technique for classifying a document according to a support vector machine algorithm (hereinafter referred to as an SVM algorithm) using a feature amount (hereinafter referred to as a feature vector) representing the characteristics of the document to be classified. Is disclosed.

Patent Document 1 discloses a technique of generating a feature vector from a training data document, learning a classification method using an SVM algorithm, and classifying a document to be classified based on the learning result.
"Text classification-" Trade fair "in learning theory", Information processing, vol142, no.1, 2001 Takashi Kawauchi, Akifumi Makinouchi, “Area Image Classification Using Morphological Operations”, IEICE Data Engineering Workshop (DEWS) 2005 Online Proceedings, 1C-i5, (2005) JP 2001-22727 A

  However, in the techniques of Non-Patent Documents 1 and 2, even if the documents represent the same content, if the included words are different, they are recognized as documents representing different meanings. Further, Patent Document 1 only discloses a technique for classifying a document, and does not disclose a technique for determining a category name into which a document is classified. Therefore, the location of the document cannot be specified from the category name.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a technique for classifying documents into categories and assigning appropriate names to the categories.

  In order to solve the above problems, the present invention provides document data input means for obtaining document data obtained by digitizing a document, category number specifying means for receiving the number of categories for classifying the document data, and document data input means. Analyzing the acquired document data, character string data generating means for generating character string data representing a character string, and feature vector generating means for detecting a feature of the character string data and generating a feature vector for each document And classifying means for classifying the document data acquired by the document data input means so that the number of categories received by the category number specifying means is based on the feature vector for each document, and the character string data. A calculation means for calculating the frequency of occurrence of words from the document group classified into each category by the classification means; To provide a document processing apparatus for and a category name determining means for determining the respective category name based on the highly frequent frequency word has.

  According to this document processing apparatus, the feature vector is generated from the feature of the character string data constituting the document data for each document data, so that the document data can be classified into a predetermined number of categories from the feature vector. In addition, for each classified category, the frequency of the words constituting the character string data is calculated, and the category name is determined based on the words with the high frequency, so that an optimum category name can be assigned to the classified category. it can.

  Further, the present invention provides document data input means for acquiring document data obtained by digitizing a document, category number specifying means for receiving the number of categories for classifying the document data, and document data acquired by the document data input means. Character string data generating means for analyzing and generating character string data representing a character string, feature vector generating means for detecting a feature of the character string data and generating a feature vector for each document, and the document data input means Classifying means for classifying each piece of document data acquired so that the number of categories received by the number-of-category specifying means is based on the feature vector for each document, and the frequency of occurrence of words constituting the character string data Is calculated from the document group classified into each category by the classification unit, and the frequent occurrence frequency calculated by the calculation unit is high Display means for displaying words, selected word receiving means for receiving selection of words displayed by the display means, and category name determining means for determining each category name based on the words received by the selected word receiving means. Provided is a document processing apparatus.

  According to this document processing apparatus, the feature vector is generated from the feature of the character string data constituting the document data for each document data, so that the document data can be classified into a predetermined number of categories from the feature vector. In addition, the frequency of words constituting the character string data is calculated for each classified category, and a word having a high frequency is displayed to allow the user to select an appropriate word as the category name. And since the word selected by the user is determined as a category name, each category can be given a name desired by the user.

  In a preferred aspect of the present invention, there is provided storage means for associating and storing a predetermined word and a related word related thereto, and the category name determination means is a word having a high frequency of occurrence calculated by the calculation means. Corresponding related words may be detected with reference to the storage means, and the category names may be determined based on both the detected related words and the frequently occurring words.

  According to this document processing apparatus, when a word having the highest frequency in a category is not used as a category name but another word related to the word is used as a category name, these words are stored in association with each other. Therefore, it is possible to determine a predetermined word as a category name by referring to the stored content.

  Further, in a preferred aspect of the present invention, there is provided storage means for associating and storing a predetermined word and a related word related to the predetermined word, and the display means displays the frequently occurring word calculated by the calculation means. Corresponding related words are detected with reference to the storage means, and both the detected related words and the frequently-occurring words are displayed, and the selected word receiving means is configured to display the words and related words displayed by the display means. The category name determination unit may receive the selection, and may determine each category name based on the word and the related word received by the selected word reception unit.

  According to this document processing apparatus, since a predetermined word and a related word related thereto are stored in association with each other, the word having the highest frequency in the category is stored in association with the word. Words can be displayed. Therefore, the user can select a category name from a word having the highest frequency of occurrence in the category and a word stored in association with the word. That is, a category name option can be given to the user.

  In another preferred aspect, the feature vector generation means determines the frequency of occurrence in the character string data for each document for each of a plurality of predetermined words, and based on the determination result, A feature vector for each document may be generated.

  First, a document in this embodiment will be defined. There are several types of document configurations, and the separation of one document (one document) and the following one document differs depending on how the documents are configured. For example, as shown in FIG. 1A, when the description items for one document are stored on one sheet, the sheet separation becomes the document separation. As shown in FIG. 1B, when one document covers a plurality of sheets, a semantic break becomes a document break. As shown in FIG. 1C, when a plurality of documents including drawing are formed on one sheet, for example, the drawing portion becomes a document delimiter. In the present embodiment, a case will be described in which a document in which one document is constituted by one sheet shown in FIG.

  FIG. 2 is a block diagram showing a configuration of a document processing system 1 which is an embodiment of a document processing apparatus according to the present invention. The image reading device 10 in FIG. 2 is a scanner device having an automatic paper feeding mechanism such as an ADF (Auto Document Feeder), for example, and optically reads and reads a paper document set on the ADF by a user page by page. The image data corresponding to the image is transferred to the document processing apparatus 20 via the communication line 12. As the communication line 12, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or the like can be used. In this embodiment, a LAN is used as the communication line 12.

The document processing device 20 makes the image data sent from the image reading device 10 into a file, classifies it, and stores it.
FIG. 3 is a block diagram showing a hardware configuration of the document processing apparatus 20. The control unit 21 is a CPU (Central Processing Unit), for example, and centrally controls each unit of the document processing device 20 by executing various software. The communication IF unit 22 is connected to the image reading device 10 via the communication line 12, receives image data transmitted from the image reading device 10 via the communication line 12, and transfers the image data to the control unit 21.

  The display unit 23 is, for example, a liquid crystal display and its drive circuit, and displays an image corresponding to the data transferred from the control unit 21. The operation unit 24 is, for example, a keyboard provided with a plurality of operators (not shown), and outputs data corresponding to the operation contents of these operators to the control unit 21.

The storage unit 25 includes a volatile storage unit 25a and a nonvolatile storage unit 25b. The volatile storage unit 25a is, for example, a RAM (Random Access Memory), and is used as a work area in which the control unit 21 operates according to various software.
The nonvolatile storage unit 25b is, for example, a hard disk, and stores and accumulates the image data in different storage areas (for example, folders) for each document type.

  The nonvolatile storage unit 25b includes OS software for causing the control unit 21 to execute an OS (Operating System), and document classification for causing the control unit 21 to implement functions unique to the document processing apparatus 20 according to the present embodiment. Software is stored. The control unit 21 executes the document classification software to extract character string data from the image data representing the document transferred from the image reading apparatus 10, performs morphological analysis on the extracted character string data, and the character string Extract the words that make up. A feature vector for each document is generated based on the extracted word. The feature vector is determined when the presence or absence of appearance in the extracted character string data is determined for each of a plurality of predetermined words (for example, predetermined hundreds of thousands of words). , A vector composed of elements having a value of 0 when they do not appear. For example, the plurality of predetermined words are (“love”, “reverse”, “National Diet”, “home run”), and the character string represented by the character string data is “a reverse full home run popped out in the last round” In this case, (0, 1, 0, 1) is generated as a feature vector.

  Then, using this feature vector, the image data transferred from the image reading apparatus 10 is classified into a predetermined number of categories. In the present embodiment, an SVM algorithm with high classification accuracy (see Non-Patent Document 1 and Non-Patent Document 2) is used as the predetermined algorithm.

  Further, the related word table shown in FIG. 4 is stored in the nonvolatile storage unit 25b. As shown in FIG. 4, the related word table is configured by a “word” field and a “related word” field, and the expressions are different, but the words used by the user with the same meaning are stored in association with each other. . For example, in “contact form”, “continuity confirmation”, and “contact confirmation form” used in the same meaning, when the control unit 21 performs character recognition of image data representing each word, the form of the character is different. Therefore, the control unit 21 recognizes these words as different ones. However, from the related word table, the control unit 21 recognizes that these words are related. In the present embodiment, it is assumed that the related word table is input in advance by the user.

  Next, an operation performed by the control unit 21 will be described. In the following, the control unit 21 of the document processing apparatus 20 operates according to the OS software and waits for a user to perform some input operation.

  When the user appropriately operates the operation unit 24 and performs an input operation for instructing execution of the document classification software, the control unit 21 reads the document classification software from the nonvolatile storage unit 25b and executes it. Hereinafter, operations performed by the control unit 21 in accordance with the document classification software will be described with reference to the drawings. In the following description, it is assumed that the user desires to classify 100 documents into five categories.

  FIG. 5 is a flowchart showing the operation performed by the control unit 21 according to the document classification software. First, the control unit 21 displays the category number input screen 30 shown in FIG. 6 on the display unit 23, and allows the user to specify the number of categories (step SA10). As shown in FIG. 6, the category number input screen 30 includes an input box 31 and a “decision” button 32. The user who visually recognizes the category number input screen 30 operates the operation unit 24, inputs the category number “5” in the category number input box 31, and presses the enter button 32 to determine the desired number of categories. be able to. The control unit 21 receives the category number data representing the user's operation content from the operation unit 24, and based on the category number data, specifies that the category number (n) designated by the user is “5”, Set “n = 5”.

  Next, the control unit 21 writes and stores the category number data transferred from the operation unit 24 in the volatile storage unit 25a, and waits for image data to be sent from the image reading apparatus 10. On the other hand, when the user sets a paper document in the ADF of the image reading apparatus 10 and performs a predetermined operation, images representing the description content of the paper document are read one by one by the image reading apparatus 10 and images corresponding to the respective images are read. Data is sequentially transferred from the image reading apparatus 10 to the document processing apparatus 20 via the communication line 12. At this time, as described above, since 100 documents are set by the user in the ADF of the image reading apparatus 10, the image data of 100 documents is sequentially transferred to the document processing apparatus 20.

  When the control unit 21 receives the image data sent from the image reading device 10 via the communication IF unit 22 (step SA12), the character string described in each image data is converted into a document unit (one sheet unit). For example, read character string data is extracted by OCR (Optical Character Recognition) or the like (step SA14). At this time, if the control unit 21 determines that the read character string data is inappropriate as Japanese using NLP (Neuro Linguistic Programming), the control unit 21 corrects the character string data to an appropriate character string. That is, for example, when an OCR process is performed on a document including the word “contact”, the “contact” may be extracted as character string data “linked” due to a font shift or the like. In such a case, the control unit 21 corrects the character string data extracted as “concatenation” to “contact” by this NLP.

  Based on the character string data extracted through such processing, the control unit 21 generates the above-described feature vector (step SA16). In the present embodiment, 100 feature vectors are generated corresponding to 100 documents transferred from the image reading apparatus 10.

  And the control part 21 classifies 100 image data according to a predetermined algorithm using the feature vector produced | generated by step SA16 (step SA18). At this time, since the number of categories instructed by the user in step SA10 is n = 5, the control unit 21 classifies 100 pieces of image data into five using the SVM algorithm. Hereinafter, the categories classified into five are referred to as categories 1 to 5 for convenience.

  Next, the control unit 21 sets a counter (not shown) to “m = 0” (step SA20). Then, in the character string data in the category classified into category 1, the word frequency is calculated (step SA22). Assume that the calculation results at this time are “contact”, “application”, and “general affairs department” in order from the word with the highest frequency of occurrence. The control unit 21 displays the category name determination screen 40 shown in FIG. 7 based on these words (step SA24). In the present embodiment, the category name determination screen 40 displays three words in descending order of frequency of occurrence, that is, “contact”, “application”, and “general affairs department”. Further, the control unit 21 refers to the “word” field of the related word table shown in FIG. 4 based on each word displayed on the display unit 23, and if there is a corresponding word, “ The words stored in the “related words” field are displayed in parallel. In FIG. 7, “continuity confirmation” and “contact confirmation” stored in association with “contact” having the highest frequency of occurrence are displayed in parallel with “contact”.

  On the category name determination screen 40, the user operates the operation unit 24 and moves the cursor to select a word. In this way, the control unit 21 displays a plurality of words with a high frequency of occurrence, so that the user can grasp words included in a character string for each category. Therefore, an appropriate category name can be determined. Furthermore, since the user can grasp a plurality of category name candidates, the choice of category name to be determined is wide for the user. In the present embodiment, the selected word is shown in reverse video. On the initial screen of the category name determination screen 40, the first word is highlighted. FIG. 7 shows a case where “Contact” is highlighted.

  When the “determine” button 41 on the category name determination screen 40 is pressed by the user (step SA26; YES), the control unit 21 specifies the highlighted word at that time as the category name. On the other hand, when the “modify” button 42 is pressed (step SA26; NO), the control unit 21 displays the category name input screen 50 shown in FIG. As shown in FIG. 8, a category name input box 51 is provided on the category name input screen 50. When the operation unit 24 is operated by the user and a category name is input to the category name input box 51 (step SA30), the control unit 21 specifies the input word as the category name.

  Next, the controller 21 increments the counter to “m = m + 1” (step SA28), and determines whether m is equal to or greater than n representing the number of categories. In this determination, if the counter value is less than n (step SA32; NO), the category names for categories 2 to 5 are not specified, so the processing after step SA22 is continued, and the counter value reaches n. If so (step SA32; YES), the control unit 21 ends this routine.

  As described above, in the present embodiment, the image data is classified into categories based on the feature vectors, and the user can determine the category name based on the frequent words for each classified category. Therefore, a uniform category name can be set even if the operating user is different. In addition, because the related word table stores words that are differently expressed but used in the same meaning, for example, in the category name determination screen 40 shown in FIG. Is “confirmed” instead of “contact” with the highest frequency, the control unit 21 displays the “confirmed” in association with the “contact” so that it can be selected. It is possible to save the trouble of inputting characters again.

  In the present embodiment, when the category name is determined, the control unit 21 displays a frequent word on the category name determination screen 40 illustrated in FIG. 7 and allows the user to select a category name. The listed table is stored in the storage unit 25 as an initial set (hereinafter referred to as an initial set table), the initial set table is searched based on frequent words, and the corresponding words are automatically assigned as category names. Also good. If the corresponding word does not exist in the initial set table, the category name determination screen 40 shown in FIG. 7 is displayed to allow the user to select it. Further, at this time, the control unit 21 may be provided with a learning function so that the category name selected by the user and a word with a high frequency of occurrence are associated and stored.

  The initial set table as described above may be provided separately from the related word table shown in FIG. 4 or may be provided together. In the case of providing them together, if there is a related word for the word stored in the “word” field in the “related word” field of the related word table, the related word is stored and the related word is stored. If there is no, leave blank. In this way, when automatically assigning category names, an initial set table that stores related words related to each word is used as an initial set. There is no need to reconfigure the set. In other words, even if a document referred to as “Contact” in Company A is referred to as “Continuous” in Company B, an initial set for Company A and an initial set for Company B can be obtained by using a related word table. There is no need to set. Therefore, it is possible to reduce the cost when resetting the initial set.

  In the present embodiment, the case where the element of the feature vector is a word is shown, but the appearance of the document may be used as the element. That is, layout analysis is performed on the image data, a character string is recognized, and a feature vector is generated from the description position or character string length of the recognized character string.

  Further, in the present embodiment, 100 documents used by the user are shown with one sheet as a document delimiter unit. For example, the document having the configuration shown in FIGS. 1B and 1C is used. You may be made to do. At this time, the control unit 21 recognizes the semantic division by the above-described layout analysis, semantic analysis, symbolic analysis, or the like, or recognizes the position of the paragraph or the drawing, thereby dividing the document unit. It may be. Further, the operation unit 24 allows the user to input a specific character or symbol as a reference for document separation, the control unit 21 detects the character or symbol in the document, and recognizes the detected position as the document separation. Also good. In short, a document having any configuration may be used as long as a break is represented. Then, the control unit 21 specifies the range of one document for each segment and performs the operation shown in FIG.

  Further, in this embodiment, after classifying a document into categories, the frequency of word occurrence for each category is calculated. However, the frequency of word occurrence may be calculated when generating a feature vector. That is, when generating a feature vector, only the presence / absence of a corresponding word in the character string data is searched for a plurality of predetermined words. The frequency of frequent occurrences may be calculated.

  In the present embodiment, the case where the image reading apparatus 10 and the document processing apparatus 20 are configured as separate hardware has been described, but both may be configured as integral hardware. At this time, the communication line 12 serves as an internal bus for connecting the image reading apparatus 10 and the document processing apparatus 20 within the hardware.

  Further, in the present embodiment, a case has been described in which software for realizing the function unique to the document processing apparatus 20 according to the present invention in the control unit 21 is stored in advance in the nonvolatile storage unit 25b. However, the software may be installed in a computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory) or a DVD (Digital Versatile Disk). In this way, a general computer device can function as the document processing device according to the present invention.

It is the figure which illustrated the format of the structure of a document. 1 is a block diagram showing an overall configuration of a document processing system according to an embodiment of the present invention. It is a block diagram which shows an example of the hardware constitutions of a document processing apparatus. It is the figure which illustrated the related word table. It is the figure which showed the flowchart which showed the operation | movement which the control part of a document processing apparatus performs. It is the figure which illustrated the category number input screen which a display part displays. It is the figure which illustrated the category name determination screen which a display part displays. It is the figure which illustrated the category name input screen which a display part displays.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Document search system, 10 ... Image reading apparatus, 12 ... Communication line, 20 ... Document processing apparatus, 21 ... Control part, 22 ... Communication IF part, 23 ... Display unit, 24 ... operation unit, 25 ... storage unit, 25a ... volatile storage unit, 25b ... non-volatile storage unit, 26 ... bus, 30 ... category number input screen , 40 ... Category name determination screen, 50 ... Category name input screen

Claims (5)

Document data input means for obtaining document data obtained by digitizing a document;
Category number specifying means for receiving the number of categories for classifying the document data;
Character string data generating means for analyzing the document data acquired by the document data input means and generating character string data representing a character string;
Feature vector generation means for detecting features of the character string data and generating a feature vector for each document;
Classifying means for classifying each document data acquired by the document data input means so that the number of categories received by the category number specifying means is based on the feature vector for each document;
Calculating means for calculating the frequency of occurrence of words constituting the character string data from the document group classified into each category by the classification means;
A document processing apparatus comprising: category name determining means for determining each category name based on a word with high frequency calculated by the calculating means. Document data input means for obtaining document data obtained by digitizing a document;
Category number specifying means for receiving the number of categories for classifying the document data;
Character string data generating means for analyzing the document data acquired by the document data input means and generating character string data representing a character string;
Feature vector generation means for detecting features of the character string data and generating a feature vector for each document;
Classifying means for classifying each document data acquired by the document data input means so that the number of categories received by the category number specifying means is based on the feature vector for each document;
Calculating means for calculating the frequency of occurrence of words constituting the character string data from the document group classified into each category by the classification means;
Display means for displaying words with a high frequency of occurrence calculated by the calculation means;
Selection word reception means for receiving selection of the word displayed by the display means;
A document processing apparatus comprising: category name determining means for determining each category name based on a word received by the selected word receiving means. Storage means for storing a predetermined word and a related word related thereto in association with each other;
The category name determining unit detects a related word corresponding to the frequently occurring word calculated by the calculating unit with reference to the storage unit, and based on both the detected related word and the frequently occurring word. The document processing apparatus according to claim 1, wherein each category name is determined. Storage means for storing a predetermined word and a related word related thereto in association with each other;
The display means detects a related word corresponding to a frequently occurring word calculated by the calculating means with reference to the storage means, and displays both the detected related word and the frequently occurring word,
The selected word accepting unit accepts selection of a word displayed by the display unit and a related word,
3. The document processing apparatus according to claim 2, wherein the category name determining unit determines each category name based on a word received by the selected word receiving unit and a related word.   The feature vector generation means determines a frequency of occurrence in the character string data for each document for each of a plurality of predetermined words, and generates a feature vector for each document based on the determination result. The document processing apparatus according to claim 1, wherein the document processing apparatus is a document processing apparatus.

Images