JPH05324730A - Document information retrieving device - Google Patents

Abstract

PURPOSE:To decrease the volume of a registered document and to speed up retrieval processing by compressing and converting input document information into compressed data and registering them at the time of document information registration. and compressing and converting input key information into compressed key data and performing the retrieval. CONSTITUTION:A character string collation part 111 collates text data analyzed by a text data analysis part 102 with text word data registered in a code conversion dictionary 104 to obtain a compressed code of word level. The collation result is transferred to a text data compression part 103, which compresses and registers the text in a text data file 106. When a key word for text retrieval and a retrieval system are inputted, they are transferred to a retrieval system analysis part 109, and the text data compression part 103 generates the compressed code of the key word. Text data take out of the text data file 106 is collated by a character collation part 111 with the key word at compression code level.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information retrieval system, and more particularly to a full-text retrieval system based on a text search system (full-text search system), which can be freely configured according to the scale and performance required of the system. The present invention relates to a document information search device having a configurable architecture, which is suitable for a free-text full-text search. It can be applied to database systems, document filing systems and word processors.

[0002]

2. Description of the Related Art In recent years, database services such as literature information and patent information have become widespread, and there is a tendency for them to become larger and more versatile. As a result, full-text search (full-text search), which enables direct reference to the text of documents such as literature information and patent information (full-text search) instead of the conventional search method using keywords or classification codes The system is also called text search).

As the name implies, a full-text search system handles the text itself of a document as search information, and enormous effort and index associated with index registration are always problems in searches using indexes such as keywords and classification codes. It is a technology that can fundamentally solve the harmful effects of search using an index such as search omission due to the difference between the registrant and the document searcher.

However, the full-text search system has some problems that do not occur in a search system using an index. The biggest problem among them is the problem of search time. Since the full-text search system searches the document text itself, it is not practical to apply it to the search of the database service information etc. which has been handled so far. For example, 2 per document
In order to perform a full-text search for 20,000 documents with a capacity of 0 KB, 400 MB of data must be searched. Even if the data is read at an execution speed of 1 MB / s on average and collation is performed at the same speed, it takes about 7 minutes to complete the search.

As a conventional device for solving the above problems,
An apparatus described in Japanese Patent Laid-Open No. 03-174652 can be mentioned. This conventional technique accelerates the reading speed of the text data by dividing / registering the document text data on a plurality of magnetic disks and extracting the text data in parallel from these magnetic disks. At the same time, create a table of the characters that appear in the text, eliminate adjuncts such as particles and conjunctions, and create a data file (repeated condensed text) in which words that appear repeatedly are excluded, and perform a full-text search. By performing a two-step pre-search before, a search speed that can be practically used is realized.

[0006]

As described above, according to the above-mentioned prior art, it is possible to complete the search processing of a document text having a huge amount of data in a time that can be practically used. This is a very useful technique for realizing a full-text search system.

However, in the above-mentioned conventional technique, the two-step pre-search method is used in order to improve the speed of searching the text data. However, in order to perform the pre-search at the time of searching, the condensed text and the character are preliminarily extracted from the text data of the text. It is necessary to create a component table and store it in the document data storage means (for example, a magnetic disk) together with the text data. This causes a problem that the document data increases by the capacity of the condensed body and the character component table in addition to the text data. Furthermore, since the pre-search method is not fundamentally a text search, a text search is indispensable for deriving the final search result. In the worst case, when the documents cannot be narrowed down by the pre-search, all the text search is performed again. The search process will be performed on the text of. Therefore, only the sum of the search processing time required for the pre-search and the search time for all the texts is required for processing, which not only leads to a reduction in the search time but also causes an increase in the search time. ..

An object of the present invention is to solve the above problems of the prior art by improving the search speed of text data, at the same time not increasing the amount of document data, and requiring a re-search of the text of the text. It is to provide a document information search device that realizes an excellent full-text search system.

[0009]

In order to achieve the above object, the document information retrieval apparatus according to the present invention focuses on a text data compression method.

According to the present invention, document information input means for inputting document information, text data storage means for registering and storing the document information, key information for searching the registered document information, and search conditions. Input means, a character string collating means for collating the key information with registered document information, a search expression converting means for generating a search condition expression from the search condition, and document information conforming to the search condition expression In a document information retrieval device having a search condition determining means for determining the above and an output means for outputting the result determined by the search condition determining means, a text data compression means for compressing and converting the input document information into compressed text data. , Text data decompression means for decompressing the compressed text data into original document information, wherein the text data storage means registers the compressed text data. Storing Te, the retrieval formula conversion means,
The input key information is compressed and converted into compressed key data to generate a search condition expression from the search condition, and the character string collating means collates the compressed key data with the registered compressed text data to determine the search condition. The means determines the compressed text data of the document information that conforms to the search condition expression given by the search expression conversion means based on the matching result output from the character string matching means, and the text data expansion means , The compressed text data of the matched document information is expanded to the original document information.

Further, it has a text data analysis unit which expands the input document information for each word that can be regarded as one semantic unit, and the text data compression means assigns a compression code to each expanded word. Compress and convert to compressed text data.

Further, it further comprises a code conversion dictionary for storing a word and a compression code that can be regarded as one semantic unit in a pair, and the text data compression means can perform compression conversion by referring to the code conversion dictionary. it can. The input means receives a word to be registered in the code conversion dictionary, and the code conversion dictionary registers the word to be registered.

The text data analysis section can perform a syntax analysis of the input document information and develop it into each word.

Further, a synonym / synonym table for storing synonyms / synonyms groups having different notations and compressed codes as a pair, storing identification information for each synonym / synonyms having different notations and storing the text, The data compression means refers to the synonym / synonym table and assigns one compression code to synonyms / synonyms groups different in notation for words in the original document text, and stores information indicating that the words are synonyms / synonyms. The identification information may be added to perform compression conversion.

The search formula conversion means compresses the input key information into compressed key data by referring to a synonym / synonym table, and generates a search condition formula from the search condition,
The character string collating means detects information indicating a synonym / synonym, skips the identification information, and collates the compressed key data with the registered compressed text data.

Further, the text data decompressing means decompresses the compressed text data to be collated at the time of collating the key information, and the character string collating means collates the key information with the decompressed document information. Good.

The text data analysis unit expands the words that are likely to be misread when the words are misread, and the text data compression means compresses the compressed code for all the expanded words. Is assigned and compressed into compressed text data, and at the time of search, the character string collating means can collate all the listed words with the compressed key data.

The text data analysis unit recognizes a portion where misalignment of words occurs, the text data compression unit does not perform compression conversion of a portion where misalignment of words occurs, and the text data storage unit does not The part where misalignment is generated is stored as text data, and when searching,
The character string collating means may also collate the key information with the text data.

The search expression conversion means recognizes a portion of the input key information in which the misreading of words occurs, and expands the words that are likely to be misread when the misreading of words occurs. , A compressed code is assigned to all the expanded words, and compressed and converted into compressed key data to generate a search condition expression from the search condition, and at the time of searching, the character string collating means causes the compressed text data for all the compressed key data. Match with the data.

A plurality of types of code conversion dictionaries are provided, the input means receives an instruction to select a dictionary to be used according to the type of document, and the text data compression means refers to the instructed code conversion dictionary to perform compression conversion. Then, add the code conversion dictionary identification information used for the compressed text after conversion,
When decompressing the text data, the text data decompression means can decompress using the corresponding code conversion dictionary by referring to the code conversion dictionary identification information. in this case,
The search expression conversion means compresses and converts the input key information into compressed key data with reference to the designated code conversion dictionary.

The text data storage means registers and stores the compressed text data together with the identifier information of the compressed text data, and the search condition discrimination means stores the compressed text data of the matched document in the text data decompression means. Instead of decompressing and outputting the original document information, the identifier information of the document is output.

Image data storage means for registering image data related to the document information at the time of registering the document information is further provided, and the search condition determining means uses the compressed text data of the matched document as the original in the text data decompressing means. Instead of decompressing and outputting the document information, the image data related to the document information may be output.

Further, an input device for inputting the document information, a storage device for storing the document information, a key input means for receiving key information for searching the document information, a central processing unit for searching the document information, In an information processing system having an output device for outputting a search result, the central processing unit compresses and converts the document information input by the input device into compressed text data and registers the compressed text data in a storage device, and the key input means. It is possible to compress and convert the key information input in step 1 into compressed key data, and collate the compressed key data with the compressed text data to search for compatible compressed text data. The CPU further includes a code conversion dictionary that stores a word that can be regarded as one semantic unit and a compression code in a pair, and the central processing unit can perform compression conversion by referring to the code conversion dictionary.

[0024]

In the present invention, when implementing a full-text search system, speedup is not required so much at the time of registration, but it is essential to shorten the processing time at the time of search, and there is a method of compressing a document text as one means for realizing it. In addition, points are used, and the minimum unit of meaning of document text is not words but words (nouns, verbs, conjunctions, etc.), and it is possible to compress the document text by using it, and use this.

A specific document information registration / retrieval method using the document compression method will be described below.

First, text data is input by the document information input means. The document information input means includes a method of reading from other information equipment through an interface, a method of reading paper surface information using a scanner and OCR, and a method of inputting from a keyboard. Next, the text data compression means compresses this text data. The text data compression means is a code conversion in which the text data analysis unit stores words expanded into word codes (arrangement of character codes) that can regard the original document information as one semantic unit and compressed codes corresponding to the words. The compression process is realized by collating the original document text with this dictionary using a dictionary and converting the compression code. Further, the synonym / synonym table may be used to collate the original document text with the synonym / synonym table, and compression code conversion may be performed to perform compression processing. Further, at the time of text data search, the search expression conversion means expands the input keyword as a synonym / synonym and then performs compression code conversion processing on this keyword as well. The character string collating unit collates the compressed / accumulated compressed text with the compressed keyword, and the search condition discriminating unit converts the search formula based on the collation result output from the character string collating unit. The compressed text data of the document information that matches the search condition expression given by the means is discriminated and the search processing is performed. As a result, advanced search processing is realized without returning to the original document code. The text data decompression means decompresses and outputs compressed text data of the matched document information again. Finally, the compressed text data is taken out according to the search and collation result, decompressed to the original document code, and then output using a display means such as a CRT or a printer.

Thus, according to the present invention, by using the text data compression means utilizing the word / compression code conversion, it is possible to reduce a huge amount of text data without reducing the information of the original document. Further, it becomes possible to support the advanced search processing without decompressing the text into the original document at the time of document search, and in particular, it is possible to provide a faster and more reliable document information search device than the conventional method for the application of the full text search system. ..

[0028]

Embodiments of the present invention will be described below with reference to the drawings.

First, FIG. 1 shows the configuration of a document information retrieval apparatus as an example of an apparatus to which the present invention is applied. In FIG. 1, 101 is a text / image data input unit, which takes in text and image data via a network or various interfaces. 102
Is a text data analysis unit, which analyzes input text data. A text data compression unit 103 compresses the text code according to the text data analysis result. A code conversion dictionary 104 holds a code data pair of a compressed code and an original character code when the code is compressed / decompressed. A synonym / synonym dictionary 105 holds synonyms / synonyms of words. A text data file 106 holds compressed text data. An image data file 107 holds image data attached to the text data. 10
Reference numeral 8 denotes an input means such as a keyboard, which inputs text data of document information at the time of document information registration, and inputs a keyword (key information) / search condition at the time of text search. A search expression analysis unit 109 separates or analyzes the input search expression into keywords and search conditions. Reference numeral 110 is a search formula conversion unit, which expands the input keyword according to the search condition and further converts it into a search word (compressed key data) suitable for the compressed text data search.
Reference numeral 111 denotes a character string collating unit, which performs matching between the input text data at the time of text registration and the words accumulated in the code conversion dictionary 104, and at the time of text search, searches the generated search word from the text data file. Matches with the read compressed text data. A search condition determination unit 112 determines a document that matches the search condition expression obtained by the search expression analysis unit with respect to the matching result obtained by the character string matching unit. Reference numeral 113 is a text data decompression unit that extracts compressed text data from the text data file and decompresses the text data with respect to the search result obtained by the search condition determination unit. 114
Is a display means such as a CRT display, and displays text and image data. A display control circuit 115 controls the display 114. Reference numeral 116 is an output means such as a printer, which can actually make a hard copy of text and image data on paper. A printer control circuit 117 controls the printer 116. 118 is the text /
An image bus that exchanges data with each other. An image input unit 119, such as a scanner, captures image information as data. A character recognition unit 120, such as an OCR, extracts character information from the captured image data and reads it.

Next, the operation of each part in FIG. 1 will be divided into a text data registration time and a text data search time, and a flow chart (problem analysis diagram, Prob
rem-Anarysis-Diagram, hereinafter referred to as PAD).

FIG. 2 shows a flowchart (PAD) which is an embodiment of the text data registration system according to the present invention.

To register text data, first, text data is input (S202). The text data to be input may be data read from another information device via the network or various interfaces using the text / image data input unit 101, or image data input using the image input unit 119 such as a scanner may be OC.
It may be input after being converted into a character code using the character recognition unit 120 such as R, or may be one in which text data is directly input using the keyboard 108. Next, the text data is analyzed as a preprocess for compressing the input text data (S203). This is performed by the text data analysis unit 102, and this portion prevents occurrence of word matching deviation when matching with a dictionary required in the next step. As a processing method, Japanese parsing processing is used, a method of separating / recognizing words / predicates / conjunctions, etc., and recognizing only a portion where a word matching error occurs
There is a method of giving a recognition result to a character string collating unit at the time of dictionary collation and removing a deviation at the time of collation. In the method of separating / recognizing words / predicates / conjunctions, etc., Japanese analysis processing is applied to the input text data to recognize words / predicates / conjunctions etc. separately. A method of recognizing only the portion where the word matching error occurs will be described later. Furthermore, the code conversion dictionary 1 is added to the document for which the analysis of the text data is completed.
04 is collated to generate a compressed code for the analysis result (S204). The code conversion dictionary 104 stores the word code of the original document text and the corresponding compressed code as 1
The dictionary is made to correspond to the pair 1 and is stored on the magnetic disk in correspondence with the size of the dictionary or is arranged on the semiconductor memory in consideration of speeding up. The text data analyzed by the text data analysis unit 102 is collated by the character string collation unit 111 with the text word data registered in the code conversion dictionary 104,
A compressed code at the word level is obtained as the matching result.
The matching result generated by the character string matching unit 111 is transferred to the text data compression unit 103. The text data compression unit 103 uses the analysis result from the text data analysis unit 102 and the above collation result (that is, the compression code at the word level) to perform text compression processing (S20).
5). Finally, the compressed and converted text data is
It is registered in the text data file 106 (S20
6). For the text data registration, an optical disk can be used to cope with the increase in capacity, or a magnetic disk or the like can be used in consideration of the search speed. The above registration process is repeated until the text data ends (S201).

In the above embodiment, the image data related to the text data is fetched (S202), the image data is registered together with the text data in the text data file 106, or the text data is stored in another file (image data file 107). It is also possible to register image data together with it. The management information of the image data related to each text data is attached to the text data and saved.

In the above embodiment, the synonym / synonym dictionary 105 is used at the time of character string matching between the text data and the code conversion dictionary 104, and the synonym or synonym is
It is also possible to regard the same meaning contents at the word level and to assign one code to the synonyms or synonyms in the original text data in the compressed text data.

Next, the code assignment and retrieval of the synonyms and the like will be described with reference to FIGS. 6, 7 and 8.
FIG. 6 shows an example of the synonym table, FIG. 7 shows an example of code allocation for the synonym compressed text conversion, and FIG. 8 shows an example of the synonym compressed keyword conversion. In order to assign one code in the compressed text data to a synonym or a synonym in the original text data, a synonym table as shown in FIG. 6 is provided in the synonym / synonym dictionary 105 or as the code conversion dictionary 104. Set up. In the synonym table, one compression code is assigned to a plurality of synonyms, and the ID number of the synonym is shown after the compression code. For example, as shown in FIG.
When the original text data such as "in the field of large computer" is input, the compressed text conversion assigns the compression code (30A2). After that, the special code (FFF0) indicates that the next code is a synonym ID, and then the synonym ID number (0001).
, And register these as compressed text. When searching, synonyms are treated as the same keyword, and synonym ID
Is skipped and matching is performed. For example, when the keyword “large-scale computer field” as shown in FIG. 8 is given, the keyword is expanded using the synonym / synonym dictionary 105 and the corresponding compressed code is assigned. When collating the compressed keyword and the compressed text data, the next synonym ID is skipped by the special code (FFF0) and collation is performed. Alternatively, at the time of conversion into a compressed keyword, it is detected that a synonym exists, and when the compression code having the synonym is collated, the subsequent special code (FFF0) and the synonym ID number are skipped. You may make it collate by doing. Further, at the time of decompression, the synonym ID number and the synonym table are used to restore the original text data.

By the flow as described above, the text data can be assigned to the compression code and the compressed text data can be registered in the text data file or the image data file.

Next, an operation flow for searching text data will be described with reference to FIG. In Figure 3,
1 is a flowchart (PAD) of an embodiment of a text data search method according to the present invention.

First, a keyword for text search and a search expression are input (S301). The keyboard 108 may be used to input the keyword and the search formula, or the text input unit 101 may be used to input the keyword and the search formula via an interface from various information devices. Further, various kinds of paper surface information such as printed matter may be input using the image input unit 119 such as a scanner, and the character code information obtained by converting the image data using the character recognition unit 120 such as OCR may be used as a keyword and a search formula. Next, the input one or more keywords and the search expression are transferred to the search expression analysis unit 109, the keywords are separated, and the search conditions between the respective keywords are analyzed / recognized (S302). According to the search conditions and the like, synonym / synonym expansion is performed on these separated keyword groups using the synonym / synonym dictionary 105 as described above (S303), or error-allowed character string expansion (input A keyword expansion process that enables a search that allows an input character error of one character or a designated character is performed on the keyword (S304). After the keyword expansion is performed, the keyword analysis process is performed as a pre-process of the compression code conversion as in the text data registration (S305). The content of the processing is to prevent the word matching deviation from occurring in the dictionary matching in the next step, and is performed by the text data analysis unit 102 as in the registration. Next, the keyword analysis result is collated with the code conversion dictionary (S306), and the compressed code of the keyword is generated by using the compressed code which is the collated result and the keyword analysis processing result (S307). The code conversion dictionary 104 is used for code conversion, and the character string collating unit 111 is used.
A keyword is collated with a code conversion dictionary, and the text data compression unit 103 performs a keyword compression process using the result to generate a keyword compression code.

Since the compression conversion of the keyword is performed in the above processing, next, the character string collation between the keyword and the text data at the compression code level is performed (S308). The compressed text data is the text data file 10
The text data taken out from No. 6 is collated with the keyword in the character string collating unit 111. The present invention is characterized in that both the keyword and the text data are compressed using the same method, and the character string search / collation is directly performed on the compressed text data. As a result, the process of decompressing the original document text at the time of collating the text data is not required, and the speed / simplification of the text collation can be achieved.
The character string collation result of the keyword and the text data is transferred to the search condition determination unit 112, and the collation result is determined according to the search condition obtained by the search expression analysis unit 109 (S309). A document or a document portion that matches the search condition is transferred from the character string collating unit 111 or is extracted from the text data file 106 again, and is input to the text data decompressing unit 113 (S310). The text data decompression unit decompresses the input compressed text data into original document text data, displays the text on the CRT display 114 from the display control circuit 115, or the text is displayed on the printer 116 via the printer control circuit 117. Print out on paper.

In the above-described embodiment, it may not be possible to eliminate all word matching deviations in the analysis processing when registering text data or inputting a search word. For example,
As shown in Fig. 9 (a), "the past, whatever the present"
When the words are duplicated as shown in, there is a possibility that the matching shift occurs. As a method for solving this in the text data analysis processing unit, in the text data search, in the search word analysis processing (S305), all the keywords that may occur due to misalignment are expanded / compressed and code conversion is performed. A method of matching the expanded keyword compression code with text data, at the time of text data registration,
Regarding the words that are likely to be misaligned, there is a method of registering some possible candidates with a mark that makes it possible to judge that they are misaligned candidates at the time of searching. There is a method in which the original document text code is registered without performing the compression code conversion, and at the time of text search, the keyword is expanded into both the compressed code and the original document text code to perform the search.

In the above method, when misaligned reading of keywords occurs during retrieval, all the words that can be read for the keywords are listed, and all of them are subjected to compression code conversion. In the case of “past past or present anytime” in the example shown in FIG. 9A, misalignment may occur in the portion “that is anyway”, so “that is ...
Compressed code conversion is performed for "Anyone" and "Are. Text data matching is performed on these expanded compressed codes.

With regard to the above method, when registering text data, a predetermined code is added before and after a word having a possibility of misalignment to clarify that there is a possibility of collation misalignment. For example, as shown in FIG. 9B, (FFFC) is used as the collation shift start code, and the first candidate compression code is described. Next, the second candidate is described by using (FFFF) as another candidate start instruction. further,
When there are other candidates, similarly, other candidates are described using (FFFF). Finally, the text data is registered using (FFFE) as the collation deviation end code. Next, the search method in the above case will be described with reference to FIGS. 10 and 11. FIG. 10 shows a state transition diagram of keyword conversion for mismatched text search. 11
Shows a mismatched text search algorithm.
As described above, the search for the compressed text including the collation error includes the special codes (FFFC), (FFFF), (FFF).
Since E) is used, it is necessary to perform special processing at the time of search when these special codes are present. In FIG. 10, in the case of collation of a normal compressed text which is not a collation deviation text as a keyword, a state transition ⇒ ⇒ transition is made to collate a character string. If the collation deviation text appears during the collation, the transition of and will be dealt with. Further, when the collation with the keyword is started in the middle of the text collation shift, it can be dealt with by the state and. In FIG. 10, as keywords
When "Anyway present" is input, the keyword is converted into a compression code and collated as a compression keyword.
According to the flow shown in FIG. 11, the compressed text is read one by one and collated. When (FFFC) of a predetermined code is read out in the compressed text, a plurality of candidates are read out as a compressed code having a possibility of collation deviation, and collation with each compressed keyword is performed.

In the above method, the original document text code is registered without performing compression code conversion for words that may be misread. When searching for text,
The keyword is expanded into both compressed code and original document text code, and both are searched to prevent the possibility of misreading.

By using the above method, the text data analysis process (S203) is omitted when the text data analysis unit 102 and the text data are registered, and the search word analysis process (S305) is omitted when the text data is searched.
The compressed code can be generated only by matching with the code conversion dictionary 104.

Further, in the above embodiment, it is possible to change the type of the code conversion dictionary according to the target text and to provide a plurality of types. For example, a code conversion dictionary corresponding to English text, a technical term corresponding dictionary, and a general office term corresponding dictionary can be considered. Which dictionary to use can be selected according to the type of document. This is selected by accepting a selection instruction at the input means 108. At this time, at the time of document text compression and document search /
The same dictionary is used for decompression. It is assumed that the code conversion dictionary is managed by adding dictionary management information to the compressed text data and storing it in the text data file 106 as to what dictionary was used for compression.

Further, in the above embodiment, the image data registered along with the text at the time of text data registration is retrieved from the image data file 107 together with the text according to the text retrieval result and displayed on the display 114 or displayed on the printer 116. It is also possible to print by using. As a text search result, instead of displaying the original text data on the display 114 or the printer 116, a method of displaying text management information such as a file name / index containing the original text data, or the original text data There is a method of displaying information such as image data related to. As a method of displaying the management information, at least one of the file name / index of the original text data, the file number, the file capacity, etc. is registered and stored in the text data file 106 as identifier information, and the search condition determination unit 112 Instead of decompressing the compressed text data of the matched document, the identifier information of the document may be output. As a method of displaying information such as image data, instead of storing the image data related to the document information in the image data file 107 and causing the search condition determination unit 112 to expand the compressed text data of the matched document, Image data related to the document information may be output.

By using the above method, it is not necessary to decompress the compressed text data into the original document text data again.

Further, in the above-described embodiment, when the character strings of the text data and the keyword are collated (S308), the distance between characters is designated (search for designating the number of characters contained in the appearance of two or more words), etc. In order to support search conditions that are difficult to support with text, the compressed text read from the text data file is decompressed into the original document text by the text data decompression unit 113 without using the technique of applying the compression code conversion to the search word. It is also possible to adopt a configuration in which this and character strings are collated with a keyword. At this time, it is not necessary to perform the search word analysis process (S305), collate the search word with the code conversion dictionary (S306), and perform the keyword compression process (S307). When matching the character string, the keyboard 108
From, it is possible to select and instruct whether to match the original document text after decompression or the compressed text.

Further, in the above embodiment, it is possible that the display control circuit and the printer control circuit perform the text data expansion. At this time, the text data decompression unit 113 is included in the display control circuit 115 and the printer control circuit 117.

FIG. 4 shows an embodiment of text compression.

FIG. 4A shows a part of the original document text. Here, the text “that is, the block diagram of the document information search device” is taken as an example. Conventionally, one code is given to one character of the original document text. For example, a shift JIS code generally used as a text code is provided with one 16-bit code for each character, and 19 16-bit codes are required in the conventional example. However, in consideration of the semantic unit of each word, the minimum unit of meaning is each word (idiom, conjunction, etc.). In this example, "that is,""document,""information,""search,""device,""block," etc. correspond to this. Therefore, the text data analysis unit 102 expands the word into the minimum unit of meaning, and the text data compression unit 103 allocates a code based on the code conversion dictionary 104 to compress the document text. In this example, as shown in FIG. 4 (b), "that is" → (01AB), "document" → (2153), "block" → (0A3
A 16-bit code shall be assigned to each word, such as B). In the code conversion dictionary 104, the word code of the original document (for example, the character code of “block”) and the compressed character code of the conversion word (for example, “0A3B”) are registered as a pair. As the word to be registered, a general-purpose Japanese dictionary may be used, or the user may register it from the input means 108. However, the code information of each character shall be supported even after the compressed text conversion. In the above embodiment, "no""figure"
"Ha" is this. Using these techniques, FIG. 4 (b)
The text data is converted by the text data compression unit 103 into compressed text data as shown in FIG.
The text data file 106 is filed.

At the time of search, a keyword for search is input, and the keyword is converted into a compressed keyword for each word in the same manner as the above-mentioned text data, and the compressed keyword and the compressed text data are collated to perform the search.

In the above embodiment, the compression conversion result is 16
The text of 9 bit codes is obtained, and the compression rate is 5
A compression effect of 0% or more is obtained. By using the above compression method, the text capacity is reduced to about 1 / th in the above embodiment.
2, the reading time from a storage medium such as a disk is halved, and at the same time, the number of times of matching the text data with the keyword at the time of search is reduced by searching the compressed text data by half. As a result, the search speed can be more than doubled.

FIG. 5 shows an example of a block diagram of an information processing system to which the present invention is applied.

In the figure, reference numeral 501 denotes a CPU (Central Processing Unit), which performs various processes on text / image data, keywords and retrieval conditional expressions, document identifier information and the like.
An image input device 502 takes in image data from a paper surface or the like. A character recognition device 503 extracts and recognizes only character components from the captured image data,
Converts document information into text data. Reference numeral 504 denotes a network / host computer connection device, which receives text / image data from various external devices via the network. Reference numeral 505 denotes a character string collation processor, which processes a character string collation operation at high speed using an automaton or other character collation algorithm. A code conversion dictionary 506 stores a character code of the original document and a compression code after compression conversion as a pair. A synonym / synonym dictionary 507 performs a synonym / synonym search for a keyword during a search. Reference numeral 508 denotes a text data file in the storage device, which stores compressed text data that has undergone compression conversion. An image data file 509 stores image data attached to the text data. 5
Reference numeral 10 is an input means such as a keyboard, which is used for inputting a keyword at the time of search or directly inputting text data. Reference numeral 511 is a display means such as a CRT and displays the search result. 512 is a CRT control circuit,
Control the CRT 511. Reference numeral 513 is an output unit such as a printer, which prints out the search result on the paper. 514 is
A printer control circuit that controls the printer 513.
The CRT 511 and the printer 513 are output devices and can output search results.

First, when registering text data, the text data is recognized / converted using the character recognition device 503 with respect to the text data via the network / host computer connection device 504 or the image data input from the image input device 502. Is input to the CPU 501. Alternatively, the text data is directly input to the CPU 501 using the keyboard 510. After the input text data is analyzed by the CPU 501,
In the character string collation processor 505, the code conversion dictionary 50
Character string collation with the word registered in No. 6 is performed, and the compression processing is performed by the CPU 501 using the collation result.
The compressed text data is stored in the text data file 508, and the image data input together with the text data is stored in the image data file 509.

Next, when retrieving text data, a keyword is input from the keyboard 510 or from the network / host computer via the network / host computer connection device 504, or via the image input device 502 and the character recognition device 503. And the search formula are input to the CPU 501. The input keyword and the search expression are analyzed by the CPU 501, the synonym / synonym dictionary 507 is used for the keyword, the synonym / synonym is expanded, and the collation from the code conversion dictionary 506 and the character string collation processor 505 is performed. Using the results, CP
The compression code is converted in U501. Next, the compressed text data extracted from the text data file 508 is subjected to character string matching by the character string matching processor 505 with respect to the above-mentioned compressed code converted keyword. The CPU 501 determines the collation result of the character string collation result, and the document matching the search result is CRT5.
11 or the printer 513 will be displayed to the searcher.

In the above system configuration diagram, when the character string collation does not require a high speed or sophisticated function, the character string collation is performed by the CPU 501, and the character string collation processor 5
05 can be deleted. Further, when the character recognition does not require speed or sophisticated functions, the CPU 501 may perform the character recognition and the character recognition device 503 may be deleted. Further, in the above system configuration diagram, the text data file 508 and the image data file 509 are separated, but it is also possible to make them one and store the text data and the image data in the same file.

According to the present invention, since the text code is compressed and stored in the text data file when the document text information is registered, a huge amount of text data can be reduced without losing the original text information. ..

In searching the document text information,
Since the compressed text data is read, the processing time required to read the disk by the text storage means is shortened as compared with the conventional original document full-text search, and as a result, the search speed is improved.

Furthermore, in consideration of the preservation of the meaning of the original document in the document compression method, the minimum meaning unit is regarded as a word, and a method of giving a new compression code to this is used. By performing the matching using the compressed code after the execution, it is possible to provide a high-speed and simple document text search method without the need to decompress the text data into the original document when searching the compressed text. In particular, the effect of this method is remarkable in usage such as full-text search application where it is necessary to search a huge amount of document text.

[0062]

As described above, according to the present invention, an excellent full-text which improves the search speed of text data, does not increase the document data, and does not require the re-search of the main text. A search system can be realized.

[Brief description of drawings]

FIG. 1 is a configuration diagram showing an embodiment of the present invention.

FIG. 2 is a flowchart for registering text data.

FIG. 3 is a flowchart for searching text data.

FIG. 4 is an example of text compression.

FIG. 5 is a system configuration diagram showing an embodiment of the present invention.

FIG. 6 shows an example of a synonym table.

FIG. 7 shows an example of synonym compression text conversion.

FIG. 8 is an example of converting a synonym compression keyword.

FIG. 9 is an example of a collation deviation process during text compression.

FIG. 10 shows an example of keyword conversion for collation deviation text search.

FIG. 11 is a mismatched text search algorithm.

[Explanation of symbols]

101 ... Text / image data input section, 102 ... Text data analysis section, 103 ... Text data compression section,
104 ... Code conversion dictionary, 105 ... Synonym / synonyms dictionary, 106 ... Text data file, 107 ... Image data file, 108 ... Keyboard, 109 ... Search expression analysis unit, 110 ... Search expression conversion unit, 111 ... Character string collation unit , 112 ... Search condition determination unit, 113 ... Text data decompression unit, 114 ... Display, 115 ... Display control circuit, 116 ... Printer, 117 ... Printer control circuit, 118 ... Text / image bus, 119 ... Image input unit, 120 ... Character recognition part.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Yoshihiro Yokoyama, inventor Yoshihiro Yokoyama, 292 Yoshida-cho, Totsuka-ku, Yokohama-shi, Kanagawa Inside the Microelectronics Equipment Development Laboratory, Hitachi, Ltd. (72) Hidefumi Masusaki Kunizu, Odawara, Kanagawa Address 2880: Hitachi Ltd. Odawara factory

Claims (19)

[Claims] 1. A document information input means for inputting document information, a text data storage means for registering and storing the document information, key information for searching the registered document information and a search condition. Input means, a character string collating means for collating the key information with registered document information, and a search formula conversion means for generating a search condition formula from the search condition,
In a document information search device having a search condition determination unit that determines document information that matches the search condition expression and an output unit that outputs a result determined by the search condition determination unit, the input document information is compressed text data. And a text data decompression unit for decompressing the compressed text data into original document information, wherein the text data storage unit registers and stores the compressed text data, and the search expression conversion is performed. The means compresses and converts the input key information into compressed key data to generate a search condition expression from the search condition, and the character string matching means matches the compressed key data with the registered compressed text data, The search condition determination means is based on the matching result output from the character string matching means, and is based on the matching result output from the search expression conversion means. Determine the compressed text data of documents matching information, wherein the text data decompression means, the document information retrieval apparatus characterized by decompressing the compressed text data of the document information that conforms to the original document information. 2. The text data analysis unit according to claim 1, further comprising a text data analysis unit that expands the input document information for each word that can be regarded as one semantic unit, and the text data compression unit for each expanded word. A document information retrieving apparatus characterized by assigning a compression code to and converting the compressed text data into compressed text data. 3. The code conversion dictionary according to claim 2, further comprising a code conversion dictionary that stores a word and a compression code that can be regarded as one semantic unit in a pair, and the text data compression unit refers to the code conversion dictionary. A document information retrieval device characterized by compression and conversion. 4. The document information search according to claim 3, wherein the input unit receives a word to be registered in the code conversion dictionary, and the code conversion dictionary registers the word to be registered. apparatus. 5. The document information search device according to claim 2, wherein the text data analysis unit performs syntax analysis on the input document information and expands it into each word. 6. The synonym of different notation according to claim 2,
The text data compression unit further includes a synonym / synonym table that stores a synonym group and a compression code as a pair and stores identification information for each synonym / synonym with different notation, and the text data compression unit stores the synonym / synonym table. Referring to a word in the original document text, one compression code is assigned to a synonym / synonym group having different notations, and information indicating that the word is a synonym / synonym and the identification information are added, and compression conversion is performed. A document information search device characterized by: 7. The search expression conversion means according to claim 6, compressing and converting the input key information into compressed key data by referring to a synonym / synonym table, and generating a search condition expression from the search condition. The document information retrieval, wherein the character string collating means detects information indicating a synonym / synonym, skips the identification information, and collates compressed key data with registered compressed text data. apparatus. 8. The text data decompressing unit according to claim 1, which decompresses the compressed text data to be collated at the time of collating the key information, and the character string collating unit collates the key information with the decompressed document information. A document information retrieval device characterized by performing. 9. The document according to claim 2, wherein the text data analysis unit recognizes a portion in which misalignment of words occurs and adds predetermined information to the portion in which misalignment of words occurs. Information retrieval device. 10. The text data analysis unit according to claim 9, wherein when a misaligned reading of a word occurs, the text is expanded and expanded, and the text data compression unit is expanded. A document information retrieving apparatus characterized in that a compression code is assigned to all of the words to compress and convert into compressed text data, and at the time of retrieval, the character string collating means collates all of the listed words with compressed key data. 11. The text data analyzing unit according to claim 2, recognizing a portion where misalignment of words occurs, and the text data compression means does not compress and convert a portion where misalignment of words occurs, and the text The data storage means stores a portion where misalignment of words occurs as text data, and the character string collating means also collates key information with the text data at the time of retrieval. 12. The search expression conversion means according to claim 2, wherein a portion of the input key information in which misalignment of words occurs is recognized, and when misalignment of words occurs, there is a possibility of misreading of words. Are expanded, and a compression code is assigned to all expanded words and compressed and converted into compressed key data to generate a search condition expression from the search condition. A document information retrieval device characterized in that the compressed key data of the above is compared with the compressed text data. 13. The code conversion dictionary according to claim 3, wherein a plurality of types of code conversion dictionaries are provided, the input means receives an instruction to select a dictionary to be used according to the type of document, and the text data compression means, the instructed code conversion dictionary. The dictionary is compressed and converted, the code conversion dictionary identification information used for the compressed text after conversion is added, and when decompressing the text data, the text data decompression means refers to the code conversion dictionary identification information and responds. A document information retrieval device characterized by decompressing using a code conversion dictionary. 14. The document information search device according to claim 13, wherein the search expression conversion means compresses and converts the key information input with reference to the designated code conversion dictionary into compressed key data. .. 15. The text data storage means according to claim 1, wherein the identifier information of the compressed text data is registered and stored together with the compressed text data, and the search condition determination means is adapted in the text data expansion means. A document information retrieving apparatus, which outputs the document identifier information instead of decompressing the compressed text data of the document to the original document information and outputting it. 16. The method according to claim 1, wherein when the document information is registered,
Image data storage means for registering image data related to the document information is further provided, and the search condition determination means decompresses the compressed text data of the matched document in the text data decompression means to output the original document information. Instead, a document information search device characterized by outputting image data related to the document information. 17. A document information registration / retrieval method in a document information retrieval device for retrieving document information, wherein when inputting document information, the input document information is compressed and converted into compressed text data and registered, and when retrieving document information, A method for registering / retrieving document information, characterized in that input key information is compressed and converted into compressed key data, and the compressed key data is collated with the compressed text data to search for compatible compressed text data. 18. An input device for inputting document information, a storage device for storing document information, key input means for receiving key information for searching document information, and a central processing unit for searching document information. In the information processing system having an output device that outputs a search result, the central processing unit compresses and converts the document information input by the input device into compressed text data and registers the compressed text data in a storage device, and the key input unit. An information processing system, characterized in that the key information input in (1) is compressed and converted into compressed key data, and the compressed key data is collated with the compressed text data to search for compatible compressed text data. 19. The code conversion dictionary according to claim 18, further comprising a code conversion dictionary for storing a word regarded as one semantic unit and a compressed code in a pair, and the central processing unit refers to the code conversion dictionary. An information processing system characterized by compression and conversion.