JP3489237B2 - Document search method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a document search method (full text search method) for rapidly searching for a document in which a specified character string exists for the entire text of a large-scale document database. In particular, databases, document management systems, document filing systems and DTPs
(Desk Top Publishing) system and the like.

[0002]

2. Description of the Related Art Conventionally, there are the following document retrieval methods that do not require index information. At the time of registration, the document is directly registered in the computer as text data encoded as characters and stored as a text database. At the time of search, the contents of all texts in the text database are read and a specified search character string (hereinafter referred to as a search term). ) Is a full-text search for documents that contain (L.
A. Hollar, “Text Retrieval
Computers ", COMPUTER, Marc
h, 1979). This full-text search is performed by scanning the entire text file, in which the texts of all the documents that make up the text database are collected as a file, character by character from the beginning, and checking whether the specified search term exists. It is characterized in that a document including the specified search term is searched for.

By doing so, indexing using a dictionary such as a thesaurus is unnecessary, and any word appearing in a document can be specified as a search term and searched.

However, the full-text search has a problem that it takes a long time to perform a search process because it scans the entire text file from the beginning and cannot be applied to a large-scale database.

In order to solve this processing time problem, Japanese Patent Application No. 2-193015 (see Japanese Patent Application Laid-Open No. 3-174652) was proposed. This aims at accelerating the reading process of text data and the search process of the search term by using the dedicated hardware, and searches the auxiliary file in which the text is compressed in advance before searching the text. This is an information search device that realizes equivalently high-speed full-text search by narrowing down the number of documents for searching the text body (hierarchical pre-search).

The hierarchical pre-search, which is a feature of this conventional example, will be described below.

Hierarchical pre-search is related to a search term by hierarchically searching two auxiliary files, that is, "character component table" and "condensed text", which are information-compressed in advance before searching text. By sieving out non-existing documents from the search target and reducing the number of documents searching for text, the search speed is equivalently accelerated. That is,
First, the character component table is narrowed down by character unit, and then the document narrowed down by the character component table search is narrowed down by word unit by condensed text search.

In this character component table search, since only character-by-character information of the registered text is recorded in the character component table,
Documents that include all characters used in the search term are search candidates. For example, if "Iran" is specified as the search term, a document in which the three letters "I", "La", and "N" exist somewhere in the text, such as the characters "Lion" and "Online" Documents containing columns will also be hit by the character component table search.

That is, the search term "Iran" is actually used.
There are cases where a large number of documents (hereinafter, referred to as noise) that are considered to include the document are output as search results although they do not include the document. In such a case, the number of documents to be subjected to the condensed text search cannot be narrowed down, so that the condensed text search takes time and a sufficient search response cannot be obtained.

As a method for solving this problem, Japanese Patent Application No. 3-
174064 (see Japanese Patent Laid-Open No. 5-174064)
Then, we proposed a concatenated character component table method that obtains a narrowing rate higher than that of a single character by combining character components.

That is, in this concatenated character component table method, information indicating whether or not a character string having a predetermined number of characters (two or more characters) exists in the text data at the time of registration is recorded in the concatenated character component table. When searching, the search term is divided into character strings of the above-mentioned predetermined number of characters before searching the condensed text, and a document including all the character strings is extracted by referring to the concatenated character component table. By doing this, documents that are not related to the input search term can be filtered out with high precision at the substring level, and it is possible to sufficiently narrow down the documents that are searched for condensed text.

For example, as shown in FIG. 2, text data
When searching (Document 1, Document 2, ..., Document N), in the case of a single character component table, the search term “Iran” corresponds to “I”, “La” and “N” indicated by arrows. The bit string to be searched is the search target, but documents including "lion" and "online", that is, both document 1 and document 2 are "a" and "la".
And "n" are included, it will be searched in the character component table even if there is no character string "Iran",
It becomes noise.

On the other hand, in the case of the concatenated character component table,
Document N as a document that includes both "Ira" and "Run"
The noise is not mixed as in the case of the single character component table.

By using the concatenated character component table in this way, documents such as document 1 and document 2 in which the characters that make up the search term are included separately can be deleted from the search target, so that a single character component can be deleted. It is possible to omit an extra condensed text search than a table. As a result, a condensed text search for documents that are sufficiently narrowed down can be performed, and an equivalently high-speed full-text search can be realized. Even with a large-scale text database, a full-text search can be performed with a practical search response. Can be executed.

With this concatenated character component, for example, in the case of a concatenated character component table of two characters, it is necessary to record the character component for all combinations of squared characters of all character types. In this known example, the capacity of the character component table is reduced by superimposing and recording the presence or absence of a plurality of concatenated character components in one entry in consideration of the appearance frequency of the concatenated character components in the text (called hashing). In addition, it is devised so that the reduction of the narrowing rate is also suppressed.

[0016]

The character component table search in the concatenated character component table method described above has the following two problems.

The first problem is that when this character component table search is applied to phonetic characters, such as English sentences, which have a small number of character types and represent meaning by a character sequence, search noise due to character combinations causes It happens a lot. For example,
A string of three letters next to each other, such as “ain” (hereinafter,
It is called "mount ain ", "p aint ", "Sp ai "
All of them are included in words such as " n ". As a result, the number of documents targeted for the condensed text search cannot be reduced because the narrowing rate of the character component table cannot be increased, and a sufficient search response cannot be obtained. This is because in a phonetic alphabet such as English the meaning is expressed by the combination of multiple consonants and vowels, there are many words that contain the same concatenated character,
This is because character strings adjacent to each other (hereinafter, referred to as sequential concatenated characters) cannot be unique to words.

The second problem is that noise is included in the search result of the character component table due to hashing. That is, since a plurality of concatenated character components are assigned to one entry in the character component table, if a certain concatenated character is specified and the corresponding entry is read, a document containing a completely different concatenated character component is obtained from the bit information. There is a possibility that
Therefore, in a large-scale document search system that registers a large number of documents, documents that are not related to a search word may not be properly screened, that is, narrowing down may not be performed accurately, which may lead to a reduction in search efficiency. It is possible to make one entry for each concatenated character without hashing for this problem. However, this is not practical because the capacity of the character component table becomes enormous.

More specifically, since there are about 8,000 character codes used in Japanese, there are 64 million kinds of concatenated characters as a combination of two characters (8,000 kinds × 8,000 kinds).
Becomes If the number of documents to be registered is 1 million, this 6,4
Since 1 million bits of document identification information must be associated with each of the 0 million types of concatenated characters, a capacity of 8 TByte (64 million types x 1 million bits) is required for the character component table. The capacity of the document body for the size of this character composition table is
Even if the capacity for one case is 20kB, 20GByte (20kB /
(1 x 1 million cases), the capacity of the character composition table will be overwhelmingly larger.

In contrast to the problems described above, the first problem to be solved by the present invention is that a character component that causes less search noise even when a word composed of alphabetic characters such as phonetic characters is designated as a search term. It is to realize a table search.

Further, a second problem to be solved by the present invention is to realize a character component table in which search noise due to hashing does not occur with a practical capacity even in a large-scale document database.

[0022]

The first object of the present invention can be solved by adopting the following configuration.

A partial character string is extracted from each of the documents stored in advance in a predetermined format, a concatenated character component table showing whether or not the partial character string exists in each document is created, and a desired character is extracted from each document. A search substring is extracted in a predetermined format from the search term input to search the document, and the search term is searched by referring to the concatenated character component table corresponding to the extracted search substring. Find documents that have substrings that match each of the constituent substrings for search,
In a document search method for filtering out documents unrelated to a search term from a search target, every predetermined m characters (m is an integer of 1 or more) from a document, a predetermined n characters (n is an integer of 2 or more) ) Is extracted as a partial character string, and a predetermined n-character (n is an integer of 2 or more) character string is extracted from the search term at every predetermined m characters (m is an integer of 1 or more). A document retrieval method characterized by extracting as a partial character string for retrieval.

Further, the second problem can be solved by the following constitution.

The above-mentioned concatenated character component table is a bit list in which appearance information of character strings is registered by writing 1 in the bit position corresponding to the document number in which a concatenated character whose appearance frequency is higher than a predetermined threshold appears. , For a concatenated character component whose appearance frequency is lower than a predetermined threshold value, has a document number list in which document numbers in which concatenated characters whose appearance frequency is lower than the predetermined appearance frequency appear are stored as a list of binary data. Calculate the type of concatenated character component that appears in the document and the number of documents in which each concatenated character component appears, and check whether the number of documents in which each concatenated character appears in the text data is greater than the specified threshold value. If it is determined that the number of occurrences of each concatenated character appearing in the document is larger than a predetermined threshold, it is determined that the bit list is found. The appearance information of the concatenated character component is registered by writing "1" in the bit position corresponding to the document number in which the concatenated character appears, and as a result of the judgment, the number of the appearing documents of each concatenated character appearing in the document is a predetermined number. If it is determined that it is smaller than the threshold value, the appearance information of each connection character component is registered as the appearance information by writing the document number in which the corresponding connection character appears in the document number list as a list of binary data. , For the concatenated characters extracted from the search term, read the bit list or document number list corresponding to the extracted concatenated characters, and in the case of the document number list, convert this to a bit list to obtain the concatenated character component table. This is a document search method characterized by acquiring.

[0026]

First, the operation of the first document search method will be described.

In this document search method, in the skip concatenated character component extraction step in the concatenated character component table creation / registration process, n character strings are cut out from the text data every m characters as skip concatenated characters, and this appearance information is concatenated. Register in the character composition table. Also in the process of searching the concatenated character component table, similarly, by cutting out n character strings every m characters from the search term and searching the concatenated character component table, words including the same partial character string such as English are extracted. Since a character component unique to a word can be taken even in a language in which a large number of characters exist, it is possible to improve the narrowing rate in the concatenated character component table search.

For example, when a character string "mountain" is specified in the search term, in the conventional method of referring to the concatenated character component table between mutually adjacent character strings, as shown in FIG. “Oun”, “unt”, “nta”, “tai” and “ain” are extracted as concatenated character components. However, for example, "ain" is also included in the words " pain ting", "Sp ain ", etc., so that a document including these words may be hit as search noise. On the other hand, for example, by taking a character string every other character, "mut", "on
Character components unique to the word can be taken, such as "a", "uti", and "nan". Therefore, the search noise caused by the existence of words that contain the same substring is significantly larger than in the conventional example. Can be reduced.

Next, the operation of the second document retrieval method of the present invention will be described.

In this document retrieval method, at the time of document registration, in the character appearance frequency calculation step, the type of concatenated character component appearing in the text data and the number of documents in which each concatenated character component appears are calculated.

Furthermore, in the character appearance frequency determination step at the time of creating and registering the concatenated character component table, whether the number of appearing documents of each concatenated character appearing in the text data is larger than a predetermined threshold value from the calculation result in the character appearance frequency calculating step. Determine whether or not.

When the character appearance frequency is larger than a predetermined threshold value, in the bit list registration step, write "1" at the bit position in the bit list corresponding to the document number in which the corresponding concatenated character appears. The appearance information of the concatenated character component is registered by.

When the character appearance frequency is smaller than the predetermined threshold value, the document number in which the corresponding concatenated character appears in the document number list registration step is written as a binary data list in the document number list. The appearance information of each concatenated character component is registered as the appearance information.

At the time of retrieval, in the concatenated character component table acquisition step, the bit list or document number list corresponding to the concatenated characters cut out from the search term is read, and in the case of the document number list, this is converted to a bit list. Thus, in the second document retrieval method of the present invention, the corresponding concatenated character component table is a bit list for concatenated characters having a high appearance frequency, and the document number for a concatenated character having a low appearance frequency. By using a list, the file size of the concatenated character component table can be greatly reduced. To be more specific, in order to configure the concatenated character composition table in the bit list format, the number of bits for all cases registered in the database is always required, but the concatenated character composition table is constructed in the document number list format. To do so, the number of bits representing the document number x the number of registered documents is sufficient. For example, assume that the total number of registrations in the database is 1 million, and that 32 bits are allocated to represent one piece of document identifier information.
If it is 0, if it is a bit string, 1 million bits = 125K
A storage area for B is required. On the other hand, in the case of the document number list format, a storage area of 32 bits × 10 cases = 320 bits = 40B will suffice.

If the number of documents including the concatenation character "composition" is 900,000 out of 1 million, if the bit string is 100, then 100
A storage area of 10,000 bits = 125KB is enough. On the other hand, in the case of the ID list format, an area of 32bit × 900,000 = 4B × 900,000 = 3.6MB is required.

Therefore, this one million cases are identified by the document identifier 3
When storing in 2 bits, 1 million bits ÷ 32 bits = 31,250
When the number of registered cases is larger than this, the file size of the concatenated character composition table can be minimized by configuring the concatenated character composition table in the bit list format and in the document number format when the number of registered cases is smaller.

Further, since the concatenated character component table search result according to the second document search method of the present invention does not include noise due to hashing, a logical product operation between these search result sets is performed.
Also in the character component table search result obtained by (AND), noise is significantly reduced compared to the conventional character component table search result for hashing, and it is possible to improve the refinement accuracy. .

[0038]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A first embodiment of the present invention will be described below with reference to FIG.

In this embodiment, the case where m = 1 and n = 3 is described as an example of a skip concatenated character component table for extracting a character string of n characters every m characters in English.

The document retrieval system to which the present invention is applied includes a display 100, a keyboard 101, and a central processing unit C.
The PU 102, the magnetic disk 110, the floppy disk drive (FDD) 106, and the main memory 200. Further, these are connected by a bus 108. Magnetic disk 110 has text 103 and condensed text 10
4, a concatenated character component table 105 and various programs 111 and tables 112 described later are stored. A floppy disk 107 stores a document to be registered.

In the main memory 200, a system control program 201, a document registration control program 202, a text registration program 203, a condensed text creation registration program 204, a concatenated character component table creation registration program 205,
The search control program 209, the search conditional expression analysis program 210, the concatenated character component table search program 211, the condensed text search program 214, the text search program 215, and the hash table 216 are read from the magnetic disk 110 and stored, and the work area 217 is also stored. Is secured.

Concatenated character component table creation registration program 205
Is a skip concatenated character extraction program 206, a concatenated character component table registration program 207, and a hash table creation program 208. The concatenated character component table search program 211 is a skip concatenated character extraction program 212.
And bit and program 213. These programs are executed under the control of the system control program 201 in response to a user's instruction from the keyboard 101.

The registration process and the search process in the hierarchical pre-search including the concatenated character component table search which is the subject of the present invention will be specifically described below.

When registering a document, as shown in FIG. 4, the system control program 201 receives a command input from the keyboard 101, and the system control program 201 receives the document registration control program 202.
To start. The document registration control program 202 is first executed by the text registration program 203 in step 1000.
Is started to read the text data of the registered document from the floppy disk 107 inserted in the floppy disk drive 106 into the work area 217 and store it as the text 103 in the magnetic disk 110. The registration document may be input not only by using the floppy disk but also by inputting from another device using a communication line (not shown in FIG. 1) or the like.

Next, in step 1001, the document registration control program 202 sends the condensed text creation registration program 20.
4 is started, the text data is divided into partial character strings at the word level using spaces and symbols as delimiters, the character string inclusion relations between the divided partial character strings are examined, and they are included in other partial character strings. Of the remaining partial character strings as the condensed text 104 and the magnetic disk 11 is removed.
Store it in 0, wait for the end of this program, and complete the registration process.

Finally, the document registration control program 202 activates the concatenated character component table creation registration program 205 in step 1002.

The processing of the concatenated character component table creation / registration program 205 will be described below with reference to FIG.

First, the concatenated character component table creation / registration program 205 activates the skip concatenated character extraction program 206 in step 1010, and the text data from the text 103 stored in the magnetic disk 110 is converted into the work area 2.
Read in 17. Then, from this text data, all three character strings are extracted every other character.

Next, in step 1011 the concatenated character component table registration program 207 is activated, and the character string extracted from the text data by the skip concatenated character extraction program 206 is converted into the concatenated character component table 1 in the work area 217.
05 is registered according to the hash table 216, and this is stored in the magnetic disk 110.

When the concatenated character component table 105 is newly registered, the concatenated character component table registration program 207 activates the hash table creation program 208, and the hash table 216 used to refer to the corresponding entry in the concatenated character component table 105 is set. At the same time as creating, all entries in the concatenated character component table 105 are initialized (cleared to “0”).

The hash table 216 created by the hash table creation program 208 is used when referring to the entry of the concatenated character component table 105 by the concatenated character component. For this hash table 216, a simple hash function or Japanese Patent Application No. 3 mentioned above
A hash method using the frequency of concatenated character components in the document database shown in No. 342695 may be used.

The above is the processing contents of the concatenated character component table preparation registration program.

At the time of retrieval, the retrieval condition expression is the keyboard 1
When input from 01, the system control program 201
This starts the search control program 209. Then, under this control program, the search conditional expression analysis program 210, the concatenated character component table search program 211, the condensed text search program 214, and the text search program 215 are sequentially activated.

Hereinafter, referring to FIG. 6, a concatenated character component table search program 211 and a condensed text search program 2
14 and the text search program 215 for the hierarchical search processing will be described in detail.

First, the search control program 209 activates the concatenated character component table search program 211 in step 1020. By executing this program, first, the skip concatenated character extraction program 212 is started, and a character string of three characters is extracted every other character from the search term in the input search condition expression and is extracted using the hash table 216. The bit list stored in the concatenated character component table entries corresponding to all the character strings is read into the work area 217. Next, the bit-and-program 213 is started, and a logical product (AND) is calculated for each bit among all the bit lists read in the work area 217. The document number corresponding to the bit that has become "1" as a result of this logical product operation is output to the search control program 209 as the result of the concatenated character component table search, and if the number of concatenated character component table searches is 0 The program 209 displays a search result of 0 here on the display.

If the number of connected character component table searches is not 0, the search control program 209 executes the condensed text search program 214 in step 1021. Here, the condensed text 104 corresponding to the document number output by the concatenated character component table search program 211 is read into the work area 217.

Then, the condensed text 104 that has been read
Is output by the condensed text search program 214, and the document number of the condensed text including the search term is output to the search control program 209.

If the number of results of the condensed text search is 0, the number of results of 0 is output to the system control program 201 and the search processing is terminated.

In addition, a single search term or a logical relationship between a plurality of search terms (AND condition or OR condition) is specified in the given search condition expression, and If the positional relationship of is not specified,
The search ends here and the condensed text search program 21
The document number output by 4 is output to the system control program 201 as a search result.

In other cases, that is, when the positional relationship in the text between a plurality of search terms is specified in the given search condition expression, the text search program 215 is started in step 1022, Perform a text search.

In the condensed text 104, it can be seen from the creation algorithm that the search can be ended by the condensed text search when a single search term is specified or only AND condition and OR condition are specified. As described above, the words existing in the text 103 are extracted without omission, and it is possible to determine whether or not the designated word appears in the text data only by searching the condensed text 104.

For example, "" information "
<AND> “systems” ”, an AND condition that means“ search for a document in which both “information” and “systems” appear ”, and“ information ”<OR>“ system ”
“Informati” described as “ems”
An OR condition, which means "search for a document in which either" on "or" systems "appears", only specifies a logical relationship between multiple search terms, and does not include the positional relationship in the text. not specified. for that reason,
Since it suffices to know the existence of “information” and “systems”, the success or failure of the search condition can be determined only by the condensed text search.

On the other hand, the "" informationatio
n ”<S>“ systems ””, a context condition having a meaning of “search for documents in which“ information ”and“ systems ”co-occur (occur simultaneously) in the same sentence (sentence)”, "" Info
“Information” and “system” described as “motion” <2W> “systems” ”.
A neighborhood condition that means "search for documents in which ems" appear within 2 words in close proximity, or "informat"
"" information "and" systems "described as" ion "<A>" systems ""
For the adjacency condition, which means that "search for documents that appear adjacent to each other", the positional relationship in the text between multiple search terms is specified, so a condensed text search that does not have word appearance position information is sufficient. The success or failure of the judgment cannot be determined, and a text search must be performed.

When the number of results of the condensed text search is not 0 and the above-mentioned context condition, neighborhood condition or adjacency condition is designated, the text search program 215 is started and the condensed text search program 21 is started.
The text data corresponding to the document number output in 4 is read from the text 103 into the work area 217. Then, the text search program 215 searches this text data, extracts those that include the given search terms and satisfy the specified condition regarding the positional relationship between the search terms, and retrieves the document number corresponding to this extracted text data. The result is output to the search control program 209.

The above is the outline of the full-text search system of the first embodiment to which the full-text search method of the present invention is applied.

The procedure for registering the concatenated character component table in this embodiment is as shown in FIG. Hereinafter, this registration procedure will be described in detail.

First, the details of the process of registering the concatenated character component table will be described. Here, as described above, the concatenated character component table creation registration program 205 activates the skip concatenated character extraction program 206.

By executing this program, text data is read from the text 103 stored on the magnetic disk 110 into the work area 217 for each document, and a character string of three characters is extracted every other character from the text data.

Next, the concatenated character component table registration program 20
7 is activated. Here, the corresponding entry is calculated using the hash table 216 for the above-mentioned concatenated character component extracted from the text data by the skip concatenated character extraction program 205, and "1" is set to the corresponding bit position to set the concatenated character. Note the presence of ingredients.

This skip concatenated character extraction program 20
In the processing of No. 6, for example, as shown in FIG.
For text data "edia", every other character is a concatenated character of three characters "Mli", "utm",
"Lie", "tmd", "iei" and "mda"
Is extracted.

Next, for the extracted character string, "1" is set to the corresponding bit position of the corresponding entry of the concatenated character component table 105 referenced through the hash table 216. Figure 8
In the example, since “Mli” exists in the document 1, “1” is set to the bit position corresponding to the document 1 of the corresponding entry referred to by using the hash table 216. Similarly, in the case of "sse", "1" is set in the corresponding entry. Less than,
Similarly, "1" is set to the corresponding entry of the concatenated character component table 105 for all the concatenated components existing in the text data. Finally, as shown in the figure, for all registered documents, a sequence of "1" and "0" (bit list)
Is completed. For example, "101 ..." of "nss"
The column of "0" is one bit list.

In this way, the concatenated character component table preparation registration program 205 prepares the concatenated character component table 105 when a document is registered, and the preparation for the hierarchical pre-search is completed.

Next, the procedure for searching the concatenated character component table will be described in detail with reference to FIG.

First, the search control program 209 activates the skip concatenated character extraction program 212 in step 1030. Here, 1 from the search term in the search condition expression
A character string of three characters is extracted every other character. However, in this embodiment, since the concatenated character component table of every three characters is used, the concatenated character cannot be obtained when the search term is less than five characters. In this case, in the present embodiment, the results of the concatenated character component table search are all hits, and the condensed text search is performed on all documents. That is, the number of this document is output, and the concatenated character component table search program 211 ends.

When a search term of five characters or more is given, the bit and program 213 extracts the bit list corresponding to the character string extracted by the skip concatenated character extraction program 212 from the concatenated character component table 105 in step 1031. It is read into the work area 217 via the hash table 216, and the logical product operation is performed for each bit between the bit lists read in step 1032. Then, in step 1033, the result of the logical product operation is "1".
The document number corresponding to the bit is calculated, and this is output as the concatenated character component table search result.

For example, as shown in FIG. 10, "Multi
When the character string "media" is given as the search term, "Mli", "utm", "lie", "tm"
The bit list of the concatenated character component table 105 corresponding to “d”, “iei”, and “mda” is the hash table 216.
A document in which all the bits in the bit list are all “1” is obtained as the search result of the concatenated character component table search.

That is, a logical product operation is performed for each bit among all the read bit lists to obtain a bit and operation result 900. In the bit list of the bit-and operation result, the document number corresponding to the bit position of "1" represents the hit document as the search result of the concatenated character component table search.

As a result, "Mli", "utm",
"Lie", "tmd", "iei" and "mda"
A document including all of will be extracted. In the example of this figure, document 1 and document N are hit documents.

As described above, in the concatenated character component table creation / registration process according to the present embodiment, at the time of document registration, a character string of three characters is extracted from the text data every other character, and the existence information of the concatenated character is extracted in advance. By registering in the composition table, a concatenated character component unique to a word can be obtained, so that it is possible to improve the narrowing rate in the concatenated character component table search at the time of search. As a result, the search amount of the condensed text in the hierarchical pre-search is reduced, so that the overall search speed can be equivalently improved. Therefore, a larger amount of full-text search can be performed in real time.

In the present embodiment, the character composition table is targeted for all character strings including "_" (space), "." (Period), "," (comma), etc. without being aware of words. Creating. Therefore, "Multimedia information system
Even when a character string containing a space such as s "is specified in the search term, it is possible to narrow down using the concatenated character component that spans between words.

In this embodiment, a case has been described in which the concatenated character component table is created with a character string of three characters every other character, that is, m = 1 and n = 3. However, the character string is extracted every other character. Even if the character string length is 2 characters or 4 characters or more, the same processing can be performed. This will be apparent from the above description.

Further, in the present embodiment, in the case of a search term of less than 5 characters, the result of the concatenated character component table search is output as all hits.
It is also possible to create a concatenated character component table of characters and use this concatenated character component table to perform a concatenated character component table search for a search term of less than 5 characters.

As described above, in the concatenated character component table search according to the present embodiment, the retrieval noise can be significantly reduced as compared with the conventional example. Therefore, the concatenated character component table search results in condensed text and text. There is no significant difference from the search results obtained by doing so. Therefore, the search result of the concatenated character component table search can be directly output to the system control program 201 as the search result of the system.

Next, a second embodiment of the present invention will be described.

In the first embodiment of the present invention, "_" (space), "." (Period), "," are recognized without being aware of words.
Character composition table is created for all character strings including (comma). However, in this method, since the skip concatenated characters spanning the words are registered in the character component table, the following two types of noise occur in the search in which the words are designated.

First, the skip concatenated character component spanning between words is hashed to the same entry as another concatenated character, resulting in noise. For example, when "angle" is specified as the search term, "age" is extracted as the skip concatenated character, but "Multimedi a i n f or
mation systems "in" _ "in the case has been hashing to the same entry as the""(space) is a skip articulated character across the" aif "is" age Multimedi a i n f ormation
Documents containing "systems" will be hit as noise.

Further, noise is generated by extracting a document in which the skip concatenated character component extracted from the search term appears across words. In other words, if the document that contains the "··· a g r e en cup ···" string in the text has been registered, "_" is a skip articulated characters across the (space) "age "Is registered in the concatenated character composition table. On the other hand, when " a n g l e " is specified for the search term, "ag
The e "will be extracted, and the above-mentioned document will be hit as search noise.

In order to solve these problems, in the second embodiment of the present invention, a word is cut out from the text at the time of extracting the connected character component, and the skip connected character component is extracted from the cut word to create the connected character component table. By doing so, the concatenated character components of the two words are not extracted, and the search noise is reduced.

This embodiment basically has the same structure as the first embodiment shown in FIG. 1, but the concatenated character component table preparation registration program 205 therein has a structure as shown in FIG.

That is, the concatenated character component table creation / registration program 205 in this embodiment comprises a word cutout program 300, a skip concatenated character extraction program 206, a concatenated character component table registration program 207 and a hash table preparation program 208.

As shown in FIG. 12, the concatenated character component table creation / registration program 205 in this embodiment first activates the word segmentation program 300 in step 1100,
Text data is read into the work area 217 from the text 103 stored in the magnetic disk 110. Then, words are cut out from this text data by separating spaces.

Next, in step 1101, the skip concatenated character extraction program 206 is activated, and all three character strings are extracted every other character from all the words cut out by the word cutout program 300.

Finally, in step 1102, the concatenated character component table registration program 207 is started, and the concatenated character extracted from the word by the skip concatenated character extraction program 206 is stored in the concatenated character component table 105 in the work area 217 in the hash table 216. It is registered according to the above, and is stored in the magnetic disk 110.

Regarding the extraction of this skip concatenated character and the registration of the concatenated character component table, for example, "Multimedi"
a information systems mus
The case where the text "t ..." is registered will be described as an example.

As shown in FIG. 13, the word cutout program 300 causes the document 1 to be "Multimediadi".
a ”,“ information ”,“ system ”
It is divided into s "," must ", ....

Next, the skip concatenated character extraction program 2
According to 06, "Mli", "utm", "lie", and "t" are used as skip concatenated character components from the word cut out.
md ”,“ iei ”,“ mda ”,“ ifr ”,“ no
m ”,“ fra ”,“ omt ”, ... Are extracted.

Furthermore, the concatenated character component table registration program 2
In 07, “1” is set to the bit position corresponding to the document 1 of the corresponding entry referred to using the hash table 216 for “Mli” cut out by the skip concatenated character extraction program 206. Similarly, "utm" is set to "1". In the same manner, “1” is set to the corresponding entry in the concatenated character component table 105 for all concatenated character components existing in the words in the text data. Finally, as shown in the figure, a column (bit list) of "1" and "0" is created for each registered document.

At the time of search, the skip concatenated character extraction program 212 is activated in the concatenated character component table search program 211, as in the first embodiment of the present invention, and every other character from the search term in the input search condition expression. Extract all 3-character strings.

Next, the bit and program 213 is started, and the concatenated character component table 1 corresponding to all the character strings extracted by the skip concatenated character extraction program 212
The bit list stored in the 05 entry is read into the work area 217 via the hash table 216, and the logical product operation is performed for each bit between all the read bit lists. The document number corresponding to the bit that has become "1" as a result of this logical product operation is output as the result of the concatenated character component table search.

As described above, in the process of creating and registering the concatenated character component table in this embodiment, after extracting a word from the text data, every three characters are separated from the character string in the word by three characters.
The character string of the character is extracted, and the existence information of the concatenated character is registered in the concatenated character component table in advance. By this word division, a concatenated character component that spans two words can be deleted, so that noise due to hashing and noise due to a skip concatenated character that spans between words can be reduced. For example, even if the document that contains the "··· a g r e en cup ···" string in the text has been registered, "_" (space)
Since the skip concatenated character "age" that crosses over is not registered in the concatenated character component table, "ag" is used as the skip concatenated character.
Even if " a n g l e " with " e " is specified as a search term, it will not be searched as noise. As a result, the narrowing rate in the concatenated character component table search can be improved. Since the amount of search for condensed text in hierarchical pre-search can be reduced, it is possible to equivalently improve the overall search speed, and thus it is possible to perform a larger amount of full-text search in real time. Become.

In this embodiment, a predetermined m characters (m
Has described the method of creating a character component table after dividing the text into words for every concatenated character (skip concatenated character) every other integer. However, for conventional contiguous contiguous characters (sequential concatenated characters), similarly, by dividing the text into words and then creating a concatenated character component table, concatenated characters including "_" (space) etc. It will be clear that the noise caused by hashing to the same entry as other concatenated characters can be reduced.

Next, a third embodiment of the present invention will be described.

In the second embodiment of the present invention, text data is divided into words in advance, and skip concatenated characters are extracted for each word, so that the skip concatenated characters extending between words are the same as other concatenated characters. It was possible to reduce the noise caused by the hashing and the noise caused by the skip concatenated character component spanning between words. However, this method has a problem that a document in which another word including the word specified in the search term as a partial character string appears is searched as noise. That is, when a document including the word "jangle" in the text is registered, "jnl" and "age" are extracted as skip concatenated characters and registered in the concatenated character component table. On the other hand, when "angle" is specified for the search term, "age" is extracted as a skip concatenated character from the search term, so it is another word that contains the search term "angle" as a substring. Documents such as "jangle" appear as noise.

To solve this problem, in the third embodiment of the present invention, a predetermined code such as a special character (hereinafter referred to as a special character) before and after a word is registered at the time of registration and search of the concatenated character component table in the document search method of the second embodiment. (Explained with special characters) is added. That is, a special character (for example, “^” here) is added, and a special character addition type concatenated character component table is created in which a concatenated character component is extracted by including the special character. As a result, it is possible to distinguish word breaks with special characters, eliminate documents in which another word that includes the search term as a partial character string appears, and reduce noise.

This embodiment basically has the same configuration as that of the first embodiment shown in FIG. 1, but the portions of the concatenated character component table creation registration program 205 and the concatenated character component table search program 211 are included therein. The configurations are as shown in FIGS. 14 and 15, respectively.

That is, the concatenated character component table creation / registration program 205 is composed of a word cutout program 300, a special character addition program 301, a skip concatenated character extraction program 206, a concatenated character component table registration program 207 and a hash table creation program 208. The character component table search program 211 is a special character addition program 302 and a skip concatenated character extraction program 212.
And a bit and program 213.

Concatenated character component table creation registration program 205
First, as shown in FIG. 16, in step 1200, the word cutout program 300 is started, and the magnetic disk 110
Text data is read into the work area 217 from the text 103 stored in. Then, words are cut out from this text data by separating spaces.

Next, in step 1201, the special character addition program 301 is started, and the word cutout program 300
The special character '^' is added before and after the word cut out by.

After that, in step 1202, the skip concatenated character extraction program 206 is activated to extract all three character strings every other character from all words to which special characters have been added by the special character addition program 300.

Finally, in step 1203, the concatenated character component table registration program 207 is activated, and the concatenated characters extracted from the words by the skip concatenated character extraction program 206 are stored in the concatenated character component table 105 in the work area 217 in the hash table 216. It is registered according to the above, and is stored in the magnetic disk 110.

At the time of search, the concatenated character component table search program 211 first executes step 12 as shown in FIG.
At 10, the special character addition program 302 is started, and the special character “^” is added before and after the search term in the search condition expression.

Next, in step 1211, the skip concatenated character extraction program 212 is activated, and the special character addition program 302 extracts all three consecutive character strings from the search term to which the special character "^" is added.

Then, in step 1212, the bit-and-program 213 is activated, and the bit list stored in the entries of the concatenated character component table 105 corresponding to all the character strings extracted by the skip concatenated character extraction program 212 is hashed. It is read into the work area 217 via the table 216, and the logical product operation is performed for each bit between all the bit lists read in step 1213.

As a result of this logical product operation, the document number corresponding to the bit that becomes "1" is output as the result of the concatenated character component table search.

The processing contents of the concatenated character component table creation registration program 205 described above will be described in detail below.

Concatenated character component table creation registration program 205
Then, first, words are cut out from the text 103 by separating spaces, and a special character “^” is added before and after each word. After that, a character string of three characters is extracted every other character from the word to which the special character “^” is added.

Regarding the extraction processing of this skip concatenated character, for example, "Multimediainfoformat"
The case where the text "ion systems must ..." is registered will be described as an example.

The word segmentation program 300 shown in FIG.
As shown in FIG. 8, the document 1 is “Multimedia”,
"Information", "systems",
It is divided into "must", ....

Next, the special character addition program 301 adds a special character "^" before and after each word cut out, and "^ Multimedia ^" and "^ info" are added.
rmation ^ ”,“ ^ systems ^ ”,“ ^ m
ust ^ ”, ...

Next, the skip concatenated character extraction program 2
According to 06, the words "^ ut", "Mli", "utm", and "li" are added from the words with special characters "^" added before and after.
e ”,“ tmd ”,“ iei ”,“ mda ”,“ ei ”
^ ”,“ ^ No ”,“ ifr ”,“ nom ”,“ fr
"a", "omt", ... Are extracted.

Finally, the concatenated character component table registration program 2
Start 07. Here, via the hash table 216, the skip concatenated character extraction program 206 sets "1" to the entry corresponding to the concatenated character component extracted from the word to which the special character "^" is added, and the concatenated character component Note the existence.

In the example of the document 1 in FIG. 18, since there is "^ ut", "1" is set to the bit position corresponding to the document 1 of the corresponding entry referred to by using the hash table 216. Similarly, "Mli" is set to "1".

Similarly, for all of the concatenated character components in the word with the special character "^" added before and after, "1" is set to the corresponding entry in the concatenated character component table 105.
Finally, as shown in the figure, a column (bit list) of '1' and '0' is created for each document in the text 103.

Next, the concatenated character component table search program 2
The processing content of 11 will be described in detail. The concatenated character component table search program 211 first adds a special character "^" before and after the search term in the search condition expression, and extracts a string of three characters from the search term.

After that, a logical product operation is performed for each bit between the bit lists corresponding to the extracted character strings, and the document number corresponding to the bit that becomes "1" is output as the concatenated character component table search result.

For example, as shown in FIG. 19, "Multi
The search term "media" becomes "^ Multimedia ^" by adding a special character. "^ ut", "Mli", "" by extracting a character string of three characters every other character from this search term. utm ”,“ l
"ie", "tmd", "iei", "mda", and "ei ^" are obtained as concatenated character components, and the bit list of the concatenated character component table 105 corresponding to these is read via the hash table 216, A document in which all the bits of all these bit lists are '1' is obtained as the search result of the concatenated character component table search.

That is, a logical product operation is performed for each bit among all the read bit lists, and a bit and operation result 900 is obtained. In the bit list of the bit-and operation result, the document number corresponding to the bit position of "1" represents the hit document as the search result of the concatenated character component table search.

By this, "^ ut", "Mli",
"Utm", "lie", "tmd", "iei",
A document including all of "mda" and "ei ^" will be extracted as a search result. In the example of FIG. 19, the document 1 is a hit document.

That is, when a document including the word “jangle” in the text is registered, “∧ag”, “jnl”, and “∧ag” as skip connecting characters are added from “∧jangle∧” with special characters added before and after the word. “Age” and “nl∧” are extracted and registered in the concatenated character component table. When "angle" is specified as the search term, "∧nl", "age", "nl∧" are extracted as skip concatenated characters from "∧angle∧" with special characters added before and after the word. I mean,
Documents containing "∧jangle∧" do not include the skip concatenation character corresponding to "∧nl", so they are excluded from the search. That is, a document including "jangle" will not be searched as noise.

As described above, in the process of creating and registering the concatenated character component table in this embodiment, when a document is registered, a word is cut out from the text, special characters are added before and after the cut out word, and then the Every three characters are extracted from the character string, the existence information of this concatenated character is registered in advance in the concatenated character component table, and special characters are added before and after the search term when searching the concatenated character component table. By performing a search after that, it is possible to distinguish before and after a word with special characters. Therefore, it is possible to avoid hitting irrelevant words that include the search term as a sub-character string inside the character string by intermediate matching, and reduce noise. As a result, the narrowing rate in the concatenated character component table search at the time of search can be improved, and the search amount of condensed text in the hierarchical presearch can be reduced, so that the overall search speed can be equivalently improved. Becomes Therefore, a larger amount of full-text search can be performed in real time.

In the present embodiment, the case where the concatenated character component table is created every other character with a character string of three characters, that is, n = 3 has been described. However, in the case where the character string length is two characters or four characters or more, It will be apparent from the above description that the same processing can be performed.

Further, in the present embodiment, a word is cut out from the text for a predetermined concatenated character every m characters (skip concatenated character), and special characters are added before and after the cut word, and then the concatenated character is added. The method of creating the ingredient table has been described. However, similar to the conventional concatenated characters (sequential concatenated characters) that are adjacent to each other, similarly, by extracting words from the text, adding special characters before and after the extracted words, and creating a concatenated character component table. It is also clear that it is possible to exclude a document in which another word including the word specified in the search term as a substring appears as noise from the search target.

Next, a fourth embodiment of the present invention will be described.

In the first embodiment of the present invention, a method has been described in which a character string of n characters is extracted every m characters from an English sentence and is registered in the concatenated character component table. However, this method has a problem that a large amount of search noise occurs because the number of concatenated character components that can be extracted when a search term having a small number of characters is specified. That is, when a document including the word "argue" in the text is registered, "age" is extracted as the skip concatenated character and registered in the concatenated character component table. In contrast, results in the "age" as the skip articulated characters are extracted when the "angle" in the search term designated, the document containing "a r g u e" is to be retrieved as noise. Furthermore, in the method of creating a concatenated character component table from concatenated characters (sequential concatenated characters) that are adjacent to each other as in the conventional example, " ang ry" and " single "
Documents in which documents that simultaneously include appear appear as noise.

To solve this problem, in the fourth embodiment of the present invention, when the concatenated character components are extracted in the document retrieval method of the first embodiment, i (i is 2
Concatenated characters of the above integers are sequentially extracted as concatenated characters, and a character string of n characters every m characters is extracted as a skip concatenated character, and a concatenated character component table of both the consecutive concatenated character component and the skip concatenated character component is extracted. Noise is reduced by narrowing down the documents to be searched by using.

This embodiment basically has the same configuration as that of the first embodiment shown in FIG. 1, but the portions of the concatenated character component table creation registration program 205 and the concatenated character component table search program 210 therein are The configurations are shown in FIGS. 20 and 21, respectively.

That is, the concatenated character component table creation / registration program 205 is composed of a sequential concatenated character extraction program 400, a skip concatenated character extraction program 206, a concatenated character component table registration program 207, and a hash table creation program 208. The program 211 includes a sequential concatenated character extraction program 401, a skip concatenated character extraction program 212, and a bit and program 213.

Concatenated character component table creation registration program 205
As shown in FIG. 22, first, in step 1300, the sequential concatenated character extraction program 400 is activated to read the text data from the text 103 stored in the magnetic disk 110 into the work area 217. And three consecutive
Extract all character strings.

Next, in step 1301, the skip concatenated character extraction program 206 is started, and the work area 217
All the character strings of 3 characters are extracted every other character from the text data taken in.

Finally, in step 1302, the concatenated character component table registration program 207 is started, and the concatenated characters extracted by the sequential concatenated character extraction program 400 and the skip concatenated character extraction program 206 are transferred to the work area 2
The hash table 21 is added to the concatenated character component table 105 in FIG.
6 is registered and stored in the magnetic disk 110.

At the time of search, the concatenated character component table search program 211, as shown in FIG.
Then, the concatenated character extraction program 401 is started, and all three consecutive character strings are extracted from the search term.

Next, in step 1311, the skip concatenated character extraction program 212 is activated to extract all three character strings every other character from the search term.

Then, in step 1312, the bit-and-program 213 is activated, and the sequential concatenated character extraction program 401 and the skip concatenated character extraction program 212 are executed.
The bit list stored in the entry of the concatenated character component table 105 corresponding to all the character strings extracted by the work area 21 via the hash table 217.
6, and the logical product operation is performed for each bit between all the read bit lists.

The document number corresponding to the bit which becomes "1" as a result of this logical product operation is output as the result of the concatenated character component table search.

The processing contents of the concatenated character component table creation registration program 205 described above will be described in detail below.

Concatenated character component table creation registration program 205
Then, first, a continuous character string of three characters and three concatenated characters every other character are extracted from the text data.

Regarding the extraction of this character string, for example,
"Multimedia informations
The case where the text “systems must ...” Is registered will be described as an example.

First, in the sequential concatenated character extraction step 400, "Mul", "ul" is selected from the document 1 shown in FIG.
t ”,“ lti ”,“ tim ”,“ ime ”,“ me
d ”,“ edi ”,“ dia ”,“ ia_ ”,“ a_
i ", ... Is extracted by the skip concatenated character extraction program 206.
m ”,“ lie ”,“ tmd ”,“ iei ”,“ md
a ”,“ ei_ ”,“ dai ”,“ i_n ”,“ ai ”
f ″, ... Are extracted.

Finally, the concatenated character component table registration program 2
07 is activated. Here, the sequential concatenated character extraction program 400 and the skip concatenated character extraction program 20 are respectively passed through the concatenated character hash table 216-a and the skip concatenated hash table 216-b.
"1" is set to the entry corresponding to the concatenated character component extracted by 6, and the existence of the concatenated character component is noted.

Next, the processing at the time of retrieval will be described in detail.

For example, as shown in FIG. 25, "Multi
"Mul", "ult", from the search term "media"
"Lti", "tim", "ime", "med",
"Edi" and "dia" are sequentially extracted as concatenated character components. Further, skip concatenated character extraction step 21
2, "Mli", "utm", "lie",
"Tmd", "iei" and "mda" are extracted as skip concatenated character components. Next, the bit and program 213 causes the bit list of the concatenated character component table 105 to pass through the consecutive concatenated hash table 215-a for the consecutive concatenated character component and the skip concatenated hash table 216-b for the skip concatenated character component. Read through. Then, a document in which all the bits of all these bit lists are '1' is obtained as the search result of the concatenated character component table search.

That is, a logical product operation is performed for each bit among all the read bit lists to obtain a logical product operation result 900. In the bit list of the bit-and operation result, the document number corresponding to the bit position of "1" represents the hit document as the search result of the concatenated character component table search. Thus, in the example of FIG. 25, the document 1 is a hit document.

As described above, in the process of creating and registering the concatenated character component table according to the present embodiment, at the time of registering a document, a character string of three consecutive characters (sequential concatenated characters) from the text and a character string of three characters every other character are registered. Take out (Skip concatenation character),
The existence information of this concatenated character is registered in the concatenated character component table in advance. At the time of searching, by searching a document that includes all the concatenated character components of both the consecutive concatenated character and the skip concatenated character, it is possible to reduce noise in the concatenated character component table search. For example, when "angle" is specified as the search term, the sequential concatenation characters are "ang", "ngl", and "gl".
e ", as the skip articulated character""to search for a document that contains a. On the other hand, in the text" age a r g u e "when the document that contains the word is registered, as a skip articulated character" age " Is extracted, but "ang",
Since "ngl" and "gle" are not extracted, they are deleted as noise. Also, if a document that contains both " ang ry" and " single " is registered, "an
Although g "," ngl ", and" gle "are registered in the concatenated character component table," age "is not extracted as a skip concatenated character, and thus can also be deleted as noise.

As a result, the narrowing rate in the concatenated character component table search can be improved, and the search amount of the condensed text in the hierarchical presearch can be reduced, so that the overall search speed is equivalently improved. . Therefore, a larger amount of full text search is possible in real time.

In the present embodiment, the case has been described in which the concatenated character component table of consecutive concatenated characters is created by a character string of three consecutive characters, but the same processing is performed when the character string length is two characters or four characters or more. It will be clear from the above description that is possible. Also, the case has been described in which the concatenated character component table of skip concatenated characters is created with a character string of three characters for every other character, but no matter how many characters are extracted for a character string, the character string length is two or four. It will be apparent that the same process can be applied to the case of more than characters.

In the present embodiment, the character component table is created for all character strings including "_" (space), "." (Period), "," (comma), etc. without being aware of words. However, as shown in the second embodiment, the method of creating a character component table after dividing the text into words, and as shown in the third embodiment, after dividing the text into words, It will be apparent that the same effect can be obtained by adding the special characters before and after the character component table.

Next, as a fifth embodiment of the present invention, a case where the document retrieval method in the fourth embodiment is applied to Japanese text will be described.

Since Japanese is an ideographic character in which each character has its own meaning, compared to English, which is a phonetic character, characters formed by adjacent character strings (sequential concatenated characters) shown in the conventional example. The component table can significantly reduce the search noise. However, when a character string composed of a combination of words is specified as a search term, a lot of noise also occurs in Japanese. For example, when a character string "moving image" is specified as a search term, a document containing both "moving image" and "image" is searched in the sequential concatenated character component table search. As a result, a document or the like in which "moving image" and "image" appear as different words at the same time even if they do not include "moving image" is hit as noise.

To solve this problem, in the fifth embodiment of the present invention, a concatenated character component table is created for a continuous character string (sequential concatenated character) of i characters (i is an integer of 2 or more) for Japanese documents. In addition, every m characters (m is an integer of 1 or more) n characters (n
By creating a concatenated character component table for character strings (skip concatenated characters) of 2 or more), concatenation with less search noise even when a character string composed of word combinations is specified in the search term Realize character component table search.

The configuration of the document search method in this embodiment is the same as in the fourth embodiment. Further, in the present embodiment, a case where i = 2 in a continuous character string of i characters in a Japanese sentence and m = 1 and n = 2 in a character string of every nth character is described with reference to an example below. .

First, the concatenated character component table creation / registration program 205 first extracts a continuous character string of two characters and two concatenated characters every other character from the text 103.

Regarding the extraction of this character string, for example,
An example will be described in which the text "image processing apparatus with automatic image quality adjustment function ..." Is input.

In the sequential concatenated character extraction step 400, "automatic", "moving image", "image quality", "quality adjustment", "adjustment", "rectifier", "function", "function" are selected from the document 1 shown in FIG. , “, ...,” “image”, “image”, “image processing”, “processing” ,. Furthermore, by the skip concatenated character extraction program 206, "self-portrait", "movement",
"Picture adjustment", "Quality adjustment", "Adjustment", "Adjustment", "Machine",
"Nohbi", ..., "ta-image", "artist", "imaging",
“Processing”, ... Is extracted.

Finally, the concatenated character component table registration program 2
07 is activated. Here, the sequential concatenated character extraction program 400 and the skip concatenated character extraction program 20 are respectively passed through the concatenated character hash table 216-a and the skip concatenated hash table 216-b.
"1" is set to the entry corresponding to the concatenated character component extracted by 6, and the existence of the concatenated character component is noted.

Next, the processing at the time of retrieval will be described in detail.

For example, as shown in FIG. 27, a step 401 of extracting consecutive concatenated characters from the search term "moving image"
Thus, the "moving image" and the "image" are sequentially extracted as concatenated character components. Further, in the skip concatenated character extraction step 212, "moving image" is extracted as a skip concatenated character component. Next, the bit and program 213
Thus, the bit list of the concatenated character component table 105 shows that the successive concatenated character component has the concatenation hash table 21.
The skip concatenated character component is read via 5-a and the skip concatenation hash table 216-b. Then, a document in which all the bits of all these bit lists are '1' is obtained as the search result of the concatenated character component table search.

That is, a logical product operation is performed for each bit among all the read bit lists to obtain a logical product operation result 900. In the bit list of the bit-and operation result, the document number corresponding to the bit position of "1" represents the hit document as the search result of the concatenated character component table search. As a result, in the example of FIG. 27, the document N is a hit document.

As described above, in the process of creating and registering the concatenated character component table in the present embodiment, a character string of two consecutive characters (sequential concatenated characters) from Japanese text data is registered at the time of document registration.
And a string of 2 characters every other character (skip concatenated character)
And the presence information of this concatenated character is registered in advance in the concatenated character component table. At the time of search, by searching for a document that contains all concatenated character components of both sequential concatenated characters and skip concatenated characters, the concatenated character component table is displayed even when a search term is specified for a character string composed of word combinations. Search noise of the search can be reduced. For example, when "moving image" is specified as the search term, a document containing "moving image" and "image" as consecutively connected characters and "moving image" as skip connected characters is searched for. On the other hand, in the text, "an image processing device equipped with an automatic image quality adjustment function ...
-When a document containing the character string "" is registered, "moving image" and "image" will be sequentially registered in the concatenated character component table, but "moving image" will be extracted as the skip concatenated character. Since it is not performed, it can be deleted as noise.

As a result, the narrowing rate in the concatenated character component table search can be improved, and the search amount of condensed text in the hierarchical presearch can be reduced, so that the overall search speed is equivalently improved. . Therefore, a larger amount of full text search is possible in real time.

In the present embodiment, the character component table is prepared for all character strings in which kanji, hiragana, katakana, alphabets, numbers, and symbols are mixed, regardless of the character type. Therefore, even when a search term such as "semiconductor laser" in which character types are mixed is specified, narrowing down can be performed using the concatenated character component that spans character types.

In the present embodiment, the case has been described in which the concatenated character component table of consecutive concatenated characters is created with a character string of two consecutive characters, but the same processing is also possible when the character string length is three characters or more. It will be apparent from the above description. Also, the case where the concatenated character component table for skip concatenated characters is created with a character string of two characters every other character has been described, but no matter how many character strings are extracted, the character string length is three characters or more. It will be apparent that the same process can be applied to the case.

In this embodiment, the case has been described in which the concatenated character component table of consecutive concatenated characters is created by a character string of two consecutive characters, but the same processing can be performed when the character string length is three characters or more. Certainly, it will be apparent from the above description. Also, the concatenated character component table for skip concatenated characters is 1
Although the case of creating a character string of two characters for each character has been described, the same processing can be performed even if the character string is extracted for every character and the character string length is three characters or more. Would be obvious.

In the present embodiment, the character component table is created for all character strings including "_" (space), "." (Period), "," (comma), etc. without paying attention to words. However, as shown in the second embodiment, the method of creating a character component table after dividing the text into words, and as shown in the third embodiment, after dividing the text into words, It will be apparent that the same processing can be performed for the method of creating a character component table after adding special characters before and after.

Next, a sixth embodiment of the present invention will be described.

In the fifth embodiment of the present invention, a character component table is created for a Japanese document targeting all character strings mixed with Kanji, Hiragana, Katakana, English letters, numbers and symbols without considering the character type. did. However, in this method, the concatenated characters that span character types are registered in the character component table, so the following types of noise occur.

First, noise is generated due to hashing of successive concatenated character components and character skip concatenated character components that span character types into the same entry as other concatenated characters. That is, when "moving image" is designated as the search term, "moving image" and "image" are sequentially extracted as the concatenated characters, and "moving image" is extracted as the skip concatenated character. On the other hand, a document containing a character string "image processing device with automatic image quality adjustment function" is registered in the text data, and the entry for the skip concatenated character "Nobi" is the same as the entry for the skip concatenated character "moving image". If the text is hashed to, this document will be extracted that contains the "Nobi", which is the skip concatenated character that crosses the Hiragana "o". <Equipment><Image><Image>
A document including the "processing device" will be hit as noise.

Further, by extracting a document in which the skip concatenated character component extracted from the search term appears across different character types, it is searched as noise. That is, "video" and "image" in the text data
When a document containing a character string that includes ", and displays the" image of feeling <motion> ... "is registered," moving image "and" image "are sequentially skipped as concatenated characters. As a concatenated character, the "moving image" that crosses the hiragana "no" is registered in the concatenated character component table. On the other hand, when "moving image" is specified for the search term, "moving image" and "image" are sequentially extracted as the concatenated characters, and "moving image" is extracted as the skip concatenated character from the search term. The document will be hit as noise.

To solve these problems, in the sixth embodiment of the present invention, the text data is divided into character types at the time of extracting the concatenated character component, and the consecutive concatenated character component and the skip concatenated character component are extracted from the divided character string. By creating a concatenated character component table by using the concatenated character component table, the concatenated character component extending over different character types is not extracted, and the noise is reduced.

This embodiment has basically the same structure as the fourth and fifth embodiments, but the concatenated character component table preparation registration program 205 shown in FIG. 20 has the structure shown in FIG.
Also, the concatenated character component table search program 2 shown in FIG.
11 has the configuration shown in FIG.

That is, the concatenated character component table creation / registration program 205 in this embodiment is the character type division program 500, the sequential concatenated character extraction program 400, the skip concatenated character extraction program 206, the concatenated character component table registration program 207, and the hash table creation program. Concatenated character component table search program 2 composed of 208
11 is a character type division program 501, a sequential concatenated character extraction program 401, a skip concatenated character extraction program 2
12 and bit and program 213.

As shown in FIG. 30, the concatenated character component table creation / registration program 205 in this embodiment first activates the character type division program 500 in step 1400 to work text data from the text 103 stored in the magnetic disk 110. The data is read into the area 217 and the text data is divided for each character type.

Next, in step 1401, the consecutive concatenated character extraction program 400 is started, and the character type division program 5
A continuous character string of two characters is extracted from the text data divided by 00 for each character type.

After that, in step 1402, the skip concatenated character extraction program 206 is activated, and a character string of two characters is extracted every other character from the text data divided by the character type division program 500 for each character type.

Finally, in step 1403, the concatenated character component table registration program 207 is activated, and the concatenated characters extracted by the sequential concatenated character extraction program 400 and the skip concatenated character extraction program 206 are transferred to the work area 2
Hash table 21 in the concatenated character component table 105 in 17
6 is registered and stored in the magnetic disk 110.

At the time of search, the concatenated character component table search program 211 first executes step 1410 as shown in FIG.
The character type division program 501 is started up, and the search term in the search condition expression is divided for each character type.

Next, in step 1411, the consecutive concatenated character extraction program 401 is started, and the character type division program 5
All consecutive two character strings are extracted from the search term divided by 01 for each character type.

Then, in step 1412, the skip concatenated character extraction program 212 is activated, and all two character strings are extracted every other character from the search term divided for each character type by the character type division program 501.

Furthermore, the concatenated character component table search program 211 executes the bit and program 2 in step 1413.
13, and the concatenated character component table 105 corresponding to all the character strings extracted by the sequential concatenated character extraction program 401 and the skip concatenated character extraction program 212.
The bit list stored in the entry is read into the work area 216 via the hash table 217, and a logical product operation is performed for each bit between all the bit lists read in step 1414.

As a result of the logical product operation, the document number corresponding to the bit that becomes "1" is output as the result of the concatenated character component table search.

The processing contents of the concatenated character component table creation registration program 205 described above will be described in detail below.

Concatenated character component table creation registration program 205
Then, first, the text data is divided for each character type, and a continuous character string of two characters and a character string of two characters every other character are extracted from the divided text data.

The process of extracting the concatenated characters will be described by taking as an example the case where the text "image processing device having an automatic image quality adjustment function" is registered.

As shown in FIG. 32, the document 1 is displayed by the character type division program 500 as shown in FIG.
It is divided into "", "", "", "", and "image processing device".

Next, the sequential concatenated character extraction program 400
The text data divided for each character type can be used as successive concatenated character components such as "automatic", "video", "image quality",
"Quality", "Adjustment", "Cleaner", "Function", "Et",
"Image", "Image processing", "Processing", "Gift", "Device",
... is extracted.

Further, by the skip concatenated character extraction program 206, “self-portrait”, “movement”, as a skip concatenated character component from the text data divided for each character type,
"Image adjustment", "Quality adjustment", "Adjustment", "Arrangement", "Image processing",
“Image”, “processing”, “laying”, ... Are extracted.

Finally, the concatenated character component table registration program 2
07 is activated. Here, the sequential concatenated character extraction program 400 and the skip concatenated character extraction program 20 are respectively passed through the concatenated character hash table 216-a and the skip concatenated hash table 216-b.
"1" is set to the entry corresponding to the concatenated character component extracted by 6, and the existence of the concatenated character component is noted.

Next, the processing at the time of retrieval will be described in detail.

First, the character type division program 501 divides the search term for each character type. In the example shown in FIG. 33, since the search term is “moving image” and is composed entirely of Chinese characters, the “moving image” is cut out as it is by character type division. Next, consecutive concatenated character extraction is performed from the character type-divided search term. By the program 206, "moving image" and "image" are sequentially extracted as concatenated character components. Furthermore, the skip connecting character extraction 212 extracts “moving image” as a skip connecting character component. Next, the bit list of the concatenated character component table 105 is executed by the bit and program 213 via the sequential concatenation hash table 215-a for the consecutive concatenated character component and the skip concatenation hash table 21 for the skip concatenated character component.
It is read via 6-b. Then, a document in which all the bits of all these bit lists are '1' is obtained as the search result of the concatenated character component table search. This allows
In the example of FIG. 33, the document N is obtained as a hit document.

As described above, in the process of creating and registering the concatenated character component table in this embodiment, the Japanese text data is divided into character types when the document is registered. Then, a character string of two consecutive characters from the text data divided for each character type
A character string of two characters (sequential concatenated character) and every other character (skip concatenated character) is taken out, and the existence information of this concatenated character is registered in the concatenated character component table in advance. Even at the time of search, by extracting the consecutive concatenated character component and the skip concatenated character component for the search term divided for each character type, the concatenated character and the skip concatenated character that span between character types are registered in the character component table, and noise is generated. Can be reduced.

[0200] For example, when a document is registered that includes "moving image" and "image" in the text data and also includes a character string "... "Movies" and "images" are extracted as, but by removing the concatenated character "moving image" that appears across character types by dividing the character string for each character type and then extracting the concatenated character components. You can Therefore, when "moving image" is specified as the search term, the consecutive moving characters "moving image" and "image" are extracted, but the skip connecting character "moving image" is not extracted. Therefore, the document can be deleted as noise.

As a result, the narrowing rate in the concatenated character component table search can be improved, and the search amount of condensed text in the hierarchical presearch can be reduced, so that the overall search speed is equivalently improved. . Therefore, a larger amount of full text search is possible in real time.

In the present embodiment, the method of dividing the text data and the search term into character types in units of kanji, hiragana, katakana, alphabets, numbers and symbols has been described. However, character type division is not performed between certain character types, and sequential concatenated character components and skip concatenated character components are extracted as continuous character strings, so that a character string with mixed character types is also specified in the search term. , It is possible to perform narrowing down with high accuracy by using the concatenated character component that extends between character types. For example, by not dividing the character type between Kanji and Katakana, it is possible to narrow down a character string such as "semiconductor laser" or "magnetic disk" with high accuracy. Further, by not performing character type division between Kanji and Hiragana, it is possible to narrow down the character strings such as "sunrise" and "reading" with high accuracy.

Further, in the present embodiment, the method of simply dividing the text data and the search term for each character type has been described, but the text data and the search term are divided by a word cutout method using a dictionary or Japanese processing. It will be apparent from the above description that similar effects can be obtained in the case of extracting successive concatenated characters and skip concatenated character components later.

In this embodiment, the method of narrowing down the documents to be searched by using the concatenated character component table of both the consecutive concatenated character and the skip concatenated character has been described. However, similar to the conventional method that uses only adjacent contiguous characters (sequential concatenated characters), similarly, by dividing the text data and the search term into character type units and creating a consecutive concatenated character component table, It is clear that the noise caused by straddling concatenated characters being hashed to the same entry as other concatenated characters can be reduced.

Next, a seventh embodiment of the present invention will be described.

In the sixth embodiment of the present invention, the text data and the search term of a Japanese document are divided for each character type, and the consecutive concatenated character and the skip concatenated character are extracted to detect noise in the concatenated character component table search. I explained how to reduce. However, this method has a problem that a document in which another word including a character string specified in the search term as a partial character string appears is searched as noise. That is, when the character string "moving image" is specified as the search term, "moving image" and "image" are sequentially extracted as the concatenated characters, and "moving image" is extracted as the skip concatenated character. On the other hand, when a document containing a character string “high-speed automatic image processing device ...” Is registered in the text data, “moving image” and “image” are also sequentially connected characters from the text data, The "moving image" is extracted as the skip concatenated character. For this reason, a document in which another word "automatic image processing device", which includes the search term "moving image" as a partial character string, appears as a noise.

In contrast to this problem, in the seventh embodiment of the present invention, the special character (for example, here, "^" is used before and after the text data and the search term divided for each character type in the document search method of the sixth embodiment. ')) Is added and the concatenated character component table of the special character addition type which extracts the concatenated character component including it is created. As a result, it is possible to distinguish the character type delimiter with the special character, eliminate a document in which another word including the search term as a partial character string appears, and reduce noise.

This embodiment basically has the same structure as the sixth embodiment, but the concatenated character component table preparation registration program 205 shown in FIG. 28 has the structure shown in FIG. 34, and FIG.
The connected character component table search program 211 shown in FIG.
The configuration shown in FIG.

That is, the concatenated character component table creation / registration program 205 in this embodiment is the character type division program 500, the special character addition program 301, the sequential concatenated character extraction program 400, the skip concatenated character extraction program 206, the concatenated character component table registration program. The concatenated character component table search program 211 includes a character type division program 501, a special character addition program 302, a sequential concatenated character extraction program 401, a skip concatenated character extraction program 212, and a bit and program 213. To be done.

As shown in FIG. 36, the concatenated character component table creation registration program 205 in this embodiment first activates the character type division program 500 in step 1500 to work the text data from the text 103 stored in the magnetic disk 110. The data is read into the area 217 and the text data is divided for each character type.

Next, in step 1501, the special character addition program 301 is started, and the character type division program 500
A special character "^" is added before and after the text data divided for each character type.

Further, in step 1502, the concatenated character extraction program 400 is activated, and the special character addition program 301 extracts a continuous character string of two characters from the text data to which the special characters are added.

Thereafter, the concatenated character component table creation registration program 205 activates the skip concatenated character extraction program 206 in step 1503, and the special character addition program 30
1 from text data with special characters added by 1
Extract all two character strings every other character.

Finally, in step 1504, the concatenated character component table registration program 207 is started, and the concatenated characters extracted by the sequential concatenated character extraction program 400 and the skip concatenated character extraction program 206 are transferred to the work area 2
The hash table 21 is added to the concatenated character component table 105 in FIG.
6 is registered and stored in the magnetic disk 110.

At the time of retrieval, the concatenated character component table search program 211 first executes step 1510 as shown in FIG.
The character type division program 501 is started up, and the search term in the search condition expression is divided for each character type.

Next, in step 1511, the special character addition program 302 is started, and the character type division program 501
The special character '^' is added before and after the search term divided for each character type.

Then, in step 1512, the concatenated character extraction program 401 is activated, and the special character addition program 302 extracts all three consecutive character strings from the search term to which the special character is added.

In step 1513, the skip concatenated character extraction program 212 is activated, and the special character addition program 302 extracts all three character strings every other character from the search term to which the special character is added.

Further, in step 1514, the bit and program 213 is activated, and the sequential concatenated character extraction program 401 and the skip concatenated character extraction program 212 are executed.
The bit list stored in the entry of the concatenated character component table 105 corresponding to all the character strings extracted by the work area 21 via the hash table 217.
6, and the logical product operation is performed for each bit between all the read bit lists.

As a result of the logical product operation, the document number corresponding to the bit which becomes "1" is output as the result of the concatenated character component table search.

The processing contents of the concatenated character component table creation registration program 205 described above will be described in detail below.

Concatenated character component table creation registration program 205
Then, first, the text data is divided for each character type, and a continuous character string of three characters and a character string of three characters at every other character are extracted from the divided text data.

The process of extracting the concatenated characters will be described by taking as an example the case where the text "high-speed automatic image processing device" is registered.

The character type division program 500 shown in FIG.
As shown in FIG. 8, the document 1 is divided into "high speed", "na", "automatic image processing device", ....

Next, by the special character addition program 301, a special character "^" is added before and after the divided text data.
Are added, and become "^ high speed ^", "^ na ^", "^ automatic image processing device ^", ....

Next, the sequential concatenated character extraction program 400
According to the following, from the text data to which special characters are added, “^ high”, “high speed”, “fast ^”, “^ na”, “na ^”, “^ self”, “automatic”, “Movie”, “image”, “image processing”, “processing”, “dressing”, “apparatus”, ... Are extracted.

Further, by the skip concatenated character extraction program 206, "^ speed", "high ^", "^ motion", "self-portrait", "moving image" are selected as skip concatenated character components from the text data to which special characters are added. , "Image processing", "image processing", "processing", "processing", "apparatus", ... Are extracted.

Finally, the concatenated character component table registration program 2
Start 07. Here, the concatenation character hash table 216-a and the skip concatenation hash table 2
16-b through sequential concatenated character extraction program 400 and skip concatenated character extraction program 206, respectively.
"1" is set to the entry corresponding to the concatenated character component extracted by, and the existence of the concatenated character component is noted.

Next, the processing at the time of retrieval will be described in detail.

First, the character type division program 501 divides the search term for each character type. In the example shown in FIG. 39, the search term is a "moving image", and since it is composed entirely of Chinese characters, the "moving image" is cut out as it is by character type division. "^" Is added before and after ",
It becomes "^ moving image ^".

Next, the sequential concatenated character extraction program 206 extracts "^" from the search term to which the special character is added.
"Moving", "moving image", "image" and "image ^" are sequentially extracted as concatenated character components. Furthermore, skip concatenated character extraction 212 causes "^ image", "moving image" and "image ^" to be skipped concatenated. Next, the concatenated character component table 105 is extracted by the bit and program 213.
The bit list of the sequential concatenation character component is passed through the sequential concatenation hash table 215-a, and the skip concatenation character component is skip concatenated hash table 2
It is read via 16-b. Then, a document in which all the bits of all these bit lists are '1' is obtained as the search result of the concatenated character component table search. As a result, in the example of the figure, the document N is obtained as a hit document.

[0232] That is, when "moving image" is specified as the search term, "^ moving" as a sequential concatenated character from "^ moving image ^" with special characters added before and after the search term,
"Movie", "image" and "image ^" are extracted as skip concatenated characters "^ image", "moving image" and "image ^", whereas from documents including "automatic image processing" Sequential concatenated character components corresponding to “^ movement” and “image ^” and skip concatenated character components corresponding to “^ image” and “image ^” are not extracted. A document in which another word "automatic image processing" that appears as a partial character string appears can be excluded from the search target as noise.

As described above, in the process of creating and registering the concatenated character component table in the present embodiment, special characters are added before and after the Japanese text data divided for each character type at the time of document registration. Then, a continuous two-character string (sequential concatenated character) and a two-character string every other character (skip concatenated character) are extracted from the text data to which the special character is added, and the existence information of this concatenated character is pre-connected. Register in the character composition table. Also during search, after adding special characters before and after the search term divided into character types, by extracting sequential concatenation characters and skip concatenation characters, another word that includes the specified search term as a substring matches in the middle. You can avoid hitting and reduce noise. That is, when the word “automatic image processing” is registered in the text, “^ self”, “automatic”, “moving image”, “image”, “image processing”,
"Process" and "R ^" are "^" as skip concatenated characters.
"Motion", "self-portrait", "moving image", "image processing", "image" and "processing ^" are extracted. On the other hand, when "moving image" is specified as the search term, sequential connection is performed. As a character "^
Motion ", video", "image" and "image ^" will be extracted as skip concatenated characters "^ image", "moving image" and "image ^". Since "^ motion" and "image ^" are not sequentially extracted as a concatenated character from a document including the word "process", the document can be deleted as noise. it can.

As a result, it is possible to improve the narrowing rate in the concatenated character component table search. Therefore, the search amount of condensed text in hierarchical pre-search can be reduced,
Equivalently, the overall search speed will be improved. Therefore, a larger amount of full text search is possible in real time.

In this embodiment, the method of narrowing down the documents to be searched by using the concatenated character component table of both the consecutive concatenated character and the skip concatenated character has been described.
However, conventional contiguous contiguous characters (sequential concatenated characters)
Similarly, with the method using only, the specified search term is a partial character string by adding special characters before and after the text data divided into character types and the search term, and extracting sequential concatenation characters and skip concatenation characters. It will be clear that the noise caused by hitting an intermediate match with another word containing as can be reduced.

Next, an eighth embodiment of the present invention will be described.

In the fifth embodiment of the present invention, a plurality of concatenated character components are assigned to one entry in the concatenated character component table.
That is, the method of realizing the concatenated character component table with a practical capacity by hashing has been described. However, in this method, when a certain concatenated character is designated and the entry of the corresponding character component table is read, there is a possibility that a document containing a completely different concatenated character component can be obtained from the bit information. Therefore, in a large-scale document search system that registers a large number of documents, sifting out documents that are not related to the search term,
That is, narrowing down may not be performed accurately, and search efficiency may be reduced.

To solve this problem, in the eighth embodiment of the present invention,
When creating a concatenated character component table, for concatenated character components with a high frequency of occurrence, a bit list in which "1" is added to the bit position corresponding to the document number in which each concatenated character appears Stores the created document number. Furthermore, for concatenated character components with low frequency of occurrence, by storing the document numbers in which each concatenated character appears as a list of binary data, a concatenated character component table with no search noise due to hashing can be stored with a practical capacity. Take the way to make it happen.

This embodiment has basically the same configuration as that of the first embodiment shown in FIG. 1, except that the concatenated character component table 1 therein.
05, the concatenated character component table creation / registration program 205 and the concatenated character component table search program 211 have the configurations shown in FIGS. 40, 41 and 42, respectively.

That is, the concatenated character component table 105 in this embodiment is the bit list 105-a as shown in FIG.
And a document number list 105-b. Also,
The concatenated character component table creation / registration program 205 is, as shown in FIG. 41, a sequential concatenated character extraction program 400, a skip concatenated character extraction program 206, a concatenated character component table registration program 207, and a character appearance frequency calculation program 6.
00, a concatenated character component table creation registration program 2
The concatenated character component table registration program 207 in 05
It is composed of a document appearance frequency determination program 601, a bit list registration program 602, and a document number list registration program 603. Further, as shown in FIG. 42, the concatenated character component table search program 211 is a sequential concatenated character extraction program 401, a skip concatenated character extraction program 2
12, a concatenated character component table acquisition program 604 and a bit and program 213.

The outline of the concatenated character component table registration processing and search processing in this embodiment will be described below.

In the present embodiment, first, the concatenated character component table preparation registration program 205 activates the character appearance frequency calculation program 600 as preprocessing of the registration processing, and reads the text data from the text 103 into the work area 217. Then, a character string of two consecutive characters (sequential concatenated characters) appearing in the text data and a character string of two characters every other character.
For (skip concatenated characters), the number of documents in which each character string appears (the number of appearing documents) is calculated.

Next, the concatenated character component table creation / registration program 205 activates the concatenated character extraction program 400 in step 1600, as shown in FIG. Then, a continuous character string of two characters appearing in the text data for each document is sequentially extracted as a concatenated character.

The concatenated character component table creation program 20
In step 1601, the skip concatenated character extraction program 206 is activated in step 1601 to extract a character string of two characters every other character as a skip concatenated character for each document.

Further, the concatenated character component table creation / registration program 205 activates the concatenated character component table registration program 207 in step 1602, and the consecutively concatenated character extraction program 4
00 and the appearance information of each concatenated character extracted by the skip concatenated character extraction program 206 are registered in the concatenated character component table.

Next, the concatenated character component table registration program 20
An outline of the processing of No. 7 is shown in FIG.

The concatenated character component table registration program 207
First, in step 1610, the character appearance frequency determination program 601 is started to check whether the appearance frequency of each concatenated character extracted by the consecutive concatenated character extraction program 400 and the skip concatenated character extraction program 206 is larger than a predetermined threshold value. judge. If it is larger, the bit list registration program 602 is started in step 1611, and "1" is written in the bit position corresponding to the document number in which each concatenated character appears, thereby recording the appearance method. If it is smaller, the document number list registration program 603 is started in step 1612, and the document number in which each concatenated character appears is registered as binary data in the document number list to record the appearance information.

The above is the outline of the registration processing.

At the time of searching, the concatenated character component table search program 211 activates the concatenated character extraction program 401 in step 1620 as shown in FIG. 45, and extracts a character string of two consecutive characters from the search term as a concatenated character. .

Further, the concatenated character component table search program 211 activates the skip concatenated character extraction program 212 in step 1621, and extracts a character string of every two characters as a skip concatenated character.

Next, the concatenated character component table search program 2
11 starts the concatenated character component table acquisition program 603 in step 1622. Concatenated character component table acquisition program 6
In step 03, as shown in FIG. 46, it is determined in step 1630 whether the character component table corresponding to each concatenated character is stored in the bit list or the document number list. If it is stored in the document number list, step 1631 is activated and the corresponding bit string is acquired as it is as a concatenated character component table. If it is stored in the document number list, step 1632 is activated, and the bit position corresponding to each document number in the document number list is set to "1" to convert it to the bit list, and the corresponding concatenation is performed. Get the character composition table of a character.

Finally, the concatenated character component table search program 211 executes the bit and program 2 in step 1623.
13 is started, and a logical product operation is performed for each bit between the bit lists acquired by the concatenated character component table acquisition program 604. The document number corresponding to the bit that becomes "1" as a result of this logical product operation is output to the search control program 209 as the result of the concatenated character component table search.

The above is the outline of the registration and search processing of the concatenated character component table according to the present invention.

The details of the method of registering and searching the concatenated character component table in the embodiment will be described below with reference to examples. In the present embodiment, the case where the number of all registered documents is 1 million and the document number is stored in the document number list as 32-bit binary data will be described.

The search process will be described first.

In this embodiment, a character table and a file pointer table are used as a management table for acquiring the concatenated character component table for the concatenated character components extracted from the search term. FIG. 47 is a diagram showing an outline of search processing using the character table and the file pointer table.

As described above, when searching the concatenated character component table, the concatenated character component table search program 211 first starts the consecutively concatenated character extraction program 401 to successively concatenate two consecutive character strings from the search term. Extract as a character. For example, when the character string "moving image" is specified in the search term, "moving image" and "image" are sequentially extracted as concatenated characters.

Then, the skip concatenated character extraction program 212 is activated, and a character string of two characters is extracted as a skip concatenated character every other character from the search term. For example,
When the character string "moving image" is specified in the search term, "moving image" is extracted as a skip concatenated character.

In the following description, for simplification,
The registration method and search of the sequential concatenated character component table will be mainly described. Further, even in the skip connection, the search can be realized by the same processing.

Next, the concatenated character component table acquisition program 60
In 4, the pointer information to the file pointer table is obtained by referring to the record corresponding to the character code of the first character of the concatenated character extracted from the search term in the character table. For example, for the consecutively-connected character "moving image", the pointer information 560 to the file pointer table is obtained by referring to the record of the character table corresponding to the character code "moving" which is the character code of the first character.

Next, the file pointer table is referred to based on the pointer information obtained as a result of referring to the character table, and the identifier of the file in which the concatenated character component table for the corresponding concatenated character is stored (hereinafter referred to as file ID). And position information in the file (a storage position from the beginning of the file, which is also called an offset hereinafter) is obtained. That is, FIG.
In the example of No. 9, based on 560 obtained as a result of referring to the character table, each record from the beginning of the file pointer table to the 560th byte and thereafter is referenced to search for a record whose second character is "image". Through the above processing, it is possible to obtain 1 as the file ID and 1,034 as the offset as information for referring to the concatenated character component table for "moving image". In the file pointer table, the record having the second character 0 is stored in the first record of the first character, and the file ID for accessing the single character component table corresponding to the first character is stored. And store the offset. That is, in the example of this figure, the file ID and offset of the character component table corresponding to one character of "motion" are stored in the 580th byte of the file pointer table. By doing so, for example, when a record of 0 is detected as the second character without collating the "image" with the second character, the concatenated character "video" does not appear in the text data. Can be determined.

Next, a concatenated character component table corresponding to the corresponding concatenated character component is acquired from the file ID and offset obtained as a result of referring to the file pointer table. In this embodiment, the file ID for the bit list
And the file ID for the document number is specified in advance. By doing so, it is possible to determine whether the concatenated character component table corresponding to each concatenated character is composed of a bit list or a document number list depending on the value of the file ID. That is, in the example shown in FIG. 47, the file corresponding to the file ID 1 is a bit list and is 2
The file corresponding to is composed of a document number list. And the file ID corresponding to the specified concatenation character
Is 1, the bit list corresponding to the number of document registrations is read from the corresponding offset position in the file 1. If the file ID is 2, the number of documents (the number of appearing documents) in which the corresponding concatenated character appears is read from the corresponding offset position in the file 2. Next, by reading the document numbers corresponding to the number of appearing documents, the list of document numbers in which the corresponding concatenated characters appear is read. Then, by converting the obtained document number list into a bit list form, a concatenated character component table corresponding to the corresponding concatenated character is obtained.

In the example of FIG. 47, a file is used as access information to the concatenated character component table corresponding to the concatenated character "moving image".
Since 1 is obtained as the ID and 875,000 is obtained as the offset, a 125 kbyte (= 1,000,000 bit) bit string “01110101” corresponding to 1 million records starting from the 875 kth byte in file 1.
"01 ...." is read. This bit string corresponds to the document number from the first bit, and "1" indicates the document containing the concatenated character "video". Also, regarding the concatenated character "image" , 2 as the file ID and 1,084 as the offset are obtained, so by referring to the 1,084th byte from the beginning of the file 2, 34 is read as the number of documents including the concatenation character “image”. By reading the document numbers corresponding to the minutes, it can be seen that the document numbers in which the "image" appears are 783, 1038, ....
From this result, it is converted into a bit string by setting '1' to the position corresponding to the 783, 1038, ... Document ID in the bit list.

Finally, in the bit and program 213, the logical product of all these bit strings is taken, and the document whose result is '1' is obtained as the search result of the concatenated character component table search.

The above is a description of the concatenated character component table search processing in this embodiment.

A method of registering a concatenated character component table and a method of creating a character table and a file pointer table for realizing such a concatenated character component table search will be described.
In this example, the number of all registered documents is 1 million,
Since the document numbers are stored as 32-bit binary data in the document number list,
Use 31,250 cases (1 million bits ÷ 32 bits / case).

First, a method of calculating the number of appearing documents of each connected character component will be described. Here, the number of appearing documents of each concatenated character component is calculated using the character appearance frequency table shown in FIG. First, in the character appearance frequency table, 0 is registered in all data as an initial state. Next, the text data is read from the text 103 for each document into the work area 217, and a continuous two-character string is extracted from the text data. Then, the number of documents in which each concatenated character component appears (the number of appearing documents) is calculated by adding 1 to the data corresponding to the character code of the extracted concatenated character in the character appearance frequency table. For text data "image processing device with automatic image quality adjustment function ...", "automatic" as a concatenated string of two consecutive characters,
"Video", "image quality", ... are extracted, 1 is added to the data corresponding to each character code in the character appearance frequency table, and the number of documents in which each concatenated character component appears in all registered documents is calculated. To do.

Next, the concatenated characters appearing in the text data are extracted by extracting the concatenated characters whose value is not 0 in the character appearance frequency table created earlier.

That is, in the example shown in FIG. 50, since the value of the character appearance frequency table for "movement" is 0, it is not extracted as a concatenated character appearing in the text data.

Since the value of the character appearance frequency table for "moving image" is not 0, it is extracted as a concatenated character appearing in the text data. Then, since the number of appearing documents is larger than the threshold value of 31,250, an area of 125 kbytes (= 1,000,000 bits) corresponding to the number of registered documents is allocated in the bit list. Further, “image” is the second character in the file pointer table, “1” is the file ID representing the bit list, and the offset is equivalent to the start offset of the area allocated in the bit list.
Write 5,000. When "0" is written as the second character, the offset value in the corresponding file pointer table is written in the data corresponding to the first character of the concatenated character component of the character table.

Next, since the value of the character appearance frequency table for "image" is not 0, it is extracted as a concatenated character appearing in the text data. Since the number of appearing documents is smaller than the threshold value of 31,250, the number of appearing documents “56” is written in the document number list, which corresponds to the number of appearing documents.
Allocate an area of bytes (4 bytes / case x 34 cases). Further, "image" is written as the second character in the file pointer table, "2" representing the document number list as the file ID, and 1,084 corresponding to the start offset of the area allocated in the document number list is written as the offset.

As described above, in the present embodiment, preparation is made for registering the concatenated character component table by allocating the storage areas of the bit list and the document number list for each concatenated character in advance.

Next, the process of registering the concatenated character component table will be described, taking as an example the case where the character string "moving image" appears as the text data of the document number 783.

First, by reading the text data from the text 103 into the work area 217 one by one, and extracting a character string of two consecutive characters appearing in the text data, "moving image" and "image" are successively connected. To extract. Next, by referring to the character table and the file pointer table in the same manner as when searching the concatenated character component table, the file ID and offset in which each concatenated character is stored are obtained. For example, the consecutive concatenated character "video"
For, the file ID is "1" and the offset is 875,000. Then, a bit string of 1 million items, that is, a 125 kbyte bit string is read from the first 875 kbyte of the bit list corresponding to "1" as the file ID, and "1" is set to the bit position corresponding to the 783th document number. It is noted that the character string "video" appeared in the document number 783 by setting. Similarly, for "images", by referring to the character table and file pointer table, "2" is set as the file ID.
Is obtained as an offset of 1,084. And file I
By referring to the 1,084th byte of the document number list corresponding to "2" as D, information that the number of appearing documents of "image" is 34 is obtained. And the following 34 in the document number list
Read the document number corresponding to the case, and from it, "0"
As binary data for the first appearing entry
It is noted that by writing 783, the character string "image" appeared in the document number 783.

The above is the details of the registration processing of the concatenated character component table in this embodiment.

As described above, in the concatenated character component table according to the present embodiment, the appearance information is a bit list for a concatenated character having a high appearance frequency, and the appearance information is a document number list for a concatenated character having a low appearance frequency. Store. By doing so, a concatenated character component table in which noise due to hashing does not occur can be realized with a practical capacity.

In this embodiment, as the preprocessing of the registration process, the appearance frequency of each concatenated character component is calculated by referring to the text data corresponding to the entire text of the registered document, and the areas of the bit list and the document number list are The method of allocating was used. However, using the statistical information in advance, a bit list is allocated to the concatenated character that is determined to have a high appearance frequency, and a capacity corresponding to the expected number of appearing documents is allocated to the concatenated character that is determined to be a low occurrence frequency. As a result, it becomes unnecessary to execute the appearance frequency calculation program.

Further, in the present embodiment, in the document retrieval method of the fifth embodiment of the present invention, the appearance information is a bit list for the concatenated characters having a high appearance frequency, and the appearance information is a bit list for the concatenated characters having a low appearance frequency. The method of storing the appearance information in the document number list has been described. However, it will be clear that it is also applicable to all the embodiments described so far.

Finally, in the present embodiment, the method of performing the registration processing of the concatenated character component table in one registration processing for one million documents has been described. Alternatively, a method of creating a bit list and a document number list and merging them later to create a concatenated character component table for 1 million cases may be used.
At this time, the capacity of the table required to create the concatenated character component table is small. Therefore, even in a computer equipped with a small amount of memory capacity, the table can be stored in the memory and the document can be registered.
It has the feature that registration time can be shortened.

[0280]

According to the present invention, a concatenated character component table in which noise due to hashing does not occur in hierarchical presearch can be realized with a practical capacity. Furthermore, even when a character string composed of phonetic characters such as English, and a character string composed of a combination of words are specified as a search term, it is possible to further improve the narrowing rate of the concatenated character component table, Since unnecessary condensed text search can be largely omitted, it becomes possible to perform full-text search with a practical response time even for a large-scale document database.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a first exemplary embodiment of the present invention.

FIG. 2 is an explanatory diagram of a conventional example.

FIG. 3 is a diagram showing an operation in a first document search method.

FIG. 4 is a PAD diagram showing a document registration procedure in the first embodiment of the present invention.

FIG. 5 is a PAD showing a processing procedure of a concatenated character component table creation registration program in the first embodiment of the present invention.

FIG. 6 is a PAD diagram showing a control procedure of hierarchical search.

FIG. 7 is a diagram showing a method of extracting concatenated character components according to the first embodiment of the present invention.

FIG. 8 is a diagram showing a method of creating a concatenated character component table in the first embodiment of the present invention.

FIG. 9 is a PAD diagram showing a method for searching a concatenated character component table in the first embodiment of the present invention.

FIG. 10 is a diagram showing a method for searching a concatenated character component table according to the first embodiment of the present invention.

FIG. 11 is a diagram showing the structure of a concatenated character component table creation / registration program in the second embodiment of the present invention.

FIG. 12 is a PAD showing a processing procedure of a concatenated character component table creation registration program in the second embodiment of the present invention.

FIG. 13 is a diagram showing a method for creating a concatenated character component table in the second embodiment of the present invention.

FIG. 14 is a diagram showing the structure of a concatenated character component table creation / registration program according to the third embodiment of the present invention.

FIG. 15 is a diagram showing the structure of a concatenated character component table search program in the third embodiment of the present invention.

FIG. 16 is a PAD showing a processing procedure of a concatenated character component table creation registration program in the third embodiment of the present invention.

FIG. 17 is a PAD showing a processing procedure of a concatenated character component table search program according to the third embodiment of the present invention.

FIG. 18 is a diagram showing a method for creating a concatenated character component table according to the third embodiment of the present invention.

FIG. 19 is a diagram showing a method for searching a concatenated character component table according to the third embodiment of the present invention.

FIG. 20 is a diagram showing the structure of a concatenated character component table creation / registration program according to the fourth embodiment of the present invention.

FIG. 21 is a diagram showing the structure of a concatenated character component table search program according to the fourth embodiment of the present invention.

FIG. 22 is a PAD showing a processing procedure of a concatenated character component table creation registration program in the fourth example of the present invention.

FIG. 23 is a PAD showing a processing procedure of a concatenated character component table search program according to the fourth embodiment of the present invention.

FIG. 24 is a diagram showing a method of creating a concatenated character component table according to the fourth embodiment of the present invention.

FIG. 25 is a diagram showing a method for searching a concatenated character component table according to the fourth embodiment of the present invention.

FIG. 26 is a diagram showing a method for creating a concatenated character component table according to the fifth embodiment of the present invention.

FIG. 27 is a diagram showing a method for searching a concatenated character component table according to the fifth embodiment of the present invention.

FIG. 28 is a diagram showing the structure of a concatenated character component table creation / registration program according to the sixth embodiment of the present invention.

FIG. 29 is a diagram showing the structure of a concatenated character component table search program according to the sixth embodiment of the present invention.

FIG. 30 is a PAD showing a processing procedure of a concatenated character component table creation / registration program according to the sixth embodiment of the present invention.

FIG. 31 is a PAD showing the processing procedure of the concatenated character component table search program in the sixth embodiment of the present invention.

FIG. 32 is a diagram showing a method for creating a concatenated character component table according to the sixth embodiment of the present invention.

FIG. 33 is a diagram showing a method for searching a concatenated character component table according to the sixth embodiment of the present invention.

FIG. 34 is a diagram showing the structure of a concatenated character component table creation / registration program according to the seventh embodiment of the present invention.

FIG. 35 is a diagram showing a structure of a concatenated character component table search program in the seventh embodiment of the present invention.

FIG. 36 is a PAD showing a processing procedure of a concatenated character component table creation registration program in the seventh embodiment of the present invention.

FIG. 37 is a PAD showing a processing procedure of a concatenated character component table search program in the seventh embodiment of the present invention.

FIG. 38 is a diagram showing a method for creating a concatenated character component table in the seventh embodiment of the present invention.

FIG. 39 is a diagram showing a method for searching a concatenated character component table according to the seventh embodiment of the present invention.

FIG. 40 is a diagram showing the structure of a concatenated character component table according to the eighth embodiment of the present invention.

FIG. 41 is a diagram showing the structure of a concatenated character component table creation / registration program according to the eighth embodiment of the present invention.

FIG. 42 is a diagram showing the structure of a concatenated character component table search program according to the eighth embodiment of the present invention.

FIG. 43 is a PAD showing a processing procedure of a concatenated character component table creation registration program in the eighth embodiment of the present invention.

FIG. 44 is a PAD showing the processing procedure of the concatenated character component table registration program in the eighth embodiment of the present invention.

FIG. 45 is a PAD showing a processing procedure of a concatenated character component table search program according to the eighth embodiment of the present invention.

FIG. 46 is a PAD showing a processing procedure of the concatenated character component table acquisition program in the eighth embodiment of the present invention.

FIG. 47 is a diagram showing a method for searching a concatenated character component table according to the eighth embodiment of the present invention.

FIG. 48 is a diagram showing a method for creating a concatenated character component table in the eighth embodiment of the present invention.

[Explanation of symbols]

100 ... Display 101 ... Keyboard 102
... CPU 103 ... Text 104 ... Condensed Text 105
... Concatenated character component table 106 ... Floppy disk drive (FDD) 107
... floppy disk 108 ... bus 110 ... magnetic disk 200
… Main memory

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Natsuko Mizutani 1099, Ozenji, Aso-ku, Kawasaki-shi, Kanagawa Hitachi, Ltd. System Development Laboratory (72) Hisamitsu Kawaguchi 1099, Ozen-ji, Aso-ku, Kawasaki, Kanagawa Hitachi, Ltd. Mfg. Co., Ltd. System Development Research Center (72) Inventor Kanji Kato 1099 Ozenji, Aso-ku, Kawasaki-shi, Kanagawa Hitachi, Ltd. System Development Research Center (72) Inventor Satoshi Asakawa 5030 Totsuka-cho, Totsuka-ku, Yokohama-shi Hitachi, Ltd. (56) References JP-A-7-319920 (JP, A) JP-A-5-174064 (JP, A) JP-A-5-81321 (JP, A) JP-A-4-274557 ( (58) Fields surveyed (Int.Cl. ⁷ , DB name) G06F 17/30 JISC file Le (JOIS)

Claims (6)

(57) [Claims] 1. A partial character string is extracted from each document stored in advance in a predetermined format, and a concatenated character component table indicating whether or not the partial character string exists in each document is created. A search substring is extracted in a predetermined format from the search term input to search for a desired document from each document, and the concatenated character component table corresponding to the extracted search substring is displayed. Search for documents that have substrings that match the search substrings that make up the search term,
In a document search method for filtering out documents unrelated to the search term from a search target, every predetermined m characters (m is an integer of 1 or more) from the document, a predetermined n characters (n is 2 or more) Character string of the integer) is extracted as the partial character string, and every predetermined m characters (m is an integer of 1 or more) from the search term, a predetermined n characters (n is an integer of 2 or more) are extracted. Is extracted as the search sub-character string, and the concatenated character component table has an appearance frequency higher than a predetermined threshold value.
Bit position corresponding to the document number where the high concatenation character appears
Bits that register appearance information of character strings by writing 1 in
List and concatenated sentences that appear less frequently than a given threshold
For character components, a sequence of occurrences that is
Document numbers in which suffixes appear are a list of binary data
And a document ID list stored in the type and each of the connecting character component appearing previously in said document
When the number of documents in which the concatenated character component appears is calculated and the calculated number of documents is larger than a predetermined threshold value.
Is the occurrence of the corresponding concatenation character for the bit list.
By writing '1' in the bit position corresponding to the document number
When the appearance information of the concatenated character component is registered and the calculated number of documents is smaller than a predetermined threshold value.
Is the document number in which the relevant concatenation character appears
Write it in the document number list as a list of data.
The appearance information of each concatenated character component is stored by, and is extracted for the concatenated character extracted from the search term.
The bit list or document number
Read the list and in case of the document number list,
A document retrieval method, characterized in that a concatenated character component table is obtained by converting it into a list . 2. The document search method according to claim 1, wherein when extracting a partial character string from the document, a word is cut out from the document, and predetermined m characters (m is 1 or more) from the cut out word. A character string of predetermined n characters (n is an integer of 2 or more) is extracted as the partial character string. 3. The document search method according to claim 2, wherein a continuous character string of predetermined n characters (n is an integer of 2 or more) is extracted from the cut word as the partial character string, A document search method, wherein a series of predetermined n characters (n is an integer of 2 or more) is extracted from a search term as the search partial character string. 4. The document search method according to claim 2, wherein a predetermined code is added before and after the cut word, and a predetermined m characters (m is 1 or more) from the word to which the predetermined code is added. Every n), a character string of a predetermined n characters (n is an integer of 2 or more) is extracted, a predetermined code is added before and after the search term, and the search term to which the predetermined code is added is extracted. Predetermined n for every predetermined m characters (m is an integer of 1 or more)
A document search method comprising extracting a character string of characters (n is an integer of 2 or more) as a search character string. 5. The document search method according to claim 1, wherein a string of predetermined i characters (i is an integer of 2 or more) is extracted from the document as the partial character string, and the partial character string is extracted. Every predetermined m characters (m is an integer of 1 or more) predetermined from the document, a predetermined n characters (n is 2)
A character string of the above integer) is extracted as the partial character string, and a series of predetermined i characters (i is an integer of 2 or more) is extracted from the search term as the search partial character string. At the same time, a predetermined n-character (n is an integer of 2 or more) character string is extracted as the search character string from the search term at every predetermined m characters (m is an integer of 1 or more). A document retrieval method characterized by. 6. extracts substrings in a predetermined format from prestored respective document, means for creating a concatenated character component table indicating whether the substring is present in each of the document Corresponding to the extracted partial character string for search, which has a unit for extracting a partial character string for search in a predetermined format from a search term input to search for a desired document from each of the documents. In the document search device that refers to the concatenated character component table and obtains a document in which a partial character string that matches each search partial character string that constitutes a search term exists, the concatenated character component table has a predetermined threshold value. A bit list for registering appearance information of a character string by writing 1 in a bit position corresponding to a document number in which a concatenated character having a higher appearance frequency, and a concatenated character component having an appearance frequency lower than a predetermined threshold value And a document number list storing, as a list of binary data, document numbers in which concatenated characters whose appearance frequency is lower than the predetermined appearance frequency are stored. Means for extracting a predetermined character string of n characters (n is an integer of 2 or more) as the partial character string for each character (m is an integer of 1 or more), and a predetermined number of m characters from the search term. Means for extracting a predetermined character string of n characters (n is an integer of 2 or more) as the search partial character string every (m is an integer of 1 or more), and a concatenated character component that appears in the document in advance. And a means for calculating the number of documents in which each concatenated character component appears, and if the calculated number of documents is larger than a predetermined threshold value, the document number in which the corresponding concatenated character appears in the bit list Means for registering the appearance information of the concatenated character component by writing '1' in the corresponding bit position; and, if the calculated number of documents is smaller than a predetermined threshold value, the document in which the corresponding concatenated character appears A means for registering the appearance information of each concatenated character component by writing the number to the document number list as a list of binary data, and a bit corresponding to the extracted concatenated character for the concatenated character extracted from the search term. A document retrieving apparatus comprising: a list or a document number list; and a means for acquiring a concatenated character component table by converting the list or the document number list into a bit list in the case of the document number list.