JP2005208852A - Summary registering apparatus, summary registration method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently generate a summary on which the retrieval intention of a user is reflected. <P>SOLUTION: A registration part 10 retrieves document data related to a designated index word from a document data group being a retrieval object, and returns the document data as the retrieval result. In a summary group storage part 20 which stores the summary data of the document data, the summary data of the document data being a new retrieval object is registered. Furthermore, the registration part 10 divides the document data being the new retrieval object into frame units, and extracts each index word from each phrase included in a frame group obtained by dividing the document data. Then, the registration part 10 preferentially selects the frame including the index word which does not overlap another phrase from the phrase group, and registers the phrase in a summary group storage part 20 as the summary data of the new document data. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention provides a new summary storage device for storing summary data of document data to be returned as a search result obtained by searching document data related to a designated index word from a search target document data group. The present invention relates to a summary registration device for registering summary data of document data.

  When a large-scale search system such as an Internet search engine is used, the contents of summary data indicating summary sentences or sample sentences included in search results may be used as clues. Thus, by determining whether or not the retrieved document data conforms to the retrieval intention based on the contents of the summary data, it is not necessary to acquire the actually retrieved document data and browse the contents. Therefore, it is possible to save time required for retrieving the intended document data.

  In general, the summary is generated by extracting important parts from all text data in the document data, using the extracted important parts as summary data (static summary), and accumulating all the text data in the document data and using the search conditions. Are classified into two types: summary data (dynamic summary).

  However, the static summary is not necessarily a summary reflecting the user's search intention because it displays certain information regardless of the keyword input by the user during the search. Therefore, it may be difficult to reduce the burden on the user who determines whether or not the search intention is satisfied. In order to solve this problem, a dynamic summarization technique for dynamically displaying information in a document related to a keyword input by a user has been researched and developed. For example, in a typical search engine such as Google, a summary is dynamically generated by a method called KWIC (Keyword in Context). Also, a technique for extracting a portion having a high degree of association with a search sentence from a document as in Patent Document 1 is being studied. Since the dynamic summary is generated based on the user's search word or search sentence, it is considered possible to generate a summary that reflects the user's search intention. However, when all the text data is accumulated for the summary generation, the database is enlarged as the information to be searched increases, and the search time and the summary generation time may become longer.

Japanese Patent Laid-Open No. 2001-167096

  An object of the present invention is to efficiently generate a summary data group covering index terms included in document data.

  The summary registration apparatus according to the present invention searches for document data related to a specified index word from a search target document data group, and stores summary data of document data to be returned as a search result. And a summary registering device for registering summary data of new search target document data, a splitting unit for splitting the new search target document data in a predetermined unit, and a split data group obtained by splitting Index word extracting means for extracting index words from each divided data included in the data, and preferentially selecting divided data including index words that do not overlap with index words included in other divided data from the divided data group Registering means for registering in the summary storage device as summary data of the new document data.

  According to the present invention, the divided data including the index words that do not overlap with the index words included in the other divided data from the divided data group obtained by dividing the document data to be searched by the dividing means is registered. Is preferentially selected and registered in the summary storage device as summary data of the new document data. Therefore, summary data storage device that preferentially covers summary data that can cover as many index terms as possible while limiting the size and number of summary data stored per document data and suppressing the capacity of the summary storage device. For example, although the document data is related to the index word specified in the search condition, the index data is not included in the summary data registered in the summary storage device, A situation in which summary data of the document data cannot be returned to the user can be suppressed.

  According to one aspect of the summary registration apparatus according to the present invention, the registration unit is divided data including an index word that does not overlap with an index word included in another divided data from among the divided data group, The divided data including more index words than the divided data is preferentially selected and registered in the summary storage device as the summary data of the new document data.

  According to the present invention, divided data including more index words can be preferentially registered in the summary storage device as summary data. For example, the user can specify a number of index words as a search condition and perform a search. In such a case, it is possible to increase the possibility that summary data including many index words can be returned.

  Embodiments of the present invention (hereinafter referred to as embodiments) will be described with reference to the drawings.

  FIG. 1 is a functional block diagram of a document search apparatus according to this embodiment. The document search apparatus includes a registration unit 10 that registers summary data of collected document data in the summary group storage unit 20, a summary group storage unit 20 that stores summary data registered by the registration unit 10, and an index specified by a user. And a search unit 30 that extracts summary data related to document data searched based on words (keywords) from the summary data group stored in the summary group storage unit 20. In the present embodiment, the document data refers mainly to text data composed of character codes defined by JIS Kanji codes, ASCII codes, or the like.

  The document search device is used, for example, when searching for document data related to the Web held by a Web server connected to the Internet and existing on the Internet. Further, it is also used when searching for document data such as instructions and office documents of devices and the like that are connected to an in-house intranet and possessed by a file server existing in the intranet.

  According to the document search device configured as described above, the summary data indicating a part of the content of the document data related to the index word specified by the user is extracted from the collected document data group, and the summary data is extracted. To the user. The user refers to the content of the returned summary data, and the search result returned from the document search device reflects his / her search intention without referring to the entire content of the document data related to the index word. It can be determined whether or not it is a thing.

  As a method of generating summary data returned as a search result in this way, for example, there is a method of generating a summary based on KWIC (Keyword in Context). This technique is a technique for accumulating all collected document data and generating and returning summary data including an index word of a search condition each time, and is widely used for summary display of a WWW search engine. However, the capacity of the database for storing the collected document data is physically limited, and the amount of document data that can be stored in the database is also limited. Further, when the database is enlarged as the document data increases, the search time and the summary generation time become longer.

  In particular, when only WWW content is searched, the size of all text per content is about several kilobytes. However, when searching for general office documents, all text data per office document is used. Is often more than 10 times the size of all text data per WWW content, and the method of storing all document data like KWIC reduces the number of office documents that can be stored in the database. There is.

  In order to solve the above-mentioned problem, a part of phrases is extracted from the collected document data in advance as summary data, and the relationship with words extracted by morphological analysis is stored in a database, so that each document data A method of limiting the size and number of summary data to be stored, reducing the database capacity, and preventing an increase in search time can be considered. However, since this method does not store all the document data, all index words included in the document data are not always included in the extracted partial phrase. Therefore, although the document data is related to the index word specified in the search condition, the summary data registered in the database does not include the index word, and the user summarizes the document data. Data may not be returned. In particular, this is more likely to occur as the amount of document data increases. If summary data including an index word cannot be returned to the user in this way, the user actually refers to the contents of the document data to determine whether or not the document data obtained as a search result is appropriate document data including the index word. Otherwise, it cannot be determined and the burden on the user increases.

  Therefore, in the present embodiment, a document search that generates summary data that can cover as many index terms contained in document data as possible while limiting the size and number of summary data stored per document data and suppressing the database capacity. Providing equipment.

  Here, the details of the registration unit 10 that cuts out part of the document data and registers it as summary data in the summary group storage unit 20 will be described with reference to the functional block diagram shown in FIG.

  The document data collection unit 11 is a module that sequentially collects document data from a document management apparatus that holds document data such as a Web server or a file server that exists in a network to which the document search apparatus is connected by robot search or the like. The phrase / index word table creation unit 12 is a module that creates a phrase / index word table as shown in FIG. 3 based on the collected document data. The creation of the phrase / index word table will be described in detail later. The registered phrase extraction unit 13 is a module that extracts a phrase to be registered as summary data based on the phrase / index word table and registers it in the summary group storage unit 20.

  Furthermore, the phrase / index word table creation method in the phrase / index word table creation unit 12 will be described in detail. The phrase / index word table creation unit 12 first divides the collected document data into a plurality of phrases (phrases) with punctuation marks as a unit, and each phrase obtained by the division is unique from the top of the document. A phrase ID is assigned. Note that the document data may be divided into other units such as a punctuation mark as a unit or a fixed number of words as a unit.

  After the division, the phrase / index word table creation unit 12 extracts index words included in each obtained phrase by so-called morphological analysis processing. Based on the result thus obtained, the phrase / index word table creation unit 12 creates a phrase / index word table as shown in FIG. In the phrase / index word table shown in FIG. 3, the phrase with phrase ID: 0 indicates that “W1” and “W3” are included as index words.

  Next, a summary data registration method in the registered phrase extraction unit 13 will be described. First, (W0, W1, W2, W3, W4,..., Wx) = (0, 0, 0, 0, 0,..., 0) is defined as the initial vector VO. Here, the number of arranged “0” s matches the number of index words extracted from the phrase group obtained by dividing the document data. Each phrase is similarly expressed as a vector. For example, in the case of the vector V0 of the phrase ID: 0, (0, 1, 0, 1, 0, ..., 0) is obtained. That is, of the phrase vector element group, only the element at the position corresponding to the index word included in the phrase is expressed as “1”.

  The registered phrase extraction unit 13 first performs a logical sum of each element of the initial vector VO and each element of the vector V0 with the phrase ID: 0. The result vector V thus obtained is (0, 1, 0, 1, 0,..., 0). Subsequently, the logical sum of each element of the result vector V obtained by taking the logical sum and each element of the vector V1 of the phrase ID: 1 is obtained. As a result, a result vector V of (0, 1, 0, 1, 1,..., 0) is obtained.

  In this way, the logical sum of each element of the result vector Vold obtained by taking the previous logical sum and each element of the phrase vector is taken while sequentially incrementing the value of the phrase ID. As a result, if there is a phrase including an element changed from “0” to “1” in the element group of the obtained result vector Vnew, the phrase is set as a registration target phrase. In other words, the fact that the element of the result vector Vold has changed from “0” to “1” means that the index word that is not included in the previously logically ORed phrase is subject to the current ORing. Since it means that the phrase is included in the phrase, the phrase is preferentially registered in the summary group storage unit 20 as summary data.

  In the above example, both the phrase ID: 0 phrase and the phrase ID: 1 phrase include the elements changed from “0” to “1” in the element group of the result vector V. Both ID: 0 and phrase ID: 1 are registration target phrases. On the other hand, if the logical sum with each element of the vector V2 of the phrase ID “2” is taken, in this case, the result vector V remains (0, 1, 0, 1, 1,..., 0). There is no change in the elements. That is, the index word included in the phrase with the phrase ID “2” means that only the index word included in the phrase with the phrase ID: 0 or the phrase ID: 1 already listed as the registration target phrase exists. Therefore, even if the phrase with the phrase ID: 2 is not registered as summary data, another phrase including the index word included in the phrase with the phrase ID: 2 is registered as summary data. Although the document data is related to the index word, the summary data registered in the summary group storage unit 20 does not include the index word, and the summary data of the document data is returned to the user. The phenomenon of not being able to do not happen.

  In this way, the document data is divided into a plurality of phrases, and phrases including index words that are not included in other phrases are preferentially extracted as registration target phrases from the phrase group. All the registration target phrases may be registered in the summary group storage unit 20 as summary data of the document data.

  However, if the number of registration target phrases extracted is large, the capacity of the summary group storage unit 20 may be reduced, and summary data may not be registered for a large amount of document data. Therefore, the registered phrase extraction unit 13 in the present embodiment limits the number of summary data that can be registered for one document data to a certain number. That is, the logical sum of the result vector V (initially the initial vector VO) in order from the phrase ID 0 is taken, and when the number of phrases listed as registration target phrases exceeds a predetermined reference value, the document data The registration of the summary data for is stopped, and the summary data registration process for another document data is performed.

  If the number of phrases listed as summary candidates does not exceed the predetermined reference value, only the registration target phrase may be registered in the summary group storage unit 20 as it is, but the process may be terminated. In order to register as much summary data as possible, in this embodiment, other phrases that are not listed as registration target phrases are extracted in ascending order of phrase IDs, and the summary data of the document data is satisfied until the reference value is satisfied. Will be registered as

  Using the summary data registered in this way, the search unit 30 returns a search result to the user as follows. That is, the search unit 30 receives an index word designated as a search condition from the user, and searches for document data related to the index word. Then, if there is document data that meets the search condition, based on the identification information of the document data, the summary data corresponding to the document data from the summary group storage unit 20, and the designated index word At least one summary data is extracted, and the summary data is returned as a search result to the user. The user determines the outline of the document data obtained as a search result from the contents of the returned summary data, and determines whether the document data satisfies the search condition of the user.

  Each functional block may be configured by hardware or software. In the case of using software, the CPU installed in the document search apparatus may read the program stored in the memory and execute the summary data registration process.

  Next, an overall operation flow of summary data registration in the registration unit 10 will be described based on the flowchart shown in FIG.

  First, the document data collected by the document data collecting unit 11 is divided into phrases by the phrase / index word table creating unit 12, and index words are extracted by morphological analysis for each phrase to create a phrase / index word table. (S101). Then, the registered phrase extracting unit 13 determines whether the total number of phrases M obtained by dividing based on the created phrase / index word table is equal to or greater than a predetermined reference value N (for example, N = 160). Is determined (S102). If the total number of phrases M is equal to or less than the reference value N as a result of the determination in S102, the registered phrase extraction unit 13 associates all phrases obtained by dividing the document data with the identification information of the document data and summarizes the data. Is registered in the summary group storage unit 20 (S103).

  On the other hand, as a result of the determination in S102, if the total number of phrases M is equal to or greater than the reference value N, the process proceeds to a phase in which phrases to be registered as summary data are selected from phrase groups obtained by subsequent division. That is, first, the registered phrase extracting unit 13 extracts a counter I indicating the value of the phrase ID of the vector Vi corresponding to the phrase to be ORed with the result vector V, and the phrase extracted as a registration target phrase from the phrase group. The counter C indicating the number of the counters is reset to “0” (S104). Then, the logical sum of each element of the phrase vector Vi having the phrase ID of the value indicated by I and each element of the result vector is calculated (S105). The result of the logical sum is set as a new result vector Vnew, and the result of the logical sum taken last time is set as a result vector Vold.

  Subsequently, the registered phrase extraction unit 13 determines whether Vnew and Vold are the same (S106). That is, the registered phrase extraction unit 13 determines whether or not the phrase of the vector Vi includes an index word that is not included in the phrase already listed as the registration target phrase.

  As a result of the determination in S106, if Vnew and Vold are not the same, it means that the phrase of the vector Vi includes an index word that is not included in the phrase already listed as the registration target phrase. Using the phrase as a registration target phrase, the value of the counter C is incremented by 1, and Vnew is changed to Vold (S107). Thereafter, the registered phrase extraction unit 13 determines whether or not the value of the counter C is the same as the reference value N (S108).

  As a result of the determination in S108, if the value of the counter C is the same as the reference value N, it means that the number of registration target phrases has already reached the reference value N. Therefore, the registered phrase extraction unit 13 registers at this time. The selection of the target phrase is stopped, and all the phrases listed as registration target phrases at the present stage are registered in the summary group storage unit 20 as summary data in association with the identification information of the document data (S109).

  On the other hand, if the result of determination in S108 is that the value of the counter C does not match the reference value N, it means that the number of registration target phrases has not yet reached the reference value N. Determines whether or not the value of counter I is M-1 (whether or not the last phrase has been reached) in order to determine whether or not all the phrases obtained by division have been ORed with the result vector (S110).

  If the result of determination in S110 is that the value of the counter I does not match M-1, an index that has not yet been ORed with the result vector, that is, is not already included in the phrase listed in the registration target phrase This means that there is a phrase that has not been determined whether or not it contains a word. Therefore, the value of the counter I is incremented by one (S111), and the phrase having the same phrase ID as the value of the incremented counter C The above process is repeated from S105.

  On the other hand, if the value of the counter I matches M−1 as a result of the determination in S110, this means that all the phrases already obtained by dividing the document data have been logically ORed. All the phrases listed as registration target phrases at the time are registered in the summary group storage unit 20 as summary data in association with the identification information of the document data (S112). Furthermore, since the number of phrases listed as registration target phrases has not yet reached the reference value N, other phrases that are not listed as registration target phrases until the value of the counter C reaches the reference value N The IDs are extracted in ascending order and registered together in the summary group storage unit 20 as summary data (S113).

  According to the present embodiment, the registration unit 10 preferentially selects and summarizes phrases including index words that are not included in other phrases from among a group of phrases obtained by dividing the collected document data. Register in the group storage unit 20. Therefore, summary data that can cover as many index terms as possible while limiting the size and number of summary data stored per document data and reducing the capacity of the summary group storage unit 20 is preferentially summarized. A document search device registered in the storage unit 20 can be provided, and the summary data registered in the summary group storage unit 20 is the document data related to the index word specified by the search condition. It is possible to suppress a situation in which the index word is not included and the summary data of the document data cannot be returned to the user.

  In the present embodiment, an example is shown in which phrase IDs are assigned in the order in which documents existing in document data appear, and a phrase / index word table is created. However, once the document data is once divided into phrases, the phrase IDs may be assigned in order from the phrase including many index words to create a phrase / index word table. By creating the phrase / index word table in this way, phrases including more index words can be preferentially registered in the summary group storage unit 20 as summary data. Therefore, when a user performs a search by specifying many index terms as a search condition, it is possible to increase the possibility that a phrase including many index terms can be returned as summary data.

It is a figure which shows the functional block of the document search apparatus in this embodiment. It is a figure which shows the functional block of the detail of the registration part in the document search device of this embodiment. It is a figure which shows an example of the phrase and index word table produced in the document inspection apparatus of this embodiment. It is a flowchart figure which shows the process which registers summary data in a summary group accumulation | storage part in the document search device of this embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 registration part, 11 document data collection part, 12 phrase and index word table preparation part, 13 registration phrase extraction part, 20 summary group accumulation | storage part, 30 search part.

Claims (6)

Document data related to the specified index word is searched from the search target document data group, and the new document data to be searched is stored in the summary storage device for storing the summary data of the document data to be returned as the search result. A summary registration device for registering summary data of
Dividing means for dividing new document data to be searched in a predetermined unit;
Index word extraction means for extracting an index word from each divided data included in the divided data group obtained by dividing;
Registration means for preferentially selecting divided data including an index word that does not overlap with an index word included in other divided data from the divided data group and registering it in the summary storage device as summary data of the new document data When,
A summary registration device comprising: The summary registration device according to claim 1,
The registration unit preferentially selects divided data including an index word that does not overlap with an index word included in another divided data from the divided data group, and includes more index words than other divided data. A summary registration device that selects and registers in the summary storage device as summary data of the new document data. A new search target document is stored in the summary storage device for storing the summary data of the document data, which is retrieved from the search target document data group and the document data related to the specified index word is returned as a search result. A summary registration method for registering summary data of data,
A division step of dividing new search target document data by a predetermined unit;
An index word extraction step for extracting an index word from each divided data included in the divided data group obtained by dividing;
A registration step of preferentially selecting divided data including an index word that does not overlap with an index word included in another divided data from the divided data group, and registering it in the summary storage device as summary data of the new document data When,
Summary registration method including In the summary registration method according to claim 3,
In the registration step, divided data including an index word that does not overlap with an index word included in another divided data from among the divided data group, and the divided data including more index words than the other divided data is prioritized. And registering the new document data as summary data in the summary storage device. A new search target document is stored in the summary storage device for storing the summary data of the document data, which is retrieved from the search target document data group and the document data related to the specified index word is returned as a search result. A program for causing a computer to execute processing for registering data summary data,
A division step of dividing new document data to be searched in a predetermined unit;
An index word extraction step for extracting an index word from each divided data included in the divided data group obtained by dividing;
A registration step of preferentially selecting divided data including an index word that does not overlap with an index word included in other divided data from the divided data group, and registering it in the summary storage device as summary data of the new document data When,
A program that causes a computer to execute. The program according to claim 5,
In the registration step, divided data including an index word that does not overlap with an index word included in another divided data in the divided data group, and the divided data including more index words than the other divided data is prioritized. And registering it in the summary storage device as summary data of the new document data.

Images