JP2011103075A - Method for extracting excerpt sentence - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately extract a principal part in a document file corresponding to a search query, as an excerpt sentence. <P>SOLUTION: A network system S includes: a database 10; a search engine 20 for searching the database 10 by using a predetermined search algorithm; a terminal apparatus 30 to be used by a user; and a data processing apparatus 40 for preparing a document index for document search. The data processing apparatus 40 divides the document data of one document file into a plurality of document blocks, evaluates the proximity of a unit sentence including a word of interest, the number of keywords included, etc. in each document block, specifies a document block most appropriate for extracting an excerpt sentence as an important document block, and extracts a sentence including a keyword from the important document block as an excerpt sentence. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an excerpt sentence extraction method for extracting a part of a sentence contained in a document file hit in the search process as an excerpt sentence and displaying it as a search result when a search process of a document file in a database is performed. .

  2. Description of the Related Art A search system that can search a document file desired by a user from a database on a worldwide Internet or a local network such as an in-house network is widely used. In such a search system, for example, when a list of hit documents when a keyword search is executed is displayed on the display screen of the terminal, usually, excerpts of each document are displayed together.

  In an Internet search using a portal site such as Yahoo (registered trademark) or Google (registered trademark), if there is an HTML summary tag described by the creator of the document file, the document related to the summary tag is used as an excerpt. Is displayed. On the other hand, if the HTML summary tag does not exist, or if it is impossible to specify the summary tag itself in a portal site in the company, a part of the document data in the document file is extracted according to a predetermined rule. Is displayed as an excerpt (see, for example, Patent Document 1). As this extraction rule, there is a method of identifying a line in which the search keyword appears first (or intermediate or final) in the document data, and extracting several lines around the specific line, or a search keyword. A method of simply extracting several lines from the top of the document data without specifying the lines is common.

JP 2008-176665 A

  The excerpt is a clue when the user who has performed the search process searches for a document that matches his search needs. However, in the mechanical excerpt sentence extraction method as described above, it is not highly likely that a description part closely related to the search keyword can be extracted as an excerpt sentence. For example, the description before and after the line in which the search keyword first appears is not necessarily the essence part of the document, and there are many cases where the main part exists in the part after the line. In this case, the user may miss this while hitting a document containing the content desired by the user.

  An object of the present invention is to provide an excerpt sentence extraction method capable of accurately extracting a main part of a document file corresponding to a search query as an excerpt sentence in a document search process for a database.

  The extracted sentence extraction method according to one aspect of the present invention that achieves the above object is to perform a search process of a document file in a database by giving a predetermined query, and to the document file that matches the query as a search result. A method of extracting the excerpt sentence when extracting and displaying a part of the contained sentence as an excerpt sentence, the step of determining one attention word from a word group included in the document data in the document file; The document data of one document file is treated as being divided into a plurality of unit sentences including a predetermined character group, and an area including a plurality of unit sentences including the attention word in a series of sentences, Defining a sentence block in the document data, and evaluating the appearance interval of the unit sentence in the series of sentences in each of the sentence blocks A step of identifying an important sentence block using the shortness of the appearance interval as an index, and a step of extracting the sentence including the attention word from the important sentence block as the excerpt sentence. (Claim 1).

  According to this configuration, for example, a word corresponding to a search keyword is determined as the attention word, a unit document including the attention word is specified, and an area in document data including a plurality of the unit documents is a single region. Identified as a document block. Then, in each document block, the appearance interval of the unit document including the word of interest is evaluated, and the document having a short appearance interval is identified as the important document block. This leads to specifying the description part where the attention word appears intensively in the document data, and it is presumed that such a description part has a detailed description related to the attention word, and as an excerpt sentence This is because there is a high possibility of having the appropriateness. Since the sentence including the attention word is extracted as an excerpt from such an important document block, the excerpt can be extracted with high accuracy.

  In the above-described configuration, the step of defining the sentence block includes a first unit sentence in which the attention word appears first in the document data, and the attention word is second in the plurality of unit sentences. A second unit sentence that appears and a third unit sentence in which the attention word appears third are specified, and a sentence region from the first unit sentence to the third unit sentence is defined as a first sentence block. , A fourth unit sentence in which the word of interest appears fourth in the document data is specified, and a sentence region from the second unit sentence to the fourth unit sentence is defined as a second sentence block. It is desirable (Claim 2).

  According to this configuration, a sentence block (first document block) is specified with three unit sentences as one set, and the next sentence block (second document block) is specified in a form in which the starting unit sentence is shifted by one. Will come to be. Since three unit sentences are included in one document block, it is easy to evaluate the appearance interval of unit sentences, and the document blocks are defined one after another while the starting unit sentences are shifted one by one. Processing can be simplified.

  In the above configuration, it is preferable that the unit sentence is one line of the sentence, and the appearance interval of the unit sentence is evaluated by a length between lines. According to this configuration, it is possible to easily divide a unit document using a tag or the like indicating a line spacing.

  Further, in the step of identifying the important sentence block, it is preferable that the appearance frequency of the attention word is further evaluated, and the important sentence block is identified by using the appearance frequency as an index. . According to this configuration, since the appearance frequency of the attention word is also evaluated as the requirement for specifying the important sentence block, it is possible to extract the extracted sentence with higher accuracy.

  In the above configuration, in the step of identifying the important sentence block, the appearance frequency of the noun word is further evaluated and the important sentence block is used as an index with the frequency of appearance of the noun word in the unit sentence including the attention word as an index. Is preferably specified (Claim 5). In general, it can be said that noun words are easier to convey information than words corresponding to adjectives, adverbs, verbs and the like. Therefore, the extraction accuracy of the excerpt sentence can be further improved by using the frequency of appearance of noun words as an index as the specific requirement for the important sentence block.

  According to the present invention, in a document search process for a database, a main part of a document file corresponding to a search query can be accurately extracted as an extracted sentence. Therefore, it is possible to contribute to the provision of a search system that allows the user to accurately obtain necessary information from the search hit list.

It is a block diagram which shows roughly the hardware constitutions of the network system to which the excerpt sentence extraction method concerning this invention is applied. It is a typical flowchart which shows the outline | summary of the search system which concerns on embodiment of this invention. It is a figure which shows an example of the display screen of a search result. It is a schematic diagram which shows the comparative example of an extract sentence extraction. It is a functional block diagram which shows the structure of a search system. It is a schematic diagram for demonstrating creation of a search index. It is explanatory drawing which shows the process of extraction sentence extraction. It is explanatory drawing which shows the process of extraction sentence extraction. It is explanatory drawing which shows the process of extraction sentence extraction. It is a figure of a table format which shows an example of a document index. It is a flowchart which shows a document search process. It is a flowchart which shows an extract sentence extraction process.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram schematically showing the hardware configuration of a network system S to which the excerpt sentence extraction method of the present invention is applied. The network system S mainly includes a database 10 that can be used on a communication network, a search engine 20 that performs a search process on the database 10 using a predetermined search algorithm, a terminal device 30 that a user uses, A data processing apparatus 40 that performs processing for creating a document index for document search is connected to be able to perform data communication via the Internet IN or the local network LN.

  The database 10 is a database in which a large number of document files to be searched are stored. When the database 10 is connected to the Internet IN, the database 10 is a set of a large number of websites each having a domain name. As such a database 10, for example, a database accessible through a portal site such as Yahoo (registered trademark) or Google (registered trademark) can be exemplified. When the database 10 is connected to a local net LN built inside a company or the like, the database 10 is a database that stores a shared document file in the company.

  The search engine 20 is equipped with a search algorithm having predetermined parameters, and performs a search process for a document file corresponding to a given query with respect to the database 10. That is, the search engine 20 analyzes a query to create a search index and extracts keywords and attributes (metadata) for each search target document file stored in the database 10 for search. Read the document index. Further, the search engine 20 performs a collation process between the document index and the search index using the search algorithm, and extracts a document file having a high degree of matching (matching a query).

  The terminal device 30 is a communication terminal 31, 32, 33, 34... Such as a personal computer, a mobile phone, or a personal digital assistant owned by many users. The terminal device 30 can access the search engine 20 and the database 10 via the Internet IN or the local net LN. For example, the terminal 31 has a keyboard 311 for inputting keywords and sentences (queries) as search processing conditions, a display 312 for displaying a search screen, a list of hit documents, document contents, and the like, and a display on the display 312. A mouse 313 for inputting a command (such as a document selection instruction or a selection end instruction) to the screen is provided.

  Each user gives a query to the search engine 20 via the keyboard 311 of the respective communication terminal 31, 32, 33, 34..., And obtains a list of document files hit by the search processing using the query. . Further, the user can select a desired document file from the list and display the contents of the document file on his / her display 312.

  The data processing device 40 performs document analysis processing for each search target document file stored in the database 10, extracts keywords and attributes, and creates a search document index. Further, in the search processing of the present embodiment, a part of the sentence included in the document file that matches the query is extracted as an excerpt, and this excerpt is also displayed on the display 312. A process of extracting the part in association with the word (word of interest) in the document index is performed. A specific configuration of the data processing device 40 will be described in detail later.

  FIG. 2 is a schematic flowchart showing an outline of the document file search process executed by the network system S. When a query is given from the terminal device 30 as processing on the search source side, the search engine 20 performs document analysis processing including, for example, morphological analysis on the query document (step S1). Next, a search index is created based on the words obtained by the document analysis process (step S2). This search index mainly consists of a group of keywords that are closely related to the question event. Then, using the search index, a document search process is performed on the database 10 including a large amount of data (step S3).

  As processing on the search destination side, the data processing device 40 periodically performs document analysis for extracting metadata (step S01) and creation of a document index (step S02). The results obtained in the search process in step S3 are output as a list (ranking display) after ranking according to the degree of match in the search algorithm to the terminal device 30 of the user who has input the query (step S4). .

  FIG. 3 is a diagram showing an example of a search result display screen displayed on the display 312 in step S4. Here, a case where an Internet search is performed with the keyword “Kansai Electric Power” and an HTML summary tag described by the creator of the document file exists for the hit document file is illustrated. As shown in FIG. 3, an excerpt sentence (summary sentence) indicating an outline of the content of the document is displayed in the lower column of the hit document title. This excerpt is a display of the above HTML summary tag.

  On the other hand, if there is no HTML summary tag as described above in a document file on the Internet, or if it is impossible to specify a summary tag on a portal site in the company, document data in the document file according to a predetermined rule Is extracted and displayed as an excerpt. The user can know the outline of the contents of the hit document file by referring to the excerpt, and can save time and effort to open each document file and check the contents. Therefore, it is important that the content of the excerpt is in line with the query. If the extracted sentence is inaccurate, the user may pass through the document file even though the document file includes a description desired by the user.

  FIG. 4 is a diagram schematically showing a conventional general extracted sentence extraction rule for a document file having no summary tag. In this example, the search keyword “A” is given from the terminal device 30 to the search engine 20 and a hit document including the keyword “A” is extracted as a result of the search process. This extraction rule specifies the line L1 in which the search keyword “A” first appears in the document data of the hit document, and extracts several lines around the specific line L1 as extracted sentences D1.

  However, the main part of the keyword “A” in the hit document is not always described near the line L1 in which the keyword “A” first appears. FIG. 4 shows an example in which the keyword “A” appears frequently in the area indicated by the symbol D2, and there is a portion where a detailed description is made regarding the keyword “A”. Originally, it is highly possible that a description part of the area D2 is suitable as an excerpt sentence, but the above extraction rule has a problem that such an area D2 cannot be extracted as an excerpt sentence.

Therefore, in the present embodiment, a sentence suitable as an excerpt sentence is defined as shown in the following (1) and (2), and processing is performed so that such a sentence is excised.
(1) There is continuity between sentences (topics).
(2) Much talk about the keyword.
As an element for mechanically determining the point (1), description sentences including keywords are close to each other. Moreover, about the point of said (2), it is mentioned that many the keywords are included.

  In order to extract an excerpt that matches such a definition, in the present embodiment, the document data of one document file is roughly divided into a plurality of document blocks, and the above-described keyword (word of interest) is included in each document block. ) And the number of keywords included. Then, a document block that most closely matches the above definition is identified as an important document block, and a sentence including a keyword is extracted as an excerpt from the important document block. In the present embodiment, such processing is executed by the data processing device 40, and detailed processing thereof will be performed in the description of the functional configuration of the network system S described below.

  Hereinafter, the extracted sentence extraction method according to the present embodiment will be described in detail with reference to FIGS. FIG. 5 is a functional block diagram showing a functional configuration of the network system S according to the present embodiment. The database 10 includes a plurality of document servers 11, 12, 13... (Web server or in-house local server) in which various document files are stored. The search engine 20 functionally includes a document index storage unit 21, a search index creation unit 22, a search processing unit 23, and a ranking display processing unit 24. The terminal device 30 includes a query input unit 301, a display unit 302, and an operation unit 303. The data processing device 40 includes a document extraction unit 41, a document analysis unit 42, a weight calculation unit 43, a storage unit 44, and an excerpt sentence extraction unit 45.

  The document index storage unit 21 stores an index (document index) for searching for a document file included in the database 10. The document index is created by the data processing device 40 at a predetermined creation timing (for example, once a day), and the stored content is updated.

  The search index creation unit 22 performs document analysis on a query (keyword or question document) given from the terminal device 30 and creates a search index. For example, when the query is a question document, a search index is created by extracting the words as independent words in the document and weighting them by referring to the appearance frequency of these words. FIG. 6 is a schematic diagram illustrating an example of a search index table. Here, the independent words A, B, C... Are extracted from the query, and weight values (Si) derived by a predetermined calculation formula are assigned to the words A, B, C. An example is shown.

  The search processing unit 23 performs a document file search process on the database 10 using a search algorithm. Specifically, the search index and the document index are collated, and a document file having a high similarity to the search index (query) is extracted. Here, a preset search algorithm is used for this search process. This search algorithm is selected from, for example, a vector space model (cosine scale), a Dicks coefficient, a Jaccard coefficient, a T score, a mutual information amount, and a Simpson coefficient. Each of these search algorithms has a calculation formula, and various parameters are set in the calculation formula.

  The ranking display processing unit 24 creates a list in which a plurality of document files hit in the search processing by the search processing unit 23 are ranked in descending order of similarity (matching degree) to the query. This list can be browsed by the terminal device 30 and is actually displayed on the display unit 302 after the search process is completed. At the time of this display, the above-mentioned excerpt sentence is displayed along with the title of each document file.

  The query input unit 301 of the terminal device 30 is a part that receives an input of a query such as a keyword for search processing from a user, and is, for example, the keyboard 311 shown in FIG.

  The display unit 302 is, for example, the display 312 shown in FIG. 1, and includes a browsing screen (query input screen) of the search engine 20, a list of hit document files and excerpts thereof, contents of document files selected from the list, and the like. Is displayed.

  The operation unit 303 is, for example, the mouse 313 illustrated in FIG. 1, and gives an operation instruction to the selection unit, the link unit, the task bar, and the like on the image displayed on the display unit 302. The user gives a selection instruction to select one of the hit document file lists displayed on the display unit 302 to display the contents and a selection end instruction to stop the display via the operation unit 303.

  The data processing device 40 creates a document index for searching for document files included in the database 10. The document extraction unit 41 of the data processing device 40 performs a process of extracting a plurality of document files from the document database 10 at predetermined intervals as a trigger for the document index creation process.

  The document analysis unit 42 performs document analysis processing such as normalization processing, document structure analysis processing, synonym processing on the document data included in each document file extracted by the document extraction unit 41, and converts the document data into word units. Divide into The normalization process is a process for deleting characters and symbols unnecessary for analysis from the analysis target document and unifying full-width and half-width characters so that the document structure analysis can be performed normally. In the document structure analysis process, the classification target documents after the normalization process are each subjected to a morphological analysis to divide the document into words, and a parsing process to perform dependency between words (nouns and verbs). And the like. The synonym process is a process in which terms having different expressions are treated as a single word by utilizing a thesaurus (synonym dictionary) that absorbs fluctuations of synonyms and notations. By such document analysis processing, independent words (keywords) are derived. A basic index (an index to which weight is not yet given) as a basis for the document index is created with the words derived here.

  The weight calculation unit 43 assigns weights to the words listed in the basic index under predetermined conditions to complete the document index. What is widely used in this weighting setting process is the appearance frequency of words, and a high weight is set for words that frequently appear in the document data.

  The storage unit 44 temporarily stores the basic index created by the document analysis unit 42 and the weight data calculated by the weight calculation unit 43, and temporarily stores various work data in the excerpt sentence extraction unit 45 described later. It is a memory used as a work area for storing. Weight calculation formulas, line spacing calculation formulas, and the like are also stored in the storage unit 44.

  The excerpt sentence extraction unit 45 performs a process of extracting an excerpt sentence including the word from the document data for each word extracted as the document index. This excerpt is a sentence that is displayed as the gist of a document containing the word when the word is used as a search keyword. Usually, a plurality of document index words are extracted from the document data of one document file, and an excerpt sentence is extracted in association with each of these words. The excerpt sentence extraction unit 45 functionally includes an attention word setting unit 451, a block division unit 452, an interline distance calculation unit 453, a block selection unit 454, and an excerpt line selection unit 455.

  The attention word setting unit 451 sequentially extracts one word from the word group of the basic index extracted by the document analysis unit 42 for each document file. Specifically, the attention word setting unit 451 performs processing for numbering the document file to be analyzed and numbering the word of each document file, and sequentially setting the extracted word as a word for extraction.

  The block division unit 452 treats the document data of one document file as being divided into a plurality of unit sentences including a predetermined character group, and includes a plurality of unit sentences including the attention word in a series of sentences. The area to be defined is defined as a sentence block in the document data. Here, it is one line of a sentence that can be preferably handled as a unit sentence. In this case, division of unit sentences can be easily performed using a tag or the like indicating a line spacing. Alternatively, a unit sentence may be determined in a predetermined number of characters (for example, 50 characters) or in units of bits. In addition, a single sentence block may include a plurality of unit sentences including the word of interest, but a preferable number of unit sentences is three. The block dividing unit 452 sets a plurality of such text blocks in the document data of one document file.

  A preferable specific example of the processing of the block dividing unit 452 will be described with reference to FIGS. As shown in FIG. 7, it is assumed that the word “A” is set as the attention word in the attention word setting unit 451 in the document data X of one document file. In a series of sentences included in the document data X, the word A is in the second line, the eighth line, the ninth line, and the 14th line among the first line to the 16th line (a plurality of unit sentences). It is assumed that it has appeared.

  In this case, as shown in FIG. 8, the block dividing unit 452 firstly displays the first unit sentence (here, the sentence on the second line) in which the word A appears first, followed by the second one. A second unit sentence (sentence on the 8th line) and a third unit sentence appearing on the 3rd (sentence on the 9th line), and sentences from the first unit sentence to the third unit sentence The area is defined as the first sentence block B1. That is, the first sentence block is specified with a unit sentence including three words A as one set.

  Next, the block division unit 452 starts from the second unit sentence (sentence on the eighth line) in which the word A appears second, and then starts the third unit sentence (9 lines) that appears third. Eye sentence) and the fourth unit sentence appearing in the fourth (sentence on the 11th line), and the sentence area from the second unit sentence to the fourth unit sentence is defined as a second sentence block B2. Define. That is, the second sentence block is specified by shifting the starting unit sentence one line downward. Similarly, the block dividing unit 452 starts from the third unit sentence (sentence on the ninth line) in which the word A appears third, and then starts the fourth unit sentence (11 that appears fourth). The fifth sentence appearing in the fifth line (sentence in the fourteenth line), and the sentence area from the third unit sentence to the fifth unit sentence is defined as the third sentence block B3. Define as Thereafter, as long as the document data X continues, a similar sentence block setting process is performed. Thus, a plurality of sentence blocks including three unit sentences including the word A in different areas are defined for the document data X. According to this method, document blocks are defined one after another while the starting unit sentences are shifted one by one, so that the processing of the block dividing unit 452 can be simplified.

  The line-to-line distance calculation unit 453 shortens the distance between lines (shortness of the appearance interval of unit sentences) based on the line-to-line distance Sn between unit sentences in which the word A appears in each sentence block B1, B2, B3. The index value for evaluating is derived. As this index value, the reciprocal of the inter-line distance square and the reciprocal of the vector size can be exemplified.

  FIG. 9A shows the first sentence block B1 illustrated in FIG. In the case of the first sentence block B1, the line-to-line distance Sn is the line-to-line distance S1 between the first unit sentence (the second line sentence) and the second unit sentence (the eighth line sentence), and the second unit. There is an inter-line distance S2 between the sentence and the third unit sentence (the sentence on the ninth line). Here, S1 = 6 (row) and S2 = 1 (row). By using these numerical values of S1 and S2, the inter-line distance sum reciprocal (F) can be obtained by the following equation (1). It can be evaluated that the larger the value of the equation (1), the shorter the total distance between rows.

The method using the reciprocal vector size (J) is a method in which the line spacing Sn is regarded as a vector element of each sentence block. In the example of FIG. 9A, the first sentence block B1 is represented by a two-dimensional vector of S1 and S2, as shown in FIG. 9B. Since S1 = 6 (row) and S2 = 1 (row), the vector b = (6, 1). The reciprocal (J) of this vector size is
J = 1 / vector b = 1 / √ (6 ² +1 ² ) = 0.164
However, it can be evaluated that the larger the value, the shorter the total distance between the rows. For example, in the case of the second sentence block B2, since S1 = 1 (line) and S2 = 2 (line), the reciprocal (J) of the vector size is
J = 1 / vector b = 1 / √ (1 ² +2 ² ) = 0.4472
It is evaluated that the total distance between lines is shorter in the second sentence block B2 than in the first sentence block B1. This vector size reciprocal (J) can be generally obtained by the following equation (2).

  The above is an index for evaluating the overall shortness between lines between unit sentences in which the word A appears. In addition to this, the interline distance calculation unit 453 also derives an index value for eliminating a sentence block including a sentence having an extremely long interline distance Sn. As this index value, the reciprocal (G) of the standard deviation between lines focusing on the standard deviation of the line distance Sn can be exemplified. This inverse line standard deviation (G) can be obtained by the following equation (3). For example, when the variation is large such as S1 = 6 (line) and S2 = 1 (line) as in the first sentence block B1, the standard deviation becomes large, so the value of the reciprocal number of interline standard deviation (G) taking the reciprocal number thereof. Becomes smaller.

  The block selection unit 454 refers to an additional determination index value in addition to the above-described vector size reciprocal (J) or the inter-line distance sum reciprocal (F) and the inter-line standard deviation reciprocal (G), and adds a plurality of sentences. A process of selecting an important sentence block including a sentence suitable as the most extracted sentence from the blocks is performed. Examples of the additional determination index values include an attention word density (H) indicating the appearance frequency of the attention word in the sentence block and a noun density (K) indicating the appearance frequency of the noun word in the sentence block. can do.

The attention word density (H) is the total number of words in one sentence block (the number of words in the document block extracted by the document analysis unit 42) H1, and the number of attention words appearing in one sentence block. When H2
H = H2 / H1 (4)
Can be obtained. The attention word density (H) increases as the appearance frequency of the attention word increases. In general, it can be said that the higher the appearance frequency of the attention word, the more detailed description is made about the attention word, and the higher the suitability as an excerpt. For example, when document blocks having the same vector size reciprocal number (J) are present side by side, it is important that the value of this attention word density (H) is larger (one where a plurality of attention words appear in one line). It will be determined as a sentence block.

The noun density (K) is defined as K1 as the total number of words in one sentence block and K2 as the number of noun words appearing in one sentence block.
K = K2 / K1 (5)
Can be obtained. In general, it can be said that noun words are easier to convey information than words corresponding to adjectives, adverbs, verbs and the like. In other words, it can be evaluated that a sentence block including a sentence in which a large number of noun words appear has a large amount of information and is highly suitable as an excerpt sentence. Therefore, the frequency of appearance of noun words can be an index as a specific requirement for important sentence blocks.

The block selection unit 454 calculates the block importance L by the following equation (6) that multiplies the above equations (1), (3), (4), and (5).
L = Linear distance reciprocal power (F) × Linear standard deviation reciprocal number (G) × Target word density (H) × Noun density (K) (6)
Alternatively, the block importance L is calculated by the following equation (7) that multiplies the equations (2), (3), (4), and (5).
Vector size reciprocal (J) x interline standard deviation reciprocal (G) x attention word density (H) x noun density (K) (7)
Further, after calculating the block importance L for each document block, the block selection unit 454 identifies the document block having the highest value of L as the important document block.

  The excerpt line selection unit 455 extracts an appropriate sentence including the attention word from the sentences of the important document block specified by the block selection part 454 as an excerpt sentence to be displayed when the attention word is used as a search keyword. Perform the process. This extraction may be, for example, the entire sentence of the important document block as a result, or only the line including the attention word in the sentence of the important document block may be extracted. The extracted excerpt is stored in the storage unit 44 in association with the document index word (attention word) of the document file, and finally incorporated in the document index table and stored in the document index storage unit 21 of the search engine 20. Is done.

  FIG. 10 is a table format showing an example of the document index table. A word in the search index is extracted for each document number representing a document file (here, word A and word B are illustrated), and a weight (Di) is assigned to each word. Further, for the word A, the excerpt sentence is associated with the line number to which the sentence belongs. The description is omitted, but the same applies to other words and other document files. Therefore, for example, when the word A is given as a search keyword and the document with the document number 1002 shown in FIG. 10 is hit, the documents with 0002 lines, 0008 lines, and 0009 lines in the document data are extracted sentences with the document number 1002. Will be displayed.

  Depending on the document file, there may be a case where the extracted sentence cannot be extracted by the above procedure. For example, there is a case where only one or two unit sentences including the word of interest appear in one document data. If there is only one unit sentence, the extracted line selection unit 455 extracts, for example, the unit sentence and the preceding and subsequent unit sentences as an extracted sentence. In addition, when there are only two unit sentences, only S1 can be extracted as the line spacing Sn. In this case, for example, S2 is substituted with a predetermined fixed value so that the above calculation can be applied. It is desirable to make it.

  Next, a search operation in the network system S according to the present embodiment will be described based on the flowchart shown in FIG. When a query for document file search is input from the terminal device 30 (step S1), the query is received by the search engine 20 via the Internet IN or the local net LN (step S2). Then, the search index creation unit analyzes the given query to extract words and obtains a weight value (Si) by a predetermined arithmetic expression to create a search index as illustrated in FIG. 6 ( Step S3).

  Subsequently, the search algorithm provided in the search processing unit 23 uses the search index created in step S3 and the document index stored in advance in the document index storage unit 21 to query each document file. The degree of match is calculated (step S4). Then, a document file having a high degree of match is specified (step S5). Further, the search processing unit 23 specifies the most relevant word that is the word most relevant to the query in the document index (step S6). This most relevant word is multiplied by the weight value (Si) of each word in the search index and the weight value (Di) (see FIG. 10) of each word in the document index, and the multiplication value Si × Di is the highest value. Is selected.

  Then, the excerpt sentence stored in association with the most relevant word is extracted (step S7). For example, if the examples of FIGS. 6 and 10 are used, if the multiplication value Si × Di for the word A is the highest value in the document file with the document number 1002, the excerpt for the word A is used as the excerpt sentence for the document file. A sentence is chosen.

  Thereafter, the ranking display processing unit 24 creates a list in which a plurality of document files hit in the search process are ranked in descending order of matching degree with the query (step S8). This list includes the title of the document file and the excerpt sentence extracted in step S7. Then, the search result list is transmitted from the search engine 20 to the terminal device 30 to which the query is input (step S9).

  Next, the excerpt sentence extraction process by the data processing device 40 will be described based on the flowchart shown in FIG. The excerpt sentence extraction unit 45 sets the counter p corresponding to the numbering assigned to each document file group to be processed to “0” (step S11), and advances the counter to p = p + 1 (step S11). S12). Then, the p-th document file (the document file to which the first number is assigned for the first time) is extracted (step S13).

  Subsequently, the attention word setting unit 451 sets the attention word. A counter q corresponding to the numbering assigned to each word group for document index extracted for the p-th document file is set to “0” (step S14), and the counter is set to q = q + 1. (Step S15). Thereby, the q-th word is extracted as the attention word, and the process of dividing the document data into a plurality of sentence blocks is executed by the block dividing unit 452 for the attention word (see step S16 / FIG. 7, FIG. 8). .

  Next, after the counter i corresponding to the numbering assigned to each of the sentence blocks is set to “0” (step S17), the counter is advanced to i = i + 1 (step S18). Then, for the i-th sentence block, the reciprocal of the inter-line distance sum or the reciprocal of the vector size and the reciprocal of the standard deviation between lines are calculated by the inter-line distance calculation unit 453, and the attention word density and the noun density are calculated by the block selection unit 454. Then, the importance level L of the i-th sentence block is obtained by the above formula (6) or formula (7) (step S19). The value of importance L is temporarily stored in the storage unit 44 (step S20).

  Thereafter, it is confirmed whether or not the sentence block for which importance L is obtained is final (step S21). If it is not final (NO in step S21), the process returns to step S18 to obtain importance L for the next sentence block. The process is repeated. On the other hand, if it is not final (YES in step S21), the block selection unit 454 selects the important sentence block having the highest importance L, and the excerpt line selection unit 455 identifies the excerpt sentence (line) from the important sentence block. (Step S22). This excerpt is stored in the document index storage unit 21.

  Subsequently, it is confirmed whether or not the word group counter q is final (step S23). If it is not final (NO in step S23), the process returns to step S15 to divide the document block for the next attention word. And the process for obtaining the importance L of the sentence block is repeated. On the other hand, if the counter q is final (YES in step S23), it is confirmed whether or not the document file counter p is final (step S24). If the counter p is not final (NO in step S24), the process returns to step S12 and the same processing is repeated for the next document file. On the other hand, if the counter p is final (YES in step S24), the process ends.

  According to the network system S according to the present embodiment described above, the main part of the document file corresponding to the search query can be accurately extracted as an excerpt in the document file search process for the database 10. Therefore, it is possible to provide a search system that allows the user to accurately obtain necessary information from the search hit list.

S Network system S
DESCRIPTION OF SYMBOLS 10 Database 20 Search engine 21 Document index memory | storage part 22 Search index creation part 23 Search processing part 24 Ranking display processing part 30 Terminal device 40 Data processing device 45 Extract sentence extraction part 451 Attention word setting part 452 Block division part 453 Interline distance calculation part 454 Block selection part 455 Extraction line selection part

Claims (5)

The excerpt in the case where a search process of a document file in a database is performed by giving a predetermined query, and a part of a sentence included in the document file matching the query is extracted and displayed as a search result A method for extracting sentences,
Determining one word of interest from a group of words included in the document data in the document file;
The document data of one document file is treated as being divided into a plurality of unit sentences including a predetermined character group, and an area including a plurality of unit sentences including the attention word in a series of sentences is defined as the document. Defining as sentence blocks in the data;
In each of the sentence blocks, evaluating an appearance interval of the unit sentence in the series of sentences, and identifying an important sentence block using the shortness of the appearance interval as an index;
Extracting a sentence including the attention word from the important sentence block as the excerpt sentence;
A method for extracting excerpts characterized by including: The step of defining as the sentence block includes:
Among the plurality of unit sentences, a first unit sentence in which the attention word appears first in the document data, a second unit sentence in which the attention word appears second, and the attention word is the first A third unit sentence that appears third is specified, and a sentence area from the first unit sentence to the third unit sentence is defined as a first sentence block;
A fourth unit sentence in which the attention word appears fourth in the document data is specified, and a sentence region from the second unit sentence to the fourth unit sentence is defined as a second sentence block. The excerpt sentence extraction method according to claim 1, wherein: The unit sentence is one line of sentences,
3. The excerpt sentence extraction method according to claim 1, wherein an appearance interval of the unit sentence is evaluated by a length between lines. In the step of identifying the important sentence block,
The excerpt sentence extraction method according to any one of claims 1 to 3, wherein the appearance frequency of the attention word is set as an evaluation target, and the important sentence block is specified by using the appearance frequency as an index. . In the step of identifying the important sentence block,
5. The important sentence block is identified by using the appearance frequency of the noun word as an evaluation target and specifying the frequency of appearance of the noun word in the unit sentence including the attention word as an index. Excerpt sentence extraction method.

Images