JP2007213157A - Example sentence retrieval device and example sentence retrieval method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that the retrieval of an example sentence which can be used for document preparation is not efficient. <P>SOLUTION: A pattern extraction part 40 is provided with a text integration part 42 for reading a document set designated by a user among document sets stored in a document storage part 70, and integrates the document set into text data; a conversion part 44 for performing conversion for segmenting the text data for every sentence; an extraction execution part 46 for extracting a word column pattern by the algorithm of system pattern mining; and a pattern information writing part 48 for making a pattern information storage part 80 store the extracted pattern by associating it with its appearance frequency and a text including it. A retrieval part 60 is provided with a retrieval execution part 62 for detecting a pattern from a keyword input by the user; a pattern output part 64 for displaying the list and frequency of the detected patterns; and an example sentence output part 66 for displaying an example sentence including the pattern selected by the user. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to search technology, and more particularly to an example sentence search apparatus for referring to phrase examples and an example sentence search method applied thereto.

  Referencing the example sentence of a phrase when creating a sentence such as a foreign language is an effective means for creating a correct document. In recent years, it has become common to save documents as electronic files, and hardware performance such as the processing speed of information processing devices and the storage capacity of electronic data has improved. Techniques for retrieving example sentences from a large number of documents obtained based on keywords have been studied. For example, as a search tool based on kwic (Keyword In Context) indexing, software that creates a word list and kwic concordance indicating the context in which the word was used from an input phrase is also provided.

On the other hand, an example sentence search can be performed using a search engine provided by a website. In this case, the user inputs a keyword into a search engine, browses a character string around the keyword displayed as a search result, and confirms an example of the keyword. In addition, a system is also provided that performs matching from a character string pattern around a keyword, extracts and displays a partial character string corresponding to a frequent phrase, and searches a document including the partial character string as an example sentence (for example, non- Patent Document 1).
Hiromoto Fujimoto et al., Text Mining Tool from Local Corpus: PortableKiwi, Proc. Of the 11th Annual Conference of the Language Processing Society of Japan

  However, when a user tries to check a frequent phrase, for example, when kwic concordance is used, it is necessary to check the phrase by himself from a large number of concordances created from keywords included in the phrase. In addition, partial information of the example can be acquired from information around the keyword, but it may be difficult to grasp the entire sentence unless the display of concordance is in sentence units. In a system that uses a search engine to extract frequent phrases, the search target is a web page, so a search specialized for the field cannot be performed, and the display result becomes enormous, resulting in poor search efficiency. When a phrase matching system is used, a simple keyword search only considers the context behind the keyword, so that necessary information may not be acquired.

  This invention is made | formed in view of such a condition, The objective is to provide the technique in which a user can acquire the useful information which concerns on an example sentence efficiently.

  One embodiment of the present invention relates to an example sentence search device. This example sentence search device includes a pattern extraction unit that extracts a word string pattern from a document set designated by a user according to a predetermined rule, a word string pattern extracted by the pattern extraction unit, and a document that belongs to the document set. A pattern information storage unit that stores pattern information that associates a sentence including a word string pattern, and a pattern information that receives a search keyword input and includes a word string pattern that includes any one of the search keyword and a phrase related to the search keyword. A search execution unit that is detected from pattern information stored in the storage unit, and a pattern information output unit that outputs at least a part of a word string pattern detected by the search execution unit and a sentence associated therewith, To do.

  Here, the “word string pattern” is a word set including a predetermined number of words and including order information. Two word sets that are the same in succession may be the same word string pattern, or two word sets that are discontinuous but appear in the same order may be the same word string pattern. Therefore, the “predetermined rule” refers to the number of words constituting the word string pattern to be extracted or the range of the numbers, whether the same word string pattern is allowed to be consecutively identical or discontinuous, or the frequency that appears in the document set Any value may be used as long as it is a condition related to extraction, such as whether it is extracted as a word string pattern when it appears many times, and a technique used for extraction may also be included.

  “Phrase having relevance to search keyword” can generally be associated with a search keyword such as a word or phrase in a different language having the same meaning as the search keyword, a synonym of the search keyword, or a combination thereof. Any of the phrases can be used.

  Another aspect of the present invention relates to an example sentence search method. This example sentence search method includes a step of receiving a search keyword input, a word string pattern that is stored in advance and extracted from a user-specified document set according to a predetermined rule, and a document belonging to the document set. A step of detecting a word string pattern including any one of a search keyword and a phrase related to the search keyword with reference to pattern information in which a sentence including the pattern is associated; and at least one of the detected word string patterns Outputting a sentence associated with the word string pattern of the part.

  Yet another embodiment of the present invention relates to a recording medium. The recording medium includes a word string pattern extracted from a document set according to a predetermined rule, an appearance frequency of the word string pattern in the document set, and a sentence included in the document belonging to the document set and including the word string pattern. It is characterized by recording in association with each other.

  It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between a method, an apparatus, a system, etc. are also effective as an aspect of the present invention.

  According to the present invention, the user can easily confirm information such as examples of desired phrases.

  FIG. 1 shows the overall configuration of an example sentence search apparatus according to the present embodiment. The example sentence search apparatus 10 includes a processor 15 that performs overall control of the example sentence search apparatus 10 and extracts a word string pattern (hereinafter also simply referred to as a pattern) from the stored document and performs a search process. The example sentence search device 10 further includes an input device 20 for a user to input instructions, a document storage unit 70 for storing and storing document data, a pattern information storage unit 80 for storing extracted pattern information, and a search result output. Output device 30. The processor 15, the input device 20, the output device 30, the document storage unit 70, and the pattern information storage unit 80 mutually transmit data via the bus 90.

  The example sentence search device 10 may also serve as a document creation device or an information processing device. In this case, the processor 15 may further execute application software that provides a document creation function and an e-mail creation function, and the input device 20 and the output device 30 process input data and output data adapted to the application software. Do each.

  The input device 20 may be any one of commonly used input devices such as a keyboard, a mouse, a trackball, or a combination thereof, such as an instruction to extract a pattern from a document set stored in the document storage unit 70, a keyword to be searched, etc. This is an interface for the user to input. The document storage unit 70 and the pattern information storage unit 80 are a hard disk, a reading device for a recording medium such as a DVD (Digital Versatile Disk) or a CD (Compact Disk), a DRAM (Dynamic Random Access Memory), an SRAM (Static Random Access Memory). Such as a memory, etc., is appropriately selected from hardware according to the amount of data and the form of the search device.

  The document storage unit 70 may be provided in a server connected to the example sentence search device 10 via a network. The document storage unit 70 stores a plurality of completed document data belonging to a category that the user wants to refer to, such as a paper in a specific field or an e-mail. For example, a user who creates an English paper accumulates English paper data in the same field that he has browsed in the past, or obtains and stores electronic data of papers previously published in a journal in the same field. You may do it. In the present embodiment, a pattern belonging to the same category, such as a paper or abstract in the same field, or a document of the same type, is extracted as a document set, and is searched for example sentences. You can efficiently search for phrases, phrase usage, and fixed phrases.

  The document storage unit 70 may store data of a plurality of document sets for each category. In this case, the user selects one document set in accordance with the category of the document he / she wants to create and performs pattern extraction described later. Prior to pattern extraction, document data is converted to text data, so the document data stored in the document storage unit 70 has a format that can be used for the conversion process.

  FIG. 2 shows the configuration of the processor 15 in more detail. The processor 15 extracts a pattern included in the document data stored in the document storage unit 70 or the document data of the document set selected by the user, and generates a pattern information file, and the user inputs A search unit 60 for searching for patterns including example keywords and example sentences is included.

  The pattern extraction unit 40 reads out the document data stored in the document storage unit 70 in accordance with a user input instruction in the input device 20, converts the text data into text data, and converts the text data into one text per line. Conversion unit 44 for converting to a file, extraction execution unit 46 for extracting a pattern that frequently appears from a text file of one sentence per line, a pattern that associates the extracted pattern with its frequency, and a sentence including the pattern A pattern information writing unit 48 for storing information in the pattern information storage unit 80 as a pattern information file is included.

  The search unit 60 is a search execution unit 62 that detects a pattern including a search keyword from data of a pattern information file stored in the pattern information storage unit 80 in accordance with a user's search keyword input in the input device 20. A pattern output unit 64 that outputs the list and the respective frequencies to the output device 30 and an example sentence output that outputs a sentence including the selected pattern as an example sentence to the output device 30 in accordance with a user's pattern selection instruction in the input device 20 Part 66 is included.

  In FIG. 2, each element described as a functional block for performing various processes can be configured by a CPU, a memory, and other LSIs in terms of hardware, and a program having a language processing function in terms of software. Etc. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

Here, the word string pattern extraction from the text data performed by the extraction execution unit 46 will be described. Assume that the following English data exists.
(1) Three types of thick branes e de Sitter and Sitter brane are considered .
(2) The cases of Dirac Proca and Maxwell fields are considered .
(3) Some issues related to quantum anomaly induced effects due to matter are considered .
(4) The example of a five dimensional BF theory with a boundary brane is considered .

  In these data, the continuous word string “are considered.” Appears at frequency 3 and the discontinuous word string “brane to considered.” Appears at frequency 2. Therefore, these are considered to be regular patterns that are frequently used in the category of documents including English sentences. The extraction execution unit 46 extracts such a fixed pattern from the document data.

  As described above, a technique for extracting an item string that frequently appears at a frequency equal to or higher than a certain threshold value from a set of continuous item strings (words, parts of speech, attributes, etc.) is called sequence pattern mining. For example, it is assumed that there is a set of item strings “ACD”, “ABC”, “CBA”, and “AAB”. According to the sequence pattern mining algorithm, information that the frequency of the pattern “A * B” is 2 and the frequency of the pattern “A * C” is 2 can be obtained from the item string set. Here, the item string included in the pattern can be set to either continuous or discontinuous. In the above example, the discontinuous pattern is also included. Many studies have been made on sequence pattern mining, and in this embodiment, any of these sequence pattern mining techniques can be introduced. Thereby, a frequent pattern can be extracted in a realistic processing time.

  For example, an n-gram PrefixSpan algorithm may be introduced as a method of sequence pattern mining (Taku Kudo et al., Text Mining Using Language Information, National Association of Language Processing Associations NLP-2002, 2002). This method uses natural language processing tools such as chunking and dependency analysis to extract patterns reflecting meaning from semi-structured data.

  Next, the operation of the present embodiment having the above configuration will be described. FIG. 3 shows a procedure for extracting a pattern and related information from the document data stored in the pattern document storage unit 70 to generate a pattern information file and storing it in the pattern information storage unit 80. First, in response to an input instruction from the user to the input device 20, the text conversion unit 42 reads a plurality of documents stored in the document storage unit 70, and converts the stored format into text data (S10). When document sets of a plurality of categories are stored in the document storage unit 70, conversion processing is performed on the document set designated by the user. Next, the conversion unit 44 converts the text data into one text file with one sentence per line (S12). The generated text file is stored in the pattern information storage unit 80.

  Next, the extraction execution unit 46 extracts a frequent pattern from the converted text file by a sequence pattern mining algorithm such as n-gram PrefixSpan (S14). Here, an appearance frequency threshold for setting a certain pattern as a “frequent pattern” is set in advance in the program, such as twice or five times. And the pattern extracted more frequently is stored as a “frequent pattern”. Similarly, the length of the word string extracted as a pattern is also set in the program as 2 words or more, 4 words or more. Alternatively, the length of the word string extracted as a pattern and the threshold value of the frequency may be changed in combination, such as reducing the threshold value of the frequency of the long word string pattern. Such setting can also be performed in the program.

  The frequent pattern extracted by the extraction execution unit 46 is stored in the pattern information storage unit 80 by the pattern information writing unit 48 (S16). At this time, the frequent pattern, the frequency in the extracted document set, the identification information of the sentence including the frequent pattern, and the like are associated with each other and written in the pattern information storage unit 80. The identification information may be, for example, a pointer indicating the storage area of the corresponding sentence in the text file generated in S12 and stored in the pattern information storage unit 80. The pattern information storage unit 80 may store a pattern information file for each document set to be extracted in one pattern extraction. With the above procedure, a database storing information such as frequent patterns included in a document set that the user wants to refer to and sentences including the patterns is completed. The pattern information file generated in the past and stored in the pattern information storage unit 80 can be used as it is in the example sentence search for the same document set.

  FIG. 4 shows a processing procedure when the user performs an example sentence search when creating a document. First, the user inputs a keyword to be searched using the input device 20 (S20). In this case, if there are a plurality of pattern information file in the pattern information storing unit 80 specifies whether to search in advance from which pattern information file. Then, the search execution unit 62 identifies the designated pattern information file from the pattern information storage unit 80, and detects a frequent pattern including the keyword (S22). Next, the pattern output unit 64 outputs data including the detected pattern and its frequency to the output device 30 (S24).

  The user selects a pattern for which an example sentence is to be confirmed from the patterns output as necessary by the input device 20 (S26). Then, the example sentence output unit 66 refers to the pattern information file in the pattern information storage unit 80, and reads a sentence including the pattern from the text file in the pattern information storage unit 80 based on the identification information associated with the selected pattern. All are read out and output to the output device 30 as an example sentence (S26). As a result, the user can know patterns that frequently appear in documents belonging to the category that the user wants to refer to, and can also check example sentences for each pattern.

  FIG. 5 shows an example of an example sentence search screen displayed on a display device corresponding to the output device 30 in the present embodiment. The example sentence search screen 100 includes a file selection command 102 for selecting from a plurality of pattern information files stored in the pattern information storage unit 80, a keyword input field 104 for inputting a keyword to be searched, and “search” for inputting a search execution instruction. An execution button 105, a database display field 106 for displaying the name of the pattern information file selected by the file selection command 102, a pattern / frequency display field 108 for displaying the pattern of the search result and its frequency, and the selected pattern. An example sentence display column 116 for displaying example sentences is included.

  When the user selects the file selection command 102 with a mouse cursor or the like included in the input device 20, a list of names of a plurality of pattern information files stored in the pattern information storage unit 80 is displayed by a pull-down menu. When the user selects one of them, the name of the pattern information file is displayed in the database display field 106 and becomes a search target of the search execution unit 62. In the example of FIG. 5, a pattern information file named “paper” is selected.

  Subsequently, in S <b> 20 of FIG. 4, the user inputs a keyword into the keyword input field 104. In the example of FIG. 5, the word “consider” is input. Then, when the user inputs a confirmation with the “search” execution button 105, the search execution unit 62 stores “consider” from the pattern information file named “paper” stored in the pattern information storage unit 80 and stores it as a frequent pattern. All the patterns that have been detected are detected. The pattern output unit 64 displays the detected pattern in the pattern display field 110 of the pattern / frequency display field 108 and the frequency of the pattern in the frequency display field 112. In the example of FIG. 5, patterns including “consider” such as “we consider the” and “considering the” are displayed in the pattern display column 110 in order of frequency.

  Next, in S26 of FIG. 4, the user selects and inputs a certain pattern from the patterns displayed in the pattern / frequency display field 108 with a mouse cursor or the like. In the example of FIG. 5, a pattern “is considered” is surrounded by a frame 114 to indicate that it is selected. Then, the example sentence output unit 66 reads all sentences including the selected pattern from the text file of the document stored in the pattern information storage unit 80 in units of sentences, and displays them in the example sentence display column 116. At this time, the pattern may be surrounded by a frame 118 or displayed in bold so that it can be seen where the selected pattern appears in the sentence.

  The present embodiment was actually applied to two document sets of a certain English abstract corpus and a certain international conference paper collection. The English abstract corpus has a total number of English texts of 65889 and a data capacity of 8.0 megabytes. The international conference papers have a total of 45115 texts and a data capacity of 4.6 megabytes. In both of these document sets, the pattern information generation process shown in FIG. 3 can be completed in several tens of seconds and the search process shown in FIG. 4 can be performed, so that practicality is sufficiently guaranteed. confirmed.

  Tables 1 and 2 show examples of search results when the present embodiment is applied to a certain paper abstract as a search target. Here, the pattern extraction condition is that the extraction frequency threshold is 5, and the extracted pattern is a continuous word string of a minimum of 2 words and a maximum of 6 words. Table 1 shows patterns and frequencies displayed in the pattern / frequency display column 108 when “consider” is used as a search keyword and Table 2 shows when “study” is used as a search keyword. Here, in the document, the process of normalizing all the changed forms such as “considered” in the document to the original form is performed in the program. Whether or not to normalize is determined automatically, for example, depending on the amount of documents to be searched, or can be specified by the user.

  From Table 1 and Table 2, it can be seen that passive expressions are frequently used for both “consider” and “study” in the document set of this paper abstract. In some cases, the two terms are used with almost the same meaning, but the user can select which expression to use by referring to the example sentence including the pattern selected from Table 1 or Table 2.

  According to this embodiment described above, pattern information can be generated from a document set of a category that the user wants to refer to, and keyword search can be performed only on the pattern information, thereby eliminating patterns with low utilization. It is easy to obtain, and it is easy to acquire a frequent pattern that is a fixed pattern in the set. Therefore, idioms that generally have the same meaning, but one is rarely used in some categories, such as category usage, grammatical bias, category-specific phrases, boilerplates, and contexts in which they are used efficiently. You can get it well and use it in your documents.

  Categories, for example, papers, e-mails, relatively large classifications, physical papers that subdivided papers, classifications of engineering papers, etc. A paper such as a paper in a journal can be easily specified by a user, and a pattern information file can be easily generated. Therefore, in addition to the above-mentioned category-specific phrases, it is possible to realize an ad hoc search function that can meet the detailed needs of the user, such as grasping local and temporary trends, trends, and content searches.

  In addition, it is easy to overview the usage status of patterns in the category from the frequency, etc., and it is easy to grasp the word feeling and usage tendency of keywords. Furthermore, since only the example sentences including the selected pattern are displayed in sentence units, only the minimum necessary example sentence can be efficiently acquired. As a result, the user can examine a more detailed example of the frequent pattern, and by imitating it, it is possible to efficiently create an accurate sentence.

  In addition, since a pattern is extracted by using a sequence pattern mining algorithm, it is possible to search for ambiguous keywords and display a list of patterns, and to specify a desired pattern. In specifying, since the example sentences of each pattern can be referred to, it is easy to select an optimal pattern.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

In this embodiment, a pattern including an input keyword is detected from the pattern information file. However, a dictionary database stored in a hard disk or the like is searched based on the keyword, and a pattern is searched using the obtained phrase as a new keyword. May be. Thereby, for example, a Japanese keyword can be input, and an English pattern and an example sentence corresponding to the Japanese keyword can be confirmed. In addition to language conversion dictionaries such as Japanese-English and Japanese-French, synonym dictionaries can be introduced. This allows synonym expansion for ambiguous keywords.

  In the present embodiment, an apparatus specialized for example sentence search has been described. The embodiment of the present invention is not limited to this, and may be executed in the same manner as other application software in a personal computer or the like as application software that provides similar functions. Also, it may be a plug-in that incorporates the same function into an application that inputs text, such as a document creation application or an e-mail application, and the pattern or usage is automatically added to the document being created by the user selecting a pattern or example sentence. An example sentence may be written.

  Further, the function of the pattern extraction unit 40 and the function of the search unit 60 may not be provided in the same device. For example, a pattern information file generated by the pattern extraction unit 40 may be generated in advance by another device, and a recording medium on which the pattern information file is recorded may be read and searched by a device having the function of the search unit 60, or via a network. The pattern information file may be downloaded and used for searching.

It is a figure which shows the whole structure of the example sentence search apparatus in this Embodiment. It is a figure which shows the structure of the processor in the example sentence search apparatus of this Embodiment in detail. It is a flowchart which shows the procedure which extracts and preserve | saves the information of a pattern in this Embodiment. It is a flowchart which shows the process sequence at the time of performing an example sentence search in this Embodiment. It is a figure which shows an example of the example sentence search screen displayed on an example sentence search apparatus in this Embodiment.

Explanation of symbols

10 example sentence search device, 15 processor, 20 input device, 30 output device, 40 pattern extraction unit, 42 text conversion unit, 44 conversion unit, 46 extraction execution unit, 48 pattern information writing unit, 60 search unit, 62 search execution unit 64 pattern output unit, 66 example sentence output unit, 70 document storage unit, 80 pattern information storage unit.

Claims (9)

A pattern extraction unit that extracts a word string pattern from a document set designated by a user according to a predetermined rule;
A pattern information storage unit that stores pattern information in which the word string pattern extracted by the pattern extraction unit is associated with a sentence that is included in the document belonging to the document set and includes the word string pattern;
A search execution unit that receives a search keyword input and detects from the pattern information stored in the pattern information storage unit a word string pattern including any of the search keyword and a phrase related to the search keyword;
A pattern information output unit for outputting at least a part of the word string pattern detected by the search execution unit and the sentence associated therewith;
An example sentence search device comprising:   The example pattern search apparatus according to claim 1, wherein the pattern extraction unit extracts the word string pattern using a sequence pattern mining technique. A pattern information storage unit for storing pattern information in which a word string pattern extracted from a document set according to a predetermined rule is associated with a document that belongs to the document set and includes the word string pattern;
A search execution unit that receives a search keyword input and detects from the pattern information stored in the pattern information storage unit a word string pattern including any of the search keyword and a phrase related to the search keyword;
A pattern information output unit for outputting at least a part of the word string pattern detected by the search execution unit and the sentence associated therewith;
An example sentence search device comprising: The pattern information storage unit stores a plurality of the pattern information corresponding to a plurality of the document sets,
4. The example sentence search device according to claim 3, wherein the search execution unit receives a selection input from the plurality of pattern information, and detects a word string pattern including the keyword from the selected pattern information. . The pattern information output unit
A pattern output unit for outputting the word string pattern detected by the search execution unit;
A sentence output unit that accepts a selection input from the word string pattern output by the pattern output unit, and outputs the sentence associated with the selected word string pattern;
The example sentence search device according to claim 1, further comprising: In the pattern information stored in the pattern information storage unit, the word string pattern is further associated with the appearance frequency of the word string pattern in the document set,
The example pattern search apparatus according to claim 5, wherein the pattern output unit further outputs the appearance frequency for each word string pattern detected by the search execution unit. Receiving search keyword input,
Refer to pattern information that associates a word string pattern that is stored in advance and extracted from a user-specified document set according to a predetermined rule with a sentence that is included in the document belonging to the document set and that includes the word string pattern. Detecting a word string pattern including any of the search keyword and a phrase related to the search keyword;
Outputting the sentence associated with at least some of the detected word string patterns;
A method for searching example sentences, comprising: A function of storing pattern information in which a word string pattern extracted from a user-specified document set according to a predetermined rule and a sentence included in the document set and including the word string pattern are associated with each other;
A function of accepting a search keyword input, and detecting a word string pattern including any of the search keyword and a phrase related to the search keyword from the pattern information;
A function of outputting at least a part of the detected word string pattern and the sentence associated therewith;
A computer program for causing a computer to realize the above. A word string pattern extracted from a document set according to a predetermined rule, an appearance frequency of the word string pattern in the document set, and a sentence included in the document belonging to the document set and including the word string pattern are associated with each other A recording medium for recording.

Images