JP3937741B2 - Document standardization - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a technique for standardizing a document by processing the document.
[0002]
[Prior art]
Text data search is required in various fields such as patent gazettes and literature database searches, but a large amount of text data is simply stored as a database and stored using pattern matching technology. It is usual to search for items containing the target word. In this case, in order to facilitate the search, a thesaurus or the like is used to search for a concept word that is similar to the word to be searched (for example, when the search word is “car”, ”Or“ vehicle ”as a search term), or a search in consideration of the difference in notation (for example,“ vehicle ”is also searched for“ vehicle ”as a search term). Has been proposed.
[0003]
In order to realize this method, when registering a document in a database, a word that is likely to be searched is derived and stored as a keyword of the document, or when searching, it is called a regular expression. Using a simple expression format, it was possible to search for words such as single-character differences. For example, in Japanese Patent Laid-Open No. 10-240742, when character string information is accumulated, an input candidate character string other than the input original character string is generated, and the input candidate character string is converted into an accumulated character string and accumulated in a database. is doing. When searching, when the searcher inputs search character string information for search, a search candidate character string that can be searched for other than the search character string is generated, and the search character string information and the accumulated stored characters are stored. The search is performed by matching the column.
[0004]
[Problems to be solved by the invention]
However, this method has a problem in that, when registering in the database, derivative words must be generated for each of a large number of words, and the processing is enormous. For example, when the term “switching” includes fluctuations such as “switching”, “switching”, “switching”, and “switching”, all these candidate character strings are generated and stored for each document to be stored. As a result, processing takes time and a huge storage capacity is required.
[0005]
Also, if there are fluctuations of different expressions in different words, for example, if you consider the words “switch” and “rewrite”, unify one to “switch” and unify the other to “rewrite” There is a problem that each must be specified because it is a separate task. Furthermore, as in the above “switching” example, when there are a plurality of notations, it is necessary to specify which notation is used.
[0006]
An object of the present invention is to solve these problems and to standardize a document, thereby reducing the time required for various subsequent document processing, for example, retrieval.
[0007]
[Means for solving the problems and their functions and effects]
The document standardization method of the present invention for solving at least a part of the above problems is as follows.
Enter a document with a certain unity,
Analyzing the document, cutting out words with grammatical information,
A predetermined standardization process is performed on the extracted word,
Outputting a document reconstructed from the standardized words
Is the gist.
[0008]
In addition, the invention of the construction method of the document database made in connection with this,
Enter a document with a certain unity,
Analyzing the document, cutting out words with grammatical information,
A predetermined standardization process is performed on the extracted word,
Accumulating documents reconstructed from the standardized words as a database
Is the gist.
[0009]
Furthermore, the invention of the document retrieval method made in connection with these is as follows:
Prior to searching for documents,
Enter a document with a certain unity,
Analyzing the document, cutting out words with grammatical information,
A predetermined standardization process is performed on the extracted word,
Documents reconstructed from the standardized words are stored in advance as a database,
When searching for documents,
Comparing a designated search word with a document stored in the database to identify a document including the search word
Is the gist.
[0010]
In this invention, a word is cut out with grammatical information by performing morphological analysis on the document, so that appropriate standardization can be performed. In other words, since word extraction is performed, appropriate standardization can be performed in units of words rather than simple replacement. A document reconstructed from standardized words may be output, for example, as a file or to a display, or may be used as a reconstructed document for database construction. In such a database, the documents are standardized and stored in principle, so that the search can be performed very easily.
[0011]
In such standardization,
The predetermined standardization process includes at least
(A) Standardization of characters to be replaced with predetermined characters,
(B) a collocation process for correcting the relationship of words having a co-occurrence relationship to a predetermined relationship;
(C) Unification processing for unifying notation fluctuations to a predetermined notation,
(D) Independent word processing for replacing an independent word with another independent word in accordance with a predetermined replacement criterion;
(E) Ancillary word processing for replacing an ancillary word with another ancillary word according to a predetermined rule
Can be included. By adopting at least one of these processes, standardization of documents can be performed at various levels.
[0012]
These standardization processes can be realized as a word replacement process by referring to a dictionary prepared in advance. Since words are extracted with grammatical information by morphological analysis, it is easy to refer to the dictionary. Such morphological analysis can also be realized by using a morphological analysis dictionary prepared in advance. Of course, it is also possible to perform morphological analysis depending on the algorithm.
[0013]
The plurality of standardization processes can be performed in various orders. For example, it is also preferable to perform the independent word process (d) after the character standardization process (a). By doing so, it is possible to reliably standardize variations of free words such as “WINDOWS” and “Windows” in half-width characters, and “WINDOWS” and “Windows” in full-width characters by a simple operation.
[0014]
It is also suitable to perform independent word processing after collocation processing (b). The collocation processing is to correct the relationship between words in the co-occurrence relationship to a predetermined relationship, and by performing the collocation processing in advance, the independent word processing can be performed more reliably. For example, in the case of performing independent word processing that replaces the conjunctive words “abdomen” + “standing” with “anger”, “abdomen” + “bad” + “standing” is once “bad” by the collocation process. If it is converted to + "belly" + "stand", it is easy to standardize to "bad" + "get angry" by the next independent word processing. Furthermore, it is also preferable to perform the notation unification process (c) at least after the independent word process (d). By doing so, it is not said that the unity of notation is broken rather than the independent word processing.
[0015]
In addition, when there are two or more standardization results in the standardization process, it is also desirable to display one of the two or more results and display that there are a plurality of results. . This is because the user performing the standardization process can know that there are a plurality of results, and can select other candidates depending on the case. It is also desirable from the aspect of candidate selection to sequentially display results other than the displayed results as the next candidate according to the user's operation.
[0016]
Each of these inventions is grasped as an invention of an apparatus that executes the above method, an invention of a program that is executed on a computer and realizes the above functions, and an invention as a recording medium on which such a program is recorded. be able to. The device may realize the above document input, morphological analysis, standardization, output, database construction, etc. by executing a program on a computer, or realized by a discrete circuit configuration. It may be. As the program, a well-known program language such as C language, Pascal, Fortran, Cobol, BASIC, or the like can be used, and an object-oriented program language, a language such as JavaScript, or the like can also be used. As the recording medium, various recording media such as a flexible disk, CD-ROM, DVD-ROM, and semiconductor memory (ROM, PROM, EEPROM, flash memory, etc.) can be used. Of course, these programs can be stored in a server placed on a network such as the Internet, and downloaded to a client computer for use.
[0017]
Other aspects of the invention
The standardization technique of the present invention can be used for translation, for example. In translation, it is effective to create a database of translation examples, and it is extremely useful in search for translation to construct such a database with plain text that is free from a trap of a document created by a translator. It is also effective to apply the same standardization when a search engine such as the Internet searches a large number of webs on the net and creates a database. This is because the creation of websites and the like is basically left to individual responsibility, so the expression of documents is not unified.
[0018]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described based on examples.
(1) Configuration of the embodiment:
First, the configuration of the embodiment will be described with reference to FIG. FIG. 1 is a schematic configuration diagram showing a system for constructing a database according to the present embodiment. This system is realized as a database server 200 connected to a large-scale network 10 such as the Internet. A client (not shown) is connected to the network 10.
[0019]
The database server 200 includes a network interface (NT-I / F) 21 that controls data exchange with the network 10 via a modem or a router 20, a CPU 22 that performs processing, a ROM 23 that stores processing programs and fixed data, A RAM 24 as a work area, a timer 25 for managing time, a display circuit 26 for managing display on the monitor 30, a hard disk (HD) 27 for storing various data to be described later, and an input interface for managing an interface with the keyboard 11 and mouse 12 (I / F) 28 and the like. Although the hard disk 27 is described as being fixed, it may be removable, or a removable storage device (for example, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, A flexible disk or the like can also be used together. In this embodiment, the processing program of the server 200 is stored in the ROM 23. However, the processing program may be stored in the hard disk 27 and expanded and executed on the RAM 24 at startup. Alternatively, it may be read from the above-described removable recording medium. Further, it may be executed by reading from another server via the network 10.
[0020]
The server 200 shown in FIG. 1 standardizes a document (text data) input from the keyboard 11 or text data taken from outside via the network 10 and finally constructs a document database on the hard disk 27. . Thereafter, search processing can be performed on the document data stored in the database, but this search processing may be performed from the server 200 or from each client connected via the network 10. .
[0021]
As described above, hardware such as the CPU 22 and the ROM 23 is provided in the server 200. By executing a program to be described later in the server 200, the configuration shown in FIG. 2 can be realized. it can. In other words, the server 200 realizes the same function as the function realizing means shown in FIG. As shown, the server 200 includes a document input unit 205, a morphological analysis unit 210, a dictionary search unit 220, a morphological analysis dictionary 230, a standardization rule database 240, a standardization processing unit 250, a log management unit 260, a document output unit 270, A log output device 280 is provided.
[0022]
Here, the document input unit 205 realizes processing for inputting a document, and inputs a document from the keyboard 11 or takes in a document stored in the hard disk 27 or the like in advance. The morphological analysis unit 210 performs morphological analysis on the text data of the input document. The morphological analysis unit 210 analyzes the morpheme of the text data mixed with kanji characters and displays the independent words and attached words constituting the text data together with the grammatical information. To get. The standardization processing unit 250 performs standardization processing on text data subjected to morphological analysis. As standardization processing to be executed,
(A) Character standardization processing (standardization of characters to be replaced with predetermined characters),
(B) collocation processing (processing for correcting the relationship of words having a co-occurrence relationship to a predetermined relationship),
(C) Unification process of notation (process to unify fluctuation of notation to a predetermined notation),
(D) Independent word processing (processing for replacing independent words with other independent words in accordance with predetermined replacement criteria),
(E) Adjunct processing (processing for replacing an adjunct with another adjunct according to a predetermined rule)
There is. All of these processes do not need to be executed, and a required process (at least one process) is executed according to a user setting.
[0023]
The document output unit 270 outputs standardized text data to the outside. In this embodiment, the text data is stored as a database in the hard disk 27. However, the text data after the standardization processing may be simply displayed on the monitor 30, or printed on a printer (not shown) or the like. It is good as a thing. Alternatively, it may be output to an external client machine via the network 10.
[0024]
The dictionary search unit 220 is for referring to the morphological analysis dictionary 230 and the standardization rule database 240. When it becomes necessary to refer to a dictionary or a database, the morphological analysis unit 210 or the standardization processing unit 250 accesses the dictionary 230 or the database 240 via the dictionary search unit 220, retrieves necessary information, and extracts a morpheme analysis unit. 210 and the standardization processing unit 250. The dictionary search unit 220 may be provided separately for each morphological analysis dictionary 230 and standardization rule database 240.
[0025]
The log management unit 260 and the log output unit 280 manage a log of standardization processing and output it. As described above, the standardization process is performed at various levels from the character standardization to the collocation process, so that the log is managed and output so that it can be referred to as necessary as to what kind of process has been performed. . The log stores information such as the document to be processed, the contents of the standardized processing that has been performed, and, as a result, errors.
[0026]
(2) Overview of processing in the embodiment:
Therefore, the standardization process realized by the standardization processing unit 250 will be described with reference to FIG. FIG. 3 is an explanatory diagram showing an outline of processing executed by the standardization processing unit 250. In this figure, the standardization processing unit 250 is described as executing all the standardization processes, but there are actually cases where at least one standardization process is executed. Which standardization process or any combination thereof is executed is determined by the user through initial settings (properties, etc.). When the standardization processing routine shown in FIG. 3 is started, first, a document reading process is executed (step S300). This process is a process corresponding to the document input unit 205, and may be a document input from the keyboard 11, or a processing document TXT (text data) already created and stored in the hard disk 27 or the like. It may be read out. Therefore, for example, an icon indicating the execution of the standardization process is displayed on the so-called desktop of the monitor 30, and the text file is dragged and dropped with the mouse 12 to start the standardization process shown in FIG. Can also be read.
[0027]
The reading of the document may be realized by reading all data at once, or may be sequentially read from the text data by using a line feed or the like as a delimiter code. If possible, it may be read in “sentence” units using punctuation marks. In any case, it is desirable to assign an identification number to each sentence and use it for management thereafter. Note that the text data may be actually expanded on the RAM 24 so that it can be processed, or may be stored in the hard disk 27 or the like so that it can be accessed randomly or sequentially after an identification number is assigned.
[0028]
After reading the document in this way, first, morphological analysis processing is performed (step S310). This is a process corresponding to the morphological analysis unit 210 and corresponds to a process of referring to the morphological analysis dictionary 230 via the dictionary search unit 220. Actually, referring to the reverse lookup dictionary IDC stored in the hard disk 27, words constituting the document are determined by morphological analysis. Details of the morphological analysis process are shown in FIG. Hereinafter, the morphological analysis process will be described with reference to FIG. Note that the reverse dictionary is a normal dictionary for kana-kanji conversion in which conversion character strings such as kanji and katakana are arranged with grammatical information using kana characters as headings, as shown in FIG. This is a dictionary arranged in reverse. Therefore, for example, it is possible to extract grammatical information such as a noun reading and a noun from a character string “school”.
[0029]
When the morphological analysis process is started, a sentence with an identification number is first identified as an object to be analyzed, and the M character (M = 1, 2,...) From the beginning of this sentence is the L character. Minutes (L = 1, 2,...) Are taken out and the reverse lookup dictionary IDC is looked up (step S12). M indicates the head position of the character string of interest, and L indicates the number of characters to be extracted. The reverse lookup dictionary reference method starts with a process of extracting M = 1, that is, L = 1, that is, one character from the head position, and referring to the dictionary to extract the corresponding word. The dictionary IDC is referenced while sequentially incrementing L. If there is no corresponding entry word, the head position M of the character string of interest is incremented, the number of characters L is returned to 1, and the dictionary is searched. When the position of the focused character or the number of characters of the sentence to be analyzed is exceeded, the dictionary reference is cut off.
[0030]
For example, referring to the reverse lookup dictionary IDC for a sentence “A car named DD is a sedan that emphasizes quality”, “DD” “to” “say” “to” “i” “u” “ The words “car” “ha” “quality” “do” “important” “do” “do” “ta” “sedan” “de” “are” “is” “a” can be extracted. Here, the kana sounds such as “I”, “U”, “A”, “Shi”, “Ta”, etc. are also extracted as words, “I” and “Uru” (sell) This is because the stem of “)” may appear in the sentence.
[0031]
In the reverse dictionary IDC, these words are stored together with the grammatical information. Therefore, the extracted words are arranged next according to the grammatical information, and a process for finding an array that does not fail is performed. For this analysis, for example, methods such as a multiple phrase longest match method and a minimum cost method are known, and a test is performed to determine which combination of predetermined words is most likely Japanese. In the present embodiment, since the minimum cost method is adopted, the cost calculation is next performed for a large number of character strings thus obtained (step S314). The cost calculation is a process of calculating the cost of a character string prepared in advance so that the score is lower for a Japanese character array than for a character string array. The rule is roughly: a self-supporting word has a cost of 2, and if an adjunct is attached to it, the cost is 0. For example, taking “quality” as an example, if “quality” + “wa”, it becomes a link between independent words + adjuncts (particles) and costs 2, “goods” + “quality” + “ "Independent words + independent words + ancillary words (particles), the cost is 4. The rules of the minimum cost method are tuned according to the actual Japanese language, and when words having a co-occurrence relationship such as “nothing” + “no” occur in the sentence, there are various costs such as “−1”. Rules are prepared.
[0032]
In this way, the above-mentioned cost is calculated for all the words obtained by referring to the reverse lookup dictionary, and a process for specifying a sentence having the lowest cost is performed (step S315). In the above example, "Quality" (independent words / nouns) + "O" rather than "Goods" (independent words / nouns) + "Quality" (independent words / nouns) + "O" (adjuncts / particles). It is judged that the (adjunct / particle) is more likely to be Japanese. Of course, this calculation is performed at least for each sentence, and an array of words is selected so as to minimize the cost of the entire sentence. Therefore, for example, if there is a cost reduction due to the co-occurrence relationship, there is a case where “quality” + “quality” + “to” is selected instead of “quality” + “to”.
[0033]
When the morpheme analysis by the minimum cost method is completed in this way, the sentence structure is analyzed (step S316). This process expresses how the words constituting a sentence are connected by logical product and logical sum, and is used, for example, when a compound sentence is separated into two sentences. In this embodiment, this point is not particularly described. After performing the above process, the process which outputs the morphological-analyzed data is performed (step S318). The data may be transferred to the next standardization process as it is, or may be temporarily stored in the hard disk 27 with an identification code.
[0034]
Next, various standardization processes are executed on the sentence thus morphologically analyzed (see FIG. 3). As standardization processing,
(1) Character standardization process (step S320)
(2) Conjunction processing (step S330)
(3) Independent word processing (step S340)
(4) Unification processing of notation (step S350)
(5) Attached word processing (step S360)
There is. As described above, each standardization process does not need to be executed all, and only a process that matches the user's intention may be performed. Further, when a plurality of standardization processes are performed, the order is not limited to the above order, and may be performed in other orders.
[0035]
First, character standardization processing will be described with reference to FIG. When the character standardization process is started, a process for referring to the standardization rule CSD is first executed (step S322). The standardization rule CSD is stored in advance in the hard disk 27, and defines what rule the standardization of characters is to be performed. These rules have defaults, but can be changed by the user. Character standardization in this embodiment means that, as shown in FIG. 7, parentheses, quotes, general symbols, alphanumeric characters, punctuation marks, punctuation marks, half-width katakana characters, name concatenation characters, and long sound symbols are replaced with certain rules. Say processing. Among these, those indicated by “x” in the column A in FIG. 7 indicate that surrounding characters do not need to be considered in replacement, and “◯” indicates that surrounding characters need to be considered. Yes. Also, column B shows the range of replacement, but “sentence” may be the range of replacement here, so for example “−” (minus sign) and “−” (long sound symbol) ) Is different from that of the morpheme analysis if it is replaced with a long sound symbol. Therefore, when a long sound symbol is replaced, the sentence structure may be changed with reference to the reverse dictionary IDC.
[0036]
As an example of character normalization, when taking a punctuation mark and a punctuation mark, these are set to be replaced by “,” “.” By default. Therefore, if the sentence “Coffee is delicious” is replaced with a character, “Coffee is good.” However, since this setting can be changed, if the setting is changed to “,” as a punctuation mark and “,” as a reading mark, “Coffee is delicious”. In addition, as shown in the column A, surrounding characters are taken into account, but if the surrounding character string is English, the replacement with “,” “.” Is the default setting. .
[0037]
To illustrate the standardization of other characters,
(A) Parentheses: “” and “” are replaced, etc.
(B) Quotation marks: “” and “” are replaced, etc.
(C) General symbols: For various symbols (for example, “:,?!”, Etc.)
(D) Alphanumeric characters: replace full-width / half-width, uppercase / lowercase characters, etc.
(E) Half-width katakana: Full-width / half-width replacement for katakana, etc.
(F) Name continuation: “Queen = Elizabeth” replaced with “Queen Elizabeth”, etc.
There is.
[0038]
Using these rules, a process of changing each character is performed (FIG. 6, step S324). Thereafter, it is determined whether or not the replacement has been completed for all the characters (step S326), and the replacement according to the rule is performed until all the characters are completed.
[0039]
After the character standardization processing described above is performed, next, co-occurrence collocation processing (FIG. 3, step S330) is executed. Details of this processing are shown in FIG. Hereinafter, a description will be given with reference to FIG. When co-occurrence collocation processing is started, attention is first focused on the phrase N of the sentence obtained by morphological analysis (step S331). At the start of processing, N = 1. Next, the phrase string search process is performed backward with reference to the co-occurrence dictionary RGD (step S332). The state of this search is shown in FIG. FIG. 9 shows a state in which co-occurrence collocation processing is performed on a sentence “I hurry to school”. By the morphological analysis, the phrase “I am” + “To school” + “Hurry up” + “I will go” is extracted. More specifically, each phrase is analyzed as an independent word + an attached word (+ an attached word...).
[0040]
First, focus on the phrase N = 1, that is, "I am", and search for phrases such as N = 2, 3, 4, ie, "To school", "Hurry up", "Go", etc. The search is to verify whether there is a phrase described in the co-occurrence dictionary RGD. Therefore, it is not a search by phrase, but a search using a phrase and its stem. While performing such a search, it is determined whether there is a phrase having a co-occurrence relationship (step S333). In the example shown in FIG. 9, there is no item corresponding to the co-occurrence dictionary for “I am”, that is, N = 2, that is, the co-occurrence relationship of “line to school” is “co-occurrence dictionary RGD”. Was found. When a phrase having a co-occurrence relationship is found, it is next determined whether or not the phrase can be replaced (step S334). If two phrases in a co-occurrence relationship are continuous, there is no need to swap them. Moreover, even if a co-occurrence relationship is found between clauses located at distant positions, there are cases where the clauses cannot be replaced due to the sentence structure. For example, in the example sentence "I will call school and then go", even if the co-occurrence relationship between "to school" and "go" is found, it will be replaced with "I will call and then go to school" It is not always possible. This is because there are restrictions on the sentence structure.
[0041]
If it is determined that the two clauses found to be in the co-occurrence relationship are separated from each other and that the clauses can be interchanged due to the sentence structure, a processing for replacing the positions of the clauses is performed (step S335). . As a result, as shown in FIG. 10, the sentence becomes “I hurry to school”. Subsequently, a collocation process is performed (step S336). That is, since a co-occurrence relationship is recognized in two consecutive phrases, this is converted into a collocation and treated as one phrase. This situation is shown in FIG. In the above embodiment, the collocation based on the co-occurrence relation has been described as collocation of two phrases into one phrase. However, in some cases, three or more phrases can be collocated into one phrase. .
[0042]
Thereafter, the focused phrase is advanced by one (step S337), and it is determined whether or not the co-occurrence relation processing has been completed for all the phrases (step S338). If not yet completed, the process returns to step S332 to perform the processing. continue. When the co-occurrence processing is completed for all the clauses, the process exits to “NEXT” and ends this routine. In the above flow, the search for the phrase having the co-occurrence relation is performed in order from the first phrase in the sentence. However, a method of performing the search by specifying the receiver of the so-called “dependency” first. If it is adopted, the search can be made sequentially from the back of the sentence. Which to search depends on the configuration of the dictionary and the search algorithm.
[0043]
When character standardization (FIG. 3, step S320) and co-occurrence collocation processing (step S330) are thus completed, next, independent word standardization processing is performed (step S340). Details of this processing are shown in FIG. When the independent word standardization process shown in FIG. 12 is started, a process for referring to the standardization rule is first performed (step S342). In this process, the default is set in advance, similar to that referred to in the character standardization, but the setting that can be changed by the user is acquired. Of course, this rule can be fixed. The standardization of independent words is basically a process of replacing different expressions between independent words having the same meaning. There are many types of such processing, for example,
(1) Replace with a more general expression: eg, tennis ball → tennis
▲ 2 ▼ Replace with plain expression: e.g. scarlet → blue
(3) Avoidance outside of common kanji: eg, caress → love and greetings → greetings
(4) Simplification of idioms: for example, every move, every move
Replacement with the character form used from (5): eg Windows → Windows, spiral up → spiral up
(6) Replacing collocations: for example, going to school → going to school
Etc. can be considered.
[0044]
These processes are actually performed by sequentially extracting independent words from the sentence to be standardized and searching the standardized dictionary IWD for independent words (step S344). The standardized dictionary IWD for independent words is configured so that the replaceable independent words described above can be referred to together with the rules to which the replaced independent words are applied. Therefore, after obtaining the standardization rules, it is possible to refer to the dictionary, read replacement words that match the rules, and perform a process of changing each word (step S346). FIG. 13 is an explanatory diagram schematically showing the state of this replacement. As shown in the figure, reference is first made to rule settings. In the figure, “◎” indicates that the replacement is set (ON), and “◯” indicates that it is not set (OFF). In the independent word standardization process, if the above (1) to (6) are taken as an example, which replacement is performed is stored as a standardization rule. Read sequentially, and search for whether or not the word to be replaced is registered in the dictionary IWD for this independent word, and if it is registered, check if it matches the replacement rule that is currently on. If it matches the replacement rule that is turned on, it will replace the independent word. The above process is repeated for all words (step S348). In the example shown in FIG. 13, (3) “I am” is replaced with “I am” because repelling outside the common Chinese characters is on. In addition, words that have been recognized as having a co-occurrence relationship and made into collocation are also replaced with other words as necessary. In this example, “Go to school” → “Go to school” is replaced. In response to this, the attached word part was also replaced with “kuyo” → “suruyo”.
[0045]
As a result, when the independent word standardization process is completed, all the words are replaced with respect to the type predetermined as the standardization rule, and the independent words are standardized at a desired level.
[0046]
After standardization of independent words, standardization processing of notation fluctuation is performed next (FIG. 3, step S350). Notation fluctuation means the ambiguity and tolerance of notation in Japanese. For example,
(1) Long-symbol fluctuation: example, user, user,
(2) Feeding kana: example, sales, sales,
(3) Shake notation: examples, wizards, wizards,
(4) Kana notation of compound words: examples, sales, sales,
▲ 5 ▼ Foreign language notation: Angel, Angel,
(6) Repeated character fluctuation: eg, Shosho-do Hall
Etc. can be illustrated.
[0047]
Since the outline of this process is similar to the independent word standardization process shown in FIG. 12, a flowchart is not shown. However, as with the independent word standardization, reference is first made to rule settings. That is, in the standardization processing of the fluctuation of the notation, taking the above (1) to (6) as an example, as shown in FIG. 15, the standardization rule DAD (see FIG. 3) ) Is read out, and then the words are sequentially read out. If the rule stored in the standardized rule DAD is applicable to this word, the normal word dictionary DIC for kana-kanji conversion is searched. . Since notation fluctuations are widely registered in this dictionary, if a word corresponding to the rule specified by the standardization rule DAD is registered in the dictionary DIC, it is read out and replaced with a word having a different notation. is there. The above process is repeated for all words.
[0048]
The standardization of independent words is slightly different from the standardization, but the standardized dictionary of independent words is constructed on the assumption that standardization in one direction is performed, whereas the fluctuation of notation is standardized in both directions. This is because it is assumed. This is because the fluctuation of the notation is not suitable for the judgment that either of the notations is correct because the allowable range is large. These notation fluctuations are widely collected in the word dictionary DIC for Kana-Kanji conversion and are associated with each other. Therefore, when standardizing the notation fluctuations, refer to the notation fluctuation standardization rules DAD and specify them. The word dictionary DIC is searched for the written notation.
[0049]
After standardizing the fluctuation of the notation in this way, the standardization processing of the attached words is performed (step S360). Since the outline of this process is almost the same as the independent word standardization process shown in FIG. 12, a flowchart is not shown, but it is basically a process of replacing different expressions between attached words having the same meaning. There are many types of such processing, for example,
(1) Simplification of repeated polite expressions: Have been → "Out" Being ”
(2) Modernization of archaic expressions: example, “Cause Whether or not → “Cause Whether or not "
(3) Regular expression of simple expressions: eg “I have to study” → “I have to study”
Can be considered.
[0050]
In practice, these processes are performed by sequentially retrieving the attached words from the sentence to be standardized and searching the attached dictionary standardized dictionary AWD. The standardized dictionary AWD for adjuncts is configured such that the replaceable adjuncts described above can be referred to together with the rules to which they are applied. Therefore, after obtaining the standardization rules, it is possible to refer to the dictionary, read replacement words that match the rules, and perform processing to change each attached word. FIG. 16 is an explanatory diagram schematically showing the state of this replacement. As shown in the figure, reference is first made to rule settings. That is, in the standardization processing of attached words, taking the above (1) to (3) as an example, which replacement is performed is stored as a standardization rule. The word is read sequentially, and it is searched whether or not the word to be replaced is registered in the dictionary AWD for this attached word, and if it is registered, whether or not it matches the replacement rule that is currently turned on. Check and replace the attached word if it matches the replacement rules that are turned on.
[0051]
As a result, when the standardization process of the attached word is completed, all the words are replaced with respect to the type predetermined as the standardization rule, and the attached word is standardized at a desired level.
[0052]
When all the standardization (steps S320 to S360) shown in FIG. 3 is completed in this way, the server 200 performs a process of registering the standardization result in the document database TDB in the hard disk 27 (step S370). This database is a full-text database of documents, and a full-text search can be performed by a search device described later.
[0053]
(3) Effects of the embodiment:
Documents registered in the document database TDB have been standardized in the form of characters, independent words, fluctuations in notation, and attached words, so that most of the writer's habits and wording differences have been eliminated. Therefore, the processed document is extremely plain text data and can be used for various purposes. For example, if it is used for construction of a full-text database such as patent gazettes and technical literatures, it is possible to reduce noise and search omissions when searching the completed database. Also, if the sentence to be translated is standardized, it can be used as one of the rough translations for machine translation. Conversely, when constructing a translation database that accumulates translation examples, differences in translators can be resolved. Furthermore, it can also be used for a wide range of comparative studies on documents, such as a study of comparing author expressions across time. In this embodiment, prior to standardization processing, text data is subjected to morphological analysis to obtain necessary grammatical information. For this reason, standardization is not limited to simple character replacement, but can be performed collectively as standardization of independent words using grammatical information, standardization of notation fluctuation, and the like. For this reason, the number of rules prepared for standardization can be reduced. Since there is grammatical information, standardization can be easily performed by referring to a dictionary for kana-kanji conversion, a dictionary for fluctuation of notation, a dictionary for replacing independent words, and the like.
[0054]
In the embodiment, standardization processing includes (1) character standardization, (2) co-occurrence collocation processing, (3) independent word standardization processing, (4) standardization processing of fluctuation, and (5) appendix word processing. Although the processing is performed in the order of standardization processing, this processing can be performed in various orders. If independent word standardization processing is performed after character standardization processing as in this embodiment, for example, half-width / full-width conversion is completed by character standardization, and then variations in independent words such as “WINDOWS” and “Windows” are obtained. Can be standardized, so that the processing can be surely performed with a simple operation.
[0055]
Similarly, it is also preferable to perform standardization processing of independent words after the collocation processing. By performing the collocation process in advance, the standardization process of independent words can be performed more reliably. In the embodiment, it has been described as a process of replacing the collocation “go to school” + “go” with “go to school”, but “go to school” + “hurry up” + “go” is once performed by collocation processing. If it was replaced with "Hurry up" + "To school" + "Go", it was easy to standardize to "immediately" + "go to school" in the next independent word processing. Further, since the process of unifying the notation fluctuation is performed after the standardization process of independent words, it cannot be said that the standardization of the notation once is lost compared to the standardization process of independent words.
[0056]
In the above-described embodiment, the case where there are two or more results in the standardization process is not particularly described. However, when there are two or more results (for example, “sales” and “sales” It is also possible to preferentially display one of them and display that there are multiple results. Such a display can be easily realized by changing the color of the standardized character on the monitor 30 or displaying “next candidate”. If it is displayed that there is a next candidate, it is possible for the user performing the processing to know that there are a plurality of results, which is preferable. To select another candidate, move the cursor to the displayed phrase and press the “next candidate” key to display the next candidate. If necessary, select the desired candidate from multiple candidates. Just choose.
[0057]
In addition, in this embodiment, since the log management unit 250 manages the standardization log, it is possible to leave the details of the standardization process performed on the input document. You can always restore the original sentence from the standardized sentence as long as you keep a log in the form of what word and what word in the sentence you entered. You can also. In addition, by analyzing the log output from the log output unit 280, it is possible to analyze which type of standardization is frequently used. It is also possible to analyze whether it is a simple colloquial sentence, etc.) or a song (whether it is easy to drop a long sound).
[0058]
(4) Description of the second embodiment:
Next, as a second embodiment of the present invention, a document search method and a search apparatus will be described. The document database TDB completed by the document standardization process described as the first embodiment is disclosed to the outside and used for free use or registered member use. At this time, a configuration in which the document database TDB is directly accessed is possible, but in order to be accessible from an unspecified number of clients via the network 10, for example, a CGI for accessing the document database TDB is provided. A site is usually prepared in the server 200, and the client 40 is usually allowed to access the document database TDB from a so-called browser via the network 10. Therefore, as a second embodiment, a method for searching for a web page using the document database TDB will be described. FIG. 17 is a flowchart showing processing at the time of search executed in the client 40. First, the client 40 that starts the search accesses a site in the server 200 prepared for the search (step S400). As a result, a search screen as shown in FIG. 18 is displayed.
[0059]
Therefore, the client inputs the search contents as a sentence in Japanese into the box KB for inputting the search character string prepared on this screen (step S410). For example, as shown in FIG. 18, a natural sentence such as “I attended school” is entered in a box KB for inputting a character string. At this time, in parallel with the input of the search text, it is monitored whether the “search” button BB is pressed (step S420). When the search button is pressed, the input text is read and the input shown in FIG. In this case, the sentence is subjected to morphological analysis, and the standardization process described in the first embodiment is performed (step S430). The search does not necessarily need to be performed based on textual input. For example, a keyword may be input and a search may be performed using one or a plurality of keywords, or a search may be performed by specifying a keyword and a search field. Also good.
[0060]
The document database TDB is searched using the search terms ("I" and "school attendance" in the example of FIG. 18) DS1 and DS2 extracted from the standardized search sentences obtained in this way (step S440). If a document having a matching sentence is found as a result of the search, the search result is output (step S450). The output search result is sent to the client via the network 10 and displayed on the screen on the client 40 side.
[0061]
According to the second embodiment described above, it is possible to perform a search using a natural Japanese sentence against a document database registered in advance as a standard. In this case, since the search is performed after standardization of a user's word habit of performing the search is reduced by standardization, a desired document can be easily found by the search. For this reason, it is not necessary to be familiar with complicated rules for inputting a search word, and a search can be easily performed even if the searcher does not have special training.
[0062]
As mentioned above, although embodiment of this invention was described, this invention is not limited to such embodiment at all, and in the range which does not deviate from the summary of this invention, it can implement in various forms. Of course. For example, the document database may be a database using keywords instead of the full-text database. It can also be applied to a translation device. Translation simply doesn't work even if you try to convert between languages using grammatical information (necessary rules grow infinitely), rather, prepare abundant examples, find examples that match the translation, It is known that a translation that is applied in such a way that it can be translated into a semantically accurate one. Therefore, it is possible to use the present invention by applying the present invention to standardized text data, standardizing the document, and using this to make it easy to specify an example.
[Brief description of the drawings]
FIG. 1 is a schematic configuration diagram showing an overall configuration in an embodiment of the present invention.
FIG. 2 is a block diagram showing a configuration for realizing standardization processing in the first embodiment.
FIG. 3 is a flowchart showing a standardization processing routine in the embodiment.
FIG. 4 is a flowchart showing a morphological analysis processing routine.
FIG. 5 is an explanatory diagram illustrating the configuration of a reverse lookup dictionary.
FIG. 6 is a flowchart showing a character standardization processing routine;
FIG. 7 is an explanatory diagram illustrating the contents of character standardization processing;
FIG. 8 is a flowchart showing a co-occurrence collocation processing routine.
FIG. 9 is an explanatory diagram showing a process of collocation.
FIG. 10 is an explanatory diagram showing a state of phrase replacement in collocation.
FIG. 11 is an explanatory diagram showing a state of collocation.
FIG. 12 is a flowchart showing an independent word standardization processing routine;
FIG. 13 is an explanatory diagram schematically showing a state of replacement of independent words.
FIG. 14 is an explanatory diagram showing an example of replacement of free-standing words when repelling outside the common kanji is on.
FIG. 15 is an explanatory diagram showing an example of a standardization rule DAD indicating which replacement is performed.
FIG. 16 is an explanatory diagram schematically showing how an attached word is replaced.
FIG. 17 is a flowchart showing processing at the time of search executed in the client 40 as the second embodiment.
FIG. 18 is an explanatory diagram showing an example of a search screen in the second embodiment.
[Explanation of symbols]
10 ... Network
11 ... Keyboard
12 ... Mouse
20 ... Router
22 ... CPU
23 ... ROM
24 ... RAM
25 ... Timer
26. Display circuit
27 ... Hard disk
30 ... Monitor
40 ... Client
200 ... Database server
205 ... Document input section
210 ... Morphological analyzer
220 ... dictionary search section
230 ... Dictionary for morphological analysis
240 ... Database
240 ... Standardized rule database
250 ... Standardization processing unit
260 ... Database
260 ... Log management section
270: Hard disk
270 ... Document output section
280 ... Log output part

Claims (16)

A document standardization method for standardizing a document using a computer,
A process of inputting a document with a certain unit from a file or a character string from the keyboard,
A process of extracting a plurality of words with grammatical information by analyzing the morpheme by referring to a dictionary stored in a storage device in advance by the computer;
A process in which the computer performs predetermined standardization on the extracted words;
Processing to output a document reconstructed from the standardized words;
The predetermined standardization process is as follows:
(B) When the plurality of extracted words have the stored co-occurrence relationship by referring to the co-occurrence relationship stored in advance in the storage device, the positional relationship of adjacent positions of the plurality of words having the relationship Conjunction processing to be corrected to
(D) an independent word process that replaces an independent word including a plurality of adjacent words whose positional relationship has been corrected with another independent word according to a predetermined replacement criterion;
(E) A document standardization method including ancillary word processing for replacing ancillary words with other ancillary words according to a predetermined rule. The standardization method according to claim 1,
The predetermined standardization process is as follows:
(A) Standardization of a character that replaces a previously stored character among characters constituting the input document with a predetermined character according to a predetermined rule;
(C) a notation unification process for unifying the fluctuation of the notation into a predetermined notation in accordance with a pre-stored rule;
A standardization method comprising at least one of The standardization method according to claim 1,
The morpheme analysis uses a morpheme analysis dictionary prepared in advance.   The standardization method according to claim 2, wherein the independent word processing (d) is performed after the character standardization processing (a).   The standardization method according to claim 1, wherein the independent word processing (d) and the adjunct word processing (e) are performed after the collocation processing (b).   3. The standardization method according to claim 2, wherein the unification processing (c) is performed at least after the independent word processing (d). The standardization method according to claim 1,
When two or more standardization results exist during the standardization process, one of the two or more results is displayed, and
A standardized method that indicates that multiple results exist. The standardization method according to claim 7, comprising:
A standardization method for sequentially displaying results other than the displayed results as next candidates according to a user operation. A method for constructing a database in which a plurality of documents are registered as a document database using a computer,
A process of inputting a document with a certain unit from a file or a character string from the keyboard,
A process of extracting a plurality of words with grammatical information by analyzing the morpheme by referring to a dictionary stored in a storage device in advance by the computer;
A process in which the computer performs predetermined standardization on the extracted words;
A process of accumulating documents reconstructed from the standardized words as a database,
The predetermined standardization process is as follows:
(B) When the plurality of extracted words have the stored co-occurrence relationship by referring to the co-occurrence relationship stored in advance in the storage device, the positional relationship of adjacent positions of the plurality of words having the relationship Conjunction processing to be corrected to
(D) an independent word process for replacing an independent word including a plurality of adjacent words whose positional relationship is corrected with another independent word according to a predetermined replacement criterion;
(E) A method for constructing a document database including an adjunct processing for replacing an adjunct with another adjunct according to a predetermined rule. A document search method for searching a document using a computer,
Prior to searching for documents,
A process of inputting a document with a certain unit from a file or a character string from the keyboard,
A process of extracting a plurality of words with grammatical information by analyzing the morpheme by referring to a dictionary stored in a storage device in advance by the computer;
A process in which the computer performs predetermined standardization on the extracted words;
(B) When the plurality of extracted words have the stored co-occurrence relationship by referring to the co-occurrence relationship stored in advance in the storage device, the positional relationship of adjacent positions of the plurality of words having the relationship Conjunction processing to be corrected to
(D) Independent word processing that replaces an independent word including a plurality of words whose positional relationship has been corrected with another independent word according to a predetermined replacement criterion;
(E) an ancillary word processing for replacing an ancillary word with another ancillary word according to a predetermined rule, a standardization process including a process for outputting a document reconstructed from the standardized word;
Documents reconstructed from the standardized words are stored in advance as a database,
When searching for documents,
A document search method in which a computer compares a designated search word with documents stored in the database, and a computer specifies a document including the search word. An apparatus for standardizing a document composed of text data using a computer,
An input means for inputting a document having a certain unit,
Morphological analysis of the document, and morphological analysis means for extracting words with grammatical information;
Standardization processing means for performing a predetermined standardization process on the extracted word;
Document output means for outputting a document reconstructed from the standardized words,
The standardization processing means includes:
(B) When the plurality of extracted words have the stored co-occurrence relationship by referring to the co-occurrence relationship stored in advance in the storage device, the positional relationship of adjacent positions of the plurality of words having the relationship Collocation processing means to correct to
(D) an independent word processing means for replacing an independent word including a plurality of words whose positional relationship is corrected with another independent word according to a predetermined replacement criterion;
(E) A document standardization device provided with an adjunct processing means for replacing an adjunct with another adjunct according to a predetermined rule. The document standardization apparatus according to claim 11,
The standardization processing means includes:
(A) a character standardization means for replacing with a predetermined character;
(C) a notation processing means for unifying the fluctuation of the notation into a predetermined notation;
Document standardization device including at least one of them. An apparatus for building a document database in which a plurality of documents are registered,
An input means for inputting a document having a certain unit,
Morphological analysis of the document, and morphological analysis means for extracting words with grammatical information;
Standardization processing means for performing a predetermined standardization process on the extracted word;
Document storage means for storing a document reconstructed from the standardized words as a database,
The standardization processing means includes:
(B) When the plurality of extracted words have the stored co-occurrence relationship by referring to the co-occurrence relationship stored in advance in the storage device, the positional relationship of adjacent positions of the plurality of words having the relationship Collocation processing means to correct to
(D) an independent word processing means for replacing an independent word including a plurality of words whose positional relationship is corrected with another independent word according to a predetermined replacement criterion;
(E) An apparatus for constructing a document database provided with an adjunct processing means for replacing an adjunct with another adjunct according to a predetermined rule. A document search device for searching for a document,
A means of entering a document with a certain unit that operates prior to document retrieval;
Morphological analysis of the document, and morphological analysis means for extracting words with grammatical information;
Standardization processing means for performing a predetermined standardization process on the extracted word;
Storage means for previously storing a document reconstructed from the standardized words as a database,
The standardization processing means includes:
(B) When the plurality of extracted words have the stored co-occurrence relationship by referring to the co-occurrence relationship stored in advance in the storage device, the positional relationship of adjacent positions of the plurality of words having the relationship Collocation processing means to correct to
(D) an independent word processing means for replacing an independent word including a plurality of words whose positional relationship is corrected with another independent word according to a predetermined replacement criterion;
(E) an ancillary word processing means for replacing an ancillary word with another ancillary word according to a predetermined rule;
A document search apparatus comprising search means that operates when searching for a document and compares a specified search word with a document stored in the database to identify a document including the search word. A program for causing a computer to perform processing for standardizing text data having a certain unit,
A function of cutting out a plurality of words with grammatical information by analyzing the text data with reference to a dictionary stored in a storage device in advance by a computer,
A function for a computer to perform a predetermined standardization on the extracted word, and a function for outputting a document reconstructed from the standardized word,
The predetermined standardization function is:
(B) When the plurality of extracted words have the stored co-occurrence relationship by referring to the co-occurrence relationship stored in advance in the storage device, the positional relationship of adjacent positions of the plurality of words having the relationship Conjunction processing to be corrected to
(D) an independent word process that replaces an independent word including a plurality of adjacent words whose positional relationship has been corrected with another independent word according to a predetermined replacement criterion;
(E) A program that realizes an ancillary word processing for replacing an ancillary word with another ancillary word according to a predetermined rule by a computer.   A recording medium on which the program according to claim 15 is recorded.

Images