JP2011113098A - Turned letter correction program and method for correcting text containing turned letter, and text analysis server - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a turned letter correction program for properly correcting a turned letter, which is detected from analysis object text information, as a word of one morpheme containing the turned letter. <P>SOLUTION: The turned letter correction program allows a computer to function as: a reference text storage means for storing correction reference text information; a turned letter extracting means for extracting one character string containing a previously registered turned letter and then dividing the character string into morphemes; a retrieval key generating means for generating a retrieval key constituted of the turned letter and at least one of the adjacent morphemes before and after the turned letter; a correction rule retrieving means for retrieving one or more correction rules similar to the turned letter with the use of the retrieval key from the analysis object text information by using the reference text storage means; and a correction rule selecting means for selecting the correction rule based on a text constitution index among the correction rules. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a technique for detecting illegal / harmful information in text information described on a website published on the Internet.

  With the spread of the Internet, various texts are described on blogs, bulletin boards, or Web sites that publish reviews. A “blog” is a site that is generally run by an individual and can be updated in a diary to express their opinions on current news and specialized topics. A “bulletin board” is a site for sequentially exchanging discussions with other people on various themes. Furthermore, “review comments” refers to a site where comments about things such as people's rumors can be described. With the spread of these sites, general users can freely send information on the Internet.

  Along with this, it has become a social problem that illegal and harmful information such as texts that slander and slander individuals and organizations and crime notice texts are described on these Web sites. Therefore, it is necessary to detect individuals and Web sites that describe these illegal / harmful information while giving consideration to freedom of expression.

  In order to analyze the text content of such a Web document, at least morphological analysis is required. The morphological analysis program divides a sentence to be analyzed into morphemes. English sentences are divided into words such as "This is a pen." On the other hand, Japanese sentences are not separately written, and therefore processing by morphological analysis is required prior to syntax analysis or the like.

  A “morpheme” refers to the smallest meaningful unit among the constituent elements of a sentence. The morphological analysis program has a dictionary in which “part of speech” and “reading” are registered for each “word”. The divided morphemes are given “part of speech” and “reading” information using a dictionary, and the sentence is analyzed using grammatical rules that define the word arrangement. Thus, the analysis accuracy of the morphological analysis program depends on the dictionary. Therefore, a sentence including words that are not registered in the dictionary cannot obtain sufficient analysis accuracy.

On the other hand, a user who posts illegal / harmful information on a Web site may use “hidden characters” to avoid specifying a specific individual or corporation. For example, the hidden character is used as follows.
(1) “Kazu Ikeda ● said this”
(2) “K DI Research Team ● Book”

  In general, in the context where the person or corporation can be identified, the “hidden character” replaces some characters of the specific name of the person or corporation with a symbol such as “●” or “○” Is. As a result, a viewer in a specific field can understand the character replaced by the hidden character and avoid direct search. For users who post illegal / harmful information, it is possible to avoid the responsibility of sending information.

Generally, a character string including these hidden characters is not registered in the dictionary of the morphological analysis program. Therefore, the accuracy of morphological analysis is reduced for the following reason.
-The entire character string including the hidden character is analyzed as an "unknown word".
-The hidden character part is independently analyzed as "symbol" or "unknown word".
-Character strings before and after the hidden character are separated from the hidden character and analyzed incorrectly.
As a result, there has been a problem that text information including hidden characters cannot be detected as illegal or harmful information.

  Conventionally, there is a technique for correcting a sentence including an unknown word into a sentence suitable for morphological analysis in contrast to detecting a hidden character as an unknown word (see Non-Patent Document 1, for example). Web sites are often described in colloquial “squeezed expressions” because of their friendliness. Examples of “complex expression” include “Usso” and “Kaisui” (read as “cute”). Since these expressions are generally not registered in the dictionary of the morphological analysis program, they are processed as “unknown words” and the morphological analysis fails.

  According to this technique, a morphological analysis is performed on an analysis target sentence that is a sentence including an unknown word, on the assumption that the analysis fails and is output as an “unknown word”. Based on the result of the morphological analysis, a search key is generated by replacing the unknown word part with an arbitrary character string. A correction candidate character string is automatically searched from the correction reference text by using the generated search key, and one or more correction rules are generated from the correction candidate character string. The generated correction rules are either (1) an expression that is frequently used in the same context, (2) whether it has changed too much from the expression of the sentence to be analyzed, or (3) the corrected sentence is in Japanese Scoring is performed using an index such as whether natural or not, and an optimal correction rule is selected. The analysis target sentence is corrected to a sentence suitable for morphological analysis according to the selected correction rule.

Kazufumi Ikeda, Tadashi Yanagihara, Kazunori Matsumoto, Yasuhiro Takishima, “Proposal and Evaluation of Unsupervised Learning Method for Correcting Expressions”, 8th Information Science and Technology Forum, 2009

  In general, a hidden character replaces some characters in a word with a symbol such as “●” or “◯”, so the portion of the hidden character itself does not become an independent morpheme. However, according to the technique described in Non-Patent Document 1, an attempt is made to correct a hidden character detected as an unknown word as an independent morpheme. For this reason, even if the face-down character is corrected to a correct character for morphological analysis, there is a high possibility that the correction itself is incorrect.

  Therefore, an object of the present invention is to provide a hidden character correction program, a method, and a sentence analysis server that can correctly correct a hidden character detected from analysis target sentence information as a morpheme word including the hidden character. To do.

According to the present invention, for a sentence information to be analyzed including a hidden character, a hidden character correction program for causing a computer to function to correct the hidden character,
Reference sentence storage means for storing corrected reference sentence information;
A hidden character extracting means for extracting one character string including a previously registered hidden character and dividing the character string into morphemes;
Search key generating means for generating a search key comprising a hidden character and at least one of the adjacent morphemes in front of and behind the hidden character;
A correction rule search means for searching for one or more correction rules that approximate a hidden character by using a search key from the analysis target sentence information using the reference sentence storage means;
The computer is caused to function as a correction rule selecting means for selecting a correction rule from correction rules based on a sentence composition index.

According to another embodiment of the program for correcting overlaid characters of the present invention,
Whether or not a corrected character string to which a correction rule is applied is temporarily generated, the corrected character string is divided into morphemes, and a character corresponding to the cover character is connected to at least one of the front and rear characters to form one morpheme Correction rule determination means for determining whether or not
In the case of connection, it is preferable that the computer is further functioned as correction rule application means for applying the corrected character string to the analysis target sentence information, and in the case of not connecting, applying the uncorrected character string to the analysis target sentence information.

According to another embodiment of the program for correcting overlaid characters of the present invention,
Preferably, the search key generation means further causes the computer to function to generate a search key using the hidden character as a regular expression.

According to another embodiment of the program for correcting overlaid characters of the present invention,
When multiple hidden characters are extracted from one character string by the hidden character extraction means,
Preferably, the search key generation means further causes the computer to function as a single search key when a plurality of hidden characters are close to each other with a predetermined number of characters or less.

According to another embodiment of the program for correcting overlaid characters of the present invention,
It is also preferred that the computer further function so that the text information to be analyzed is text information described by an unspecified number of users in a blog (Weblog), bulletin board, and / or word-of-mouth comment published via the network.

According to another embodiment of the program for correcting overlaid characters of the present invention,
The correction rule selection means includes (1) the appearance frequency in the correction rule, (2) the edit distance between the cover letter and the correction morpheme based on the correction rule, and / or (3) before and after correction as a sentence composition index. It is also preferable to further cause the computer to function so as to select one correction rule based on the difference between the morphological analysis cost value and the later one.

According to another embodiment of the program for correcting overlaid characters of the present invention,
It is also preferable that the computer further function so as to further include illegal harmful information detecting means for determining whether or not to include the illegal / harmful keyword registered in advance with respect to the analysis target sentence information in which the hidden character is corrected.

According to the present invention, it is a sentence analysis server that acquires analysis target sentence information including a hidden character from another public server via a network, and corrects the hidden character,
Reference sentence storage means for storing corrected reference sentence information;
A hidden character extracting means for extracting one character string including a previously registered hidden character and dividing the sentence into morphemes;
Search key generating means for generating a search key comprising a hidden character and at least one of the adjacent morphemes in front of and behind the hidden character;
A correction rule search means for searching for one or more correction rules that approximate a hidden character by using a search key from the analysis target sentence information using the reference sentence storage means;
And a correction rule selection means for selecting a correction rule based on a sentence composition index from among the correction rules.

According to another embodiment of the sentence analysis server of the present invention,
Whether or not a corrected character string to which a correction rule is applied is temporarily generated, the corrected character string is divided into morphemes, and a character corresponding to the cover character is connected to at least one of the front and rear characters to form one morpheme Correction rule determination means for determining whether or not
It is preferable to further include correction rule application means for applying the corrected character string to the analysis target sentence information when connecting, and applying the pre-correction character string to the analysis target sentence information when not connecting.

  According to the program for correcting hidden characters, the method, and the sentence analysis server of the present invention, the hidden characters detected from the analysis target sentence information can be corrected correctly as a morpheme word including the hidden characters. This increases the accuracy of morphological analysis even for text information including hidden characters, and increases the accuracy of detecting illegal / harmful information including hidden characters.

It is a functional block diagram of the hidden character correction program in this invention. It is explanatory drawing of the correction rule determination and correction rule application in this invention. It is a system configuration | structure figure of the text analysis server in this invention. It is a sequence diagram of a system in the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a functional configuration diagram of the program for correcting a hidden character in the present invention.

  According to FIG. 1, the hidden character correction program 1 includes a reference sentence storage unit 11, a hidden character storage unit 12, a hidden character extraction unit 13, a search key generation unit 14, a correction rule search unit 15, and a correction rule selection unit 17. And a correction rule determination unit 16 and a correction rule application unit 18. These functional units other than the reference text storage unit 11 can be realized by executing a program that causes a computer installed in the apparatus to function. The processing flow of each functional unit can be executed as a hidden character correction method using a computer.

  The reference sentence storage unit 11 stores a set of corrected reference sentence information. For example, a sentence with high analysis accuracy of morphological analysis such as a sentence of a newspaper article is used as the correction reference sentence.

  The hidden character storage unit 12 registers and stores the hidden character to be corrected in advance. For example, there are “●”, “◯”, “■”, “□”, or “×” as the face-down character. The hidden character storage unit 12 is referred to by the hidden character extraction unit 13.

  The hidden character extraction unit 13 inputs analysis target sentence information. The analysis target sentence information may be sentence information described by an unspecified number of users such as a blog, a bulletin board, or a word-of-mouth comment published on a website. Then, the hidden character extraction unit 13 searches for the hidden characters registered in advance in the hidden character storage unit 12 from the analysis target sentence information. If the analysis target sentence includes a hidden character, the hidden character extraction unit 13 extracts one character string including the previously registered hidden character from the analysis target sentence information. The character string including the hidden character may be one sentence or one sentence including the hidden character, or may be a combination of the character string including the hidden character and a certain length before and after the character.

Next, the hidden character extraction unit 13 divides the character string including the extracted hidden characters into morphemes by morphological analysis. Here, a character string including the face-down character “Kazu Ikeda ●” will be described as an example.
Character string including hidden characters: Kazu Ikeda ● is a morphological analysis result: Ikeda / Kazu / ● (unknown word or symbol) / is a hidden character is generally not registered in the dictionary for morphological analysis, so “●” is an unknown word It is processed. However, if the concealed character is registered in the morphological analysis dictionary as “symbol”, “●” is analyzed as “symbol”.

  The hidden character extraction unit 13 outputs a character string including the hidden character whose form has been analyzed to the search key generation unit 14. Further, the hidden character extraction unit 13 outputs the analysis target sentence information in which the hidden character is detected to the correction rule application unit 18.

The search key generation unit 14 inputs a character string including a hidden character divided for each morpheme. The search key generation unit 14 generates a “search key” composed of a hidden character and at least one of the adjacent morphemes in front of and behind the hidden character. Here, the explanation will be made assuming that the concealed character, the front two morphemes adjacent thereto, and the rear one morpheme are used for generating the search key.
Input character string: Ikeda / Wa / ● / is a search key: Ikeda Kazu * (where "*" indicates one or more arbitrary character strings)

  The search key generation unit 14 generates a search key that combines the regular expression and the character string adjacent to the hidden character, using the hidden character as a regular expression (for example, “*” representing a wild card). Of course, the regular expression may be one that matches any single character (for example, “?” That represents any single character in the regular expression).

If a single character string extracted by the hidden character extraction unit 13 includes a plurality of hidden characters, the search key generation unit 14 can display a plurality of regular expressions if these hidden characters are close to each other with a predetermined number of characters or less. Generate a search key containing. Here, a character string including a hidden character “K • DI Institute Kazu Ikeda •” will be described as an example.
Input character string: K / ● / DI / Laboratory / Ikeda / Wa / ● / is a search key: K * DI Laboratories Ikeda * (where “*” indicates one or more arbitrary character strings). )

  Moreover, the adjacent morpheme should just have at least one of the front and back with respect to a hidden character. Of course, two or more adjacent morphemes may be continuous. Further, two or more concealed characters may be continuous. The search key generation unit 14 outputs the generated search key to the correction rule search unit 15.

  The correction rule search unit 15 inputs a search key. The correction rule search unit 15 searches the reference sentence storage unit 11 for a correction candidate character string including the “search key”. Then, the correction rule search unit 15 determines that the part corresponding to the regular expression in the extracted correction candidate character string is a part that approximates the hidden character, and extracts it as a correction rule. The correction rule refers to a rule for character string conversion from a character string including a hidden character (for example, “Kazushi Ikeda”) to a correction candidate character string (for example, “Kazufumi Ikeda”).

For example, the correction rule search unit 15 obtains the following correction candidate character strings by searching. The correction rule search unit 15 extracts one or more correction rules from the portion corresponding to the regular expression in the extracted correction candidate character string. Here, an explanation will be given by taking as an example a character string that includes the face-down character “Kazu Ikeda ● said.”
Search key: Ikeda Kazu * (where "*" indicates one or more arbitrary character strings)
Correction candidate string: Kazu Ikeda / History / Ha
Kazu Ikeda / Utako / Ha
Kazu Ikeda / About / Ha
Kazu Ikeda / to / has revised rule: ● ⇒ History
● ⇒Singer
● ⇒
● ⇒ and

  Two or more correction rules may be obtained by a search using a search key. The correction rule search unit 15 outputs all the correction rules obtained by the search using the search key to the correction rule determination unit 16.

  The correction rule determination unit 16 temporarily generates a corrected character string to which all correction rules are applied, morphologically analyzes the corrected character string, and the character corresponding to the cover character is connected to at least one of the front and rear characters It is then determined whether or not one morpheme is configured. If there are one or more correction rules that are connected to form one morpheme, the correction rule determination unit 16 outputs the correction rules to the correction rule selection unit 17. On the other hand, if there is no correction rule that is connected to form one morpheme, the correction rule determination unit 16 may output the pre-correction character string to the correction rule application unit 18.

  When there are two or more input correction rules, the correction rule selection unit 17 selects one correction rule suitable for the context based on the sentence composition index.

  The sentence composition index includes (1) the appearance frequency in the correction rule, (2) the edit distance between the hidden letter and the corrected morpheme based on the correction rule, and / or (3) the morphological analysis cost before and after the correction. An index calculated from the difference between values. The correction rule selection unit 17 selects one correction rule based on the sentence composition index.

For example, it is assumed that the correction rule selection unit 17 inputs the following correction rule.
Correction rule: ● ⇒ History
● ⇒Singer

(1) Appearance Frequency in Correction Rule The appearance frequency in the correction rule refers to the frequency at which the search result character string corresponding to the searched correction rule appears. The following table shows an example of scoring based on the appearance frequency of the search result character string.

  A character string having a high appearance frequency is considered to be an expression that is frequently used in a context similar to the context in which the hidden character appears, and is likely to be a correction candidate character string. On the other hand, an expression that is not frequently used in a similar context is highly likely not to be a correction candidate character string. Therefore, the correction rule having a high appearance frequency has a high score. The score may be independent of the number of searches by dividing the appearance frequency by the number of searches and normalizing the score.

(2) Edit distance between face-down characters and corrected morphemes based on correction rules Edit distance is an index that indicates how different two character strings are, and converts one character string into the other. Is given as the minimum number of insertions, deletions, and substitutions necessary to do. In many cases, a modified morpheme based on a modification rule is obtained by inserting, deleting, or replacing a small number of characters in an unknown word. For example, in the editing from “Forum” to “Farm”, the method of replacing “o” with “a” and deleting “La” is the minimum number of editing times, so the editing distance is 2 It is. The following table shows an example of scoring based on edit distance. A correction rule with a small editing distance has a high score.

(3) Difference in morpheme analysis cost value The morpheme analysis cost value is an index representing how probable the word break is among a plurality of word breaks. The morpheme analysis cost value is calculated from, for example, the appearance probability (occurrence cost) of a single word or the probability that a plurality of words appear continuously (concatenation cost). The morphological analysis cost value is used as an index for evaluating the fitness in the context of the correction rule.

  The morpheme analysis cost value of the entire sentence is calculated by accumulating the sum of the concatenation cost of each morpheme from the beginning of the sentence to the end of the sentence and the word occurrence cost (accumulated cost). When an expression having a high degree of fitness in context is generated by applying the correction rule, the occurrence cost and the concatenation cost around the expression are reduced, so the morphological analysis cost value of the entire sentence is reduced. On the other hand, when an expression with low fitness in the context is generated, the occurrence cost and the connection cost around the expression increase, and the morphological analysis cost value of the entire sentence increases.

  Here, a score based on the morphological analysis cost value is calculated from the difference between the morphological analysis cost value of the entire sentence after application of the correction rule and the morphological analysis cost value of the entire sentence before the correction. If the expression generated by the correction rule adapts to the context, the calculated score will be high.

The sentence composition index (score) includes (1) the appearance frequency (freq) in the correction rule, (2) the edit distance (dist) between the hidden letter and the correction morpheme based on the correction rule, and / or (3) the morpheme From the difference (cost) of the analysis cost value, for example, it is calculated by the following calculation formula.
score = α ・ freq + β ・ dist + γ ・ cost
Here, α, β, and γ are weighting functions, and optimum values can be calculated by applying and learning correction rules. The correction rule can set a threshold value to be applied. When the threshold is set low, the number of correction rules to be applied increases, but the erroneous application of the correction rules included therein also increases. On the other hand, when the threshold value is set high, the number of correction rules to be applied decreases, but the erroneous application of the correction rules included therein can also be reduced.

  The correction rule selection unit 17 selects a correction rule based on the sentence composition index. Of course, a plurality of correction rules may be selected. Further, the correction rule selection unit 17 may select a correction rule having a maximum score that is equal to or greater than a threshold value. The correction rule selection unit 17 outputs the selected one correction rule to the correction rule application unit 18.

  The correction rule application unit 18 inputs analysis target sentence information. When the correction rule is input from the correction rule selection unit 17, the correction rule application unit 18 applies the correction rule to the analysis target sentence information. On the other hand, when the determination result is input from the correction rule determination unit 16, the correction rule application unit 18 may apply the pre-correction character string to the analysis target sentence.

  The feature of the present invention is that an appropriate correction rule is applied to the analysis target sentence. Therefore, an embodiment in the correction rule determination unit 16 will be described with reference to FIG.

  FIG. 2 is an explanatory diagram of correction rule determination in the present invention.

(S21) The correction rule determination unit 16 temporarily generates a corrected character string to which the correction rule is applied, and divides the corrected character string into morphemes. As a result of the morphological analysis, it is determined whether or not the character corresponding to the hidden character is connected to at least one of the front and rear characters to form one morpheme. Of course, the characters corresponding to the hidden characters may be connected to both the front and rear characters to form one morpheme.

(S22) Here, an explanation will be given by taking as an example a sentence to be analyzed that includes a hidden character such as “Kazu Ikeda ● said,”. For the analysis target sentence information, corrected character strings “Kazufumi Ikeda” and “To Kazu Ikeda” are temporarily generated.

In the example of “Kazufumi Ikeda”, the replaced character “History” is connected to the preceding character “Wa” to form one morpheme.
Analysis target sentence: Kazu Ikeda ● Had said: Temporarily generated corrected character string: Kazufumi Ikeda is a morphological analysis result: Ikeda / Kazufumi / ha In this case, the correction rule determination unit 16 selects a correction rule as a correction rule To the unit 17.

(S23) On the other hand, in the example of “Ikeda Kazu ni”, the replaced character “ni” is not concatenated with the front character “sum” or the rear character “ha”, but only with “ni”. Constitute one morpheme.
Analyzed text: Kazu Ikeda ● Had said: Temporarily generated corrected character string: To Kazu Ikeda Morphological analysis result: Ikeda / Kazu / Ni / ha In this case, the correction rule determination unit 16 sets the correction rule as a correction rule. The data is not output to the selection unit 17.

(S24) In the input correction rule, when there is no character corresponding to the hidden character connected to at least one of the front and rear characters to form one morpheme, the correction rule determination unit 16 corrects the determination result You may output to the rule application part 18. FIG. The correction rule application unit 18 may apply the pre-correction character string to the analysis target sentence. In this case, the sentence to be analyzed is not amended as “Kazu Ikeda has said this”.

  FIG. 3 is a system configuration diagram of the sentence analysis server in the present invention.

  According to FIG. 3, the sentence analysis server 2 includes a communication interface unit 20, an analysis target sentence input unit 21, and a hidden character correction function unit 22. The sentence analysis server 2 connects to the Internet via the communication interface unit 20.

  Moreover, according to FIG. 3, the text analysis server 2 can communicate with the Web server 3 via the Internet. The Web server 3 is connected from the poster terminal 4 for contributors.

  The Web server 3 publishes Web documents such as blog text and word-of-mouth comments, which are analysis target sentences, received from the poster terminal 4. The sentence analysis server 2 acquires the Web document as an analysis target sentence from the Web server 3 via the Internet.

  The analysis target sentence input unit 21 receives the analysis target sentence via the communication interface unit 20. The analysis target sentence is output to the covert letter correction function unit 22.

  The hidden character correction function unit 22 is exactly the same as the function configuration unit described above with reference to FIG. The hidden character correction function unit 22 inputs an analysis target sentence from the analysis target sentence input unit 21 and outputs a corrected sentence.

  FIG. 4 is a sequence diagram of the system according to the present invention.

(S401) The contributor terminal 4 posts the blog text, which is the analysis target sentence, to the Web server 3.
(S402) The sentence analysis server 2 receives the analysis target sentence information (“Kazu Ikeda ● said this”) from the Web server 3. The analysis target sentence is output to the covert letter correction function unit 22.
(S403) The hidden character correction function unit 22 searches the previously registered hidden character from the analysis target sentence information. When the analysis target sentence includes a hidden character, the hidden character correction function unit 22 extracts one character string including the previously registered hidden character from the analysis target sentence information. The hidden character correction function unit 22 further divides the character string into morphemes by morphological analysis.
(S404) The hidden character correcting function unit 22 generates a search key including the hidden character extracted in S403 and at least one of the adjacent morphemes in front and rear of the hidden character.
(S405) The hidden character correction function unit 22 searches the correction reference text information for one or more correction rules that approximate the hidden character using a search key.
(S406) The covert character correcting function unit 22 temporarily generates a corrected character string to which the correction rule is applied, divides the corrected character string into morphemes, and the character corresponding to the covert character is at least one of the front and rear characters To determine whether one morpheme is configured.
(S407) The hidden character correction function unit 22 selects one correction rule based on the sentence composition index from among the correction rules determined to be connected in S406 to form one morpheme.
(S408) The covert-letter correction function unit 22 applies the selected correction rule to the analysis target sentence information, and outputs the corrected sentence (“Kazufumi Ikeda said this”).

  As described above in detail, according to the program for correcting hidden characters, the method, and the sentence analysis server of the present invention, the hidden characters detected in the analysis target sentence information are correctly corrected as a morpheme word including the hidden characters. be able to. This increases the accuracy of morphological analysis even for text information including hidden characters, and increases the accuracy of detecting illegal / harmful information including hidden characters.

  The present invention is also applied to a filtering technique for preventing access to a website describing illegal / harmful information detected by the present invention. Filtering refers to a technique for preventing access to a website when the website that the user is trying to access describes illegal or harmful information.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Fuzzy character correction program 11 Reference text memory | storage part 12 Fuzzy character memory | storage part 13 Fuzzy character extraction part 14 Search key generation part 15 Correction rule search part 16 Correction rule determination part 17 Correction rule selection part 18 Correction rule application part 2 Text analysis server 20 Communication interface part 21 analysis object sentence input part 22 obscure character correction function part 3 Web server 4 posting terminal

Claims (11)

A hidden character correction program for causing a computer to function to correct a hidden character for analysis target sentence information including the hidden character,
Reference sentence storage means for storing corrected reference sentence information;
A hidden character extracting means for extracting one character string including a previously registered hidden character and dividing the character string into morphemes;
Search key generating means for generating a search key comprising the hidden character and at least one of the adjacent morphemes in front of and behind the hidden character;
Correction rule search means for searching for one or more correction rules that approximate the hidden character using the search key from the analysis target sentence information using the reference sentence storage means;
A hidden letter correction program that causes a computer to function as correction rule selection means for selecting one correction rule based on a sentence composition index from one or more of the correction rules. For one or more correction rules searched by the correction rule search means, the corrected character string is divided into morphemes for each correction rule, and the character corresponding to the cover character is connected to at least one of the front and rear characters. And a correction rule determination means for determining whether or not to constitute one morpheme,
The correction rule selection means further causes the computer to select one correction rule based on the sentence composition index from one or more correction rules determined to be connected by the correction rule determination means. A program to fix up the character.   3. The program for correcting a hidden character according to claim 1, wherein the search key generating unit further causes the computer to generate a search key by using the hidden character as a regular expression of one character. When a plurality of hidden characters are extracted from one character string by the hidden character extracting means,
4. The program for correcting a hidden character according to claim 3, wherein the search key generating unit further causes the computer to function as one search key when a plurality of the hidden characters are close to each other with a predetermined number of characters or less. .   The analysis target text information further causes the computer to function as text information described by an unspecified number of users in a blog (Weblog), bulletin board, and / or word-of-mouth comment published via a network. The hidden letter correction program according to any one of claims 1 to 4.   The correction rule selection means includes (1) an appearance frequency in the correction rule, (2) an edit distance between the hidden character and a correction morpheme based on the correction rule, and / or (3) The hidden character according to any one of claims 1 to 5, wherein the computer further functions to select one correction rule based on a difference between morphological analysis cost values before and after the correction. Fix program.   The computer further functions so as to further include illegal harmful information detection means for determining whether or not to include the illegal / harmful keyword registered in advance for the analysis target sentence information in which the hidden character is corrected. The hidden character correction program according to any one of claims 1 to 6. A hidden character correction method that corrects a hidden character using a computer for analysis target sentence information including the hidden character,
It has a reference sentence storage unit that stores corrected reference sentence information,
A first step of extracting one character string including a pre-registered hidden character and dividing the character string into morphemes;
A second step of generating a search key comprising the hidden character and at least one of the adjacent morphemes in front of and behind the hidden character;
A third step of searching for one or more correction rules approximating the hidden character using the search key from the analysis target sentence information using the reference sentence storage unit;
And a fourth step of selecting one correction rule based on a sentence composition index from one or more of the correction rules. For one or more correction rules searched in the third step, the corrected character string is divided into morphemes for each correction rule, and the character corresponding to the cover character is connected to at least one of the front and rear characters. A determination step of determining whether or not to constitute one morpheme;
9. The fourth step according to claim 8, wherein one correction rule is selected based on the sentence composition index from one or more correction rules determined to be connected in the determination step. Hidden character correction method. A sentence analysis server that obtains analysis target sentence information including a hidden character from another public server via a network, and corrects the hidden character,
Reference sentence storage means for storing corrected reference sentence information;
A hidden character extracting means for extracting one character string including a previously registered hidden character and dividing the sentence into morphemes;
Search key generating means for generating a search key comprising the hidden character and at least one of the adjacent morphemes in front of and behind the hidden character;
Correction rule search means for searching for one or more correction rules that approximate the hidden character using the search key from the analysis target sentence information using the reference sentence storage means;
A sentence analysis server, comprising: a modification rule selection unit that selects one modification rule from one or more of the modification rules based on a sentence composition index. For one or more correction rules searched by the correction rule search means, the corrected character string is divided into morphemes for each correction rule, and the character corresponding to the cover character is connected to at least one of the front and rear characters. And a correction rule determination means for determining whether or not to constitute one morpheme,
The correction rule selection unit selects one correction rule based on the sentence composition index from one or more correction rules determined to be connected by the correction rule determination unit. The sentence analysis server described in.

Images