JP5704732B2 - Program particle complementation program, apparatus, server, and method for target sentence - Google Patents

Description

  The present invention relates to a technique for analyzing the contents of text information described on a website published on the Internet.

  With the spread of the Internet, various texts are described on blogs, bulletin boards, or Web sites that publish reviews. A “blog” is a site that is generally run by an individual and can be updated in a diary to express their opinions on current news and specialized topics. A “bulletin board” is a site for sequentially exchanging discussions with other people on various themes. Furthermore, “review comments” refers to a site where comments about things such as people's rumors can be described. With the spread of these sites, general users can freely send information on the Internet.

  By analyzing these text information in detail, opinions of an unspecified number of general users can be collected. For example, the present invention can be applied to reputation analysis regarding products and services and information retrieval for filtering illegal / harmful information. Such analysis of sentence information requires a technique for acquiring a relationship between morphemes using dependency analysis or case analysis. However, a sentence described by particle dropping or inversion in the sentence information may fail in dependency analysis or case analysis.

  As a conventional technique, there is a technique for statistically analyzing a tendency unique to a written word such as particle dropping or inversion and improving dependency analysis accuracy of a sentence having particle dropping or inversion (for example, see Non-Patent Document 1). This technique is determined based on a heuristic rule such as “a noun with a particle dropping is highly likely to be applied to the immediately following predicate”.

  There is also a technique for improving dependency analysis accuracy using a model based on ME (maximum entropy method) (see, for example, Non-Patent Document 2). Furthermore, there is a technique for improving dependency analysis accuracy by dividing a colloquial document into sentence units and classifying them into fine units called “sections” (see, for example, Non-Patent Document 3).

Mikio Yamamoto, Satoshi Kobayashi, Seiichi Nakagawa, “Analysis and Analysis of Particle Dropping and Inversion in Spoken Dialogue”, Transactions of Information Processing Society of Japan, vol.33 No.11, pp.1322-1330, 1992, [online] [August 24, 2010] Internet <URL: http://ci.nii.ac.jp/naid/110002723414> Kiyotaka Uchimoto, Satoshi Sekine, Hitoshi Isahara, “Japanese dependency analysis using a model based on the maximum entropy method”, IPSJ Journal 2001, [online], [August 24, 2010], Internet < URL: http://ci.nii.ac.jp/naid/110002725061> Masahiro Ohno, Shigeki Matsubara, Hideki Tsujioka, Yasuyoshi Inagaki, “Dependency analysis of monologue sentences based on the beginning detection of clauses”, IPSJ Transactions 2009, [online], [August 24, 2010], Internet <URL: http://ci.nii.ac.jp/naid/110006549568>

  However, sentence information written by an unspecified number of general users, such as electronic bulletin boards and blogs on the Internet, includes many colloquial expressions. For this reason, there is a problem that the accuracy of dependency analysis and case analysis decreases. Many of the causes of such a decrease in accuracy are based on particle dropping expressions such as “I ate ramen” (I ate ramen) and “I have fast feet” (I have fast feet).

  According to the technique described in Non-Patent Document 1, a noun with a particle dropping is determined to be related to the immediately following verb with a high probability. However, in the case of colloquial expressions, there are many sentences that are not particles, even if they are sentences in the form of “noun + verb”. Eventually, in the case of sentence information including colloquial expressions for an unspecified number of general users, the analysis accuracy of the sentence information is lowered.

  In addition, according to the technique described in Non-Patent Document 2, a learning document to which a dependency analysis result is assigned is required, so that manual labeling is required. Furthermore, according to the technique described in Non-Patent Document 3, it is considered that the effect of improving the analysis accuracy of a document including particle dropping is small.

  Therefore, the present invention detects a particle dropping presence / absence in the target sentence information and complements the dropped particle, thereby improving the analysis accuracy of the target sentence information, apparatus, server, and method The purpose is to provide.

According to the present invention, there is a particle complement program for causing a computer to function to complement a particle for target sentence information including a particle missing expression,
A reference sentence storage means for storing reference sentence information that does not include a particle omission expression;
A particle missing sentence generating means for generating particle missing sentence information by deleting a particle from the reference sentence information;
2 class pattern discriminators are constructed using particle missing sentence information as positive example data and reference sentence information as negative example data . Whether or not the input target sentence information is a particle missing expression using the pattern classifier And an identification engine means for extracting a plurality of particle candidate expressions corresponding to the particle dropping expression
Appearance frequency counting means that counts the appearance frequency in the reference sentence information stored in the reference sentence storage means for each candidate particle expression,
The computer is caused to function as a particle dropping complement means for complementing the particle in the expression with the particle having the highest appearance frequency with respect to the target sentence information .

According to another embodiment of the particle complement program of the present invention,
It is also preferred that the computer further functions so that the identification engine means is based on a Support Vector Machine or based on a rule base.

According to another embodiment of the particle complement program of the present invention,
The standard sentence information is publicly available and is sentence information described by a specific user who is trusted,
It is also preferable that the target text information is open to the public and that the computer further functions to be text information described by an unspecified number of users.

According to the present invention, for a target sentence information including a particle removal expression, a sentence analysis device that complements the particle,
A reference sentence storage means for storing reference sentence information that does not include a particle omission expression;
A particle missing sentence generating means for generating particle missing sentence information by deleting a particle from the reference sentence information;
2 class pattern discriminators are constructed using particle missing sentence information as positive example data and reference sentence information as negative example data . Whether or not the input target sentence information is a particle missing expression using the pattern classifier And an identification engine means for extracting a plurality of particle candidate expressions corresponding to the particle dropping expression
Appearance frequency counting means that counts the appearance frequency in the reference sentence information stored in the reference sentence storage means for each candidate particle expression,
A particle omission complementing means for complementing a particle in an expression with a particle having the highest appearance frequency with respect to target sentence information is provided.

According to the present invention, it is a sentence analysis server that acquires target sentence information including a particle removal expression from another public server via a network, and complements the particle,
A reference sentence storage means for storing reference sentence information that does not include a particle omission expression;
A particle missing sentence generating means for generating particle missing sentence information by deleting a particle from the reference sentence information;
2 class pattern discriminators are constructed using particle missing sentence information as positive example data and reference sentence information as negative example data . Whether or not the input target sentence information is a particle missing expression using the pattern classifier And an identification engine means for extracting a plurality of particle candidate expressions corresponding to the particle dropping expression
Appearance frequency counting means that counts the appearance frequency in the reference sentence information stored in the reference sentence storage means for each candidate particle expression,
A particle omission complementing means for complementing a particle in an expression with a particle having the highest appearance frequency with respect to target sentence information is provided.

According to the present invention, there is a particle omission complementing method for complementing the particle with respect to the target sentence information including the particle omission expression using an apparatus equipped with a computer,
It has a reference sentence storage unit that stores reference sentence information that does not include particle removal expressions,
A first step of generating particle missing sentence information by deleting particles from the reference sentence information;
A second step of constructing a two-class pattern discriminator, with the particle missing sentence information as positive example data and the reference sentence information as negative example data ;
A third step of identifying whether or not the input target sentence information is a particle dropping expression using a pattern discriminator, and extracting a plurality of particle-with-expression candidates corresponding to the particle dropping expression;
A fourth step of counting the frequency of appearance in the reference sentence information stored in the reference sentence storage unit for each candidate particle with expression;
A fifth step of complementing the particle in the expression with a particle having the highest appearance frequency with respect to the target sentence information .

  According to the particle omission completion program, the apparatus, the server, and the method of the present invention, it is possible to improve the analysis accuracy of the target sentence information by detecting the presence or absence of the particle in the target sentence information and complementing the dropped particle. it can. In particular, according to the present invention, since only existing newspaper documents can be used as target sentence information, the analysis accuracy is improved and the versatility is high.

It is a functional block diagram of the particle omission complementation program in this invention. It is a part of speech system diagram showing a part of speech and a part of speech fine classification. It is explanatory drawing which detects and complements a particle omission from sentence information. It is explanatory drawing which detects and complements particle omission from a part of speech sequence. It is a system block diagram which has a text analysis server in this invention. It is a sequence diagram in the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a functional configuration diagram of a particle dropping complement program according to the present invention.

  The particle complement program of the present invention causes the computer to function to complement the particle with respect to the target sentence information including the particle dropping expression. Here, the particle complement program is divided into a “particle missing detection function” and a “particle complement function”.

  According to FIG. 1, the particle missing complement program 1 includes a reference sentence accumulating unit 10, a particle missing sentence generating unit 11 and an identification engine unit 12 as a “particle missing detection function”.

  The reference sentence storage unit 10 stores reference sentence information that does not include a particle dropping expression. The “reference text information” is text information that is publicly disclosed and described by a specific user who is trusted. For example, it is preferably a set of sentences such as newspaper article sentences with few particles dropped and high morphological analysis accuracy.

  The particle missing sentence generation unit 11 generates particle missing sentence information by deleting the particle from the reference sentence information. The particle dropping sentence generation unit 11 divides the reference sentence information into morphemes, and deletes those whose morpheme part of speech becomes “particles”. A “morpheme” refers to the smallest meaningful unit among the constituent elements of a sentence. Further, a dictionary in which “part of speech” is registered for each “word” is required.

  The identification engine unit 12 generates a learning database using the particle missing sentence information as positive example data and the reference sentence information as negative example data. The identification engine unit 12 may be based on a support vector machine or a rule base (for example, C4.5).

  According to the identification engine of the support vector machine, it does not generate a clear rule and looks black box. Positive example data and negative example data are generated as support vectors. The “support vector machine” is an identification algorithm using supervised learning and is applied to pattern recognition. The support vector machine configures two classes (positive examples / negative examples) of pattern discriminators using linear input elements, and learns parameters of the linear input elements.

  Further, according to the rule-based identification engine, a clear rule is generated from positive example data and negative example data. “C4.5” is an algorithm for generating a decision tree for use in class classification, and is a statistical class classifier. This uses a concept of information entropy to generate a decision tree from a set of positive example data and negative example data.

  After generating the learning database, the identification engine unit 12 inputs target sentence information including a particle dropping expression. “Target sentence information” is privately disclosed and is sentence information described by an unspecified number of users. For example, it is text information described on an electronic bulletin board or blog on the Internet, and includes many colloquial expressions.

  The identification engine unit 12 specifies a particle dropping point from the input target sentence information using the learning database. Here, one or more candidate expressions with particles corresponding to the particle dropping expression are extracted. The expression with particles is not limited to one candidate, and a plurality of candidates may be extracted.

  Further, the “participant drop detection function” in the particle dropout supplement program of FIG. 1 may further include a part of speech extraction unit 15. The identification engine unit 12 can enhance the learning effect by learning and identifying only the part of speech string without learning and identifying the sentence information (word string) itself. Specifically, it can be expected that the storage capacity is small and the calculation amount is also small.

  FIG. 2 is an explanatory diagram showing a part of speech system in the part of speech extraction unit.

  The part-of-speech extraction unit 15 divides sentence information into morphemes and associates a part-of-speech system with each morpheme. That is, the morpheme string of sentence information is converted into a part of speech string. The “part of speech string” is composed of a plurality of part of speech strings. As shown in FIG. 2, “part of speech” is represented by the part of speech itself and one or more part of speech subcategories.

  According to FIG. 1, the particle missing sentence information of the positive example data output from the particle missing sentence generation unit 11 is converted into a particle missing part of speech string by the part of speech extraction unit 15. The particle part-of-speech string is input to the identification engine unit 12 as positive example data. In addition, the reference sentence information of the negative example data output from the reference sentence storage unit 10 is also converted into a part-of-speech string with a particle by the part-of-speech extraction unit 15. Then, the part-of-speech string with the particle is input to the identification engine unit 12 as negative example data. Thereby, the identification engine unit 12 can generate a learning database based on the part of speech string.

  Similarly, the part-of-speech extraction unit 15 also converts target sentence information to be input to the identification engine unit 12 into a part-of-speech string. Target sentence information based on the part-of-speech string is input to the identification engine unit 12. As a result, the identification engine unit 12 can extract a candidate part-of-speech string with a candidate using a learning database based on the part-of-speech string.

  According to FIG. 1, the “particle auxiliary function” further includes an appearance frequency counting unit 13 and a particle dropping complementing unit 14.

  The appearance frequency counting unit 13 counts the appearance frequency in the reference sentence information stored in the reference sentence storage unit 10 for each candidate particle expression. When the identification engine is based on a part of speech string, the appearance frequency in the reference sentence information stored in the reference sentence storage unit 10 is counted for each candidate part-of-speech part with a particle.

  The particle omission complementing unit 14 complements the target sentence information with the particle in the expression with the particle having the highest appearance frequency. Further, when the identification engine is based on a part-of-speech string, the particle in the part-of-speech part-with-participant string having the highest appearance frequency is supplemented to the target sentence information.

  FIG. 3 is an explanatory diagram showing complementation of particle dropping sentences based on sentence information.

(S31) It is assumed that the following two pieces of reference text information are stored in the reference text storage unit 10.
"I was eating ramen"
"I can't catch up because my feet are fast"
The reference sentence information is output as positive example data to the particle dropping sentence generation unit 11 and is output as negative example data to the identification engine unit 12.

(S32) The particle missing sentence generation unit 11 generates particle missing sentence information obtained by deleting the particle from the reference sentence information.
"I was eating ramen"->"I was eating ramen"
"Foot is fast and can't catch up"->"Foot is fast and can't catch up"
The generated particle missing sentence information is output to the identification engine unit 12 as positive example data.

(S33) The identification engine unit 12 generates a learning database from the reference sentence information of the positive example data and the reference sentence information of the negative example data.

(S34) It is assumed that the following three pieces of target sentence information are input to the identification engine unit 12.
"I ate ramen"

(S35) The identification engine unit 12 outputs a candidate particle existence expression. The expression with particles preferably corresponds to the following pattern, for example.
(Pattern 1) “Noun” + “Complementary particle” + “Verb”
(Pattern 2) “Noun” + “Complementary particle”
(Pattern 3) “Participant to complement” + “Verb”
For example, for “Ramen has been eaten”, the following expression with a particle is output to the appearance frequency counting unit 13.
"Eat ramen +"
"Ramen +"
"Eat +"
"Ramen + eat +"
"Ramen + is"
"Ga + eat"
...

(S36) The appearance frequency counting unit 13 uses the reference sentence storage unit 10 to count the appearance frequency of the candidate particles with expressions.
“Eat Ramen +”: 10 times “Ramen + Eat”: 50 times “Eat + Eat”: 100 times “Ramen + Eat +”: 0 times “Ramen + Eat”: 3 times “Eat + Eat”: 5 times
In addition, it is also preferable to weight each appearance frequency according to the pattern mentioned above. For example, pattern 1 is weighted more than patterns 2 and 3.

(S37) The particle dropping complement complementing unit 14 complements the target sentence information with the particle in the expression with the particle having the highest appearance frequency. For example, it is complemented as follows.
"I ate ramen"->"I ate ramen"

  FIG. 4 is an explanatory diagram showing complementation of particle dropping sentences based on the part of speech string.

(S41) The reference text information similar to S31 in FIG. 3 is output as positive example data to the particle dropping unit 11 and as negative example data to the part of speech extraction unit 15.

(S42) As in S32 of FIG. 3 described above, the particle missing sentence generation unit 11 generates particle missing sentence information in which the particle is deleted from the reference sentence information. The generated particle missing sentence information is output to the part of speech extraction unit 15 as positive example data.

(S43) The quality extraction unit 15 converts the particle missing sentence information input from the particle missing sentence generation unit 11 into a particle missing part of speech string. For example, it is converted as follows.
"I was eating ramen"
->"(Noun / general) (verb / independence)"
"I can't catch up with my feet fast"
->"(Noun / general) (adjective / independence) (verb / independence)"
Then, the particle part-of-speech part string is input to the identification engine unit 12 as positive example data.

(S44) The quality extraction unit 15 converts the reference sentence information input from the reference sentence storage unit 10 into a part-of-speech string with a particle. For example, it is converted as follows.
"I was eating ramen"
->"(Noun / general) + + (verb / independence)"
"I can't catch up because my feet are fast"
-> “(Noun / general) + ga + (adjective / independence) (verb / independence)”
The part-of-speech string with particles is input to the identification engine unit 12 as negative example data.

(S45) The identification engine unit 12 generates a learning database from the particle part-of-speech sequence of positive example data and the part-of-speech sequence with particle of negative example data.

(S46) It is assumed that the following three pieces of target sentence information are input to the identification engine unit 12.
"I ate ramen"

(S47) The identification engine unit 12 outputs a candidate part-of-speech string with a candidate particle.
For example, for “Ramen has been eaten”, the following part-of-speech string with particles is output to the appearance frequency counting unit 13.
“(Noun / general) +” + (verb / independence) ”
"(Noun / general) +"
"O + (verb, independence)"
“(Noun / general) + ga + (verb / independence)”
"(Noun / general) +"
"Ga + (verb, independence)"
...

(S48) The appearance frequency counting unit 13 counts the appearance frequency in the reference sentence storage unit 10 for candidate particle part-of-speech strings with candidates.
“(Noun / general) (particle / case particle in general) (verb / independence)” 1500 times “(noun / general) (particle / case particle in general)”: 900 times “(particle / case particle in general) (verb / "Independence)": 4500 times "(Noun / general) (particles / corresponding particles) (verb / independence)": 10 times "(Noun / general) (particles / corresponding particles)": 200 times (Verb / independence) ”: 350 times
In addition, the reference sentence storage unit 10 may store the part of speech sequence based on the reference sentence information in advance, or the particle part of speech with a particle output by the part of speech extraction unit 15 as negative example data may be stored in advance. It may be.

(S49) The particle dropping complementing unit 14 complements the target sentence information with the particles in the particle part-of-speech sequence having the highest appearance frequency. For example, it is complemented as follows.
"I ate ramen"->"I ate ramen"

  FIG. 5 is a system configuration diagram of the sentence analysis server according to the present invention.

  According to FIG. 5, the sentence analysis server 2 includes a communication interface unit, a particle dropping complement function unit, and a sentence content analysis unit. The sentence analysis server 2 is connected to the Internet via a communication interface unit.

  Moreover, according to FIG. 5, the text analysis server 2 can communicate with the Web server 3 via the Internet. The Web server 3 is connected from the poster terminal 4 for contributors.

  The Web server 3 publishes Web documents such as blog text and word-of-mouth comments, which are analysis target sentences, received from the poster terminal 4. The sentence analysis server 2 acquires the Web document as an analysis target sentence from the Web server 3 via the Internet.

  The particle omission complementation function unit receives the target sentence information via the communication interface unit. Complement the particle omission for the target sentence information. The target sentence information supplemented with the particle omission is output to the sentence content analysis unit. The text content analysis unit can analyze text content from various viewpoints and classify target text information into a specific category.

  FIG. 6 is a sequence diagram of the system according to the present invention.

(S601) The particle missing sentence information is generated by deleting the particle from the reference sentence information.
(S602) When the learning engine generates a part-of-speech string learning database, it converts the particle missing sentence information of the positive example data into a particle missing part-of-speech string and converts the reference sentence information of negative example data into a part-of-speech string with a particle.
(S603) The identification engine generates a learning database using the missing particle information as positive example data and the reference sentence information as negative example data.

(S611) The terminal 4 for contributors posts the blog text that is the target sentence information to the Web server 3. It is assumed that the target sentence information includes a particle dropping expression.
(S612) The sentence analysis server 2 receives the target sentence information (“I ate ramen”) from the Web server 3.

(S613) When the identification engine has generated a part-of-speech string learning database, the target sentence information is converted into a part-of-speech string.
(S614) The identification engine specifies a particle dropping expression of the target sentence information, and outputs a candidate particle-with expression.
(S615) The frequency of appearance is counted by using the reference sentence accumulating unit for the candidate particles with expressions.
(S616) The particle in the expression with the particle having the highest appearance frequency is supplemented with respect to the target sentence information.
(S617) The sentence content analysis process is executed based on the target sentence information supplemented with the particle missing expression.

  As described above in detail, according to the particle omission complementation program, the apparatus, the server, and the method of the present invention, the target sentence is detected by detecting the presence or absence of the particle omission in the object sentence information and complementing the dropped particle. Information analysis accuracy can be improved. In particular, according to the present invention, since only existing newspaper documents can be used as target sentence information, the analysis accuracy is improved and the versatility is high.

  In the present invention, it is possible to find and complement a particle dropping, which is a factor that lowers dependency analysis accuracy, with respect to a colloquial expression (particle dropping expression) frequently seen in sentences written by general users. . This corrects colloquial sentences into natural and easy-to-read sentences.

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Particle removal complement 10 Reference sentence storage part 11 Particle drop sentence generation part 12 Identification engine part 13 Appearance frequency counting part 14 Particle drop completion part 15 Part of speech extraction part 2 Text analysis server 3 Web server 4 Posting terminal

Claims (6)

A particle complement program for causing a computer to function to complement a particle for target sentence information including a particle dropping expression,
A reference sentence storage means for storing reference sentence information that does not include a particle omission expression;
From the reference sentence information, a particle missing sentence generating means for generating particle missing sentence information by deleting a particle,
The particle sentence sentence information is used as positive example data, the reference sentence information is used as negative example data, and a two-class pattern classifier is formed. An identification engine means for identifying whether or not the expression is present, and extracting a plurality of candidate particles with corresponding particles corresponding to the particle dropping expression ;
Appearance frequency counting means for counting the appearance frequency in the reference sentence information stored in the reference sentence storage means for each candidate expression with particles,
A particle complement program for causing a computer to function as a particle drop complement means for complementing the particle in the expression with a particle having the highest appearance frequency with respect to the target sentence information . The identification engine unit, based on the support vector machine (Support Vector Machine), or particle complementing program according to claim 1, characterized in that those based on rule-based, the computer to further function as a. The reference sentence information is publicly available and is sentence information described by a specific user who is trusted,
3. The particle complementing according to claim 1, wherein the target sentence information is publicly disclosed and further functions as a sentence information written by an unspecified number of users. program. A sentence analysis device that complements the particle with respect to target sentence information including a particle omission expression,
A reference sentence storage means for storing reference sentence information that does not include a particle omission expression;
From the reference sentence information, a particle missing sentence generating means for generating particle missing sentence information by deleting a particle,
The particle sentence sentence information is used as positive example data, the reference sentence information is used as negative example data, and a two-class pattern classifier is formed. An identification engine means for identifying whether or not the expression is present, and extracting a plurality of candidate particles with corresponding particles corresponding to the particle dropping expression ;
Appearance frequency counting means for counting the appearance frequency in the reference sentence information stored in the reference sentence storage means for each candidate expression with particles,
A sentence analysis apparatus comprising: a particle omission complementing means for complementing a particle in an expression with a particle having the highest appearance frequency with respect to the target sentence information . A sentence analysis server that obtains target sentence information including a particle omission expression from another public server via a network, and complements the particle;
A reference sentence storage means for storing reference sentence information that does not include a particle omission expression;
From the reference sentence information, a particle missing sentence generating means for generating particle missing sentence information by deleting a particle,
The particle sentence sentence information is used as positive example data, the reference sentence information is used as negative example data, and a two-class pattern classifier is formed. An identification engine means for identifying whether or not the expression is present, and extracting a plurality of candidate particles with corresponding particles corresponding to the particle dropping expression ;
Appearance frequency counting means for counting the appearance frequency in the reference sentence information stored in the reference sentence storage means for each candidate expression with particles,
A sentence analysis server , comprising: a particle omission complementing means for complementing a particle in an expression with a particle having the highest appearance frequency with respect to the target sentence information . Using a device equipped with a computer, for a target sentence information including a particle dropping expression, a particle dropping completion method for complementing the particle,
It has a reference sentence storage unit that stores reference sentence information that does not include particle removal expressions,
A first step of generating particle missing sentence information by deleting a particle from the reference sentence information;
A second step of configuring a two-class pattern discriminator with the particle missing sentence information as positive example data and the reference sentence information as negative example data ;
A third step of extracting a plurality of candidate particles with a particle corresponding to the particle dropping expression when the input target text information is identified as a particle dropping expression using the pattern classifier; ,
A fourth step of counting the frequency of appearance in the reference sentence information stored in the reference sentence storage unit for each candidate particle expression that is a candidate;
A particle omission complementing method comprising: a fifth step of complementing a particle in an expression with a particle having the highest appearance frequency with respect to the target sentence information .

Images