JP6042789B2 - Profile word extraction device, profile word extraction method, and profile word extraction program - Google Patents

Description

  The present invention relates to the field of information extraction technology, and more particularly to a profile word extraction apparatus, method, and program for microblogging.

  A microblog represented by Twitter (registered trademark) is a type of blog that transmits information and communicates by posting a short document. A user using a microblog can post a self-introduction document (profile document) in a free description format. Words that are characteristically appearing in the profile document and that clearly represent the user's characteristics (for example, words such as Star Wars, Arashi, and Cat that often express the user's interest) are called profile words. A device that extracts profile words from microblog data is a profile word extraction device.

  Unlike social networking services (SNS) such as Facebook (registered trademark) that can describe a specific movie name, artist name, etc. for each dedicated input field for movies, music, etc., microblogging has no dedicated input field, How to write a profile sentence depends on the user. Therefore, it is difficult to automatically extract profile words on a rule basis.

  Therefore, in the prior art, for example, words that appear in a profile document are clustered using a topic model such as Late Dirichlet Allocation (see Non-Patent Document 1), and words having a high topic probability are extracted as profile words. A word having a low topic probability is a general-purpose word that appears in every topic, so it is determined that it is not appropriate as a profile word and is not extracted.

D. Blei, A .; Ng, and M.M. Jordan. “Lent Dirichlet Allocation”. Journal of Machine Learning Research, 3, January 2003, pp. 993-1022

  Since profile documents are in a free description format, in profile documents, in addition to profile words, unfavorable words (tweets, follow me, etc.) that appear in documents other than profile documents (posted documents, etc.) ) Is also included. If the profile word is extracted using only the profile document as an information source as in the prior art, it cannot be determined whether the word is such an undesirable word. For this reason, an unfavorable word may be erroneously extracted as a profile word, and the profile word extraction accuracy may be reduced.

  The present invention solves the above-described problems, and an object of the present invention is to provide a profile word extraction apparatus, method, and program capable of accurately extracting profile words.

  In order to solve the above problems, a profile word extraction apparatus according to the present invention includes a document database storing a plurality of documents, and a profile document having profile words representing user characteristics for each document stored in the document database. And at least a user's posted document or hash tag document, and information source dividing means for obtaining a profile document set for each user and obtaining at least a document set of either a posted document set or a hash tag document set. Each document acquired by the information source dividing means is morphologically analyzed and divided into word groups, the appearance frequency of each word is counted, profile document word appearance frequency information, and at least posted document word appearance frequency information or hash Word appearance frequency counting means for acquiring any of the tag document word appearance frequency information, Using each word appearance frequency information acquired by the word appearance frequency counting means, an odds ratio between the profile document word appearance frequency and at least one of the posted document word appearance frequency and the hash tag document word appearance frequency is calculated. Profile word extracting means for calculating and determining whether or not the odds ratio matches a set extraction condition and extracting a word determined to be matched as a profile word.

  According to the above configuration, it is possible to determine words that have appeared characteristically in the profile document by comparing the appearance frequencies of words in a plurality of information sources divided from the document. Accordingly, it is possible to prevent erroneous extraction of an inappropriate word as a profile word that appears in addition to the profile document, and it is possible to accurately extract the profile word.

  According to the present invention, it is possible to prevent erroneous extraction of an inappropriate word as a profile word that appears in addition to a profile document, and it is possible to accurately extract a profile word.

1 is a configuration diagram of a profile word extraction device according to an embodiment of the present invention. Explanatory drawing which shows the structure of the data used by each part of the apparatus of FIG. The flowchart of the process which each part of the apparatus of FIG. 1 performs. The example of the process of the profile word extraction part of the apparatus of FIG. 1 is represented, (a) is a graph which shows the frequency of the word which appeared in the profile document, (b) is a graph which shows the frequency of the word which appeared in the contribution document.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments. FIG. 1 shows a configuration diagram of an embodiment of the present invention. FIG. 1 is a profile word extraction device that receives a microblog database (DB) storing a plurality of microblog documents and outputs a set of profile words.

  In FIG. 1, reference numeral 101 denotes a microblog DB in which microblog documents are stored.

  102 divides each document stored in the microblog DB 101 into a profile document having a profile word representing a user characteristic, a user posted document, and a hash tag document, and a profile document set for each user. 103, an information source dividing unit serving as an information source dividing unit that acquires the posted document set 104 and the hash tag document set 105.

  106 morphologically analyzes each document of each document set 103, 104, 105 acquired by the information source dividing unit 102, divides it into word groups, counts the appearance frequency of each word, and generates profile document word appearance This is a word appearance frequency counting unit as word appearance frequency counting means for acquiring the frequency information 107, the posted document word appearance frequency information 108, and the hash tag document word appearance frequency information 109.

  110 is an odds ratio between the profile document word appearance frequency and the posted document word appearance frequency, the profile document word appearance frequency and the hash tag, using the word appearance frequency information 107 to 109 acquired by the word appearance frequency counting unit 106. As a profile word extraction unit that calculates an odds ratio with the document word appearance frequency, determines whether the odds ratio meets a set extraction condition, and extracts a word determined to be matched as a profile word set 111 Profile word extraction unit.

  The profile word extraction device of FIG. 1 is configured by a computer, for example, and includes hardware resources of a normal computer, such as ROM, RAM, CPU, input device, output device, communication interface, hard disk, recording medium, and driving device thereof. Yes.

  As a result of the cooperation between the hardware resource and the software resource (OS, application, etc.), the profile word extraction device, as shown in FIG. 1, the microblog DB 101, the information source dividing unit 102, the word appearance frequency counting unit 106, and The profile word extraction unit 110 is implemented.

  The microblog DB 101 is constructed in storage means / storage means such as a hard disk or RAM, and the profile word document set 103, the posted document set 104, the hash tag document set 105, and the word appearance frequency counter acquired by the information source dividing unit 102. The profile document word appearance frequency information 107, the posted document word appearance frequency information 108, and the hash tag document word appearance frequency information 109 acquired by the unit 106 are stored in a storage unit / storage unit such as a RAM.

  The profile word set 111 extracted by the profile word extraction unit 110 is stored in storage means / storage means such as a RAM, or is output to the outside.

  Here, FIG. 2 shows the data structure used in this embodiment. The microblog DB 101 stores microblog documents of a plurality of users (FIGS. 2A and 2B), and stores a plurality of microblog documents per user. One microblog document includes id, text, description, and hashtags. id is a unique identification number for identifying the user. The text is a document posted by the user, and is composed of a combination of a posted document and a hash tag (consisting of individual words constituting hashtags and # characters indicating hash tags). The description is a profile document. hashtags is a list of hash tags included in the text.

  Since there are a plurality of microblog documents per user, the profile document, the posted document, and the hash tag document processed by the information source dividing unit 102 are shown in FIGS. 2 (c), 2 (d), and 2 (e). It is obtained by concatenating profile documents, posted documents, and hash tags included in the microblog document by line feeds and grouping them into one document for each user. When there are a plurality of hash tags, each is connected by a half-width space as shown in FIG.

  Examples of word appearance frequency information of each document set processed by the word appearance frequency counter unit 106 are shown in FIGS. 2 (f), (g), and (h). Even if the same word appears multiple times, it is counted as 1 if it is a document of the same user. Note that the number of appearances may be counted (for example, ramen and lovers in the profile document word appearance frequency in FIG. 2F are counted as 2). In the former case, it is robust even in situations where there are unique users (for example, it is possible to suppress the influence of spammy users who describe the same word in large quantities), and in the latter case, high appearance Frequency words become more important.

  Next, the operation of the apparatus configured as described above will be described with reference to the flowchart of FIG.

  First, the information source dividing unit 103 executes the processes of the following steps s1-1 to s1-6.

  [Step s1-1] Steps s1-2 to s1-6 are repeated for each user (id) included in the input microblog DB 101.

  [S1-2] The microblog document set generated by the user with the id selected in step s1-1 is acquired from the microblog DB 101.

  [Step s1-3] Steps s1-4 to s1-5 are performed for each microblog document included in the microblog document set acquired in step s1-2.

  [Step s1-4] The microblog document selected in step s1-3 is processed. A description is extracted as a profile document, a post document is obtained by removing a hash tag from a text, a hash tag is extracted as a hash tag document, and individual hash tags are extracted by a single-byte space. The profile document, the posted document, and the hash tag document are concatenated with a line feed at each repetition of step s1-3 and stored so that there is one document for each user (for example, FIGS. 2C, 2D, and 2E). )). Since one profile document, one posted document, and one hash tag document exist for each user, when processing is completed for all users, the profile document set 103, posted document set 104, and hash tag document set 105 are stored.

  [Step s1-5] It is determined whether or not to repeat the process starting at step s1-3.

  [Step s1-6] The end of repetition starting at step s1-6 is determined.

  Next, the word appearance frequency counting unit 106 executes the processing of the following steps s2-1 to s2-7.

  [Step s2-1] Steps s2-2 to s2-7 are repeated for each profile document set 103, posted document set 104, and hash tag document set 105.

  [Step s2-2] Steps s2-3 to s2-6 are repeated for each document included in the target document set selected in step s2-1.

  [Step s2-3] The target document is subjected to morphological analysis, and the document is divided into words with parts of speech. A conventional product such as MeCab (http://mecab.googlecode.com/svn/trunk/mecab/doc/index.html) can be used as the morphological analyzer.

  [Step s2-4] From the word group with part of speech obtained in step s2-3, a word corresponding to the stop word defined in advance and a word not corresponding to the extraction target part of speech (noun, etc.) defined in advance exclude.

  [Step s2-5] The appearance frequency of words remaining after the processing of step s2-4 is counted. As described in the description of the data structure used in the present invention in FIG. 2, even if the same word appears multiple times, it may be counted as 1, or it may be counted as many times as it appeared. The counted information is stored as profile document word appearance frequency information 107, posted document word appearance frequency information 108, and hash tag document word appearance frequency information 109, respectively.

  [Step s2-6] It is determined whether or not to repeat the process starting at step s2-2.

  [Step s2-7] It is determined whether or not to repeat the process starting at step s2-1.

  Next, the profile word extraction part 110 performs the process of the following steps s3-1 to s3-7.

  [Step s3-1] The processing in steps s3-2 to s3-6 is repeated for each word counted in the profile document set.

[Step s3-2] Let the word selected in step s3-1 be w. The odds ratio OR _t is calculated using the profile document word appearance frequency information 107 and the posted document word appearance frequency information 108. The sum of the word frequencies that appear in the profile document is n _{p *} , and the word frequency of w is n _pw . Similarly, it is assumed that the total frequency of words appearing in the posted document is n _{t *} and the word frequency of w is n _tw . At this time, OR _t is calculated as in the following equation (1).

[Step s3-3] The odds ratio OR _h is calculated using the profile document word appearance frequency information 107 and the hash tag document word appearance frequency information 109. Let n _{h * be} the sum of the word frequencies that appear in the hashtag document, and n _{hw be} the word frequency of w. At this time, OR _h is calculated as follows.

[Step s3-4] It is determined whether or not the odds ratio meets a pre-defined extraction condition. First, it is confirmed whether or not the odds ratios OR _t and OR _h are both greater than the threshold before the pre-defined condition conformity judgment. Otherwise, the process proceeds to step s3-6 with the determination result No. If so, make a pre-defined condition compliance decision. Note that 1 is normally used as the threshold value, but other values may be used. In the pre-definition, it is assumed that at least one odds ratio is statistically significant, or all odds ratios are statistically significant. Whether the odds ratio is statistically significant is determined by a test in the 95% confidence interval of the odds ratio (Confidence interval). The confidence interval for the odds ratio OR _t is

  Ask for.

Similarly, the confidence interval for the odds ratio OR _h is

  Ask for.

  In addition, the value of q in the above formulas (3) and (4) is 1.96 in the 95% confidence interval (value read from the normal distribution table). When changing the test conditions such as 99% confidence interval, the value of q changes accordingly. For example, in the case of 99% confidence interval, it is 2.58. The test conditions are set in advance. When the confidence interval does not include 1 (when the confidence interval does not sandwich 1), the odds ratio is determined to be significant. If it is determined in advance that at least one odds ratio is statistically significant, if either one is significant, the process proceeds to step s3-5, and otherwise, the process proceeds to step s3-6. Proceed to If all the odds ratios are statistically significant in the predefined definition, if both are significant, the process proceeds to step s3-5 as Yes, otherwise proceeds to step s3-6.

  FIG. 4 shows an example of processing in step s3-4. In the example of FIG. 4, the word “student” is extracted as a profile word under a pre-defined condition that at least one odds ratio only needs to be statistically significant.

  In FIG. 4A, odds Odds (A, student) in which the word “student” appears in profile document A is Odds (A, student) = a / b = 100 / (1000-100) = 0.11. Odds Odds (A, Tweet) in which the word “tweet” appears in the profile document A is Odds (A, Tweet) = a / b = 50 / (1000-50) = 0.053 (a Is the appearance frequency of the word selected as a processing target in the profile document, and b is a value obtained by subtracting a from the sum of the word frequencies appearing in the profile document.

  In FIG. 4B, odds Odds (B, student) in which the word “student” appears in the posted document B is Odds (B, student) = c / d = 100 / (10000-100) = 0. Odds Odds (B, Tweet) in which the word “tweet” appears in the posted document B is Odds (B, Tweet) = c / d = 1000 / (10000−1000) = 0.11. c is the appearance frequency of the word selected as the processing target in the posted document, and d is a value obtained by subtracting c from the sum of the word frequencies appearing in the posted document.

  Therefore, the odds ratio Odds (A, student) / Odds (B, student) between the profile document word appearance frequency and the posted document word appearance frequency when the processing target word is “student” is 0.11 / 0. Since 01 = 11> 1, it may be unique because it is greater than 1.

  On the other hand, the odds ratio Odds (A, Tweet) / Odds (B, Tweet) between the profile document word appearance frequency and the posted document word appearance frequency when the processing target word is “tweet” is 0.053 / 0. .11 = 0.48 <1 and is not unique because it is smaller than 1.

  Then, for the word “student” having a unique possibility, the odds ratio test in the 95% confidence interval is performed using the same equation (5) as the equation (3) (q = 1.96, OR = 11). A = 100, b = 900, c = 100, d = 9900).

  The 95% confidence interval of the odds ratio OR obtained by the equation (5) is [8.29, 14.63], and is determined to be significant (conforms to the extraction condition) because it does not include 1 in the confidence interval. Is.

  [Step s3-5] The word w is extracted as a profile word.

  [Step s3-6] The end of repetition starting at step s3-1 is determined.

  [Step s3-7] The extracted all profile words are output as the profile word set 111.

  As described above, in the present embodiment example, information sources other than profile documents (posted documents and hash tag documents) are used to determine whether or not they appear characteristically in profile documents, and word appearance frequency information in each information source. It is judged by comparing. Thereby, it is possible to prevent erroneous extraction of an inappropriate word as a profile word appearing other than the profile document, and it is possible to accurately extract the profile word.

  In the present embodiment, three information sources are used, but two information sources using either one of a posted document and a hash tag document may be used. Three or more may be used. Furthermore, the information source to be added may be a document obtained from other media such as a newspaper or a Web page as well as a microblog. What is necessary is just to be able to compare the appearance frequency and co-occurrence information of words in the profile document and other documents.

  Further, the present invention can be realized by configuring a part or all of the functions of each means in the profile word extraction device of the present embodiment by a computer program and executing the program using the computer. It goes without saying that the procedure in the profile word extraction method of the above can be configured by a computer program, and the program can be executed by the computer, and the program for realizing the function by the computer can be read by the computer, For example, FD (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk) -ROM, DVD (Digital V rsatile Disk) -ROM, CD-R, CD-RW, HDD, and recorded in a removable disk, or stored, it is possible or distribute. It is also possible to provide the above program through a network such as the Internet or electronic mail.

101 ... Microblog DB
DESCRIPTION OF SYMBOLS 102 ... Information source division part 103 ... Profile document set 104 ... Post document collection 105 ... Hash tag document set 106 ... Word appearance frequency count part 107 ... Profile document word appearance frequency information 108 ... Post document word appearance frequency information 109 ... Hash tag document Word appearance frequency information 110 ... Profile word extraction unit 111 ... Profile word set

Claims (5)

A document database that stores multiple documents;
Each document stored in the document database is divided into a profile document having a profile word representing a user's characteristics and at least a user's posted document or hashtag document, and a profile document set for each user and at least a posted document Information source dividing means for acquiring either a document set or a hash tag document set;
Each document acquired by the information source dividing means is morphologically analyzed and divided into word groups, the appearance frequency of each word is counted, profile document word appearance frequency information, and at least posted document word appearance frequency information or hash Word appearance frequency counting means for acquiring any information of the tag document word appearance frequency information;
Using each word appearance frequency information acquired by the word appearance frequency counting means, an odds ratio between the profile document word appearance frequency and at least one of the posted document word appearance frequency and the hash tag document word appearance frequency Profile word extracting means for calculating whether or not the odds ratio meets a set extraction condition, and extracting a word determined to match as a profile word;
A profile word extraction device characterized by comprising:   The determination as to whether or not the extraction condition in the profile word extraction means is met is performed by determining whether or not the calculated odds ratio is statistically significant by testing a confidence interval based on a preset test condition. The profile word extracting device according to claim 1, wherein the profile word extracting device is implemented by performing the following. An information source dividing unit divides a plurality of documents stored in a document database into a profile document having a profile word representing a user characteristic and at least a user posted document or a hash tag document, and a profile for each user. An information source dividing step for obtaining the document set and at least one of the posted document set and the hash tag document set;
The word appearance frequency counting means morphologically analyzes each document acquired by the information source dividing means, divides it into word groups, counts the appearance frequency of each word, and profile document word appearance frequency information and at least a post A word appearance frequency counting step for obtaining any information of document word appearance frequency information or hash tag document word appearance frequency information;
The profile word extraction means uses each word appearance frequency information acquired by the word appearance frequency counting means, and the profile document word appearance frequency and at least one of the posted document word appearance frequency and the hash tag document word appearance frequency A profile word extraction step of calculating an odds ratio with the appearance frequency, determining whether or not the odds ratio meets a set extraction condition, and extracting a word determined to match as a profile word;
A profile word extraction method characterized by comprising:   The determination as to whether or not the extraction condition in the profile word extraction step is met is performed by determining whether or not the calculated odds ratio is statistically significant by testing a confidence interval based on a preset verification condition. The profile word extracting method according to claim 3, wherein the profile word extracting method is performed by performing the following.   The profile word extraction program which makes a computer function as each means of Claim 1 or 2.

Images