JP5955817B2 - Extraction apparatus, extraction method and program - Google Patents

Description

  The present invention relates to an extraction device, an extraction method, and a program.

  In recent years, with the spread of portable information devices such as smartphones, communication via social networking services (hereinafter referred to as SNS) has permeated people's daily lives and has exploded. In this SNS, the user can easily post and share his / her recent situation and feeling through the portable information device. Therefore, since the general public's opinions can be obtained in a timely manner from the SNS, the use for marketing measures is attracting attention.

  For example, in SNS, an API (Application Programming Interface) for obtaining posted data is disclosed, and analysis and prediction using data posted in SNS is starting to be performed. Since data posted by this SNS is written by a large number of users, the total amount of data is enormous, and the content is a set of time-series data (hereinafter referred to as stream data) that differs depending on the user to be written. . The amount of stream data and the content of the stream data (keywords (words) included in the stream data) vary depending on the life cycle of the user, general event topics, etc.

  An extraction method (see Patent Document 1) for extracting a keyword with a high degree of attention by calculating / detecting the time transition of the frequency of appearance of keywords (words) included in these enormous stream data and the time series change width, There is an extraction method for extracting potential keywords (see Patent Document 2).

JP 2005-316899 A JP 2008-152634 A

  However, in the conventional extraction method described above, even if there is a keyword that has a strong relationship within a specific period due to some trigger (factor) even if it is irrelevant in the long term, that keyword is regarded as a potential keyword. Also could not be extracted. In addition, due to the extraction of such keywords, the analysis accuracy in marketing measures may be reduced.

  The present invention has been made in view of the above, and an object of the present invention is to provide an extraction device, an extraction method, and a program capable of accurately extracting relevant keywords from stream data.

  In order to solve the above-described problem and achieve the object, the extraction device according to the embodiment totals the appearance frequency of the word appearing in the input stream data for each predetermined time for each predetermined word. A totaling unit, an appearance frequency recording unit that accumulates the frequency of appearance of each of the aggregated words for each predetermined time, and records the frequency of appearance of each word up to the present, and the occurrence of each of the aggregated words for every predetermined time A burst detection unit for detecting a burst of the word at the predetermined time based on the frequency and the frequency of appearance of each recorded word up to the present; and for each word at which the burst is detected, at the predetermined time Receiving a burst feature amount information recording unit that stores burst feature amount information including the appearance frequency, a period as an extraction condition, and a reference word; A related word extraction unit that normalizes the appearance frequency of each word in the first feature amount information and extracts a word whose appearance frequency after normalization is similar to the appearance frequency after normalization of the reference word as a related word It is characterized by.

  According to the extraction device according to the embodiment, there is an effect that it is possible to accurately extract related words from stream data.

FIG. 1 is a block diagram illustrating a functional configuration of the extraction device according to the embodiment. FIG. 2 is a diagram for explaining the data structure of stream data. FIG. 3 is a diagram for explaining the data structure of dictionary information. FIG. 4 is a diagram for explaining the data structure of long-time frequency information. FIG. 5 is a diagram for explaining the data structure of the short-time frequency information. FIG. 6 is a diagram for explaining the data structure of burst feature information. FIG. 7 is a flowchart illustrating an example of the operation of the extraction device according to the embodiment. FIG. 8 is a flowchart illustrating an example of the operation of the extraction device according to the embodiment. FIG. 9 is a flowchart illustrating an example of the operation of the extraction device according to the embodiment. FIG. 10 is a diagram illustrating that the processing in the extraction device according to the embodiment is specifically realized using a computer.

  Hereinafter, an extraction apparatus, an extraction method, and a program according to embodiments will be described in detail with reference to the accompanying drawings. In the following description, common constituent elements are given common reference numerals, and redundant description is omitted.

  In the extraction device according to the embodiment described below, for the stream data that is a set of data that is continuously entered in SNS and the like, the appearance frequency of each word appearing in the stream data is tabulated, The latest long-term appearance frequency for each word is updated and recorded based on the counting result. And based on the appearance frequency for a long time and the appearance frequency for every short time, the burst of a word which is a phenomenon in which the appearance frequency of a predetermined word increases rapidly in the short time is detected. Then, burst feature information including the detected burst occurrence time and appearance frequency is recorded. Then, based on the burst feature amount information obtained by normalizing the appearance frequency within the period, and receiving the period as the extraction condition and the reference word, the relationship with the reference word (the appearance frequency after normalization is the reference word) And the related words are extracted.

  The “short time” described above may be a predetermined time set as an appropriate value in the apparatus, and is set as 10 minutes in the present embodiment. In addition, “long time” does not mean a time period of a certain period longer than “short time”, but means the total time of words recorded so far, which is the extraction time point at which extraction is performed. . The “period as the extraction condition” and “reference word” may be values set in advance in the memory of the device or values specified by the user via an operation device such as a keyboard or a mouse. .

  FIG. 1 is a block diagram illustrating a functional configuration of the extraction device according to the embodiment. As shown in FIG. 1, the extraction device 100 has a short-time frequency counting unit 101, a burst detection / determination unit 102, and a long-time frequency information record as a functional configuration realized by using a computer (details will be described later). Unit 103, long-time frequency calculation / update unit 104, burst feature amount information recording unit 105, and related word extraction unit 106.

  The short-time frequency totaling unit 101 totals the appearance frequencies of the words appearing in the input stream data 200 for each word predetermined in the dictionary information 300 every short time. The short-time frequency totaling unit 101 outputs the appearance frequency totaled for each word to the burst detection / determination unit 102 as short-time frequency information 500.

  The burst detection / determination unit 102 uses the short-time frequency information 500 tabulated by the short-time frequency tabulation unit 101 and the long-time frequency information 400 recorded in the long-time frequency information recording unit 103 to generate a word in a short time. Detect bursts. The burst detection / determination unit 102 outputs burst feature amount information 510 including the appearance frequency for each word in which a burst is detected to the burst feature amount information recording unit 105.

  The long-time frequency information recording unit 103 records the long-time frequency information 400 in which the appearance frequency (frequency information for each short time) of each summarized word is accumulated. The long-time frequency calculation / update unit 104 displays the appearance frequency of each word for a short time (frequency information for each short time) and the current long-time frequency information recorded by the long-time frequency information recording unit 103 up to the present time. 400, the appearance frequency of each word up to now is calculated, and the long-term frequency information 400 recorded by the long-term frequency information recording unit 103 is updated.

  The burst feature amount information recording unit 105 records the burst feature amount information 510 output from the burst detection / determination unit 102. The related word extraction unit 106 receives the extraction condition (period, reference word), and acquires all the burst feature amount information within the period from the burst feature amount information 510 recorded by the burst feature amount information recording unit 105. . Next, the related word extracting unit 106 normalizes the appearance frequency of each word in the burst feature amount information within the period, and uses a word whose appearance frequency after normalization is similar to the appearance frequency after normalization of the reference word as a related word. Extract.

  FIG. 2 is a diagram for explaining the data structure of the stream data 200. As shown in FIG. 2, the stream data 200 includes a document ID 201 uniquely assigned to identify a document, a document announcement period 202 that is an entry period in SNS and the like, and an entered document content 203. Have.

  FIG. 3 is a diagram for explaining the data structure of the dictionary information 300. As shown in FIG. 3, a large number of words 302 are registered in advance in the dictionary information 300. Specifically, the dictionary information 300 has a word 302 for each uniquely assigned word ID 301.

  FIG. 4 is a diagram for explaining the data structure of the long-time frequency information 400. As shown in FIG. 4, the long-term frequency information 400 is obtained by summing up the word 402, the average frequency 403 obtained by summing up appearance frequencies every short time, and the word 402 for each uniquely assigned word ID 401. And an observation time 404 indicating a period.

  FIG. 5 is a diagram for explaining the data structure of the short-time frequency information 500. As shown in FIG. 5, the short-time frequency information 500 includes a word 502, a period 503, an appearance frequency 504 that is an appearance frequency, and a normalized value 505 for each uniquely assigned word ID 501. The normalized value 505 is blank (N / A (Not Available)).

  FIG. 6 is a diagram for explaining the data structure of the burst feature information 510. As shown in FIG. 6, the burst feature quantity information 510 includes a word 512, a period 513, an appearance frequency 514, and a normalized value 515 for each uniquely assigned word ID 511. The normalized value 515 describes a value when normalized by the related word extraction unit 106.

  Here, operation | movement of the extraction apparatus 100 by the function structure mentioned above is demonstrated in detail with reference to FIGS. 7 to 9 are flowcharts illustrating an example of the operation of the extraction device according to the embodiment.

  More specifically, FIG. 7 is a flowchart showing calculation of word burst feature values in real time. FIG. 8 is a flowchart showing related word extraction processing based on extraction conditions (period, reference word). FIG. 9 is a flowchart showing normalization of the appearance frequency of each word in the burst feature information.

  As shown in FIG. 7, when the processing for calculating the burst feature amount of the word in real time is started, the short-time frequency counting unit 101 reads the stream data 200 (S11). Next, the short-time frequency counting unit 101 performs morphological analysis on the read stream data 200 and extracts a word list in the stream data 200 (S12).

  For example, in the example of FIG. 2, a word list such as “Tomorrow”, “Ring”, “Eclipse”, “Expectation” is extracted from the document content 203 with the document ID “0001”.

  In consideration of the time required for the processing illustrated in FIGS. 7 to 9, it is assumed that a framework (for example, Jubatus (registered trademark)) capable of distributed processing in real time is used.

  Next, the loop processing of S13 to S21 is performed for each word in the word list extracted in S12.

  When the loop process is started (S13), the short-time frequency counting unit 101 reads the dictionary information 300 and determines whether or not the word that is the processing target in the loop process exists in the dictionary in the word list. (S14). When the word does not exist in the dictionary (S14: NO), the short-time frequency counting unit 101 advances the process to S21 and shifts to a loop process for the next word.

  When the word is present in the dictionary (S14: YES), the short-time frequency counting unit 101 calculates the number of appearances of the word within the designated short time, the word ID of the word, the word, the designated short time, The calculated number of appearances is stored in the memory as short-time frequency information 500 (S15).

  In the present embodiment, when the short period is a partial period of 10 minutes and the document announcement period 202 of the stream data 200 matches the period 503 of the short time frequency information 500 of the word, the appearance frequency of the word in a short time is set. to add. For example, in the stream data 200 of the document ID “0001”, the document announcement period 202 is “2012/05/20 18:31:01 to 2012/05/20 18:40:00” (see FIG. 2). The number of appearances 504 of “2012/05/20 18:31:01 to 2012/05/20 18:40:00” of the word “eclipse” in the short time frequency information 500 of the word is increased by one. In this embodiment, the short-time value is set to 10 minutes, but this is an example of short-time specification, and an appropriate value can be selected by the application system.

  Next, the long-time frequency calculation / update unit 104 acquires the current long-time frequency information of the corresponding word from the long-time frequency information 400 recorded in the long-time frequency information recording unit 103 (S16). Specifically, when the word is “eclipse”, the average frequency 403 and the observation time 404 for “eclipse” of the word ID “0001” are acquired (see FIG. 4).

  Next, the long-time frequency calculation / update unit 104 uses the following formulas (1) and (2) based on the short-time appearance count calculated in S15 and the long-time frequency information acquired in S16. The latest long-term appearance frequency and the latest observation time (latest long-term frequency information) of the corresponding word are calculated (S17).

  Next, the long-term frequency calculation / update unit 104 outputs the latest long-term frequency information calculated in S17 to the long-term frequency information recording unit 103, and the long-term frequency information recording unit 103 sets the latest long-term frequency information long. The time frequency information 400 is stored (S18).

  Next, the burst detection / determination unit 102 detects a burst of words in a short time using the short-time appearance count and the current long-time appearance frequency (appearance count). Specifically, the burst detection / determination unit 102 determines whether or not the following expression (3) is satisfied (S19), thereby determining whether or not the short-time appearance count is a word burst. . If the expression (3) is satisfied and the number of appearances in a short time is a burst of words (S19: YES), the process proceeds to S20. When Expression (3) is not satisfied (S19: NO), the process proceeds to S21.

  Note that α in Expression (3) is “2” in the present embodiment, but an appropriate value can be selected by the application destination system.

  In S20, the burst detection / determination unit 102 outputs burst feature amount information 510 having the number of short-time appearances of the word determined as a burst as a burst feature amount to the burst feature amount information recording unit 105, and burst feature amount information The recording unit 105 stores the output burst feature amount information 510.

  As illustrated in FIG. 8, when the related word extraction process based on the extraction condition (period, reference word) is started, the related word extraction unit 106 uses the extraction device 100 user (another system or user) to extract the extraction condition. And a reference word are acquired (S31).

  In the present embodiment, it is assumed that the target period is “2012/05/19 00: 00: 00-2012 / 06/09 24:00:00”, the reference word is “sunglasses”, and the extraction condition is designated. Will be explained.

  Next, the related word extraction unit 106 acquires burst feature amount information (records) of all words within the corresponding period specified as the extraction condition from the burst feature amount information 510 of the burst feature amount information recording unit 105 (S32). . Next, the related word extraction unit 106 normalizes the burst feature value for each word within the corresponding period (S33).

  Specifically, as shown in FIG. 9, when the process is started in S33, the first loop process of S41 to S48 is performed for each word.

  When the first loop process is started (S41), the related word extraction unit 106 sets a record that is the minimum value of the period 513 from all the records of the corresponding word as a reference record (S42). For example, in the example of FIG. 6, the period designated as the period “2012/05/19 00:00:00 to 2012/06/09 24:00:00” for the word “eclipse” with the word ID “0001”. Since the minimum value is the first record, the first record of the burst feature information 510 is set as the reference record for the word “eclipse”. The reference record is similarly determined for “lunar eclipse”, “Venus”, “sunglasses”, and the like.

  Next, the related word extraction unit 106 sets the normalized value 515 of the reference record to “1” (S43). For example, for the word “eclipse”, the normalized value 515 of the first record of the burst feature information 510 is set to “1”.

  Next, the related word extraction unit 106 performs the second loop process of S44 to S47 for each record other than the reference record in order to normalize the number of appearances when the reference record is “1”.

  Specifically, the related word extraction unit 106 calculates a division of “the number of appearances of the current record / the number of appearances of the reference record” (S45). For example, in the example of FIG. 6, the number of appearances 514 of the second record of the burst feature information 510 is “250”, and the number of appearances 514 of the reference record (first) is “230”. The division is calculated by “250” of the current record and “230” of the reference record, and 250/230 = 1.087 is calculated.

  Next, the related word extraction unit 106 sets the calculation result of S45 to the normalized value 515 of the current record (S46). Therefore, in the above-described example, “1.087” is set to the normalized value 515 of the second record of the burst feature information 510.

  Returning to FIG. 8, after S <b> 33, the related word extraction unit 106 uses the normalized value of the burst feature value for each short time of each word for words other than the reference word (hereinafter, non-reference word), Based on the following formula (4), the similarity between the non-reference word and the reference word is calculated (S34).

  Specifically, as shown in Equation (4) described above, the variable k indicating the short time is set to 1 to n, the normalized value of the short time k of the non-reference word, and the normalization of the short time k of the reference word The values are compared in time series order to calculate the similarity of the time series transition between the non-reference word and the reference word.

  Next, the related word extraction unit 106 sorts words in descending order of similarity based on the similarity calculated in S34 (S35). Next, the related word extraction unit 106 returns the sorted words as extracted related words (S36). Specifically, from the sorted word list, a certain number (for example, 30) of words from the beginning are returned as related words that are related to the reference word within the period. In the present embodiment, the first 30 words of the sorted word list are set to be output as related words, but this is an example of designation, and an appropriate value can be selected by the application system. When the related word is returned to the system in S36, the returned related word is displayed on the system, and the user is notified.

  As described above, the extraction apparatus 100 adds up the appearance frequencies of the words appearing in the input stream data 200 for each predetermined time for each short time. And the appearance frequency for every short time of each totaled word is accumulate | stored, and the long time frequency information 400 until now in each word is recorded. Then, based on the frequency of appearance of each counted word for a short time and the frequency of occurrence of each recorded word up to the present, a burst of words in a short time is detected, and for each word for which a burst is detected The burst feature amount information 510 including the appearance frequency in a short time is recorded. Then, the period as the extraction condition and the reference word are accepted, the appearance frequency of each word in the burst feature amount information 510 within the accepted period is normalized, and the appearance frequency after normalization is after the normalization of the reference word Words similar in appearance frequency are extracted as related words.

  Therefore, the extraction device 100 can accurately extract keywords (words) related to the reference word from the extraction device 100. In addition, since the stream data 200 is a large amount of data, when all the time-series change information within the observed period is recorded for the words included in the stream data 200, the related words are found. Sometimes takes a long time and becomes a bottleneck in the time series analysis process. However, since the appearance frequency of words in a short time is recorded as the burst feature information 510, it is possible to efficiently find a potential related word.

  In addition, the extraction apparatus 100 extracts, as the related words, a word whose transition in the time series order of the appearance frequency after normalization is similar to the transition in the time series order of the appearance frequency after the normalization of the reference word. By analyzing the time series change rate of the appearance frequency of words included in 200, even if there are many words, there is no linguistic relevance among the temporarily related words Therefore, it is possible to effectively extract related words that are normally buried.

  For example, it is assumed that “sun eclipse”, “lunar eclipse”, and “passing the sun on Venus” are in one period such as three months as examples of general events and topics. In this case, with the conventional extraction method, “lunar eclipse” directly related to “eclipse” can be extracted as a related word. However, although not directly related, it was not possible to extract “Venus” related to “Venus's passage through the sun”, which is observed using “sunglasses” in the same way as “eclipse”, as a related word.

  On the other hand, in the above-described embodiment, the time series of occurrences of the occurrence number of “sun eclipse” observed using “sunglasses” and the appearance number of “Venus” observed similarly using “sunglasses”. Based on similar similarity, it is possible to extract “Venus” as a related word.

  FIG. 10 is a diagram illustrating that the processing in the extraction device 100 according to the embodiment is specifically realized using a computer. As illustrated in FIG. 10, the computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031. The disk drive interface 1040 is connected to the disk drive 1041. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to the display 1061, for example.

  The hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the above program is stored in, for example, the hard disk drive 1031 as a program module 1093 in which a command to be executed by the computer 1000 is described. For example, a program module 1093 for executing information processing similar to the functional configuration illustrated in FIG. 1 is stored in the hard disk drive 1031.

  In addition, setting data necessary for processing in the above-described embodiment is stored as program data 1094 in, for example, the memory 1010 or the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1031 to the RAM 1012 as necessary, and executes them.

  The program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1031, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive or the like. Alternatively, the program module 1093 and the program data 1094 are stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.) and read by the CPU 1020 via the network interface 1070. May be issued.

  DESCRIPTION OF SYMBOLS 100 ... Extraction apparatus, 101 ... Short time frequency totaling part, 102 ... Burst detection / determination part, 103 ... Long time frequency information recording part, 104 ... Long time frequency calculation / update part, 105 ... Burst feature-value information recording part, 106 ... related word extraction unit, 200 ... stream data, 201 ... document ID, 202 ... document announcement period, 203 ... document content, 300 ... dictionary information, 301, 401, 501, 511 ... word ID, 302, 402, 502, 512 ... Word, 400 ... Long-term frequency information, 403 ... Average frequency, 404 ... Observation time, 500 ... Short-time frequency information, 503, 513 ... Period, 504, 514 ... Number of appearances, 505, 515 ... Normalized value, 510 ... Burst feature information, 1000 ... computer.

Claims (5)

For each predetermined word, a counting unit that counts the appearance frequency of the word in the input stream data every predetermined time;
Appearance frequency recording unit for accumulating the appearance frequency of each of the aggregated words every predetermined time, and recording the appearance frequency of each word up to the present,
A burst detection unit that detects a burst of the word at the predetermined time based on the frequency of occurrence of each of the counted words at a predetermined time and the frequency of appearance of the recorded words up to the present;
A burst feature amount information recording unit that records burst feature amount information including an appearance frequency in the predetermined time for each word in which the burst is detected;
Accepting a period as an extraction condition and a reference word, normalizing the appearance frequency of each word in the burst feature information within the period, and the appearance frequency after normalization is the appearance frequency after normalization of the reference word A related word extraction unit that extracts words similar to
An extraction device comprising:   The related word extracting unit extracts, as related words, a word whose transition in time series order of appearance frequency after normalization is similar to transition in time series order of appearance frequency after normalization of the reference word The extraction device according to claim 1.   The extraction apparatus according to claim 1, wherein the related word extraction unit sorts and outputs the extracted related words based on a rank similar to the reference word. An extraction method executed by an extraction device,
For each predetermined word, a counting step of counting the frequency of appearance of the word in the input stream data every predetermined time;
Appearance frequency recording step of accumulating the appearance frequency of each of the counted words for each predetermined time, and recording the appearance frequency of each word up to the present,
A burst detection step of detecting a burst of the word at the predetermined time based on the frequency of occurrence of the totaled words for each predetermined time and the frequency of occurrence of the recorded words up to the present;
Burst feature amount information recording step for recording burst feature amount information including the appearance frequency in the predetermined time for each word in which the burst is detected;
Accepting a period as an extraction condition and a reference word, normalizing the appearance frequency of each word in the burst feature information within the period, and the appearance frequency after normalization is the appearance frequency after normalization of the reference word A related word extraction step of extracting a word similar to as a related word;
The extraction method characterized by including. In the computer of the extraction device,
For each predetermined word, a counting step of counting the frequency of appearance of the word in the input stream data every predetermined time;
Appearance frequency recording step of accumulating the appearance frequency of each of the counted words for each predetermined time, and recording the appearance frequency of each word up to the present,
A burst detection step of detecting a burst of the word at the predetermined time based on the frequency of occurrence of the totaled words for each predetermined time and the frequency of occurrence of the recorded words up to the present;
Burst feature amount information recording step for recording burst feature amount information including the appearance frequency in the predetermined time for each word in which the burst is detected;
Accepting a period as an extraction condition and a reference word, normalizing the appearance frequency of each word in the burst feature information within the period, and the appearance frequency after normalization is the appearance frequency after normalization of the reference word A related word extraction step of extracting a word similar to as a related word;
A program that executes

Images