JP6325132B2 - Data collection device and data collection method - Google Patents

Description

  The present invention relates to a data collection device and a data collection method.

  Patent Document 1 states that “When a user selects an article for which a related article is to be searched, a search expression that dynamically matches the number of hits closest to the reference search number is automatically generated while dynamically changing the related article. It is displayed.

  Patent Document 2 discloses that “a keyword input accepting unit that accepts a keyword input by a user, a number of cases in which a search expression determined by the accepted user input keyword is established in a target document, A search result presenting means for presenting to a user, a related word generating means for generating a related word of the search expression presented by the search result presenting means, and a search expression including the presented keyword and the generated related word It has a search result prediction presenting means for obtaining the number of cases established in the document and presenting the number of cases and the generated related words to the user.

JP 2005-100136 A JP-A-5-314182

  In Patent Document 1, a search expression that dynamically matches the number of hits that is closest to the reference number of searches is dynamically changed. May not include information that the user is expecting. Further, in Patent Document 2, since the user selects a related word and designates a new keyword based on the presented information, the same problem as in Patent Document 1 may occur.

  An object of the present invention is to provide a data collection apparatus and a data collection method capable of accurately collecting information intended by a user when collecting data performed by specifying a search condition.

  One aspect of the present invention for achieving the above object is an information collection apparatus that collects information satisfying a search condition from a data group, and searches the data group using each of a plurality of search conditions, Based on the result of classifying the plurality of search conditions into a plurality of clusters according to the number of hits of each of the search conditions, the search condition candidates used for the search are specified.

  In addition, the subject which this application discloses, and its solution method are clarified by the column of the form for inventing, and drawing.

  According to the present invention, it is possible to accurately collect information intended by a user when collecting data performed by specifying a search condition.

1 is a diagram illustrating a schematic configuration of a data collection system 1. FIG. 3 is a diagram for explaining a concept of processing of the data collection device 10. FIG. It is an example of the hardware of the information processing apparatus 100 used for implementation | achievement of the data collection device 10 and the server apparatus 20. FIG. It is a data flow figure explaining the function with which data collection device 10 is provided, and the data which data collection device 10 manages. 3 is an example of collected data 402; It is an example of a setting search word 403. It is an example of a synonym dictionary 405. It is an example of an antonym dictionary 406. This is an example of hit count data 408. It is an example of the search condition data 411. It is a flowchart explaining search condition generation processing S1100. It is a flowchart explaining collection data acquisition processing S1200. It is a flowchart explaining search processing S1300. It is a flowchart explaining cluster generation processing S1400. It is a flowchart explaining exclusion condition determination processing S1500. It is an example of an exclusion condition setting screen 1600. It is an example of hit number data 408A in 2nd Example. It is a figure which shows the example of determination of the similarity which the cluster production | generation part 409 of 3rd Example performs.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, components having the same function and configuration may be denoted by the same reference numerals and redundant description may be omitted.

= First embodiment =
FIG. 1 is a diagram showing a schematic configuration of a data collection system 1 described as an embodiment. As shown in FIG. 1, the data collection system 1 includes a data collection device 10 and a server device 20. The data collection device 10 and the server device 20 are connected to be communicable via the communication network 5. The communication network 5 is, for example, the Internet or a dedicated line.

  The server device 20 is a device that provides information (data) to other devices accessed via the communication network 5 (for example, a Web server, an SNS server (SNS: Social Network Service), an open data server, etc.) Function as.

  The data collection device 10 accesses the server device 20 via the communication network 5 and acquires data from the server device 20. The data acquired by the data collection device 10 is useful for, for example, trend analysis and causal relationship analysis regarding a specific topic.

  The data collection device 10 collects information (hit information) that satisfies the search condition (search formula) from the data group acquired from the server device 20. In the above collection, the data collection device 10 searches the data group using a plurality of search conditions, and based on the result of classifying the plurality of search conditions into a plurality of clusters according to the number of hits of each of the plurality of search conditions, A search condition candidate used for searching the data group is specified.

  As shown in FIG. 2, the data collection device 10 excludes, from the search condition candidates, search conditions belonging to a cluster whose average value is not included in a preset range in the above specification. For example, the data collection device 10 excludes the search condition (h, i, j) belonging to a cluster (cluster C in the figure) whose average value exceeds a preset range from the candidate targets.

  As described above, the data collection apparatus 10 classifies the search conditions into a plurality of clusters and specifies search condition candidates in units of clusters. Therefore, it is possible to effectively exclude search conditions belonging to clusters with a very large number of hits. It is possible to specify a search condition that can obtain a search result that does not include information that is not intended by the user (hereinafter also referred to as noise). Note that search conditions belonging to clusters with a very large number of hits are excluded from candidates because, for example, keywords with ambiguousness are included in the search conditions and a large amount of information not intended by the user is included. To do.

  As shown in FIG. 2, the data collection device 10 uses the related word dictionary (synonym dictionary, synonym dictionary, materialization) for the plurality of search conditions, for example, related words of words included in a certain search condition. It is generated by searching from a dictionary etc. and replacing it. Therefore, it is possible to efficiently generate a search condition that is a selection target of search condition candidates used for the search without imposing a burden on the user.

  The data collection device 10 classifies the plurality of clusters of the search condition by, for example, the k-means method. In addition, the data collection device 10 includes a user interface that presents and specifies a search condition specified as a search condition candidate used for search, and accepts specification of the search condition used for search from the user. Therefore, the user can finally determine the search condition suitable for efficiently acquiring the target information.

  FIG. 3 is an example of hardware of the information processing apparatus 100 used for realizing the data collection apparatus 10 and the server apparatus 20. As illustrated in FIG. 1, the information processing apparatus 100 includes a processor 101, a storage device 102, an input device 104, an output device 105, and a communication device 106. These are connected to be communicable via a communication means such as a bus.

  The processor 101 is configured using, for example, a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The storage device 102 is a device that stores programs and data. For example, ROM (Read Only Memory), RAM (Random Access Memory), NVRAM (Non Volatile RAM), hard disk drive, SSD (Solid State Drive), optical type A storage device or the like. The input device 104 is a user interface that receives input of information and instructions from the user, and is a keyboard, a mouse, a touch panel, or the like, for example. The output device 105 is a user interface that provides information to the user, such as a graphic card or a liquid crystal monitor. The communication device 106 is a communication interface that communicates with other devices via the communication network 5, and is, for example, a NIC (Network Interface Card) or a wireless LAN interface.

  FIG. 4 is a data flow diagram for explaining functions provided in the data collection device 10 and data managed by the data collection device 10.

  As shown in the figure, the data collection device 10 includes functions of a data acquisition unit 401, a search condition generation unit 404, a data search unit 407, a cluster generation unit 409, an exclusion condition determination unit 410, and an exclusion condition selection unit 412. Prepare. These functions are realized by the processor 101 reading and executing a program stored in the main storage device 102. These functions may be realized by hardware (ASIC (Application Specific Integrated Circuit) or the like). Further, a plurality of these functions may be realized by a single piece of hardware, or a configuration in which these functions are distributed or emphasized by a plurality of pieces of hardware. Each function shown in the figure is merely set for convenience in order to facilitate understanding of the function of the data collection device 10, and the classification method and name of each function are not limited to the modes shown here. .

  As shown in the figure, the data collection device 10 manages collected data 402, a setting search word 403, a synonym dictionary 405, a synonym dictionary 406, hit count data 408, search condition data 411, and the like. The data collection device 10 manages these data as, for example, a database table provided by a DBMS (DataBase Management System). In the following, the synonym dictionary 405 and the synonym dictionary 406 are exemplified as related word dictionaries used for generating a plurality of search conditions, but other types of dictionaries (for example, subordinate concepts of a word) are used as related word dictionaries. A materialized dictionary describing words may be used.

  Among the functions shown in the figure, the data acquisition unit 401 acquires data (SNS post data, article data such as news articles, open data, etc.) from the server device 20 via the communication network 5. Specifically, the data acquisition unit 401 acquires data periodically (for example, once a week) from the server device 20 by a technique such as API (Application Programming Interface) or crawling. The data acquisition unit 401 manages the acquired data group as collected data 402.

  FIG. 5 shows an example of the collected data 402. The data for one line (one record) shown in FIG. As shown in the figure, the collected data 402 includes items of a provision date and time 4021, a source 4022, a provider ID 4023, and a body text 4024. Among these, the provision date 4021 stores the date when the collected data 402 was provided (for example, the date when the collected data 402 was posted to the SNS). The source 4022 stores information indicating the type of acquisition source of the collected data 402. For example, “SNS” is stored if the acquisition source is an SNS server, and “news” is stored if the acquisition source is a Web server providing a news article. In the provider ID 4023, information (user name, account name, etc.) indicating the provider (contributor, contributor, etc.) of the collected data 402 is stored. In the body 4024, information corresponding to the body of the collected data 402 is stored.

  Returning to FIG. 4, for example, the search condition generation unit 404 acquires a search condition word group registered in advance by the user from the set search word 403, and from the synonym dictionary 405, synonyms for each word of the search condition are From the antonym dictionary 406, an antonym for each word of the search condition is acquired, and a plurality of search conditions are generated by combining the word of the search condition set by the user with the acquired synonym and antonym.

  FIG. 6 shows an example of the setting search word 403. The data for one line (for one record) shown in the figure corresponds to one setting search word 403. The set search word 403 stores words used as search conditions set in advance by the user. As shown in the figure, the set search word 403 includes items of a search condition ID 4031 and search conditions (search word 1 (4032), search word 2 (4033), search word 3 (4034),...)). Including. Among these, the search condition ID 4031 stores identification information for specifying the search condition. In search conditions (search word 1 (4032), search word 2 (4033), search word 3 (4034),...)), Words used as search conditions set in advance by the user are stored.

  FIG. 7 shows an example of the synonym dictionary 405 (synonym dictionary). The data for one line (one record) shown in the figure corresponds to one synonym dictionary 405. The synonym dictionary 405 includes one word (target word 4051) and one or more synonyms (synonyms 4052, 4053, 4054,...) About the word. For example, the synonym dictionary 405 of the record on the first line in the figure includes the target word 4051 “future”, the synonym 4052 “future”, and the synonym 4053 “future”.

  FIG. 8 shows an example of the antonym dictionary 406 (antonym dictionary). The data for one line (one record) shown in the figure corresponds to one synonym dictionary 406. The antonym dictionary 406 includes one word (target word 4061) and one or more antonyms (antonyms 4062, 4063, 4064,...) For the word. For example, the antonym dictionary 406 of the record on the second line in the figure includes an object word 4061 “relief”, an antonym 4062 “anxiety”, and an antonym 4063 “anxiety”.

  Returning to FIG. 4, the data search unit 407 searches the collected data 402 using the plurality of search conditions generated by the search condition generation unit 404 and manages the number of hits of each of the plurality of search conditions as hit number data 408. .

  FIG. 9 shows an example of hit count data 408. The data for one line (for one record) shown in the figure corresponds to one hit number data 408. The hit count data 408 includes search conditions (search word candidate 1 (4081), search word candidate 2 (4082), search word candidate 3 (4083),...)) Counted by the data search unit 407, and the search conditions. And the number of hits 4084. Among these, the search condition (search word candidate 1 (4081), search word candidate 2 (4082), search word candidate 3 (4083),...)) Includes the search condition word group in the set search word 403 and the set search. A combination of synonyms and synonyms of each word in the word group of the search condition stored in the word 403 is stored. For example, in the same figure, the search condition word group included in the setting search word 403 (telephone, pause), “pause” is replaced with a synonym (telephone, abort), (telephone, pause), (telephone , Pause) and “telephone” (pause) in which “telephone” is replaced with a synonym are stored.

  Returning to FIG. 4, the cluster generation unit 409 classifies the plurality of search conditions generated by the search condition generation unit 404 into clusters according to the hit count 4084 of the respective hit count data 408.

  The exclusion condition determination unit 410 identifies search condition candidates (identifies search conditions to be excluded from candidates) based on the result of classifying the search conditions into clusters.

  The exclusion condition selection unit 412 presents the contents of the search condition data 411 (search conditions to be candidates, search conditions to be excluded from candidates) to the user, and accepts specification of search conditions used for the search from the user.

  FIG. 10 shows an example of the search condition data 411. The data for one line (for one record) shown in the figure corresponds to one search condition data 411. As shown in the figure, the search condition data 411 includes a search condition ID 4111, search conditions (search word candidate 1 (4112), search word candidate 2 (4113), search word candidate 3 (4114),...), Exclusion. An object determination result 4115 and an exclusion selection result 4116. Among these, the search condition ID 4111 stores identification information for specifying the search condition. For the search conditions (search word candidate 1 (4112), search word candidate 2 (4113), search word candidate 3 (4114),...), The search conditions (search word candidate 1 (4081), Search word candidate 2 (4082), search word candidate 3 (4083),...) Are stored. The exclusion target determination result 4115 stores the result of determining whether the exclusion condition determination unit 410 is an exclusion target. For example, when the search condition is determined to be an exclusion target, “1” is stored in the exclusion target determination result 4115, and “0” is stored otherwise. The exclusion selection result 4116 stores the result of the user selecting whether to exclude the search condition determined to be excluded through the user interface of the exclusion condition selection unit 412. For example, if the search condition is not excluded, “-” is stored. Further, when the search condition is an exclusion target and the user selects exclusion, “1” is stored. If the search condition is an exclusion target and the user has not selected exclusion, “0” is stored.

  Next, processing performed by the data collection device 10 will be described.

<Search condition generation process>
FIG. 11 illustrates a process in which the search condition generation unit 404 of the data collection apparatus 10 generates a search condition for the collected data 402 based on the set search word 403, the synonym dictionary 405, and the synonym dictionary 406 (hereinafter referred to as a search condition generation process). This is a flowchart for explaining (referred to as S1100). For example, the data collection device 10 starts the search condition generation processing S1100 when a start operation of the processing is performed on the input device 14. Hereinafter, the search condition generation processing S1100 will be described with reference to FIG. In the following description, a case where there are two search words 501 (when search word 4032 and search word 4033 are set in FIG. 6) will be described as an example.

  As shown in the figure, the search condition generation unit 404 first acquires the first search word 1 (4032) from the set search word 403 (S1101). Hereinafter, a list of related words of the first search word 1 (4032) is expressed as a related word list W1, and an nth related word included in the related word list W1 is expressed as W1 [n].

  Subsequently, the search condition generation unit 404 acquires the synonym of the first search word 1 (4032) from the synonym dictionary 405 (hereinafter, the total number of acquired synonyms is expressed as W1SN). Synonyms are registered in the search word 1 related word list W1 [1] to W1 [W1SN] (S1102). It is assumed that the first search word 1 (4032) is registered in W1 [0].

  Subsequently, the search condition generation unit 404 acquires the antonym of the first search word 1 (4032) from the antonym dictionary 406 (hereinafter, the total number of acquired antonyms is expressed as W1AN), and the acquired antonym is The search word 1 related word list W1 [W1SN + 1] to W1 [W1SN + W1AN] is registered (S1103).

  Next, the search condition generation unit 404 acquires the second search word 2 (4033) from the set search word 403 (S1104). Hereinafter, a list of related words of the second search word 2 (4033) is expressed as a related word list W2, and an nth related word included in the related word list W2 is expressed as W2 [n].

  Subsequently, the search condition generation unit 404 acquires the synonym of the second search word 2 (4033) from the synonym dictionary 405 (hereinafter, the total number of acquired synonyms is expressed as W2SN). Synonyms are registered in the search word 1 related word list W2 [1] to W2 [W2SN] (S1105). It is assumed that the second search word 2 (4033) is registered in W2 [0].

  Subsequently, the search condition generation unit 404 acquires an antonym of the second search word 2 (4033) from the antonym dictionary 406 (hereinafter, the total number of acquired antonyms is expressed as W2AN). The search word 2 related word lists W2 [W2SN + 1] to W2 [W2SN + W2AN] are registered (S1106).

  In S1107, the search condition generation unit 404 substitutes 0 for a variable I1 used as an index for specifying a related word in the related word list.

  In S1108, the search condition generation unit 404 substitutes 0 for a variable I2 used as an index for specifying a related word in the related word list.

  Subsequently, the search condition generation unit 404 uses the search condition (search word candidate 1 (4081), search word candidate 2 (4082)) of the hit number data 408 as a set of related words (W1 [I1], W2 [I2]). The search word candidate 3 (4083),...) Or the search condition data 411 is registered (S1109).

  Subsequently, the search condition generation unit 404 repeats the process of S1109 while repeatedly incrementing I2 (S1111) until I2 becomes W2SN + W2AN (until the condition of S1110 is satisfied (S1110: YES)).

  When the condition of S1110 is satisfied (S1110: YES), the search condition generation unit 404 repeatedly increments I1 until I1 becomes W1SN + W1AN (until the condition of S1112 is satisfied (S1112: YES)) (S1113). , S1108 to S1111 are repeated.

  Here, the processing of S1107 to S1113 is a process of selecting related words one by one from the related word list of each search word (search word 1, search word 2) and registering a set of these related words as search condition data 411. Is equivalent to executing for all combinations of related terms. Thus, for example, when the related word list of the first search word 1 (4032) is (safe, relief) and the related word list of the second search word 2 (4033) is (future, future). , (Reliable, Future), (Anhui, Future), (Reliable, Future), and (Anhui, Future) are registered as search conditions.

  By the way, although the case where there are two search words 501 has been described above as an example, the number of search words 501 may be any number. When the number of search words 501 is n, for example, related word lists W1 to Wn are created for each search word, and a set of related words (W1 [I1], W2 [I2],..., Wn [In] ) Is registered as a search condition.

<Collecting data acquisition process>
FIG. 12 illustrates processing in which the data acquisition unit 401 of the data collection device 10 acquires data (collected data 402) from the server device 20 via the communication network 5 (hereinafter referred to as collected data acquisition processing S1200). It is a flowchart. For example, the data collection device 10 executes the collected data acquisition process S1200 after the search condition generation process S1100 ends. Hereinafter, the collected data acquisition process S1200 will be described with reference to FIG.

  As shown in the figure, the data acquisition unit 401 acquires data from the server device 20 through the communication network 5 and stores it as collected data 402 (S1101). For example, the data acquisition unit 401 acquires data provided by the server device 20 specified in advance by the user. Further, for example, the data acquisition unit 401 acquires data in a period specified by the user among the data included in the server device 20.

<Search process>
FIG. 13 is a flowchart illustrating a process in which the data search unit 407 of the data collection device 10 searches the collected data 402 (hereinafter referred to as search process S1300). For example, the data collection device 10 executes the search process S1300 after the collection data acquisition process S1200 ends. Hereinafter, the search processing S1300 will be described with reference to FIG.

  As shown in the figure, first, the data search unit 407 obtains a list of search conditions and the total number of search conditions included in the list from the search condition data 411 generated by the search condition generation unit 404 (S1301). . Hereinafter, the search condition list is denoted by S, the I-th search condition in the search condition list is denoted by S [I], and the total number is denoted by SCN.

  Subsequently, the data search unit 407 substitutes 0 for the variable I (S1302). The variable I is an index for designating a search condition.

  Subsequently, the data search unit 407 acquires from the collected data 402 what hits the search condition S [I] (S1303).

  Subsequently, the data search unit 407 obtains the number (hit number) of the collected data 402 acquired in S1303, sets the obtained value as the hit number 4084, and hits the currently selected search condition S [I]. Data 408 is generated (S1305).

  The data search unit 407 increments I (S1306) and performs the processing of S1303 to S1304 until I = SCN, that is, for all search conditions in the search condition list (S) (S1305: YES). Run repeatedly.

  By executing the above search processing S1300, the number of collected data 402 (number of hits) that hit the search condition can be acquired for each of the plurality of search conditions generated by the search condition generating unit 404.

<Cluster generation processing>
In FIG. 14, the cluster generation unit 409 of the data collection device 10 uses the search conditions (search word candidate 1 (4081), search word candidate 2 (4082), search word candidate 3 (4083)) registered in the hit count data 408. ,... Is a flowchart for explaining a process of classifying a cluster (hereinafter referred to as a cluster generation process S1400). For example, the data collection device 10 executes the cluster generation process S1400 after the search process S1300 ends. Hereinafter, a case where the “k-means method” is used as a clustering method will be described as an example.

  As shown in the figure, first, the cluster generation unit 409 assigns (randomly) each hit number 4084 (each record of the hit number data 408) of the hit number data 408 to any of the K clusters (S1401). The number K of clusters can be preset by the user, for example. It should be noted that the user can search for a search condition more suitable for acquiring the target information with high accuracy by adjusting the number K of clusters. Hereinafter, the number of I-th hits included in the list of hit counts 4084 of the hit count data 408 is denoted as v [I], and the total number of hits (records) included in the list of hit counts 4084 is denoted as N.

  Subsequently, the cluster generation unit 409 substitutes 0 for a variable k which is an index for specifying a cluster (S1402).

  Subsequently, the cluster generation unit 409 obtains an average value of the number v [n] belonging to the cluster specified by the index k, and sets this as the center value C [k] of the cluster specified by the index k (S1403). .

  Subsequently, the cluster generation unit 409 repeatedly executes the processing of S1403 while incrementing the variable k (S1405) until the variable k becomes K, and the center value C [k] (k = 0 to K) of each cluster. Is calculated.

  Subsequently, the cluster generation unit 409 substitutes 0 for a variable n that is an index for specifying one of the lists of hit counts 4084 (S1406).

  Subsequently, the cluster generation unit 409 assigns the hit count v [n] to the cluster k having the shortest distance between the cluster center value C [k] and v [n] (S1407). The distance between the cluster center value C [k] and the number of hits v [n] is obtained as an absolute value of C [k] −v [n], for example.

  Subsequently, the cluster generation unit 409 increments the variable n (S1409), repeatedly executes the process of S1407 until the variable n becomes N (S1408), and the number of hits v [n] (that is, the search condition ( Search word candidate 1 (4081), search word candidate 2 (4082), search word candidate 3 (4083),...)) Are reassigned to the cluster having the shortest distance from the cluster center value C [k].

  Next, the cluster generation unit 409 determines whether or not a change in cluster assignment has occurred during the repeated execution of the processing of S1407 (S1410). When the cluster assignment is changed (S1410: Yes), the cluster generation unit 409 executes the processes of S1402 to S1410 again. On the other hand, if the search condition cluster assignment has not been changed during the repeated execution of the process of S1407 (S1410: No), the cluster generation process S1400 ends.

  According to the above mechanism, the number of hits registered in the hit number data 408 (that is, the search condition (search word candidate 1 (4081), search word candidate 2 (4082), search word candidate 3 (4083),... )) Can be classified into a plurality of clusters composed of search conditions with a close number of cases. In the above, the “k-means method” is used, but other clustering methods may be used.

<Exclusion condition judgment processing>
FIG. 15 illustrates a process (hereinafter referred to as “exclusion condition determination process S1500”) performed by the exclusion condition determination unit 410 of the data collection apparatus 10 for the purpose of excluding search conditions belonging to a cluster to which many search cases belong. It is a flowchart to do. For example, the data collection device 10 executes the exclusion condition determination process S1500 after the cluster generation process S1400 ends. Hereinafter, the exclusion condition determination processing S1500 will be described with reference to FIG.

  As shown in the figure, the exclusion condition determination unit 410 first substitutes 0 for a variable k that is an index for specifying a cluster (S1501).

尚、クラスタの内容をユーザが確認してユーザが変数ｐを指定する構成としてもよい。ユーザは変数ｐを調節することで、目的とする情報を取得するのにより適した検索条件を抽出することができる。Next, the exclusion condition determination unit 410 determines whether or not the cluster center value C [k] is larger than the variable p (S1502). Here, the variable p is a threshold value for determining whether to exclude the search condition. For example, the variable p is obtained from the following equation, where Cm is an average value of the center values C [k] of all clusters, variance Cv, and α is a parameter predetermined by the user.

A configuration in which the user confirms the contents of the cluster and the user designates the variable p may be adopted. By adjusting the variable p, the user can extract a search condition that is more suitable for acquiring target information.

  When the center value C [k] of the cluster is larger than the variable p (S1502: Yes), the exclusion condition determination unit 410 determines that the search condition corresponding to the number v [n] included in the cluster k is an exclusion target and determines The result is set as an exclusion target determination result 4115 of the search condition data 411 (S1503). The exclusion condition determination unit 410 increments k (S1505), and repeatedly executes the processing of S1502 to S1503 until k = K (S1504).

  Through the above exclusion condition determination processing S1500, search conditions belonging to a cluster with a large number of searches can be specified as exclusion targets. For example, when the hit count data 408 has the contents shown in FIG. 9, search conditions such as (phone, pause) and (telephone, pause) are excluded. In this example, the word “pause” is ambiguous, and the search result includes a lot of data (noise) that is different from the topic to be originally collected. By performing the exclusion condition determination processing S1500 in this way, it is possible to efficiently exclude a search condition that includes a lot of data (noise) that the user does not intend in the search result.

  FIG. 16 is an example of a screen (hereinafter, referred to as an exclusion condition setting screen 1600) displayed on the display device 205 when the exclusion condition selection unit 412 allows the user to determine a search condition to be finally excluded. As shown in the figure, the exclusion condition setting screen 1600 includes a search condition display field 1601, a search condition exclusion designation field 1602, and an exclusion execution instruction button 1603. The search condition display field 1601 displays a group of search condition words determined to be excluded by the exclusion condition determination unit 410. In the search condition exclusion designation field 1602, a field (for example, a check box) for designating whether or not the user excludes the search condition displayed on the search condition display unit 1601 is displayed. The exclusion execution instruction button 1603 is a user interface for instructing the data collection device 10 to execute processing for excluding the search condition specified in the search condition exclusion specification field 1602. The contents of the exclusion designation made via the exclusion condition setting screen 1600 are reflected in the exclusion selection result 4116 of the search condition data 411 in FIG.

  The user can easily confirm the search condition (in other words, the search condition candidate presented by the data collection device 10) determined to be excluded by the exclusion condition determination unit 410 via the exclusion condition setting screen 1600. Further, the user can determine the search condition to be excluded through the exclusion condition setting screen 1600 by himself / herself.

  As described above, the data collection device 10 according to the present embodiment classifies search conditions into a plurality of clusters, and specifies search condition candidates in units of clusters. Therefore, a search belonging to a cluster having a very large number of hits. Conditions can be excluded, and search condition candidates that can obtain a search result that does not include information (noise) that is not intended by the user can be specified.

= Second Example =
The basic structure of the second embodiment is the same as that of the first embodiment, but the number of search cases is weighted according to the data acquisition source when the number of search cases is tabulated. In the following, description will be made centering on the parts that differ from the first embodiment.

  FIG. 17 is an example of hit count data 408 (hereinafter referred to as hit count data 408A) in the second embodiment. As shown in the figure, the hit number data 408A in the second embodiment is based on the search conditions (search word candidate 1 (4081), search word candidate 2 (4082), search word candidate 3 (4083),...). ), The number of searches by acquisition source 1 (1701), the number of searches by acquisition source 2 (1702), and the search score 1703.

  Of these, the number of searches by acquisition source 1 (1701), the number of searches by acquisition source 2 (1702),... Are the search conditions (search word candidate 1 (4081), search word candidate 2 (4082)) in the collected data 402. , The search word candidate 3 (4083),...) Is totaled for each acquisition source of the collection data 402. For example, the number of search cases 1 (1701) by acquisition source in the figure is the number of data including the search word candidate 1 (4081) and the search word candidate 2 (4082) in the collected data 402 acquired from the SNS server. The number of searches by acquisition source 2 (1702) is the number of data including search word candidate 1 (4081) and search word candidate 2 (4082) in the collected data 402 acquired from the Web server that provides the news article. The search score 1703 is a value obtained by weighting and adding each of the number of searches by acquisition source 1 (1701), the number of searches by acquisition source 2 (1702),... According to the acquisition source.

  The cluster generation unit 409 performs cluster generation processing S1400 using the search score 1703 as the number of hits, and classifies the search conditions into clusters. In this way, by setting the weight according to the acquisition source of the collected data 402 and adjusting the number of hits of each search condition, and then classifying the search conditions into clusters, the characteristics of the acquisition source are taken into consideration The search condition candidate can be obtained. For example, the higher the reliability (credibility) of the information provided by the acquisition source of the collected data 402, the higher the reliability (credibility) information can be collected by setting a higher weight to the acquisition source. become.

= Third Example =
In the third embodiment, for a plurality of search conditions, the number of collected data 402 that hit the search condition is counted for each predetermined time range, the degree of similarity of the transition of these numbers is calculated, and the obtained similarity The search condition is classified into a plurality of clusters based on the degree.

  In the third embodiment, the data search unit 407 includes search conditions (search word candidate 1 (4081), search word candidate 2 (4082), search word candidate 3 (4083),...) Stored in the hit count data 408. (1) is read, and the number of collected data 402 hitting the search condition is counted for each search condition for each predetermined time range. For example, when a predetermined time range is assumed to be a year unit, the number of collection data 402 that hit the search condition out of the collection data 402 posted in 2011 for the specified search condition, posted in 2012. The number of collected data 402 that hit the search condition in each fiscal year, such as the number of collected data 402 that hit the search condition in the collected data 402, is totaled. In the hit number data 408 in the third embodiment, the number of collected data 402 hitting the search condition for each predetermined time range is stored.

上式において、ａ（ｔ）は、時間範囲ｔにおいて検索条件ａに合致する収集データ４０２の件数であり、ｂ（ｔ）は時間範囲ｔにおいて検索条件ｂに合致する収集データ４０２の件数であり、ｋは時間ずれに対応するパラメータである。ｋは、例えば、ユーザが予め指定してもよいし、ヒット件数データ４０８から推定してもよい。The cluster generation unit 409 of the third embodiment measures the degree of similarity in the number of collected data 402 that hit each other for two search conditions. For example, the cluster generation unit 409 obtains the similarity of the transition of the number of collected data 402 that hits the search condition a and the search condition b from the following expression.

In the above formula, a (t) is the number of collected data 402 that matches the search condition a in the time range t, and b (t) is the number of collected data 402 that matches the search condition b in the time range t. , K are parameters corresponding to the time lag. For example, k may be designated in advance by the user or may be estimated from the hit number data 408.

  FIG. 18 shows a similarity determination example performed by the cluster generation unit 409 of the third embodiment. In the figure, the frame indicated by reference numeral 1801 is an example of a set of search conditions having a high degree of similarity, and the frame indicated by reference numeral 1802 is an example of a set of search conditions having a low degree of similarity. The set of search conditions in the frame indicated by reference numeral 1801 indicates that the similarity between the search conditions in which different place names such as “Tokyo” and “Osaka” are added to the product name “Product A” is large. On the other hand, the set of search conditions within the frame denoted by reference numeral 1802 indicates that the similarity between the search conditions is small, in which different place names such as “Tokyo” and “Osaka” are added to the product name of the product B.

  In this example, for example, a group of search conditions whose similarity exceeds a predetermined threshold value belongs to the same cluster, and a group of search conditions whose similarity is less than the above threshold value belongs to a different cluster. To. In this way, by classifying search conditions into clusters according to the similarity of search conditions, it is possible to classify search conditions into clusters while taking into account the similarity of the number of hits in each search condition over time. The search condition can be determined in consideration of the similarity of the number of cases with time. As another effect of determining the similarity of the number of hits over time, for example, whether or not information propagation has occurred on a specific topic based on the similarity of the transition of the number of hits for each search condition. Can be determined.

  By the way, this invention is not limited to above-described embodiment, Various modifications are included. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to one having all the configurations described. Further, a part of the configuration of an embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of an embodiment. In addition, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. In addition, each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function can be stored in a recording device such as a memory, a hard disk, or an SSD, or a recording medium such as an IC card, an SD card, or a DVD.

  The control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

DESCRIPTION OF SYMBOLS 1 Data collection system, 5 Communication network, 10 Data collection apparatus, 20 Server apparatus, 100 Information processing apparatus, 401 Data acquisition part, 402 Collection data, 403 Setting search word, 404 Search condition generation part, 405 Synonym dictionary, 406 Synonym Dictionary 407 Data search unit, 408 hit count data, 409 cluster generation unit, 410 exclusion condition determination unit, 411 search condition data, 412 exclusion condition selection unit, S1100 search condition generation process, S1200 collection data acquisition process, S1300 search process, S1400 Cluster generation process, S1500 exclusion condition determination process, S1600 exclusion condition setting screen

Claims (15)

An information processing apparatus that collects information satisfying a search condition from a data group,
Search the data group using each of a plurality of search conditions,
A data collection device that identifies candidates for the search condition used for the search based on a result of classifying the plurality of search conditions into a plurality of clusters according to the number of hits of each of the plurality of search conditions. The data collection device according to claim 1,
A data collection device for obtaining an average value of the number of hits for each of the search conditions belonging to each of the clusters, and identifying the search condition candidates based on the average value. The data collection device according to claim 2,
A data collection device that excludes, from the candidate targets, the search condition belonging to the cluster whose average value is not included in a preset range. The data collection device according to any one of claims 1 to 3,
A data collection device that automatically generates a search condition different from the search condition by searching for a related word of a word included in the search condition from a related word dictionary and replacing it. The data collection device according to claim 4,
The data collection device, wherein the related word dictionary is at least one of a synonym dictionary, an antonym dictionary, and an instantiation dictionary. The data collection device according to claim 1,
A data collection device that classifies the plurality of search conditions into clusters by a k-means method. The data collection device according to claim 1,
A data collection device comprising a user interface for presenting and selecting a search condition specified as a candidate for the search condition used for the search. The data collection device according to claim 1,
Total the number of hits by data acquisition source of the data group,
After obtaining the total number of hits for each acquisition source after weighting the number of hits for each acquisition source,
A data collection device that identifies candidates for the search condition used for the search based on a result of classifying the plurality of search conditions into a plurality of clusters according to the total value of each of the plurality of search conditions. The data collection device according to claim 1,
A data collection device that classifies the plurality of search conditions into clusters according to the similarity of time-series changes in the number of hits of each of the plurality of search conditions. The data collection device according to claim 1,
A data collection device that is communicably connected to the Internet and accesses the data group via the Internet. An information collection method for collecting information satisfying a search condition from a data group,
Information processing device
A first step of searching the data group using each of a plurality of search conditions;
A data collection method comprising: a second step of identifying candidates for the search condition used for the search based on a result of classifying the plurality of search conditions into a plurality of clusters according to the number of hits of each of the plurality of search conditions. The data collection method according to claim 11, comprising:
In the second step, the information processing apparatus obtains an average value of the number of hits of each of the search conditions belonging to each of the clusters, and specifies the search condition candidates based on the average value. Collection method. The data collection method according to claim 12, comprising:
The data collection method, wherein, in the second step, the information processing apparatus excludes the search condition belonging to the cluster that is not included in a preset range from the candidates. The data collection method according to claim 11, comprising:
The information processing apparatus is
A step of counting the number of hits by data acquisition source of the data group; and
Calculating the total number of hits for each acquisition source after weighting the number of hits for each acquisition source;
Further including
In the second step, based on a result of classifying the plurality of search conditions into a plurality of clusters according to the total value of each of the plurality of search conditions, the search condition candidates used for the search are specified. Method. The data collection method according to claim 11, comprising:
The data collection method, wherein the information processing apparatus classifies the plurality of search conditions into clusters according to the similarity of time-series changes in the number of hits of the plurality of search conditions in the second step.

Images