JP2015026103A - Data gathering apparatus, data gathering method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To secure real-time property and integrity of data, to efficiently gather data on a network.SOLUTION: A data gathering apparatus 100 includes: a search condition storage unit 110 which stores a search condition; a long-period search unit 120 which executes full search on a database (DB) server on a network on the basis of the search condition; a short-period search unit 130 which searches the DB server so as to give high priority to the latest data at high frequency on the basis of the search condition; a search incidental information storage unit 141 which stores a search result, a search unit which acquires the search result, and a search time; a search result storage unit 142 which stores the search results without redundancy; a short-period adjustment unit 150 which decides a search period of the short-period search unit 130 on the basis of the data in the search incidental information storage unit 141; and a long-period adjustment unit 160 which adjusts a search period of the long-period search unit 120 on the basis of the data in the search incidental information storage unit 141.

Description

  The present invention relates to a data collection device, a data collection method, and a program that efficiently collect data on a network.

  When data such as an SNS (social networking service) or a bulletin board existing on a network is used for marketing or the like, it is necessary to store the data once in the system and perform a search or the like on the stored data. This is because the information on the network is not guaranteed to be permanent, and old data is deleted on the SNS, bulletin board, and the like.

  Therefore, a possible method for collecting and storing data from an SNS or a bulletin board is to collect all data on the SNS or the bulletin board and store it in a search DB (DataBase). For example, a service such as Twitter shown in Non-Patent Document 1 provides a Stream API in which the server side distributes data in real time on the system side that collects and stores from SNS, bulletin boards, and the like in order to meet these demands. Furthermore, if a database for searching a system that collects and stores data from an SNS or bulletin board can be linked with a database such as an SNS or bulletin board, an existing replication as shown in Non-Patent Document 2 can be used. Using technology, it is possible to copy all data at high speed.

  The above-mentioned StreamAPI transmits complete data to the system side in real time by immediately transmitting data when data such as SNS or bulletin board is updated, that is, when there is a new writing from the user. It was possible. However, many services do not permit copying of all of these data from a commercial and performance standpoint, and for a system that collects and stores data from SNS, bulletin boards, etc. There is a problem that enormous costs occur because data is held.

  Therefore, as shown in Patent Document 1 and Patent Document 2, a method has been proposed in which only data required by the user is collected by specifying a keyword. Patent Document 1 proposes a technique for collecting data by visiting links of pages having corresponding keywords. Patent Document 2 proposes a technique for efficiently collecting only Japanese data by paying attention to particles. .

JP 2005-346598 A JP 2012-032903 A

Twitter, [Search May 15, 2013 Internet <URL: http: // twitter. com />] Ken Mishima, Hiroshi Nakamura, “Concurrent control method for eager replication middleware”, IEICE Transactions Volume D, Vol. J93-D, No. 3, pp. 232-240, 2010

  However, in the methods shown in Patent Document 1 and Patent Document 2, all data is searched over time so that there is no unacquired data, but new data is sequentially generated by SNS / bulletin board, etc. The data that occurred after the start of the search could not be obtained. For this reason, the methods shown in Patent Document 1 and Patent Document 2 have a problem that it is impossible to achieve both real-time property and completeness of data.

  In view of the above problems, an object of the present invention is to provide a data collection device and a data collection method that efficiently collect data on a network while achieving both real-time property and completeness of data.

  The present invention proposes the following matters in order to solve the above problems. In addition, in order to make an understanding easy, although the code | symbol corresponding to embodiment of this invention is attached | subjected and demonstrated, it is not limited to this.

  (1) The present invention is a data collection device that collects data on a network, and includes search condition storage means (for example, the search condition storage unit 110 in FIG. 1) that stores search conditions set by a user, Based on the search condition, a long-cycle search unit (for example, the long-cycle search unit 120 in FIG. 1) that performs a full search for the data server on the network and the data server on the network based on the search condition. In addition, the short cycle search unit (for example, the short cycle search unit 130 in FIG. 1) that executes a search that gives priority to the latest data more frequently than the long cycle search unit, the long cycle search unit, and the short cycle search Search associated information storage means (for example, FIG. 1) that stores the search result acquired by the means in association with the search means that acquired the search result and the time when the search result was acquired. Search associated information storage unit 141) and search result storage unit (for example, search result storage unit 142 in FIG. 1) that stores the search results acquired by the long cycle search unit and the short cycle search unit by eliminating duplication. ) And an average per unit time of data that matches the search condition in the database server, which is obtained as the number of search results acquired per unit time by the long cycle search means with reference to the search accompanying information storage means Based on the increase number and the number of search results acquired per unit time by the short cycle search unit, a short cycle adjustment unit (for example, the short cycle adjustment in FIG. 1) that determines a search cycle performed by the short cycle search unit. 150) and the search-accompanying information storage means, and the search result obtained by the long-cycle search means is compared with the search result obtained by the short-cycle search means. A data collection device is proposed that includes a long cycle adjustment unit (for example, the long cycle adjustment unit 160 in FIG. 1) that adjusts a search cycle executed by the long cycle search unit based on the degree of complexity.

  (2) In the data collection device according to (1), the long cycle adjustment unit refers to the search-accompanying information storage unit, and the long cycle search unit stores search results obtained by previous and subsequent searches. A data collection device is proposed that adjusts the search cycle executed by the long cycle search means based on the degree of overlap.

  (3) In the data collection device according to (1) or (2), the short cycle adjustment unit refers to the search accompanying information storage unit, and with respect to the search result acquired by the long cycle search unit, A data collection device is proposed that adjusts the search cycle executed by the short cycle search unit based on the degree of duplication of the search results acquired by the short cycle search unit.

  (4) According to the present invention, in the data collection device of (1) to (3), the short cycle adjustment means is acquired by a previous and subsequent search in the short cycle search means with reference to the search accompanying information storage means. A data collection device is proposed that adjusts the search cycle executed by the short cycle search means based on the degree of duplication of the search results.

  (5) In the data collection device of (1) to (4), the present invention refers to the search associated information storage unit when the search condition storage unit stores a plurality of the search conditions, Based on the acquisition rate and acquisition amount of the search results of each search condition stored in the search condition storage means, resource allocation determining means (for example, the resource shown in FIG. 3) determines the resource allocation ratio to each search condition. Resource determining unit 180), and resource control for allocating resources based on the distribution ratio of resources to each search condition to the process for executing the search for each search condition in the short cycle search means and the long cycle search means A data collection device including means (for example, the resource control unit 190 in FIG. 3) is proposed.

  (6) According to the present invention, in the data collection device of (1) to (5), the short-cycle search unit stores data accumulated in the database server after the search by the long-cycle search unit is started. A data collection device characterized by being a search target has been proposed.

  (7) The present invention relates to the data collection device according to (1) to (6), in response to receiving the search condition from the user, to the data server on the network based on the received search condition. An on-demand search unit (for example, on-demand search unit 170 in FIG. 1) that performs a search, and the search result storage unit stores the search result acquired by the on-demand search unit without duplication. The data collection device characterized by

  (8) The present invention provides a search condition storage means for storing search conditions set by a user, a long cycle search means, a short cycle search means, a search associated information storage means, a search result storage means, a short cycle adjustment means, and a long A data collection method in a data collection apparatus including a cycle adjustment unit, wherein the long cycle search unit performs a full search for a database server on a network based on the search condition (for example, FIG. Step S1), and the short cycle search means executes a search for giving priority to the latest data to the database server on the network more frequently than the long cycle search means based on the search condition. 2 (for example, step S2 in FIG. 2) and the search-accompanying information storage means are acquired in the first step and the second step. A third step (for example, step S3 in FIG. 2) for storing the search result obtained by acquiring the search result and the time when the search result was acquired in association with the search result obtained, and the search result storage unit However, a fourth step (for example, step S4 in FIG. 2) for storing the search results obtained in the first step and the second step by eliminating duplication, and the short cycle adjusting means, An average increase in data per unit time that matches the search condition in the database server, which is obtained as the number of search results acquired per unit time by the long-cycle search means with reference to the search accompanying information storage means And a fifth step of determining a search cycle performed by the short cycle search unit based on the number of search results acquired per unit time by the short cycle search unit (for example, FIG. Step S6) of 2 and the long cycle adjustment means refer to the search accompanying information storage means, and the search results obtained by the long cycle search means overlap with the search results obtained by the short cycle search means A data collection method including a sixth step (for example, step S7 in FIG. 2) for adjusting a search cycle executed by the long cycle search means based on the degree is proposed.

  (9) The present invention provides a search condition storage means for storing a search condition set by a user, a long cycle search means, a short cycle search means, a search associated information storage means, a search result storage means, a short cycle adjustment means, and a long A program for causing a computer to execute a data collection method in a data collection apparatus including a cycle adjustment unit, wherein the long cycle search unit executes a full search for a database server on a network based on the search condition. The first step (for example, step S1 in FIG. 2), and the short cycle search means causes the database server on the network to update the latest data more frequently than the long cycle search means based on the search conditions. And a second step (for example, step S2 in FIG. 2) for executing a search that prioritizes the search, and the search accompanying information storage means The third step (for example, FIG. 2) stores the search result acquired in the step 2 and the second step in association with the search means that acquired the search result and the time when the search result was acquired. Step S3) and a fourth step (for example, step S4 in FIG. 2) in which the search result storage means stores the search results obtained in the first step and the second step with duplicates eliminated. ) And the short cycle adjusting unit refers to the search accompanying information storage unit, and the long cycle search unit finds the number of search results acquired per unit time, and matches the search condition in the database server A search performed by the short cycle search unit based on the average increase number of data per unit time and the number of search results acquired per unit time by the short cycle search unit A fifth step (for example, step S6 in FIG. 2) for determining the period of the search, and the long cycle adjustment means refer to the search associated information storage means, and for the search result acquired by the short cycle search means, A sixth step (for example, step S7 in FIG. 2) for adjusting a search cycle executed by the long cycle search unit based on the degree of duplication of the search results acquired by the long cycle search unit. A program to execute it is proposed.

  According to the present invention, it is possible to efficiently collect data on a network while achieving both real-time property and completeness of data.

It is a figure which shows the structure of the data collection device which concerns on the 1st Embodiment of this invention. It is a figure which shows the flowchart of the data collection process in the data collection device which concerns on the 1st Embodiment of this invention. It is a figure which shows the structure of the data collection device which concerns on the 2nd Embodiment of this invention. It is a figure which shows an example of the data set of the words A, B, and C in a database server. It is a figure which shows the structure of the data collection device of this invention applied to Twitter.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the constituent elements in the present embodiment can be appropriately replaced with existing constituent elements and the like, and various variations including combinations with other existing constituent elements are possible. Therefore, the description of the present embodiment does not limit the contents of the invention described in the claims.

<First Embodiment>
<Configuration of data collection device>
FIG. 1 is a diagram showing a configuration of a data collection apparatus 100 according to the first embodiment of the present invention. Here, the data collection device 100 is a device that efficiently collects data from a database server on the network, for example, a database server that stores data such as an SNS or a bulletin board. As shown in FIG. 1, the data collection device 100 includes a search condition storage unit 110, a long cycle search unit 120, a short cycle search unit 130, a data storage unit 140, a short cycle adjustment unit 150, a long cycle adjustment unit 160, and an on-state. The demand search unit 170 is configured.

  The search condition storage unit 110 stores search conditions for data that the user wants to collect from a database server on the network.

  The long cycle search unit 120 searches all the database servers on the network based on the search conditions stored in the search condition storage unit 110 in a cycle longer than the search cycle of the short cycle search unit 130. When the number of search target data and the search target period are set in advance, the long-cycle search unit 120 executes all searches within the set range of the search target data and the search target period. . The long cycle search unit 120 first executes a search with a search cycle set by a user or the like, and executes a search with the adjusted search cycle when the long cycle adjustment unit 160 adjusts the search cycle.

  Here, in the present embodiment, the long-period search unit 120 follows a link in a web page and obtains a search result by repeating an operation of collecting another web page, or a search method that follows a link, or HTTP. A web page is searched by a keyword using the GET command, and a search is performed by any of the keyword search methods in which a web page matching the keyword is obtained as a search result. In the case of the keyword search method, for example, an API provided by Twitter (registered trademark) can be used.

  The short cycle search unit 130 frequently searches the database server on the network based on the search conditions stored in the search condition storage unit 110 with a short cycle such as a second unit. To do. Similar to the long-cycle search unit 120, the short-cycle search unit 130 performs a search by a search method that tracks links or a keyword search method. As with the long cycle search unit 120, the short cycle search unit 130 first performs a search using a search cycle set by a user or the like, and is adjusted when the short cycle adjustment unit 150 adjusts the search cycle. Perform a search at the search interval.

  The short cycle search unit 130 may search all data in the database server, or may search data stored in the database server after the search by the long cycle search unit 120 is started. If the data accumulated in the database server after the start of the search by the long-cycle search unit 120 is set as a search target, the long-cycle search unit 120 efficiently performs the processing when the resource of the data collection device 100 is limited. The search and the search by the short cycle search unit 130 can be executed.

  The data storage unit 140 includes a search accompanying information storage unit 141 and a search result storage unit 142.

  The search accompanying information storage unit 141 stores information used when the cycle is adjusted by the long cycle adjustment unit 160 and the short cycle adjustment unit 150. Specifically, the search accompanying information storage unit 141 uses the search results acquired by the long cycle search unit 120 and the short cycle search unit 130 as information for specifying the search unit that acquired the search results and the time when the search results are acquired. Are stored in association with each other. The data in the search associated information storage unit 141 is deleted after the cycle is adjusted by the long cycle adjusting unit 160 and the short cycle adjusting unit 150, or when a preset interval or capacity is exceeded.

  The search result storage unit 142 stores the search results acquired by the long-cycle search unit 120 and the short-cycle search unit 130 without duplication. The search result storage unit 142 uses, for example, an in-memory DB with high input / output speed. Since the data stored in the search result storage unit 142 has both real-time characteristics and completeness, it can be effectively used for marketing and business planning.

  The short cycle adjustment unit 150 refers to the search result acquired by the short cycle search unit 130 stored in the search incidental information storage unit 141, and the appearance frequency [times / s] of the search condition per unit time and 1 Calculate the number of average search results [number of times / times] obtained in a single search. Then, the short cycle adjustment unit 150 divides the appearance frequency [times / s] per unit time of the search condition by the number [number / times] of average search results acquired by one search, and the short cycle search unit 130. The number of search results acquired per unit time (hereinafter referred to as the number of short-cycle acquisitions) [number / s] is calculated.

  In addition, the short cycle adjustment unit 150 refers to the search result acquired by the long cycle search unit 120 stored in the search accompanying information storage unit 141, and the appearance frequency [times / s] of the search condition per unit time is obtained. The number of average search results acquired in one search [number of times / times] is calculated. Then, the short cycle adjustment unit 150 divides the appearance frequency [times / s] per unit time of the search condition by the number [number / times] of average search results acquired in one search, and the long cycle search unit 120. The number of search results obtained per unit time [number / s] is calculated. The number of search results obtained per unit time by the long-cycle search unit 120 calculated in this way corresponds to the average increase number per unit time of data that matches the search condition in the database server.

  Then, the short cycle adjustment unit 150 adjusts the cycle of the short cycle search unit 130 so that the number of short cycle acquisitions and the average increase number are the same. Specifically, when the number of short cycle acquisitions is larger than the average increase number, the cycle is shortened. On the other hand, when the number of short cycle acquisitions is smaller than the average increase number, the search cycle is lengthened. Thereby, the cycle of the short cycle search unit 130 can be set to an optimal cycle by determining the number of acquired search results.

  Further, the short cycle adjustment unit 150 refers to the search accompanying information storage unit 141, and based on the degree of duplication of the search results acquired by the short cycle search unit 130 with respect to the search results acquired by the long cycle search unit 120, The search cycle executed by the short cycle search unit 130 is adjusted. Specifically, the search result acquired by the long cycle search unit 120 that does not overlap with the search result acquired by the short cycle search unit 130, that is, the search result acquired by the long cycle search unit 120, but the short cycle search unit 130. If there is a search result that could not be acquired in step 1, the search cycle of the short cycle search unit 130 is shortened. Thereby, it is possible to adjust to a search cycle in which search result leakage does not occur.

  Further, the short cycle adjustment unit 150 refers to the search accompanying information storage unit 141, and the short cycle search unit 130 executes the short cycle search unit 130 based on the degree of duplication of search results obtained by previous and subsequent searches. Adjust the search interval. Specifically, in the short cycle search unit 130, when the search result acquired in the previous search and the search result acquired in the subsequent search overlap, the search cycle of the short cycle search unit 130 Lengthen. Note that the longer the search results acquired in the previous search and the search results acquired in the subsequent search, the longer the search cycle. Thereby, it can adjust to the suitable search period which can use a resource efficiently.

  In this way, by adjusting the search cycle of the short cycle search unit 130 by the short cycle adjustment unit 150, it is possible to cope with fluctuations in the data amount of the database server and the appearance frequency of the search conditions caused by the day of the week, time, and contents. The search cycle of the short cycle search unit 130 can be adjusted.

  The long cycle adjustment unit 160 refers to the search-accompanying information storage unit 141 and determines the long cycle based on the degree of duplication of the search results acquired by the long cycle search unit 120 with respect to the search results acquired by the short cycle search unit 130. The search cycle performed by the search unit 120 is adjusted. Specifically, the search result acquired by the short cycle search unit 130 that does not overlap with the search result acquired by the long cycle search unit 120, that is, the search result acquired by the short cycle search unit 130, but the long cycle search unit 120. If there is a search result that cannot be acquired in step 1, the search cycle of the long cycle search unit 120 is shortened. As a result, since data has been deleted from the database server over time, data that could not be acquired by the long-cycle search unit 120 can be found, and the integrity of the data can be maintained.

  Further, the long cycle adjustment unit 160 refers to the search accompanying information storage unit 141, and the long cycle search unit 120 executes the long cycle search unit 120 based on the degree of overlap of the search results obtained by the previous and subsequent searches. Adjust the search interval. Specifically, in the long cycle search unit 120, if the search results acquired in the subsequent search include all the search results acquired in the previous search, the search cycle is lengthened. Thereby, it can adjust to the suitable search period which can use a resource efficiently.

  Note that the short cycle adjustment unit 150 and the long cycle adjustment unit 160 adjust the search cycle according to a preset day of the week, date, and interval.

  The on-demand search unit 170 searches the database server on the network based on the received search condition in response to receiving the search condition input by the user. The on-demand search unit 170 stores the search result in the search result storage unit 142. Thereby, it is possible to perform a search even for search conditions that are not stored in the search condition storage unit 110. The on-demand search unit 170 may not be included in the data collection device 100.

  The flowchart of the data collection process in the data collection device 100 according to the first embodiment of the present invention will be described with reference to FIG. Note that step S1 and step S2 are independent of each other, and are started when a search start instruction is received.

  First, in step S <b> 1, the long cycle search unit 120 selects a data server on the network based on the search condition stored in the search condition storage unit 110 in a cycle longer than the search cycle of the short cycle search unit 130. To search all.

  Next, in step S2, the short cycle search unit 130 updates the data server on the network frequently with a short cycle such as a second unit based on the search condition stored in the search condition storage unit 110. Search by prioritizing data.

  Next, in step S3, the search-accompanying information storage unit 141 acquires the search results acquired by the long-cycle search unit 120 and the short-cycle search unit 130, information for specifying the search unit that acquired the search results, and the search results. Stored in association with the time information.

  Next, in step S4, the search result storage unit 142 stores the search results acquired by the long cycle search unit 120 and the short cycle search unit 130 without duplication. Note that the order of step S3 and step S4 may be reversed or simultaneous.

  Next, in step S5, the short cycle adjustment unit 150 and the long cycle adjustment unit 160 determine whether or not to adjust the search cycle. If it is determined that the search period is to be adjusted (YES), the process proceeds to step S6. On the other hand, if it is determined that the search period is not to be adjusted (NO), the process proceeds to step S8.

  Next, in step S <b> 6, the short cycle adjustment unit 150 refers to the search result stored in the search accompanying information storage unit 141 and adjusts the search cycle of the short cycle search unit 130.

  Next, in step S <b> 7, the long cycle adjustment unit 160 refers to the search result stored in the search accompanying information storage unit 141 and adjusts the search cycle of the long cycle search unit 120. Note that the order of step S6 and step S7 may be reversed or simultaneous.

  Next, in step S8, the long cycle search unit 120 and the short cycle search unit 130 determine whether or not there is an end instruction. If it is determined that there is an end instruction (YES), the process ends. On the other hand, if it is determined that there is no end instruction (NO), the process returns to steps S1 and S2.

  As described above, according to the present embodiment, it is impossible to acquire the latest data by searching the latest data frequently in a short cycle with the short cycle search unit 130 together with all the searches by the long cycle search unit 120. In addition, data accumulated in the database server after the search is started by the long cycle search unit 120 can be acquired. As a result, it is possible to efficiently collect data on the network while achieving both real-time property and completeness of the data. In addition, by adjusting the search cycle of the long-cycle search unit 120 and the short-cycle search unit 130 based on the collected search results, the data on the network can be efficiently made compatible with both real-time and completeness of data. Can be collected well.

<Second Embodiment>
A second embodiment of the present invention will be described with reference to FIG. Note that the data collection device according to the present embodiment controls a resource allocated to each search condition when there are a plurality of search conditions, and stores a database server such as an SNS or a bulletin board on the network. A device that efficiently collects data from a server. In addition, about the component which attaches | subjects the same code | symbol as 1st Embodiment, since it has the same function, the detailed description is abbreviate | omitted.

<Configuration of data collection device>
FIG. 3 is a diagram showing the configuration of the data collection apparatus 101 according to the second embodiment of the present invention. As shown in FIG. 3, the data collection device 101 includes a search condition storage unit 110, a long cycle search unit 121, a short cycle search unit 131, a data storage unit 140, a long cycle adjustment unit 160, a short cycle adjustment unit 150, and resource allocation. It comprises a determination unit 180 and a resource control unit 190.

  When there are a plurality of search conditions stored in the search condition storage unit 110, the long cycle search unit 121 and the short cycle search unit 131 execute a search process for each search condition in parallel. Note that the long cycle search unit 121 and the short cycle search unit 131 have the functions described in the first embodiment.

When there are a plurality of search conditions stored in the search condition storage unit 110, the resource allocation determination unit 180 refers to the search accompanying information storage unit 141 and determines the acquisition rate and the acquisition amount of the search results of each search condition. Based on this, the distribution ratio of resources to each search condition is determined. Specifically, the resource allocation determining unit 180, as gain coefficient T _i of the search results for each search condition is the same, a variable of N r by the expression (1) and (2) below By solving the Nth order simultaneous equations, the resource allocation to each search condition is determined.

  The resource control unit 190 allocates resources to the process for executing the search for each search condition based on the resource allocation to each search condition determined by the resource allocation determination unit 180. Specifically, the resource control unit 190 adjusts the number of search processes and the search frequency based on the assigned resources.

  By the way, when there are a plurality of search conditions, it may not be efficient to allocate the same resource to each search condition. For example, when searching for a database server in which words A, B, and C have a data set as shown in FIG. 4, if resources are equally allocated to words A, B, and C, words A and B are sufficient. In some cases, however, data could not be sufficiently obtained for word C due to insufficient resources.

  In the present embodiment, by providing the above-described resource allocation determination unit 180 and resource control unit 190 in the data collection device 100, when there are a plurality of search conditions, the number of search results obtained by each search condition is the same. Resources can be allocated. As a result, data can be efficiently collected even when there are a plurality of search conditions.

<Example>
An example in which the data collection apparatus 101 of this embodiment is applied to Twitter will be described. In addition, about the component which attaches | subjects the same code | symbol as 1st Embodiment and 2nd Embodiment, since it has the same function, the detailed description is abbreviate | omitted.

  In this example, as shown in FIG. 5, the data collection device 200 includes a search condition storage unit 110, a search unit 210, a data storage unit 140, a Srrem processing unit 220, a search engine 230, and an altitude analyzer 240. . The Srrem processing unit 220, the search engine 230, and the advanced analyzer 240 are processing units that analyze the data stored in the data storage unit 140 for marketing and the like, and may be provided in another device. Good.

  The search unit 210 includes a keyword search unit 211 that performs a keyword-designated search and a user search unit 212 that performs a user-designated search. Tweets used for marketing etc. can be collected mainly by keywords and users for all Twitter tweets, and in this example, data is collected by two search methods, keyword search and user search. .

  Each processing unit of the search unit 210 operates in parallel by at least one process, and for example, collects Tweet from a Twitter database server using the Twitter REST API. When data is acquired for a plurality of keywords or users according to the user's request, the number of processes in each processing unit is increased as necessary.

  The keyword search unit 211 includes a keyword long cycle search unit 211a, a keyword short cycle search unit 211b, a hiragana search unit 211c, and a keyword on-demand search unit 211d.

  The keyword long cycle search unit 211a searches all tweets including keywords stored in the search condition storage unit 110 from the Twitter database server. The search range of the keyword long cycle search unit 211a may be limited to the number of search targets and the search period. The keyword short cycle search unit 211b preferentially searches the latest Tweet from the Twitter database server based on the search conditions stored in the search condition storage unit 110 with a short cycle such as seconds. .

  The hiragana search unit 211c uses the technique described in Patent Document 2 to search for a Japanese Tweet using hiragana as a keyword. The keyword on-demand search unit 211d searches the Twitter database server for a Tweet based on the keyword received from the user when the user wants to collect a Tweet associated with a keyword not stored in the search condition storage unit 110.

  The user search unit 212 includes an existing user search unit 212a, a new user detection unit 212b, and a user on-demand search unit 212c.

  The existing user search unit 212a searches all tweets from the Twitter database server based on the user ID stored in the search condition storage unit 110. Also, all Tweets are searched based on the user ID of the user detected by the new user detection unit 212b. In this case, Tweet may be searched according to the priority of the user by the method of Patent Document 3 (Japanese Patent Laid-Open No. 2012-216168). In this manner, real-time performance can be guaranteed on a daily basis for a Twitter user who is requesting collection of Tweet.

  The new user detection unit 212b detects a user who has posted Tweet acquired by the keyword search unit 211 and is not stored in the search condition storage unit 110, and acquires a user ID. Based on the user ID acquired by the new user detection unit 212b, the existing user search unit 212a described above searches the Twitter database for Twitter.

  The user on-demand search unit 212c searches for Twitter from the Twitter database server based on the user ID received from the user.

  The in-memory DB 142 corresponds to the search result storage unit 142, and stores the search results of the search units 211a to 211d and 212a to 212c of the search unit 210. As the in-memory DB 142, for example, Redis [May 15, 2013 search, Internet <URL: http: // redis. iotp: //>]. Note that the in-memory DB 142 can be parallelized.

  Each search unit 211a-d, 212a-c operates in parallel to achieve data integrity and real-time property, and when there are a plurality of search conditions, each search unit 211a-d, 212a-c Each has multiple parallel processes. Like Twitter, even if each piece is small data, if it is inserted into the database asynchronously and in parallel, the insertion performance of the database is greatly reduced. Furthermore, since the search results of the search units 211a to 211d and 212a to 212c include overlapping Tweets, the insertion performance is likely to deteriorate. In order to solve these problems, by using an in-memory DB with high input / output speed in the search result storage unit 142, the data is stored once, so that the duplicate Tweet is removed and the data is streamed into one stream. Can do.

  The stream processing unit 220 refers to the in-memory DB 142 and performs an analysis process that can be processed for each Tweet, such as counting the number of cases. Note that the stream processing unit 220 can also be parallelized.

  The search engine 230 searches the in-memory DB 142 based on an instruction from the advanced analyzer 240 and passes the search result to the advanced analyzer 240. The search engine 230 includes, for example, Apache Solr [May 15, 2013 Search Internet <URL: http: // lucene. apache. org / solr />].

  The advanced analyzer 240 acquires necessary Tweet via the search engine 230 from the in-memory DB 142 storing Tweet collected from the Twitter database server, and performs analysis for marketing and the like. For example, Non-Patent Document 5 [Kazufumi Ikeda, Motoshi Hattori, Tomohiro Ono, Teruo Higashino, “Twitter Contributor Profile Estimation Method for Market Analysis”, Information Processing Research Reports IPSJ-CDS Vol. 2, no. 1, pp. 82-93 (2012)], Non-Patent Document 6 [Kazufumi Ikeda, Motoshi Hattori, Tomohiro Ono, Hideki Aso, "Proposal of early detection method of communication quality deterioration tendency by Twitter analysis", FIT 2012] Analysis processing is implemented.

  In this way, when the data collection device is used for collecting Twitter's Tweet, it is possible to efficiently collect Tweet while achieving both real-time performance and completeness. Thereby, the analysis for marketing etc. can be performed using collected Tweet.

  As described above, according to the present embodiment, when there are a plurality of search conditions, the allocation of resources to be allocated to each search condition is determined, and the resources of each search condition are controlled according to the determined allocation. Even when there are a plurality of search conditions, data can be efficiently collected.

  The data collection device of the present invention can be realized by recording the processing of the data collection device on a recording medium readable by a computer system, causing the data collection device recorded on the recording medium to read and execute the processing. . The computer system here includes an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW (World Wide Web) system is used. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  The embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to the embodiments, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 100 Data collection device 110 Search condition memory | storage part 120 Long period search part 130 Short period search part 140 Data storage part 141 Search accompanying information storage part 142 Search result storage part 150 Short period adjustment part 160 Long period adjustment part 170 Image data management part

Claims (9)

A data collection device for collecting data on a network,
Search condition storage means for storing search conditions set by the user;
Long-term search means for performing a full search on the database server on the network based on the search conditions;
A short-cycle search unit that executes a search that prioritizes the latest data more frequently than the long-cycle search unit based on the search condition;
Search associated information storage means for storing the search results acquired by the long cycle search means and the short cycle search means in association with the search means that acquired the search results and the time when the search results were acquired;
Search result storage means for storing the search results acquired by the long cycle search means and the short cycle search means by eliminating duplication;
An average increase in data per unit time that matches the search condition in the database server, which is obtained as the number of search results acquired per unit time by the long-cycle search means with reference to the search accompanying information storage means A short cycle adjusting unit that determines a search cycle performed by the short cycle search unit based on the number of search results acquired per unit time by the short cycle search unit;
A search performed by the long-period search unit with reference to the search-accompanying information storage unit and based on the degree of duplication of the search result acquired by the long-cycle search unit with respect to the search result acquired by the short-cycle search unit Long cycle adjusting means for adjusting the cycle of
A data collection device comprising:   The long cycle adjustment means refers to the search accompanying information storage means, and the long cycle search means performs a search performed by the long cycle search means based on the degree of overlap of search results obtained by previous and subsequent searches. The data collection device according to claim 1, wherein the period is adjusted.   The short cycle adjustment means refers to the search-accompanying information storage means, based on the degree of duplication of the search results acquired by the short cycle search means with respect to the search results acquired by the long cycle search means. The data collection device according to claim 1 or 2, wherein a cycle of search executed by the cycle search means is adjusted.   The short cycle adjustment means refers to the search-accompanying information storage means, and the short cycle search means performs a search performed by the short cycle search means based on the degree of overlap of the search results obtained by previous and subsequent searches. The data collection device according to claim 1, wherein the period is adjusted. When the search condition storage unit stores a plurality of the search conditions,
Referring to the search-accompanying information storage unit, the distribution ratio of resources to each search condition is determined based on the acquisition rate and acquisition amount of the search result of each search condition stored in the search condition storage unit Resource allocation determination means;
The resource control means for allocating resources to the process of executing the search of each search condition in the short cycle search means and the long cycle search means based on a distribution ratio of resources to each search condition. The data collection device according to any one of claims 1 to 4.   6. The short-term search unit uses data accumulated in the database server after the search by the long-cycle search unit is started as a search target. The data collection device described. In response to receiving a search condition from the user, the database server on the network includes an on-demand search means for performing a search based on the received search condition,
The data collection device according to claim 1, wherein the search result storage unit stores the search result acquired by the on-demand search unit without duplication. Data collection comprising search condition storage means for storing search conditions set by the user, long cycle search means, short cycle search means, search accompanying information storage means, search result storage means, short cycle adjustment means, and long cycle adjustment means A data collection method in an apparatus, comprising:
A first step in which the long-period search means performs a full search on a database server on a network based on the search conditions;
A second step in which the short cycle search means performs a search for giving priority to the latest data to the database server on the network more frequently than the long cycle search means based on the search condition;
The search accompanying information storage means stores the search results acquired in the first step and the second step in association with the search means that acquired the search results and the time when the search results were acquired. A third step,
A fourth step in which the search result storage means stores the search results acquired in the first step and the second step by eliminating duplication; and
A unit of data that matches the search condition in the database server, wherein the short cycle adjustment unit is obtained as the number of search results acquired per unit time by the long cycle search unit with reference to the search accompanying information storage unit A fifth step of determining a search cycle to be executed by the short cycle search unit based on the average increase number per time and the number of search results acquired per unit time by the short cycle search unit;
The long cycle adjustment means refers to the search-accompanying information storage means, and based on the degree of duplication of the search results acquired by the long cycle search means with respect to the search results acquired by the short cycle search means, A sixth step of adjusting a search cycle executed by the cycle search means;
Data collection method including. Data collection comprising search condition storage means for storing search conditions set by the user, long cycle search means, short cycle search means, search accompanying information storage means, search result storage means, short cycle adjustment means, and long cycle adjustment means A program for causing a computer to execute a data collection method in an apparatus,
A first step in which the long-period search means performs a full search on a database server on a network based on the search conditions;
A second step in which the short cycle search means performs a search for giving priority to the latest data to the database server on the network more frequently than the long cycle search means based on the search condition;
The search accompanying information storage means stores the search results acquired in the first step and the second step in association with the search means that acquired the search results and the time when the search results were acquired. A third step,
A fourth step in which the search result storage means stores the search results acquired in the first step and the second step by eliminating duplication; and
A unit of data that matches the search condition in the database server, wherein the short cycle adjustment unit is obtained as the number of search results acquired per unit time by the long cycle search unit with reference to the search accompanying information storage unit A fifth step of determining a search cycle to be executed by the short cycle search unit based on the average increase number per time and the number of search results acquired per unit time by the short cycle search unit;
The long cycle adjustment means refers to the search-accompanying information storage means, and based on the degree of duplication of the search results acquired by the long cycle search means with respect to the search results acquired by the short cycle search means, A sixth step of adjusting a search cycle executed by the cycle search means;
A program that causes a computer to execute.

Images