JP6385224B2 - Data collection device - Google Patents

Description

  The present invention relates to a data collection apparatus that can collect necessary data in real time in a system that handles a large amount of data (various amounts of data) such as SNS (social) data, multimedia data, and sensor data.

  For example, in a conventional data collection device, in order to suppress the load when collecting a large amount of large amounts of data, the load is checked by scheduling the data collection and checking the load (for example, Patent Document 1). reference).

JP 2006-79488 A

  However, when the load on the machine is reduced by scheduling, collection of a large amount of data is temporarily waited, so that necessary data cannot be collected in real time. For example, there has been a problem that it is impossible to confirm highly urgent information such as hospital operating conditions, vacant floor conditions, and railway operation conditions that need to be checked in real time in the event of an emergency such as a disaster.

  The present invention has been made to solve the above-described problems, and is a data collection device capable of reducing the load on a machine when collecting a large amount of data and collecting necessary data in real time. The purpose is to provide.

Data collecting device according to the present invention, a wide large amount of data, a data collecting device for collecting from an external server of the data collection in accordance with the load state of the machine, the normal load data is increased in the machine than at If the data collection level is high, the data collection priority is determined based on the set definition data, and the data collection priority is stored in the database. Data with high urgency is preferentially stored in the database from data with low priority, and the data with high urgency is text collected from the data collection server for determining the priority of data collection. The data includes data, and the text data includes character string information with high urgency, and the data with low urgency is the above Data does not include the text data or, is characterized in that the data does not include the character string information of urgency height above text data.

  According to the data collection device of the present invention, when the load state of a machine is high, data is collected according to the priority of data collection, and data with high urgency is given priority among data determined to have low priority. Since data can be stored in the database and collected, it is possible to update data with high priority and data with high urgency in real time while reducing the load on the machine.

It is a block diagram which shows the structure of the data collection device in Embodiment 1 of this invention. 3 is a diagram illustrating a flowchart of the data collection device according to Embodiment 1. FIG. It is a block diagram which shows the structure of the data collection device in Embodiment 2 of this invention. FIG. 10 is a diagram illustrating a flowchart of a data collection device in a second embodiment. It is a block diagram which shows the structure of the data collection device in Embodiment 3 of this invention. FIG. 11 is a diagram illustrating a flowchart of a data collection device in a third embodiment. It is a block diagram which shows the structure of the data collection device in Embodiment 4 of this invention. FIG. 10 is a diagram illustrating a flowchart of a data collection device in a fourth embodiment. It is a block diagram which shows the structure of the data collection device in Embodiment 5 of this invention. FIG. 10 is a diagram showing a flowchart of a data collection device in a fifth embodiment.

It is a figure which shows the example of data collection object of a data collection device required for description of Embodiment 1 of this invention. It is a figure which illustrates the data collected by the data collection device required for description of Embodiment 1 of this invention. It is the figure which illustrated the facility name of the data collected by the data collection device, information, a use, and importance required for description of Embodiment 1 of this invention. It is the figure which illustrated increase of the collection data at the time of the normal time of the data collected by a data collection device required for description of Embodiment 1 of this invention, and a disaster occurrence. It is the figure which illustrated prioritization of the data collection in a facility unit required for description of Embodiment 1 of this invention. It is the figure which illustrated prioritization of the data collection required for description of Embodiment 2 of this invention in a data unit.

Embodiment 1 FIG.
FIG. 11 shows an example of an object to be collected by the data collection apparatus of the present invention. As shown in FIG. 11, as area information for the entire area, data handled by the facilities (large-scale buildings, water facilities (waterworks and sewerage), hospitals, railways, toll roads, factories, airports, SNS, etc.) existing in the area Target of collection. FIG. 12 shows an example of data handled. FIG. 13 shows an example of the name of the facility that delivered the data, the information item (large and small classification), the use of the information, and the importance (priority) of the information. Here, information for grasping the usage status of each facility in an emergency such as a disaster is positioned as highly important information. Furthermore, an example of data increase is shown in FIG. It shows that data such as disaster information / emergency information on SNS, train delay information on railways, accident information, etc. will increase when a disaster occurs. It is necessary to preferentially store the latest information in the database of the data collection device so that the information required by citizens and government officials in the event of a disaster can be confirmed in real time.

Therefore, in the data collection device of the present invention, by giving priority to a large amount of various data, when scheduling data collection when the machine is under heavy load, the priority of the data is checked and the priority is high. Can be preferentially collected. Therefore, it is possible to collect necessary high-priority data in real time while reducing the load on the machine by postponing the collection of low-priority data. Here, FIG. 15 shows an example of data collected by the data collection device. This is an example of lowering the priority of data collection for large-scale buildings and water facilities that do not hold high-priority information and preferentially collecting data of other facilities (data described in the colored areas).
Furthermore, in the data collection device of the present invention, it is possible to analyze the character string information for text data (details will be described later) out of low priority data. Text data including information can be collected in real time.

  1 is a block diagram showing a configuration of a data collection device 100 according to Embodiment 1 of the present invention. In FIG. 1, the data collection device 100 includes, as data to be managed, a database 101 that stores and manages a large amount of data, and data collection priority definition data 102 that defines data collection priorities. Also, as functions, a plurality (N) of data collection servers 111 to 112 that distribute a large amount of data, a plurality (N) of data collection units 1 and 2 that receive the respective data, and a CPU of the machine And a load check unit 3 for checking the load state of the memory, a data collection priority setting unit 4 for setting the data collection priority in the data collection priority definition data 102, and data set by the data collection priority setting unit 4 A data collection priority determination unit 5 that determines the priority of data collection from the collection priority definition data 102 is provided. Furthermore, a text information priority determination unit that determines priority of text information from high priority text definition data 103 in which character string information having high urgency is defined for text data among data collected from each data collection server 6.

Here, the text data is data of character string information collected and held by the data collection servers 111 to 112. The data collected by the data collection device 100 includes numerical data and image data in addition to text data (character string information). Of the data collected by the data collection device 100, text data that is character string information is information attached to data such as event information including the degree of urgency such as a failure.
This text data is not added and may not be included for data that is not required to be updated in real time (low urgency). In the data collection process to be described later, first, the level of priority is determined for each of the data collection servers 111 to 112 that perform data distribution, and the server priority is low and stored in the database 101 in the next stage. The selection of data to be collected in the database 101 is advanced by the procedure of determining the presence or absence of text data for the missing data.

Next, the data collection process will be described with reference to FIG. FIG. 2 is a flowchart showing a processing procedure of data collection processing of the data collection unit 1 in the data collection device.
In FIG. 2, when the data collection process of the data collection unit 1 starts, first, the load state of the CPU and the memory of the machine is acquired from the load check unit 3 (S201). It is checked whether the load is low (S202). If the load is low, a large amount of data is received from the data collection server 111 to be collected (S203), and the collected data is stored in the database 101 (S204). ). When the load is high, the data collection priority is acquired from the data collection priority determination unit 5 (S205). It is checked whether the data received by the data collection unit 1 has a high priority for data collection (S206). If the priority is high, the processes of S203 and S204 are performed. If the priority is low, it is checked whether the collected data includes text data (S207). If there are only numerical data and image data, and text data is not included, the processing from S201 is performed again. When text data is included, the priority of the text information is acquired from the text information priority determination unit 6 (S208). It is checked whether the text information has high priority (high urgency) (S209). If the text information has high priority, the processing of S203 and S204 is performed. If the priority is low, the process from S201 is performed again.

  As shown in FIG. 2, for example, data of the data collection server (hospital-A) defined as high priority in the data collection priority definition data 102 is stored in the database 101. Data collection server (Large-scale building-B) that includes text data in the collected data among the data of the low-priority data collection servers (large-scale building-A, large-scale building-B, water facility-A) Since the text data includes character string information with a high degree of urgency (such as “self-generated failure”), the data with a high degree of urgency is preferentially stored in the database 101. The same procedure is performed for the other data collection units 2.

In this way, it is determined whether or not data with high urgency is included in the data collection target server by checking the presence or absence of text data in the data collected from the data collection servers 111 to 112 by the data collection device 100. Further, by checking the character string information of the text data, it is possible to determine which data in the server has a high degree of urgency. Then, data determined to have a high degree of urgency can be preferentially stored in the database 101, and data can be updated in real time. A series of data for data that is determined to be low in the priority determination at the server level, and that is determined to have no text data or low in the text data check or text information check priority determination. in the collection process without storage in the database 101, again, such that the process from the load state acquisition step S 2 01, the process is repeated.

  As described above, according to the data collection device 100 of the first embodiment of the present invention, data can be collected by checking the load state of the machine and the priority of data collection, thereby reducing the load state of the machine. In addition, the priority is high and data necessary for the user can be collected in real time. Furthermore, data with a high degree of urgency can be selected from data determined to have a low priority for data collection based on text data and can be preferentially collected in the database 101 in real time for the user. Information can be provided.

Embodiment 2. FIG.
FIG. 3 is a block diagram showing the configuration of the data collection device 100 according to Embodiment 2 of the present invention. In addition, the same number is attached | subjected the location with the same function as FIG. 1 which shows the said Embodiment 1, and description is abbreviate | omitted. The data collection device 100 according to the first embodiment determines the priority for each of the data collection servers 111 to 112 that distribute data, for each of a large amount of data received by the data collection unit 1. Next, the processing procedure for determining the priority based on the text data is described. In the data collection device 100 according to the second embodiment, the priority is not set for each server, but a plurality of data distributed by the server. The data collection processing is performed in the order of setting the priority for each data, determining the priority for each data, and then determining the priority based on the text data.

  In FIG. 3, the data collection device 100 has data priority definition data 104 that defines the priority of individual data as data to be managed. Further, as a function, for a plurality (N units) of data collection servers 111 to 112, N data collection units 201 to 202 for collecting the respective data, and storage of various kinds of large amounts of data in the database 101 The data priority setting unit 7 sets the priority of the data to the data priority definition data 104, and the data priority definition data 104 set by the data priority setting unit 7 is used to determine the priority of each data item of a large amount of data. A data priority determination unit 8 is included.

Next, the data collection processing of the data collection unit 201 will be described with reference to the flowchart of FIG. In addition, the same number is attached | subjected to the location of the process similar to FIG. 2 which shows the said Embodiment 1, and description is abbreviate | omitted.
In FIG. 4, when the machine is in a low load state check (S202), the data priority is acquired from the data priority determination unit 8 (S301). It is checked whether the priority of the data is high (S302). If the priority is high, the processing of S203 and S204 is performed. If the priority is low, it is checked whether the collected data is text data (S303). When it is not text data (when there is no text data), the processing is performed again from S201. In the case of text data, the priority of the text information is acquired from the text information priority determination unit 6 (S304). Whether the priority of the text information is high is checked (S305). If the priority is high, the processing of S203 and S204 is performed. If the priority is low, the process from S201 is performed again.

  As shown in FIG. 4, for example, data of a data target (the number of vacant beds in Hospital-A, etc.) set as high priority in the data priority definition data 104 is stored in the database 101 and set as low priority. The data target (power consumption of large-scale building-A, etc.) is repeatedly executed from the load state acquisition. In addition, in the data object set as low priority and the collected data is the data object of the text data (event information of the large-scale building-B), the character string information with high urgency in the text data (“self-generated failure” Etc.), the data is preferentially stored in the database 101.

As described above, according to the data collection device 100 of the second embodiment of the present invention, since the priority of data collection can be checked in units of data, machine load adjustment according to the size of each data and necessary data Real-time collection. Furthermore, text data with a high degree of urgency can be collected in real time.
FIG. 16 shows an example of collected data that illustrates the prioritization of data collection in units of data. It is an example of lowering the priority of data collection for low-priority information and preferentially collecting high-priority data (data written in the colored areas). For information on low-priority data collection Only the highly urgent information of the text data is preferentially collected, and the information is stored in the database 101.

Embodiment 3 FIG.
FIG. 5 is a block diagram showing the configuration of the data collection device 100 according to Embodiment 3 of the present invention. In addition, the same number is attached | subjected the location with the same function as FIG. 1 which shows the said Embodiment 1, and FIG. 2 which shows Embodiment 2, and description is abbreviate | omitted.
In FIG. 5, the data collection device 100 includes a plurality (N) of data collection units 301 to 302 that collect respective data with respect to a plurality (N) of data collection servers 111 to 112 as functions.

Next, the data collection process will be described with reference to FIG. In addition, the same number is attached | subjected to the location of the process similar to FIG. 2 which shows the said Embodiment 1, and FIG. 4 which shows Embodiment 2, and description is abbreviate | omitted.
In FIG. 6, when the priority is high in the data collection server unit check (S206), the data priority is acquired from the data priority determination unit 8 (S301), and the data priority is high. Check (S302).
Note that the data collection device 100 preferentially determines data with high urgency among data with high urgency in the server with low priority and data with low priority among individual data by determination using text data. Data collection is the same as in the first and second embodiments, and is common to the fourth and fifth embodiments described later.

  As described above, according to the data collection device 100 of the third embodiment of the present invention, the priority of data collection can be checked by sequentially focusing on the data collection server unit and the data unit, thereby reducing the machine load. Further, it is possible to collect necessary data in real time.

Embodiment 4 FIG.
FIG. 7 is a block diagram showing the configuration of the data collection device 100 according to Embodiment 4 of the present invention. In addition, the same number is attached | subjected the location with the same function as FIG. 1 which shows the said Embodiment 1, and description is abbreviate | omitted.
In FIG. 7, the data collection device 100 includes a data collection priority schedule definition file 105 that defines data collection priority and low-priority collection schedules as data to be managed. As a function, for a plurality (N) of data collection servers 111 to 112, a plurality (N) of data collection units 401 to 402 for collecting the respective data, a data collection priority and a low priority collection schedule The data collection schedule setting unit 9 to be set in the data collection priority schedule definition file 105, the data collection priority is determined from the data collection priority schedule definition file 105 set in the data collection schedule setting unit 9, and the data collection priority A data collection schedule determination unit 10 that acquires a collection schedule for a server determined to have a low priority is provided.

Next, the data collection process will be described with reference to FIG. In addition, the same number is attached | subjected to the location of the process similar to FIG. 2 which shows the said Embodiment 1, and description is abbreviate | omitted.
In FIG. 8, when the priority is low in the data collection high priority check (S206), the data collection schedule is acquired from the data collection schedule determination unit 10 (S401), and the data collection processing is waited until the data collection time. (S302) and S201 are repeated. For example, when the collection period is set to 60 seconds in the data collection priority schedule definition file, the process is repeatedly executed after acquiring the load state after waiting for 60 seconds.

  As described above, according to the data collection device 100 of the fourth embodiment of the present invention, it is possible to eliminate load state acquisition repetitive processing for a server that is a low-priority data collection target and reduce the load on the machine. Can be reduced.

Embodiment 5. FIG.
FIG. 9 is a block diagram showing the configuration of the data collection device 100 according to Embodiment 5 of the present invention. In addition, the same number is attached | subjected the location with the same function as FIG. 3 which shows the said Embodiment 2, and description is abbreviate | omitted.
In FIG. 9, the data collection device 100 includes a data priority schedule definition file 106 that defines a data priority and a low priority collection schedule as data to be managed. As functions, a plurality (N) of data collection servers 501 to 502 for collecting the respective data to a plurality (N) of data collection servers 111 to 112, data priority and low priority collection schedule data The data schedule setting unit 11 to be set in the priority schedule definition file 106, the data priority is determined from the data priority schedule definition file 106 set in the data schedule setting unit 11, and the data with low priority is collected. A data schedule determination unit 12 that acquires a schedule is included.

Next, the data collection process will be described with reference to FIG. In addition, the same number is attached | subjected to the location of the process similar to FIG. 4 which shows the said Embodiment 2, and description is abbreviate | omitted.
In FIG. 10, when the priority check is low (S301), the data collection schedule is acquired from the data schedule determination unit 12 (S501), and the process waits until the data collection time (S502). The process of S201 is repeated.

  As described above, according to the data collection device 100 of the fifth embodiment of the present invention, it is possible to eliminate load state acquisition repetitive processing for low priority data, and to reduce the load on the machine. .

  It should be noted that the present invention can be freely combined with each other within the scope of the invention, and each embodiment can be appropriately modified or omitted.

1, 2, 201, 202, 301, 302, 401, 402, 501, 502 Data collection unit, 3 Load check unit, 4 Data collection priority setting unit, 5 Data collection priority determination unit, 6 Text information priority determination Unit, 7 data priority setting unit, 8 data priority determination unit, 9 data collection schedule setting unit, 10 data collection schedule determination unit, 11 data schedule setting unit, 12 data schedule determination unit, 100 data collection device, 101 database, 102 data collection priority definition data, 103 high priority text definition data, 104 data priority definition data, 105 data collection priority schedule definition file, 106 data priority schedule definition file, 111, 112 data collection server.

Claims (6)

A data collection device that collects a large amount of data from an external data collection server according to the load state of the machine,
When the amount of data increases compared to normal times and the load on the machine is high, the priority of data collection is determined based on the set definition data for the various types of received data. Store high data in the database and store data with high urgency in the database preferentially from data with low data collection priority .
The high urgency data includes text data collected from the data collection server for determining the priority of data collection, and the text data includes character string information with a high urgency level.
The data collection device characterized in that the data with low urgency is data that does not include the text data, or data that does not include character string information with high urgency in the text data . Using the data collection server that distributes a large amount of data as a unit, the level of data collection priority is determined, and the data received from the data collection server determined to have a high priority is stored in the database. The data collection device according to claim 1 . For each of a large amount of data received from the data collection server, the priority of data collection is determined in units of each of the data, and the data determined to have a high priority is stored in the database. The data collection device according to claim 1 .   The data stored in the database is data having high data collection priority of the data collection server as a data distribution source, and data having high data collection priority in units of the data. The data collection device according to claim 1. For the data priority of the data collection it is determined to be low, following the timing of data collection, the data collection device of any one of claims 1 to 4, characterized in that to set after a predetermined waiting time . For the data priority of the data collection is determined to be low, the timing of the next data collection, regularly claim 1, characterized in that the data acquisition time that is repeated 4 of any one claim Data collection device.