JP2023151992A - Data collection system, method, and program - Google Patents

Abstract

To achieve data order guarantee and high scalability.SOLUTION: Data receiving units 103 and 113 perform mutually synchronized time management, add a timestamp representing a time based on the time management to data received from the data source, and transmit data and association information that associates data identification information and the timestamp to data transfer units 104 and 114, the data transfer units 104 and 114 receive data and association information from the data receiving units 103 and 113, temporarily save the association information and transfer the data and the association information to a data storage unit 105, and cancel the temporary storage of the association information after confirming that the data has been saved in the data storage unit 105.SELECTED DRAWING: Figure 1

Description

TECHNICAL FIELD This disclosure relates to techniques for collecting data.

Data hubs are attracting attention in order to link data from multiple information systems. Data hubs are characterized by low data loss and high scalability. In recent years, the areas to which data hubs are applied have expanded, and the data handled by data hubs has also become more diverse. For example, a data hub is required not only to collect numerical information output from devices, but also to use it for file transfer. File transfer often requires guaranteeing the order of data. At the same time, a large amount of data is output from the information systems that perform batch processing to the data hub.

Patent Document 1 discloses a technique for guaranteeing the order of data. The data processing system disclosed in Patent Document 1 includes a router connected to a network, a load distribution server connected to the router, a plurality of reception servers connected to the load distribution server, and a plurality of reception servers connected to the router. It consists of a processing server. The router adds a timestamp to the input packet. The reception server adds the latest time of the timestamps added to the plurality of packets as the reception time of the transaction to the transaction that is divided into a plurality of packets having the same identifier and transmitted. The processing server sorts and processes a plurality of transactions received from a plurality of reception servers according to transaction reception times assigned to transactions for which a preset maximum delay time has elapsed.

Japanese Patent Application Publication No. 2012-129857

As described above, in the technology of Patent Document 1, the router adds a timestamp to the packet. The router is the route that packets always take, and by adding a timestamp there, the order of the packets can be guaranteed.

However, in order to guarantee ordering, a single router adds timestamps to packets, so even if the packet processing load is distributed among multiple reception servers, there is a limit to the scalability of the system as a whole. .

One of the purposes of the present disclosure is to provide a technology that achieves data order guarantee and high scalability.

A data collection system according to one aspect of the present disclosure includes a plurality of data receiving units that share the processing of accepting data from a data source, and is provided corresponding to each of the data receiving units, and the data receiving unit receives data from a data source. and a data storage section that receives and stores the data transferred from the plurality of data transfer sections, and the plurality of data reception sections are synchronized with each other. A time stamp indicating the time based on the time management is assigned to the received data, and the data is associated with the identification information of the data and the time stamp. information to the data transfer unit, and the data transfer unit receives the data and the association information from the data reception unit, temporarily stores the association information, and transmits the data and the association information. The association information is transferred to the data storage unit, and after confirming that the data has been saved in the data storage unit, the temporary storage of the association information is canceled.

According to one aspect of the present disclosure, it is possible to guarantee data order and achieve high scalability.

FIG. 1 is a block diagram showing an information collection system according to the present embodiment. FIG. 2 is a sequence diagram for explaining processing when no failure occurs in the data receiving section and the data transfer section in the information collection system shown in FIG. 1. FIG. 3 is a flowchart for explaining details of processing in the data receiving section shown in FIG. 2. FIG. 3 is a flowchart for explaining details of processing in the data transfer unit shown in FIG. 2. FIG. FIG. 2 is a diagram showing an example of the configuration of the in-process state management database shown in FIG. 1. FIG. FIG. 2 is a diagram illustrating an example of the configuration of a data storage destination database illustrated in FIG. 1. FIG. FIG. 2 is a sequence diagram for explaining processing when a failure occurs in a data receiving unit during data collection processing in the information collection system shown in FIG. 1; FIG. 2 is a sequence diagram for explaining processing when a failure occurs in the data receiving section after data is transferred from the data receiving section to the data transferring section in the information collection system shown in FIG. 1; FIG. 2 is a sequence diagram for explaining processing when a failure occurs in the data transfer unit before data is transferred from the data transfer unit to the data storage unit in the information collection system shown in FIG. 1; FIG. 2 is a sequence diagram for explaining processing when a failure occurs in the data transfer unit while transferring data from the data transfer unit to the data storage unit in the information collection system shown in FIG. 1; 11 is a flowchart for explaining details of processing in the data transfer unit shown in FIGS. 9 and 10. FIG. FIG. 2 is a sequence diagram for explaining another example of processing when no failure occurs in the data receiving section and the data transfer section in the information collection system shown in FIG. 1; 13 is a flowchart for explaining details of processing in the data transfer unit shown in FIG. 12. FIG. FIG. 2 is a diagram showing an example of the configuration of the in-process state management database shown in FIG. 1. FIG. FIG. 2 is a sequence diagram for explaining another example of processing when a failure occurs in the data transfer unit before data is transferred from the data transfer unit to the data storage unit in the information collection system shown in FIG. 1; 16 is a flowchart for explaining details of processing in the data transfer unit shown in FIG. 15. FIG.

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the information collection system of this embodiment.

As shown in FIG. 1, the data collection system 100 of this embodiment is a system that transfers data transmitted from data sources 101 and 111 for data utilization 106, and includes a load balancing unit 102, a data It includes receiving sections 103 and 113, data transfer sections 104 and 114, and data storage section 105. Note that the number of data receiving units 103, 113 and data transferring units 104, 114 is not limited to two.

The load distribution unit 102 distributes and transmits data transmitted from the data sources 101 and 111 to the data reception units 103 and 113 using a predetermined load distribution method such as round robin. Note that the method for distributing data is not limited to the load distribution method, and the distributed data may also be referred to as chunks.

The data receiving units 103 and 113 share the processing of receiving data transmitted from the data sources 101 and 111 via the load distribution unit 102, and each performs mutually synchronized time management. The time synchronization generation units 131 and 132 generate and add a timestamp representing the time based on time management to the received data, and the received data is associated with an ID and a timestamp that serve as identification information of the data. The associated association information is transmitted to the data transfer units 104 and 114.

The data transfer units 104 and 114 are provided corresponding to the data reception units 103 and 113, respectively, and are configured to transfer data received by the data reception units 103 and 113. and correspondence information, temporarily saves the correspondence information, transfers the data and correspondence information to the data storage unit 105, and takes action after confirming that the data has been saved in the data storage unit 105. Cancel temporary storage of tag information. Furthermore, the data transfer units 104 and 114 generate new records in the in-process state management databases 141 and 142 using the data and association information received from the data reception units 103 and 113, and The data and association information received from 113 are saved in the in-process state management databases 141 and 142, and the in-process state to be the transfer state is set. In addition, when the data transfer units 104 and 114 encounter a failure after receiving data and association information from the data reception units 103 and 113 and restart from the failure, the data transfer units 104 and 114 send a response prompting data retransmission to the data reception units 103 and 114. 113 to the data sources 101 and 111, and also holds the association information, and uses the time stamp in the held association information for the retransmitted data.

The data storage unit 105 receives and stores the data transferred from the data transfer units 104 and 114, and is indicated by the time stamp in the association information transferred from the data transfer units 104 and 114. The time is set as the data creation time, and the data and the creation time are stored in the data storage database 151 in association with each other.

Processing in the information collection system configured as described above will be described below.

First, a description will be given of processing when no failure occurs in the data receiving sections 103, 113 and the data transferring sections 104, 114.

FIG. 2 is a sequence diagram for explaining processing when no failure occurs in the data receiving units 103, 113 and data transfer units 104, 114 in the information collection system shown in FIG. In addition, in FIG. 2, the case where data is transferred to the data source 101, the data receiver 103, and the data transfer unit 104 among the data sources 101, 111, the data receivers 103, 113, and the data transfer units 104, 114 is shown as an example. I will list and explain.

When data is transmitted from the data source 101 in step 201, the load distribution unit 102 determines a data reception unit as a transfer destination using a predetermined load distribution method in step 202, and sends the data to the determined data reception unit 103. In step 222, the data is transferred.

The data receiving unit 103 that has received the data transferred from the load balancing unit 102 performs the process in step 203.

FIG. 3 is a flowchart for explaining details of the processing in data receiving section 103 shown in FIG. 2.

In the process in step 203, when the data receiving unit 103 receives the data transferred from the load balancing unit 102, first in step 231, the time synchronization generating unit 131 performs high-precision time synchronization in order to perform mutually synchronized time management. A time stamp generated by (for example, GPS (Global Positioning System) or PTP (Precision Time Protocol)) is acquired, and in step 232, a data ID for identifying the data is generated. Note that the data ID consists of, for example, "file name" + "data source ID" (IP address, client ID, etc.), and files with the same name from the same data source are given the same ID. Even files with the same name are assigned different IDs if they are from different data sources. Note that highly accurate time synchronization is possible by managing time using GPS or PTP.

Next, in step 233, the data receiving unit 103 combines the time stamp acquired from the time synchronization generating unit 131, the data name, and the data ID to create a triplet that serves as association information.

Next, in step 234, the data receiving unit 103 transfers the data transferred from the load distribution unit 102 and the triplet created in step 233 to the data transfer unit 104, and in step 705, the process in step 203 is performed. finish.

The data transfer unit 104, which has received the data and triplet transferred from the data reception unit 103, performs the process in step 204.

FIG. 4 is a flowchart for explaining details of processing in data transfer unit 104 shown in FIG. 2.

In the process in step 204, upon receiving the data and triplet transferred from the data receiving unit 103, the data transfer unit 104 first checks in step 801 whether or not the received triplet is stored in the in-process status management database 141. to decide. This is done using the data ID within the triplet.

If the received triplet is not stored in the in-process state management database 141, the data transfer unit 104 generates a new record in the in-process state management database 141 in step 241, and the data transfer unit 104 generates a new record in the in-process state management database 141, The data and the triplet are saved in the in-process state management database 141, and the in-process state, which is the transfer state, is set to "receive".

Further, if the received triplet is stored in the in-process state management database 141, the data transfer unit 104 sets the in-process state, which is the transfer state of the stored triplet, to "transferring" in step 242. Change to

FIG. 5 is a diagram showing an example of the configuration of the in-process status management databases 141 and 142 shown in FIG. 1.

As shown in FIG. 5, in the in-process state management databases 141 and 142 shown in FIG. .

Next, in step 243, the data transfer unit 104 transfers the data and triplet received from the data reception unit 103 to the data storage unit 105, and waits for a response from the data storage unit 105 in step 802.

When the data storage unit 105 receives the data and its triplet transferred from the data transfer unit 104, in the process in step 205, the data storage unit 105 stores the received data in the data storage database 151 in step 251, and in step 252, An ACK is returned to the data transfer unit 104 indicating that the data received from the data transfer unit 104 has been saved in the data storage database 151 in step 251.

At this time, in step 253, the data storage unit 105 sets the time indicated by the timestamp in the triplet as the data creation time, associates the data with the creation time, and stores it in the data storage database 151. .

FIG. 6 is a diagram showing an example of the configuration of the data storage database 151 shown in FIG. 1. As shown in FIG.

As shown in FIG. 6, in the data storage database 151 shown in FIG. It is stored in association with data content 1104 indicating the content.

In this way, since the time stamp in the data receiving section 103 is used as the data creation time, the creation time can be set using highly accurate times managed by a plurality of data receiving sections.

When the data transfer unit 104 receives the ACK from the data storage unit 105, in step 244, the data transfer unit 104 transfers the data for the three sets corresponding to the ACK from the data storage unit 105 among the three sets stored in the in-process state management database 141. After confirming that the data has been saved in the data storage unit 105, the transfer status (in progress) is changed to "transfer completed".

Thereafter, in step 245, the data transfer unit 104 returns an ACK for the triplet corresponding to the ACK from the data storage unit 105 to the data receiving unit 103, and in step 246, The temporary storage is canceled and deleted, and the process in step 204 is ended in step 807.

Upon receiving the ACK from the data transfer section 104, the data reception section 103 returns an ACK to the load distribution section 102 in step 261 in step 206, and upon receiving the ACK from the data reception section 103, the load distribution section 102 In step 202, ACK is returned to the data source 101 in step 223.

Thereafter, the data source 101 deletes the data sent to the load distribution unit 102 in step 212.

In this way, data order is guaranteed based on time management in multiple data receiving units that distribute the load, so it is possible to simultaneously achieve data order guarantee and high scalability.

Next, a description will be given of processing when a failure occurs in the data receiving units 103 and 113 during data collection processing.

FIG. 7 is a sequence diagram for explaining processing when a failure occurs in the data receiving unit 103 during data collection processing in the information collection system shown in FIG.

After the processing from step 201 in the data source 101 to step 233 in the data receiving unit 103 shown in FIG. If a failure is detected using, for example, a timeout or a heartbeat, the load balancing unit 102 returns a negative ACK to the data source 101 in step 322, indicating that the data collection process has failed.

Thereafter, even if the data receiving unit 103 recovers from the failure in step 302, it does not retain the data sent from the load balancing unit 102, so the same process as shown in FIG. 2 is performed. become.

Next, a description will be given of processing when a failure occurs in the data receiving section 103 after data is transferred from the data receiving section 103 to the data transferring section 104.

FIG. 8 is a sequence diagram for explaining processing when a failure occurs in the data receiving unit 103 after data is transferred from the data receiving unit 103 to the data transferring unit 104 in the information collection system shown in FIG.

Even if a failure occurs in step 401 in the data receiving unit 103 after the processing in step 201 in the data source 101 shown in FIG. Since the data and its triplet have been transferred to the transfer unit 104, the same processing as shown in FIG. 2 is performed in the data transfer unit 104 and the data storage unit 105.

Further, when the data transfer unit 104 returns ACK in step 245, if the data reception unit 103 has recovered from the failure in step 402, the subsequent data reception unit 103, load distribution unit 102, and data source 101 In this case, processing similar to that shown in FIG. 2 is performed.

Next, a description will be given of processing when a failure occurs in the data transfer unit 104 before data is transferred from the data transfer unit 104 to the data storage unit 105.

FIG. 9 is a sequence diagram for explaining processing when a failure occurs in the data transfer unit 104 before data is transferred from the data transfer unit 104 to the data storage unit 105 in the information collection system shown in FIG. be.

If a failure occurs in step 501 in the data transfer unit 104 after the process from step 201 in the data source 101 to step 241 in the data transfer unit 104 shown in FIG. After recovering from the failure in step 502, the data transfer unit 104 detects, as step 541, a triple whose status is "receiving" from the in-process status management database 141 in step 504.

Then, the data transfer unit 104 returns a negative ACK indicating that the data collection process has failed to the data reception unit 103 in step 542. At this time, the data transfer unit 104 does not delete the detected triple from the in-process state management database 141.

Upon receiving the negative ACK from the data transfer unit 104, the data receiving unit 103 returns the negative ACK to the load distribution unit 102 in step 506 as step 561.

When the load balancing unit 102 receives the negative ACK from the data receiving unit 103, the load distribution unit 102 returns the negative ACK to the data source 101 in step 521 in step 202.

Thereafter, the series of processes shown in FIG. 2 will be performed again, but since the data transfer unit 104 has not deleted the triples detected from the in-process state management database 141, the retained data will not be included in the retransmitted data. The previous timestamp will be used.

In this way, since the previous timestamp is used when retransmitting data, it is possible to prevent the order of data from being changed due to retransmission.

Next, a description will be given of processing when a failure occurs in the data transfer unit 104 while data is being transferred from the data transfer unit 104 to the data storage unit 105.

FIG. 10 is a sequence diagram for explaining processing when a failure occurs in the data transfer unit 104 while transferring data from the data transfer unit 104 to the data storage unit 105 in the information collection system shown in FIG. .

If a failure occurs in step 601 in the data transfer unit 104 after the process from step 201 in the data source 101 to the process in step 243 in the data transfer unit 104 shown in FIG. In step 205, although the data transferred from the data storage unit 105 is stored in the data storage destination database 151, it is unclear whether all the data has been transferred.

Therefore, when the data transfer unit 104 recovers from the failure in step 602, it performs the process in step 604.

FIG. 11 is a flowchart for explaining details of the processing in the data transfer unit 104 shown in FIGS. 9 and 10.

When the data transfer unit 104 starts processing in step 901, it first determines in step 902 whether there is a triple whose status is "receiving" in the in-process status management database 141.

If there is a triplet whose status is "receiving" in the in-process status management database 141, as shown in FIG. Return to 103.

If there is no triplet with the status "receiving" in the in-process status management database 141, or if there is a triplet with the status "receiving" in the in-progress status management database 141, and the status is "receiving". After returning a negative ACK regarding the triplet "transferring" to the data receiving unit 103, the data transfer unit 104 determines in step 903 whether there is a triplet whose status is "transferring" in the in-process status management database 141.

If there is a triple whose status is "transferring" in the in-process state management database 141, in step 641, a triple whose state is "transferring" is detected in the in-process state management database 141. In step 642, the data storage unit 105 is requested to delete the data indicated by the detected triplet of data IDs, and in step 931, an ACK from the data storage unit 105 is awaited.

In step 651, data storage unit 105 deletes data from data storage database 151 in accordance with the request from data transfer unit 104, and in step 652, returns ACK to data transfer unit 104 indicating that the data has been deleted.

When the data transfer unit 104 receives the ACK from the data storage unit 105, in step 604, the data transfer unit 104 sends a negative ACK indicating that the data collection processing of the triplet of data requested to be deleted from the data storage unit 105 has failed in step 643, and receives the data. It is returned to section 103.

Furthermore, the data transfer unit 104 does not delete the triplet stored in the in-process status management database 141, but changes its status from "transferring" to "receiving" in step 644.

The data transfer unit 104 then ends the process in step 904.

Upon receiving the negative ACK from the data transfer unit 104, the data receiving unit 103 returns the negative ACK to the load distribution unit 102 in step 661 in step 662.

When the load distribution unit 102 receives the negative ACK from the data reception unit 103, the load distribution unit 102 returns the negative ACK to the data source 101 in step 202 as step 621.

Thereafter, the series of processes shown in FIG. 2 will be performed again, but since the data transfer unit 104 has not deleted the triples detected from the in-process state management database 141, the retained data will not be included in the retransmitted data. The previous timestamp will be used.

(Other embodiments)

Below, an embodiment will be described in which, in the information collection system shown in FIG. 1, the timing at which data can be deleted on the data sources 101 and 111 is advanced by using resources on the information collection system 1 side.

First, a description will be given of processing when no failure occurs in the data receiving sections 103, 113 and the data transferring sections 104, 114.

FIG. 12 is a sequence diagram for explaining another example of processing when no failure occurs in the data reception units 103, 113 and data transfer units 104, 114 in the information collection system shown in FIG. .

After the process in step 201 in the data source 101 shown in FIG. 2 and the process in step 243 in the data transfer unit 104 are performed, the process in step 1204 is performed in the data transfer unit 104.

FIG. 13 is a flowchart for explaining details of processing in data transfer unit 104 shown in FIG. 12.

In the process in step 1204, upon receiving the data and triplet transferred from the data receiving unit 103, the data transfer unit 104 first checks in step 1801 whether the received triplet and data are stored in the in-process state management database 141. judge whether This is done using the data ID within the triplet.

If the received triplet and data are not stored in the in-process state management database 141, the data transfer unit 104 generates a new record in the in-process state management database 141 in step 1241, and transfers the data from the data receiving unit 103 to the in-process state management database 141. The received data and triplet are stored in the in-process state management database 141, and the in-process state, which is the transfer state, is set to "receiving".

The data transfer unit 104 saves the data and triples received from the data receiving unit 103 in the in-process state management database 141, and after setting the in-process state that becomes the transfer state to “reception”, or the in-process state If the received triplet and data are stored in the management database 141, the data transfer unit 104 returns ACK regarding the received triplet to the data receiving unit 103 in step 1201.

Upon receiving the ACK from the data transfer section 104, the data reception section 103 returns an ACK to the load distribution section 102 in step 261 in step 206, and upon receiving the ACK from the data reception section 103, the load distribution section 102 In step 202, ACK is returned to the data source 101 in step 223.

Thereafter, the data source 101 deletes the data sent to the load distribution unit 102 in step 212.

In step 1242, the data transfer unit 104 changes the in-progress state, which is the transfer state of the stored triplet, to "transferring".

FIG. 14 is a diagram showing an example of the configuration of the in-process state management databases 141 and 142 shown in FIG. 1.

As shown in FIG. 14, the in-process status management databases 141 and 142 shown in FIG. are set in association with each other.

Next, in step 243, the data transfer unit 104 transfers the data and triplet received from the data reception unit 103 to the data storage unit 105, and waits for a response from the data storage unit 105 in step 802.

When the data storage unit 105 receives the data and its triplet transferred from the data transfer unit 104, in the process in step 205, the data storage unit 105 stores the received data in the data storage database 151 in step 251, and in step 252, An ACK is returned to the data transfer unit 104 indicating that the data received from the data transfer unit 104 has been saved in the data storage database 151 in step 251.

At this time, in step 253, the data storage unit 105 sets the time indicated by the timestamp in the triplet as the data creation time, associates the data with the creation time, and stores it in the data storage database 151. .

When the data transfer unit 104 receives the ACK from the data storage unit 105, in step 1244, the data transfer unit 104 selects the triplet and data corresponding to the ACK from the data storage unit 105 from among the triplet and data stored in the in-process state management database 141. After confirming that the data has been saved in the data storage unit 105, the transfer status of the data is changed to "transfer completed".

Thereafter, in step 1245, the data transfer unit 104 cancels the temporary storage of the triplets and data stored in the in-process state management database 141 and deletes them, and in step 1405 ends the process in step 1204.

In this way, the data transfer unit 104 temporarily stores the data together with the triplet, transmits an ACK indicating that the data has been stored to the data source 101 via the data receiving unit 103, and stores the triplet and the data. Since the temporary storage of the triplet and data is canceled after confirming that the data is transferred to the data storage unit 105 and stored in the data storage unit 105, data can be deleted at an early stage in the data source.

Next, a description will be given of processing when a failure occurs in the data transfer unit 104 before data is transferred from the data transfer unit 104 to the data storage unit 105.

FIG. 15 is for explaining another example of processing when a failure occurs in the data transfer unit 104 before data is transferred from the data transfer unit 104 to the data storage unit 105 in the information collection system shown in FIG. FIG.

If a failure occurs in step 1301 in the data transfer unit 104 after the process from step 201 in the data source 101 to step 1201 in the data transfer unit 104 shown in FIG. After recovering from the failure in step 1302, the data transfer unit 104 performs the process in step 1304.

In step 1341, the data transfer unit 104 detects a triplet and data whose status is "receiving" from the in-process status management database 141, and in step 1342, changes the status to "transferring".

Next, in step 1343, the data transfer unit 104 transfers the data and triplet stored in the in-process state management database 141 to the data storage unit 105.

When the data storage unit 105 receives the data and its triplet transferred from the data transfer unit 104, in the process in step 205, the data storage unit 105 stores the received data in the data storage database 151 in step 251, and in step 252, An ACK is returned to the data transfer unit 104 indicating that the data received from the data transfer unit 104 has been saved in the data storage database 151 in step 251.

At this time, in step 253, the data storage unit 105 sets the time indicated by the timestamp in the triplet as the data creation time, associates the data with the creation time, and stores it in the data storage database 151. .

When the data transfer unit 104 receives the ACK from the data storage unit 105, in step 1244, the data transfer unit 104 selects the triplet and data corresponding to the ACK from the data storage unit 105 from among the triplet and data stored in the in-process state management database 141. After confirming that the data has been saved in the data storage unit 105, the transfer status of the data is changed to "transfer completed".

Thereafter, in step 1245, the data transfer unit 104 cancels the temporary storage of the triplet and data stored in the in-process state management database 141 and deletes them.

FIG. 16 is a flowchart for explaining details of processing in data transfer unit 104 shown in FIG. 15.

In the process in step 1204, when the data transfer unit 104 starts the process in step 1501, first in step 1502, the data transfer unit 104 stores a triple whose status is “receiving” and data in the in-process status management database 141. determine whether it is saved.

If the in-process status management database 141 stores the triplet and data whose status is "receiving," the data transfer unit 104 performs the process of step 1201 shown in FIG. 13 in step 1521.

Further, if the in-process state management database 141 does not store the triplet and the data in which the state is "receiving", the data transfer unit 104 stores the state in the in-process state management database 141 as "transferring" in step 1503. ” triplet and determine whether the data is saved.

If the in-process status management database 141 stores a triplet and data whose status is "transferring," the data transfer unit 104 sends the data ID of the triplet to the data storage unit 105 in step 1531. It requests deletion of the indicated data and waits for an ACK from the data storage unit 105 in step 1532.

After that, when the data transfer unit 104 receives an ACK from the data storage unit 105 indicating that the data has been deleted, the data transfer unit 104 performs the process of step 243 in FIG. 13 in step 1533.

The data transfer unit 104 performs the series of processes described above, and ends at step 1504.

Note that the method executed by the information collection system of this embodiment can also be written as a software program. Further, the software program can be stored in a storage medium, and can also be provided externally via a network. The information collection system of this embodiment can be realized by a computer executing the software program.

The embodiments of the present invention described above are illustrative examples of the present invention, and are not intended to limit the scope of the present invention only to those embodiments. Those skilled in the art can implement the invention in various other ways without departing from the scope of the invention.

100... Data collection system, 101, 111... Data source, 102... Addition distribution section, 103, 113... Data receiving section, 104, 114... Data transfer section, 105... Data storage section, 131, 132... Time synchronization generation section, 141, 142...In-progress status management database, 151...Data storage destination database

Claims (11)

multiple data receiving units that share the processing of receiving data from data sources;
a plurality of data transfer units that are provided corresponding to each of the data reception units and that transfer data received by the data reception units;
a data storage unit that receives and stores the data transferred from the plurality of data transfer units;
has
The plurality of data receiving units perform time management that is synchronized with each other, and add a timestamp representing the time based on the time management to the received data, and identify the data and the identification information of the data. transmitting association information associating the timestamp with the data transfer unit;
The data transfer unit receives the data and the association information from the data reception unit, temporarily stores the association information, and transfers the data and the association information to the data storage unit, canceling the temporary storage of the association information after confirming that the data has been saved in the data storage unit;
Data collection system. The data storage unit sets the time indicated by the timestamp in the association information as the creation time of the data, and stores the data in association with the creation time.
The data collection system according to claim 1. When the data transfer unit encounters a failure after receiving the data and the association information and restarts from the failure, the data transfer unit transmits a response requesting retransmission of the data to the data source via the data reception unit, and retaining the association information and using the timestamp in the association information for retransmitted data;
The data collection system according to claim 1. The data transfer unit temporarily stores the data together with the association information, transmits a response indicating that the data has been saved to the data source via the data reception unit, and transfers the association information and transferring the data to the data storage unit, confirming that the data has been stored in the data storage unit, and then canceling temporary storage of the association information and the data;
The data collection system according to claim 1. The plurality of data receiving units perform mutually synchronized time management using a Global Positioning System or Precision Time Protocol.
The data collection system according to claim 1. a plurality of data reception sections that share the processing of receiving data from a data source; and a plurality of data transfer sections that are provided corresponding to each of the data reception sections and that transfer the data received by the data reception sections. , a data storage unit that receives and stores the data transferred from the plurality of data transfer units, the data collection method in a data collection system comprising:
The plurality of data receiving units perform time management that is synchronized with each other, and add a timestamp representing the time based on the time management to the received data, and combine the data with identification information of the data. transmitting association information associating the timestamp with the data transfer unit;
The data transfer unit receives the data and the association information from the data reception unit, temporarily stores the association information, and transfers the data and the association information to the data storage unit, canceling the temporary storage of the association information after confirming that the data has been saved in the data storage unit;
Data collection methods. the data storage unit sets the time indicated by the timestamp in the association information as the creation time of the data, and stores the data in association with the creation time;
The data collection method according to claim 6. If the data transfer unit encounters a failure after receiving the data and the association information and restarts from the failure, the data transfer unit transmits a response requesting retransmission of the data to the data source via the data reception unit, and The mapping information is retained and the timestamp in the mapping information is used for retransmitted data.
The data collection method according to claim 6. a plurality of data reception sections that share the processing of receiving data from a data source; and a plurality of data transfer sections that are provided corresponding to each of the data reception sections and that transfer the data received by the data reception sections. , a data storage unit that receives and stores the data transferred from the plurality of data transfer units, the data collection program for causing a computer to operate a data collection system,
The plurality of data receiving units perform time management that is synchronized with each other, and add a timestamp representing the time based on the time management to the received data, and combine the data with identification information of the data. transmitting association information associating the timestamp with the data transfer unit;
The data transfer unit receives the data and the association information from the data reception unit, temporarily stores the association information, and transfers the data and the association information to the data storage unit, A data collection program for causing the computer to cancel temporary storage of the association information after confirming that the data has been stored in the data storage unit. causing a computer to execute, in which the data storage unit sets the time indicated by the timestamp in the association information as the creation time of the data, and stores the data in association with the creation time;
The data collection program according to claim 9. If the data transfer unit encounters a failure after receiving the data and the association information and restarts from the failure, the data transfer unit transmits a response requesting retransmission of the data to the data source via the data reception unit, and causing a computer to hold association information and use a time stamp in the association information for retransmitted data;
The data collection program according to claim 9.

Images