JP5570469B2 - Distributed data management system and method - Google Patents

Description

  The present invention relates to a distributed data management system and method, and in particular, collects and accumulates time series data (data stream) generated from a web camera, a sensing device, a portable terminal with a GPS function, and the like on a network for use. The present invention relates to a non-disruptive distributed data management system and method for executing processing in a form suitable for a user's purpose and providing results.

  In response to a connection request from an append client, the append section is paid out, an information section that manages the process configuration in the system, an append section that receives and queues data appended from the append client, and a record that is extracted from the queue In a network environment in which filter units that perform various processes are distributed and distributed, there is a distributed data management method for the purpose of load distribution, scale-out, and availability improvement (see, for example, Non-Patent Document 1). In the method of Non-Patent Document 1, it is realized by a process configuration in which each of the appending unit and the filter unit is distributed, and each server has one process of port mapper that manages the appending unit and the filter unit in the server. Multicasts the process configuration information that records the process configuration in the system, distributes it to all processes in the system via the port mapper, and checks the survival status of the process and the addition status of new processes. In the appending part, a record ID and time stamp for management are assigned to the record input from the appending client, and then the record is accumulated in the queue. In the filter part, from the process configuration information received by multicast from the information part Acquire a list of connectable appending parts, select one appending part by autonomous distributed data acquisition processing (for example, round robin algorithm), connect, acquire a record with information source ID from the queue (Pull), The step of performing processing determined according to the information source ID is repeated.

  As described above, in the method of Non-Patent Document 1, the filter unit acquires data from the appending unit at a convenient timing (Pull) and performs processing, so the process can be terminated at its own timing, and a new A filter unit can be added at any timing to participate in the system. Therefore, if a new filter unit is added in advance and the operating (old) filter unit is stopped, the filter unit can be replaced (switched over) without stopping the processing flow.

“Design and Evaluation of Write-once / Reference Data Management System”, Akama et al., IPSJ Transactions Vol.49 No.2, February 2008, pp749-764.

  However, in the above-mentioned method of Non-Patent Document 1, the appending unit receiving data from the appending client stops the appending unit as long as data writing from the appending client does not stop, or writes to another appending unit. Cannot be switched (switched over) (cannot cope with planned server shutdown due to OS version upgrade or patch application).

  In addition, since there is no means for suppressing data writing from the appending client in the appending unit, writing of data with a large data size from the appending client in a situation where the free memory capacity of the server operating the appending unit is insufficient. If there is, it is possible to cause a server error or a server stop.

  The object of the present invention is to collect and accumulate time series data (data stream) generated from a web camera, a sensing device, a portable terminal with a GPS function, etc. on a network, and to form a data suitable for the purpose of the user. It is to realize a distributed data management system capable of executing processing without interruption.

In order to achieve the above object, the present invention provides
Append client (AC) that accepts and appends data from information sources that generate time-series data such as web cameras, sensing devices, and mobile devices with GPS functions,
An appending unit (Q) that includes a data storage device and receives and queues data appended from the appending client (AC);
A filter unit (F) that acquires data from the appending unit (Q) and executes a predefined process;
In a network environment where there is an information part (I) that pays out the appending part (Q) in response to a connection request from the appending client (AC),
The write-once client (AC) makes a connection request to the information part (I) by the connection request means, acquires connection information (IP address, port number, etc.) of the write-once part (Q) as the connection destination together with the information source ID, Add data to the appending part (Q) by data sending means,
The appending part (Q) receives the data transmitted from the appending client (AC) by the data accepting means, assigns the record ID and time stamp for management, and stores them in the data storage device in FIFO order.
The filter unit (F) performs autonomous distributed data acquisition processing (for example, a round robin algorithm) by the data acquisition unit, selects one additional recording unit (Q), and obtains information from the data storage device of the additional recording unit (Q). Acquire data with ID (Pull) and repeat the steps of performing predefined processing for each information source ID by the data processing means,
The information part (I) receives a connection request from the additional writing client (AC) by the connection request receiving means, selects at least one additional writing part (Q) from the unused additional writing parts (Q), and 1 Return one information source ID and connection information (IP address, port number, etc.) of at least one additional writing part (Q),
When the appending unit (Q) receives data from the appending client (AC) using the status notification means, the appending client (AC) is notified of itself (the appending unit (Q)) or the switchover state of the server on which it operates or the data reception standby state. By notifying, time series data (data stream) generated from web camera, sensing device, mobile terminal with GPS function, etc. is collected and accumulated on the network and processed in a form that suits the user's purpose. Provide a non-disruptive distributed data management system that executes and provides results.

  As described above, according to the present invention (Claims 1 and 2), time-series data (data stream) generated from a web camera, a sensing device, a GPS-equipped mobile terminal, etc. every moment is added to the network. It is possible to realize a non-disruptive distributed data management system that collects and accumulates in (Q), and the filter unit (F) executes processing in a form that suits the user's purpose and provides the result.

  According to the present invention (Claims 3 and 4), the appending unit (Q) is in a switchover state with respect to the appending client (AC) when the appending unit (Q) receives data from the appending client (AC). The notification client (AC) makes a request to issue a new appending part (Q), and resumes appending data to the added appending part (Q). A management system can be realized.

  According to the present invention (Claim 5), the appending part (Q) is in a data reception standby state with respect to the appending client (AC) when the appending part (Q) accepts data from the appending client (AC). Since the append client (AC) waits for additional data to be added to the appending part (Q) until the specified condition is met, a server error due to the memory of the server on which the appending part (Q) is operating, etc. And a non-stop distributed data management system that suppresses server stoppage.

1 is a configuration diagram of a non-stop distributed data management system according to an embodiment of the present invention. FIG. 1 is a configuration example of a non-stop distributed data management system according to an embodiment of the present invention. It is a flowchart of operation | movement of the write-once client (AC) of one Embodiment of this invention. It is a flowchart of operation | movement of the postscript part (Q) of one Embodiment of this invention. It is a flowchart of operation | movement of the filter part (F) of one Embodiment of this invention. It is a flowchart of operation | movement of the information part (I) of one Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram of a non-stop distributed data management system according to an embodiment of the present invention.

  The non-stop distributed data management system shown in FIG. 1 includes an additional recording client (AC) 100, an additional recording unit (Q) 200, a filter unit (F) 300, and an information unit (I) 400.

  The additional recording client (AC) 100 includes a connection request unit 110 and a data transmission unit 120, the additional recording unit 200 includes a data reception unit 210, a data storage device 220, and a state notification unit 230, and the filter unit (F) 300 includes The data acquisition unit 310 and the data processing unit 320 are included, and the information unit (I) 400 includes a connection request reception unit 410.

  In the non-stop distributed data management system of this embodiment, the append client (AC) 100 receives data from an information source that generates time series data such as a web camera, a sensing device, and a mobile terminal with a GPS function, and requests connection. As a result of the connection request from the unit 110 to the information unit (I) 400, the data transmission unit 120 adds data to the additional recording unit (Q) 200 that has been paid out. In the appending unit (Q) 200, the data receiving unit 210 queues the data appended from the appending client (AC) 100 to the data storage device 220 mounted as a queue or the like in the order of FIFO. In the filter unit (F) 300, the data acquisition unit 310 acquires data from the data storage device 220, and the data processing unit 320 repeats the steps of executing a process defined in advance for each information source ID.

  In the distributed data management system of this embodiment, each process of the appending unit (Q) 200 holds the data storage device 220 managed by itself, but the data storage device is placed outside the appending unit (Q) process. It is also possible to arrange. Moreover, it is also possible to employ a configuration in which a plurality of information units (I) 400, additional recording clients (AC 100), additional recording units (Q) 200, and filter units (F) 300 are arranged.

  FIG. 2 is a configuration example of a non-stop distributed data management system according to an embodiment of the present invention.

  The additional recording unit (Q) 200 can be arranged in one process or a plurality of processes on the physical server. The filter unit (F) 300 can be arranged in one process or in a plurality of processes on the physical server. It is also possible to arrange the processes of the appending unit (Q) 200 and the filter unit (F) 300 in a mixed manner on the same server.

  A process called a port mapper (P) is arranged on each physical server to manage the additional recording unit (Q) 200 and the filter unit (F) 300, and the additional recording unit (Q) 200 and the filter unit (F) 300 under the management. It is also possible to adopt a configuration in which various commands (for example, start, stop, existence confirmation, execution of specified command, etc.) are notified to each process.

  In the present embodiment, the information unit (I) 400 is arranged on a different physical server. However, the information unit (I) 400 is arranged on the same server as the appending unit (Q) 200 and the filter unit (F) 300. It is also possible.

  FIG. 3 is a flowchart of the operation of the write-once client (AC) according to an embodiment of the present invention.

  In the flowchart of FIG. 6, when the additional recording client (AC) 100 makes a connection request to the information unit (I) 400 by the connection requesting unit 110 and when the data transmission unit 120 adds data to the additional recording unit (Q) 200. Shows the operation.

  Step 101) The connection request unit 110 of the additional recording client (AC) 100 makes a connection request for paying out the additional recording unit (Q) to the information unit (I) 400.

  Step 102) The data transmitting unit 120 of the additional recording client (AC) 100 acquires connection information (IP address, port number, etc.) of the additional recording unit (Q) from the information unit (I) 400 together with the information source ID.

  Step 103) If there is no data from the information source, repeat Step 103, and if there is data, go to the next step.

  Step 104) The data transmitting unit 120 of the additional recording client (AC) 100 adds data to the additional recording unit (Q) 200 together with the information source ID.

  Step 105) If there is a switchover status notification from the additional recording section (Q) 200, the process proceeds to the next step, and if not, the process proceeds to step 109.

  Step 106) If the switchover status notification is received, if there is a notification of the switchover destination IP address, the process proceeds to the next step, and if not, the process proceeds to Step 108.

  Step 107) The switchover destination IP address is notified, a connection request is made to the information unit (I) 400, and the process returns to Step 102.

  Step 108) The connection request unit 110 makes a connection request to the information unit (I) 400 by designating the IP address of the additional recording unit (Q) 200 that has notified the switchover state, and returns to Step 102.

  Step 109) If there is a data reception standby state notification from the additional writing section (Q) 200, the process proceeds to the next step, and if not, the process returns to Step 103.

  Step 110) Wait until the designated time elapses or until the designated server state is changed (for example, until the designated free memory capacity is reached), and return to Step 103.

  Next, an operation related to the additional recording unit (Q) 200 will be described.

  FIG. 4 is a flowchart of the operation of the additional recording section (Q) according to the embodiment of the present invention.

  The operation when the appending unit (Q) 200 accepts data from the appending client (AC) 100 by the data accepting unit 210 and stores it in the data storage device 220, and accepts data from the appending client (AC) 100 by the status notifying unit 230 The operation at the time of notifying itself (additional part (Q) 200) or the state of the server in which it operates to the additional recording client (AC) 100 is shown.

  Step 201) The data reception unit 210 proceeds to the next step if the data storage device 220 has a free space, and proceeds to step 207 if not.

  Step 202) The data receiving unit 210 receives the data additionally written together with the information source ID from the additional writing client (AC) 100.

  Step 203) The data reception unit 210 gives a record ID for management.

  Step 204) The data reception unit 210 gives a time stamp at the time of reception.

  Step 205) The data receiving unit 210 stores the data in the data storage device 220 in the FIFO order.

  Step 206) If a stop command is notified through a command or the like, the process ends. If not, the process proceeds to Step 208.

  Step 207) Exception processing such as outputting an error log is executed and the processing returns to Step 201.

  Step 208) If it is a switchover state, proceed to the next step, and if it is not a switchover state, proceed to Step 213.

  Step 209) If there is a notification of the switchover destination IP address, go to the next step, otherwise go to Step 211. The processing from step 209 to step 212 is processing when the IP address of the switchover destination server is specified (corresponding to claim 3).

  Step 210) The switchover destination IP address is designated to the write-once client (AC) 100 to notify the switchover state, and the process proceeds to Step 212.

  Step 211) The switchover information is notified to the write-once client (AC) 100 by designating its own IP address.

  Step 212) If there is no record in the data storage device 220 of itself, the process ends. If it does exist, Step 212 is repeated.

  Step 213) If it is in the data reception standby state, the process proceeds to the next step, and if it is not in the data reception standby state, the process returns to Step 201.

  Step 214) The write-once client (AC) 100 is notified of the data reception standby state, and the process returns to Step 201. This is the data reception standby state notification process in this step (corresponding to claim 5).

  Next, the operation of the filter unit (F) 300 will be described.

  FIG. 5 is a flowchart of the operation of the filter unit (F) according to the embodiment of the present invention.

  The filter unit (F) 300 performs autonomous distributed data acquisition processing (for example, a round robin algorithm) by the data acquisition unit 310 to select one additional recording unit (Q) 200, and the data storage device of the additional recording unit (Q) 200 An operation when acquiring (Pull) data together with an information source ID from 220 and an operation when performing a process defined in advance for each information source ID on the data acquired by the data processing unit 310 are shown.

  Step 301) The data acquisition unit 310 selects one additional recording unit (Q) 200 by an autonomous distributed data acquisition process (for example, a round robin algorithm).

  Step 302) If there is a record in the data storage device 220 of the additional writing section (Q) 200, the process proceeds to the next step, and if not, the process returns to step 301.

  Step 303) The data acquisition unit 310 acquires the data together with the information source ID from the data storage device 220 of the additional recording unit (Q) 200 in the FIFO order.

  Step 304) The data processing unit 320 performs a predefined process on the acquired data for each information source ID.

  Step 305) If a stop command is notified through a command or the like, the process ends. If not, the process returns to Step 301.

  Next, the operation of the information unit (I) 400 will be described.

  FIG. 6 is a flowchart of the operation of the information unit (I) in one embodiment of the present invention.

  Below, the information unit (I) 400 receives a connection request from the write-once client (AC) 100 by the connection request reception unit 410, and returns connection information and one information source ID of at least one additional write-in unit (Q). The operation is shown.

Step 401) If there is a connection request from the write-once client (AC) 100, the connection request receiving unit 410 proceeds to the next step, and if not, returns to Step 401. As a connection request,
(1) Request for connection of IP address designation of appending part (Q) 200;
(2) Connection request for specifying the IP address of the switchover destination;
(3) Connection request without IP address designation;
There is.

If step 402) a first recordable portion (Q) 200 ₁ of the connection request notifies the IP address advances to the next step, the process proceeds to step 404 otherwise.

Step 403) connection when the request is the above described (1), the first write-once portion (Q) a second write-once section operating outside 200 ₁ IP address (Q) 200 ₂ of at least one connection The information and one information source ID are returned to the write-once client (AC) 100, and the process returns to step 401.

  Step 404) If the connection request notifies the switchover destination IP address, the process proceeds to the next step; otherwise, the process proceeds to Step 406.

Step 405) When the connection request is the above-mentioned (2), adds the second recordable section (Q) 200 ₂ of at least one or more connection information and one source ID that runs the IP address notified client (AC) Return to 100 and return to step 401.

Step 406) connection If the request is above (3) is to return at least one or more connection information and one source ID of the unused appending section (Q) 200 _n to append the client (AC) 100 Return to step 401.

  The operation of the constituent elements of each of the additional recording client (AC) 100, additional recording unit (Q) 200, filter unit (F) 300, and information unit (I) 400 of the above distributed data management system is constructed as a program. It can be installed in a computer and executed, or distributed via a network.

  The present invention is not limited to the above-described embodiment, and various modifications and applications are possible within the scope of the claims.

100 Append client (AC)
110 connection request unit 120 data transmission / reception unit 200 additional recording unit (Q)
210 Data reception unit 220 Data storage device 230 Status notification unit 300 Filter unit (F)
310 Data Acquisition Unit 320 Data Processing Unit 400 Information Unit (I)
410 Connection request acceptance unit

Claims (5)

Append client (AC) that accepts and appends data from information sources that generate time-series data from time to time,
An appending unit (Q) that includes a data storage device and receives and queues data appended from the appending client (AC);
A filter unit (F) that acquires data from the appending unit (Q) and executes a predefined process;
A distributed data management system in a network environment in which an information unit (I) that issues a write-out unit (Q) in response to a connection request from the write-once client (AC) exists,
The append client (AC)
A connection request is made to the information section (I), and connection information (IP address, port number, etc.) of the additional recording section (Q) as a connection destination is acquired together with the information source ID , and from the additional recording section (Q) Based on the notified status information, connection request means for making a connection request to the information unit (I) ,
Data transmitting means for receiving data from the information source and adding data to the appending unit (Q) together with the information source ID;
The appending part (Q)
Data receiving means for receiving data added by the additional recording client (AC) and giving a record ID for management, giving a time stamp of reception, and queuing the data in a data storage device in FIFO order;
A status notification means for notifying the additional recording client (AC) itself (additional recording section (Q)) or a switchover state or a data reception standby state of the server on which the additional recording client (AC) operates when receiving data from the additional recording client (AC);
Comprising
The filter part (F)
Autonomous distributed data acquisition processing is performed from at least one additional recording section (Q) to select one additional recording section (Q), and data is acquired together with the information source ID from the data storage device of the additional recording section (Q) ( Pull) data acquisition means,
Data processing means for executing a process defined in advance for each acquired information source ID for each information source ID,
The information part (I)
Accepting a connection request from the additional recording client, selecting at least one additional recording section (Q) from unused additional recording sections (Q), one information source ID and at least one additional recording section ( Q) A non-stop distributed data management system comprising connection request accepting means for returning connection information of Q). An appending client (AC) that accepts data from an information source that generates time series data from time to time and a data storage device, and an appending unit that receives and queues the data appended from the appending client (AC) ( Q) and
A filter unit (F) that acquires data from the appending unit (Q) and executes a predefined process;
In a network environment in which there is an information unit (I) that issues the additional recording unit (Q) in response to a connection request from the additional recording client (AC),
The write-once client (AC) makes a connection request to the information part (I) by a connection request means, and obtains connection information (IP address, port number, etc.) of the write-once part (Q) as a connection destination together with the information source ID. Then, add data to the additional recording section (Q) by the data transmission means,
The appending part (Q) receives the data transmitted from the appending client (AC) by the data receiving means, stores the management record ID and time stamp, and then stores them in the data storage device in FIFO order,
The filter unit (F) performs an autonomous distributed data acquisition process by a data acquisition unit, selects one additional recording unit (Q), and acquires data together with an information source ID from the data storage device of the additional recording unit (Q). (Pull) and repeat the step of performing a predefined process for each information source ID by the data processing means,
The information unit (I) receives a connection request from the additional recording client (AC) by a connection request receiving unit, and selects at least one additional recording unit (Q) from unused additional recording units (Q). , Return the connection information (IP address, port number, etc.) of one information source ID and the at least one additional writing part (Q),
The additional recording unit (Q) is switched to the additional recording client (AC) as status information of itself (additional recording unit (Q)) or a server on which it operates when data is received from the additional recording client (AC) by the status notification unit. Over status or data reception standby status ,
If the status information notified from the additional recording unit (Q) is a switchover state, the additional recording client (AC) notifies the information unit (I) of the IP address of the switchover destination and makes a connection request. <br/> A distributed data management method characterized by the above. When a switchover command is notified to the first appending part (Q),
The first additional recording unit (Q) shifts to the switchover state, and the additional recording unit (Q) notifies the additional recording unit (Q) from the first additional recording unit (Q) by the state notification unit of the additional recording unit at the time of the next additional data recording from the additional recording client (AC). Notifying the client (AC) that the IP address of itself (additional part (Q)) and the first additional part (Q) are in a switchover state,
The postscript client (AC) that has received the notification notifies the information unit (I) of the IP address of the first postscript unit (Q) by the connection request means and makes a new connection request,
The information unit (I) is at least one second appending unit (Q) that operates other than the IP address of the first appending unit (Q) notified from the appending client (AC) by the connection request accepting unit. Return one or more connection information and one information source ID to the append client (AC),
3. The distributed data according to claim 2, wherein the additional recording client (AC) that has received the payout from the second additional recording unit (Q) resumes the additional data recording by the data transmission unit with respect to the second additional recording unit (Q). Management method. When you specify the switchover destination IP address when notifying the switchover command to the first appending part (Q),
The first additional recording unit (Q) shifts to the switchover state, and the additional recording unit (Q) notifies the additional recording unit (Q) from the first additional recording unit (Q) by the state notification unit of the additional recording unit at the time of the next additional data recording from the additional recording client (AC). Notifying the client (AC) that the switchover destination IP address and the first appending part (Q) are in the switchover state,
The append client (AC) that has received the notification notifies the information unit (I) of the IP address of the switchover destination by the connection request means and makes a new connection request,
The information part (I) is at least one connection information (IP address, port number) of the second additional recording part (Q) operating at the IP address notified from the additional recording client (AC) by the connection request accepting means. Etc.) and one information source ID to the append client (AC),
3. The distributed data according to claim 2, wherein the additional recording client (AC) that has received the payout from the second additional recording unit (Q) resumes the additional data recording by the data transmission unit with respect to the second additional recording unit (Q). Management method. When a data reception standby instruction is sent to the first appending part (Q),
The first additional recording section (Q) shifts to the data reception standby state, and the additional recording client from the first additional recording section (Q) by the state notification means of the additional recording section at the next additional data recording from the additional recording client (AC) (AC) is notified that the first appending part (Q) is in a data reception standby state,
The additional recording client (AC) that has received the notification resumes data addition to the first additional recording unit (Q) after a lapse of a specified time or after a change to a specified server state. Distributed data management method.

Images