JP2014225111A - Data processing device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent all data processing programs from automatically concentrating on a buffer device in which a congestion state occurs.SOLUTION: When congestion occurs in a buffer device, a data processing device inhibits all data processing programs from concentrating on the buffer device in which the congestion occurs by mixing data processing programs that operates according to a rule to preferentially select the buffer device in which the congestion occurs and data processing programs that operates according to a rule to select buffer devices in the same order as before even when congestion occurs.

Description

  The present invention relates to a data processing apparatus, method, and program, and more particularly, to a data processing apparatus, method, and program for optimizing data processing for autonomously processing data stored in a buffer device in a stream processing system.

  There is a system in which a plurality of data processing programs process stream data stored in a plurality of buffer devices in parallel.

  Each data processing program needs to determine from which buffer device data is acquired in order to perform data processing in parallel. The rules for determining which buffer device to select to acquire and process data (hereinafter referred to as “PULL strategy”) are used by the data processing program to select all buffer devices in order, A method of selecting a buffer device can be considered.

  If the processing capacity of the data processing program that processes the data amount stored in the buffer device is sufficient, these simple pull strategies also operate effectively, and parallel processing can be executed without any problems.

  In addition, when there is a lot of data input to some buffer devices and a congested state occurs, the autonomously distributed data processing device is stored in the congested buffer device to prevent the buffer device from failing. Data needs to be processed intensively to eliminate congestion. The data processing device monitors the state in the system, and when congestion occurs, this is realized by switching the internal state so that the corresponding buffer device is processed intensively (for example, patents). Reference 1).

  FIG. 1 is a flowchart of the operation of a data processing program in the prior art.

  The data processing program determines whether or not congestion has occurred (step 10). If no congestion has occurred (FIG. 2), the acquisition buffer device is determined by a prescribed procedure (step 11). Then, data is acquired from the buffer device (step 13), and data processing is performed (step 14). When congestion occurs (FIG. 3), the congested buffer device is determined as the acquisition buffer device (step 12), data is acquired from the buffer device (step 13), and data processing is performed (step 13). 14).

  As shown in FIG. 3, when congestion occurs in the buffer device A, the buffer device A notifies the data processing device of the congestion, and the data processing device uses the data in the buffer device A to eliminate the congestion as soon as possible. Is to be obtained and processed with priority. At this time, depending on the PULL strategy, the processing of the buffer device B may not be performed until the congestion of the buffer device A is resolved.

  Various PULL strategies can be considered, and an example is shown below.

  PULL_RR: Round-robin PULL strategy. When this strategy is specified, data acquisition is attempted in order from a plurality of buffer devices.

  -PULL_RAND: When this strategy is specified, when selecting a plurality of buffer devices, a pseudo random number R is generated, and the pseudo random number R is divided by the number n of buffer devices (R% n, etc.). Determine from which buffer device to try to get data.

  PULL_RR_JAM: This is a PULL strategy that performs the same operation as PULL_RR when congestion does not occur, but tries data preferentially from the congested buffer device when congestion occurs.

  PULL_BIAS_RR: A round-robin PULL strategy that acquires each buffer device at a specified rate, and separately sets which buffer device is acquired at which rate.

  The choice of PULL strategy should be chosen according to the various requirements of stream processing. An example of the requirement is latency until data processing is completed. In the case where the latency is to be kept as low as possible, it is most important to secure a sufficient number so that the data processing program does not run out.

  However, even if the data processing program itself is not insufficient, when there are a large number of buffer devices, the timing at which the data processing program selects (circulates) the buffer device is delayed, and latency is increased. .

  When it is necessary to perform processing within a certain period when it is added to the buffer device, there is a risk that a specific buffer device will not be selected indefinitely if a PULL strategy such as PULL_RAND is used. When a PULL strategy such as PULL_RR_JAM corresponding to congestion is selected, there is a risk that the buffer device is not selected when another buffer device is congested.

  In such a case, a PULL strategy such as PULL_RR that reliably cycles through all buffer devices is more suitable.

  As another case, there is a buffer device that usually receives a small amount of data, but a large amount of data arrives at regular or irregular intervals, and this buffer device must be preferentially processed. There is such a requirement.

  In this case, with the PULL strategy such as PULL_RAND or PULL_RR that tries to acquire data at a certain rate for each buffer device, there is a risk that it will take time to process the data of the buffer device that a large amount of data has reached. is there.

  In such cases, using a PULL strategy that supports congestion, such as PULL_RR_JAM, can efficiently select a buffer device with a large amount of additional data, and shorten the time until congestion is eliminated. is there.

  In the opposite case, it is necessary to maintain the performance of the buffer device data that is always input, but usually there are few inputs, so that a large amount of data arrives temporarily at regular or irregular intervals. In some cases, there is a buffer device that performs processing on a best effort basis.

In such a case, if only the PULL strategy corresponding to congestion is used, such as PULL_RR_JAM, the data processing program is occupied by the response to the processing of the congested buffer device, and the buffer device that is always input is not selected, There is a risk that data processing will be delayed.
In such a case, a PULL strategy such as PULL_RR or PULL_RAND that selects all buffer devices is more suitable.

  As described above, it can be understood that the requirement cannot be satisfied depending on the situation when data processing is performed with only one PULL strategy. Further, it can be seen that a system in which a plurality of input characteristics and latency requirements up to processing are mixed cannot always satisfy the requirements.

Japanese Patent No. 4764300

  However, in the above prior art, since each data processing program is autonomously distributed, each data processing program concentrates on a congested buffer device. As a result, an excessively optimized state occurs in which the number of data processing programs that execute processing of the buffer device that is not congested is greatly reduced and processing is delayed.

  A conventional problem will be described with reference to FIG.

  When data input concentration (congestion) occurs, the buffer device A notifies the data processing device of the data input concentration in order to prevent buffer overflow (FIG. 4A).

  Upon receiving the notification, the data processing program of the data processing device preferentially acquires data from the buffer device where data input is concentrated and executes the data processing (FIG. 4B).

  The above-mentioned response to congestion is a function necessary for eliminating congestion in a short time. However, accesses from data processing programs are excessively concentrated, and buffers other than the buffer device where data input is concentrated Latency increases because device data cannot be processed.

  The present invention has been made in view of the above points, and in a system that processes stream data stored in a plurality of buffer devices in parallel by a plurality of autonomously distributed data processing programs, when a congestion state occurs, the present invention Another object of the present invention is to provide a data processing apparatus, method and program capable of preventing all data processing programs from concentrating on a congested buffer device.

In order to solve the above problem, according to one aspect of the present invention, in a stream processing system, a plurality of autonomously processing data accumulated in a plurality of buffer devices that temporarily hold input data from a data source A data processing apparatus having the data processing program of
When congestion occurs in the buffer device, a data processing program that operates according to a rule that preferentially selects a buffer device in which congestion has occurred, and even if congestion occurs, the buffer is processed in the same order as before the congestion occurs There is provided a data processing apparatus having means for controlling both of data processing programs that operate according to a rule for selecting an apparatus.

  As described above, the present invention retains state management information and sets a condition corresponding to the congestion state of the buffer device in advance for the autonomously distributed data processing program. By setting this condition, not only the data processing program that concentrates on the congested buffer device but also the data processing program that does not concentrate on the buffer device even if a congested state occurs is mixed with the congested state in the buffer device. It is possible to prevent all data processing programs from autonomously concentrating on the congested buffer device when a congested state occurs by preventing the processing of the buffer device that has not been performed.

It is a flowchart of operation | movement of the data processing program of a prior art. It is a figure which shows the state in which the congestion in the prior art has not generate | occur | produced. It is a figure which shows the state which congestion generate | occur | produced in a prior art. It is a figure for demonstrating the subject of a prior art. It is a system configuration example on the premise of the present invention. It is a system configuration figure in a 1st embodiment of the present invention. It is a flowchart of the data reception in the buffer apparatus in the 1st Embodiment of this invention, and the notification process of the congestion state to a data processor. It is a flowchart of the whole data processing in the data processing in the 1st Embodiment of this invention. It is a flowchart of the data processing in the data processor in the 1st Embodiment of this invention. It is a flowchart of the buffer apparatus selection in the data processor in the 1st Embodiment of this invention. It is a flowchart of the data acquisition process in the data processor and buffer apparatus in the 1st Embodiment of this invention. It is a system block diagram in the 2nd Embodiment of this invention. It is a flowchart of the data reception in the buffer apparatus in the 2nd Embodiment of this invention, and the notification process of the congestion state to a data processor. It is a flowchart of the data processing of the whole data processing in the data processing apparatus in the 2nd Embodiment of this invention. It is a flowchart of the data processing in the data processor in the 2nd Embodiment of this invention. It is a flowchart of the buffer apparatus selection in the data processor in the 2nd Embodiment of this invention. It is a flowchart of the data acquisition process in the data processor in the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  Before describing the embodiment of the present invention, a system on which the present invention is based will be described.

  FIG. 5 is a system configuration example on the premise of the present invention.

  The present invention is a PULL-type stream distributed processing system in which a portion (buffer device 10) that temporarily holds input data from a data source 20 and a data processing device 100 are separated, and the buffer device 10, data The processing apparatus 100 exists on a plurality of hardware.

  In addition, a plurality of data processing programs exist in the form of processes or threads on the hardware of one data processing apparatus 100 (hereinafter, the data processing program existing in the form of a process or thread is referred to as a “data processing program”). "). The data processing program autonomously polls the buffer device in a certain order (PULL strategy), and if data is held, acquires it and performs processing.

  Since the data input rate from the data source 20 is not always constant, when the data input is congested, the data processing program can share the fact that the data is congested in the system so that the data processing program of the congested buffer device performs data processing. Is given priority.

  According to the present invention, the above processing is performed in the PULL-type stream processing system without deteriorating the latency until a plurality of data sources are processed.

[First embodiment]
In the present embodiment, the data processing program in the data processing device selects a buffer device for each processing based on the selected PULL strategy. In the present embodiment, the PULL strategy is determined when the data processing program in the data processing apparatus is activated.

  FIG. 6 shows a system configuration in the first embodiment of the present invention.

The system shown in the figure includes buffer devices 10A and 10B, a data source 20, and data processing devices 100 ₁ and 100 ₂ .

  Each buffer device 10 includes a storage unit 11, threshold information 12, and a communication unit 13, and each data processing device 100 includes a communication unit 110, a PULL strategy configuration setting unit 120, a buffer device selection unit 130, a state management unit 140, A data processing program 150 is included. The buffer device selection unit 130 selects the buffer device 10 based on the PULL strategy held inside.

  The state management unit 140 has a list of buffer devices (buffer device identification information) and a list of congestion buffer devices (congestion buffer device identification information).

  Further, it is assumed that a program (hereinafter referred to as an additional recording client 21) for additionally writing data generated in the data source 20 to the buffer device 10 exists in the preceding stage of the buffer device 10.

  Each data processing program 150 selects the buffer device 10 that acquires (fetches) data for each processing based on the selected pull strategy, and acquires data. The PULL strategy in this embodiment uses both a method that sets a congested buffer device as an acquisition destination when congestion occurs and a method that targets all buffer devices even when congestion occurs.

  Hereinafter, the operation of the present embodiment will be described with reference to the flowcharts of FIGS.

  First, the buffer device 10 notifies the data processing device 100 when each data D is additionally recorded from an external additional recording program or the like and is congested. The flow of this operation is shown in the flowchart of FIG.

  FIG. 7 is a flowchart of data reception in the buffer device and notification processing of the congestion state to the data processing device according to the first embodiment of the present invention.

  The start of the flowchart of FIG. 7 is described from the time when the above-described additional recording client transmits data to the communication unit 13. Data D is the target data of the stream processing generated at the data source 20, and data D at a certain time t is represented as Dt.

  The buffer device 10 receives Dt by the communication unit 13 (step 101). The communication unit 13 can receive with an arbitrary communication protocol, but normally the IP protocol is used. The Dt received by the communication unit 13 is temporarily stored in the storage unit 11 of the buffer device 10 (step 102) and waits for processing by the data processing device 100. The received data is stored in the storage unit 11 of the buffer device 10 in the same manner as the data Dt-1 before the arrival of the data Dt and the subsequent data Dt + 1. At this time, the storage unit 11 refers to the threshold information 12 (step 103), and if the amount of data stored in the storage unit at that time exceeds the threshold information, notifies the data processing device as a congestion notification. (Steps 104 and 105).

  As a method for specifying threshold information, the number of data D added or the total capacity of data D can be used.

  Although any protocol can be used for the congestion notification in step 105, the IP protocol is usually used. Furthermore, the congestion notification needs to include information for uniquely specifying which storage unit 11 of the buffer device 10 exceeds the threshold.

  As an example, in the implementation using the IP protocol, the congestion notification includes the IP address and communication port number of the congested buffer device. I can know.

  As another example, in this system, a correspondence table between buffer device ID information that can identify each buffer device 10 and the IP address and communication port number on which the buffer device operates is managed. The data processing device 100 can also know which buffer device is congested by specifying the IP address and communication port from the buffer device ID information on each data processing device 100 side including only ID information.

  When the data processing apparatus 100 receives the congestion notification (step 106), the data processing apparatus 100 stores information on the buffer apparatus 10 once congested in the state management unit 140 (step 107).

  The data processing device 100 acquires data from the buffer device 10 according to the contents of the state management unit 140 and the PULL strategy configuration setting unit 120 and performs data processing. The data processing sequence control in the data processing apparatus at this time is shown in the flowchart of FIG.

  FIG. 8 is a flowchart of the entire data processing in the data processing apparatus according to the first embodiment of the present invention.

  The start of the flowchart of FIG. 8 is described from the time when the data processing program 150 is started. The PULL strategy configuration is acquired at the start of the data processing program.

  When the data processing program 150 is activated (step 201), the PULL strategy configuration setting unit 120 buffers the data processing program 150 in any order according to the number of buffer devices 10 and the state of the state management unit 140. Whether to acquire the data of the device 10 (PULL strategy) is described and held in a memory (not shown) (step 202).

  The following is an example of a PULL strategy.

  (1) As an example of the PULL strategy, there is a round-robin PULL strategy (hereinafter, PULL_RR). When PULL_RR is specified, data is acquired sequentially from a plurality of buffer devices. When the identifiers of the buffer devices 10 are BUFF_1, BUFF_2... BUFF_n, the data processing program tries to acquire data in the order of BUFF_1, BUFF_2.

  (2) There is a random PULL strategy (hereinafter, PULL_RAND). When PULL_RAND is specified, when selecting a plurality of buffer devices 10, the data processing program 150 generates a pseudo random number R, and the remainder obtained by dividing the pseudo random number R by the number n of buffer devices (R% n, etc.) To determine from which buffer device the data acquisition is to be attempted. If the buffer device identifiers of the buffer devices are BUFF_1, BUFF_2... BUFF_n, and a = R% n, BUFF_a is selected and data acquisition is attempted.

  (3) There is a round robin PULL strategy (hereinafter, PULL_BIAS_RR) for acquiring each buffer device 10 at a specified rate. When PULL_BIAS_RR is designated, it is necessary to separately set which buffer device 10 is to be acquired at which rate. As an example, if the buffer device BUFF_1 and the buffer device BUFF_2 are set to acquire 1: 2, respectively, the data processing program 150 selects the buffer device three times and tries to acquire the data once. Attempt to acquire data, and twice, select BUFF_2 and try to acquire data.

  (4) When there is no information indicating the buffer device 10 that is congested in the state management unit 140, the same operation as the round-robin PULL strategy (PULL_RR) is performed, but the buffer device that is congested in the state management unit 140 In the case where information indicating the existence is stored, there is a PULL strategy (hereinafter referred to as PULL_RR_JAM) in which data acquisition is preferentially performed from a congested buffer device. If PULL_RR_JAM is specified and the buffer manager identifier is BUFF_1, BUFF_2 ... BUFF_j ... BUFF_n, and information indicating that BUFF_j is congested exists in the state management unit 140, data The processing program 150 tries to acquire data only from the buffer device BUFF_j.

  One or more data processing programs 150 exist on the data processing apparatus 100, and one or more data processing apparatuses 100 exist in the entire system, and each data processing program 150 acquires the data of the buffer device 10 and Attempt processing.

  In the system of the present invention, it is possible to specify a plurality of PULL strategies as PULL strategies. For example, two data processing programs (a data processing program PG_A and a data processing program PG_B) can be activated in one data processing apparatus, and individual designations such as PULL_RR for PG_A and PULL_RAND for PG_B can be performed.

  When the PULL strategy is determined, the activation of the data processing program 150 is completed (step 204), and the data processing flow loop (FIG. 9) is started (step 205).

  FIG. 9 is a flowchart of data processing in the data processing device according to the first embodiment of the present invention.

  The data processing program 150 makes a buffer device selection request to the buffer device selection unit 130 (step 301). The buffer device selection unit 130 selects a buffer device by the processing shown in FIG. 10, and returns it to the data processing program 150 (step 302). The data processing program 150 acquires the buffer device selection result (step 303), and performs the processing shown in FIG. 11 (step 304).

  In the loop of the data processing flow, as shown in FIG. 9, the buffer device selection unit 130 performs the buffer device selection flow (FIG. 10), and the data processing program 150 sequentially performs the data acquisition and processing flow (FIG. 11). carry out.

  FIG. 10 is a flowchart of buffer device selection in the data processing device according to the first embodiment of the present invention.

  In the buffer device selection flow shown in FIG. 10, the buffer device selection unit 130 branches according to the PULL strategy acquired by the PULL strategy configuration setting unit 120. As an example, an example is shown in which two data processing programs PG_A and PG_B operating with different PULL strategies are operating. In PG_A, PULL_RR_JAM is specified as PULL strategy 1 whose operation is changed according to information held by the state management unit. PULL strategy 2 specifies PULL_RAND that operates regardless of the information held by the state management unit. Each data processing program branches according to the designated PULL strategy (step 401).

  Since the data processing program PG_A is specified to operate in the PULL strategy 1, when the buffer device selection is started (step 402), the presence of the congested buffer device is confirmed with the state management unit 140. The buffer device list is acquired (step 403). If it is found that there is no information on the buffer device in which congestion has occurred in the buffer device list (step 404, No), one of the buffer devices is stored in the PULL_RR procedure for all buffer devices. The acquisition buffer device is selected (step 405). If it is found that congestion has occurred (step 404, Yes), the data acquisition destination buffer device is selected from the list of congested buffer devices (step 406).

  Since PG_B is specified to operate in PULL strategy 2 (steps 401 and 407), one of the buffer devices is selected as a data acquisition destination buffer device in the PULL_RR procedure for all buffer devices (steps 401 and 407). Step 408).

  The buffer device selection unit 130 returns the buffer device selection result to the data processing program 150 (step 409).

  FIG. 11 is a flowchart of data acquisition processing in the data processing device and the buffer device according to the first embodiment of the present invention.

  In the flowchart of FIG. 11, the communication unit 110 makes a data acquisition request to the buffer device 10 that has already been selected as the data acquisition destination buffer device in the process shown in FIG. 10 (step 501). If there is data to be returned, the buffer device 10 returns the data (steps 502 and 503). If the data does not exist, the buffer device 10 returns a response indicating that the data does not exist (steps 502 and 504). Only when data is returned, the data processing program executes data processing (steps 505 and 506). Data processing differs depending on the application of the system, but is not specified here.

[Second Embodiment]
In the present embodiment, although the PULL strategy is not clearly separated, a conditional example is shown in which a conditional branch is performed in the buffer device selection unit and a plurality of PULL strategies are mixed.

  In the data processing program according to the present embodiment, when there is no congestion, a plurality of buffer devices are searched from the buffer device while circulating through a determined PULL strategy (for example, random, order of identifier values of the buffer device). Acquire data and process it. On the other hand, if congestion occurs, the data processing program corresponds to congestion at a rate of α in the processing capacity in one system (0 ≦ α ≦ 1). For example, as one specific example, a method of increasing the data processing program corresponding to the congestion by increasing the alpha value based on the congestion duration time, or the congestion level (for example, the number of congested buffer devices) The value of α is increased based on (1) to increase the data processing programs corresponding to congestion step by step.

  FIG. 12 shows a system configuration in the second embodiment of the present invention.

  In the system shown in the figure, the buffer device 10 includes a storage unit 11, threshold information 12, and a communication unit 13 as in the first embodiment, and the data processing device 200 includes a communication unit 210, a buffer device selection unit, and the like. 230, a congestion handling setting unit 220, a state management unit 240, and a data processing program 250. Further, it is assumed that a program (hereinafter referred to as an additional recording client 21) for additionally writing data generated by the data source 20 to the buffer device 10 exists in the preceding stage of the buffer device 10.

  Each data processing program selects a buffer device from which data is acquired (fetched) and acquires data each time processing is performed based on the selected pull strategy. The flow of this operation will be described with reference to FIGS.

  FIG. 13 is a flowchart of data reception in the buffer device and notification processing of the congestion state to the data processing device according to the second embodiment of the present invention.

  The buffer device 10 notifies the data processing device 200 when each data D is additionally recorded from an external additional recording program or the like and is congested.

  The start of FIG. 13 is described from the time when the above-described additional recording client transmits data to the communication unit 13. Data D is the target data of the stream processing generated at the data source 20, and data D at a certain time t is represented as Dt.

  The buffer device 10 receives Dt by the communication unit 13 (step 601). The communication unit 13 can receive with an arbitrary communication protocol, but normally the IP protocol is used. The Dt received by the communication unit 13 is temporarily stored in the storage unit 11 of the buffer device 10 and waits to be processed by the data processing device 200 (steps 602 and 603). The received data is stored in the storage unit 11 of the buffer device 10 in the same manner as the data Dt-1 before the arrival of the data Dt and the subsequent data Dt + 1. At this time, the storage unit 11 refers to the threshold information (step 604), and if the amount of data stored in the storage unit 11 at that time exceeds the threshold information, the storage unit 11 sends a congestion notification to the data processing device 200. Notification is performed (steps 605 and 606).

  As a method for specifying threshold information, the number of data D added or the total capacity of data D can be used.

  Any protocol can be used for the congestion notification, but the IP protocol is usually used.

  Furthermore, the congestion notification needs to include information for uniquely specifying which storage unit 11 of the buffer device 10 exceeds the threshold. As an example, in the implementation using the IP protocol, the congestion notification includes the IP address and the communication port number of the buffer device 10 that is congested. I can know.

  As another example, in this system, a correspondence table between buffer device ID information that can identify each buffer device 10 and the IP address and communication port number on which the buffer device operates is managed. The data processing device 200 can know which buffer device is congested also by specifying the IP address and the communication port from the buffer device ID information on each data processing device 200 side including only the ID information.

  When receiving the congestion notification, the data processing device 200 stores information on the buffer device once congested in the state management unit 240.

  The data processing device 200 performs data processing by acquiring data from the buffer device 10 according to the contents of the state management unit 240 and the congestion handling setting. Data processing in the data processing apparatus 200 at this time will be described with reference to FIGS.

  FIG. 14 is a flowchart of data processing for the entire data processing in the data processing apparatus according to the second embodiment of the present invention.

  The start of FIG. 14 is described from the start of the data processing program 250. When the data processing program is activated (step 701), a data processing flow loop (the flow of FIG. 15) is started (step 702).

  In the loop of the data processing flow, as shown in FIG. 15, the data processing program 250 sequentially executes a buffer device selection flow (FIG. 16), data acquisition, and processing flow (FIG. 17).

  FIG. 16 is a flowchart of buffer device selection in the data processing device according to the second embodiment of the present invention.

  The buffer device selection unit 230 obtains a list of the buffer devices 10 from the state management unit 240 (step 801), and from this result, it is possible to know whether congestion has occurred and the buffer device in which congestion has occurred. In the buffer device selection flow shown in FIG. 16, as a result, when there is no congestion, the data processing program 250 selects the data acquisition destination buffer device for all the buffer devices. As the PULL strategy to be executed at this time, any PULL strategy such as the above-described PULL_RAND and PULL_RR can be set. If congestion occurs in the buffer device 10 (step 802, Yes), further congestion correspondence setting is acquired (step 803), and it is determined whether the data processing program should cope with the congestion (step 803). 804). At this time, it is necessary to adjust the α value indicating how much congestion should be handled (step 805).

  As an example of the adjustment method, the total number of buffer devices SUM_buff and the number of congested buffer devices SUM_jam are obtained from α = SUM_jam / SUM_buff.

  In this random number determination, a random number between 0 and 1 is generated by this α (step 806). If the random number is 0 or more and less than α, the data processing program selects a buffer device for a buffer device in a congested state. (Step 807). If the random number is not less than α and not more than 1, the data processing program 250 selects a buffer device for the entire buffer device (step 808). The buffer selection result is output to the data processing program 250 (step 809).

  The buffer device selection procedure is the same as the state where the PULL strategy corresponding to congestion and the PULL strategy not corresponding to congestion are mixed according to the state of the state management unit without explicitly mixing multiple PULL strategies. Function can be realized.

  FIG. 17 is a flowchart of data acquisition processing in the data processing device according to the second embodiment of the present invention.

  In the processing of FIG. 17, the communication unit 210 makes a data acquisition request to the buffer device 10 that has already been selected as the data acquisition destination buffer device in the processing of FIG. 16 (step 9901). If there is data to be returned, the buffer device 10 returns the data (step 902). If the data does not exist, the buffer device 10 returns a response indicating that the data does not exist (step 903). Only when data is returned, the data processing program executes data processing (step 904). Data processing differs depending on the application of the system, but is not specified here.

  The effects of the present invention are specifically shown below.

  In one case, there are a number of buffer devices, and there is a possibility that congestion will occur due to the arrival of a large amount of data temporarily. Buffer device BUFF_1 that needs to be preferentially processed and has a low flow rate but low latency. An example will be given of a case where PULL_RR (a PULL strategy not corresponding to congestion) and PULL_RR_JAM (a PULL strategy corresponding to congestion) are mixed in a system in which buffer devices BUFF_2 that need to perform processing are mixed.

  When congestion does not occur, PULL_RR and PULL_RR_JAM select the buffer device by round robin in the same manner. As a result, all the buffer devices are selected within a certain period. When congestion occurs in some buffer devices, the data processing program operating with PULL_RR always cycles through BUFF_1 and BUFF_2 at a constant cycle, so that the requirements of BUFF_2 can be satisfied. Further, PULL_RR_JAM preferentially selects a congested BUFF_1, and therefore tries to process a large amount of data that arrives at BUFF_1 and accumulates in a short time.

  As described above, when a single PULL strategy is selected, there may be cases where the requirements cannot be met sufficiently in an environment where data with different input and processing requirements are mixed. There is an effect of the system of the present invention which can be mixed.

  As the data processing apparatus in the stream processing system, the operations of the constituent elements of the data processing apparatus 100 in FIG. 6 in the first embodiment and the data processing apparatus 200 in FIG. 12 in the second embodiment are constructed as programs. It can be installed and executed on a computer to be used, or can be distributed via a network.

  The present invention is not limited to the first and second embodiments described above, and various modifications and applications can be made within the scope of the claims.

DESCRIPTION OF SYMBOLS 10 Buffer apparatus 11 Accumulation part 12 Threshold information 13 Communication part 20 Data source 21 Additional recording client 100,200 Data processing apparatus 110,210 Communication part 120 Pull strategy configuration setting part 130,230 Buffer apparatus selection part 140,240 State management part 150, 250 Data processing program 220 Congestion handling setting unit

Claims (8)

In a stream processing system, a data processing device having a plurality of data processing programs for autonomously processing data accumulated in a plurality of buffer devices that temporarily hold input data from a data source,
When congestion occurs in the buffer device, a data processing program that operates according to a rule that preferentially selects a buffer device in which congestion has occurred, and even if congestion occurs, the buffer is processed in the same order as before the congestion occurs A data processing apparatus comprising means for controlling both data processing programs that operate according to a rule for selecting an apparatus. When acquiring congestion notification from the buffer device, state management means for holding state information of the congestion buffer device;
As the rule, rule A for determining a data acquisition destination buffer device from the buffer device in which congestion occurs if congestion occurs, and data acquisition destination buffer device in all buffer devices if no congestion occurs A rule configuration setting means for setting a first rule including a rule B for determining the buffer, a second rule not corresponding to congestion of the buffer device, and notifying the rule in response to a request;
When a buffer device selection request is input from the data processing program, referring to the state information of the congestion buffer device of the state management unit, the first rule and the second rule acquired from the rule configuration setting unit Buffer device selection means for notifying the data processing program of the buffer device selected according to the specified rule among the rules,
Have
The data processing program is
The data processing apparatus according to claim 1, wherein data is acquired from the buffer apparatus selected by the buffer apparatus selecting means. When acquiring congestion notification from the buffer device, state management means for holding state information of the congestion buffer device;
A congestion correspondence setting means for setting a ratio of α (0 ≦ α ≦ 1) value indicating whether or not to cope with congestion when congestion occurs in the buffer device;
Referring to the state information of the state management means, when there is no congestion, a plurality of buffer devices are circulated according to a rule that determines in what order the data of the buffer devices is acquired. When data is acquired and congestion occurs, the α value is acquired from the congestion response setting unit, a random number determination is performed to select a buffer device, and the selected buffer device is notified to the data processing program Buffer device selection means for
Have
The buffer device selection means includes:
A random number between 0 and 1 is generated. When the random number is 0 or more and less than α, the buffer device in the congested state is used as the data acquisition destination buffer device. When the random number is α or more and 1 or less, all buffer devices are selected. The data processing apparatus according to claim 1, further comprising: The buffer selection means includes
4. The data processing apparatus according to claim 3, further comprising means for increasing the α value based on a congestion duration or the number of congested buffer apparatuses and increasing the number of the data processing programs in stages. In a stream processing system, a data processing method having a plurality of data processing programs for autonomously processing data stored in a plurality of buffer devices that temporarily hold input data from a data source,
Data processing device
When congestion occurs in the buffer device, a data processing program that operates according to a rule that preferentially selects a buffer device in which congestion has occurred, and even if congestion occurs, the buffer is processed in the same order as before the congestion occurs A data processing method characterized by controlling both data processing programs that operate according to a rule for selecting a device. The data processing device includes state management means, rule configuration setting means, buffer device selection means, and the data processing program.
When the state management means obtains a congestion notification from the buffer device, a state management step for holding state information of the congestion buffer device;
The rule configuration setting means, as the rule, determines rule A to determine a data acquisition destination buffer device from the buffer device in which congestion has occurred if congestion has occurred, and all buffer devices if congestion has not occurred A rule configuration setting step for setting a first rule including rule B for determining a buffer device from which data is acquired from the second rule, a second rule not corresponding to congestion of the buffer device, and notifying the rule in response to a request When,
When the buffer device selection means receives a buffer device selection request from the data processing program, the buffer device selection means refers to the state information of the congestion buffer device and the first rule obtained from the rule configuration setting means and the first rule A buffer device selection step of notifying the data processing program of the buffer device selected according to the specified rule of the two rules;
The data processing method according to claim 5, wherein: The data processing device has state management means, congestion response setting means, buffer device selection means, and the data processing program,
When the state management means obtains a congestion notification from the buffer device, a state management step for holding state information of the congestion buffer device;
A congestion handling setting step in which the congestion handling setting means sets a ratio of an α (0 ≦ α ≦ 1) value indicating whether or not to deal with congestion when congestion occurs in the buffer device;
When the buffer device selection means refers to the state information and no congestion has occurred, the buffer device selection unit conforms to a rule that determines in what order the buffer device data is acquired in a plurality of buffer devices. If the congestion occurs, the buffer device is selected by random number determination by acquiring the α value from the congestion response setting means, and the selected buffer device is stored in the data processing program. A buffer device selection step to notify
And
In the buffer device selection step,
A random number between 0 and 1 is generated. When the random number is 0 or more and less than α, the buffer device in the congested state is used as the data acquisition destination buffer device. When the random number is α or more and 1 or less, all buffer devices are selected. The data processing method according to claim 5. Computer
The data processing program for functioning as each means of the data processing apparatus of any one of Claims 1 thru | or 4.

Images