JP2008083808A - Filter processor, data storage device, shared information management device, filter processing method and filter processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To prevent reduction of processing capability of the whole system even when receiving various stream data from an information source. <P>SOLUTION: A filter processing means 10 (10A-10N) used for a stream data management system 1 receives queue state information showing a queue state in a data storage means 20 from each of the data storage means 20 storing the stream data. The filter processing means 10 selects the data storage means 20 that is an acquisition source of the stream data on the basis of the received queue state information. The filter processing means 10 acquires the stream data from the selected data storage means 20, and executes prescribed filter processing to the stream data. When the filter processing means 10 downloads file data from a shared information management means 30 and adds a filter processing function, the filter processing means 10 confirms to the shared information management means 30 whether a restart can be performed or not. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a stream data processing technique.

2. Description of the Related Art Conventionally, there is a stream data processing technique in which various data are collected by various sensors, IC tags, mobile phones, etc., and the collected data is subjected to filter processing and distributed to a predetermined usage destination. As such stream data processing techniques, telegraphCQ (see Non-Patent Document 1) and borealis (see Non-Patent Document 2) have been proposed.
Sirish Chandrasekaran, Owen Cooper, AmoI Deshpande, Michael J. Franklin, Joseph M. Hellerstein, Wei Hong, Sailesh Krishnamurthy, Sam Madden, Vijayshankar Raman, Fred Reiss, and Mehul A. Shah., “Te1egraphCQ: Continuous Dataflow Processing for an Uncertain Dataflow Processing for ancercer World, ”CIDR (2003) DJAbadi, Y.Ahmad, M.Balazinska, U.Cetintemel, M.Cherniack, JHHwang, W.Lindner, ASMaskey, A.Rasin, E.Ryvkina, N.Tatbul, Y.Xing, and Stan Zdonik, “The Design of the Borealis Stream Processing Engine, ”CIDR (2005)

  The telegraph CQ described above executes stream data processing according to changes in the properties of input data, but only a single server executes the processing. That is, in the telegraphCQ described above, it is not considered that a plurality of servers execute stream data processing.

  In the above-described borealis, a plurality of servers execute stream data processing. After the optimum arrangement of each server is determined, stream data processing migration processing (from a server with a high load to a server with a low load). Processing to migrate operations). However, if such a migration process is executed in each server, the migration load increases. In particular, data input from various information sources such as various sensors and IC tags is accepted, and data input from the information source is also performed. When the change in quantity is severe, the processing capacity of the entire system may be reduced.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a filter processing apparatus and the like that solves the above-described problems and prevents the processing capacity of the entire system from being lowered even when stream data is received from various information sources. To do.

  In order to solve the above-described problem, the invention described in claim 1 is a filter processing device used in a stream data management system that performs filtering of stream data by a plurality of filter processing devices, and accepts input of various data. From each of the input unit and the data storage devices that store the stream data transmitted from the plurality of information sources, the queue state information indicating the queue state in the data storage device is received via the input unit, and the received queue Based on the status information, the selection processing unit that selects the data storage device from which the stream data is acquired, and the data storage device selected by the selection processing unit, the data is stored in the data storage device via the input unit. A data acquisition processing unit for acquiring the stream data and the filter processing device For each service ID assigned to the stream data, a storage unit that stores a filter operation describing the contents of the filter process to be executed, a filter processing unit that executes a filter process based on the filter operation on the stream data, and The filter operation to be executed on the acquired stream data is determined with reference to the service ID table indicating the filter processing to be executed on the stream data and the service ID attached to the acquired stream data. Then, the determined filter operation is read from the storage unit, a filter execution control unit for causing the filter processing unit to execute a filter process based on the read filter operation, and stream data after the filter process are output. And configured to include an output unit.

  According to such a configuration, each filter processing device in the stream data management system acquires stream data from a plurality of data storage devices. At this time, each filter processing device determines a data storage device that acquires stream data based on the queue state in each data storage device. For example, the filter processing device acquires stream data from a data storage device having a higher input throughput. Thereby, even if it does not exchange the processing load information between filter processing apparatuses, the filter processing load of each filter processing apparatus can be balanced. That is, in the stream data management system, the processing load is not concentrated on a specific filter processing device. As a result, when the stream data management system receives stream data having different flow characteristics, the specific filter processing device does not become a bottleneck, and the processing capacity of the entire system is not reduced.

  According to a second aspect of the present invention, in the filter processing device according to the first aspect, the filter operation includes information on an output destination of the stream data after the filter processing, and the filter processing unit includes the filter operation. Based on this, the output destination of the stream data after the filtering process is determined.

  According to such a configuration, the filter processing apparatus can output the stream data after the filter processing to a predetermined output destination. For example, when the stream data is viewing data of a television program in each home, the filter processing device can perform the filtering process on the stream data and then output the stream data to the server of the company that conducts the audience rating survey. That is, it is possible to control the output destination of the stream data after the filtering process. As described above, when the filter operation includes the information on the output destination of the stream data after the filter processing, when the filter processing function is added in the filter processing apparatus, the administrator of the system can output the output destination. This is very convenient because there is no need to manually add.

  According to a third aspect of the present invention, in the filter processing device according to the first or second aspect, the queue status information in each of the data storage devices includes input throughput, output throughput, and CPU cost in the data storage device. Including at least one of the values, and the selection processing unit prioritizes the one having the larger value based on at least one of the input throughput, the output throughput, and the CPU cost in each of the data storage devices. In particular, the data storage device from which the stream data is acquired is selected.

  According to such a configuration, the selection processing unit of each filter processing device selects a data storage device that acquires stream data based on the input throughput, output throughput, and CPU (Central Processing Unit) cost of each data storage device. Therefore, the stream data can be preferentially acquired from the data storage device that is more congested.

  According to a fourth aspect of the present invention, in the filter processing device according to any one of the first to third aspects, the filter execution control unit has a filter processing function required for the acquired stream data. When the filter determination unit that determines whether or not the filter processing unit is present and the filter acquisition determination unit determine that the filter processing function necessary for the acquired stream data is not present in the filter processing unit ( From the shared information management device storing the file necessary for the filter processing and (2) restart information indicating the number of the filter processing devices permitted to restart in the stream data management system, the filter processing To the filter acquisition unit that acquires necessary files for addition to the filter processing unit and the shared information management device. A restart inquiry unit that inquires whether or not to permit restart of the filter processing device, and a response to permit restart of the own filter processing device from the shared information management device at the restart inquiry unit A restart unit that restarts the own filter processing device and notifies the shared information management device of the completion of restart of the own filter processing device when the restart is completed. It was.

  According to such a configuration, each filter processing device acquires a file necessary for the filter processing that is lacking from the shared information management device and adds the file to the filter processing unit. Therefore, each filter processing device can execute the filter processing even when the type of stream data changes. When each filter processing device adds a filter processing function, it is necessary to restart the filter processing device, but each filter processing device has received a notification from the shared information management device that the restart is permitted. When performing a reboot. As a result, it is possible to prevent the number of filter processing devices whose filter processing is stopped due to restarting from exceeding a predetermined number and the processing capacity of the entire system from being significantly reduced.

  According to a fifth aspect of the present invention, in the filter processing device according to the fourth aspect, the restart inquiry unit permits the restart of its own filter processing device from the shared information management device even if a predetermined time has elapsed. When a response to this effect is not received, it is again inquired whether or not the restart of the own filter processing device is permitted.

  According to such a configuration, since a predetermined number or more of the filter processing apparatuses are being restarted in the entire system, even if the own filter processing apparatus cannot be restarted, the time is shifted and the own filter processing apparatus Can be restarted.

  According to a sixth aspect of the present invention, there is provided a data storage device for transmitting the stream data to the filter processing device according to any one of the first to fifth aspects. When at least one of the input throughput, the output throughput, and the CPU cost changes to a predetermined threshold value or more, the queue status information in the data storage device is transmitted to the filter processing device.

  According to such a configuration, when there is a change in the queue state (input throughput, output throughput, and CPU cost) of the data storage device, the filter processing apparatus quickly captures the change, and acquires the queue state information after the change. Based on this, it is possible to determine a data storage device from which stream data is acquired.

  According to a seventh aspect of the present invention, there is provided a file necessary for the filter processing in the filter processing device according to the fourth or fifth aspect, and restart information indicating the number of the filter processing devices permitted to be restarted. A shared information management device comprising a storage unit for storing, a file data transmission unit for transmitting file data necessary for the filter processing in response to a request from the filter processing device, and a restart from the filter processing device If the inquiry about whether or not to permit is received, it is determined whether or not the number of filter processing devices permitted to restart in the restart information is positive, and the number of filter processing devices permitted to restart Is determined to be positive, a restart determination processing unit that transmits a response to permit the restart of the filter processing device to the filter processing device; When the response indicating that the restart is permitted is transmitted to the data processing device, the number of filter processing devices that are permitted to be restarted in the restart information is subtracted, and the filter processing device is restarted from the filter processing device. And a restart information update unit that adds the number of filter processing devices that are permitted to restart in the restart information when a notification that the start is completed is received.

  According to such a configuration, the shared information management apparatus transmits file data necessary for the filtering process in response to a request from the filtering process apparatus, and thus causes the filtering process apparatus to perform various stream data filtering processes. be able to. In addition, since this shared information management device manages the number of filter processing devices that are allowed to be restarted in the entire system, the number of filter processing devices in which the filter processing is stopped because of restarting exceeds a predetermined number, It is possible to prevent the processing capacity from being significantly reduced.

  The invention according to claim 8 is that each of the filter processing devices used in the stream data management system that performs filtering of stream data by a plurality of filter processing devices, from each of the one or more data storage devices that store the stream data, Receiving queue state information indicating a queue state in the data storage device, selecting the data storage device from which the stream data is acquired based on the received queue state information, and the selected A step of acquiring stream data stored in the data storage device from the data storage device, and a service ID table indicating a filtering process to be performed on the stream data for each service ID attached to the stream data; , The obtained stream A filter operation to be performed on the acquired stream data is determined with reference to the service ID attached to the data, the determined filter operation is read from the storage unit, and a filter process based on the read filter operation is performed The filter processing method is characterized in that the step of causing the filter processing unit to execute and the step of outputting the stream data after the filter processing are executed.

  According to such a method, each of the filter processing devices in the stream data management system acquires stream data from a plurality of data storage devices. At this time, each filter processing device determines a data storage device that acquires stream data based on the queue state in each data storage device. For example, the filter processing device acquires stream data from a data storage device having a higher input throughput. Thereby, it is possible to balance the filter processing between the respective filter processing devices without exchanging processing load information between the filter processing devices. That is, in the stream data management system, the processing load is not concentrated on a specific filter processing device. As a result, when the stream data management system receives stream data having different flow characteristics, the specific filter processing device does not become a bottleneck, and the processing capacity of the entire system is not reduced.

  According to a ninth aspect of the present invention, in the filter processing method according to the eighth aspect, the queue status information in each of the data storage devices is at least one of input throughput, output throughput, and CPU cost in the data storage device. When the filter processing device selects the data storage device from which the stream data is acquired, at least one value of input throughput, output throughput, and CPU cost in each of the data storage devices Based on the above, the filtering method is characterized in that the data storage device from which the stream data is obtained is selected with priority from the one of the large values.

  According to such a method, the selection processing unit of each filter processing device selects a data storage device that acquires stream data on the basis of the input throughput, output throughput, and CPU cost of each data storage device, so that it is more congested. Stream data can be preferentially acquired from the existing data storage device.

  According to a tenth aspect of the present invention, there is provided a filter processing program that causes a computer to execute the filter processing method according to the eighth or ninth aspect.

  According to such a filter processing program, it is possible to cause a computer to execute the filter processing method according to claim 8 or claim 9.

  According to the present invention, in the stream data management system, even when the data input amount of the information source changes drastically, the processing capacity of the entire system can be prevented from being significantly reduced. Moreover, since the filter processing apparatus in the stream data management system can add a filter processing function, it can execute filter processing on various stream data. Furthermore, even when such a filtering function is added, the processing capacity of the entire system can be prevented from being significantly reduced.

  Hereinafter, the best mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration example of a system including a stream data management system according to the present embodiment. As shown in FIG. 1, the system attaches a service ID (SID) to data acquired from various information sources, transmits the data as stream data, and a client 50 (50A, 50B) and the received stream data. The stream data management system 1 that performs filtering, the output destination 60 (60A to 60N) that is the output destination of the filtered data, and the search client 70 (70A to 70N) that searches for data from the output destination 60 It is comprised including.

  The stream data management system 1 includes a queue allocating unit 40 that instructs the client 50 to specify a data storage unit (device) 20 that is a transmission destination of the stream data, and a data storage unit (device) that stores the stream data transmitted from the client 50. ) 20 (20A, 20B,..., 20N), filter processing means (apparatus) 10 (10A to 10N) for filtering stream data, and a filter processing unit 124 (details will be described later) in the filter processing means 10 And a shared information management means (device) 30 for storing file data and the like for realizing the function.

  Further, the output destination 60 includes a storage processing unit 62 that stores data filtered in the stream data management system 1 in a DB (database) 61, a DB 61 that stores the filtered data, and requests from the search client 70. And a search processing unit 63 that searches the DB 61 for predetermined data.

  (1) The client 50 and the stream data management system 1 are connected to each other, and (2) the stream data management system 1 and the output destination 60 are connected by a transmission path (not shown). Furthermore, although the number of data storage means 20, filter processing means 10, and output destinations 60 is N in FIG. 1, it is not limited to this.

<Overview of operation>
Here, an outline of the operation of the system of FIG. 1 will be described. First, the client 50 acquires data from various information sources and attaches SIDs corresponding to the data. This SID may include SID version information. Then, the client 50 transmits the data with the SID as stream data to the stream data management system 1. The queue assignment unit 40 of the stream data management system 1 selects the data storage unit 20 that is the transmission destination of the stream data. Then, the client 50 is instructed to the data storage unit 20 that is the transmission destination of the stream data, and the client 50 transmits the stream data to the instructed data storage unit 20. Each of the data storage means 20 notifies each filter processing means 10 of this queue state when its own queue state (input throughput, output throughput, CPU cost, etc.) has changed by a predetermined threshold value or more.

  Subsequently, the filter processing unit 10 determines the data storage unit 20 that acquires the stream data based on the queue state notified from each data storage unit 20. For example, when the input throughput in the data storage unit 20B among the data storage units 20 (20A to 20N) is a relatively large value, the data storage unit 20B is determined as the stream data acquisition source.

  Next, the filter processing unit 10 acquires stream data from the determined data storage unit 20 and performs a filter process. The filtering process to be performed on each stream data is determined based on the SID attached to the stream data. Here, when the filter processing unit 124 does not have the filter processing function necessary for the stream data, the filter processing unit 10 acquires the file data from the shared information management unit 30 and adds the filter processing function. . Then, the filter processing means 10 executes the filter processing after adding the function.

  The filter processing unit 10 restarts when adding a function by acquiring file data from the shared information management unit 30. At this time, the filter processing means 10 first inquires of the shared information management means 30 whether or not it can be restarted. Here, the shared information management means 30 records the number (restart upper limit value) of the filter processing means 10 that may stop the filter processing by restarting the entire stream data management system 1 at the present time. Then, the shared information management means 30 notifies the filter processing means 10 that the restart is permitted when the restart upper limit value is not exceeded even if the filter processing means 10 that has received this inquiry is restarted. . On the other hand, when the filter processing unit 10 is restarted, when the restart upper limit is exceeded, the shared information management unit 30 notifies the filter processing unit 10 that the restart is not permitted. By doing in this way, in the stream data management system 1, it can prevent that the filter processing means 10 exceeding the predetermined number stops for restart, and the filter processing capability of the whole system falls remarkably.

  After that, the filter processing unit 10 outputs the stream data after the filter processing to a predetermined output destination 60. Then, the filtered data is accumulated in the DB 61 at the output destination 60, and the data in the DB 61 can be searched by the search client 70.

  Further, the client 50, the queue allocation means 40, the data storage means 20, the shared information management means 30, the filter processing means 10, the output destination 60, and the search client 70 are input / output interfaces, arithmetic processing such as a CPU (Central Processing Unit), etc. And a computer including a storage unit such as a random access memory (RAM), a read only memory (ROM), and a hard disk drive (HDD). Each processing unit is realized by a program execution process by a CPU or a dedicated circuit.

  The data storage means 20 (20A to 20N) may be realized by separate computers (devices), or may be realized by a single computer. Further, the filter processing means 10 (10A to 10N) may be realized by separate computers (devices), or may be realized by a single computer. That is, the data storage means 20 (20A to 20N) and the filter processing means 10 (10A to 10N) may be realized by processes or may be realized by a computer (machine). Further, the queue allocation unit 40 may further include a load balancer, and the load balancer may allocate the stream data to each data storage unit 20.

<Data storage means>
Next, the data storage means 20 will be described in detail with reference to FIG. FIG. 2 is a functional block diagram showing details of the data storage means of FIG.

  The data storage unit 20 is a unit that temporarily stores the stream data transmitted from the client 50. As shown in FIG. 2, the data storage unit 20 monitors the status of the queue 241 in the input unit 21 that receives the stream data input from the client 50 and the storage unit 24, and the queue state exceeds a predetermined threshold value. When there is a change, the processing unit 22 for transmitting the queue status information (information such as input throughput, output throughput and CPU cost) in the data storage means 20, and for outputting the stream data and the queue status information described above The output unit 23 includes a storage unit 24 in which a queue 241 of accumulated stream data is formed.

  The input unit 21 and the output unit 23 are realized by an input / output interface capable of inputting / outputting data to / from the client 50 and the filter processing unit 10, and the storage unit 24 is realized by a RAM, an HDD, or the like. The processing unit 22 is realized by an execution process of a program stored in the storage unit 24 by the CPU, a dedicated circuit, and the like. In the storage unit 24, the stream data accumulated as the queue 241 is transmitted to the filter processing unit 10 in response to a request from the filter processing unit 10.

  The processing unit 22 includes a queue state monitoring unit 221 and a queue state information transmission unit 222. The queue state monitoring unit 221 monitors the queue state (input throughput, output throughput, CPU cost, etc.) in its own data storage means 20. Then, the queue state monitoring unit 221 monitors whether or not the queue state has changed beyond a threshold value indicated by a queue state information notification threshold value 321 (details will be described later) of the shared information management unit 30.

  When the queue state monitoring unit 221 changes the queue state to a threshold value indicated by the queue state information notification threshold 321 of the shared information management unit 30, the queue state information transmission unit 222 changes the queue state information in the data storage unit 20 to each queue state information. It transmits to the filter processing means 10. At this time, identification information of the data storage means 20 is attached to the queue state information. Further, the transmission of the queue state information is performed by, for example, broadcast transmission to each filter processing means 10 or the like.

  In this way, the data storage unit 20 can quickly notify each filter processing unit 10 of its own queue state when its own queue state has changed. Note that the queue status information notification threshold 321 is copied from the shared information management unit 30 by the data storage unit 20 and stored in the storage unit 24, and the queue state monitoring unit 221 stores the queue status information stored in the storage unit 24. The queue state may be monitored with reference to the state information notification threshold value 321.

<Filter processing means>
Next, returning to FIG. 1, the filter processing means 10 will be described in detail. As shown in FIG. 1, the filter processing unit 10 includes an input unit 11 that receives input of stream data, a processing unit 12 that acquires stream data from the data storage unit 20 and performs filter processing, and data that has undergone filter processing. Output unit 13 and a storage unit 14 for storing a program for realizing the function of the filter processing unit 10 and a queue state table 141 indicating the queue state of each data storage unit 20. The The input unit 11 and the output unit 13 are also realized by an input / output interface capable of inputting / outputting data to / from the data storage unit 20 and the output destination 60, and the storage unit 14 is realized by a RAM, an HDD, or the like. The processing unit 12 is realized by an execution process of a program stored in the storage unit 14 by the CPU, a dedicated circuit, and the like.

  The processing unit 12 includes a selection processing unit 121, a data acquisition processing unit 122, a filter execution control unit 123, and filter processing units 124 (124A to 124N).

  The selection processing unit 121 receives the queue status information (information indicating input throughput, output throughput, CPU cost (filter cost), etc.) transmitted from the data storage unit 20 via the input unit 11. This information is received and reflected in the queue status table 141 (details will be described later). Then, the selection processing unit 121 selects (determines) the data storage unit 20 that is the stream data acquisition source based on the queue state information in each data storage unit 20 described in the queue state table 141. Details of the selection process of the data storage unit 20 at this time will be described later using a specific example. For example, the data storage unit 20 having a larger input throughput value is selected with higher probability (priority).

  The data acquisition processing unit 122 receives stream data from the data storage unit 20 selected by the selection processing unit 121 via the input unit 11. Note that the timing of acquiring the stream data may be, for example, when there is no data processed in its own filter processing means 10 or when the data being processed falls below a predetermined threshold.

  The filter execution control unit 123 refers to the SID attached to the stream data acquired from the data acquisition processing unit 122 and the SID table 322 (details will be described later) of the shared information management unit 30, and executes this stream data. Determine the filter processing (application procedure of filter operation) to be performed. Then, the filter execution control unit 123 reads the filter operation (FOP) from the storage unit 14 and causes the filter processing unit 124 to execute the filter process described by the FOP.

  The filter execution control unit 123 receives the file data for realizing the filter processing function from the shared information management unit 30 when the filter processing unit 10 does not have the filter processing function to be executed on the stream data. Download (including filter operation) and add functions. Details of the execution control unit 123 will be described later with reference to FIG.

  Based on an instruction from the filter execution control unit 123, the filter processing unit 124 performs filter processing on the stream data. The content of the filter processing in the filter processing unit 124 is described by a filter operation (FOP), and the filter processing unit 124 executes the filter processing based on this FOP. This FOP is, for example, “Convert image format from JPEG to Bitmap format” or “Normalize image size to fit feature extraction routine”. Further, this FOP may include information regarding the output destination of the processed data such as “output the processed data to the output destination 60A”. This FOP is stored in the storage unit 14 and read by the filter execution control unit 123.

  The storage unit 14 stores the above-described FOP, a program (not shown) that realizes the function of the filter processing unit 10, and a queue state table 141. The queue status table 141 is information indicating the queue status for each data storage unit 20. Table 1 below illustrates the queue state table 141.

  For example, in the queue status table 141 illustrated in Table 1, the input throughput of the data storage unit 20 with ID “1” is “10”, the output throughput is “10”, and the CPU cost is “2”. Indicates.

  The input throughput is an input amount per unit time of data input to the data storage unit 20. The output throughput is an output amount per unit time when the data stored in the data storage unit 20 is output to the filter processing unit 10. The input amount and the output amount can be quantified by the number of tuples (records) in the database.

  Furthermore, the CPU cost is defined by, for example, the filter processing time per tuple. The filter processing time is a time from when the data storage unit 20 passes the data to the filter processing unit 10 until the data is processed. That is, the data storage unit 20 records the time when the data is passed to the filter processing unit 10. When the filter processing unit 10 completes the filtering process of the data, the filter processing unit 10 transmits a notification to that effect to the data storage unit 20. Measure the filter processing time. The data storage unit 20 determines that the CPU cost increases as the measured filter processing time increases.

  Here, the selection processing of the data storage unit 20 by the selection processing unit 121 with reference to the queue state table 141 will be described with reference to FIG. FIG. 3 is a diagram for explaining selection processing of the data storage means by the selection processing unit of FIG. Here, an example will be described in which the selection processing unit 121 selects the data storage unit 20 that is the stream data acquisition source based on the input throughput value.

(1) First, the selection processing unit 121 obtains the sum (10 + 5 + 5 = 20) of input throughput values in the queue state table 141.
(2) Next, the selection processing unit 121 generates a random value r between 1 and 20 (sum of input throughput values).
(3) Then, the selection processing unit 121 selects the data storage unit 20 that acquires data based on the random value r generated in the above (2). For example, if the random value r generated in (2) satisfies 1 ≦ r ≦ 10, the data storage means 20 with ID “1” is selected, and the random value r satisfies 11 ≦ r ≦ 15. If there is, the data storage means 20 with ID “2” is selected, and if the random value r satisfies 16 ≦ r ≦ 20, the data storage means 20 with ID “3” is selected.

  That is, when the number of data storage means 20 is 3, the probability that the data storage means 20 with ID “1” is selected is 50%, and the probability that the data storage means 20 with ID “2” is selected is 25%. , The probability that the data storage means 20 with ID “3” is selected is 25%. That is, the selection processing unit 121 of each filter processing unit 10 can select the data storage unit 20 having a higher load with a high probability when selecting the data storage unit 20 that acquires the stream data.

  Here, the selection processing unit 121 selects the data storage unit 20 that acquires the stream data based on the input throughput value of the data storage unit 20, but the output throughput or CPU cost value is determined. The data storage means 20 that acquires the stream data may be selected. Alternatively, the data storage unit 20 may be selected in consideration of all values of input throughput, output throughput, and CPU cost. That is, the data storage means 20 that is the data acquisition source is selected based on at least one of the input throughput, output throughput, and CPU cost in the data storage means 20.

<Filter execution control unit>
Next, the details of the filter execution control unit 123 will be described with reference to FIG. FIG. 4 is a functional block diagram showing details of the filter execution control unit of FIG.

  As shown in FIG. 4, the filter execution control unit 123 includes a filter determination unit 126, a filter acquisition unit 127, a restart inquiry unit 128, a restart unit 129, and a filter processing application unit 130. The

  The filter determination unit 126 refers to the SID and the version of the SID attached to the stream data output from the data acquisition processing unit 122 and the SID table 322 (details will be described later) of the shared information management unit 30 to refer to this stream. Determine the filter operation to be performed on the data. Further, the filter determination unit 126 determines whether or not the filter processing unit 124 has a filter processing function necessary for the stream data.

  The filter acquisition unit 127 acquires the file data 323 from the shared information management unit 30 when the filter determination unit 126 determines that the filter processing function necessary for the stream data is not in the filter processing unit 124. Then, by installing the file data 323, a filter processing function is added to the filter processing means 10.

  After the filter acquisition unit 127 acquires the file data 323, the restart inquiry unit 128 inquires of the shared information management unit 30 whether or not the restart of its own filter processing unit 10 is permitted.

  When the restart unit 129 receives a response from the shared information management unit 30 to permit the restart of the own filter processing unit 10, the restart unit 129 restarts the own filter processing unit 10. In addition, when the restart is completed, the restart unit 129 notifies the shared information management unit 30 of the completion of restart of its own filter processing unit 10. In this way, the shared information management unit 30 is notified whether each filter processing unit 10 is in a state where the filter processing is stopped because it is being restarted.

  The filter processing application unit 130 refers to the SID and the version of the SID attached to the stream data, and the SID table 322 (details will be described later) of the shared information management unit 30 to perform a filter operation to be performed on the stream data. Apply. That is, with reference to the schema information shown in the SID table 322 and the FOP (filter operation) sequence, it is determined in what procedure the filtering process should be executed on this stream data. Then, the filter processing unit 124 is caused to execute the filter process according to the determined procedure.

  Note that the timing for acquiring the file data 323 from the shared information management unit 30 may be when the filter processing unit 10 detects that there is no filter processing function necessary for the stream data, or may be every predetermined period. .

<Shared information management means>
Next, the shared information management unit 30 will be described in detail with reference to FIG. FIG. 5 is a functional block diagram of the shared information management means of FIG. 1 (see FIGS. 1 and 2 as appropriate).

  As shown in FIG. 5, the shared information management unit 30 realizes the functions of the input unit 31 that controls the input of various data and the various data and shared information management unit 30 that are referred to by the filter processing unit 10 (see FIG. 1). A storage unit 32 that stores a program to be executed, an output unit 33 that controls the output of various data, and a processing unit 34 that controls the entire shared information management unit 30. The input unit 31 and the output unit 33 are realized by an input / output interface capable of inputting / outputting data to / from the data storage unit 20, the filter processing unit 10, and the like, and the storage unit 32 is realized by a RAM, an HDD, or the like. The processing unit 34 is realized by a program execution process by a CPU, a dedicated circuit, or the like. In FIG. 5, the program is not shown.

  The storage unit 32 stores a program (not shown) that realizes the function of the shared information management unit 30, a queue state information notification threshold value 321, a table 322, file data 323, and restart information 324.

  The queue status information notification threshold 321 is a threshold for the queue status monitoring unit 221 of the data storage unit 20 to determine whether or not to transmit queue status information. This queue state information notification threshold value 321 is set with conditions such as when at least one of input throughput, output throughput, CPU cost, etc. has changed by 20% of the whole (that is, increased or decreased). To do.

  The SID table 322 is a table showing the content of the filter processing to be performed on the stream data to which the SID is attached for each SID. The SID table 322 is illustrated in Table 2 below.

  The SID table 322 illustrated in Table 2 includes a SID, a version of the SID, schema information of the stream data, and a filter operation (FOP) sequence to be applied to the stream data having the SID and the version (the FOP to be applied). Identification information and filter operation application procedure).

  For example, in the SID table 322 illustrated in Table 2, the schema information of the stream data of the SID “10” and the SID version “1.0” is “NAME char (10), TEL char (10), ADDRESS char (100)”. AGE int ”, and indicates that the filter processing“ FOP102, FOP103, FOP104 ”should be applied to this stream data in the order“ FOP102 → FOP103 → FOP104 ”. The value in parentheses in the schema information is the window size when the filter processing unit 124 performs the filter processing. For example, when the filter processing unit 124 performs a filter process on “NAME” in the data, it indicates that 10 data (records) are necessary.

  The SID table 322 is referred to when the filter execution control unit 123 of the filter processing unit 10 determines the content of the filter process to be executed on the stream data. The information in the SID table 322 can be rewritten via the input unit 31.

  The file data 323 is file data that realizes the function of the filter processing unit 124 of the filter processing means 10. This file data is, for example, the latest version of file data used in the stream data management system 1 and includes the above-described FOP. The file data 323 is transmitted to the filter processing unit 10 that is the transmission source of the request by the filter transmission unit 341 based on the request from the filter processing unit 10.

  The restart information 324 is information indicating the number of filter processing units 10 that are permitted to restart in the stream data management system 1. For example, in the entire stream data management system 1, when the number of filter processing means 10 that may stop the filter processing by restart is “10”, “10” is set in the restart information 324 as an initial value. Keep it. The restart information 324 is referred to by the restart determination processing unit 342 (described later) and rewritten by the restart information updating unit 343 (described later).

  Information stored in the storage unit 32 is input via the input unit 31.

  Next, the processing unit 34 will be described. The processing unit 34 includes a filter transmission unit 341, a restart determination processing unit 342, and a restart information update unit 343.

  Upon receiving a file data acquisition request from the filter acquisition unit 127 (see FIG. 4) of the filter processing unit 10, the filter transmission unit (file data transmission unit) 341 receives the file data 323 via the output unit 33 accordingly. Send.

  When the restart determination processing unit 342 receives an inquiry from the restart inquiry unit 128 (see FIG. 4) of the filter processing unit 10 as to whether or not the restart of the filter processing unit 10 is permitted, the restart determination unit 342 displays the restart information 324. With reference to this, it is determined whether or not the restart of the filter processing means 10 is permitted. For example, when the restart determination processing unit 342 receives an inquiry as to whether or not to permit restart of the filter processing unit 10, the number of filter processing units 10 that permit restart in the restart information 324 is correct. If so, a response is sent to the filter processing means 10 to permit the restart of the filter processing means 10. On the other hand, when the number of filter processing devices permitted to restart in the restart information 324 is 0 or less, the restart determination processing unit 342 does not permit the filter processing unit 10 to restart the filter processing unit 10. Send the response.

  When the restart determination processing unit 342 transmits a response indicating that the restart of the filter processing unit 10 is permitted to the filter processing unit 10, the restart information update unit 343 performs restart in the restart information 324. Reduce the number of allowed filter processing devices by one. Further, when a restart completion notification is received from the restart unit 129 (see FIG. 4) of the filter processing unit 10, the number of filter processing devices that are permitted to restart is increased by one.

  By doing in this way, the shared information management means 30 can keep the number of the filter processing means 10 that has stopped the filter processing due to the restart in the stream data management system 1 below a predetermined number.

  The shared information management unit 30 may disclose the SID associated with the FOP provided by the file data 323 to each filter processing unit 10 when the file data 323 is updated to the latest one. Good. By doing in this way, each filter processing means 10 can immediately know that the filter processing function provided therein needs to be updated.

<Operation procedure>
Next, the operation of the stream data management system 1 will be described. First, a procedure until stream data is stored in the data storage unit 20 will be described with reference to FIG. FIG. 6 is a flowchart showing a procedure until stream data is stored in the data storage means of FIG.

  First, the client 50 transmits a request for specifying the data storage means 20 that is the storage destination of the stream data from the client 50 to the queue assignment means 40 (S601). Upon receiving this, the queue allocation unit 40 selects the data storage unit 20 and designates it as the selected data storage unit 20 (S602). Here, the data storage means 20 is selected, for example, by selecting the data storage means 20 having the shortest queue among all the data storage means 20 in the stream data management system 1. By doing so, queues are formed in each data storage means 20 with little bias.

  Next, when receiving the designation of the data storage unit 20, the client 50 transmits data (stream data) to the designated data storage unit 20 (S603). The data transmitted here includes the SID.

  Then, when receiving data from the client 50, the data storage means 20 stores this data in the storage unit 24 (see FIG. 2) (S604). In this way, data is stored in the data storage unit 20.

  Next, a procedure for updating the queue state table 141 (see FIG. 1) will be described with reference to FIG. FIG. 7 is a flowchart showing a procedure for updating the queue state table of FIG.

  The data storage unit 20 stores data transmitted from the client 50 (S701), and the queue state monitoring unit 221 (see FIG. 2) monitors the queue state in the data storage unit 20 (S702). The queue state monitoring unit 221 determines whether or not there is a change in the queue state beyond the threshold indicated by the queue state information notification threshold 321 of the shared information management unit 30 (S703).

  Here, when the queue state monitoring unit 221 determines that there is a change in the queue state beyond the threshold indicated by the queue state information notification threshold 321 of the shared information management unit 30 (Yes in S703), the queue state information transmission unit 222 transmits the queue status information in the data storage means 20 to each filter processing means 10 (S704).

  When the selection processing unit 121 of the filter processing unit 10 receives queue state information from each data storage unit 20, the selection processing unit 121 stores this information in the queue state table 141. That is, the queue state table 141 is updated (S705).

  Next, the filter processing procedure of the filter processing means 10 will be described with reference to FIGS. 8 and 9 are flowcharts showing the filter processing procedure of the filter processing means of FIG. 1 (see FIGS. 1 and 4 as appropriate).

  First, when it is detected that there is no data to be processed in the filter processing means 10 (Yes in S801), the selection processing unit 121 refers to the queue state table 141 and selects the data storage means 20 that acquires data next. (S802). Details of the selection procedure at this time will be described later with reference to FIG. Then, the data acquisition processing unit 122 acquires data from the selected data storage unit 20 (S803).

  Next, the filter determination unit 126 (see FIG. 4) refers to the SID information (SID and its version) attached to this data and the SID table 322, and performs a filtering function necessary for this stream data. Is determined (S804). Then, it is determined whether or not the filter processing function necessary for the stream data is present in its own filter processing means 10 (S805). Here, when the necessary filter processing function does not exist in its own filter processing means 10 (No in S805), the filter acquisition unit 127 acquires the necessary filter processing function from the shared information management means 30 (S806). That is, the filter acquisition unit 127 acquires the file data 323 from the shared information management unit 30 and adds a filter processing function to the filter processing unit 124 using the file data 323.

  On the other hand, when the necessary filter processing function is present in the own filter processing means 10 (Yes in S805), the process proceeds to S901 in FIG. 9, and the restart inquiry unit 128 instructs the shared information management means 30 to the own filter processing means 10. An inquiry is made as to whether or not to permit restart of the system (S901). That is, a message for inquiring whether or not to permit restarting of its own filter processing means 10 is transmitted.

  Then, the restart inquiry unit 128 of the filter processing unit 10 waits for a response from the shared information management unit 30 as to whether or not to permit restart. Here, when the response indicating that the restart of the own filter processing unit 10 is permitted from the shared information management unit 30 (Yes in S902), the restart unit 129 restarts the own filter processing unit 10. (S903). Then, when the restart is completed, the restart unit 129 transmits a notification of restart completion to the shared information management unit 30 (S904). Next, the filter execution control unit 123 executes the filter process using the filter processing function acquired in S806 (S905). That is, the filter execution control unit 123 determines the content of the filter process for this stream data according to the SID table 322, and reads a predetermined FOP for executing this filter process from the storage unit 14. Then, the filter execution control unit 123 causes the filter processing unit 124 to execute the filter processing in a predetermined order based on this FOP, and outputs the data after the filter processing to a predetermined output destination 60. Then, the process returns to S801 in FIG.

  In S902 of FIG. 9, when the restart inquiry unit 128 of the filter processing unit 10 receives a notification from the shared information management unit 30 that does not permit the restart of its own filter processing unit 10, or its own filter When a response to permit the restart of the processing means 10 has not been received (No in S902), after a predetermined time has elapsed (Yes in S906), the process returns to S901. Then, the filter processing unit 10 restart inquiry unit 128 inquires of the shared information management unit 30 whether or not the restart of the filter processing unit 10 is permitted. By doing in this way, since the other filter processing means 10 is being restarted, even if its own filter processing means 10 cannot be restarted, it can be restarted with a time shift.

  Next, a procedure (S802 in FIG. 8) in which the selection processing unit 121 selects the data storage unit 20 to acquire data next will be described in detail with reference to FIG. FIG. 10 is a flowchart showing a procedure for selecting the data storage means for acquiring data next by the selection processing unit of FIG. 1 (see FIG. 1 as appropriate).

  First, when the filter processing unit 10 detects that there is no data to be processed, the selection processing unit 121 refers to the queue state table 141 and calculates the sum of loads (processing loads) in all the data storage units 20. (S1001). For example, the sum of input throughput values in all data storage means 20 is obtained.

  The selection processing unit 121 generates a random value r between the values of 1 to the sum of loads (the sum of loads calculated in S1001) (S1002). The generation of the random value r at this time may use a known technique.

  The selection processing unit 121 then sums S [1] (the load in the first data storage unit 20) to S [j] (the load in the jth data storage unit 20) ≦ r ≦ S [1] ( J that satisfies the sum from the load in the first data storage unit 20) to S [j + 1] (the load in the j + 1th data storage unit 20) is determined and output (S1003). That is, the selection processing unit 121 determines the data storage unit 20 that acquires data next. When the number of data storage means 20 in the stream data management system 1 is N, j is a value that satisfies 1 ≦ j ≦ N.

  By doing in this way, each filter processing means 10 can select the data storage means 20 with larger load when selecting the data storage means 20 for acquiring the stream data.

  Next, an operation procedure of the shared information management unit 30 that has received an inquiry from the filter processing unit 10 as to whether or not to permit restart will be described with reference to FIG. FIG. 11 is a flowchart showing an operation procedure of the shared information management unit that has received an inquiry as to whether or not to permit restart from the filter processing unit of FIG. 1 (see FIG. 5 as appropriate).

  First, when the restart determination processing unit 342 (see FIG. 5) of the shared information management unit 30 receives an inquiry from the filter processing unit 10 as to whether or not to allow restart (Yes in S1101), the restart information 324 is stored. Reference is made (S1102).

  Then, the restart determination processing unit 342 determines whether or not the number of the filter processing means 10 permitting restart in the restart information 324 is positive (S1103).

  Here, when the number of filter processing means 10 permitting restart in the restart information 324 is positive (Yes in S1103), the restart determination processing unit 342 sends the filter processing means 10 to the filter processing means 10. A response to permit the restart is transmitted (S1104).

  Then, the restart information update unit 343 reduces the number of filter processing devices permitted to restart by 1 in the restart information 324 (S1105).

  Thereafter, the restart determination processing unit 342 of the shared information management unit 30 waits for a restart completion notification from the filter processing unit 10 and receives a restart completion notification from the filter processing unit 10 (Yes in S1106). In the restart information 324, the restart information update unit 343 of the shared information management unit 30 increases the number of filter processing devices that are permitted to restart by one (S1107). Then, the process returns to S1101.

  On the other hand, in S1103, when the number of the filter processing means 10 permitting restart of the restart information 324 is 0 or less (No in S1103), the restart determination processing unit 342 of the shared information management means 30 is the filter processing means. A response to the effect that the restart of the filter processing means 10 is not permitted is transmitted to 10 (S1108). Then, the process returns to S1101.

  By doing in this way, the shared information management means 30 can make the number of the filter processing means 10 which have stopped the filter processing because of the restart in the stream data management system 1 not more than a predetermined number.

<Example of embodiment>
Next, an example of an embodiment of the above-described system will be described using a specific example. First, a case where the stream data management system 1 performs video monitoring will be described as an example.

  Here, a case will be described as an example in which it is monitored whether or not an object (person) reflected in a video to be monitored has followed a trajectory other than a trajectory designated in advance.

  When the client 50 receives a video from a surveillance camera or the like, the client 50 decomposes the video into frames. Then, the decomposed frame data (stream data) is transmitted to the data storage means 20 instructed from the queue assignment means 40. Information relating to the SID and its SID version is attached to this data. The data storage means 20 that has received the data accumulates this data as a queue 241 (see FIG. 2).

  Each filter processing means 10 determines the data storage means 20 for acquiring data from the queue state in the data storage means 20 and acquires data from the determined data storage means 20. Then, with reference to the SID and version of the acquired data and the SID table 322 (see FIG. 4) of the shared information management unit 30, the filtering process to be performed on this data is determined. For example, the filter processing unit 10 determines to execute the following FOPs in order.

(FOP11) Image format is converted from JPEG (Joint Photographic Experts Group) to Bitmap format,
(FOP12) normalize the image size to fit the feature extraction routine,
(FOP13) Black and white luminance image information is extracted from the color information,
(FOP14) Detecting a difference between the monochrome luminance image information and a background image registered in advance,
(FOP15) Label the area for the difference,
(FOP16) The area below the specified number of pixels is removed as noise,
(FOP17) For each region, the center of gravity, area, shape, etc. are detected,
(FOP18) XML (eXtensible Markup Language) tag the result,
(FOP19) Send as XML data to output destination 60A

  The output destination 60A that has received the XML data sorts the data for each information source of the XML data, obtains a sequence arranged in time order, and obtains the locus of the object in the video. When the trajectory is similar (or not similar) to the trajectory designated in advance, an alarm is issued to the supervisor.

  Here, in the stream data management system 1, when there are a plurality of persons appearing in the video, the following FOP needs to be added to each filter processing means 10 when extracting the relationship (parent-child, gender, etc.) between the persons. become.

(FOP1A1) For each label, extract the relationship between its objects,
(FOP1A2) XML tag the result

  Here, a case where the above-described FOP is added to each filter processing means 10 will be described.

  First, a system administrator or the like registers file data 323 (see FIG. 4) for realizing the above-described FOP1A1 and FOP1A2 in the shared information management means 30 in advance. Then, when there is no more data being processed in the filter processing unit 10, the filter processing unit 10 confirms that the new file data 323 exists in the shared information management unit 30, and this new file data 323. Is acquired from the shared information management means 30. Then, the filter processing means 10 inquires of the shared information management means 30 whether or not it can be restarted, and restarts when receiving a response indicating that it may be restarted. Then, the filter processing means 10 adds the functions of FOP1A1 and FOP1A2 with the file data 323. While the filter processing unit 10 is being restarted, the filter processing is temporarily stopped, but the filter processing is continued in the other filter processing units 10. Then, the filter processing means 10 that has completed the restart executes filter processing on the data by using the functions of the added FOP1A1 and FOP1A2.

  In this way, even when a new filter process is added to the filter processing unit 10, the processing capacity of the entire system is not significantly reduced.

  Next, it will be shown together with a specific example that video data used for a specific purpose can be dealt with by adding a new FOP to each filter processing means 10 when use other than the original purpose becomes necessary. .

  For example, it is assumed that the data storage unit 20 has extracted and stored feature information for video data from a surveillance camera installed in a station A. In addition, it is assumed that the data storage unit 20 stores video data from a monitoring camera in a shopping street in front of the station A for crime prevention.

  On the other hand, each filter processing means 10 extracts what people flow in what time zone and how from each data while maintaining the conventional filter processing, and the real estate company in front of the station. When it is desired to transmit to a marketing DB for vacant stores, the following FOP may be added by the same procedure as described above.

(FOP1B1) Extract from each data what kind of people and how they flow,
(FOP1B2) Send to the marketing DB for the vacant store of the real estate company in front of the station

  Next, a case where the stream data management system 1 acquires the audience rating of a television program in each home will be described.

  The audience rating sensor (client 50) of each home stores data including the ID of this home, the date and time of viewing, and channel information (viewing rate information) every predetermined period or each time the TV channel changes. It transmits to the data storage means 20. Each filter processing means 10 acquires such data from each data storage means 20 and performs the following filter processing.

(FOP21) Transfer to audience rating database as it is

  At this time, suppose that the marketing company receives provision of audience rating information from a household that has agreed to provide audience rating information in order to perform marketing utilizing audience ratings. At this time, each filter processing means 10 refers to an agreement DB (database) indicating a home ID agreed to provide audience rating information to the marketing company, and has agreed to provide audience rating information to the marketing company. Is sent to the DB of this marketing company. In addition, this agreement DB is installed in the shared information management means 30, for example.

  In this case, the FOP is as follows.

(FOP21 ′) Transfer to audience rating survey company DB and process FOP22 (FOP22) Referring to agreement DB, home ID attached to received data is registered in agreement DB Transfer the data to the marketing company's database.

  Each filter processing unit 10 acquires the file data 323 for realizing the FOP 21 ′ and FOP 22 from the shared information management unit 30 and replaces “FOP 21” with “FOP 21 ′, FOP 22”. Then, in the SID table 322 in the shared information management means 30, FOP sequences (see Table 2) corresponding to the SID of the service for transferring audience rating information to the audience rating research company DB are changed from “FOP21” to “FOP21”. ', FOP22 ".

  By doing in this way, each filter processing means 10 can add or change filter processing without interfering with the conventional audience rating information collection service.

  Note that the FOP replacement in each filter processing means 10 is performed sequentially from the filter processing means 10 that no longer has data to be processed, for example. In addition, a mechanism for changing the processing logic and a date and time when the processing logic is changed may be designated in the FOP itself. Further, the filter processing unit 10 may change such processing logic with reference to a predetermined DB.

  In addition, when the processing amount in the filter processing unit 10 varies greatly depending on the stream data (data), for example, a plurality of filter processing units 10 are autonomously connected to each other as in the present embodiment in which a large number of subjects are input. The method of executing the filter processing is less likely to cause processing delay as a whole system compared to the case of processing a single stream data by a specific filter processing means.

  Next, the stream data management system 1 collects an execution log and a communication log in this device from network devices such as a Web server, an application server, a DB server, a Web client, and an administrator client, and an abnormal access to the customer information DB A case where monitoring is performed will be described.

here,
(1) When there are three or more consecutive login password authentication errors to the customer information DB (2) When 100 or more searches are performed on the customer information DB (3) Authentication ID or password When an attack including single-auto characters is detected for a character string (4) When logging in and referring with the administrator authority of the customer information DB (5) When processing for outputting the customer information DB is performed (6) When the customer information DB is accessed with administrator authority at night or on a holiday, when the access to the customer information DB is abnormal, the filter processing means 10 performs the following processing by the filter processing unit 124 (see FIG. 1). Execute.

-Delete unnecessary characters in the data-Unify the character code of the data-Format and normalize the data representation format-Select a log for a specific server, user, service-Data according to a specific protocol Extract ・ Integrate data from multiple information sources such as Web server and DB server ・ Detect access that does not match access pattern that can be judged safe

Further, the filter processing means 10 uses the filter processing unit 124 in addition to the processing described above.
・ When abnormal data related to operator operations such as login is detected, the data is output to the output destination 60B. ・ When abnormal data related to end user operations such as special characters mixed in the authentication data is detected, A process of outputting data to the output destination 60C may be added to control the data output destination.

Furthermore, the filter processing means 10 may add the following processing.
・ Check that the login account when logging in to the customer information DB is the ID included in the registered employee DB. ・ If login to the customer information DB occurs due to an ID other than the registered employee, manage the information by e-mail. An alert to the person

  Even in such a case, since the FOP that realizes the above-described processing is added by the same procedure as described above, it is possible to prevent the monitoring of the existing access abnormality from being stopped.

  When the stream data management system 1 monitors an access error, the client 50 or the data storage unit 20 may sign a signature in order to prevent falsification of the input stream data (log data).

  Note that the filter processing unit 10 may execute the filter process using the average value of the stream data that has been subjected to the filter process in the past. This is called WindowQuery. For example, the filter processing means 10 uses a time stamp value or the like in the data stored in the DB 61 of the output destination 60 to select a predetermined period or a predetermined number from the data of this DB 61. These data may be read out, an average value of the data may be calculated, and the filtering process may be executed.

  In the above-described embodiment, the queue state monitoring unit 221 of the data storage unit 20 monitors whether the queue state has changed beyond the threshold value indicated by the queue state information notification threshold value 321 of the shared information management unit 30. However, for example, it may be monitored whether or not data exceeding a predetermined amount is stored in the storage unit 24 of the data storage unit 20. In this case, as the queue state information notification threshold value 321, for example, a value when 80% of the buffer in the storage unit 24 is exceeded is set, and the queue state monitoring unit 221 sets 80% of the buffer in the storage unit 24. Monitor whether more data is accumulated. By doing so, the filter processing unit 10 can prevent the occurrence of congestion in the data storage unit 20 in advance, so that it is possible to prevent the filter processing in the stream data management system 1 from being lowered.

  The filter processing unit 10, the data storage unit 20, and the shared information management unit 30 according to the present embodiment can be realized by a program that executes the processing as described above, and the program is stored in a computer-readable storage medium. It can be stored and provided. It is also possible to provide the program through a network such as the Internet.

It is the figure which showed the structural example of the system containing the stream data management system of this Embodiment. It is the functional block diagram which showed the detail of the data storage means of FIG. It is a figure explaining the selection process of the data storage means by the selection process part of FIG. It is the functional block diagram which showed the detail of the filter execution control part of FIG. It is a functional block diagram of the shared information management means of FIG. 2 is a flowchart showing a procedure until stream data is stored in a data storage unit in FIG. 1. FIG. 7 is a flowchart showing a procedure for updating the queue state table of FIG. It is the flowchart which showed the filter processing procedure of the filter processing means of FIG. It is the flowchart which showed the filter processing procedure of the filter processing means of FIG. It is the flowchart which showed the procedure in which the selection process part of FIG. 1 selects the data storage means which acquires data next. 3 is a flowchart showing an operation procedure of a shared information management unit that has received an inquiry as to whether or not to permit restart from the filter processing unit of FIG. 1.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stream data management system 10 Filter processing means 11, 21, 31 Input part 12, 22, 32 Processing part 13, 23, 33 Output part 14, 24, 34 Storage part 20 (20A-20N) Data storage means 30 Shared information management Means 40 Queue allocation means 50 Client 60 (60A to 60N) Output destination 61 DB (database)
62 accumulation processing unit 63 search processing unit 70 search client 121 selection processing unit 122 data acquisition processing unit 123 filter execution control unit 124 filter processing unit 126 filter determination unit 127 filter acquisition unit 128 restart inquiry unit 129 restart unit 130 filter processing application Unit 141 queue state table 221 queue state monitoring unit 222 queue state information transmission unit 241 queue 321 queue state information notification threshold 322 SID (service ID) table 323 file data 324 restart information 341 filter transmission unit 342 restart determination processing unit 343 re Startup information update unit

Claims (10)

A filter processing device used in a stream data management system that performs filtering of stream data by a plurality of filter processing devices,
An input unit for receiving input of various data;
From each of the data storage devices that store stream data transmitted from a plurality of information sources, queue status information indicating the queue status in the data storage device is received via the input unit, and based on the received queue status information A selection processing unit that selects the data storage device from which the stream data is acquired;
A data acquisition processing unit that acquires stream data stored in the data storage device from the data storage device selected by the selection processing unit via the input unit;
A storage unit for storing a filter operation describing the content of the filter processing executed by the filter processing device;
A filter processing unit that performs a filtering process based on the filter operation on the stream data;
For each service ID attached to the stream data, the service ID table indicating the filtering process to be performed on the stream data and the service ID attached to the acquired stream data are referred to, A filter execution control unit that determines a filter operation to be performed on the stream data, reads the determined filter operation from the storage unit, and causes the filter processing unit to execute a filter process based on the read filter operation;
An output unit for outputting the filtered stream data;
A filter processing apparatus comprising: The filter operation includes information on an output destination of the stream data after the filtering process,
The filter processing apparatus according to claim 1, wherein the filter processing unit determines an output destination of the stream data after the filter processing based on the filter operation. The queue status information in each of the data storage devices includes at least one value of input throughput, output throughput, and CPU cost in the data storage device,
The selection processing unit
Based on at least one of the input throughput, output throughput, and CPU cost in each of the data storage devices, the data storage device that is the stream data acquisition source in priority from the one of the higher values. The filter processing apparatus according to claim 1, wherein the filter processing apparatus is selected. The filter execution control unit
A filter determination unit for determining whether or not the filter processing function necessary for the acquired stream data is in the filter processing unit;
When the filter acquisition determining unit determines that the filter processing function necessary for the acquired stream data is not present in the filter processing unit, (1) a file required for the filter processing, and (2) the stream A filter that acquires a file necessary for the filter processing from a shared information management device that stores restart information indicating the number of filter processing devices that are permitted to be restarted in a data management system, and adds the file to the filter processing unit An acquisition unit;
A restart inquiring unit for inquiring whether or not to permit restart of the filter processing apparatus of the shared information management apparatus;
When the restart inquiry unit receives a response from the shared information management device to permit the restart of its own filter processing device, when it restarts its own filter processing device and completes the restart The filter processing according to claim 1, further comprising: a restart unit that notifies the shared information management device of completion of restart of the own filter processing device. apparatus. The restart inquiry unit
Whether or not to permit restart of the own filter processing device again when a response to permit restart of the own filter processing device is not received from the shared information management device even after a predetermined time has elapsed. The filter processing apparatus according to claim 4, wherein the filter processing apparatus inquires about whether or not. A data storage device that transmits the stream data to the filter processing device according to any one of claims 1 to 5,
When at least one of the input throughput per unit time, the output throughput, and the CPU cost in the data storage device changes to a predetermined threshold value or more, the queue status information in the data storage device is sent to the filter processing device. A data storage device characterized by transmitting the data. 6. A shared information management comprising a storage unit for storing files necessary for the filter processing in the filter processing device according to claim 4 and restart information indicating the number of the filter processing devices permitted to be restarted. A device,
In response to a request from the filter processing device, a file data transmission unit that transmits file data necessary for the filter processing;
When receiving an inquiry from the filter processing device as to whether or not to allow restart, it is determined whether or not the number of filter processing devices that permit restart in the restart information is positive, and the restart A restart determination processing unit that transmits a response to permit the restart of the filter processing device to the filter processing device,
When a response indicating that the restart is permitted is transmitted to the filter processing device, the number of filter processing devices that are permitted to be restarted in the restart information is subtracted, and the filter processing device A restart information update unit that adds the number of filter processing devices that are allowed to restart in the restart information when a notification that the restart has been completed is received;
A shared information management apparatus comprising: Each of the filter processing devices used in the stream data management system that performs stream data filtering by a plurality of filter processing devices,
Receiving queue status information indicating a queue status in the data storage device from each of the one or more data storage devices that store the stream data;
Selecting the data storage device from which the stream data is acquired based on the received queue status information;
Obtaining the stream data stored in the data storage device from the selected data storage device;
For each service ID attached to the stream data, the service ID table indicating the filtering process to be performed on the stream data and the service ID attached to the acquired stream data are referred to, Determining a filter operation to be performed on the stream data, reading the determined filter operation from the storage unit, and causing the filter processing unit to perform a filter process based on the read filter operation;
Outputting the filtered stream data; and
The filtering method characterized by performing. The queue status information in each of the data storage devices includes at least one value of input throughput, output throughput, and CPU cost in the data storage device,
The filter processing device includes:
When the data storage device from which the stream data is acquired is selected, one of the values is large based on at least one of the input throughput, output throughput, and CPU cost of each of the data storage devices 9. The filter processing method according to claim 8, wherein the data storage device from which the stream data is acquired is selected with priority.   A filter processing program for causing a computer to execute the filter processing method according to claim 8 or 9.

Images