JP4909290B2 - Stream data processing method, stream data processing program, and stream data processing system - Google Patents

Description

  The present invention relates to a stream data processing technique.

2. Description of the Related Art Conventionally, there is a stream data processing technique in which various data are collected by various sensors, IC tags, mobile phones, etc., and the collected data is subjected to filter processing and distributed to a predetermined usage destination. As such a stream data processing technique, telegraphCQ (see Non-Patent Document 1) has been proposed. In such a stream data processing technique, a technique has also been proposed in which received data is accumulated in a DB (database) and the data can be searched (see Non-Patent Document 2).
Sirish Chandrasekaran, Owen Cooper, AmoIDeshpande, Michael J. Franklin, Joseph M. Hellerstein, Wei Hong, SaileshKrishnamurthy, Sam Madden, Vijayshankar Raman, Fred Reiss, andMehul A. Shah., “TelegraphCQ: Continuous Dataflow Processing for an Uncertain CIDR (2003) Yamada et al. Real-world information management platform integrating stream processing and database, IEICE, Data Engineering Workshop DEWS 2006, 2006

  In any of the above-described techniques, a single server executes stream data processing. That is, it is not considered that the plurality of servers execute the stream data processing. For this reason, when stream data is received from a large number of information sources, there is a problem that the processing capacity of the entire system is reduced. In addition, when the server acquires stream data from a plurality of stream data acquisition destinations, even if the stream data acquisition destination is changed, the server performs a filtering process without duplication or omission on the series of stream data. Was not taken into account.

  Therefore, the present invention solves the above-described problem, and even when stream data is received from various information sources, the processing capacity of the entire system is not deteriorated, and the server duplicates a series of stream data. The purpose is to perform filtering without leaks.

  The invention according to claim 1 is: (1) a plurality of data storage devices that store stream data transmitted from a plurality of information source terminals; and (2) a filtering process for the stream data acquired from the data storage device. (3) For each SID indicating the type of the stream data, a processing procedure showing the filtering processing to be performed on the stream data having the SID and an application procedure of the filtering processing A stream data processing method in a stream data processing system comprising an information management device that transmits information to the filter processing device, wherein each of the data storage devices stores a storage data storage unit that stores the stream data, and the stream data To the stored data storage unit and to its own queue unit One or more additional recording units that perform product, and an accumulated data acquisition unit that acquires the stream data from the accumulated data storage unit and transmits the acquired stream data to the additional recording unit, When receiving from the device a queue type change notification including the change of the additional recording unit that is the acquisition destination of the stream data by the filter processing device and the SID of the stream data in which the additional recording unit that is the acquisition destination is changed, Of the processing procedure information, the information regarding the SID of the stream data is reflected on the change contents of the appending unit, the queue type change notification is transmitted to the data storage device, and the appending unit serving as the new acquisition destination is provided. The data storage device transmits the queue type change notification to the appending unit as the new acquisition destination, and changes the queue type. The appending unit that has received the knowledge records the SID of the stream data being processed in the appending unit and the reception time of the queue type change notification in the appending unit information of the storage unit of the data storage device, and the accumulated data When the acquisition unit acquires the stream data from the accumulated data storage unit, the acquisition unit refers to the additional recording unit information of the storage unit and determines that the queue type of the additional recording unit that is the transmission destination of the acquired stream data has been changed. In this case, the reception time of the queue type change notification is referred to the reception time of the queue type change notification indicated in the additional recording section information and the reception time of the acquired stream data. The earlier stream data is transmitted to the appending part before the change, and the reception time is after the reception time of the change notification of the queue type. The stream data processing method is characterized in that the stream data is not transmitted to the post-change appending unit.

  The invention according to claim 6 is: (1) an accumulated data storage unit for accumulating stream data transmitted from a plurality of information source terminals, an output of the stream data to the accumulated data storage unit, and an own queue unit A plurality of data storage devices including one or more additional recording units that perform storage, and an accumulated data acquisition unit that acquires the stream data from the stored data storage unit and transmits the acquired stream data to the additional recording unit; 2) A plurality of filter processing devices that execute filter processing on the stream data acquired from the data storage device, and (3) for each SID that indicates the type of the stream data, for the stream data having this SID. Processing procedure information indicating the filtering processing and the application procedure of the filtering processing is transmitted to the filtering device In the information management apparatus, which is a computer used in a stream data processing system including the information management apparatus, a change of the appending unit that is the acquisition destination of the stream data by the filter processing apparatus from an external device, and the acquisition destination And a step of receiving a queue type change notification including the SID of stream data to be changed, and a step of reflecting the change contents of the write-once portion in the information regarding the SID of the stream data in the processing procedure information. Transmitting the queue type change notification to the data storage device, and adding the new acquisition destination to the data storage device, which is a computer including an additional recording unit to be the new acquisition destination. Sending the queue type change notification to a section, and the queue type In the appending unit that has received the update notification, the step of recording the SID of the stream data being processed in the appending unit and the reception time of the queue type change notification in the appending unit information of the storage unit of the data storage device; When the stream data is acquired from the accumulated data storage unit, it is determined that the queue type of the additional recording unit that is the transmission destination of the acquired stream data is changed by referring to the additional recording unit information of the storage unit In this case, the reception time of the queue type change notification is referred to the reception time of the queue type change notification indicated in the additional recording section information and the reception time of the acquired stream data. The previous stream data is transmitted to the post-change appending unit, and the reception time is received by the queue type change notification. The stream data processing program is characterized in that the stream data after the transmission time is not transmitted to the post-change appending unit.

  The invention according to claim 7 is: (1) an accumulated data storage unit for accumulating stream data transmitted from a plurality of information source terminals, an output of the stream data to the accumulated data storage unit, and an own queue unit A plurality of data storage devices including one or more additional recording units that perform storage, and an accumulated data acquisition unit that acquires the stream data from the stored data storage unit and transmits the acquired stream data to the additional recording unit; 2) A plurality of filter processing devices that execute filter processing on the stream data acquired from the data storage device, and (3) for each SID that indicates the type of the stream data, for the stream data having this SID. Processing procedure information indicating the filtering processing and the application procedure of the filtering processing is transmitted to the filtering device A stream data processing system comprising: an information management device, wherein the information management device changes, from an external device, a change in an additional recording unit that is an acquisition destination of the stream data by the filter processing device, and an additional writing that is the acquisition destination When the queue type change notification including the SID of the stream data to be changed is received, the change content of the appending unit is reflected in the information regarding the SID of the stream data in the processing procedure information, and the stream data A data refilter control unit that transmits the queue type change notification to the data storage device including the appending unit as a new acquisition destination, and when the data storage device receives the queue type change notification, A process management unit that transmits the queue type change notification to the appending unit as a new acquisition destination, When the recording / adding unit receives the queue type change notification, the recording unit adds the SID of the stream data being processed in the additional recording unit and the reception time of the queue type change notification to the appending unit information of the storage unit of the data storage device When the stream data is acquired from the stored data storage unit, the stored data acquisition unit refers to the additional recording unit information of the storage unit, and transmits the acquired stream data to the additional recording unit. When it is determined that the queue type has been changed, the reception time of the stream data is determined by referring to the reception time of the queue type change notification indicated in the additional recording unit information and the reception time of the acquired stream data. The stream data prior to the reception time of the type change notification is transmitted to the appending unit before the change, and the reception time is the queue The stream data processing system is characterized in that the stream data after the reception time of the type change notification is not transmitted to the post-change appending unit.

  By doing so, the stream data processing system uses a plurality of data storage devices and a plurality of filter processing devices, so that even when stream data is received from various information sources, the processing capacity of the entire system can be improved. It can be prevented from decreasing. Even when the filter processing device changes the data storage device (data recording device appending unit) from which the stream data is acquired, the filter processing device performs the filtering process on a series of stream data without omission or duplication. Can do.

  According to a second aspect of the present invention, in the stream data processing method according to the first aspect, when the information management apparatus receives a transmission destination assignment request for the stream data including the SID from the information source terminal, For each SID stored in the storage unit, with reference to the system information indicating the identification information of the data storage device that is the transmission destination of the stream data of the SID, the transmission destination of the stream data to the information source terminal The identification information of the data storage device is transmitted, and the information source terminal transmits the stream data to the data storage device corresponding to the identification information transmitted from the information management device, and the stored data of the data storage device Each of the acquisition units outputs the acquired stream data to the appending unit when acquiring the stream data. Output to the accumulated data storage unit, the filter processing device selects the data storage device from which the stream data is acquired based on the queue state in the queue unit of the appending unit of the data storage device, and the selection The stream data obtained from the appending unit of the data storage device obtained and the stream data obtained with reference to the processing procedure information received from the information management device and the SID attached to the obtained stream data The filter processing to be executed is determined, the determined filter processing is executed, and each of the data storage devices receives stream data from the information source terminal, and stores the received stream data in the stored data storage unit. When output, the SID of this stream data, the reception time of this stream data, Additional write data position information in which identification information of the data storage device and information indicating the storage area of the stream data are associated with each other is created and transmitted to the information management device, and the information management device is transmitted from each of the data storage devices. When the stream data search request including the SID of the stream data and the reception time is received from the external device, the SID included in the search request and the The stream data processing method is characterized in that the additional data position information regarding the stream data is retrieved from the storage unit using the reception time as a key, and the retrieved additional data position information is transmitted to the external device.

  According to such a method, since the stream data processing system uses a plurality of data storage devices and a plurality of filter processing devices, even when receiving stream data from various information sources, the processing capacity of the entire system Can be prevented from dropping. In addition, when each of the filter processing devices acquires stream data from the data storage device, the filter processing device determines a data storage device from which the stream data is acquired based on a queue state in each data storage device. For example, the filter processing device acquires stream data from a data storage device having a higher input throughput. Thereby, even if it does not exchange the processing load information between filter processing apparatuses, the filter processing load of each filter processing apparatus can be balanced. Further, each data storage device transmits the stream data received from the information source terminal to the queue unit and also stores it in the stored data storage unit. Each of the data storage devices transmits the storage position of the stream data stored in the own data storage device to the information management device, and the information management device stores this storage location (additional data position information) in the storage unit. In this way, when the information management apparatus receives a search request for stream data to be refiltered from the external apparatus, the information management apparatus can respond to the storage position of the stream data to the external apparatus. That is, the external device can know the storage position of the stream data to be refiltered. The external device that has received such additional write data position information may transmit the additional write data position information as it is when transmitting the refilter processing instruction to the information management apparatus. Of the stream data indicated in the information, a predetermined stream data may be selected and a refiltering instruction may be transmitted.

  According to a third aspect of the present invention, in the filter processing method according to the second aspect, when the information management apparatus receives a transmission destination assignment request for the stream data including the SID from the information source terminal, the storage unit The system information indicating the identification information of the data storage device that is the transmission destination of the stream data of the SID is stored for each SID, and the transmission destination of the stream data to the information source terminal is referred to When transmitting the identification information of the data storage device and receiving from the external device refilter instruction information indicating additional data position information regarding the stream data to be refiltered and the content of the refilter processing, Issue a refilter SID to be used for the stream data to be refiltered, and store it in the system information of the storage Information that associates the re-filter SID with identification information of a data storage device that stores stream data having the SID is added, and the issued re-filter SID and the processing procedure information in the storage unit are added. By adding information associated with the contents of the refiltering process for the stream data having this SID, the contents of the refiltering process are reflected in the processing procedure information, and the process reflecting the contents of the refiltering process is performed. Procedure information is transmitted to the filter processing device, one of the plurality of data storage devices is selected, and the re-filter including the re-filter SID issued to the selected data storage device is selected. The data storage device that has transmitted the filtering process instruction information and received the refiltering instruction information receives the received refiltering instruction. The stream data that is the target of the refiltering process is acquired from the storage area of the data storage device indicated in the information, and the refiltering SID is given to the stream data that is the target of the acquired refiltering process, Using the refilter SID included in the refilter instruction information as a key, the data management device acquires identification information of the data storage device to be connected to the data storage device from the information management device, and the data indicated by the acquired identification information When connecting to a storage device and transmitting the stream data to the data storage device, the filter processing device acquires the stream data having the SID for refiltering from the additional recording unit of the data storage device, and the information Refer to the processing procedure information reflecting the contents of the re-filtering process received from the management apparatus, and The stream data processing method is characterized in that filter processing is executed.

  According to such a method, the external apparatus can receive the storage position of the stream data to be refiltered from the information management apparatus. That is, the external device transmits refilter instruction information including the storage position of the stream data to be refiltered to the information management device. Then, the information management device causes the data storage device in the system to acquire the stream data from the data storage device indicated by the refilter processing instruction information. Then, the data storage device that has acquired this stream data flows this stream data into the queue section of its own data storage device. Next, when the filter processing apparatus acquires the stream data to be refiltered into the queue unit of the data storage apparatus, the filter processing apparatus receives the processing procedure information received from the information management apparatus (the contents of the refiltering process). With reference to the reflected processing procedure information, re-filtering processing can be executed on this stream data.

  According to a fourth aspect of the present invention, in the stream data processing method according to the third aspect, the information management apparatus receives an SID, a reception time, and a stream of the stream data to be refiltered from the external apparatus. When refilter instruction information indicating the content of the refilter process to be performed on the data is received, the stream data is received from the storage unit using the SID and reception time of the stream data to be refiltered as a key. The postscript data position information relating to the rewrite processing is searched for, and the refilter instruction information indicating the content of the refilter process is sent to any one of the plurality of data storage devices. The stream data processing method is characterized by transmitting.

  According to such a method, the information management apparatus can execute the refiltering process in the filter processing apparatus without transmitting additional data position information regarding the stream data to be refiltered to the external apparatus.

  According to a fifth aspect of the present invention, in the stream data processing method according to the third or fourth aspect, the content of the refiltering process indicated in the received refiltering instruction information by the information management device is the stream. If it is determined whether or not the window processing for the data is included, and the content of the refiltering process indicated in the received refiltering instruction information does not include the window processing for the stream data, the refiltering instruction information When the data storage device that is the transmission destination is selected, the additional write data position information related to the stream data retrieved from the storage unit is divided for each data storage device having the stream data, and the divided additional write data position information Is included in the refilter instruction information, and the data storage having the stream data is included. And a stream data processing method and transmitting to the device, respectively.

  According to such a method, if the content of the refiltering process does not need to perform window processing on the stream data, the information management apparatus stores each data storage apparatus having the stream data in its own data storage apparatus. Stream data is acquired from the data storage unit and poured into the appending unit. Therefore, each data storage device can efficiently read out the stream data to be refiltered.

  According to the present invention, even when the stream data processing system receives stream data from various information sources, the processing capacity of the entire system is not reduced. Further, the stream data processing system can perform a filtering process on a series of stream data without omission or duplication.

  Hereinafter, the best mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described by dividing it into a first embodiment to a third embodiment. Note that the stream data in each embodiment is, for example, data in XML (Extensible Markup Language) format.

<< First Embodiment >>
FIG. 1 is a block diagram illustrating a stream data processing system according to the first embodiment. A stream data processing system (hereinafter, abbreviated as a system as appropriate) adds a service ID (SID) to data created in time series such as measurement data and log data, and adds a client (information) that transmits this data as stream data Source terminal) 50 (50A to 50N), data storage device 20 (20A to 20N) that stores stream data transmitted from the append client 50, and obtains stream data from the data storage device 20 and performs filter processing. Filter processing apparatus 10 (10A to 10N), database apparatus 60 (60A to 60N) for storing the stream data after the filtering process, and reference client (external apparatus) 70 (70A to 70A to retrieve data from the database apparatus 60) 70N), the data storage device 20 and the fill Configured to include an information management apparatus 30 that manages various information for operating the processor 10. The SID is identification information for identifying the type of stream data and the transmission source (information source) of the stream data.

  In addition, (1) between the additional recording client 50 and the information management device 30 and the data storage device 20 (20A to 20N), (2) between the information management device 30 and the data storage device 20 (20A to 20N), (3) Between the data storage devices 20 (20A to 20N), (4) Between the data storage devices 20 (20A to 20N) and the filter processing device 10 (10A to 10N), (5) The filter processing device 10 (10A) 10N) and the database device 60 (60A to 60N), and (6) the database device 60 (60A to 60N) and the reference client 70 (70A to 70N) are connected by a network (not shown). The Furthermore, although the number of the additional recording client 50, the data storage device 20, the filter processing device 10, the database device 60, and the reference client 70 is N, it is not limited to this.

<Append client>
The additional recording client (external device) 50 is a computer having a communication interface and the like. The additional recording client 50 sequentially transmits the data with the SID assigned to the additional recording client 50 to the data storage device 20 as stream data.

  The stream data to be transmitted at this time is data measured in time series such as temperature data of various places, computer logs, and the like, and is described in XML format, for example. Further, it is assumed that the appending units 221 and 222 of the data storage device 20 that is the connection destination of the appending client 50, that is, the transmission destination of the stream data, is instructed (allocated) by the information management device 30.

<Information management device>
The information management device 30 accumulates various data used for the filter processing, assigns the connection destination of the additional recording client 50, records stream data storage information (additional data position information) in the data storage device 20 (20A to 20N), A refilter instruction from the reference client 70 is received.

  Such an information management device 30 includes a storage unit 32, a connection destination allocation unit 31, a data refilter control unit 33, a data position management unit 34, an information management unit 35, and a restart processing unit 36. . Although not shown here, the information management device 30 is a computer including an input unit that controls input of various data, an output unit that controls output, a CPU (Central Processing Unit), a memory, a communication interface, and the like. It is realized by.

  The storage unit 32 includes system information 321, information source information (SID information) 322, schema information 323, processing procedure information 324, data location management DB (database) 325, and restart information 326 in a predetermined area.

  The system information 321 is information indicating the transmission destination of the stream data of the SID for each SID. The transmission destination here is, for example, information (see Table 1) indicating identification information (such as an IP address) of the data storage device 20 and identification information (ID) of the additional recording units 221 and 222 in the data storage device 20. This system information 321 is referred to by the connection destination assignment unit 31 and the accumulated data acquisition unit 25 (described later) of the data accumulation device 20. Note that the port numbers assigned to the additional recording units 221 and 222 may be used as the IDs of the additional recording units 221 and 222.

  FIG. 2 is a diagram illustrating information source information and schema information of FIG.

  The information source information (SID information) 322 is definition information of the SID, as shown by reference numeral 201 in FIG. 2, identification information (SCHEMA) of schema information of the stream data having the SID, and the stream data Is information indicating the filter identification information (FILTER), the window size (WINDOW_SIZE) of this stream data, and the like. The information source information indicated by reference numeral 201 in FIG. 2 is the definition information of the SID “DML001”, the schema of this stream data is the schema defined in “DML001_SCHEMA”, and the filter used for this stream data is “XpathFilter (“ update ")", indicating that the window size of this stream data is "1".

  The schema information 323 is definition information of a stream data schema having the SID. The schema information indicated by reference numeral 202 in FIG. 2 is “DML001_SCHEMA”, that is, schema information of the SID “DML001”. The schema of this stream data is “col_t” is “timestamp”, and “col_id” is “int”. "Col_d" indicates "xml".

  The processing procedure information 324 indicates, for each SID, a filtering process to be applied by the filter processing apparatus 10 to the stream data of the SID, that is, a filter operation (FOP) sequence (an FOP to be applied and an FOP application procedure). It is. Further, as illustrated in Table 2, this processing procedure information includes stream data schema information (schema definition information) and a window size when performing filtering (refer to values in parentheses in the schema information of Table 2). You may go out.

  For example, the schema information of the SID “10” in the processing procedure information shown in Table 2 indicates that 10 data (records) are required when the filter processing related to “NAME” is performed. Further, it is indicated that FOPs “FOP102, FOP103, FOP104” are applied to this stream data. The schema information is the same as that stored in the schema information 323.

  The data position management DB 325 stores additional write data position information indicating the storage position of the stream data stored in each data storage device 20 (20A to 20N) in the system. As illustrated in FIG. 6 to be described later, the additional data position information corresponds to the SID of the stream data, the time when the data storage device 20 received the stream data, and the storage position (additional data position) of the stream data. This is what I remembered. The additional recording data position is identification information of the data storage device 20 that stores the stream data and information that indicates the storage position in the data storage device 20.

  As described above, the information management apparatus 30 grasps the storage position of the stream data stored in each data storage apparatus 20 by the data position management DB 325, so that the information management apparatus 30 stores the stream stored in each data storage apparatus 20. Stream data to be refiltered can be efficiently found from the data. Note that this additional write data position information is received from each data storage device 20. Further, the time at which the stream data is stored in the stored data storage unit 24 (described later) of the data storage device 20 may be used instead of the time (reception time) at which the stream data in the additional data position information is received. For example, B-Tree, B + Tree, R-Tree, Red Black Tree, or the like is used as a method for managing additional data location information in the data location management DB 325.

  Returning to the description of FIG. The restart information 326 is information indicating the number of filter processing devices 10 that are permitted to restart in the system. For example, when the number of filter processing apparatuses 10 that may stop the filter processing by restarting is “10” in the entire system, “10” is set as the initial value of the restart information 326. The restart information 326 is referred to when the restart processing unit 36 (described later) determines whether or not to permit restart of the filter processing apparatus 10.

  When the connection destination assignment unit 31 receives a connection destination assignment request from an external device (the append client 50 or the data storage device 20), the external device connects from the system information 321 using the SID included in the connection destination assignment request as a key. The connection destination (additional recording units 221 and 222 in the data storage device 20) to be searched is searched. Then, the connection destination assigning unit 31 instructs the searched connection destinations (additional recording units 221 and 222 in the data storage device 20) as the connection destination of the external device. Note that the connection destination instructed here is, for example, a combination of the identification information (IP address or the like) of the data storage device 20 and the port numbers of the appending units 221 and 222. Further, the connection destination assigning unit 31 may select a connection destination with a smaller processing load in consideration of the load state of each data storage device 20. In other words, the connection destination assigning unit 31 may change the connection destination to balance the load between the data storage devices 20.

  FIG. 3 is a block diagram showing an outline of the operation of the stream data processing system of FIG. As shown in FIG. 3, the data refilter control unit 33 receives information indicating the refilter instruction (SID of stream data to be refiltered, time interval (or start time), etc. from the reference client 70, When the content of the refiltering process is received, an arbitrary machine (data storage device 20) in the system is determined. Thereafter, the data refilter control unit 33 transmits a refilter instruction to the process management unit 26 (described later) of the determined data storage device 20. Here, the re-filtering process includes, for example, a change in threshold value and parameters used for the filtering process, a change in the filtering process procedure, and the like. Further, the refilter instruction includes additional data position information regarding the stream data to be refiltered. The additional data location information is acquired from the data location management DB 325 by the data location management unit 34.

  At this time, the data re-filter control unit 33 issues a new SID (re-filter SID) for re-filter processing (for re-filter) and includes it in the re-filter instruction. The data refilter control unit 33 sends the information about the refiltering SID and stream data having the SID to the information management unit 35 (described later), system information 321, information source information 322, schema information 323, An instruction to additionally register the processing procedure information 324 is given.

  For example, when the data refilter control unit 33 receives a refilter instruction to perform refiltering on stream data of SID “10” and time interval “A to B” (or time “A”), A new SID “30” for re-filtering is issued for the stream data. Then, the data refilter control unit 33 adds information on the stream data having the SID to the information management unit 35 (described later) to the system information 321, the information source information 322, the schema information 323, and the processing procedure information 324. Instruct. In addition, the data refilter control unit 33 informs the information management unit 35 (described later) to the processing procedure information 324 related to the refiltering SID in the contents of the refiltering process (for example, the threshold value of the filter). Instructions to add changes). Note that the data refilter control unit 33 issues a new SID (refilter SID) in this way and assigns it to the stream data to be refiltered even if it is originally stream data having the same SID. This is because the refiltering process can be applied only to the stream data to be refiltered.

  The data position management unit 34 stores additional data position information received from each data storage device 20 in the data position management DB 325.

  FIG. 4 is a block diagram showing an outline of the operation of the stream data processing system of FIG. For example, as shown in FIG. 4, when the data position management unit 34 receives additional data location information including the SID, time, and additional data location from each additional recording unit 221, 222 (described later) of the data storage device 20, Data position information is additionally registered in the data position management DB 325 sequentially. In addition, the data position management unit 34 performs a refilter instruction (or a request for retrieving additional data position information) including the time interval (the time from the start to the end of the reception time) of the stream data or the start time and the SID. Is received, the additional data location information is retrieved from the data location management DB 325 using the time interval and the SID as a key. That is, it is searched in which storage location of which data storage device 20 the stream data indicated by the refilter instruction is stored. Note that the data position management unit 34 may transmit the searched additional data position information to the reference client 70 as an additional data position list.

  Returning to the description of FIG. The information management unit 35 adds or changes each piece of information in the storage unit 32. For example, the threshold value change information of the filtering process that is the content of the refiltering process transmitted from the reference client 70 is reflected in the processing procedure information 324. Further, in response to a request from each filter processing device 10, information source information 322, schema information 323, processing procedure information 324, and the like are transmitted to the filter processing device 10. As a result, the contents of the refiltering process newly added to the information management apparatus 30 can be reflected on the filter processing apparatus 10.

  When the restart processing unit 36 receives an inquiry from the filter processing device 10 as to whether or not to permit restart, the restart processing unit 36 refers to the restart information 326 and determines whether or not to permit restart of the filter processing device 10. Respond. For example, if the value indicated in the restart information 326 is “1” or more, restart is permitted, but if it is “0”, restart is not permitted. Further, when the restart processing unit 36 permits the restart of one filter processing apparatus 10, the value of the restart information is decreased by “1”. When the restart of the filter processing device 10 is completed, the restart processing unit 36 restores the value of the restart information 326 to the original value. By doing so, the number of filter processing devices that stop the filter processing by restarting in the system can be reduced to a predetermined number or less, and it is possible to prevent the efficiency of the filter processing in the system from being extremely lowered.

  The storage unit described above is realized by a RAM (Random Access Memory), a ROM (Read Only Memory), a HDD (Hard Disk Drive), or the like. The connection destination assignment unit 31, the data refilter control unit 33, the data position management unit 34, the information management unit 35, and the restart processing unit 36 are realized by a program execution process by the CPU or dedicated hardware. When realizing by program execution processing by the CPU, the storage unit 32 stores these programs.

<Data storage device>
Each of the data storage devices 20 (20A to 20N) temporarily stores the stream data transmitted from the additional recording client 50. Such a data storage device 20 includes an input unit 21, an output unit 23, additional recording units 221 and 222, a stored data storage unit 24, a stored data acquisition unit 25, and a process management unit 26. In addition, the number of additional recording parts 221 and 222 is not limited to the number shown in FIG.

  The input unit 21 receives stream data from the write-once client 50. The output unit 23 outputs the stream data output from the appending units 221 and 222 to the filter processing device 10. Further, each of the additional recording units 221 and 222 outputs the received stream data to the queue unit 226 (described later) and accumulates it in the accumulated data storage unit 24. Such additional recording units 221 and 222 include a stream reception unit 223, an additional recording data storage unit 224, a queue management unit 225, a queue unit 226, and an additional recording data position information transmission unit 227.

  The stream receiving unit 223 receives stream data from the additional recording client 50 via the input unit 21. Then, the received stream data is output to the additional write data storage unit 224 and the queue management unit 225.

  The additional data accumulation unit 224 accumulates the stream data in the accumulation data storage unit 24. The additional data storage unit 224 preferably stores the stream data received from the additional client 50 in the stored data storage unit 24 as it is. For example, when a data base management system (DBMS) or the like is used for managing stream data in the accumulated data storage unit 24, various journals, checkpoints, and the like accompanying data transaction processing are required. This is because the efficiency is poor when data is appended.

  The data file stored in the stored data storage unit 24 is divided by a predetermined time or a predetermined data size. At the end of each divided data file, the path to the next data file is described. FIG. 5 is a diagram illustrating data files stored in the stored data storage unit of FIG. The accumulated data storage unit 24 accumulates a data file including Type (data type), Length (length), Value (content of data), and the like. For example, the additional data storage unit 224 divides and stores the data file into files A, B, and C with a predetermined data size. As shown in FIG. 5, the path to the next file, file B, is written at the end of file A, and the path to the next file, file C, is written at the end of file B. By doing in this way, when the filter processing apparatus 10 performs the refiltering process on the data stored in the stored data storage unit 24, the refiltering process can be performed without interruption. Further, by storing the data file by dividing the file at a predetermined data size or at a predetermined time as described above, it becomes easier to perform recovery and backup when a failure occurs in the data file.

  Returning to the description of FIG. The queue management unit 225 outputs the data (stream data) output from the stream reception unit 223 to the queue unit 226. When there is a stream data acquisition request from the filter processing device 10, the stream data stored in the queue unit 226 is transmitted to the filter processing device 10.

  When the stream data is accumulated (stored) in the accumulated data storage unit 24 in the additional recording units 221 and 222, the additional recording data position information transmission unit 227 creates additional recording data position information of the stream data. Information is transmitted to the information management device 30. Note that the additional write data position indicated in the additional write data position information is described by, for example, a URI (Uniform Resource Identifier), a file path, a location ID, and the like.

  The queue unit 226 accumulates the stream data output from the queue management unit 225 as a queue.

  The accumulated data storage unit 24 accumulates the stream data output from the appending units 221 and 222.

  As shown in FIG. 3, when the accumulated data acquisition unit 25 receives the refilter instruction from the process management unit 26, based on the refilter instruction, the accumulated data acquisition unit 20 or another data accumulation device 20 Stream data is acquired from the accumulated data storage unit 24. Then, the refiltering SID instructed from the process management unit 26 is added to the acquired stream data. Here, the accumulated data acquisition unit 25 reads to the end of the data file in the accumulated data storage unit 24 (see FIG. 5), and when the path to the next data file is described, the next data file indicated by this path is displayed. Read the data file. Then, the accumulated data acquisition unit 25 inquires of the connection destination allocation unit 31 of the information management apparatus 30 about the transmission destination of the stream data to which the refiltering SID is assigned, and is instructed by the connection destination allocation unit 31. This stream data is poured into the additional recording unit (for example, the additional recording unit 222) of the data storage device 20. The stream data is accumulated in the queue unit 226 of the appending units 221 and 222, and then acquired by the filter processing apparatus 10 and refiltered.

  The process management unit 26 controls the accumulated data acquisition unit 25 and the additional recording units 221 and 222 in the data storage device 20. For example, when the process management unit 26 receives a refilter instruction from the information management apparatus 30, based on the contents of the instruction, the process management unit 26 sends stream data to be refiltered to the accumulated data acquisition unit 25 of the own data storage apparatus 20. The acquisition of is instructed.

  The input unit 21 and the output unit 23 are realized by an input / output interface capable of inputting / outputting data to / from the write-once client 50 and the filter processing apparatus 10.

  The data storage device 20 is also realized by a computer, and the stored data storage unit 24 is realized by, for example, a RAM, an HDD, or the like. Further, the accumulated data acquisition unit 25, the process management unit 26, the stream reception unit 223 of the additional recording units 221, 222, the additional data storage unit 224, the queue management unit 225, and the additional data position information transmission unit 227 perform program execution processing by the CPU. Alternatively, it is realized by dedicated hardware or the like. The queue unit 226 is formed in the RAM or the like of the data storage device 20.

  The queue management unit 225 monitors the queue state (input throughput, output throughput, CPU cost, etc.) of the queue unit 226. Then, when the queue state changes to a predetermined threshold value or more, the queue state information in the data storage device 20 may be transmitted to each filter processing device 10. At this time, the queue state information is added with identification information of its own data storage device 20. In addition, transmission of the queue status information at this time is performed by broadcast transmission to each filter processing device 10 or the like. In this way, the data storage device 20 can quickly notify each filter processing device 10 of its own queue state when its own queue state has changed.

<Filter processing device>
The filter processing device 10 acquires stream data from the data storage device 20 (20A to 20N), and executes filter processing on the acquired stream data. Such a filter processing apparatus 10 includes an input unit 11, an output unit 13, and a processing unit 12 (12A, 12B). The number of processing units is not limited to the number shown in FIG.

  The input unit 11 receives stream data from the data storage device 20. The output unit 13 outputs the filtered stream data to the database device 60.

  The processing unit 12 includes a queue selection unit 121, a data acquisition unit 122, a filter execution control unit 123, filter processing units 124 (124A to 124N), and a restart inquiry unit 125.

  The queue selection unit 121 selects the data storage device 20 that is the stream data acquisition source. The selection of the data storage device 20 at this time is, for example, information indicating queue status information (input throughput, output throughput, CPU cost (filter cost), etc.) of the data storage device 20 transmitted from the data storage device 20. ), The data storage device 20 having a larger input throughput value is selected with higher probability (priority).

  The data acquisition unit 122 acquires stream data from the data storage device 20 selected by the queue selection unit 121. Note that the timing of acquiring this stream data may be, for example, when there is no data processed in its own filter processing device 10 or when the data being processed falls below a predetermined threshold.

  The filter execution control unit 123 refers to the SID attached to the stream data acquired from the data acquisition unit 122 and the information source information 322, schema information 323, and processing procedure information 324 received from the information management apparatus 30. Then, a filter process to be executed on the stream data having this SID is determined. Here, the filtering process includes an FOP and an application procedure of the FOP. Then, the filter execution control unit 123 causes the filter processing unit 124 to execute the determined filter processing. The information source information 322, schema information 323, and processing procedure information 324 acquired from the information management device 30 are stored in a storage unit (not shown) of the filter processing device 10 and are referred to as appropriate when determining the filter processing. .

  Note that the filter execution control unit 123 receives the stream data to which the refiltering SID is added when the filter processing device 10 does not have the filter processing function to be executed on the stream data (for example, When there is no processing procedure information for the SID), file data (including a filter operation) for realizing the filter processing function is downloaded from the information management apparatus 30 and the function is added. By doing in this way, the filter processing apparatus 10 can perform a re-filter process reliably.

  Based on an instruction from the filter execution control unit 123, the filter processing unit 124 performs filter processing on the stream data. The contents of the filter processing in the filter processing unit 124 are described in FOP, and the filter processing unit 124 executes the filter processing based on this FOP. This FOP is, for example, “to extract stream data having temperature information of X ° C. or higher” or “to output processed data to the database device 60A”. This FOP is stored as part of the processing procedure information in a storage unit (not shown) of the filter processing apparatus 10.

  Note that after the filter acquisition unit 127 acquires the schema information 323, the restart inquiry unit 125 inquires of the information management device 30 whether or not to permit restart of its own filter processing device 10. Then, when receiving a response from the information management device 30 to permit the restart of the own filter processing device 10, the own filter processing device 10 is restarted. By doing so, in the stream data processing system, it is possible to prevent the filter processing devices 10 exceeding a predetermined number from being stopped due to restart, and the filter processing capability of the entire system from being significantly reduced.

  Thereafter, the filter processing unit 124 outputs the stream data after the filtering process to the database device 60 via the output unit 13.

  The filter processing apparatus 10 is also realized by a computer including a CPU, RAM, HDD, communication interface, and the like. The queue selection unit 121, the data acquisition unit 122, the filter execution control unit 123, the filter processing unit 124, and the restart inquiry unit 125 are realized by a program execution process by the CPU or dedicated hardware.

<Database device>
The database device 60 includes a receiving unit 62 (62A to 62N) that receives stream data filtered by the filter processing device 10, a DB (database) 61 (61A to 61N) that stores the stream data, and a search from the reference client 70. A search unit 63 (63A to 63N) that searches data from the DB 61 (61A to 61N) in response to a request is provided. The data storage device 20 is also realized by a computer having a CPU, RAM, HDD, communication interface, and the like. The reception unit 62 and the search unit 63 are realized by a program execution process by the CPU or dedicated hardware. The DB 61 is realized by an HDD or the like. Instead of the search unit 63, a push-type delivery unit that transmits data stored in the DB 61 to the reference client 70 may be used.

<Reference client>
The reference client 70 retrieves data from the database device 60, and indicates to the information management device 30 a refilter instruction (contents of refilter processing, SID of stream data to be refiltered, time interval, etc. Information) and instructing refiltering of the stream data stored in the data storage device 20. This reference client 70 is also realized by a computer having a CPU, RAM, HDD, communication interface, and the like.

  The filter processing device 10 (10A to 10N) and the data storage device 20 (20A to 20N) may be realized by separate computers (devices) or may be realized by a single computer. You may do it.

<Operation procedure>
Next, the operation procedure of the system shown in FIG. 1 will be described with reference to FIG. 1 and FIG. 6 and FIG. 6 and 7 are diagrams showing an operation procedure of the stream data processing system of FIG. First, the procedure in which the information management apparatus 30 acquires and stores the additional write data position information will be described with reference to FIG.

  When the additional recording client 50 receives the data (stream data) 100 from the information source, the additional recording client 50 sends an additional recording destination payout request including the SID (for example, “SID-kana99”) attached to the data to the connection destination allocation unit 31 of the information management apparatus 30. (S41). That is, the additional recording client 50 inquires of the information management apparatus 30 about the additional recording units 221 and 222 of the transmission destination of the stream data (the IP address of the data storage device 20 including the additional recording unit and the port number for connecting to the additional recording unit). .

  Upon receiving the additional write destination payout request transmitted in S41, the connection destination assigning unit 31 searches the system information 321 using the SID indicated in the additional write destination payout request as a key. The connection destination assigning unit 31 then adds the IP address of the data storage device 20 with the additional recording unit (for example, the additional recording unit 221) of the transmission destination of the stream data and the additional recording destination indicating the Port (port number) of the additional recording unit 221. Payout (S42).

  The additional recording client 50 connects to the additional recording unit (for example, the additional recording unit 221) paid out in S42 (S43). Then, the write-once client 50 transmits, for example, the write-once data 100 having “SID-kana99” to the connected write-once unit 221. That is, the stream data is added (S44).

  When the additional data 221 is received by the stream reception unit 223 (see FIG. 1), the additional data 221 acquires the reception time (reception time) of the additional data 100 (S45). That is, the reception time of the additional data 100 is recorded. Further, the additional recording unit 221 stores the received stream data (additional data) in the stored data storage unit 24 by the additional data storage unit 224 (see FIG. 1) (S46). Next, the additional write data position information transmission unit 227 creates additional write data position information of the accumulated additional write data 100. Then, the additional write data position information transmission unit 227 transmits the created additional write data position information to the information management apparatus 30 (S47). This additional write data position information includes the SID ("SID-kana99") of the accumulated additional write data 100, the reception time ("2007/02 / 20T 10:00") acquired in S45, and the additional write data position ("macA.ntt"). /App99/b.log "). The additional data position is “macA.ntt” in the data storage device 20 that stores the additional data 100, and the storage area in the stored data storage unit 24 of the data storage device 20 is “/app99/b.log”. ".

  The data position management unit 34 of the information management device 30 that has received such additional write data position information stores the additional write data position information in the data position storage DB 61 (see the additional write data position information 401 in FIG. 6).

  Further, the additional recording unit 221 of the data storage device 20 stores the additional recording data 100 in the queue unit 226 by the queue management unit 225 (see FIG. 1) (S48: queue management).

  Thereafter, when the additional recording unit 221 receives the additional recording data 101 having the same SID “SID-kana99” by the stream reception unit 223 (see FIG. 1) (S49), the additional recording unit 101 reads the additional recording data 101 in the same manner as described above. The reception time (reception time) is acquired (S50). Further, when the additional data storage unit 224 accumulates the additional data 101 in the accumulated data storage unit 24 (S51), the additional recording unit 221 stores the additional data location information (SID (SID ()) of the stored data 101 in the same manner as described above. “SID-kana99”), the reception time acquired in S50 (“2007/02 / 20T 11:00”), and the postscript data position (“macA.ntt/app99/b.log”) are created, and this The postscript data position information is transmitted to the information management device 30 (S52).

  The data position management unit 34 of the information management apparatus 30 that has received such additional write data position information stores the additional write data position information in the data position storage DB 61 (see reference numeral 404 in FIG. 6).

  Then, the additional recording unit 221 of the data storage device 20 stores the received additional recording data 101 in the queue unit 226 (S53: queue management).

  By repeating the processing as described above, each data storage device 20 stores stream data, and the information management device 30 stores additional data position information of the stream data stored in each data storage device 20 in the data position storage DB 61. I will do it.

  Next, a procedure from when the reference client 70 transmits a refilter instruction until the filter processing apparatus 10 executes the refiltering process will be described with reference to FIG.

  The reference client 70 transmits a refilter instruction including the SID of the stream data to be refiltered, the time interval, and the content of the refiltering process to the data refilter control unit 33 of the information management device 30 (S61). ).

  The data refilter control unit 33 of the information management apparatus 30 that has received the refilter instruction issues this refiltering SID (refilter SID) and assigns the refilter SID (S62). That is, the data refilter control unit 33 provides the information management unit 35 with the issued refiltering SID and information on the stream data having the SID, the system information 321, the information source information 322, the schema information 323, and the processing procedure information. 324 is instructed to register.

  Further, the data refilter control unit 33 registers the refilter process in the process procedure information 324 (S63). In other words, the data refilter control unit 33 provides the information management unit 35 with the contents of the refiltering process indicated by the refiltering instruction as the processing procedure information 324 related to the refiltering SID (for example, changing the threshold value in the filter, etc. ). Further, the data refilter control unit 33 sends the refiltering SID and the additional recording units 221 and 222 of the data storage device 20 that is the transmission destination of the stream data having the SID to the system information 321 to the information management unit 35. Information associated with information is added.

  Next, the data refilter control unit 33 searches the data position management DB 325 for the additional data position information of the stream data to be subjected to the refiltering process (S64). Specifically, the data refilter control unit 33 adds data to the data position management unit 34 (see FIG. 1) using the SID included in the refilter instruction and the time interval (or start time) as keys. Instructs to retrieve data location information. Then, the search result is received.

  Then, the data refilter control unit 33 determines one machine (data storage device 20) to be a refilter addition client from the data storage devices 20 in the system (S65). The machine (data storage device 20) at this time is determined by, for example, round robin.

  Next, the data refilter control unit 33 transmits a refilter instruction including the additional data position information searched in S64 and the refiltering SID to the process management unit 26 of the determined data storage device 20 ( S66).

  The process management unit 26 of the data storage device 20 that has received the refilter instruction from the data refilter control unit 33 activates the accumulated data acquisition unit 25 and activates the own data storage device 20 as a refilter addition client (S67). . Then, the refilter instruction including the additional data position information received in S66 and the refilter SID is transferred to the accumulated data acquisition unit 25 (S68).

  When the re-filtering instruction is received, the accumulated data acquiring unit 25 transmits an additional write destination payout request to the connection destination assigning unit 31 of the information management device 30 (S69). The additional write destination payout request at this time includes the refiltering SID included in the refiltering instruction. That is, the accumulated data acquisition unit 25 reads the stream data that is the target of refiltering, and then inquires of the additional recording units 221 and 222 to the connection destination allocation unit 31.

  Then, the connection destination assigning unit 31 that has received such a payout request uses the refiltering SID as a key to add the stream data having the refiltering SID from the system information 321 (see FIG. 1). The IP address and Port (port number) are searched. Then, the connection destination assignment unit 31 pays out the searched IP address and Port (port number) as an additional write destination (S70).

  The accumulated data acquisition unit 25 then connects to the additional recording unit 221 indicated by the IP address and Port (port number) issued from the connection destination allocation unit 31 (S71).

  Then, the accumulated data acquisition unit 25 re-filters the data (stream data) from the storage position (storage area) of the accumulated data storage unit 24 indicated in the additional data position information of the refilter instruction transferred in S68. Is read (S72).

  In S72, if the data to be refiltered is stored in the stored data storage unit 24 of the own data storage device 20, it is read from the stored data storage unit 24 of the own data storage device 20, and the other data If the data is stored in the stored data storage unit 24 of the data storage device 20, it is read from the stored data storage unit 24 of the other data storage device 20 via a network (not shown).

  When the accumulated data acquisition unit 25 reads the data to be refiltered in this way, the SID of the read data (additional data) is replaced with the refilter ID included in the refilter instruction transmitted in S68. , Flow into the additional recording unit 221 connected in S71. That is, the stream data is added to the appending unit 221 (S73). Then, the additional recording unit 221 performs queue management in the same manner as S48 and S53 of FIG. 6 described above (S74).

  Thereafter, when the filter processing apparatus 10 acquires data from the additional recording unit 221 of the data storage apparatus 20 (S75), the filter processing apparatus 10 performs a filtering process on this data (S76). That is, if the data acquired from the appending unit 221 is data having a refiltering SID, the refiltering process is performed on this data. Next, the processing of S75 and S76 of FIG. 7 will be described in detail with reference to FIG. FIG. 8 is a diagram for explaining the processing of S75 and S76 of FIG. 7 in detail.

  The data acquisition unit 122 of FIG. 1 checks whether or not there is data to be processed (S701), and if it is detected that there is no data to be processed (Yes in S701), the queue selection unit 121 next The data storage device 20 for acquiring data is selected (S702). The selection of the data storage device 20 at this time is preferentially selected, for example, as the data storage device 20 having a larger input throughput value based on the queue status information transmitted from each data storage device 20 as described above. To do.

  The data acquisition unit 122 acquires data from the data storage device 20 selected in S702 (S703). The acquired data is output to the filter execution control unit 123.

  Next, the filter execution control unit 123 refers to the processing procedure information 324 and determines a necessary filter processing function for this data using the SID attached to the acquired data as a key (S704). Then, the filter execution control unit 123 determines whether or not the filter processing function necessary for the data is in its own filter processing apparatus 10 (S705). Here, when the necessary filter processing function is not in the own filter processing apparatus 10 (No in S705), the filter execution control unit 123 receives the necessary filter processing function (information source information 322, schema, etc.) from the information management apparatus 30. Information 323, processing procedure information 324, etc.) is acquired (S706). For example, when the data acquired by the filter execution control unit 123 is data with a refiltering SID, the filter processing function corresponding to the refiltering SID has not yet been downloaded (acquired) to the filter processing apparatus 10. Most likely not. Therefore, when receiving the stream data including the re-filter SID, the filter execution control unit 123 downloads the filter processing function corresponding to the SID from the information management apparatus 30. That is, the information management device 30 transmits the filter processing function corresponding to this SID to the filter processing device 10. Then, the filter execution control unit 123 of the filter processing apparatus 10 adds a filter processing function to the filter processing unit 124.

  On the other hand, when the necessary filter processing function is present in the own filter processing apparatus 10 (Yes in S705), the process proceeds to S707.

  Then, the filter execution control unit 123 executes filter processing on the data acquired in S703 (S707). That is, the filter execution control unit 123 refers to the SID of the stream data, the information source information 322, the schema information 323, and the processing procedure information 324 to determine the content of the filtering process for the stream data, and this filter FOP for executing the process is read from the storage unit. Then, the filter execution control unit 123 causes the filter processing unit 124 to execute the filter processing in a predetermined order based on this FOP, and the filter processing unit 124 outputs this data to the predetermined database device 60 after the filter processing (S708). ).

  In this way, the information management apparatus 30 can cause each filter processing apparatus 10 to execute the refiltering process.

  Although not described here, after the filter execution control unit 123 of the filter processing apparatus 10 acquires a necessary filter processing function from the information management apparatus 30 in S706 of FIG. Then, the information management device 30 is inquired whether or not the restart of its own filter processing device 10 is permitted. Then, when the restart processing unit 36 receives a restart permission response from the information management device 30, the restart processing unit 36 restarts its own filter processing device 10. Next, when the restart processing unit 36 completes the restart, the restart processing unit 36 transmits a restart completion notification to the information management apparatus 30. On the other hand, if the restart processing unit 36 fails to receive a response indicating that the restart is permitted from the information management device 30, the restart processing unit 36 does not restart its own filter processing device 10. For example, the restart processing unit 36 permits the restart after a predetermined time has elapsed. Inquire whether or not. By doing in this way, even if the filter processing apparatus 10 of its own cannot be restarted due to restart of the other filter processing apparatus 10, it can be restarted with a time shift. Accordingly, it is possible to prevent the filter processing capacity of the entire system from being significantly reduced.

<< Second Embodiment >>
Next, a second embodiment will be described. In the second embodiment, each data storage device that stores stream data to be refiltered reads out the stream data to be refiltered from its stored data storage unit and flows it into the appending unit (stream). (Addition of data). That is, when stream data to be refiltered is stored across a plurality of data storage devices, each data storage device executes (transmission) pouring of the stream data into the appending unit. And FIG. 9 is a block diagram illustrating a stream data processing system according to the second embodiment. The outline of the second embodiment will be described with reference to FIG. Constituent elements similar to those in the first embodiment described above are denoted by the same reference numerals, and description thereof is omitted.

  As shown in FIG. 9, the information management apparatus 30 in the stream data processing system of the second embodiment includes a data refilter control unit 33A. When the data refilter control unit 33A receives a refilter instruction including the SID and time interval of the stream data to be refiltered from the reference client 70 and the content of the refilter process, the data relocation control unit 34 To identify the machine where the data exists. That is, the data location management unit 34 refers to the additional data location information in the data location management DB 325, searches the data storage device 20 that stores the stream data to be subjected to the refiltering process, and the presence of this stream data. The machine (data storage device 20) to be executed is specified. Then, the data refilter control unit 33 determines the data storage device 20 in which the stream data exists as a data storage device 20 (machine) that additionally writes the stream data, and sends it to the process management unit 26 of the data storage device 20. On the other hand, it instructs to read stream data.

  Thereafter, each of the data storage devices 20 that has received such an instruction reads the stream data to be refiltered from the stored data storage unit 24 of the own data storage device 20 by the stored data acquisition unit 25. Then, the accumulated data acquisition unit 25 inquires about the additional recording destination into which the stream data is to be sent to the connection destination allocation unit 31 of the information management device 30 and flows the stream data into the additional recording unit 222 instructed from the connection destination allocation unit 31. Next, each filter processing apparatus 10 acquires the stream data to be refiltered from the queue unit 226 of each data storage apparatus 20 and executes the refiltering process. Then, the data after the refilter process is transmitted to the database device 60. That is, each of the data storage devices 20 reads out the stream data to be refiltered in parallel and flows it into the appending unit 222.

<Operation procedure>
The operation procedure of the stream data processing system according to the second embodiment will be described with reference to FIG. FIG. 10 is a diagram showing an operation procedure of the stream data processing system of FIG.

  Since the process of S91 of FIG. 10 is the same as the process of S61 of FIG. 7, description is abbreviate | omitted and it demonstrates from S92.

  The data refilter control unit 33A of the information management device 30 determines whether or not the content of the refiltering process indicated in the refiltering instruction transmitted in S91 includes a window process (S92). The determination as to whether or not the content of the refiltering process includes window processing may be made from the content of the refiltering process itself included in the refiltering instruction or may be registered in the information source information 322. The determination may be made with reference to the information source information 322 regarding the SID of the stream data to be refiltered. When determining with reference to the information source information 322, the determination may be made based on whether or not the window size defined in the information source information 322 of the SID (see reference numeral 201 in FIG. 2) is “2” or more.

  For example, in the information source information of the SID “DML001” illustrated by reference numeral 201 in FIG. 2, the window size is “1”. Therefore, the data refilter control unit 33A determines that the content of the refilter process does not include the window process.

  When it is determined in S92 in FIG. 10 that the content of the refiltering process includes the window process (Yes in S92), the data refilter control unit 33A performs the same process as in S62 and subsequent steps in FIG. In other words, any one of the data storage devices 20 selected by the data refilter control unit 33A reads out the stream data to be refiltered from each data storage device 20 and flows it into the appending units 221 and 222.

  On the other hand, when it is determined in S92 that the content of the refiltering process does not include the window process (No in S92), the information management apparatus 30 allocates the refiltering SID (S62 and S63 in FIG. 7 described above). S93) and re-filter processing registration (S94). Then, similarly to S64, the additional data location information is searched from the data location management DB 325 using the SID indicated in the refilter instruction and the time interval (or the start time) as keys (S95).

  The data refilter control unit 33A refers to the identification information of the data storage device 20 indicated by the additional data position in the additional data position information (list of additional data position information) searched in S95. The storage device 20 is divided (S96).

  For example, the data refilter control unit 33A includes an instruction of SID “SID-kana99” and time interval “2007/02 / 20T 10: 0 to 2007/02/02 / 20T 12:00” in the refilter instruction. In this case, additional data location information 402, 403, and 405 satisfying this condition are retrieved from the data location management DB 325 illustrated in FIG. Here, when the description part of the additional data position information 402, 403, 405 of the additional data position information (the first part of the additional data position information) is seen, it is “macA.ntt”. Some of them are “macB.ntt”. That is, the stream data to be refiltered is stored in different data storage devices 20. Therefore, the data re-filter control unit 33A uses the additional data position information 402, 403, and 405 as the additional data position information 402 and 403 that starts with “macA.ntt” and the additional data position information 405 that starts with “macB.ntt”. And divide into That is, the additional data position information 402 and 403 transmitted to the data storage device 20 of “macA.ntt” and the additional data position information 405 transmitted to the data storage device 20 of “macB.ntt” are divided.

  Then, the data refilter control unit 33A transmits a refilter instruction including the additional data position information divided in S96 to each data storage device 20 that stores the stream data to be refiltered (S97). For example, the data refilter control unit 33A transmits a refilter instruction including the additional data position information 402 and 403 to the data storage device 20 of “macA.ntt”, and sends the data storage device 20 of “macB.ntt” to the data storage device 20 of “macB.ntt”. A re-filter instruction including additional data position information 405 is transmitted. Then, the stream data to be refiltered in each data storage device 20 is read out, poured into the appending units 221 and 222, and the filter processing procedure by the filter processing device 10 is the same as in the first embodiment, so the explanation will be given. Omitted.

  In this way, when stream data to be refiltered is stored across a plurality of data storage devices 20, the refiltering target is stored in the respective data storage devices 20 from the stored data storage unit 24. Since the stream data is read into the queue unit 226, the stream data to be refiltered can be efficiently read out.

<< Third Embodiment >>
Next, a third embodiment of the present invention will be described. In the stream data processing system according to the third embodiment, even when the filter processing apparatus 10 changes the additional recording unit (the additional recording units 221, 222, etc.) from which data is acquired, stream data (data) can be obtained without omission or duplication. This filtering process is performed. Constituent elements similar to those of the above-described embodiments are given the same reference numerals, and description thereof is omitted.

  FIG. 12 is a block diagram illustrating a stream data processing system according to the third embodiment. As illustrated in FIG. 12, the data storage device 20 of the stream data processing system according to the third embodiment includes additional writing unit information 81. The additional recording unit information 81 is information indicating the SID of data being processed in the additional recording units 221, 222, the queue type of the additional recording units 221, 222, the change time of the queue type, and the like.

  This queue type will be described later with reference to FIG. That is, (1) a pattern in which data received by the appending units 221 and 222 is output to the accumulated data storage unit 24 and data acquisition from the filter processing apparatus 10 is accepted (this pattern is assumed to be a queue type “ B ”), (2) a pattern in which the appending units 221 and 222 output data to the accumulated data storage unit 24 but does not accept data acquisition from the filter processing apparatus 10 (assuming this pattern is a queue type“ A ”) ).

  The accumulated data acquisition unit 25 refers to the additional recording unit information 81, determines the queue type of each additional recording unit 221, 222, and controls the flow of data into each additional recording unit 221, 222. By doing in this way, in the filter processing apparatus 10, the data to be filtered are prevented from being leaked or duplicated. For example, even when the data acquisition destination is changed from the additional recording unit 222 to the additional recording unit 221, the filter processing apparatus 10 can perform a filtering process on a series of stream data without omission or duplication. The additional recording section information 81 is stored in a storage means (not shown) such as a RAM in each data storage device 20. It should be noted that the queue type and the filter processing content related to the stream data of the SID are recorded in the processing procedure information 324 in FIG. In the processing procedure information 324, for example, when the information management apparatus 30 receives a queue type change notification from an external device (such as the reference client 70), the contents indicated in the queue type change notification are recorded. The queue type change notification includes the SID of the stream data whose queue type is changed.

  FIG. 13 is a diagram for conceptually explaining the operation outline of the stream data processing system according to the third embodiment. An outline of the operation of the stream data processing system according to the third embodiment will be described with reference to FIG. Here, the filter processing apparatus 10 acquires the data accumulated in the accumulated data storage unit 24 by the additional recording unit 221 via the additional recording unit 222 and performs the filtering process. Thereafter, the filter processing apparatus 10 will be described by taking, as an example, a case in which data is directly acquired from the appending unit 221 to perform filter processing, and thereafter data acquisition from the appending unit 222 is stopped. Here, for the sake of simplicity of explanation, the additional recording units 221, 222, the accumulated data storage unit 24, the accumulated data acquisition unit 25, the process management unit 26, the additional recording unit information 81, and the filter processing device of the configuration shown in FIG. Configurations other than 10 are omitted.

  (1) First, when the additional recording unit 221 receives stream data (data), the additional recording unit 221 outputs (accumulates) this data to the accumulated data storage unit 24. At this stage, in the processing procedure information 324 of FIG. 12, the queue type of the additional recording unit that handles the stream data of the SID “1” is “A”, and the filtering process by the filter processing apparatus 10 is “Null”. Is recorded. The filter processing apparatus 10 does not acquire data from the additional recording unit 221 based on such processing procedure information 324. Further, the additional recording unit 221 writes the SID (for example, “1”) of the data being processed and the queue type (for example, “A”) in the additional recording unit information 81.

  (2) Next, when the filter processing apparatus 10 starts the filtering process on the data accumulated in the accumulated data storage unit 24 by the additional recording unit 221, the following is performed. First, in the processing procedure information 324 of FIG. 12, for example, the execution of the filtering process “P” is registered for the SID “2”. Then, the accumulated data acquisition unit 25 reads the data accumulated by the appending unit 221 (data of SID “1”) from the accumulated data storage unit 24 and flows it into the appending unit 222. Then, the additional recording unit 222 changes the SID of the data (the data of the SID “1”) that is inserted by the accumulated data acquisition unit 25 to “2”. Then, the filter processing apparatus 10 acquires data of the SID “2” via the additional recording unit 222 based on the processing procedure information 324, and starts filter processing (for example, “P”).

  (3) The process management unit 26 instructs the appending unit 221 to change the queue type. This queue type change instruction is made when the information management device 30 in FIG. 12 receives a queue type change notification from the reference client 70 or the like, and this information management device 30 changes the queue type to each data storage device 20. This is done by sending a notification. At this time, the information management apparatus 30 is changed to the processing procedure information 324 to execute the filtering process “P” with the queue type “B” for the data of the SID “1”, for example. Register. Upon receiving such a queue type change instruction, the appending unit 221 accumulates the received data in the accumulated data storage unit 24 and accumulates the received data in the queue management unit 225 of FIG. Responds to data acquisition requests. The additional recording unit 221 then changes the queue type (for example, “B”), the SID of the stream data being processed (for example, “1”), and the change time of the queue type (for example, “2008/01 / 11T11: 00”). Is written in the additional recording section information 81. The queue type change time is, for example, the time when the appending unit 221 receives a queue type change instruction.

  (4) Next, when the accumulated data acquisition unit 25 refers to the additional recording unit information 81 and detects that the queue type of the SID being processed in the additional recording unit 221 is changed, the following processing is performed. . First, the accumulated data acquisition unit 25 refers to the additional recording unit information 81 and reads the time when the queue type is changed. Then, when data is read from the accumulated data storage unit 25 and flowed into the appending unit 222, the data up to that time (change time) is flowed while referring to the reception time attached to the data. Finish pouring into. That is, after changing the queue type, the filter processing apparatus 10 performs data acquisition from the additional recording units (the additional recording units 221 and 222) in parallel for a predetermined time, but the accumulated data acquisition unit 25 transfers to the additional recording unit 222. Since the flow of the data is finished by the change time of the queue type, as a result, duplication of data processed by the filter processing apparatus 10 is eliminated.

  Thus, the filter processing apparatus 10 acquires data directly from the appending unit 221 while acquiring the data (for example, data of SID “1”) while the appending unit 221 is only accumulating in the accumulated data storage unit 24. You can get it. That is, the filter processing apparatus 10 can filter all data after the specified time (queue type change time) without duplication for data having a specific SID.

<Operation procedure>
The operation procedure of the stream data processing system of the third embodiment will be described with reference to FIG. FIG. 14 is a diagram showing an operation procedure of the stream data processing system of FIG. Also in this case, the filter processing apparatus 10 acquires the data once accumulated in the accumulated data storage unit 24 by the additional recording unit 221 via the additional recording unit 222 and performs the filtering process. Thereafter, the filter processing apparatus 10 will be described by taking as an example a case where data is directly acquired from the appending unit 221 and data acquisition from the appending unit 222 is stopped.

  Although not shown here, the accumulated data acquisition unit 25 stores the data accumulated in the accumulated data storage unit 24 by the appending unit 221 according to the procedure shown in S61 to S72 of FIG. The data is read as refiltering data and added to the appending unit 222.

  That is, when the reference client 70 transmits a refilter instruction including the SID of the stream data to be refiltered, the start time, and the content of the refilter process to the data refilter control unit 33 of the information management device 30. (S61 in FIG. 7), using this as a trigger, the data refilter control unit 33 issues and assigns a refiltering SID, registers refiltering processing, and searches additional data position information (S62 to S64). Then, the data refilter control unit 33 determines a machine (data storage device 20) to be a refilter addition client (S65), and instructs the data storage device 20 to perform refiltering (S66). Then, the process management unit 26 that has received such a refiltering instruction activates its own data storage device 20 as a refilter additional recording client (S67), and sends the refiltering SID and additional data position information to the stored data acquisition unit 25. Including re-filtering instruction (S68). Then, the accumulated data acquisition unit 25 transmits an additional write destination payout request including the SID for refiltering to the connection destination assigning unit 31 of the information management device 30 (S69), and as a response, the payout of the additional write unit (the additional write unit 222) (S70). Thereafter, the accumulated data acquisition unit 25 connects to the appending unit 222 that has received this payout (S71), reads out from the accumulated data storage unit 24 (S72), and adds a refiltering SID (eg, “2”) to the data. At the same time, preparation is made for additional writing to the additional writing section 222 (S73).

  First, the additional recording unit 221 in FIG. 12 accumulates data in the accumulated data storage unit 24 (S100), and the accumulated data acquisition unit 25 reads data from the accumulated data storage unit 24 (S101). At this stage, the filter processing apparatus 10 does not acquire data from the additional recording unit 221.

  When the process management unit 26 of the data storage device 20 of FIG. 12 receives the queue type change notification from the information management device 30, the process management unit 26 transmits information instructing the change of the queue type to the appending unit 221 (S102). Then, the additional recording unit 221 that has received the information for instructing the change of the queue type writes information on the change of the queue type in the additional recording unit information 81 (S103). For example, the additional recording unit 221 writes the queue type after the queue type, the SID, and the reception time of the information instructing the change of the queue type (queue type change time) in the additional recording unit information 81.

  The accumulated data acquisition unit 25 refers to the additional recording unit information 81 (S104), and determines whether there is a change in the queue type (S105). If there is no change in queue type (No in S105), S106 and 107 are skipped and the process proceeds to S108. S108 will be described later. On the other hand, when the queue type has been changed with reference to the additional recording unit information 81 (Yes in S105), the change time of the queue type is read, and the reading end time of the data of the SID from the accumulated data storage unit 24 Is set (S106). This read end time is recorded in the storage unit in the data storage device 20.

  The accumulated data acquisition unit 25 refers to the time attached to the data for which the SID has been changed among the data read from the accumulated data storage unit 24, and the end time set in S106. It is determined whether the data is earlier (S107). Here, the accumulated data acquisition unit 25 adds the data that is determined to be data before the end time set in S106 (Yes in S107) to the additional recording unit 222 (S108), and the additional recording unit 222 performs queue management. Perform (S109). The accumulated data acquisition unit 25 adds the data to the appending unit 222 and then returns to S101. And the filter processing apparatus 10 acquires data from this additional recording part 222 (S110), and performs a filter process (S111). On the other hand, when it is determined that the data read from the stored data storage unit 24 is data after the end time set in S106 (No in S107), the appending to the appending unit 222 is terminated (S112). As a result, the filter processing apparatus 10 does not acquire the stream data received by the appending unit 221 from the appending unit 222. That is, the filter processing apparatus 10 can also acquire data directly from the additional recording unit 221 while acquiring the data accumulated by the additional recording unit 221 before changing the queue type via the additional recording unit 222. In this way, the filter processing apparatus 10 can perform the filtering process without omission on the series of stream data received by the additional recording unit 221. In addition, since the same data acquired from the additional recording unit 221 is not acquired from the additional recording unit 222, the filtering process is not performed repeatedly.

  The present invention is not limited to the above-described embodiments, and can be widely modified.

  For example, after receiving the refilter instruction from the reference client 70 in the information management apparatus 30 (after S61 in FIG. 7), the additional data position information (additional data position list) that is the target of this refilter instruction is sent to the reference client 70. You may make it transmit. FIG. 11 is a diagram illustrating an operation procedure of the information management apparatus and the reference client in FIGS. 1 and 9.

  That is, as shown in FIG. 11, the data location management unit 34 of the information management device 30 receives a search request for additional data location information including the SID and the time interval from the reference client 70 (S151). Then, the data location management unit 34 searches the additional data location information from the data location management DB 325 using the SID and time interval included in the search request as keys, and a result list (additional data location list) summarizing the search results. ) Is created (S152). Then, the data position management unit 34 returns (sends) the created additional write data position list as a search result for the search request (S153).

  Then, the reference client 70 that has received such an additional write data position list sends a refilter instruction including the contents of the refiltering process and the additional write data position information indicated in the additional write data position list to the data refilter of the information management device 30. The data is transmitted to the control unit 33A (S154).

  Thereafter, the data refilter control unit 33A assigns the SID for the additional recording filter and registers the refiltering process, and then follows the additional data position information included in the refiltering instruction transmitted from the reference client 70 in S54. Then, the data storage device 20 is instructed to refilter.

  In S54, the reference client 70 selects predetermined information from the additional data position information indicated in the additional data position list received from the information management apparatus 30, and refilters the selected additional data position information. It may be included and transmitted. By doing in this way, the reference client 70 has the number of data, which data storage device 20 stores the stream data searched by specifying the SID and the time interval, and how much data there is. It is possible to instruct refiltering after confirming the above.

  When the reference client 70 once transmits the refilter instruction after receiving the additional data position list in this way, the reference client 70 transmits identification information indicating this list instead of the additional data position list itself. It may be.

  In each of the above-described embodiments, the timing at which the data storage device 20 transmits the additional data position information of the own data storage device 20 to the information management device 30 is determined from the connection destination assignment unit 31 of the data storage device 20. It may be when the connection destination is paid out, when the additional recording client 50 is connected to the data storage device 20, or when the data storage device 20 receives stream data from the additional recording client 50.

  Further, the window processing in the filter processing apparatus 10 of each of the above-described embodiments includes a past predetermined time interval in addition to the window processing (Touple-based Window) using data for the past n items based on the current data. Various window processes such as a window process using time data (Time-based Window) may be used.

  Further, the stream data transmitted by the additional recording client 50 may be CLOB (Character Large Object) type data or BLOB (Binary Large OBject) type data other than XML data.

  Furthermore, in the first embodiment described above, the information management device 30 selects one data storage device 20 as a refilter appending client, but may select a plurality of data storage devices 20. Good.

  Further, in the second embodiment described above, when the content of the refiltering process does not include the window process, the information management apparatus 30 adds each of the data storage apparatuses 20 that store the stream data to be refiltered to the additional client. (That is, parallel processing in each data storage device 20 is instructed), but is not limited to this. For example, when the refilter instruction received from the reference client 70 includes the flag for requesting parallel processing, the information management device 30 assigns each data storage device 20 as an additional recording client and requests this parallel processing. When the flag to be included is not included, any one of the data storage devices 20 may be assigned as an additional recording client as in the first embodiment.

  Moreover, the system of each embodiment described above can be applied to various fields.

  For example, this system may be used to monitor the internal temperature of each refrigerator at a maintenance center of a refrigerator manufacturer. In this case, the write-once client 50 in FIG. 1 transmits stream data indicating the refrigerator internal temperature every predetermined time, and the filter processing device 10 uses the internal temperature (specific column (for example, , Recorded in C1) exceeds a predetermined threshold value, it is detected as abnormal. Then, the reference client 70 such as the maintenance center receives the detection result via the database device 60.

  In such a case, when the reference client 70 receives a predetermined refrigerator abnormality detection, the maintenance center of the refrigerator manufacturer may want to investigate the process until such abnormality occurs.

  In such a case, the following may be performed. That is, the SID indicating the type of the refrigerator in which the abnormality is detected and the threshold value for detecting the abnormality of the internal temperature in the filter processing unit 124 of the filter processing device 10 are changed. Then, the information management device 30 repeats the process of flowing the past stream data of the data storage device 20 again to the filter processing device 10 after changing the threshold value several times.

  Using a specific example, if you want to find out how often the internal temperature is 3 ° C or higher, go back the past 3 months, the reference client 70, the SID of the refrigerator where the abnormality was detected, the past 3 months When the internal temperature is 3 ° C. or higher, a re-filter instruction including the content of the re-filter process of abnormality detection is transmitted to the information management device 30. Similarly, if you want to find out how often the internal temperature goes back over the past year, the SID of the refrigerator where the abnormality was detected from the reference client 70, the time interval of the past year When the internal temperature is 0 ° C. or higher, a re-filter instruction including the content of the re-filter process of abnormality detection is transmitted to the information management device 30.

  By repeating this process several times, the temperature inside each refrigerator gradually increased to the temperature at which the abnormality was detected, or suddenly increased, and when Such analysis can be performed.

  In addition, this system may be used for monitoring the temperature in various parts of the country. In this case, for example, each of the plurality of additional recording clients 50 measures the temperature every minute. Next, stream data in which the SID assigned to each place is added to the measured temperature data is transmitted to the data storage device 20. The filter processing device 10 is set to issue an alarm when temperature data of 40 ° C. or higher is detected.

  Here, it is assumed that the temperature of the filter processing apparatus 10 is monitored for five years, and an alarm of 40 ° C. or higher is issued at a point A one day. In such a case, the reference client 70 issues a refilter instruction including the SID of the point A, the time interval of the past three years, and the content of the refilter process of issuing an alarm when the temperature is 38 ° C. or higher. Send to. By doing in this way, about the point A, the data whose temperature is 38 degreeC or more can be investigated among the data of the temperature for the past three years.

  That is, according to this system, since processing is performed using a plurality of data storage devices 20 and filter processing devices 10, a large number of stream data such as the temperature inside the vast number of refrigerators and the temperature monitoring at various points throughout the country are provided. The filtering process can be executed efficiently. Further, even when past stream data is stored in a plurality of data storage devices 20 in this way, the information management device 30 grasps in which data storage device 20 the past stream data is stored based on the additional data position information. Therefore, stream data to be refiltered can be searched efficiently.

  This system is also effective when a new filter is added to the information management apparatus 30 and the past stream data is re-flowed again with this new filter in the filter processing apparatus 10.

  For example, when using the person detection tool for detecting a person from stream data of a street corner monitoring image in the filter processing apparatus 10 of this system and measuring the flow rate of the person at that point, the person detection tool improves the person detection accuracy (For example, when the parent-child relationship can be discriminated, or when the movement direction of the person can be detected, etc.) First, the person detection tool in the filter processing apparatus 10 is a person with improved person detection accuracy. Replace with the detection tool. Then, based on the refiltering instruction from the additional recording client 50, the system performs refiltering on the past stream data with the new person detection tool. In this way, when a new filter is created, past stream data can be rewashed using this new filter.

  The filter processing apparatus 10 to which the new filter processing has been added in this way temporarily stops for restarting. As described above, the information processing apparatus 30 is restarted by the restart processing unit 36 as described above. Limit the number of ten. Therefore, it is possible to prevent a large number of filter processing apparatuses 10 from being stopped due to the addition of new filter processing, and the filter processing capability of the entire system to be significantly reduced.

  Furthermore, the present system may be applied to monitoring of a home gateway log, monitoring of a device state of a mobile phone, and the like other than the above-described example.

  The information management device 30, the data storage device 20, and the filter processing device 10 according to the present embodiment can be realized by a program that executes the processing as described above, and the program can be read by a computer-readable storage medium (CD -ROM etc.) and can be provided. It is also possible to provide the program through a network.

It is a block diagram which shows the stream data processing system of 1st Embodiment. It is the figure which illustrated the information source information and schema information of FIG. It is a figure which shows the operation | movement outline | summary of the stream data processing system of FIG. It is a figure which shows the operation | movement outline | summary of the stream data processing system of FIG. It is the figure which illustrated the data file accumulate | stored in the accumulation | storage data storage part of FIG. It is a figure which shows the operation | movement procedure of the stream data processing system of FIG. It is a figure which shows the operation | movement procedure of the stream data processing system of FIG. It is a figure explaining the process of S75 and S76 of FIG. 7 in detail. It is a block diagram which shows the stream data processing system of 2nd Embodiment. It is a figure which shows the operation | movement procedure of the stream data processing system of FIG. It is a figure which shows the operation | movement procedure of the information management apparatus of FIG. 1 and FIG. 9, and a reference client. It is a block diagram which shows the stream data processing system of 3rd Embodiment. It is a figure which illustrates notionally the operation | movement outline | summary of the stream data processing system of 3rd Embodiment. It is a figure which shows the operation | movement procedure of the stream data processing system of FIG.

Explanation of symbols

10 (10A to 10N) Filter processing device 11, 21 Input unit 12, 32 Processing unit 13, 23 Output unit 20 (20A to 20N) Data storage device 24 Stored data storage unit 25 Stored data acquisition unit 26 Process management unit 30 Information management Device 31 Connection destination allocation unit 33, 33A Data refilter control unit 34 Data location management unit 35 Information management unit 36 Restart processing unit 50 (50A to 50N) Additional recording client (information source terminal)
60 Database device 61 Data position storage DB
62 Reception unit 63 Search unit 70 Reference client (external device)
81 Additional Recording Section Information 121 Queue Selection Section 122 Data Acquisition Section 123 Filter Execution Control Section 124 Filter Processing Section 125 Restart Inquiry Section 126 Processing Procedure Information 127 Filter Acquisition Section 124 Filter Processing Sections 221 and 222 Additional Recording Section 223 Stream Acceptance Section 224 Additional Recording Data Storage unit 225 Queue management unit 226 Queue unit 227 Additional write data position information transmission unit 321 System information 322 Information source information 323 Schema information 324 Processing procedure information 325 Data position management DB
326 Restart information

Claims (7)

(1) A plurality of data storage devices that store stream data transmitted from a plurality of information source terminals, and (2) a plurality of filter processing devices that perform a filtering process on the stream data acquired from the data storage device. (3) For each SID indicating the type of the stream data, processing procedure information indicating the filter processing to be executed on the stream data having this SID and the application procedure of the filter processing is transmitted to the filter processing device. A stream data processing method in a stream data processing system comprising an information management device,
Each of the data storage devices includes a storage data storage unit that stores the stream data, one or more appending units that output the stream data to the storage data storage unit and store the stream data in its own queue unit, and the storage A storage data acquisition unit that acquires the stream data from a data storage unit and transmits the acquired stream data to the appending unit;
The information management device includes:
When a queue type change notification is received from an external device, including a change of the appending unit that is the acquisition destination of the stream data by the filter processing device and an SID of the stream data in which the appending unit that is the acquisition destination is changed,
Of the processing procedure information, reflect the change of the appending unit in the information related to the SID of the stream data,
Sending the queue type change notification to the data storage device;
The data storage device including the additional recording unit as the new acquisition destination,
Send the queue type change notification to the appending unit that is the new acquisition destination,
The appending unit that has received the queue type change notification,
Record the SID of the stream data being processed in the appending unit and the reception time of the queue type change notification in the appending unit information of the storage unit of the data storage device,
The accumulated data acquisition unit
When acquiring the stream data from the accumulated data storage unit, referring to the additional recording unit information of the storage unit, if it is determined that the queue type of the additional recording unit as a transmission destination of the acquired stream data has been changed,
With reference to the reception time of the queue type change notification indicated in the additional recording section information and the reception time of the acquired stream data, the reception time of the stream data is earlier than the reception time of the change notification of the queue type. The stream data is transmitted to the appending unit before the change, and the stream data whose reception time is after the reception time of the change notification of the queue type is not transmitted to the appending unit before the change Data processing method. The information management device includes:
When receiving a destination allocation request for the stream data including the SID from the information source terminal, for each SID stored in the storage unit, identification of the data storage device that is the destination of the stream data of the SID With reference to the system information indicating the information, the identification information of the data storage device that is the transmission destination of this stream data is transmitted to the information source terminal,
The information source terminal is
Transmitting the stream data to the data storage device corresponding to the identification information transmitted from the information management device;
Each of the stored data acquisition units of the data storage device, when acquiring the stream data, outputs the acquired stream data to the appending unit and outputs to the stored data storage unit,
The filter processing device includes:
Based on the queue state in the queue unit of the appending unit of the data storage device, select the data storage device as the acquisition destination of the stream data,
Obtaining the stream data from the appending unit of the selected data storage device;
Referring to the processing procedure information received from the information management device and the SID attached to the acquired stream data, a filtering process to be executed on the acquired stream data is determined, and the determined filtering process Run
Each of the data storage devices is
When stream data is received from the information source terminal and the received stream data is output to the stored data storage unit, the SID of the stream data, the reception time of the stream data, the identification information of the own data storage device, and Create additional write data position information in association with the information indicating the storage area of the stream data, and transmit to the information management device,
The information management device includes:
Storing the additional data location information transmitted from each of the data storage devices in a storage unit;
When receiving the stream data search request including the SID and reception time of the stream data from the external device,
Using the SID and the reception time included in the search request as keys, the additional data location information regarding the stream data is searched from the storage unit,
The stream data processing method according to claim 1, wherein the searched postscript data position information is transmitted to the external device. The information management device includes:
When receiving a destination allocation request for the stream data including the SID from the information source terminal, for each SID stored in the storage unit, identification of the data storage device that is the destination of the stream data of the SID With reference to the system information indicating the information, the identification information of the data storage device that is the transmission destination of this stream data is transmitted to the information source terminal,
When receiving from the external device refilter instruction information indicating additional data position information relating to the stream data to be refiltered and the content of the refilter processing,
Issue a refilter SID to be used for the stream data to be refiltered,
Adding information that associates the re-filtering SID with identification information of a data storage device that stores stream data having the SID, to the system information of the storage unit;
By adding information relating to the issued refilter SID and the content of the refilter process for the stream data having the SID to the processing procedure information of the storage unit, the content of the refilter process is added. Reflected in the processing procedure information,
Processing procedure information reflecting the contents of the refiltering process is transmitted to the filter processing apparatus;
Select one of the plurality of data storage devices,
Transmitting the refiltering instruction information including the issued refiltering SID to the selected data storage device;
The data storage device that has received the refilter processing instruction information,
From the storage area of the data storage device indicated in the received refilter instruction information, obtain the stream data that is the target of the refiltering process,
The refiltering SID is given to the stream data that is the target of the refiltering acquired,
Using the refilter SID included in the refilter instruction information as a key, from the information management device, obtain identification information of the data storage device to which the data storage device is to be connected,
Connect to the data storage device indicated by the acquired identification information, and transmit the stream data to the data storage device,
The filter processing device includes:
When the stream data having the SID for refiltering is acquired from the appending unit of the data storage device,
3. The stream data processing method according to claim 2, wherein the filtering process is executed with reference to processing procedure information reflecting the contents of the re-filtering process received from the information management apparatus. The information management device includes:
When receiving the refilter instruction information indicating the SID of the stream data to be refiltered, the reception time and the content of the refilter process to be performed on the stream data from the external device,
Using the SID and reception time of the stream data to be refiltered as a key, the additional data location information regarding the stream data is retrieved from the storage unit,
4. The re-filter instruction information indicating the searched additional write data position information and the content of the re-filter process is transmitted to any one of the plurality of data storage devices. 2. The stream data processing method described in 1. The information management device includes:
Determining whether the content of the refiltering process indicated in the received refiltering instruction information includes a window process for the stream data;
When it is determined that the content of the refilter process indicated in the received refilter instruction information does not include the window process for the stream data,
When selecting the data storage device that is the transmission destination of the refiltering instruction information, the additional data position information regarding the stream data retrieved from the storage unit is divided for each data storage device having the stream data, 5. The stream data processing method according to claim 3, wherein the additional write data position information is included in the refilter instruction information and transmitted to each data storage device having the stream data. (1) an accumulated data storage unit that accumulates stream data transmitted from a plurality of information source terminals, one or more appending units that perform output of the stream data to the accumulated data storage unit and accumulation in the own queue unit, And a plurality of data storage devices including a storage data acquisition unit that acquires the stream data from the storage data storage unit and transmits the acquired stream data to the appending unit, and (2) acquired from the data storage device A plurality of filter processing devices that perform filter processing on the stream data; and (3) for each SID indicating the type of the stream data, the filter processing to be executed on the stream data having this SID and the filter processing thereof An information management device that transmits processing procedure information indicating an application procedure to the filter processing device. In the information management device is a computer used for the stream data processing system,
Receiving, from an external device, a queue type change notification including the change of the additional recording unit that is the acquisition destination of the stream data by the filter processing device and the SID of the stream data in which the additional recording unit that is the acquisition destination is changed; ,
Of the processing procedure information, reflecting the change content of the additional recording unit in information related to the SID of the stream data;
Sending the queue type change notification to the data storage device;
And execute
In the data storage device, which is a computer provided with an additional recording section to be the new acquisition destination,
Sending the queue type change notification to the appending unit as the new acquisition destination;
In the appending unit that has received the queue type change notification,
Recording the SID of the stream data being processed in the additional recording unit and the reception time of the queue type change notification in the additional recording unit information of the storage unit of the data storage device;
When the stream data is acquired from the accumulated data storage unit, it is determined that the queue type of the additional recording unit that is the transmission destination of the acquired stream data has been changed by referring to the additional recording unit information of the storage unit If
With reference to the reception time of the queue type change notification indicated in the additional recording section information and the reception time of the acquired stream data, the reception time of the stream data is earlier than the reception time of the change notification of the queue type. The stream data is transmitted to the appending unit before the change, and the stream data whose reception time is after the reception time of the change notification of the queue type is not transmitted to the appending unit before the change Data processing program. (1) an accumulated data storage unit that accumulates stream data transmitted from a plurality of information source terminals, one or more appending units that perform output of the stream data to the accumulated data storage unit and accumulation in the own queue unit, And a plurality of data storage devices including a storage data acquisition unit that acquires the stream data from the storage data storage unit and transmits the acquired stream data to the appending unit, and (2) acquired from the data storage device A plurality of filter processing devices that perform filter processing on the stream data; and (3) for each SID indicating the type of the stream data, the filter processing to be executed on the stream data having this SID and the filter processing thereof An information management device that transmits processing procedure information indicating an application procedure to the filter processing device. A stream data processing system,
The information management device includes:
When a queue type change notification is received from an external device, including a change of the appending unit that is the acquisition destination of the stream data by the filter processing device and an SID of the stream data in which the appending unit that is the acquisition destination is changed,
Of the processing procedure information, the change type of the additional recording unit is reflected in the information regarding the SID of the stream data, and the queue type change is made to the data storage device including the additional recording unit as a new acquisition destination of the stream data. A data refilter control unit for sending notifications,
The data storage device includes:
When the queue type change notification is received, the process management unit transmits the queue type change notification to the additional recording unit as the new acquisition destination,
The appending part is
When the queue type change notification is received, the SID of the stream data being processed by the additional recording unit and the reception time of the queue type change notification are recorded in the additional recording unit information of the storage unit of the data storage device,
The accumulated data acquisition unit
When acquiring the stream data from the accumulated data storage unit, referring to the additional recording unit information of the storage unit, if it is determined that the queue type of the additional recording unit as a transmission destination of the acquired stream data has been changed,
With reference to the reception time of the queue type change notification indicated in the additional recording section information and the reception time of the acquired stream data, the reception time of the stream data is earlier than the reception time of the change notification of the queue type. The stream data is transmitted to the appending unit before the change, and the stream data whose reception time is after the reception time of the change notification of the queue type is not transmitted to the appending unit before the change Data processing system.

Images