JP2009087190A - Stream data analysis speed-up device, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To speed up an analysis of a high speed data stream by using a plurality of applications a high speed operation by eliminating redundant calculation as much as possible when performing the analysis. <P>SOLUTION: This stream data analysis speed-up device applied to an information processor for successively analyzing data to be outputted by a data output means which outputs data to be analyzed from moment to moment is provided with: a data storage means for time-sequentially storing data output by a data output means under the consideration of the generation time of data; a data analyzing means for analyzing the data stored by the data storage means in parallel as a plurality of applications, wherein each analytic instance independently operates; and an event data analyzing library means for providing a low rank analyzing means common to the plurality of data analyzing means, and for quickening the parallel execution of data analysis by saving redundant calculation. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a stream data analysis speed-up device, a stream data analysis speed-up method, and a stream data analysis speed-up program that sequentially analyze a large amount of information that is constantly and momentarily generated, and particularly, a large amount of sensor data. The present invention relates to a stream data analysis acceleration device, a stream data analysis acceleration method, and a stream data analysis acceleration program.

  A database management system (DBMS) is widely used as means for simultaneously analyzing a large amount of data by a plurality of applications. In particular, since the performance of an application that uses a database is also determined according to the processing performance of the DBMS, it is extremely important to improve the processing performance of the DBMS.

  Since a database handles a large amount of data, in many DBMS execution environments, a large-capacity storage device is connected to a computer on which the DBMS is executed, and the database data is stored on the storage device. For this reason, when processing related to the database is performed, access to the storage device occurs, and the data access performance in the storage device greatly affects the performance of the DBMS. Therefore, it is important to optimize access in a storage device in a system in which a DBMS operates.

  Patent Document 1 discloses a technique for improving DBMS performance by reducing useless access and prefetching necessary data. Japanese Patent Application Laid-Open No. 2004-151867 discusses prefetch execution, determination of the amount thereof, memory buffer in consideration of the execution plan of the inquiry, data access characteristics, memory buffer amount, I / O load, etc. It describes that the performance of the DBMS is improved by improving the I / O access performance by performing management or the like.

  Patent Document 2 discloses a technique for speeding up a database by copying some data stored in the database onto a RAM memory having a high access speed. In Patent Document 2, the number of accesses to each database file is constantly counted by the CPU 1 dedicated to database management, and a frequently used table is extracted in a timely manner from a plurality of tables in the database placed on the disk. It is described that data is automatically copied onto a RAM memory having a high access speed. When accessing the database from the general arithmetic processing CPU 2, the database on the RAM is preferentially accessed. As a result, the average access time of the database can be shortened and speeded up.

  In Patent Document 3, in a storage device in which a logical storage device used for access by a computer is placed on a physical storage device that actually stores data, the logical storage device is disposed on the physical storage device. A technique for improving the access performance of a storage device by dynamically changing the storage device is disclosed. By moving a part of data stored in a physical storage device with high access frequency to another physical storage device using the dynamic allocation change function, the access frequency of a specific physical storage device is prevented from increasing. This improves the performance of the storage device as a whole. Patent Document 3 also discloses an automatic execution method for high-performance processing using an arrangement dynamic change function.

US Pat. No. 5,317,727 JP-A-11-31093 Japanese Patent No. 3641872

  In the systems described in Patent Documents 1 and 2, it is the locality of data access that has the effect of speeding up. Only a small amount of data (hereinafter referred to as working data) is accessed many times in the current processing for a large amount of data in the database. Therefore, in the methods described in Patent Documents 1 and 2, a faster storage device such as a memory is used for working data processing, and a slow storage device such as a disk is used for data that is not accessed frequently. Therefore, the whole process is speeded up.

  Here, the working data of the data stream processing assumed in the present invention will be examined. First, with reference to FIG. 28, data stream processing targeted by the present invention will be described. FIG. 28 is an explanatory diagram for explaining an example of the data stream processing. As shown in FIG. 28, data always flows (this data is called event data). A general stream data analysis apparatus sequentially inputs event data to an analysis process (analysis function F in the example shown in FIG. 28) to obtain an analysis result. In many general data stream processes, a general stream data analysis apparatus does not input each event data as an input to the analysis process, but inputs it to the analysis process in a certain block. An analysis window is used to obtain the input chunk from the flowing event data. Here, the analysis window defines a group of event data to be passed to the input of analysis processing, and is defined by a period, the number of events, and the like.

  FIG. 28 shows a case where an analysis window is created in a fixed time unit. This fixed time is called a span. In the example shown in FIG. 28, the data of the analysis window-t1 is first input to the analysis process F and the analysis result is output. Next, the data of the analysis window-t2 is input to the analysis process F, and the analysis result is output. Thereafter, the analysis window is shifted in time by the same method, and the application of the analysis process F is repeated.

  As described above, data stream processing is a calculation method in which analysis processing is repeatedly applied to flowing data. From the viewpoint of each data, there is only read access from the analysis process once, not many times. Therefore, it is considered that there is no working data where access is concentrated for a certain period. Alternatively, it is considered that only a very short time is valid as working data. In the data stream processing, since there is no working data, the methods described in Patent Documents 1 and 2 cannot be expected to increase the speed.

  The technique described in Patent Document 3 is a technique for increasing the speed by distributing centralized access of hardware. Conversely, it can be said that there is no effect if there is no part where access is concentrated.

  As already described, the data stream processing assumed in the present invention is processing by a very distributed processing method in which working data always changes, and does not concentrate on accessing a certain storage device. Furthermore, the source of event data and the like is a sensor such as RFID (Radio Frequency Identification), which is physically dispersed from the beginning. As described above, in the data stream processing, the effect of speeding up by the technique described in Patent Document 3 cannot be expected.

  Therefore, the present invention provides a stream data analysis that can eliminate a redundant calculation as much as possible when analyzing a large amount of a high-speed data stream that is generated constantly and constantly by a plurality of applications. An object is to provide a speed-up device, a stream data analysis speed-up method, and a stream data analysis speed-up program.

  A stream data analysis acceleration device according to the present invention is a stream data analysis acceleration device applied to an information processing device that sequentially analyzes data output by a data output means that outputs data to be analyzed every moment. In order to analyze the data output by the data output means in consideration of the generation time of the data, the data storage means stored in time series and the data stored in the data storage means are analyzed in parallel as a plurality of applications. A library that provides data analysis means in which each instance of analysis operates independently and a lower-level analysis means common to a plurality of data analysis means, and is redundant with respect to parallel execution of a plurality of data analysis And an event data analysis library means for speeding up the calculation.

  The stream data analysis speed-up method according to the present invention is a stream data analysis speed-up method for sequentially analyzing data output every moment, and is a processing step for managing generated event data. Data flow processing for temporarily storing in the storage means and deleting from the storage means when no longer needed, event data analysis processing for analyzing event data stored in the data flow processing in parallel by a plurality of analysis methods, and events Data analysis processing is divided into multiple subtasks, and for subtasks commonly used for multiple event analysis processing, event data common analysis processing that speeds up analysis by reusing past calculation results and event data common In order for the analysis process to analyze at high speed, the past calculation results are stored and checked for reuse. That includes the data reuse processing, data flow processing, event data analyzing process, characterized in that to operate the event data common analysis and data reuse processing in parallel.

  A stream data analysis acceleration program according to the present invention is a stream data analysis acceleration program for sequentially analyzing data that is output every moment, in order to manage and analyze event data generated by a computer. Data flow processing that temporarily stores data in the storage means and deletes it from the storage means when it is no longer needed, and event data analysis processing that analyzes event data stored in the data flow processing in parallel using multiple analysis methods Event data analysis processing is divided into multiple subtasks, and event data common analysis processing that speeds up analysis by reusing past calculation results for subtasks commonly used for multiple event analysis processing, In order for the event data common analysis processing to analyze at high speed, the past calculation results are stored and reused. Characterized in that to execute the data reuse process for checking the possibility of.

  Advantageous Effects of Invention According to the present invention, when analyzing a large amount of a high-speed data stream that is generated constantly and momentarily with a plurality of applications, it is possible to increase the speed by eliminating redundant calculations as much as possible. is there.

Embodiment 1.
A first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration example of the stream data analysis speed-up device according to the first embodiment.

  Stream data analysis acceleration devices are applied to information processing devices that sequentially analyze large amounts of information (stream data) that is generated constantly and momentarily. Especially, multiple applications can analyze a large amount of sensor data simultaneously. Applied to an information processing apparatus.

  The stream data analysis acceleration device shown in FIG. 1 includes a sensor 1, an analysis queue 2, data analysis means 3, and event data analysis library means 4. Since the stream data analysis speed-up device has these functions, the data analysis means 3 can be expected to speed up.

  The sensor 1 generates data to be analyzed every moment. That is, the sensor 1 converts a predetermined detection value into a predetermined signal and outputs it as data to the analysis queue 2. Hereinafter, the output of data by the sensor 1 may be described as the generation of data by the sensor 1.

  The analysis queue 2 stores data from the sensor 1 in time series in order to analyze in consideration of the data generation time. The analysis queue 2 is realized by a storage unit such as a memory, for example. Hereinafter, the data stored in the analysis queue 2 may be referred to as “analysis queue 2 data”.

  The data analysis means 3 is an application that analyzes data in the analysis queue 2. The stream data analysis speed-up device has a plurality of data analysis means 3, for example. In each data analysis means 3, the analysis instance operates independently.

  The event data analysis library means 4 is a library that provides sub (lower) analysis means common to the plurality of data analysis means 3. The event data analysis library means 4 is a library for speeding up the parallel execution of a plurality of data analyzes by eliminating redundant calculations as much as possible.

  The event data analysis library means 4 includes sub-analysis interface means 401, sub-analysis method determination means 402, sub-analysis means 403, and sub-analysis result shared memory 404.

  The sub analysis interface unit 401 provides an interface when the data analysis unit 3 uses the event data analysis library unit 4.

  When the sub-analysis interface unit 401 is called, the sub-analysis method determination unit 402 determines whether to perform calculation using data stored in the analysis queue 2 or to divert past calculation results.

  The sub-analysis unit 403 analyzes the sub-analysis provided by the sub-analysis interface unit 401 using the data in the analysis queue 2.

  The sub-analysis result sharing memory 404 temporarily stores the analysis result of the sub-analysis unit 403 for reuse while considering the time axis.

  The sensor 1 is means for outputting data to be analyzed, and any means may be used as long as it is means for outputting data momentarily. The sensor 1 is, for example, a means for outputting temperature data at intervals of one minute, a means for reading data on an IC chip on which product information is written, such as an RFID (Radio Frequency Identification) system, and generating data. There may be.

  In the following description, data output from the sensor 1 is referred to as event data. The format of the event data is not particularly limited, but has fields of “time: date and time when the event occurred” and “content: information about the event that has occurred”.

  The analysis queue 2 stores event data from a plurality of sensors 1 as one series. Since the event data is generated every moment, it is necessary to delete from the memory (the analysis queue 2) in order from the old event data. For this realization, a method called a ring buffer may be used.

  FIG. 2 is an explanatory diagram for explaining an example of the ring buffer system. As shown in FIG. 2, the ring buffer method treats the memory as if it were arranged on a circle, and a Wtite Pointer that indicates the position of the memory that stores data next, and a Read Pointer that indicates the position of the memory that reads data next. It is a method with. In the ring buffer method, data is stored and read in a certain direction (clockwise in the figure), so that the data moves in the same direction every time both the Write Pointer and Read Pointer are accessed. In the example shown in FIG. 2, the memory is indexed from 1 to 8. FIG. 2A shows a state where index 1 is stored and index 3 is stored, and Wtite Pointer points to the fourth. FIG. 2B shows a state where all the memories up to the eighth have been used. Since Write Pointer indicates the first, the first data is overwritten by the next write. By using the ring buffer method, event data that occurs every moment can be managed while deleting from the memory in order from the old event data. It should be noted that a sufficient memory size is required so that the Write Pointer can pass the ReadPointer, that is, data that has not yet been read is not overwritten.

  It is assumed that the data analysis means 3 has a plurality of instances, and an ID is assigned to each instance. Using this ID, it is possible to communicate by specifying an instance of the data analysis means 3 from another means (used by the analysis data shift means 406 in the second embodiment).

  The data analysis means 3 stores an analysis window that defines data to be analyzed for each instance. The analysis window is information including start information, span information, and slide information, and is information indicating an analysis unit of data to be analyzed.

  The start information is a start pointer of the analysis window and corresponds to a time-series offset value of event data. The span information is information indicating the size of the analysis window. Therefore, the event data in the analysis window is event data from start information event data to start information + span information. The slide information is information indicating a width to move the analysis window when a certain analysis is completed and the next analysis is performed.

  In addition, there are a method of specifying these values by a time axis and a method of specifying by the number of data. For example, in the case of start information, date and time are specified when specifying on the time axis, and when specifying by the number of data, a serial number is specified for event data by this number. In the first embodiment, a method of designating on the time axis will be described. However, processing can be performed in the same manner even if designated by the number of data. The entire stream data analysis speed-up device has a clock and can measure time and duration.

  The operation of each instance of the data analysis unit 3 will be described. Each instance repeats the following process.

  In the first step, the instance acquires event data from the analysis queue 2. At this time, acquiring a plurality of data at a time is a general analysis method, and an analysis window is used. The analysis window represents a range of data input from the data in the analysis queue 2 to the data analysis means 3. The instance acquires event data by sliding the analysis window in order from the oldest data. In the example shown in FIG. 28, the range indicated by the analysis window -t1 is the event data first acquired in the first step. Thereafter, each time the first step is repeatedly executed, event data is acquired by sliding the analysis window-t2 and the analysis window-t3.

  In the second step, the instance analyzes the acquired event data. The instance receives the set of event data obtained in the first step, analyzes it, and outputs the result. Usually, a plurality of instances of the data analysis means 3 analyze the data in the analysis queue 2 in parallel.

  FIG. 3 is an explanatory diagram for explaining an example of processing (parallel analysis) in which instances analyze data in parallel. FIG. 3 illustrates a case where each instance performs an analysis process using the analysis function F, an analysis process using the analysis function G, and an analysis process using the analysis function H. As shown in FIG. 3, each instance basically operates independently, and the definition of the analysis window is not necessarily the same. In the example shown in FIG. 3, the span is small in the order of analysis G, analysis F, and analysis H. In order to analyze the instances of the plurality of data analysis means 3 independently and in parallel, it is necessary to independently manage to what extent each instance of the data analysis means 3 has analyzed. This can be realized by making the Read Pointer of the ring buffer of the analysis window to be analyzed next correspond to the start information of the analysis window for each instance of the data analysis means 3. FIG. 4 is an explanatory diagram for explaining an example of a ring buffer method in parallel analysis.

  In the first embodiment, a one-dimensional least square method will be described to show a specific example of analysis. FIG. 5 is an explanatory diagram for explaining an example of the least square method. The one-dimensional least-squares method is a method of approximating a set of event data for a certain period using a specific linear function, and the residual function is used so that the linear approximation function is a good approximation to the value of each event data. This is a method for determining a coefficient that minimizes the sum of squares of differences. In the example shown in FIG. 5, the horizontal axis represents time, and the vertical axis represents event data values. In the example shown in FIG. 5, a total of n event data from t1 to tn is data to be analyzed. At this time, the one-dimensional least square method is approximated by the linear equation shown in FIG. This approximate straight line can be used to predict the value of event data at time tx in the future, for example.

The method for obtaining the approximate straight line is widely disclosed. Hereinafter, a method for obtaining the approximate straight line will be described.
First, E is obtained from temporary variables A to E.
A = SUM [(Y_ti) * (Y_ti)]
B = SUM [(X_ti) * (X_ti)]
C = SUM [Y_ti]
D = SUM [Y_ti * X_ti]
E = SUM [X_ti]

  Y_ti means the value of event data at time ti, X_ti means the value of time ti, and SUM means the sum obtained by changing i from 1 to n.

When the approximate function is y = aX + b, the values of the coefficients a and b can be calculated by the following equations.
a = (nD-CE) / (nB-E * E)
b = (BC-DE) / (nB-E * E)

  In order to simplify the description, the one-dimensional least square method will be described below, but the analysis method is not limited to this.

  In the third step, the instance changes the analysis window. When the analysis of the second step is finished, the instance changes the analysis window according to the slide parameters. The method of sliding depends on the analysis method and is not particularly limited.

  FIG. 6 is an explanatory diagram for explaining an example of sliding analysis windows. As shown in FIG. 6 (a), the span and slide may be set to the same value, and all the data may be processed only once. As shown in FIG. 6 (b), the analysis windows may overlap. As shown in FIG. 6C, there may be data that is not processed. After changing the analysis window, the process returns to the first step.

  The event data analysis library unit 4 is a unit that enters between the data analysis unit 3 and the analysis queue 2 to speed up the analysis of the data analysis unit 3. Hereinafter, it is assumed that common processing is included when each data analysis means 3 is divided into sub-tasks. For example, the data analysis means 3 often executes statistical processing and the like, and analysis processing such as average, variance, and standard deviation can be considered as a subtask. Further, in the least square method, processing such as calculation of the sum of values such as calculation of temporary variables C and E, and calculation of the sum of root mean squares such as calculation of temporary variables A and B are subtasks.

  It should be noted here that the effect of the present invention cannot be obtained simply by adding the event data analysis library means 4 to the configuration of a general system. FIG. 7 is an explanatory diagram for comparing a configuration example of the stream data analysis apparatus. FIG. 7A shows a general system configuration that does not use the event data analysis library means 4. FIG. 7B shows a system configuration of the first embodiment using the event data analysis library means 4.

  As an implementation method of the data analysis means 3, the event data analysis library means 4 is implemented at the source code level so as to be implemented while calling as a subtask. In the example shown in FIG. 7, the analysis process F () executed by the data analysis unit 3 can execute Sub-l () and Sub-m () that can be executed by using the event data analysis library unit 4 as a subtask. And have been implemented. In a general scheme, these subtasks are all implemented in F (). The present invention speeds up the data analysis means 3 created on the assumption that the event data analysis library means 4 is used.

  Hereinafter, the configuration in the event data analysis library means 4 will be described.

  The sub-analysis interface unit 401 is an API (Application Interface) of the event data analysis library unit 4 and shows a list of subtasks implemented by the event data analysis library unit 4.

  In the example shown in FIG. 7, four subtasks of Sub-l (), Sub-m (), Sub-n (), and Sub-o () are disclosed. The realization method is the same as the API of a normal library. For example, the data analysis means 3 makes a call with the public subtask name, and the linker can convert the subtask name into a function pointer in the event data analysis library means 4 at the time of compilation, thereby executing the subtask. .

  The parameters of the subtask interface are data to be analyzed. An analysis window is designated for the data to be analyzed. That is, the sub-analysis interface unit 401 inputs data to be analyzed shown in the analysis window as a parameter.

  FIG. 8 is an explanatory diagram illustrating an example of registration in the sub-analysis result shared memory 404. The sub-analysis result sharing memory 404 stores information including a sub-task name 404-1, a type 404-2, start information 404-3, a span 404-4, and a calculation result 404-5.

  The subtask name 404-1 means the name of the subtask handled by the sub-analysis interface unit 401. The type 404-2 means the type of the analysis window, and takes any value of “period” and “number”. The start information 404-3 means the top information of the analysis window. In the case of a period, it is a time, and in the case of a number, it is a serial number of event data. The span 404-4 means information on the span of the analysis window. In the case of a period, it is time, and in the case of a number, it is the number of event data. The calculation result 404-5 means a calculation result to be reused.

  In the example shown in FIG. 8, the first record has a subtask named X total, an analysis window in the period, and the result of calculating event data from time 10:00:00 to 10:01:00, 123.

  The sub-analysis method determining unit 402 searches and inserts data stored in the sub-analysis result shared memory 404. The operation of the sub-analysis method determining unit 402 will be described later.

  The sub-analysis unit 403 is a unit that executes a subtask published by the sub-analysis interface unit 401. The sub-analysis unit 403 receives the sub-task name and the analysis window parameter to be analyzed from the sub-analysis interface unit 401 and analyzes the data in the analysis queue 2.

  The sub-analysis method determination unit 402 functions as a main control unit for operations in the event data analysis library unit 4. FIG. 9 is a flowchart showing an example of the operation of the sub-analysis method determining unit 402.

  In step S402-1, the data analysis unit 3 calls a subtask through the sub analysis interface unit 401. The data analysis unit 3 passes the subtask name and the analysis window information, which is a parameter for specifying the analysis target data, to the subanalysis interface unit 401 when the subtask is called.

  In step S402-2, the sub-analysis method determining unit 402 searches the sub-analysis result shared memory 404 to search for a reusable calculation result. For example, the sub-analysis method determining unit 402 searches the following conditions for each field using AND conditions. For example, the sub-analysis method determining unit 402 “subtask name 404-1 is the subtask name called in step S402-1”, “analysis window whose type 404-2 is called in step S402-1” “The value of the type of information”, “The start information 404-3 is the value of the start information of the analysis window information called in step S402-1,” and “Span 404-4 are the values of step S402-1. The data satisfying all the conditions of “the value of the span information of the analysis window information called in” is extracted from the sub-analysis result shared memory 404. Note that only when the registration method in the sub-analysis result shared memory 404 is not found under the above conditions, a record is registered in step S402-5, so this search result is at most one.

  In step S402-3, the sub-analysis method determination unit 402 determines whether or not the data can be reused based on the search result of the sub-analysis result shared memory 404. If the search result is found, it can be reused (YES), and the process proceeds to step S402-6. If no search result is found, it cannot be reused (NO), and the process proceeds to step S402-4 to analyze the data in the analysis queue.

  If the data cannot be reused, in step S402-4, the sub-analysis unit 403 acquires data in the analysis queue 2 based on the condition of the analysis window, and executes analysis corresponding to the subtask name. The sub-analysis method determination unit 402 performs analysis and outputs it as a calculation result.

  In step S402-5, the sub-analysis unit 403 registers in the sub-analysis result shared memory 404 via the sub-analysis method determination unit 402 in order to reuse the calculation result of step S402-4.

  A method for registering the calculation result calculated in step S402-4 will be described. The sub-analysis method determination unit 402 newly adds a record in the sub-analysis result shared memory 404 and registers the following values. That is, the sub-analysis method determining means 402 registers the sub-task name called in step S402-1 as the sub-task name 404-1, and the type of analysis window information called in step S402-1 as the type 404-2. Is registered, the start information 404-3 is registered as the start information value of the analysis window information called in step S402-1, and the analysis window information is called as span 404-4 in step S402-1. The span information value is registered, and the calculation result calculated in step S402-4 is registered as the calculation result 404-5.

  In step S402-6, the sub-analysis method determination unit 402 returns the subtask calculation result to the data analysis unit 3 that is the caller. The sub-analysis method determining means 402 returns the calculation result in the record searched in step S402-2 or the calculation result calculated in step S402-4 as the calculation result. That is, if the data can be reused, the sub-analysis method determination unit 402 returns the calculation result retrieved in step S402-2 to the data analysis unit 3, and if the data cannot be reused. The calculation result calculated in step S402-4 is returned to the data analysis means 3.

  The stream data analysis speed-up device can be realized by a computer, and each component constituting the stream data analysis speed-up device, that is, the data analysis means 3 and the event data analysis library means 4 are computer processing devices ( CPU) can be realized as a program for realizing the functions described above. The fact that each component constituting the stream data analysis acceleration device can be realized by a computer and can be realized as a program is not limited to the first embodiment, but is the same in other embodiments. .

  Next, the operation of the first embodiment will be described with reference to FIG. FIG. 10 is a flowchart showing an example of the operation of the stream data analysis speed-up device in the first embodiment. In the first embodiment, the data flow process F1, the event data analysis process F2, the event data common analysis process F3, and the data reuse process F4 operate in parallel. For example, the control unit (not shown) of the stream data analysis acceleration device operates in parallel the data flow process F1, the event data analysis process F2, the event data common analysis process F3, and the data reuse process F4. Let

  The data flow process F1 is a process step for managing generated event data, and is a process for temporarily storing the event data in a memory for analysis and deleting it from the memory when it is no longer necessary.

  The event data analysis process F2 is a process of analyzing the event data stored in the data flow process F1 in parallel by a plurality of analysis methods.

  The event data common analysis process F3 speeds up the analysis by dividing the event data analysis process F2 into a plurality of subtasks and reusing past calculation results for the subtasks commonly used in the plurality of event analysis processes. It is processing.

  The data reuse process F4 is a process for storing past calculation results and checking the possibility of reuse in order for the event data common analysis process F3 to analyze at high speed.

  The data flow process F1 may be realized using the sensor 1 and the analysis queue 2.

  The event data analysis process F2 is realized by the data analysis means 3. The data analysis means 3 consists of a plurality of instances, and usually a plurality of instances execute analysis in parallel. As shown in FIG. 10, the data analysis unit 3 executes, for example, analysis F and analysis G in parallel. Each analysis instance executes an individual analysis process F21 or a common analysis process F22 as two sub-analysis processes.

  The individual analysis process F21 is a process for performing an analysis in cooperation with the data flow process F1 directly without using the event data common analysis process F3. This is a general method and does not have the effect of speeding up.

  The common analysis process F22 is a process for performing high-speed analysis using the event data common analysis process F3.

  The distinction between the individual analysis process F21 and the common analysis process F22 to be executed is determined not statically but statically. That is, since the list of the processing contents of the event data common analysis process F3 is disclosed in advance, a part of the event data analysis process F2 is executed using the event data common analysis process F3. To implement.

  The event data common analysis process F3 may be implemented using the event data analysis library means 4. The event data common analysis process F3 does not operate when the system is started. At the timing when the event data analysis processing F2 calls the event data analysis library means 4, an instance for processing the call is generated and operated.

  The overall operation flow is a processing flow (see FIG. 9) of the sub-analysis method determination unit 402 which is the main control unit of the event data analysis library unit 4. Note that when this processing flow ends, this instance disappears. There are the following two methods for obtaining the calculation result in this processing flow.

  One is a method F31 of calculating from event data, and is a method of analyzing event data in cooperation with the data flow processing F1.

  The other is a method F32 for reusing past calculation results, which is a method for reusing past calculation results in cooperation with the data reuse processing F4.

  Whether the method F31 for calculating from event data or the method F32 for reusing past calculation results is executed is determined by conditional branching in step S402-3 in FIG.

  The data reuse process F4 may be realized by the sub-analysis result shared memory 404.

  As described above, according to the first embodiment, there is an effect that a partial analysis process common to a plurality of data analysis means can be processed at high speed. The reason is that it has event data analysis library means for reusing past calculation results for a plurality of data analysis executions in parallel. That is, in the parallel execution of a plurality of data analysis processes, the past calculation results are reused, so that the processing can be performed at high speed.

Embodiment 2. FIG.
Next, a second embodiment of the present invention will be described with reference to the drawings. FIG. 11 is a block diagram illustrating a configuration example of the stream data analysis speed increasing device according to the second embodiment.

  Referring to FIG. 11, the second embodiment of the present invention is the latest call history means 407, data analysis scheduling means 405, analysis data that are not provided in the event data analysis library means 4 of the first embodiment. Shift means 406. By configuring as shown in FIG. 11, the reuse rate of the past calculation results stored in the sub-analysis result shared memory 404 is increased, and higher speed can be expected.

  The latest call history unit 407 is a storage unit that always stores the latest one in the history of the data analysis unit 3 calling the sub-analysis interface unit 401.

  The data analysis scheduling means 405 synchronizes the analysis windows with respect to a situation where the plurality of data analysis means 3 are executed in parallel and independently of each other.

  The analysis data shift unit 406 receives the result of the data analysis scheduling unit 405 and shifts the analysis window held by each data analysis unit 3 on the time axis.

  The purpose of the data analysis scheduling means 405 will be described with reference to FIGS. FIG. 12 is an explanatory diagram for explaining an example in which the analysis windows are not synchronized. In the example shown in FIG. 12, the analysis window of analysis A is represented by a solid line, and the analysis window of analysis B is represented by a wavy line. Both types of analysis windows (period or number) and spans are the same, and start information is different. Since the start timing is different, reuse using the sub-analysis result shared memory 404 is not performed between these two analyses.

  On the other hand, as shown in FIG. 13, the data analysis scheduling means 405 shifts the analysis window of analysis A, so that analysis B and the calculation result can be reused. FIG. 13 is an explanatory diagram for explaining an example of the synchronization processing of the analysis window.

  When the analysis window is shifted, the analysis result changes because the data to be analyzed changes. However, in the data stream analysis, there is no problem because the data is regarded as a flow. That is, in the data stream analysis, the analysis start timing is not important. This is because the data stream analysis repeatedly analyzes data in a certain segment unit and analyzes the overall tendency. Hereinafter, the method of shifting the analysis window for the data analysis means 3 to synchronize will be described.

  First, the latest call history unit 407 will be described. The latest call history unit 407 stores history information indicating a history of the data analysis unit 3 calling the sub analysis interface unit 401. FIG. 14 is an explanatory diagram illustrating an example of the structure of a table in which history information is registered. More specifically, the table in which the history information is registered is stored in a storage device such as a hard disk device or a memory of the stream data analysis speedup device.

  The data analysis means ID 407-1 means the ID of the data analysis means 3 that called the sub analysis interface means 401 immediately before.

  The subtask name 407-2 means the name of the subtask that called the immediately preceding subanalysis interface unit 401.

  The type 407-3 means the type of analysis window that called the immediately preceding sub-analysis interface unit 401.

  The start information 407-4 means the start information of the analysis window that called the immediately preceding sub-analysis interface unit 401.

  A span 407-5 means a span of an analysis window that calls the immediately preceding sub-analysis interface unit 401.

  The latest call history unit 407 updates the table shown in FIG. 14 by performing the process shown in FIG. 15 every time the data analysis unit 3 calls the sub-analysis interface unit 401. FIG. 15 is a flowchart showing an example of the operation of the latest call history unit 407.

  In step S407-1, the latest call history unit 407 searches the table shown in FIG. 14 for an old history. The latest call history means 407 searches the values of the fields in the table shown in FIG. 14 under the following conditions. For example, the latest call history unit 407 indicates that “the data analysis unit ID 407-1 is the ID of the data analysis unit 3 that called the current sub-analysis interface unit 401”, “the subtask name 407-2 is the current sub "The name of the subtask that called the analysis interface means 401", "the type 407-3 is the type of the analysis window that called the current sub-analysis interface means 401", and "span 407-5 is the current subtask History information that satisfies the condition that it is the span of the analysis window that called the analysis interface means is extracted from the table.

  In step S407-2, the latest call history unit 407 determines whether history information has been extracted in the process of step S407-1. If a record is found in the process of step S407-1 (YES), the process proceeds to step S407-3. If no record is found in the process of step S407-1 (NO), the process proceeds to step S407-4.

  In step S407-3, the latest call history unit 407 deletes the record searched in step S407-1.

  In step S407-4, the latest call history unit 407 adds the call information of the current sub-analysis interface unit 401 to the table as new history information.

  FIG. 16 is a flowchart showing an example of the operation of the data analysis scheduling unit 405.

  In step S405-1, the data analysis scheduling unit 405 performs grouping by grouping records having the same value of the subtask name, type, and span from the latest call history unit 407.

  In the case of the example shown in FIG. 14, it is divided into two groups as shown in FIG. FIG. 17 is an explanatory diagram illustrating an example of grouped history information. Hereinafter, each of the grouped tables is referred to as a history information table, and the entire set of history information tables is referred to as a scheduling candidate. In the example illustrated in FIG. 17, the scheduling candidate includes two history information tables.

  In step S405-2, the data analysis scheduling unit 405 selects one history information table from the scheduling candidates.

  In step S405-3, the data analysis scheduling unit 405 determines whether the history information table can be selected. If it is selectable (YES), the process proceeds to step S405-4. If it cannot be selected (NO), it means that all the history information tables in the scheduling candidates have been processed, and the processing is terminated. In the following description, it is assumed that group 1 (see FIG. 17A) is selected from the history information table shown in FIG. 17 as the history information table.

  In step S405-4, the data analysis scheduling unit 405 performs further fine grouping. The data analysis scheduling unit 405 performs grouping into the same group if the field values of the start information are the same. FIG. 18 is an explanatory diagram for explaining an example of history information that is further finely grouped. In the example shown in FIG. 17, the data analysis scheduling unit 405 performs grouping into three groups as shown in FIG. Hereinafter, the three groups are subgroup 1-1 (see FIG. 18A), subgroup 1-2 (see FIG. 18B), and subgroup 1-3 (FIG. 18C), respectively. See).

  In step S405-5, the data analysis scheduling unit 405 counts the number of elements for each subgroup. In the example shown in FIG. 18, the number of elements in the subgroup 1-1 is 2 records, and the number of elements in the subgroup 1-2 and the subgroup 1-3 is 1 record.

  In step S405-6, the data analysis scheduling unit 405 determines a subgroup serving as a reference for synchronization. The data analysis scheduling means 405 is based on the subgroup having the largest number of elements in step S405-5. When there are a plurality of subgroups having the highest number, all are set as reference subgroups.

  In step S405-7, the data analysis scheduling unit 405 outputs the result to the analysis data shift unit 406. The reference subgroup table and the non-reference subgroup table are distinguished from each other and sent to the analysis data shift means 406.

  In step S405-8, the data analysis scheduling unit 405 deletes the group for which the scheduling process has been completed in step S405-7 from the scheduling candidates. For example, the data analysis scheduling unit 405 deletes the group 1.

  The analysis data shift means 406 receives the result from the data analysis scheduling means 405 and detects a deviation from the reference subgroup for a subgroup that is not the reference subgroup. Further, the analysis data shift means 406 requests each data analysis means 3 to shift the value of the start information of the subgroup based on the detected result.

  The shift width is calculated by comparing the value of the start information with the value of the reference subgroup. In the example shown in FIG. 18, the shift width of the subgroup 1-2 is +20 seconds (10: 00: 20-10: 00: 00), and the shift width of the subgroup 1-3 is −10 seconds (09: 59: 50-10: 00: 00).

  When the deviation width is a positive value, it indicates that the deviation is advanced from the reference, and when it is negative, it indicates that the deviation is delayed from the reference. In the example shown in the figure, the result is that the data analysis means G is 20 seconds earlier and the data analysis means H is 10 seconds late. Therefore, the data analysis means G is requested to delay 20 seconds, and the data analysis means H is requested to delay. Asks you to advance 10 seconds. Upon receiving the request, the instance of the data analysis means 3 changes the setting of the analysis window stored therein as requested.

  Next, the operation of the second embodiment will be described with reference to FIG. FIG. 19 is a flowchart showing an example of the operation of the stream data analysis speed-up device in the second embodiment.

  In the process of the second embodiment, a data analysis scheduling process F5 is added to the process of the first embodiment (see FIG. 10). The data analysis scheduling process F5 is a process for synchronizing analysis windows in a plurality of instances of the event data analysis process F2. By performing the processing as shown in FIG. 19, in the second embodiment, the reuse rate of past calculation results increases, and the processing speed can be further increased.

  The data analysis scheduling process F5 is a process including a call history management process F51, a data analysis scheduling process F52, and a shift request process F53.

  The call history management process F51 is a process of generating original data for extracting an instance of the event data analysis process F2 that is likely to be synchronized by temporarily recording the call of the event data common analysis process F3.

  The data analysis scheduling process F52 is a process of analyzing the call history data accumulated in the call history management process F51 and controlling the analysis start timing of the analysis window of the event data common analysis process F3.

  The shift request process F53 is a process for requesting to shift the analysis data of each event data analysis process F2 based on the result of the data analysis scheduling process F52.

  The data analysis scheduling process F5 may be realized by the latest call history means 407, the data analysis scheduling means 405, and the analysis data shift means 406.

  The call history management process F51 may be realized by the latest call history means 407 performing the process shown in FIG.

  The data analysis scheduling process F52 may be realized by the data analysis scheduling unit 405 performing the process shown in FIG.

  The shift request process F53 may be realized by the analysis data shift means 406.

  As described above, according to the second embodiment, there is an effect that high-speed processing can be performed even when a plurality of data analysis units execute analysis independently of each other. The reason is that data analysis scheduling means that monitors the analysis window, which is a data analysis unit of the data analysis means, and schedules the analysis windows to be overlapped by a plurality of data analysis means, and analysis that changes the analysis timing based on this result This is because it has data shift means.

Embodiment 3. FIG.
Next, a third embodiment of the present invention will be described with reference to the drawings. FIG. 20 is a block diagram illustrating a configuration example of the stream data analysis speed-up device according to the third embodiment.

  Referring to FIG. 20, the third embodiment of the present invention includes sub-analysis result shared memory management means 408 that is not provided in the event data analysis library means 4 of the first and second embodiments. . The sub-analysis result shared memory management unit 408 performs processing for deleting unnecessary entries from the analysis results stored in the sub-analysis result shared memory 404. With the configuration as shown in FIG. 20, the memory utilization efficiency can be increased.

  The sub-analysis result shared memory management unit 408 includes an analysis queue monitoring unit 40802 and a reuse data deletion unit 40801.

  The analysis queue monitoring unit 40802 performs processing for monitoring data deleted from the analysis queue 2.

  The reuse data deletion unit 40801 receives the result of the analysis queue monitoring unit 40802, searches the calculation result stored in the sub-analysis result shared memory 404, and performs a process of searching for and deleting an entry that is not reused. .

  The analysis queue monitoring unit 40802 monitors overwritten data when data is added to the analysis queue 2. The monitoring item is an attribute of time of overwritten data (hereinafter, processing completion time). In other words, the processing completion time means the time of old data that has already been analyzed by the data analysis means 3 and may be deleted from the sub-analysis result shared memory 404.

  The reuse data deletion unit 40801 deletes, for example, a record that matches the condition that “the value obtained by adding the span to the start information is older than the processing completion time” from the sub-analysis result shared memory 404.

  Next, the operation of the third embodiment will be described with reference to FIG. FIG. 21 is a flowchart illustrating an example of the operation of the stream data analysis speed-up device according to the third embodiment.

  In the process of the third embodiment, a reuse data deletion process F6 is added to the process of the first embodiment (see FIG. 10) and the process of the second embodiment (see FIG. 19). The reuse data deletion process F6 is a process of deleting an unnecessary entry in the analysis result stored in the sub-analysis result shared memory 404.

  The reuse data deletion process F6 is a process including an overwrite monitoring process F61 and a deletion process F62.

  The overwrite monitoring process F61 is a process of monitoring the data flow process F1 and detecting the time of data that has already been deleted from the analysis queue 2.

  The deletion process F62 is a process for receiving the result of the overwrite monitoring process F61 and deleting data that is not reused from the past calculation results stored in the data reuse process F4.

  The reuse data deletion process F6 may be realized by the sub-analysis result shared memory management unit 408.

  The overwrite monitoring process F61 may be realized by the analysis queue monitoring unit 40802.

  The deletion process F62 may be realized by the reuse data deletion unit 40801.

  According to the third embodiment, there is an effect that stream data can be permanently analyzed at high speed. The reason is that by having a sub-analysis result shared memory management means for deleting a calculation result that is not reused in the future among the calculation results of the sub-analysis result shared memory that stores the calculation result to be reused, This is because only a certain amount of memory is used even in the analysis.

Embodiment 4 FIG.
Next, a fourth embodiment of the present invention will be described with reference to the drawings. FIG. 22 is a block diagram illustrating a configuration example of the stream data analysis acceleration device of the fourth embodiment.

  Referring to FIG. 22, the fourth embodiment of the present invention includes a calculation cost estimation unit 40803 that is not provided in the sub-analysis result shared memory management unit 408 of the third embodiment. The calculation cost estimation unit 40803 stores an estimation of the calculation cost for the analysis executed by the sub analysis unit 403.

  In the fourth embodiment, the operation of the reuse data deletion unit 40801 is performed from the calculation results stored in the sub-analysis result shared memory 404 based on the calculation cost data calculated by the calculation cost estimation unit 40803. Modify to delete entries that are not reused.

  Furthermore, a “speedup degree” for storing an estimated value of the speedup effect when reused is added to the field of the sub-analysis result shared memory 404. FIG. 23 is an explanatory diagram illustrating a registration example of the sub-analysis result shared memory 404 according to the fourth embodiment. In addition to the subtask name 404-1, the type 404-2, the start information 404-3, the span 404-4, and the calculation result 404-5 (see FIG. 8), the sub-analysis result shared memory 404 includes the speed-up degree 404-. Information including 6 is stored. The speed-up degree 404-6 is a field for storing a difference in calculation cost between when the sub-analysis unit 403 analyzes and when the calculation result of the sub-analysis result shared memory 404 is reused.

  FIG. 24 is an explanatory diagram illustrating a registration example of a table in which the value calculated by the calculation cost estimation unit 40803 is set. The table shown in FIG. 24 is set by a user, for example. Specifically, the table in which the value calculated by the calculation cost estimation unit 40803 is set is stored in a storage device such as a hard disk device or a memory of the stream data analysis acceleration device.

  A subtask name 40803-1 is a name of an analysis executed by the sub-analysis unit 403. A span 40803-2 is a span of an analysis window for analysis executed by the sub-analysis unit 403. The calculation cost 40803-3 is an estimated value of the calculation cost when the subtask indicated by the subtask name 40803-1 is executed with the value set as the span 40803-2. The unit of calculation cost 40803-3 may be processing time.

  The reuse data deletion unit 40801 calculates the speedup degree using the data of the calculation cost estimation unit 40803 and registers it in the sub-analysis result shared memory 404. In other words, the reuse data deletion unit 40801 uses the subtask name 404-1 and span 404-4 stored in the sub-analysis result shared memory 404 as keys to set a table in which the value calculated by the calculation cost estimation unit 40803 is set ( (Refer to FIG. 24) and the value of the calculation cost 40803-3 is extracted and set as the acceleration degree 404-6 in the sub-analysis result shared memory 404.

  The reason why the calculation cost 40803-3 is set as the acceleration degree 404-6 will be described. The reuse process only searches the sub-analysis result shared memory 404, and the processing cost is constant regardless of the sub-task name and span. Therefore, it can be considered that the reuse process is fast enough to ignore the processing cost. Therefore, the speed-up effect can be approximated by the analysis cost of the sub-analysis unit 403.

  In the example shown in the third record of FIG. 23, the subtask name 404-1 is the X-square sum, and the span 404-4 is 1:00. When the table shown in FIG. 24 is searched with these values, 1000 is extracted as the calculation cost 40803-3. Therefore, the speed-up degree 404-6 is 1000.

  When the speed-up degrees 404-6 of all the sub-analysis result shared memories 404 are calculated, the reuse data deletion unit 40801 deletes the data in the order from the smaller value of the speed-up degree 404-6, that is, the record having the lower reuse effect. To do. In the example shown in FIG. 23, the first record and the second record are deleted preferentially over the third record and the fourth record.

  According to the fourth embodiment, even when there is little memory, there is an effect that processing can be performed at high speed. The reason is that there is a calculation cost estimation means for estimating the cost for each analysis and controlling so as to store the calculation result of reuse by giving priority.

Embodiment 5. FIG.
Next, a fifth embodiment of the present invention will be described with reference to the drawings. FIG. 25 is a block diagram illustrating a configuration example of a stream data analysis speed increasing device according to the fifth embodiment.

  Referring to FIG. 25, the fifth embodiment of the present invention includes a sub-analysis observation unit 409 that is not provided in the event data analysis library unit 4 of the fourth embodiment. The sub-analysis observation unit 409 observes the time taken by the sub-analysis unit 403 for each calculation, and records the observation result as the calculation cost 40803-3 of the calculation cost estimation unit 40803.

  The sub-analysis observation unit 409 receives notification from the sub-analysis unit 403 when the calculation is started and when the calculation is completed. For example, the sub-analysis observation unit 409 receives a notification including a sub-task name, a span, and information indicating “start” when the sub-analysis unit 403 starts calculation. For example, when the sub-analysis unit 403 finishes the calculation, the sub-analysis observation unit 409 receives a notification including a sub-task name, a span, and information indicating “end”.

  The sub-analysis observation unit 409 stores the time when the start and end are notified, and calculates the execution time for the set of sub-task name and span from this difference. The sub-analysis observation unit 409 registers the calculated result as the calculation cost 40803-3 of the table stored in the calculation cost estimation unit 40803. At this time, if the calculation cost 40803-3 has already been registered, an already registered value and an average value of newly registered values may be registered as the calculation cost. However, this is only an example, and another method such as overwriting may be used.

  According to the fifth embodiment, there is an effect that overhead for high-speed processing can be minimized. The reason is that there is a reuse shared memory management timing judging means for judging the timing of arranging the reuse memory of the calculation result while monitoring the state of resources such as the remaining amount of memory.

Embodiment 6. FIG.
Next, a sixth embodiment of the present invention will be described with reference to the drawings. The configuration of the stream data analysis speed-up device of the sixth embodiment is the same as that of the fifth embodiment shown in FIG.

  In the sixth embodiment, a “reuse count” that stores the number of times the calculation result is reused is added to the field of the sub-analysis result shared memory 404.

  Further, when the reuse data deletion unit 40801 deletes the data, the reuse data deletion unit 40801 is modified so that it is deleted from data with a low frequency of use.

  FIG. 26 is an explanatory diagram illustrating a registration example of the sub-analysis result shared memory 404 according to the sixth embodiment. The sub-analysis result shared memory 404 is in addition to the sub-task name 404-1, the type 404-2, the start information 404-3, the span 404-4, the calculation result 404-5, and the speed-up degree 404-6 (see FIG. 23). Thus, information including the number of reuses 404-7 is stored.

  For the reuse count 404-7, for example, a value of 0 is set when the sub-analysis method determination unit 402 registers in the sub-analysis result shared memory 404 (step S402-5). Thereafter, each time the sub-analysis method determining means 402 reuses the data in the sub-analysis result shared memory 404, the value is increased by one.

Embodiment 7. FIG.
Next, a seventh embodiment of the present invention will be described with reference to the drawings. FIG. 27 is a block diagram illustrating a configuration example of the stream data analysis speed increasing device according to the seventh embodiment.

  Referring to FIG. 27, the seventh embodiment of the present invention includes a reuse shared memory management timing determination unit 410 in addition to the third, fourth, fifth and sixth embodiments. The reuse shared memory management timing determination unit 410 determines the timing at which the sub-analysis result shared memory management unit 408 executes a process of deleting data with a low possibility of reuse from the sub-analysis result shared memory 404.

  The reuse shared memory management timing determination unit 410 monitors the remaining amount of memory and the like, for example, when the value becomes below a certain value, the sub analysis result shared memory management unit 408 has a low possibility of reuse. It may be requested to delete data. However, this is an example, and various methods such as predicting a future memory usage rate from a history of the usage rate of the used memory and notifying at a timing that does not exceed the resource limit may be considered.

  In addition, each embodiment of this invention can achieve the objective of this invention by providing the structure shown below.

  In an information processing device that sequentially analyzes a large amount of information that is generated constantly and constantly, especially for an information processing device that simultaneously analyzes a large amount of sensor data by multiple applications, the data to be analyzed is from moment to moment Sensors to be generated, analysis queues for storing data in time series in order to analyze the data from the sensors in consideration of the data generation time, and means for analyzing the analysis queue data in parallel by multiple applications It is a library that provides data analysis means that each analysis instance operates independently and sub-analysis means common to multiple data analysis means, and is as redundant as possible for parallel execution of multiple data analysis By having event data analysis library means that speeds up the process by eliminating unnecessary calculations, Speed up can be expected.

  The event data analysis library means uses the analysis queue when the sub-analysis interface means is called and the sub-analysis interface means for providing an interface for the data analysis means to use the event data analysis library means. The sub-analysis method determining means for determining whether the past calculation result is diverted, the sub-analysis means for analyzing the sub-analysis provided by the sub-analysis interface means using the data of the analysis queue, and the sub-analysis means By configuring the analysis result from the sub-analysis result shared memory that temporarily stores the analysis result for reuse while taking the time axis into consideration, it is possible to expect a high-speed data analysis means.

  Furthermore, the data analysis means always stores the latest call history of the sub-analysis interface means, and the situation where a plurality of data analysis means are executed in parallel and independently of each other. The data analysis scheduling means for synchronizing the analysis windows with each other, and the analysis data shift means for shifting the analysis windows held by each data analysis means on the time axis in response to the results of the data analysis scheduling means, are added to the configuration. Even when a plurality of data analysis units execute analysis independently of each other, high-speed processing can be performed.

  By adding sub-analysis result shared memory management means for deleting unnecessary entries in the analysis result stored in the sub-analysis result shared memory, stream data can be permanently analyzed at high speed.

  The sub-analysis result shared memory management means receives the results of the analysis queue monitoring means for monitoring the deleted data in the analysis queue data and the analysis queue monitoring means, and stores them in the sub-analysis result shared memory By comprising reuse data deletion means for searching for and deleting entries that are not reused from the calculation results, stream data can be analyzed permanently at high speed.

  In addition, the sub-analysis result shared memory management means stores an estimate of the calculation cost for the analysis executed by the sub-analysis means. In addition to the configuration of the calculation cost estimation means, the operation of the reuse data deletion means is estimated as the calculation cost. Based on the calculation cost data calculated by the means, it was modified to delete entries that are not reused from the calculation results stored in the sub analysis result shared memory, and further reused in the field of the sub analysis result shared memory By adding a speedup degree that stores an estimated value of the speedup effect in the case, processing can be performed at high speed even when there is little memory.

  By observing the time taken for each calculation of the sub-analysis means and adding the sub-analysis observation means for recording the observation result to the calculation cost estimation means, the processing can be performed at high speed even when the memory is small.

  Add a field to the sub-analysis result shared memory management means to store the number of times the calculation result has been reused, and modify the data so that it is deleted from less frequently used data when the reuse data deletion means deletes the data. Thus, even when there is little memory, it can be processed at high speed.

  The sub-analysis result shared memory management means adds to the configuration a reuse shared memory management timing determination means that measures the timing of executing the process of deleting data that is unlikely to be reused from the sub-analysis result shared memory. The overhead for high-speed processing can be minimized.

  In the embodiment described above, a certificate generation / distribution system having characteristic configurations as shown in the following (1) to (9) is shown.

  (1) A stream data analysis accelerating device applied to an information processing device that sequentially analyzes data output by a data output unit that outputs data to be analyzed every moment, output from the data output unit In order to analyze the data in consideration of the generation time of the data, data storage means for storing in time series (for example, realized by the analysis queue 2), and data stored in the data storage means as a plurality of applications Data analysis means that performs analysis in parallel and each analysis instance operates independently (for example, realized by the data analysis means 3) and lower-level analysis means common to a plurality of data analysis means are provided. This is an event data analysis library that speeds up the parallel execution of multiple data analyzes by omitting redundant calculations. (For example, implemented in the event the data analysis library unit 4) and stream data analysis speed device characterized by comprising a.

  (2) The event data analysis library means includes sub-analysis interface means (for example, realized by the sub-analysis interface means 401) that provides an interface when the data analysis means uses the event data analysis library means, and data analysis means Sub-analysis method decision means (for example, sub-analysis method) for determining whether to use the data stored in the data storage means or calculate using past calculation results when the sub-analysis interface means is called And sub-analysis means (for example, realized by the sub-analysis means 403) that uses the data stored in the data storage means to analyze the lower-order analysis provided by the sub-analysis interface means. The results of analysis by the sub-analysis means can be reused in consideration of the time axis. Sub analysis result shared storage means for temporarily storing the order (for example, implemented in the sub-analysis results shared memory 404) and a data stream analysis optimization device comprising. The data stream analysis speedup device configured as described above can process a partial analysis process common to a plurality of data analysis means at high speed.

  (3) The event data analysis library means is the latest call history means (for example, the latest call history means 407) that always stores the latest history information among the history information indicating the history of the data analysis means calling the sub-analysis interface means. Data analysis scheduling means for synchronizing the analysis windows indicating the analysis units of the data to be analyzed with respect to the situation in which a plurality of data analysis means are executed in parallel and independently from each other (for example, Data analysis scheduling means 405) and an analysis data shift means (for example, analysis data shift means 406) for shifting the analysis window held by each data analysis means on the time axis in response to the results of the data analysis scheduling means. Data stream analysis speed-up device including: The data stream analysis speed-up device configured as described above can perform high-speed processing even when a plurality of data analysis units execute analysis independently of each other.

  (4) The event data analysis library means is sub-analysis result shared storage management means (for example, sub-analysis result shared memory management) that deletes unnecessary entries from the analysis results stored in the sub-analysis result shared storage means. A data stream analysis speed-up device including: Since the data stream analysis speedup device configured as described above uses only a fixed amount of memory, it can analyze at high speed.

  (5) The sub-analysis result shared storage unit management unit includes a data storage unit monitoring unit for monitoring data deleted from the data storage unit (for example, realized by the analysis queue monitoring unit 40802), and a data storage unit monitoring unit. Based on the result monitored by the sub-analysis result shared storage means, a reuse data deletion means (for example, reuse data deletion means 40801) that searches for and deletes an entry that is not reused from the calculation results stored in the sub-analysis result shared storage means. A data stream analysis speed-up device. The data stream analysis speed-up device configured as described above can process at high speed even when the memory is small.

  (6) The sub-analysis result shared storage management unit includes a calculation cost estimation unit (for example, realized by a calculation cost estimation unit 40803) that stores an estimation of a calculation cost related to the analysis executed by the sub-analysis unit. The sub-analysis result shared storage unit stores speed-up degree information indicating an estimated value of the speed-up effect when reused, and the reuse data deletion unit sub-analyzes the calculation cost data calculated by the calculation cost estimation unit. Data stream analysis that stores information as speedup degree information in the result sharing storage means, and extracts and deletes entries that are not reused from the calculation results stored in the sub-analysis result shared storage means based on the speedup degree information High speed device. The data stream analysis speedup device thus configured can minimize the overhead for the speedup processing.

  (7) The event data analysis library means observes the time taken by the sub-analysis means, and records the observation result as the calculation cost estimate in the calculation cost estimation means (for example, the sub-analysis observation means 409). Data stream analysis speed-up device including

  (8) The sub-analysis result sharing storage unit stores information indicating the number of times the calculation result has been reused, and the reuse data deletion unit re-calculates the calculation result when deleting the data from the sub-analysis result sharing storage unit. A data stream analysis speed-up device that deletes data that is not frequently used based on information indicating the number of times it has been used.

  (9) The sub-analysis result shared storage unit management unit is a reuse shared storage unit management timing determination unit that measures the timing of executing the process of deleting data that is unlikely to be reused from the sub-analysis result shared storage unit ( (E.g., realized by the reuse shared memory management timing determination means 410).

  INDUSTRIAL APPLICABILITY The present invention can be effectively applied to an information processing apparatus that sequentially analyzes a large amount of information that is generated constantly and momentarily, and particularly, an information processing apparatus that simultaneously analyzes a large amount of sensor data by a plurality of applications. It can be applied effectively.

It is a block diagram which shows the structural example of the stream data analysis speed-up apparatus of 1st Embodiment. It is explanatory drawing for demonstrating an example of a ring buffer system. It is explanatory drawing for demonstrating an example of the process (parallel analysis) in which an instance analyzes data in parallel. It is explanatory drawing for demonstrating an example of the ring buffer system in parallel analysis. It is explanatory drawing for demonstrating an example of the least squares method. It is explanatory drawing for demonstrating the example of sliding of an analysis window. It is explanatory drawing for comparing the structural example of a stream data analysis apparatus. It is explanatory drawing which shows the example of registration of the sub-analysis result shared memory 404. 5 is a flowchart showing an example of the operation of a sub-analysis method determination unit 402. It is a flowchart which shows an example of operation | movement of the stream data analysis speed-up apparatus in 1st Embodiment. It is a block diagram which shows the structural example of the stream data analysis speed-up apparatus of 2nd Embodiment. It is explanatory drawing for demonstrating the example when the synchronization of an analysis window is not taken. It is explanatory drawing for demonstrating an example of the synchronous process of an analysis window. It is explanatory drawing which shows the example of the structure of the table in which historical information is registered. 7 is a flowchart showing an example of the operation of latest call history means 407. It is a flowchart which shows an example of operation | movement of the data analysis scheduling means 405. It is explanatory drawing for demonstrating the example of the log | history information divided into groups. It is explanatory drawing for demonstrating the example of the log | history information further divided into groups. It is a flowchart which shows an example of operation | movement of the stream data analysis speed-up apparatus in 2nd Embodiment. It is a block diagram which shows the structural example of the stream data analysis speed-up apparatus of 3rd Embodiment. It is a flowchart which shows an example of operation | movement of the stream data analysis speed-up apparatus in 3rd Embodiment. It is a block diagram which shows the structural example of the stream data analysis speed-up apparatus of 4th Embodiment. It is explanatory drawing which shows the example of registration of the sub-analysis result shared memory 404 in 4th Embodiment. It is explanatory drawing which shows the example of registration of the table in which the value calculated by the calculation cost estimation means 40803 is set. It is a block diagram which shows the structural example of the stream data analysis speed-up apparatus of 5th Embodiment. It is explanatory drawing which shows the example of registration of the sub-analysis result shared memory 404 in 6th Embodiment. It is a block diagram which shows the structural example of the stream data analysis speed-up apparatus of 7th Embodiment. It is explanatory drawing for demonstrating an example of a data stream process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Sensor 2 Analysis queue 3 Data analysis means 4 Event data analysis library means 401 Sub analysis interface means 402 Sub analysis method determination means 403 Sub analysis means 404 Sub analysis result shared memory

Claims (16)

A stream data analysis acceleration device applied to an information processing device that sequentially analyzes data output by a data output means that outputs data to be analyzed every moment,
Data storage means for storing the data output by the data output means in time series in order to analyze the data generation time in consideration;
A means for analyzing the data stored in the data storage means in parallel as a plurality of applications, each of which is an instance of an analysis,
A library providing low-level analysis means common to a plurality of the data analysis means, and comprising event data analysis library means for speeding up the parallel execution of a plurality of data analyzes by omitting redundant calculations. Stream data analysis speed-up device characterized by Event data analysis library means
Sub-analysis interface means for providing an interface when the data analysis means uses the event data analysis library means;
Sub-analysis method determining means for determining whether to calculate using the data stored in the data storage means or to use the past calculation result when the data analysis means calls the sub-analysis interface means When,
Sub-analysis means for analyzing using the data stored in the data storage means for the lower-level analysis provided by the sub-analysis interface means;
The data stream analysis speed-up device according to claim 1, further comprising: a sub-analysis result sharing storage unit that temporarily stores the result analyzed by the sub-analysis unit for reuse while considering a time axis. Event data analysis library means
Of the history information indicating the history of data analysis means calling the sub-analysis interface means, the latest call history means for always storing the latest history information,
A data analysis scheduling means for synchronizing an analysis window indicating an analysis unit of data to be analyzed with respect to a situation in which a plurality of data analysis means are executed in parallel and independently of each other;
The data stream analysis speed-up device according to claim 1, further comprising: an analysis data shift unit that receives the result of the data analysis scheduling unit and shifts an analysis window held by each data analysis unit on a time axis.   The event data analysis library means includes sub-analysis result shared storage management means for deleting an unnecessary entry from the analysis results stored in the sub-analysis result shared storage means. The data stream analysis speed-up device according to any one of the above items. Sub-analysis result shared storage management means
Data storage monitoring means for monitoring data deleted from the data storage means;
Reusable data deletion means for searching for and deleting entries that are not reused from among the calculation results stored in the sub-analysis result shared storage means based on the results monitored by the data storage monitoring means. 4. The data stream analysis acceleration device according to 4. The sub-analysis result shared storage management unit has a calculation cost estimation unit that stores an estimation of the calculation cost for the analysis performed by the sub-analysis unit,
The sub-analysis result shared storage means stores speed-up degree information indicating an estimated value of the speed-up effect when reused,
The reuse data deletion means stores the calculation cost data calculated by the calculation cost estimation means in the sub-analysis result shared storage means as speed-up degree information, and based on the speed-up degree information, the sub-analysis result shared storage means The data stream analysis speed-up device according to claim 4 or 5, wherein an entry that is not reused is extracted and deleted from the stored calculation results.   7. The data stream analysis high-speed data processor according to claim 6, wherein the event data analysis library means includes sub-analysis observation means for observing the time taken for the calculation by the sub-analysis means and recording the observation result as an estimation of the calculation cost in the calculation cost estimation means. Device. The sub-analysis result shared storage means stores information indicating the number of times the calculation result has been reused,
The reusable data deleting means deletes the data from the less frequently used data based on the information indicating the number of times the calculation result is reused when deleting the data from the sub-analysis result shared storage means. The data stream analysis speed-up device according to any one of the above. The sub-analysis result shared storage unit management unit includes a reuse shared storage unit management timing determination unit that measures the timing of executing processing for deleting data with a low possibility of reuse from the sub-analysis result shared storage unit. The data stream analysis speed-up device according to any one of claims 4 to 8. A stream data analysis speed-up method that sequentially analyzes data that is output every moment,
Data flow processing for managing the event data that has occurred, temporarily storing in the storage means for analysis, and deleting from the storage means when no longer needed,
Event data analysis processing for analyzing event data stored in the data flow processing in parallel by a plurality of analysis methods;
The event data analysis process is divided into a plurality of subtasks, and the event data common analysis process for speeding up the analysis by reusing past calculation results for the subtasks commonly used for the plurality of event analysis processes;
In order for the event data common analysis process to analyze at high speed, it includes a data reuse process for storing past calculation results and checking the possibility of reuse,
A data stream analysis speeding-up method, wherein the data flow processing, event data analysis processing, event data common analysis processing, and data reuse processing are operated in parallel.   11. The data stream analysis speed-up method according to claim 10, further comprising a data analysis scheduling process for synchronizing analysis windows in a plurality of instances of the event data analysis process. In the data analysis scheduling process, a call history management process for generating original data for extracting an instance of the event data analysis process that is likely to be synchronized by temporarily recording a call of the event data common analysis process,
Analyzing the call history data accumulated in the call history management process, a data analysis scheduling process for controlling the analysis start timing of the analysis window of the event data common analysis process,
The data stream analysis speeding-up method according to claim 11, further comprising: a shift request process for requesting to shift analysis data of each event data analysis process based on a result of the data analysis scheduling process.   The data stream analysis speed-up method according to any one of claims 10 to 12, further comprising a reuse data deletion process for deleting an unnecessary entry in the analysis result stored in the storage means. In the reuse data deletion process, the overwriting monitoring process for monitoring the data flow process and detecting the time of the data already deleted from the data storage means;
14. The data stream analysis speed-up according to claim 13, further comprising: a deletion process that deletes data that is not reused from past calculation results stored in the data reuse process in response to the result of the overwrite monitoring process. Method. A stream data analysis acceleration program for sequentially analyzing data output every moment,
On the computer,
Data flow processing for managing and analyzing generated event data temporarily in the storage means for analysis, and deleting from the storage means when no longer needed,
Event data analysis processing for analyzing event data stored in the data flow processing in parallel by a plurality of analysis methods;
The event data analysis process is divided into a plurality of subtasks, and the event data common analysis process for speeding up the analysis by reusing past calculation results for the subtasks commonly used for the plurality of event analysis processes;
A data stream analysis acceleration program for storing a past calculation result and executing a data reuse process for checking the possibility of reuse in order for the event data common analysis process to analyze at high speed.   16. The data stream analysis acceleration program according to claim 15, for executing data analysis scheduling processing for synchronizing analysis windows in a plurality of instances of event data analysis processing.

Images