JP5169117B2 - Data stream analysis system and apparatus, method and program used therefor - Google Patents

Description

  The present invention relates to a data stream analysis system, an information processing apparatus, a memory storage method, and a memory storage program that sequentially analyze information that occurs continuously.

  Patent Document 1 and Patent Document 2 describe technologies relating to memory allocation for sequentially analyzing information generated continuously.

  In Patent Document 1, a dedicated memory pool for use by specific software and a general-purpose memory pool mainly used by other software are provided in the memory, and when the memory is allocated to the specific software, the remaining dedicated memory pool is stored. There is described an information processing apparatus including a memory allocation unit that allocates a part of a general-purpose memory pool when the amount is insufficient.

  Patent Document 2 discloses a memory management system that manages a shared memory divided into a part that can be assigned to all CPUs (shared part) and a part that can be assigned only to a predetermined CPU (dedicated part). Memory demand forecasting means for obtaining the memory demand forecast amount of each CPU, and when securing memory in response to a memory securing request from the CPU, if the memory capacity of the dedicated part is insufficient, the memory amount for the demand forecast capacity is A memory management system is described that includes a memory securing means that moves from a shared part to a dedicated part.

JP-A-8-339324 JP-A-10-320358

  However, if a large amount of data is generated constantly and constantly, the memory for analysis cannot be secured and the sensor data may be lost. For example, when information detected by a sensor (hereinafter referred to as sensor data) is analyzed by a general-purpose computer, even if a memory for temporarily holding sensor data is provided in the sensor, it is only temporary. In order to analyze, it is necessary to store the sensor data in the memory of a general-purpose computer. When analyzing sensor data of a plurality of sensors, sensor data generated from the plurality of sensors must be stored in a memory of a general-purpose computer. In general, the memory securing function provided by the OS of the general-purpose computer is slow in processing, and depending on the frequency of sensor data generation, the memory securing cannot catch up and the sensor data may be lost.

  When the information processing apparatus described in Patent Document 1 is used to process sensor data generated every moment, access concentrates on the memory allocation means. Further, in the configuration of the information processing apparatus disclosed in Patent Document 1, exclusive control is required when the memory allocation unit secures the memory. For this reason, if the occurrence of data with different occurrence frequencies overlaps, copying of the sensor data with a high occurrence frequency (processing) will not be in time while processing the data, and the data will be overwritten and erased by newly generated sensor data. there is a possibility.

  Note that when the memory allocation means is distributed by the number of sensors using the technique described in Patent Document 1, the management of the memory capacity becomes complicated and the use efficiency of the memory may be reduced.

  In the memory management system described in Patent Document 2, a memory area that each CPU can secure without performing an exclusive process is prepared in advance, and when the memory capacity becomes insufficient, the demand forecast capacity starts from the shared portion. By shifting the amount of memory, the exclusion process associated with accessing the shared portion is reduced as much as possible. However, the memory management system described in Patent Document 2 is aimed at optimizing the processing of each CPU in a multi-CPU, and performs high-speed memory allocation processing for sensor data generated from thousands or tens of thousands of sensors. This is not taken into account. For example, if the method described in Patent Document 2 is used in the memory securing process for sensor data, the memory securing process for each sensor data can be speeded up evenly, but the sensor is not frequently updated. In contrast, a dedicated memory area is allocated. For this reason, since the memory area for memory management is required, the memory use efficiency is significantly reduced.

  Accordingly, an object of the present invention is to make it possible to balance the memory usage efficiency and the execution speed in a data stream analysis system that sequentially analyzes continuously generated information. In the present invention, a data stream is a collection of data that is generated permanently (continuously) while changing its value as a data flow, which is an occurrence of data that changes every moment. Point to. In addition, when there are a plurality of generation sources that permanently generate data while changing the value, it may indicate a collection of data generated from the plurality of generation sources.

A data stream analysis system according to the present invention is a data stream analysis system for analyzing continuously generated data, and a data storage means in which memory areas for storing data for analysis are allocated as a plurality of memory pools; A plurality of memory management means that correspond one-to-one with any one of the plurality of memory pools, and that collectively perform procedures for the associated memory pool, and are independent for each type of pre-divided data When the data to be analyzed is generated, one or more memory storage means for storing the data in the memory area, and the correspondence relationship for storing information indicating the correspondence between the memory storage means and the memory management means comprising storage means, and optimizing means for optimizing the correspondence between the memory storage means and the memory management unit, multiple Memory management means are divided into a dedicated memory management unit for performing a shared memory management means for performing procedures with respect to the memory pool after performing exclusive control, the procedure for the memory pool without exclusive control, optimization means, Optimize the correspondence between memory storage means and memory management means based on the frequency of data generation by type, the number of memory management means functioning as shared memory management means, and the number of memory management means functioning as dedicated memory management means The correspondence relationship storage means stores information indicating the correspondence between the memory storage means optimized by the optimization means and the memory management means. When the data to be analyzed is generated , the memory storage means The memory area secured from any of the plurality of memory pools using the memory management means associated with the storage means Characterized by storing over data.

An information processing apparatus according to the present invention is an information processing apparatus used for analyzing continuously generated data, and a data storage in which memory areas for storing data for analysis are allocated as a plurality of memory pools Means, a plurality of memory management means for performing one-to-one correspondence with any one of the plurality of memory pools and performing a procedure for the associated one memory pool, and for pre-sorted data When data to be analyzed operates independently for each type and data to be analyzed is generated, information indicating the correspondence between one or more memory storage means for storing the data in the memory area, the memory storage means, and the memory management means is displayed. includes a correspondence relationship storing means for storing, and optimizing means for optimizing the correspondence between the memory storage means and the memory management unit, a plurality of memory management Stage is divided into a dedicated memory management unit for performing a shared memory management means for performing procedures with respect to the memory pool after performing exclusive control, the procedure for the memory pool without exclusive control, optimization means, Type Optimize the correspondence between the memory storage means and the memory management means based on the occurrence frequency of the data, the number of memory management means functioning as the shared memory management means and the number of memory management means functioning as the dedicated memory management means, The correspondence relationship storage means stores information indicating the correspondence between the memory storage means optimized by the optimization means and the memory management means, and when the data to be analyzed is generated, the memory storage means The data is stored in a memory area secured from one of a plurality of memory pools using the memory management means associated with And wherein the Rukoto.

In addition, the memory storage method according to the present invention includes a storage unit in which memory areas for storing continuously generated data are divided and assigned to a plurality of memory pools, and any one of the plurality of memory pools. Storage method applied to an information processing apparatus that includes a plurality of memory management means that perform one-to-one correspondence with a memory pool that is associated with each other, and that performs a plurality of memory management units. The method is divided into a shared memory management means that performs a procedure for a memory pool after performing exclusive control and a dedicated memory management means that performs a procedure for a memory pool without performing exclusive control . Frequency of occurrence, number of memory management means functioning as shared memory management means, and memory management means functioning as dedicated memory management means Based on the number of, correspondence between the memory storage means and to optimize the association between memory management unit, optimized memory storage means and memory management means operating independently of each type of data previously divided stores information indicating, memory storage means, upon receiving the data those said memory storage means is processed, the memory area secured by using the memory management means are associated with the memory storage means The data is stored.

The memory storage program according to the present invention is a memory storage program applied to an information processing apparatus that analyzes continuously generated data, and a memory area for storing continuously generated data is: A memory storage process that operates independently for each type of data that has been pre-divided into a computer having storage means that is divided and assigned to a plurality of memory pools, and one-to-one correspondence with any one of the plurality of memory pools , A plurality of memory management processes that collectively perform a procedure for one associated memory pool, a plurality of memory management processes , a shared memory management process that performs a procedure for a memory pool after performing an exclusive control, and an exclusive control Generation of data by type after dividing into dedicated memory management processing that performs procedures for memory pools without performing A memory storing process that operates independently for each type of data divided in advance based on the number of memory management means functioning as shared memory management means and the number of memory management means functioning as dedicated memory management means, and the memory An optimization process for optimizing the correspondence with the management process, and a process for storing information indicating the correspondence between the optimized memory storage process and the memory management process are executed. Among the memory management processes corresponding to any one of the plurality of memory pools when receiving the type of data targeted for the storage process, the memory management process associated with the memory storage process The data is stored in the memory area secured using, and is associated with exclusive control according to the settings in the memory management process. Or to perform procedures for one memory pool are, or, characterized in that to perform the procedure for a memory pool is associated not perform exclusive control.

  According to the present invention, it is possible to balance the memory usage efficiency and the execution speed in a data stream analysis system that sequentially analyzes information that occurs continuously.

Embodiment 1. FIG.
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is an explanatory diagram showing a configuration example of a data stream analysis system according to the first embodiment of the present invention. The data stream analysis system shown in FIG. 1 includes a computer 100 for sequentially analyzing data continuously generated from a data generation source. The computer 100 includes a plurality of memory storage units 11 (11-1 to 11- n), a plurality of memory management means 12 (12-1 to 12-m), a correspondence relation storage means 13, and a data analysis means 14.

  The memory storage unit 11 is a processing unit that operates independently for each type of data divided in advance. When data to be analyzed is generated, a memory area in the computer 100 (specifically, a data analysis unit) The data is stored in a memory area 14 can be referred to. This memory area is allocated as a plurality of memory pools to data storage means (not shown) provided in the computer 100. When the type of data that is to be processed by the memory storage unit 11 is generated, the memory storage unit 11 uses the memory management unit 12 associated with the memory storage unit 11 as a data storage unit included in the computer 100. The data is stored in a memory area secured from any of the plurality of allocated memory pools. The association between the memory storage unit 11 and the memory management unit 12 is in accordance with the information held by the data category-memory association management unit 14.

  Here, the data type may be classified based on, for example, an input method of data to be analyzed, a generation source or a generation factor, or a combination thereof. Any type can be used as long as it can be discriminated in an interface (for example, an API provided by a port device driver, a protocol stack, or a file system) with an input unit included in the computer 100. In consideration of parallel operation when storing the data to be stored in the memory, it is preferable to classify the data according to the frequency of data generation.

  Each memory storage unit 11 is implemented as, for example, an instance of a processing entity (in this case, a process, a thread, a task, etc.) in program processing that runs on the computer 100. Note that “n” in the memory storage means 11-1 to 11-n in FIG. 1 indicates the number of instances in the program processing that operate as the memory storage means 11 (here, the same as the number of data types). Yes. Hereinafter, each memory storage unit 11 refers to each instance operating as the memory storage unit 11. It is assumed that which memory storage means 11 is operated in advance for the type of data.

  Next, a specific example of the memory storage unit 11 will be shown. For example, if the data generation source (for example, an RFID reader that reads information from an RFID tag) is connected to a USB port or the like of the computer 100 via a HUB, the memory storage unit 11 includes a USB port driver. This is processing means for performing processing for receiving (inputting) data output from the data generation source via the. Note that the type of data may be determined by, for example, a device number assigned in advance for each data generation source.

  Further, for example, when a data generation source (for example, a server device that outputs log information) is connected via a network, the memory storage unit 11 uses the data generation source ( Alternatively, it is a processing means for performing a process of receiving (receiving) data transmitted by a communication device connected to the data generation source. The data receiving operation includes a case where data is requested from the data generation source and received as a response. The type of data may be determined uniquely by the receiving port, for example, or may be determined by information indicating the data source (IP address or the like). If the data to be analyzed is log information, the type of data may be classified according to the log generation factor. Of course, the log may be classified according to the source that generated the log, or may be classified by a combination thereof.

  Further, for example, when the computer 100 includes a data generation source (for example, other processing means that operates on the computer 100), the memory storage means 11 may store data via a file system or OS, for example. It becomes a processing means which performs the process which receives the data which the generation source output. For example, when the data generation source outputs data in a file format, the memory storage unit 11 serves as a processing unit that performs a process of reading the data via the file system. Further, for example, when the data generation source outputs data as an inter-thread message, the memory storage unit 11 is a processing unit that performs a process of receiving the data via the OS. The data receiving operation includes a case where the memory storage unit 11 voluntarily confirms the presence / absence of output according to a predetermined cycle and reads it. The type of data may be uniquely determined by specifying a destination by a file name or an inter-thread message source, for example. In addition, it is also possible to combine these connection forms.

  The memory management means 12 is associated with one of a plurality of memory pools to which a memory area for storing data is assigned, and is associated with one memory pool (hereinafter, a management target memory pool). Collective procedures for Here, the memory pool is a collection of memory areas allotted for use in program processing. A memory area acquired from a memory pool according to a predetermined procedure may be called a memory block. Therefore, it can be said that the memory pool is a memory area handled as an aggregate of a plurality of memory blocks. Note that the memory management unit 12 may generate a management target memory pool (assignment to a real memory area).

  The memory management means 12 manages at least the use status (whether reserved or released) of the memory area allocated as the managed memory pool for the managed memory pool while other means are used. (Specifically, the memory area is secured from the management target memory pool in response to a request from the memory storage unit 11), or in response to a request from another unit (specifically, the data analysis unit 14). Collectively perform processing to release the memory area secured from the memory pool. The procedure for the management target memory pool includes processing for referring to or updating information indicating the usage status of the management target memory pool.

  In this embodiment, the memory management unit 12 functions as a shared memory management unit that performs a procedure for the management target memory pool after performing exclusive control, and performs a procedure for the management target memory pool without performing exclusive control. It can be divided into those that function as dedicated memory management means. It should be noted that at least one or more shared memory management means and at least one or more dedicated memory management means are included. For example, each memory management unit 12 can set whether the management target memory pool is used exclusively for one data type or shared from a plurality of data types, and is shared by switching the setting. You may make it switch between the case where it functions as a memory management means, and the case where it functions as a dedicated memory management means.

  Each memory management unit 12 is implemented as, for example, a procedure function group on program processing that operates on the computer 100 and an instance of a variable used therefor. Note that “m” in the memory management means 12-1 to 12 -m in FIG. 1 indicates the number of instances on the program processing functioning as the memory management means 12 (here, the same as the number of memory pools). . The number of memory pools may be smaller than the number of data types (m <n). Hereinafter, each memory management unit 12 refers to each instance that functions as the memory management unit 12.

  For example, each memory management unit 12 may be able to switch between two types of modes, a thread safe mode and a thread free mode, in providing a function for performing a procedure for the management target memory pool to other units. . The thread safe mode is a mode in which simultaneous calls from multiple processes are forcibly converted into sequential calls by performing exclusive control on certain data and procedures (in this case, a series of procedures for the managed memory pool). It is. The thread safe mode is slow because of exclusive control. The thread free mode is a mode that does not perform exclusive control on the assumption that there are no simultaneous calls from a plurality of processes. Since the exclusive control is not performed, the thread free mode can operate at a very high speed as compared with the thread safe mode.

  The correspondence relationship storage unit 13 stores information indicating the correspondence between the memory storage unit 11 and the memory management unit 12. The correspondence relationship storage unit 13 may store, for example, information in which an identifier for identifying the memory storage unit 11 and an identifier for identifying the memory management unit 12 are associated with each other. Further, each memory storage means 11 is associated with a data type in a one-to-one relationship, and each memory management means 12 is associated with a memory pool in a one-to-one relationship. For example, 13 may store information in which a data type ID for identifying a data type is associated with a memory pool ID for identifying a memory pool. In addition, for example, information in which a data type ID is associated with a pointer to each memory management unit 12 (a pointer variable indicating each instance implemented as the memory management unit 12) may be stored.

  The data analysis means 14 analyzes continuously generated data. The data analysis unit 14 is stored in the memory included in the computer 100 by the memory storage unit 11 (specifically, one of the memory areas allocated as a plurality of memory pools in the data storage unit included in the computer 100). Read data for analysis. For example, the data analysis unit 14 may receive notification that the data has been stored in the memory area from the memory storage unit 11, and sequentially read and analyze the data stored in the memory area. Note that the number of the data analysis means 14 (the number of instances of the processing entity operating as the data analysis means 14) may be any number. For example, it may be implemented as one instance that analyzes all types of data collectively, or may be implemented as a plurality of instances that operate independently for each type of data.

  Hereinafter, a description will be given using a more specific configuration example. FIG. 2 is a block diagram showing a more specific configuration example of the data stream analysis system according to the present embodiment. The data stream analysis system shown in FIG. 2 is an example of a data stream analysis system that analyzes sensor data generated from sensors, and a plurality of sensors 200-1 to 200- that are sources of sensor data to be analyzed. n and a computer 100 for analyzing sensor data generated from the sensors 200-1 to 200-n. In this example, it is assumed that the types of sensor data are classified according to the generation source of the sensor data.

  The computer 100 also includes sensor data receiving means 11-1 to 11-n, memory management means 12-1 to 12-m, sensor-memory correspondence management means 13, data analysis means 14, and correspondence optimization. Means 15, sensor attribute management means 16, and exclusive control means 17 are included.

  The sensor 200 is a means for generating sensor data to be analyzed and may be any means for generating data that changes every moment. For example, the sensor 200 measures the temperature at intervals of 1 minute and generates data thereof, or means for reading the information of an IC chip in which product information is written, such as an RFID system, with an RFID reader and generating the data. Etc.

  Sensor data generated by the sensors 200 is temporarily stored in a data storage means in each sensor 200. In this example, it is assumed that the data analysis unit 17 cannot directly access the data storage unit in the sensor 200. Therefore, in order to analyze the generated sensor data, it is necessary to store the sensor data in a memory area accessible from the data analysis means 17. In this example, it is assumed that sensor data generated from a plurality of sensors 200 (sensors 200-1 to 200-n) is analyzed by one computer 100.

  The sensor data receiving means 11 is a processing means that operates independently for each type of sensor data (ie, for each source of sensor data) that has been divided in advance, and when the sensor data to be analyzed is generated, the computer The sensor data is stored in a memory area in 100 (specifically, a memory area that can be referred to by the data analysis means 14). A memory area for storing sensor data to be analyzed is allocated as a plurality of memory pools 121 to a data storage device (not shown) provided in the computer 100. The sensor data receiving means 11 is a processing means corresponding to the memory storage means 11 in FIG. In this example, it is assumed that an identifier for identifying each sensor 200 is assigned to the sensor 200 that is a generation source of the sensor data. Hereinafter, this identifier is referred to as a sensor ID. The sensor ID is also an identifier for identifying each instance that operates as the sensor data receiving unit 11, and is also an identifier for identifying the type of sensor data. The sensor ID may be a serial number, for example. In this example, it is assumed that sensor IDs S1, S2,..., Sn (n is the number of sensors) are assigned.

  The memory management unit 12 is associated with one of a plurality of memory pools to which a memory area for storing sensor data is allocated, and is associated with one memory pool (memory to be managed). Collective procedures for pool). The memory management unit 12 includes a free memory management unit 122, a memory acquisition / release unit 123, and an access mode storage unit 124 as functional blocks for realizing these functions. In this example, it is assumed that an identifier for identifying each memory pool is assigned to a memory pool for storing sensor data for analysis. Hereinafter, this identifier is referred to as a memory ID. The memory ID is also an identifier for identifying each instance that functions as the memory management unit 12. The memory ID may be a serial number, for example. In this example, it is assumed that memory IDs M1, M2,..., Mm (m is the number of memory pools) are assigned.

  The free memory management unit 122 manages at least the usage status of the memory area allocated as the management target memory pool for the management target memory pool 121. As an initialization process, the free memory management unit 122 uses a system call provided by the OS running on the computer 100 before starting the analysis of sensor data, and a data storage unit included in the computer 100 The management target memory pool 121 may be generated by securing a memory area to be used as the management target memory pool 121 from (not shown). The OS operating on the computer 100 may be a general-purpose OS such as Windows (registered trademark) or LINUX.

  Specifically, a system call for securing a memory area to be handled as a memory block may be called for a predetermined number of memory blocks. At that time, the size of the memory block is designated. If the size of the sensor data of all the sensors 200 is common, the size of the memory block may be used. If not common, the largest sensor data size may be used. Then, a set of pointers to the reserved memory area (memory block) obtained as a return value of the system call may be held as entity information of the management target memory pool 121. Hereinafter, a pointer to a memory block is referred to as a memory block pointer. The number of memory blocks may be set based on the amount of memory installed in the execution machine, the frequency of occurrence of sensor data, and the like. It should be noted that since the memory area is not necessarily secured from the OS according to the conditions, more strictly, exception processing when the memory area is not secured is necessary.

  The free memory management unit 122 designates a memory size required as the total memory capacity of the management target memory pool 121 indicated by a predetermined memory block size and the number of memory blocks, and batches the memory pool. It is also possible to secure a memory area handled as In such a case, a set of pointers obtained by adding the size of the memory block as an offset from the pointer to the reserved memory area (memory pool) obtained as a return value of the system call is set as the management target memory pool 121. May be held as the entity information. In this way, the number of OS system calls can be reduced. Further, in such a case, when collective securing from the OS fails, securing for each memory block may be attempted.

  When the management target memory pool 121 is generated (the real memory area is allocated), the free memory management unit 122 initializes information indicating the usage status of the memory area allocated as the management target memory pool 121. For example, the free memory management unit 122 uses, as information indicating an unused memory block, a set of pointers to memory blocks in a state where the memory block is not secured by another processing unit (hereinafter referred to as a free state). You may make it hold | maintain. This can be realized by a stack or a queue.

  The free memory management unit 122 may hold a pointer to a memory block in a free state using, for example, an available memory queue as shown in FIG. In the free memory management unit 122, for example, when the management target memory pool 121 is generated, all the memory block pointers may be stored in the available memory queue. Thereafter, the memory block pointer is extracted from the available memory queue in response to a memory block allocation request (Pop), or the pointer to the memory block to be released in response to a memory block release request is returned to the available memory queue. (Push)

  The memory acquisition / release means 123 provides a procedure for acquiring a memory block from the management target memory pool 121 and a procedure for releasing the acquired memory block to other means. Specifically, the memory acquisition / release unit 123 is implemented as a provision function that prescribes a procedure for acquiring a memory block from the management target memory pool 121 and a procedure for releasing the acquired memory block. Is done.

  The memory acquisition / release unit 123 acquires a free memory block from the management target memory pool 121 in response to a memory block acquisition request from another unit (specifically, the sensor data receiving unit 11). In this example, if the memory acquisition / release unit 123 acquires a memory block in a free state by taking out (Pop) one memory block pointer from the available memory queue managed by the free memory management unit 122. Good. FIG. 4 is an explanatory diagram showing a change in the free state of the management target memory pool 121 related to acquisition of a memory block. FIG. 4A is an explanatory diagram illustrating an example of a free state of the management target memory pool 121 before acquiring the memory block. FIG. 4B is an explanatory diagram showing an example of a free state of the management target memory pool 121 after acquiring the memory block. As shown in FIG. 4A, for example, when the memory block pointer at the head of the available memory queue is a pointer to the memory block MB1, the memory acquisition / release unit 123 sets the free memory management unit 122 to A pointer to the memory block MB1 is taken out from the available memory queue. Specifically, after referring to the entry related to the memory block pointer at the head, the entry is deleted from the available memory queue.

  In the memory acquisition / release means 123, when the memory management means 12 functions as a shared memory management means, a series of accesses to the available memory queue (one memory block pointer is extracted from the available memory queue and A critical process is defined as a series of processing until an entry is deleted. Specifically, exclusive control is started before one memory block pointer is extracted from the available memory queue or before the presence of an entry is confirmed, and after the entry related to the extracted memory block pointer is deleted, the exclusive control is released. That's fine. By referring to the entry at the head of the available memory queue by the memory acquisition / release means 123 of a memory management means 12 by calling from a certain sensor data receiving means 11, before deleting this entry from the queue, This is to prevent the same memory area from being provided if the memory acquisition / release means 123 of the same memory management means 12 can access this entry by a call from the sensor data accepting means 11.

  Note that the pointer to the memory block MB1 acquired at this time is passed to another means (specifically, the sensor data receiving means 11) that called the memory acquisition / release means 123, and the sensor data to be analyzed is sent to the memory block MB1. Used as information indicating the storage location. Specifically, the sensor data receiving means 11 stores the sensor data in the memory block MB1 pointed to by the memory block pointer and passes it to the data analysis means 14. FIG. 4 shows an example in which the memory block pointer is delivered to the data analysis means 14 using the data analysis memory queue. Here, it is added at the end of the data analysis memory queue.

  On the other hand, the memory acquisition / release means 123 releases the acquired memory block in response to a memory block release request from another means (specifically, the data analysis means 14). The memory acquisition / release unit 123 may release the memory block by adding (Push) the memory block pointer transferred to the available memory queue managed by the free memory management unit 122. FIG. 5 is an explanatory diagram showing a change in the free state of the management target memory pool 121 related to the release of the memory block. FIG. 5A is an explanatory diagram illustrating an example of a free state of the management target memory pool 121 before the memory block is released. FIG. 5B is an explanatory diagram showing an example of a free state of the management target memory pool 121 after the memory block is released. As shown in FIG. 5A, for example, when the memory block storing the sensor data analyzed by the data analysis unit 14 is a pointer to the memory block MB1, the memory acquisition / release unit 123 includes A memory block release request designating a pointer to the memory block MB1 is made. In such a case, the memory acquisition / release unit 123 uses the free memory management unit 122 to add a pointer to the memory block MB1 to the available memory queue. Specifically, an entry in which information related to the designated memory block pointer is registered is added to the available memory queue. Here, it is added at the end of the available memory queue. By adding an entry to the available memory queue, the memory block pointer becomes empty and can be used as a memory area for storing other sensor data.

  If there is a possibility that the consistency of the queue may not be obtained due to simultaneous access by a plurality of processes at the time of releasing the memory block, it depends on whether the memory management unit 12 functions as a shared memory management unit. Thus, exclusive control may be performed. In this example, the sensor data receiving unit 11 and the data analyzing unit 14 are mounted so that access to the management target memory pool 121 does not compete by synchronizing or mounting as the same instance. Shall.

  Whether the memory management unit 12 functions as the shared memory management unit 12 or the dedicated memory management unit 12 may be determined according to information stored in the access mode storage unit 124. Note that the memory acquisition / release means 123 may refer to the information stored in the access mode storage means 124 every time a memory block acquisition request or a memory block release request is called. Note that the memory management unit 12 that functions as the shared memory management unit and the memory management unit 12 that functions as the dedicated memory management unit may be defined separately in advance. In such a case, the access mode storage unit 124 may be omitted.

  The access mode storage unit 124 stores a flag indicating whether or not exclusive control is necessary for the procedure for the management target memory pool 121. This flag may be 1 bit.

  The sensor-memory correspondence management unit 13 stores information indicating the correspondence between the sensor data receiving unit 11 and the memory management unit 12. In this example, the sensor-memory correspondence management unit 13 stores information in which the sensor ID and the memory ID are associated with each other as shown in FIG. In FIG. 6, sensor ID = S1 and memory ID = M1 are associated with each other. Further, the sensor ID = S2 is associated with the memory ID = M2. Further, the sensor ID = S3 and the memory ID = M1 are associated with each other. Therefore, it can be seen that the sensor data received by the sensor data receiving unit 11 identified by the sensor ID = S1 is stored in the memory block secured by using the memory management unit 12 identified by the memory ID = M1. Further, since the memory ID = M1 is associated with the sensor ID = S1 and the sensor ID = S3, the memory management unit 12 identified by the memory ID = M1 can function as a shared memory management unit. Recognize.

  The data analysis means 14 reads and analyzes the sensor data stored in the memory area in the computer 100 as needed by the sensor data reception means 11. In this example, it is assumed that the data analysis unit 14 is associated with the sensor data reception unit 11 on a one-to-one basis. For example, the sensor data reception unit 11 may specify which data analysis unit 14 is to perform analysis processing by storing a pointer to the corresponding data analysis unit 14.

  As shown in FIG. 4B, the data analysis unit 14 has a data analysis queue that holds information (memory block pointer) indicating the storage destination of the sensor data to be analyzed in order of generation time. May be. An example of the requirement for the data analysis means 14 is to sequentially analyze sensor data that is permanently generated while changing from moment to moment. Here, “sequential” means that the contents of past sensor data whose analysis has been completed are presupposed to be deleted, and are not used again for analysis. Therefore, the data analysis unit 14 sets the latest sensor data for a certain period as an analysis target. A specific analysis method is not particularly limited. For example, sensor data statistical processing, pattern matching, or classification may be performed using a widely known algorithm. The data analysis means 14 uses the memory management means 12 that manages the memory pool from which the memory area (memory block) storing the sensor data is acquired when the analysis of the sensor data is completed. , Release the memory block.

  Correspondence optimization means 15 includes the occurrence frequency of each type of data (in this example, the occurrence frequency of data of each sensor), the number of shared memory management means (the number of shared memory pools), and dedicated memory management means. The correspondence between the sensor data receiving means 11 and the memory management means 12 is optimized based on the number of memory pools (the number of dedicated memory pools). For example, the correspondence optimization unit 15 allocates a dedicated memory pool to each high-frequency sensor based on the attribute information regarding the occurrence frequency of each sensor stored in the sensor attribute management unit 16, and The correspondence between the sensor data receiving unit 11 and the memory management unit 12 may be optimized by equally allocating the shared memory pool. Each sensor is assigned either a dedicated memory pool or a shared memory pool. When a dedicated memory pool is allocated, the sensor data receiving means 11 associated with the sensor and the memory management means 12 functioning as the dedicated memory management means may be associated with each other on a one-to-one basis. In addition, when a shared memory pool is allocated, one memory management unit 12 functioning as a shared memory management unit may be associated with a plurality of sensor data receiving units 11 associated with the sensor.

  The sensor attribute management means 16 stores information regarding the frequency of data generation for each type of data (for each sensor). For example, for each sensor, information indicating whether the sensor is a high-frequency sensor or a low-frequency sensor may be stored. Here, the high-frequency sensor refers to a sensor that is frequently updated and has a large influence when exclusive control is performed. The low-frequency sensor is a sensor that is rarely updated and has little influence even if exclusive control is performed. However, this is an example, and the present invention is not limited to this. For example, information indicating the frequency of data generation may be stored for each sensor in five steps or ten steps. The sensor attribute management means 16 may acquire information related to the occurrence frequency of data of each sensor, for example, by receiving the information via an input means included in the computer 100, or is stored in a setting file or the like in advance. You may acquire by reading the content.

  The exclusive control unit 17 performs exclusive control in response to a request from the memory acquisition / release unit 123. Here, exclusive control means that only one processing entity is allowed to enter the critical section at the same time. The exclusive control can be realized by using, for example, a mutex or a semaphore generally provided as an API in a general-purpose OS.

  Next, the operation of this embodiment will be described with reference to FIG. FIG. 7 is a flowchart showing an operation example of the data stream analysis system according to the present embodiment. As shown in FIG. 7, first, information on the frequency of data generation of each sensor is set (stored) in the sensor attribute management means 16 (step S11). In step S11, the sensor attribute management unit 16 obtains information related to the frequency of data generation of each sensor, for example, via an input unit provided in the computer 100, or is stored in a setting file or the like in advance. What is necessary is just to acquire by reading the content and to store in a predetermined memory area.

  Next, the correspondence relationship optimization unit 15 optimizes the correspondence between the sensor data reception unit 11 and the memory management unit 12 based on the information regarding the frequency of data generation of each sensor held in the sensor attribute management unit 16. (Step S12). The correspondence optimization unit 15 assigns the memory management unit 12 to each sensor data receiving unit 11 by using an algorithm as shown in FIG. May be optimized. FIG. 8 is a flowchart showing an example of an algorithm for optimization by the correspondence optimization means 15. Hereinafter, the number of sensors (that is, the number of sensor data receiving means 11) is referred to as SNUM, and the number of memory pools (that is, the number of memory management means 12) is referred to as MNUM. Note that MpNUM in FIG. 8 indicates the number of memory management means 12 functioning as dedicated memory management means. If the number of the memory management means 12 functioning as the shared memory management means is McNUM, there is a relationship of MNUM = MpNUM + McNUM. In this example, MpNUM and McNUM are each set to a numerical value of 1 or more. Note that McNUM may be zero depending on the total number of sensors or the number of low-frequency sensors.

  As shown in FIG. 8, the correspondence optimization unit 15 first compares the number of sensors classified as high-frequency sensors with the number of MpNUMs (step S121). MpNUM is the maximum number of memory management means 12 that can function as dedicated memory management means when the dedicated memory management means and shared memory management means are switched by setting. Note that the maximum number of memory management means 12 that can function as dedicated memory management means is MNUM when the number of high-frequency sensors = SNUM, and MNUM-1 otherwise.

  If the number of sensors classified as high-frequency sensors is larger (No in step S121), a dedicated memory pool cannot be assigned to all high-frequency sensors, and the process proceeds to step S122. Try to solve. If the number of sensors classified as the high-frequency sensor is smaller (Yes in step S121), the process may proceed to step S123 as it is.

  In step S122, a process of transferring some or all of the sensors classified as high-frequency sensors to low-frequency sensors is performed to reduce the number of high-frequency sensors. For example, one sensor randomly selected from the sensors classified as the high frequency sensor may be classified as the low frequency sensor. If more detailed frequency information of each sensor is known, the sensor with the lowest frequency among the sensors classified as the high-frequency sensor may be classified as the low-frequency sensor. The number of high-frequency sensors is reduced by repeating the process in step S122.

  In step S123, a dedicated memory pool is allocated to each high frequency sensor. That is, each sensor data reception means 11 associated with the high frequency sensor and the memory management means 12 functioning as the dedicated memory management means are associated with each other on a one-to-one basis. For example, the correspondence relationship optimization unit 15 sequentially associates the memory management unit 12 with each sensor data reception unit 11 associated with the high-frequency sensor, and sets the memory management unit 12 to exclusive control off. The mode may be set.

  Finally, the correspondence relationship optimization unit 15 assigns the remaining memory pools evenly to the low-frequency sensors (step S124). That is, the remaining memory management means 12 is shared memory management means, and is equally allocated to the sensor data receiving means 11 associated with the low frequency sensor. For example, when SNUM = 10 (high frequency = 2, low frequency = 8) and MNUM = 5, of the five memory management units 12, two memory management units 12 serve as dedicated memory management units as high frequency sensors. If assigned, the remaining memory management means 12 becomes three. The correspondence relationship optimization unit 15 may share the remaining three memory management units 12 with the eight low-frequency sensors. For example, if the remaining three memory management means 12 are a, b, and c, respectively, the a and b memory management means 12 are assigned to be shared by three low-frequency sensors, and the c memory management means 12 is assigned. What is necessary is just to allocate so that it may be shared by two low frequency sensors.

  In this way, the correspondence relationship optimization unit 15 reflects the information indicating the correspondence relationship to the sensor-memory correspondence management unit 13 when the optimization of the correspondence between the sensor data receiving unit 11 and the memory management unit 12 is completed. . The processing so far is preprocessing for analyzing sensor data.

  Each sensor starts sensing (step S13 in FIG. 7). For example, sensor data that changes from moment to moment is generated by starting the execution of each sensor. Each sensor generates sensor data asynchronously.

  When sensor data is generated by the start of sensing by each sensor, the sensor data receiving means 11 receives the sensor data according to the type of sensor data (here, the generation source of the sensor data) (step S14). First, the sensor data receiving means 11 specifies the memory management means 12 to be used based on the information indicating the correspondence between the sensor data receiving means 11 and the memory management means 12 held by the sensor-memory correspondence management means 13. (Step S15). The sensor data receiving means 11 is a sensor-memory correspondence managing means 13 that obtains a memory ID associated with a sensor ID assigned to a sensor that is processed by the sensor data receiving means 11. What is necessary is just to specify whether the management means 12 is used.

  Then, a memory block for storing sensor data is secured using the specified memory management means 12, and the received sensor data is stored in the secured memory block (step S16). Thus, the sensor data is copied to the memory in the computer 100. In the memory management unit 12, the memory acquisition / release unit 123 performs the process of taking out the memory block pointer from the available memory queue via the free memory management unit 122. At this time, if the memory management unit 12 functions as a shared memory management unit, a process of taking out a memory block pointer from the available memory queue via the free memory management unit 122 after performing exclusive control. Is done. The sensor data receiving unit 11 may pass the sensor data to the memory management unit 12 and the memory management unit 12 may perform a copy process.

  The data analysis unit 14 analyzes the sensor data copied to the memory in the computer 100 (step S17). If there is sensor data for which analysis has been completed, the data analysis unit 14 releases the memory block storing the sensor data (step S18). Note that the series of processing in steps S14 to S17 is performed asynchronously by each sensor data receiving unit 11 operating independently each time sensor data is generated.

  As described above, according to the present embodiment, it is possible to balance the memory usage efficiency and the execution speed. In the present embodiment, the speed reduction due to the centralized access to the memory management unit 12 can be reduced. The reason is that either a dedicated memory pool or a shared memory pool is allocated to the sensor data receiving means 11 that performs the memory storage process of the sensor data of each sensor asynchronously based on the attribute information relating to the data occurrence frequency. This is because the correspondence between the sensor data receiving means 11 and the memory management means 12 can be optimized.

  Further, in the present embodiment, speed reduction due to exclusive control accompanying access to the memory management unit 12 can be reduced. The exclusive control that is the cause of the reduction in the speed is necessary because the memory securing requests compete among the plurality of sensor data receiving means 11. In the present embodiment, by associating the sensor data receiving unit 11 with the memory management unit 12, a dedicated memory pool can be given to the sensor data receiving unit 11 that is associated with a frequently occurring sensor. This is because it is possible to eliminate contention between the sensor data receiving means 11 having a high memory securing request frequency.

Embodiment 2. FIG.
Next, a second embodiment of the present invention will be described. FIG. 9 is a block diagram illustrating a configuration example of a data stream analysis system according to the second embodiment. As shown in FIG. 9, the data stream analysis system according to the present embodiment is further provided with an occurrence frequency classification means 18 as compared with the first embodiment shown in FIG. The difference is that the means 125 is included.

  The usage status monitoring unit 125 monitors the timing at which the memory block is reserved, and measures the frequency at which the memory block is reserved for each type of sensor data (that is, for each sensor). For example, when the memory acquisition / release unit 123 performs the memory block acquisition process, the usage status monitoring unit 125 determines the sensor ID associated with the caller sensor data reception unit 11. It may be a means for determining and counting the number of calls for each sensor ID. Note that the sensor ID associated with the caller sensor data reception unit 11 may be passed as an argument when the memory acquisition / release unit 123 is used.

  The occurrence frequency classification unit 18 classifies each sensor based on the occurrence frequency of data based on the monitoring result by the usage state monitoring unit 125.

  FIG. 10 is an explanatory diagram illustrating an example of a monitoring result by the usage status monitoring unit 125. As shown in FIG. 10, the usage status monitoring unit 125 may count and hold the number of memory acquisition request calls for each sensor ID, for example, in units of one hour. In the example illustrated in FIG. 10, for example, it is indicated that the sensor data reception unit 11 identified by the sensor ID = S1 has been called 60 times during one hour until 10:00. The unit time may not be 1 hour. The processing of the usage status monitoring unit 125 may be performed by the memory acquisition / release unit 123. At that time, the memory acquisition / release means 123 may count the data internally and write it in a table as shown in FIG. 10 at regular intervals. Further, the counting may be performed by the memory acquisition / release unit 123, and the utilization status monitoring unit 125 may reflect the table. For example, the usage status monitoring unit 125 may make an inquiry to the memory acquisition / release unit 123 to update this table.

  The occurrence frequency classification unit 18 may classify each sensor into a high-frequency sensor and a low-frequency sensor, for example. FIG. 11 is an explanatory diagram illustrating an example of a sensor classification method. As shown in FIG. 11, for example, by setting a certain threshold regarding the occurrence frequency, it is determined whether or not the average number of calls for memory acquisition requests within a certain period indicated by the monitoring result is larger than this threshold. May be classified. In FIG. 11, the horizontal axis represents time, and the vertical axis represents the monitoring result. Further, FIG. 11 shows an example in which the determination is made based on the average number of calls for memory acquisition requests over four periods. For example, in FIG. 11, since the average number of calls for memory acquisition requests within a certain period from each sensor data accepting unit 11 associated with the sensor of sensor ID = S1, S2 is higher than the threshold value, sensor ID = Each sensor identified by S1 and S2 is classified as a high-frequency sensor. Also, for example, in FIG. 11, the average number of calls for memory acquisition requests within a certain period from the sensor data receiving means 11 associated with the sensor with sensor ID = S3 is lower than the threshold value, so sensor ID = S3 Are identified as low-frequency sensors. Note that the occurrence frequency classification unit 18 may classify the sensors by clustering the sensors into high-frequency sensors and low-frequency sensors using, for example, a K-average method.

  Next, the operation of this embodiment will be described. FIG. 12 is a flowchart showing an operation example of the data stream analysis system in the present embodiment. In FIG. 12, the same operations as those in the first embodiment shown in FIG. In FIG. 12, a dashed arrow means that the original step of the arrow affects the step ahead of the arrow asynchronously.

  In the present embodiment, after step S16 (sensor data memory storage process), a process of logging information indicating the status of the memory acquisition request for each sensor data receiving means 11 is performed (step S21). In step S21, for example, the usage status monitoring unit 125 writes the number of calls for a memory acquisition request from each sensor data receiving unit 11 in a table as shown in FIG.

  Further, in the present embodiment, another flow (hereinafter referred to as a subflow) that is executed asynchronously operates in parallel with a series of memory analysis processing (steps S14 to S17, hereinafter referred to as a main flow) (step S22). ~ Repeat S24).

  In this sub-flow, first, the occurrence frequency classification means 18 classifies each sensor based on the occurrence frequency of data in response to the monitoring result by the utilization status monitoring means 125 in step S21 (step S22). The occurrence frequency classification unit 18 regards the average of the number of calls for memory acquisition requests within a certain period for each sensor data reception unit 11 indicated by the monitoring result by the usage state monitoring unit 125 as the occurrence frequency of data for each sensor. Each sensor may be classified. Then, the classification result is reflected in the sensor attribute management means 16 (step S23). Correspondence relation optimizing means 15 optimizes the correspondence between sensor data receiving means 11 and memory management means 12 in response to the update of the sensor attributes (step S24).

  In this example, the logging result of step S21 in the main flow is asynchronously referred to in step S22 of the subflow, and sensor classification is performed. Then, the optimization result in step S24 of the subflow that has received the classification result is referred to asynchronously in step S15 of the main flow, and the used memory management means is specified.

  According to this embodiment, even when the frequency of occurrence of data of each sensor is not known in advance, the system can automatically create sensor attributes and optimally allocate memory. Other points are the same as those in the first embodiment.

Embodiment 3. FIG.
Next, a third embodiment of the present invention will be described. FIG. 13 is a block diagram illustrating a configuration example of a data stream analysis system according to the third embodiment. As shown in FIG. 13, the data stream analysis system according to the present embodiment is different from the second embodiment shown in FIG. 9 in that it further includes a data generation number predicting means 19.

  The data generation number predicting unit 19 predicts the frequency of data generation of each sensor in the future based on the monitoring result by the usage state monitoring unit 125. Note that the frequency of data generation of each sensor predicted by the data generation number prediction unit 19 may be stored in addition to, for example, a table serving as an output destination of the monitoring result by the usage status monitoring unit 125. The data generation number prediction means 19 may be separately stored in a different table.

  The prediction method of the data generation number prediction means 19 is not particularly limited. For example, the least square method may be used. FIG. 14 is an explanatory diagram showing an example of a prediction result by the data generation number prediction means 19. Here, an example using the least square method is shown. In the least squares method, the monitoring results in each period are approximated by a function, and a coefficient that minimizes the residual sum of squares is determined so that the assumed approximate function is more approximate to the measured value. It is a method to do. FIG. 14 shows an example approximated by a linear function. Note that the approximation is not limited to a linear function, but may be approximated by a second-order or higher-order function. If the function can be approximated, the future can be predicted by inputting the future time into this function. In the example shown in FIG. 14, the data occurrence frequency of the sensor identified by sensor ID = S1 is predicted to be i1 at t1, which is a future time.

  Next, the operation of this embodiment will be described. FIG. 15 is a flowchart showing an operation example of the data stream analysis system in the present embodiment. In FIG. 15, the same operations as those of the second embodiment shown in FIG. In FIG. 15 as well, a broken line arrow means that the original step of the arrow affects the step ahead of the arrow asynchronously.

  In the present embodiment, before the subflow step S22 (sensor classification process), a process of predicting the frequency of future data generation from the monitoring result by the usage status monitoring unit 125 is performed (step S31). In step S <b> 31, the data generation number prediction unit 19 predicts the frequency of future data generation of each sensor based on the monitoring result by the usage state monitoring unit 125. In step S22, the occurrence frequency classification unit 18 may classify each sensor based on the occurrence frequency of future data of each sensor predicted by the data generation number prediction unit 19.

  In this example, the logging result of step S21 in the main flow is asynchronously referred to in step S31 of the subflow, and the occurrence frequency of future data is predicted. Then, the optimization result in step S24 of the subflow that has received the prediction result is asynchronously referred to in step S15 of the main flow, and the used memory management means is specified.

  According to the present embodiment, memory allocation can be performed optimally in consideration of the future data generation state, not the current data generation state. Other points are the same as those in the second embodiment.

Embodiment 4 FIG.
Next, a fourth embodiment of the present invention will be described. FIG. 16 is a block diagram illustrating a configuration example of a data stream analysis system according to the fourth embodiment. As shown in FIG. 16, the data stream analysis system according to the present embodiment is different from the second embodiment shown in FIG. 9 in that the memory management unit 12 includes an average analysis life monitoring unit 126.

  The average analysis lifetime monitoring unit 126 monitors the timing at which the memory block is secured and released, and after the memory area is secured for each type of sensor data (that is, for each sensor) until the memory area is released. The average (hereinafter referred to as the average analysis lifetime) of the time required for the analysis (ie, the time required for data analysis) is measured.

  In this embodiment, in order to realize the average analysis life monitoring means 126, as shown in FIG. 17, information indicating the time when the memory block is secured is stored in each memory block of the memory pool. For this purpose (hereinafter referred to as the header area).

  For example, when the memory acquisition / release unit 123 performs the memory block acquisition process, the average analysis lifetime monitoring unit 126 stores the date and time at that time in the header area of the acquired memory block. Then, in response to the memory block release processing being performed by the memory acquisition / release means 123, the date and time (release date and time) at that time and the header area of the memory block pointed to by the designated memory block pointer are stored. The time required for data analysis (analysis lifetime) is calculated from the difference from the date and time (acquisition date). The average analysis life monitoring means 126 may temporarily store the analysis life for each sensor, and obtain the average value at regular intervals.

  FIG. 18 is an explanatory diagram showing an example of an observation result by the average analysis life monitoring unit 126. As shown in FIG. 18, the average analysis life monitoring unit 126 may have a table that stores the average analysis life for each sensor ID in units of one hour, for example, as a means for indicating an observation result. The sensor ID may be passed as an argument when using the memory acquisition / release means 123 related to memory block release, or as an argument when using the memory acquisition / release means 123 related to memory block acquisition. It may be stored in the header area after being passed. In FIG. 18, the time indicates the time when the average value is calculated. The average analysis life indicates an average value of analysis life after the time when the previous average is calculated. Note that the unit of the average analysis lifetime in FIG. 18 is milliseconds.

  In the present embodiment, the sensor attribute management unit 16 stores attribute information related to the occurrence frequency of data and attribute information related to the analysis lifetime for each sensor. The attribute information related to the analysis life may be information indicating whether the sensor is a long life sensor or a short life sensor, for example. Here, the longevity sensor refers to a sensor that requires a lot of time for analysis and requires a lot of memory for storing sensor data. A short-lived sensor is a sensor that does not require much time for analysis, and that does not require a memory for storing sensor data. However, the classification relating to the analysis life is an example, and for example, the analysis life of the data may be expressed in five or ten stages.

  In order to reduce the waste of memory allocation, it is preferable to distinguish between sensors that perform analysis that uses a large amount of memory and sensors that perform analysis that requires less memory. For example, two types of memory pools, a large size memory pool and a small size memory pool, may be prepared and selectively allocated according to the analysis life for each sensor. A large-sized memory pool refers to a memory pool in which a large capacity is secured as a whole memory pool. The small-sized memory pool is a memory pool in which a relatively small capacity is secured as a whole memory pool. Note that the capacity of the entire memory pool can be adjusted by the number of memory blocks. In this embodiment, the correspondence relationship optimization unit 15 performs an average analysis when allocating the memory management unit 12 functioning as the dedicated memory management unit or the memory management unit 12 functioning as the shared memory management unit according to the frequency of data generation. By allocating the memory management means 12 associated with the memory pool having the largest possible memory capacity to the sensor whose lifetime is longer than the predetermined reference time, the memory storage means 11 and the memory management means 12 are Optimize mapping.

  FIG. 19 is a flowchart illustrating an example of an optimization algorithm by the correspondence relationship optimization unit 15 in the present embodiment. Note that MbNUM in FIG. 19 indicates the number of large memory pools. If the number of small-sized memory pools is MsNUM, there is a relationship of MNUM = MbNUM + MsNUM. In this example, MbNUM and MsNUM are each a numerical value of 1 or more. MbNUM may be 0 or MsNUM may be 0 depending on the number of longevity sensors or the number of short-life sensors. MbNUM and MsNUM are determined in advance and are not changed during operation.

  Note that steps S121 and 122 in FIG. 19 are the same as steps S121 and 122 shown in FIG.

  When the problem of the number of memory management means 12 that can function as dedicated memory management means for the number of sensors classified as high-frequency sensors is solved, the correspondence optimization means 15 determines the number of sensors classified as longevity sensors. Is compared with the number of MbNUMs (step S221).

  Here, if the number of sensors classified as longevity sensors is larger (No in step S221), a dedicated large memory pool cannot be allocated to all longevity sensors. Try to solve the problem. If the number of sensors classified as longevity sensors is smaller (Yes in step S221), the process may proceed to step S223 as it is.

  In step S222, a process of transferring some or all of the sensors classified as longevity sensors to shortlife sensors is performed to reduce the number of longevity sensors. For example, one sensor randomly selected from the sensors classified as longevity sensors may be classified as a short-life sensor. When more detailed analysis life information of each sensor is known, a sensor having the shortest analysis life may be classified as a short-life sensor among sensors classified as long-life sensors. The number of longevity sensors is reduced by repeating the process in step S122.

  In step S223, a large dedicated memory pool is allocated to each long-lived high-frequency sensor. That is, each sensor data receiving means 11 associated with a long-lived and high-frequency sensor and memory management means 12 associated with a large-sized memory pool and functioning as dedicated memory management means 12 One-to-one correspondence. For example, the correspondence relationship optimization unit 15 sequentially associates the memory management unit 12 with each sensor data reception unit 11 associated with the high-frequency sensor, and sets the memory management unit 12 to exclusive control off. The mode may be set.

  Next, the correspondence relationship optimization unit 15 allocates a small-sized dedicated memory pool to each short-lived high-frequency sensor. That is, each sensor data receiving means 11 associated with a short-lived and high-frequency sensor, and a memory management means 12 associated with a small-sized memory pool and functioning as a dedicated memory management means One-to-one correspondence. If there are not enough small-sized memory pools, large-sized memory pools may be allocated. For example, the correspondence relationship optimization unit 15 sequentially associates the memory management unit 12 associated with the small-sized memory pool with each sensor data reception unit 11 associated with the low-frequency sensor, The memory management unit 12 may be set to the exclusive control off mode.

  Finally, the correspondence relationship optimization unit 15 evenly allocates the remaining memory pool to the low-frequency sensors (step S224). If a large-sized memory pool remains, a large-sized memory pool may be allocated as much as possible to a combination including a long-lived low-frequency sensor.

  Next, the operation of this embodiment will be described. FIG. 20 is a flowchart showing an operation example of the data stream analysis system in the present embodiment. In FIG. 20, the same operations as those in the second embodiment shown in FIG. 12 are denoted by the same step numbers and description thereof is omitted. In FIG. 20, the dashed arrow means that the original step of the arrow affects the step ahead of the arrow asynchronously.

  In the present embodiment, in step S16 (sensor data memory storage process) of the main flow, the average analysis lifetime monitoring unit 126 adds a process of storing the acquisition date and time in the header area of the acquired memory block. In step S18 (memory book release process) of the main flow, the average analysis lifetime monitoring unit 126 adds a process of acquiring the acquisition date and the release date stored in the header area of the memory block to be released. Note that the analysis life of the sensor data may be obtained.

  After step S18, the average analysis life monitoring unit 126 obtains the analysis life of the sensor data and, if necessary, obtains the average analysis life for each type of sensor data (step S41).

  In the present embodiment, after the sub-flow step S22 (sensor classification process), the process of setting the monitoring result (average analysis life for each type of sensor data) by the average analysis life monitoring means 126 in the sensor attribute management means 16 (Step S42). For example, the sensor attribute management means 16 acquires the average analysis life for each type of sensor data calculated by the average analysis life monitoring means 126 and updates the attribute information related to the analysis life of the sensor attribute management means 16. Also good. Corresponding relationship optimization means 15 receives the fact that the sensor attribute has been updated, and optimizes the correspondence between sensor data receiving means 11 and memory management means 12 (step S43). The correspondence optimization unit 15 optimizes the correspondence between the sensor data reception unit 11 and the memory management unit 12 in consideration of the data generation frequency and the analysis lifetime.

  In the present embodiment, a decrease in memory efficiency due to the distribution of the memory management unit 12 can be reduced. The reason is that memory efficiency can be improved by performing memory allocation using memory pools of different sizes in consideration of not only the frequency of occurrence of each sensor data type but also the average analysis lifetime. Note that this embodiment can be implemented, for example, as a mode in which the average analysis life monitoring unit 126 is added to the third embodiment.

  In addition, each component in each embodiment of the present invention can be realized not only by hardware but also by a computer and a program. The program is provided by being recorded on a computer-readable functional medium such as a magnetic disk or a semiconductor memory, and is read by the computer when the computer is started up, etc. To function as.

  In the above embodiment, a data storage means in which memory areas for storing data for analysis are allocated as a plurality of memory pools (for example, each memory pool 121), and any one of the plurality of memory pools. A plurality of memory management means (for example, each memory management means 12) that correspond to a memory pool on a one-to-one basis and perform a procedure for one associated memory pool, and are independent for each type of data that has been divided in advance. When the data to be analyzed is generated, the correspondence between the one or more memory storage means for storing the data in the memory area, the memory storage means (for example, each memory storage means 11), and the memory management means And a correspondence relationship storage means (for example, correspondence relationship storage means 13) for storing information indicating the attachment, and the plurality of memory management means include exclusive control. When the memory storage means generates data to be analyzed, it is divided into a shared memory management means that performs a procedure for the memory pool after being performed and a dedicated memory management means that performs a procedure for the memory pool without performing exclusive control. A configuration of a data stream analysis system that stores the data in a memory area secured from one of a plurality of memory pools using a memory management unit associated with the memory storage unit is shown.

  In the above embodiment, the configuration of the data stream analysis system using the type of data classified based on the input method of data to be analyzed, the generation source or the generation factor, or a combination thereof is shown as the type of data. Yes.

  In the above embodiment, the memory storage means and the memory are based on the frequency of occurrence of data by type, the number of memory management means functioning as shared memory management means, and the number of memory management means functioning as dedicated memory management means. An optimization unit (for example, a correspondence relationship optimization unit 15) that optimizes the correspondence with the management unit is provided, and the correspondence relationship storage unit associates the data type optimized by the optimization unit with the memory management unit. The configuration of a data stream analysis system that stores information indicating attachment is shown.

  In the above-described embodiment, the optimization unit has a one-to-one correspondence between the memory management unit that functions as a dedicated memory management unit and the memory storage unit that has a data generation frequency higher than a predetermined threshold. After the allocation, the configuration of the data stream analysis system that optimizes the correspondence between the memory storage means and the memory management means by equally allocating the memory management means functioning as the shared memory management means to the remaining memory storage means. It is shown.

  Further, in the above embodiment, the number of memory storage means whose processing means is a type whose data generation frequency is higher than a predetermined threshold with respect to the number of memory management means that the optimization means functions as the dedicated memory management means. If there is a large amount of memory, a memory management unit that functions as a shared memory management unit is assigned to a part or all of the memory storage unit to optimize the correspondence between the memory storage unit and the memory management unit. The configuration of the data stream analysis system is shown.

  Further, in the above embodiment, the number of memory storage means whose processing means is a type whose data generation frequency is higher than a predetermined threshold with respect to the number of memory management means that the optimization means functions as the dedicated memory management means. If there is a large amount of memory, a memory management unit that functions as a shared memory management unit is assigned to a part or all of the memory storage unit to optimize the correspondence between the memory storage unit and the memory management unit. The configuration of the data stream analysis system is shown.

  Further, the above embodiment includes a frequency measurement unit (for example, usage status monitoring unit 125) that measures a memory allocation frequency, which is a frequency at which a memory area is allocated, for each type of data. A configuration of a data stream analysis system is shown that optimizes the correspondence between the memory storage means and the memory management means, with the memory allocation frequency for each type measured by the measurement means as the occurrence frequency of data for each type.

  In the above embodiment, the data generation frequency predicting unit (for example, the data generation number predicting unit 19) that predicts the data generation frequency for each future type based on the memory allocation frequency for each type measured by the frequency measuring unit. And the optimization unit has a configuration of a data stream analysis system that updates the correspondence between the memory storage unit and the memory management unit based on the generation frequency of data for each type predicted by the data generation frequency prediction unit. It is shown.

  Further, the above embodiment is a data stream analysis system including a data storage unit in which a memory area for storing data to be analyzed is divided and allocated to a plurality of memory pools having different memory capacities. Storage means based on the frequency of occurrence of data by type, average analysis time by type, number of memory management means functioning as shared memory management means, and number of memory management means functioning as dedicated memory management means A configuration of a data stream analysis system associating means with memory management means is shown (for example, see Embodiment 4).

  In the above embodiment, when the optimization unit assigns the memory management unit functioning as the dedicated memory management unit or the memory management unit functioning as the shared memory management unit according to the frequency of occurrence of data, the average analysis time The memory storage means and the memory management are allocated by allocating a memory management means associated with a memory pool having a memory capacity as large as possible to a memory storage means whose processing target is longer than a predetermined reference time. A configuration of a data stream analysis system for optimizing the correspondence between means is shown.

  In the above embodiment, the memory allocation period for measuring the average memory allocation period, which is an average value of the period from when the memory area is actually allocated from the memory pool until the memory area is released, for each data type. Measuring means (for example, average analysis life monitoring means 126), and the optimization means sets the average memory securing period for each type measured by the memory securing period measuring means as the average analysis time for each type, and the memory storing means and the memory The configuration of a data stream analysis system that optimizes the correspondence with the management means is shown.

  The present invention is not limited to a data stream analysis system that analyzes continuously generated data, and can be suitably applied to any system that needs to store continuously generated data in a memory.

It is explanatory drawing which shows the structural example of the data stream analysis system in 1st Embodiment. It is a block diagram which shows the more specific structural example of the data stream analysis system by 1st Embodiment. It is explanatory drawing which shows an example of the free state management using an available memory queue. It is explanatory drawing which shows the change of the empty state of the management object memory pool 121 concerning acquisition of a memory block. It is explanatory drawing which shows the change of the empty state of the management object memory pool 121 concerning release of a memory block. It is explanatory drawing which shows the example of the sensor-memory correspondence management means. It is a flowchart which shows the operation example of the data stream analysis system by 1st Embodiment. 4 is a flowchart showing an example of an algorithm for optimization by the correspondence relationship optimization unit 15. It is a block diagram which shows the structural example of the data stream analysis system by 2nd Embodiment. It is explanatory drawing which shows an example of the monitoring result by the usage condition monitoring means. It is explanatory drawing which shows an example of the classification method of a sensor. It is a flowchart which shows the operation example of the data stream analysis system in 2nd Embodiment. It is a block diagram which shows the structural example of the data stream analysis system by 3rd Embodiment. It is explanatory drawing which shows an example of the prediction result by the data generation number prediction means. It is a flowchart which shows the operation example of the data stream analysis system in 3rd Embodiment. It is a block diagram which shows the structural example of the data stream analysis system by 4th Embodiment. It is explanatory drawing which shows an example of the memory pool 121 in 4th Embodiment. It is explanatory drawing which shows an example of the observation result by the average analysis lifetime monitoring means 126. FIG. It is a flowchart which shows the example of the algorithm of the optimization by the corresponding relationship optimization means 15 in 4th Embodiment. It is a flowchart which shows the operation example of the data stream analysis system in 4th Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Computer 11 Memory storage means, sensor data reception means 12 Memory management means 121 Memory pool (managed memory pool)
122 free memory management means 123 memory acquisition / release means 124 access mode storage means 125 usage status monitoring means 126 average analysis life monitoring means 13 correspondence relation storage means, sensor-memory correspondence management means 14 data analysis means 15 correspondence relation optimization means 16 Sensor attribute management means 17 Exclusive control means 18 Occurrence frequency classification means 19 Data generation number prediction means 200 Sensor

Claims (12)

A data stream analysis system for analyzing continuously generated data,
Data storage means in which memory areas for storing the data for analysis are allocated as a plurality of memory pools;
A plurality of memory management means that correspond one-to-one with any one of the plurality of memory pools and collectively perform procedures for the associated one memory pool;
One or more memory storage means that operate independently for each type of pre-divided data and store the data in the memory area when data to be analyzed is generated;
Correspondence storage means for storing information indicating the correspondence between the memory storage means and the memory management means ;
Optimizing means for optimizing the correspondence between the memory storage means and the memory management means ,
The plurality of memory management means include
Shared memory management means for performing a procedure for the memory pool after performing exclusive control;
It is divided into dedicated memory management means that performs procedures for the memory pool without performing exclusive control,
The optimization means includes the memory storage means and the memory based on the occurrence frequency of data by type, the number of memory management means functioning as shared memory management means, and the number of memory management means functioning as dedicated memory management means. Optimize the correspondence with management means,
The correspondence storage means stores information indicating the correspondence between the memory storage means optimized by the optimization means and the memory management means,
When the data to be analyzed is generated, the memory storage unit stores the data in a memory area secured from one of the plurality of memory pools using a memory management unit associated with the memory storage unit. A data stream analysis system characterized by that. The data stream analysis system according to claim 1, wherein the data type is a type classified based on a method of inputting data to be analyzed, a generation source or a generation factor, or a combination thereof. The optimization means allocates one-to-one memory management means functioning as dedicated memory management means to the memory storage means for which the data generation frequency is higher than a predetermined threshold, and then stores the remaining memory. Optimize the correspondence between the memory storage means and the memory management means by equally allocating the memory management means functioning as the shared memory management means to the means
The data stream analysis system according to claim 1 or 2 . If the number of memory storage means that process data having a frequency higher than a predetermined threshold is larger than the number of memory management means that function as dedicated memory management means, the optimization means By allocating a memory management unit that functions as a shared memory management unit to a part or all of the storage unit, the correspondence between the memory storage unit and the memory management unit is optimized.
The data stream analysis system according to any one of claims 1 to 3 . For each type of data, it has a frequency measurement means for measuring the memory allocation frequency, which is the frequency with which the memory area is allocated,
Optimization means, claim from claim 1 for optimizing the correspondence between the frequency and the occurrence frequency of the data by type by type of memory allocation frequency measured by the measuring means, memory storage means and a memory management unit data stream analysis system according to any one of the four. Based on the memory allocation frequency by type measured by the frequency measurement unit, the data generation frequency prediction unit for predicting the data generation frequency by type in the future,
Optimization means, the data occurrence frequency based on the occurrence frequency of the data by the type predicted by the prediction means, claim 1 for updating the correspondence between the memory storage means and memory management unit of claim 4 The data stream analysis system according to any one of the above. A memory region for storing data to be analyzed is a data stream analysis system provided with data storage means that is divided and allocated to a plurality of memory pools having different memory capacities,
The data stream analysis system according to any one of claims 1 to 6 , wherein the optimization unit further associates the memory storage unit with the memory management unit based on an average analysis time for each type. The optimization unit is a type in which an average analysis time is longer than a predetermined reference time when allocating a memory management unit functioning as a dedicated memory management unit or a memory management unit functioning as a shared memory management unit according to the occurrence frequency of data. By optimizing the correspondence between the memory storage means and the memory management means by assigning the memory management means associated with the memory pool having the largest possible memory capacity to the memory storage means that are subject to processing The data stream analysis system according to claim 7 . For each type of data, a memory securing period measuring means for measuring an average memory securing period, which is an average value of a period from when a memory area is actually secured from the memory pool to when the memory area is released,
Optimization means according to claim 8 in which the average memory allocation period different type measured by the memory allocation period measuring means as an average analysis time different kind, to optimize the correspondence between the memory storage means and memory management unit Data stream analysis system. An information processing apparatus used for analyzing continuously generated data,
Data storage means in which memory areas for storing the data for analysis are allocated as a plurality of memory pools;
A plurality of memory management means that correspond one-to-one with any one of the plurality of memory pools and collectively perform procedures for the associated one memory pool;
One or more memory storage means that operate independently for each type of pre-divided data and store the data in the memory area when data to be analyzed is generated;
Correspondence storage means for storing information indicating the correspondence between the memory storage means and the memory management means ;
Optimizing means for optimizing the correspondence between the memory storage means and the memory management means ,
The plurality of memory management means include
Shared memory management means for performing a procedure for the memory pool after performing exclusive control;
It is divided into dedicated memory management means that performs procedures for the memory pool without performing exclusive control,
The optimization means includes the memory storage means and the memory based on the occurrence frequency of data by type, the number of memory management means functioning as shared memory management means, and the number of memory management means functioning as dedicated memory management means. Optimize the correspondence with management means,
The correspondence storage means stores information indicating the correspondence between the memory storage means optimized by the optimization means and the memory management means,
When the data to be analyzed is generated, the memory storage unit stores the data in a memory area secured from one of the plurality of memory pools using a memory management unit associated with the memory storage unit. An information processing apparatus characterized by that. A memory area for storing continuously generated data is allocated to a plurality of memory pools in a one-to-one correspondence with one of the plurality of memory pools. A memory storage method applied to an information processing apparatus comprising a plurality of memory management means that collectively perform a procedure for one associated memory pool,
After dividing the plurality of memory management means into shared memory management means for performing a procedure for the memory pool after performing exclusive control and dedicated memory management means for performing a procedure for the memory pool without performing exclusive control, Based on the frequency of occurrence of data by type, the number of memory management means functioning as shared memory management means, and the number of memory management means functioning as dedicated memory management means, it operates independently for each type of data divided in advance. optimize correspondence between memory storage means and said memory management unit,
Storing information indicating the correspondence between the optimized memory storage means and the memory management means ;
It said memory storage means, upon receiving the data those said memory storage means is processed, storing the data in the memory area secured by using the memory management means are associated with the memory storage means A memory storage method characterized by the above. A program for storing a memory applied to an information processing apparatus that analyzes continuously generated data,
A memory area for storing continuously generated data is divided into a plurality of memory pools, and a computer having storage means is allocated.
Memory storage processing that operates independently for each type of pre-divided data ,
A plurality of memory management processes which are associated with any one of the plurality of memory pools on a one-to-one basis and collectively perform a procedure for the associated memory pool ;
The plurality of memory management processes are divided into a shared memory management process for performing a procedure for a memory pool after performing exclusive control, and a dedicated memory management process for performing a procedure for a memory pool without performing exclusive control. Based on the frequency of occurrence of data by type, the number of memory management means functioning as shared memory management means, and the number of memory management means functioning as dedicated memory management means, it operates independently for each type of data divided in advance. An optimization process for optimizing the correspondence between the memory storage process and the memory management process, and
Executing a process of storing information indicating the correspondence between the optimized memory storage process and the memory management process ;
In the memory storage process, when the type of data targeted by the memory storage process is received, among the memory management processes corresponding to any one of the plurality of memory pools, the memory storage process Store the data in the memory area secured using the associated memory management process;
In the memory management process, according to the setting, one memory pool is made to perform a procedure for one memory pool associated with it after performing exclusive control or without performing exclusive control. A program for storing memory to perform procedures for.

Images