JP6457747B2 - Framework providing apparatus for processing large-capacity sequentially collected data and data processing method thereof - Google Patents

Description

  The present invention relates to a system for processing a large amount of data. More specifically, the present invention relates to a function for collecting, storing, processing, and managing various types of data to be processed based on a user definition. The present invention relates to a framework providing apparatus and a data processing method therefor for providing a configuration flexibility and a management function capable of efficiently processing data.

  Recently, with the spread of smartphones, digital data used in collection, processing, storage, management, and use in information systems has been rapidly increasing due to the expansion of information technology based on the development of the Internet of Things technology. As an example, in the pharmaceutical and pharmaceutical distribution industries, we promote identification and management of individual drugs using RFID technology, beyond drug management using barcodes to prevent illegal distribution of counterfeit drugs and counterfeit drugs. However, the amount of data generated due to various business processes during the manufacturing and distribution process of pharmaceuticals is expected to increase explosively.

  In addition, when the Internet of Things technology spreads, a large amount of sensor data will be generated, and various services that use this data are expected to appear. The diversity of sensor data, the increase in new sensor devices, and user needs In an environment where services are frequently changed due to changes in the network, there is a problem that it is difficult to apply one batch processing method.

  Various technologies for efficiently collecting, storing, and processing large amounts of data have emerged from the background described above. However, in response to changes in user needs and the diversity of data, it has been adapted to user definitions. Data collection, storage, and processing functions can be flexibly configured, and a method for efficiently managing a large amount of data generated from a plurality of data sources in terms of data scale is required. In particular, in an environment that is subordinated to a specific industry or business area, is sustained for a long time, is generated and accumulated in large numbers, and many stakeholders have to reflect different needs for the data, A framework to effectively support this is required as a particularly important factor.

  In order to solve the above-mentioned problems of the prior art, the present invention can be made variable by configuring functions such as user-defined collection, storage, processing, management, and search for various types of data. It is an object of the present invention to provide a framework providing apparatus and a data processing method thereof that can flexibly cope with user needs and arbitrary data changes.

  Another object of the present invention is to store and manage a large amount of data in a distributed store so as not to be biased to a specific area in an environment in which various types of data and an increase rate of the data cannot be predicted. It is possible to provide a framework providing apparatus and a data processing method thereof that can satisfy a service request even if an error occurs.

  The object of the present invention is not limited to the above-mentioned object, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

  A data processing method according to one aspect of the present invention for achieving the above-described object of the present invention is a data processing method in a framework providing apparatus for processing a large amount of sequentially collected data. Pre-processing user-defined infrastructure for the generated source data, storing the source data in a distributed data store based on a user-defined storage rule, and based on a user-defined data processing rule Processing the source data or the storage information of the source data to generate processed data, registering a data distribution rule for distributing and storing the processed data in the distributed data store, and Based on the data distribution rule, the processed data is distributed to the distributed data store for storage. With the method comprising the steps of, a.

  The data processing method according to one aspect of the present invention refers to a storage rule defined by the user or the data distribution rule when there is a user search in the source data or the processed data stored in the distributed data store. And transferring the source data or the processed data to a user terminal.

  In one embodiment, the step of pre-processing the user-defined base includes generating a data collection process based on an external system linkage function registered via a user interface and a data pre-processing function.

  In one embodiment, storing in the distributed data store includes generating a source data storage process based on user-defined source data storage rules registered via a user interface.

  In one embodiment, the processing data generating step generates at least one source data processing process based on a user-defined infrastructure data processing function registered through a user interface and the processing data storage rule. Has steps.

  In one embodiment, the step of registering the data distribution rule includes a key value generation rule capable of storing the processed data evenly when the distributed data store is a key-value distributed data store. A step of registering.

  A data processing method according to another embodiment of the present invention includes a step of generating a data collection process based on an external system linkage function and a data pre-processing function registered via a user interface, and via a user interface. Generating a source data storage process based on registered user-defined source data storage rules, a user-defined infrastructure data processing function registered via a user interface, and at least based on the processed data storage rules Creating one or more source data processing processes; registering data distribution rules for distributing and storing the processed data processes in the distributed data store; and processing data distribution processes based on the data distribution rules Generating and each of the above Comprising generating an integrated process for processing a user-defined base data by the relationship set between the processes.

  Meanwhile, a framework providing apparatus according to another aspect of the present invention for achieving the object of the present invention is a non-volatile storage in which program code (code) for providing a framework for processing large-capacity sequentially collected data is stored. A memory and at least one processor for executing the program code;

  Here, the framework distributes the source data collected from an external data providing apparatus based on a data collection module that performs user-defined infrastructure preprocessing and a storage rule that defines the source data by the user. A data storage module for storing data in a type data store, a data processing module for processing the source data or storage information of the source data based on a data processing rule defined by a user to generate processed data, and the processed data A data management policy module for registering data distribution rules to be distributed and stored in a distributed data store is provided.

  In one embodiment, the data collection module provides a user interface capable of registering a user-defined infrastructure preprocessing function including an external system linkage function and data analysis.

  In one embodiment, the data storage module provides a user interface capable of registering user-defined source data storage rules.

  In one embodiment, the data storage module registers the data processing module when necessary data extraction work or processing to the source data is necessary from the source data. Based on the defined data processing rule, the source data is transferred to the data processing module, or the storage information of the source data is notified.

  In one embodiment, the data processing module provides a user interface capable of registering a data processing rule defined by the user and the processing data storage rule.

  In one embodiment, the data processing module processes the source data or the stored information of the source data based on a data processing rule defined by a user in an immediate manner or periodically.

  In one embodiment, the data management policy module provides a user interface for registering and retrieving the data distribution rule, and the data distribution rule when the distributed data store is a key-value distributed data store. Is a key value generation rule capable of storing the processed data evenly.

  According to the present invention, the collection, storage, processing, search, and management functions can be defined by the user for various types of data that are generated and collected in large quantities according to the business sequence involving multiple stakeholders. By providing a framework that can be configured on the basis of additional storage and processing capacity as needed to meet arbitrarily changing and increasing user needs and data Respond flexibly and efficiently, and minimize costs. Especially in environments where data is expected to grow exponentially over time, it provides a highly scalable and highly available framework to efficiently manage large volumes of data and Can provide a satisfying service.

It is a figure which shows the structure of the framework of the framework provision apparatus for processing the large-capacity sequential collection data based on embodiment of this invention. It is a figure which shows an example of the system using the framework provision apparatus for processing the large-capacity sequential collection data based on embodiment of this invention. It is a figure which shows the other example of the system using the framework provision apparatus for processing the large-capacity sequential collection data based on embodiment of this invention. It is a figure which shows an example which expands the data storage space of a distributed store based on embodiment of this invention. It is a figure for demonstrating the storage system of the user-defined basic | foundation data in a key value distributed type store based on embodiment of this invention. It is a figure for demonstrating the storage system of the user-defined basic | foundation data in a key value distributed type store based on embodiment of this invention. It is a figure for comparing and explaining the situation where data was distributed and stored in a key / value distributed store. It is a figure for comparing and explaining the situation where data was distributed and stored in a key / value distributed store. It is a figure which shows the data processing method with the framework provision apparatus which concerns on embodiment of this invention.

BEST MODE FOR CARRYING OUT THE INVENTION

  Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described in detail below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various forms. The embodiments merely provide a complete disclosure of the present invention. The present invention is only defined by the scope of the claims, and is provided to fully inform those skilled in the art of the technical field to which the invention pertains. Note that the terms used in this specification are for describing the embodiments and do not limit the present invention. In this specification, the singular includes the plural unless specifically stated otherwise.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, in adding reference numerals to the components of each drawing, it should be noted that the same components have the same symbols even if they are displayed on other drawings. Further, in describing the present invention, when it is determined that a specific description of a related known configuration or function obscures the gist of the present invention, a detailed description thereof is omitted.

  A framework providing apparatus according to the present invention includes a non-volatile memory storing program code for providing a framework for processing large-capacity sequentially collected data, and at least one processor for executing the program code. Here, the device that provides the framework may be an external data providing device, a distributed data store, or a server device that plays a role of data collection, processing and / or processing, search, etc. between user terminals.

  In the framework providing apparatus according to the present invention, the execution of the program code can be processed in units of one “task” or “process”, and the execution result of the program code and a large amount of sequentially collected data are processed. A framework is generated. FIG. 1 shows a configuration of a framework provided from a framework providing apparatus for processing large-capacity sequentially collected data according to an embodiment of the present invention.

  Referring to FIG. 1, a large-capacity business sequential collection data processing framework according to an embodiment of the present invention includes a data collection module 101, a data storage module 102, a data processing module 104, a data management policy module 105, and a module configuration manager. 106, and a data service provider 107.

  The data collection module 101 collects data in conjunction with an external data providing system that provides various types of data, and performs pre-processing on a user-defined basis for the collected data (hereinafter, source data). The source data that has undergone the preprocessing process is transferred to the data storage module 102.

  The data collection module 101 provides a user interface capable of registering a user-defined base preprocessing function such as an external data providing system interlocking function and data analysis.

  As a result, even if the linkage method with the external data providing system is changed, the user can change to the external change by changing only the external data providing system linkage function of the data collection module 101 without changing the data storage module 102. Can respond.

  The data collection module 101 operates in the Pull method and the Push method, but the Pull method is a method of collecting data by periodically making a data request to the external data providing system, and the Push method is in a data reception standby state. A method of collecting data transferred from an external data providing system.

  The data storage module 102 provides a function for storing the source of data transferred from the data collection module 101 in the distributed store 103 and an interface for registering source data storage rules based on user definitions. Users can use the data storage module 102 to store, maintain, and manage source data in the distributed store 103 so that they can smoothly provide new services or change existing services in the future. It can be used for error verification of data processing results. Further, the user can register the data processing module 104 in the data storage module 102 when necessary data extraction work or source data processing work is required from the source data transferred to the data storage module 102. When the data processing module 104 registered in the data storage module 102 exists, the data storage module 102 transfers the corresponding source data immediately or stores it in the distributed store 103 according to the data processing method of the data processing module 104. Notify the source data storage information (data provider, data generation time, data storage path, etc.).

  On the other hand, the distributed store 103 provides a function of storing and managing source data received from the data storage module 102 and processed data generated by the data processing module 104. The distributed store 103 is configured by connecting a plurality of information devices capable of storing data directly or connected to a network, and by simply adding a device as necessary, the data storage space and the processing capability can be improved.

  FIG. 4 relates to the step of expanding the storage space in a situation where the data storage space of the distributed store 401 constituting the framework described in the present invention is insufficient. FIG. 4a shows an environment in which a distributed store 401 composed of a plurality of distributed nodes (402-1, 402-2,..., 402-n) is constructed. FIG. 4b shows an embodiment in which the data storage space is expanded by adding a new distributed node (402-n + 1) to the distributed store 401 in a situation where the amount of data increases rapidly.

  The data processing module 104 shown in FIG. 1 processes the data transferred or notified from the data storage module 102 based on the data processing rules defined by the user, and the processed data generated as a result of the processing is distributed. A function of storing in the store 103 is provided.

  The data processing module 104 provides a user interface capable of registering user-defined infrastructure data processing functions and processing data storage rules. The user-defined infrastructure data processing rules registered in the data processing module 104 can operate in a periodic manner or an immediate manner according to user settings.

  Here, the method performed periodically performs the user-defined infrastructure data processing function at regular time intervals, and the method performed immediately performs the user-defined infrastructure data processing function immediately after the data is transferred or notified.

  As a result of performing the user-defined infrastructure data processing function, the data processing module 104 determines that data is concentrated and stored in a specific area of the distributed store 103 when storing the generated processed data in the distributed store 103. In order to prevent this, the processing data distribution rule information provided from the data management policy module 105 is used to store the processing data. Hereinafter, the data management policy module 105 will be described in detail.

  The data management policy module 105 prevents the performance of the distributed store 103 from being deteriorated by storing the processed data generated from the data processing module 104 in a specific area in the distributed store 103. A user interface capable of registering and searching for data distribution rules for storing processed data evenly in all areas of the distributed store 103 is provided. In the case of the key-value distributed store among the embodiments of the distributed store 103, the data is distributed according to the key value of the data. By registering with, the performance degradation of the distributed store 103 can be prevented.

  Hereinafter, an example of a method for storing data based on the key value generation rule in the key-value distributed type store will be described with reference to FIG. FIG. 5 is a diagram for explaining a storage method of user-defined infrastructure data in a key / value distributed store according to an embodiment of the present invention.

  The table 501 in FIG. 5A is an example in which the distribution event key value is configured in the order of occurrence time and distribution event ID, and the recently generated distribution event is stored in the end portion of the table. A table 502 in FIG. 5B is an example in which the key value of the distribution event is configured in the order of the distribution event ID and the generation time, and the distribution event is collected and stored in units of distribution event ID.

  FIG. 6 is a comparison of the situation where distribution event data is distributed and stored in the key-value distributed store 601 using the example of the table 501 in FIG. 5a and the table 502 in FIG. 5b, and the key-value distributed type. The store 601 includes a plurality of distributed nodes 602, and a large amount of distribution event data is distributed and stored in the data storage areas 603 of the plurality of distributed nodes 602.

  In the case of FIG. 6 a, distribution event data that has recently occurred is stored in a specific data area 603, so that the load on the distribution event storage work is concentrated on a specific distributed node 602, and the key-value distributed store 601. On the other hand, in the case of FIG. 6B, the distribution event data that has recently occurred is distributed to and stored in a plurality of distributed nodes 602, thereby distributing the load on the distribution event storage work. Can do.

  On the other hand, the module configuration manager 106 shown in FIG. 1 sets user-defined infrastructure data processing steps by setting the relationship among the data collection module 101, the data store module 102, the data processing module 104, and the data management policy module 105. Provide functions that can be created, searched, and managed.

  The data service provider 107 provides a data service using the data stored in the distributed store 103 as a result of processing the user-defined infrastructure data generated by the module configuration manager 106.

  For example, when providing a data service, the data service provider 107 uses the data distribution rule information registered in the data management policy module 105 to provide the data service for efficient retrieval of necessary data.

  Hereinafter, an example of a system using a framework providing apparatus for processing large-capacity sequentially collected data according to an embodiment of the present invention will be described with reference to FIG.

  FIG. 2 is a conceptual diagram showing an example of a system using the framework providing apparatus described in the present invention. From a sensor information search service using sensor data generated from the sensor network 251 and a distribution / distribution network 252. The present invention relates to a distribution information search service that utilizes generated distribution event data.

  First, a user who intends to provide a sensor information search service using the framework shown in FIG. 1 can generate a sensor data processing process 210 and a sensor information search service 220.

  The user uses the framework to generate a sensor data collection module 211, a sensor data storage module 212, a sensor data processing module 213, a time series pattern extraction module 214, a sensor data management policy module 215, and a time series pattern management policy module 216. Then, the relationship between the modules generated using the module configuration management 106 of FIG. 1 can be set to configure the sensor data processing process 210.

  On the other hand, the “module” in FIG. 2 can be reused as an intermediate program generated by the user through the framework of FIG. 1 or another program that performs a specific function as an intermediate process. Means that The final process such as the sensor data processing process or the distribution event processing process is generated by setting the relationship to the plurality of “module” sets.

  Further, referring back to FIG. 2, the user uses the user interface that can register the external data providing system interlocking function and the data preprocessing function provided from the data collection module 101 of FIG. By registering the processing function, the sensor data collection module 211 can be generated.

  Further, the user can generate the sensor data storage module 212 by registering the sensor data storage rule using the source data storage rule registration user interface provided from the data storage module 102 of FIG.

  In addition, the user registers the sensor data processing function and the sensor data storage rule using the user interface that can register the user-defined infrastructure data processing function and the processing data storage rule provided from the data processing module 104 of FIG. Thus, the time series pattern extraction module 214 can be generated by generating the sensor data processing module 213 and registering the time series pattern extraction function and the time series pattern storage rule.

  Further, in the case of the sensor data processing function, the user needs to immediately process the sensor data collected from the sensor network, so the sensor data processing module 213 is set to a method of performing immediately, and in the case of the time series pattern extraction function Since this is an operation that requires processing for sensor data collected for a certain period of time, the time-series pattern extraction module 214 can be set to perform periodically.

  Also, the user registers the sensor data distribution rule using the data distribution rule registration user interface provided from the data management policy module 105 of FIG. By registering the pattern data distribution rule, the time series pattern management policy module 216 can be generated.

  Further, the user generates a sensor data statistical search service 221 and a time series pattern search service 222 by using the data management policy module 105 and the distributed store 103 access function provided by the data service provider 107 of FIG. can do.

  The sensor data statistical search service 221 uses the sensor data distribution rule information registered in the sensor data management policy module 215 in response to the sensor data statistical search request to determine the storage area of the sensor data that satisfies the search condition, Sensor statistical search requests can be efficiently processed by statistical processing on sensor data in the area.

  The time series pattern search service 222 uses a time series pattern data distribution rule information registered in the time series pattern management policy module 216 in response to a time series pattern search request, and a time series pattern storage area that satisfies the corresponding search conditions. And a time series pattern search request can be processed efficiently.

  Further, when the user needs to collect new types of sensor data in the future, the sensor data processing process is performed by changing the sensor data preprocessing function of the sensor data collection module 211 and the sensor data storage rule of the sensor data storage module 212. If a new information retrieval service that utilizes sensor data can be modified and expanded, add a data processing module, data management policy, and data service provider for the service to the framework. Thus, the sensor information search service 220 can be extended.

  On the other hand, a user who intends to provide a distribution information search service using the framework shown in FIG. 1 can generate a distribution event processing process 230 and a distribution information search service 240.

  The user generates a distribution event collection module 231, a distribution event storage module 232, a distribution event processing module 233, a distribution history extraction module 234, a distribution event management policy module 235, and a distribution history management policy module 236, and manages the module configuration of FIG. The distribution event processing process is configured by setting the relationship of the modules generated using the person 106.

  The user uses the external system linkage function and the data pre-processing function registration interface provided from the data collection module 101 of FIG. 1, and the HTTP server function for collecting distribution events from the distribution / distribution network and the distribution in the XML format. By registering the preprocessing function for the event message, the distribution event collection module 231 can be generated.

  Further, the user can generate the distribution event storage module 232 by registering the distribution event storage rule using the source data storage rule registration interface provided from the data storage module 102 of FIG.

  Further, the user registers the distribution event processing function and the distribution event storage rule by using the user-defined infrastructure data processing function and the processing data storage rule registration interface provided from the data processing module 104 of FIG. By registering the event processing module 233, the distribution history extraction function and the distribution history storage rule, the distribution history extraction module 234 can be generated. In addition, since the distribution event processing function and the distribution history extraction function are required to be processed in real time, the user sets the data processing method of the distribution event processing module 233 and the distribution history extraction module 234 to a method for performing immediately.

  The user registers the distribution event management rule module 235 and the distribution history distribution rule by registering the distribution event distribution rule using the data distribution rule registration interface provided from the data management policy module 105 of FIG. Thus, the distribution history management policy module 236 can be generated.

  Further, the user can generate the distribution event search service 241 and the distribution history search service 242 using the data management policy module 105 and the distributed store 103 access function provided by the data service provider 107 of FIG. . In response to the distribution event search request, the distribution event search service 241 uses the distribution event distribution rule information registered in the distribution event management policy module 235 to determine the storage area of the distribution event that satisfies the corresponding condition. Distribution event search requests can be efficiently processed by distribution event search. The distribution history search service 242 uses the distribution history distribution rule information registered in the distribution history management policy module 236 in response to a distribution history search request for a specific product, and determines an area in which the history data of the corresponding product is stored. , It is possible to quickly respond to the distribution history search request.

  In addition, when the user needs an inventory information search service for products stored in each distribution warehouse in the future, the user creates a data processing module, a data management policy module, and a data service provider for the corresponding service. The distribution event is transferred from the distribution event storage module 232 to provide an inventory information search service.

  On the other hand, the framework providing apparatus according to the present invention can provide a method capable of efficiently processing data in various fields using the framework.

  FIG. 3 illustrates various distribution / distribution data sharing systems 301 that provide functions for collecting, processing, and searching distribution event data generated in the distribution process of goods in the distribution / distribution network 302 using the framework described in the present invention. Embodiments are shown.

  The distribution / distribution data sharing system 301 provides various environments of application systems by providing an environment in which a large number of distribution / distribution participants can share distribution events that occur in the entire distribution process from manufacture to sale of products. Can be built. The drug distribution history tracking system 303 in FIG. 3 prevents an accident in which illegal drugs are distributed by monitoring distribution of drugs in real time by providing distribution events for drugs from the distribution / distribution data sharing system 301. can do. The inventory management system 304 in FIG. 3 is provided with the quantity of goods in and out of each physical distribution / distribution base from the physical distribution / distribution data sharing system 301, and an out-of-stock or excess problem occurs by grasping the stock status of each commodity. An inventory management plan can be developed before The expiration date management system 305 of FIG. 3 is provided with a distribution event for food from the distribution / distribution data sharing system 301, and can prevent the problem of harming consumers' health due to distribution of expired food. The illegal transaction tracking system 306 of FIG. 3 is provided with the transaction data of the product from the logistics / distribution data sharing system 301, and can track the status of illegal transactions for the product.

  Hereinafter, a data processing method performed by the framework providing apparatus according to the embodiment of the present invention will be described in detail with reference to FIG. Data processing is performed by intermediate processes created by user definitions, respectively, and the framework provides the user with a user interface that allows the intermediate processes to be created.

  More specifically, the framework providing apparatus according to the embodiment of the present invention generates a data collection process based on the external system linkage function and the data preprocessing function registered via the user interface, A source data storage process is generated based on the source data storage rules defined by the user registered through the user interface.

  In addition, the framework providing device generates at least one source data processing process based on the user-defined infrastructure data processing function registered through the user interface and the processing data storage rule, A data distribution rule to be distributed and stored in the distributed data store is registered, and a processed data distribution process is generated based on the data distribution rule.

  Finally, the framework providing apparatus generates an integrated process for processing the user-defined infrastructure data according to the relationship setting between the processes and processes the data.

  Referring to FIG. 7, first, the data collection module 101 collects data in conjunction with an external data providing system that provides various types of data, and the collected data (hereinafter referred to as source data) is a user. Predefine the definition base. The source data that has undergone the preprocessing process is transferred to the data storage module 102 (S10).

  Then, the data storage module 102 stores the data source transferred from the data collection module 101 in the distributed store 103 (S20). At this time, the data storage module 102 stores, maintains, and manages source data based on user-defined source data storage rules, so that it is necessary to provide a new service or change an existing service in the future. Can also be used for error verification of data processing results.

  Hereinafter, the data processing module 104 processes the data transferred or notified from the data storage module 102 based on the data processing rules defined by the user, and stores the processed data generated as a result of the processing in the distributed store 103. The function to perform is provided (S30).

  On the other hand, the data management policy module 105 prevents the problem that the performance of the distributed store 103 deteriorates due to the processing data generated from the data processing module 104 being stored in a specific area of the distributed store 103. In addition, a user interface capable of registering and searching for data distribution rules for storing processed data evenly in all areas of the distributed store 103 is provided (S40). When the data distribution rule for distributing and storing the processed data is registered through the user interface, the data processing module 104 distributes and stores the processed data in the distributed store 103 based on the data distribution rule (S50). ).

  Thereafter, when there is a search request for the source data or the processed data stored in the distributed store 103 from the data service provider 107, the data service provider 107 performs an efficient search for data necessary for providing the data service. Therefore, the data distribution rule information registered in the data management policy module 105 is utilized to provide a data service to the user (S60).

  On the other hand, the above-described data processing method in the framework providing apparatus according to the present invention can be implemented as a computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording media that store data that can be decoded by a computer system. For example, there are ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage device, and the like. A computer-readable recording medium is distributed in a computer system connected to a computer network, and stored and executed as a code that can be read in a distributed manner.

  Those having ordinary knowledge in the technical field to which the present invention pertains will understand that the present invention can be implemented in other specific forms without changing the technical idea and essential features thereof. Therefore, it should be understood that the above-described embodiment is illustrative in all aspects and not restrictive. The protection scope of the present invention is indicated by the following claims rather than the above detailed description, and all modifications or variations derived from the claims and their equivalents are within the scope of the present invention. It must be interpreted as included.

Claims (13)

In a data processing method in a framework providing apparatus for processing a large amount of sequentially collected data,
Pre-processing user-defined infrastructure for source data collected from an external data provider;
Storing the source data in a distributed data store based on user-defined storage rules;
Processing the source data or stored information of the source data based on a user-defined data processing rule to generate processed data;
Registering a data distribution rule for distributing and storing the processed data in the distributed data store; and distributing and storing the processed data in the distributed data store based on the data distribution rule When,
A data processing method.   When there is a user search in the source data or the processed data stored in the distributed data store, the source data or the processed data is referred to the user terminal by referring to the storage rule or the data distribution rule defined by the user. The data processing method according to claim 1, further comprising a transferring step. The step of pre-processing the user-defined infrastructure includes:
The data processing method according to claim 1, further comprising a step of generating a data collection process based on an external system linkage function and a data preprocessing function registered via a user interface. Storing in the distributed data store comprises:
The data processing method according to claim 1, further comprising the step of generating a source data storage process based on user-defined source data storage rules registered via a user interface. The step of generating the machining data includes:
The data processing method according to claim 1, further comprising: generating at least one source data processing process based on a user-defined infrastructure data processing function registered through a user interface and a data processing rule defined by the user. . The step of registering the data distribution rule includes:
The data processing method according to claim 1, further comprising a step of registering a key value generation rule capable of storing the processed data evenly when the distributed data store is a key-value distributed data store. In an apparatus having a non-volatile memory storing a program code (code) for providing a framework for processing a large amount of sequentially collected data, and at least one processor for executing the program code,
The framework is
A data collection module that performs user-defined infrastructure preprocessing on source data collected from an external data providing device;
A data storage module for storing the source data in a distributed data store based on storage rules defined by a user;
A data processing module that processes the source data or stored information of the source data based on a user-defined data processing rule to generate processed data;
A framework providing apparatus for providing a data management policy module for registering a data distribution rule for distributing and storing the processed data in the distributed data store. The data collection module includes:
8. The framework providing apparatus according to claim 7 , which provides a user interface capable of registering a user-defined infrastructure preprocessing function having an external system linkage function and data analysis. The data storage module includes:
8. The framework providing apparatus according to claim 7 , which provides a user interface capable of registering user-defined source data storage rules. The data storage module includes:
When necessary data extraction work from the source data or processing work to the source data is necessary, the data processing module is registered, and the source data is converted into the source data based on a data processing rule defined by the user in the data processing module. The framework providing apparatus according to claim 7 or 9 , wherein the framework providing apparatus notifies the storage information of the source data by transferring to a data processing module. The data processing module includes:
The framework providing apparatus according to claim 7 , wherein a user interface capable of registering the data processing rule defined by the user and the data distribution rule is provided. The data processing module includes:
12. The framework providing apparatus according to claim 7 or 11 , wherein processing is performed by a method in which the source data or storage information of the source data is immediately or periodically performed based on a data processing rule defined by a user. The data management policy module includes:
A user interface that can register and search the data distribution rule is provided, and when the distributed data store is a key-value distributed data store, the data distribution rule is a key value generation rule that can store the processed data equally. The framework providing device according to claim 7 .

Images