CN105989163A - Data real-time processing method and system - Google Patents

Abstract

The invention discloses a method and system for real-time data processing. The method is applied to a real-time data processing system. The method includes: acquiring source system data change information; and real-time processing the data change information according to a first preset processing strategy Performing the first data processing to form the processed first data; loading the first data into the memory database; based on the data processing requirements, performing the second processing on the first data in the memory database in real time according to the second preset processing strategy Data processing, to obtain data processing results.

Description

Data real-time processing method and system

technical field

The invention relates to information processing technology in the field of information processing, in particular to a method and system for real-time data processing.

Background technique

With the development of social informatization, the value of data (specifically, business operation data) is increasing day by day, and the timeliness requirements for operational information analysis are also increasing. It is required to achieve real-time analysis and real-time decision-making through real-time monitoring of business acceptance data. .

Existing data analysis generally includes two types:

The first method: performing statistical analysis directly on the original business system may occupy a large amount of resources of the business system, which in turn will cause a great impact on the normal operation of the business system, and it is difficult to guarantee the timeliness of analysis.

The second type: build a data warehouse system for analysis; often when the business system is idle, data is extracted from the database table on a daily or monthly basis, stored in the database, and complex analysis can be performed, but this method is based on an independently running database. The lack of real-time performance of data processing leads to varying degrees of lag in problem discovery.

At present, there are some suitable data synchronization methods in the industry, but the existing data synchronization methods in the industry can basically only meet the needs of synchronizing data on the business system to another system, and cannot support complex data operations, resulting in the inability to accept business data. Real-time analysis, when finding problems and making corresponding decisions, there will also be time delays.

Therefore, based on the above, it is an urgent problem to be solved in the prior art to provide a data processing method with fast response speed and low time delay.

Contents of the invention

In view of this, the embodiments of the present invention expect to provide a real-time data processing method and system, so as to at least partially solve the problem of large data processing time delay in the prior art.

In order to achieve the above object, technical solution of the present invention is achieved in that way:

The first aspect of the embodiments of the present invention provides a real-time data processing method, the method is applied to a real-time data processing system, and the method includes:

Obtain source system data change information;

performing first data processing on the data change information in real time according to a first preset processing strategy to form processed first data;

loading the first data into an in-memory database;

Based on the data processing requirement, the second data processing is performed on the first data in the internal memory database in real time according to the second preset processing strategy to obtain a data processing result.

Preferably,

The acquisition of source system data change information includes:

When the database log of the source system changes, acquire the changed database log information;

Analyzing the changed database log information in real time and converting the data format of the changed database log information to form data suitable for data stream processing.

Preferably,

The performing the first data processing on the data change information in real time according to the first preset processing strategy to form the processed first data includes:

Perform data cleaning and data summary processing on the data change information in real time, so as to standardize and reduce the weight of the first data loaded into the memory database.

Preferably,

The real-time data summary processing of the data change information includes:

Carry out light summary processing and/or high summary processing on the data of the data cleaning;

Wherein, the light summary processing includes performing data summary processing at a first time interval;

The high-level summary processing package performs data summary processing at a second time interval;

The first time interval is not greater than the second time interval.

Preferably,

The second data processing is performed on the first data in the internal memory database in real time according to the second preset processing strategy based on the data processing requirements, and the data processing results are obtained, including:

Based on the business requirements of the data, real-time statistics, real-time analysis, real-time monitoring and real-time decision-making are performed on the first data in the memory database to form a processing result.

Preferably,

The method also includes at least one of the following:

Perform abnormal monitoring on the real-time data processing of the data real-time processing system;

performing load balancing processing on the real-time data processing of the data real-time processing system;

Perform route adaptation processing on the real-time data processing of the data real-time processing system.

Preferably,

The load balancing processing of the real-time data processing of the data real-time processing system includes at least one of the following:

performing load balancing among service hosts performing the first data processing;

Perform load balancing among service hosts performing the second data processing.

Preferably,

The routing adaptation processing of the real-time data processing of the data real-time processing system includes:

Establishing a routing adaptation relationship between the service host performing the second data processing and the specified service request.

The second aspect of the embodiment of the present invention provides a real-time data processing system, the system comprising:

The data acquisition module is used to acquire the source system data change information;

The first data processing module is configured to perform first data processing on the data change information in real time according to a first preset processing strategy to form processed first data;

a data storage module, configured to load the first data into the memory database;

The second data processing module is configured to perform second data processing on the first data in the in-memory database in real time according to a second preset processing strategy based on data processing requirements, and obtain a data processing result.

Preferably,

The data acquisition module includes:

The replication submodule is used to obtain the changed database log information when the database log of the source system changes;

The parsing sub-module is used for parsing the changed database log information in real time and converting the data format of the changed database log information to form data suitable for data flow processing.

Preferably,

The first data processing module is configured to perform data cleaning and data summary processing on the data change information in real time, so as to standardize and reduce the weight of the first data loaded into the memory database.

Preferably,

The second data processing module is configured to perform real-time statistics, real-time analysis, real-time monitoring, and real-time decision-making on the first data in the memory database based on the business requirements of the data to form processing results.

Preferably,

The system also includes a system management module;

The system management module is configured to monitor abnormalities in the real-time data processing of the data real-time processing system; and/or perform load balancing processing on the real-time data processing of the data real-time processing system; and/or perform load balancing processing on the data real-time processing system; The data of the real-time processing system is processed in real time for routing adaptation processing. The real-time data processing method described in the embodiment of the present invention is applied to a real-time data processing system independent of a source system, and the real-time data processing system is provided based on the real-time data processing method. First of all, the real-time data processing system is independent of the source system, so it will not occupy the resources of the source system, and thus has little impact on the normal operation of the source system. Secondly, the real-time data processing method and system described in the embodiments of the present invention use an in-memory database to store data. Compared with the prior art that uses a disk to store data, it reduces the number of times data is written to and read from a disk. time, and the method described in this embodiment can use real-time processing stream processing technology to perform the above data processing, obviously the data processing cycle is small, and has the advantages of small delay and fast response.

Description of drawings

Fig. 1 is a schematic flow chart of a data real-time processing method according to an embodiment of the present invention;

Fig. 2 is the structural representation of the data real-time processing system described in the embodiment of the present invention;

FIG. 3 is a schematic flowchart of source system data change information according to an embodiment of the present invention;

4 is a schematic flow chart of the first data processing described in the embodiment of the present invention;

Fig. 5 is a schematic diagram of the system management process described in the embodiment of the present invention.

detailed description

The technical solutions of the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

Method example:

As shown in Figure 1, this embodiment provides a method for real-time data processing, the method is applied in a real-time data processing system, and the method includes:

Step S110: Acquiring source system data change information;

Step S120: Perform first data processing on the data change information in real time according to a first preset processing strategy to form processed first data;

Step S130: loading the first data into the memory database;

Step S140: Based on data processing requirements, perform second data processing on the first data in the internal memory database in real time according to a second preset processing strategy to obtain a data processing result.

The source system may be various types of service systems, such as a service acceptance system, a billing system, and the like.

The real-time data processing method of this embodiment is applied to a system dedicated to data processing independent of the source system, so that during the data processing process, it will obviously not occupy the resources of the source system, thereby avoiding the source system The operation of the system causes interference, which leads to problems such as slow response speed of the source system to execute business requests.

The data change information acquired in step S110 may specifically be data such as business acceptance change log data of the source system.

In step S130, the first data after the first data processing is directly loaded into the memory database. The in-memory database is a database composed of internal memory. When performing the second data processing, the data is directly obtained from the in-memory database, which is much faster than obtaining and reading data from a disk. If data processing is usually carried out in the memory of the service host, the speed of migrating data from one part of the memory to another is very fast, or when performing the first data processing, directly in the memory corresponding to the memory database In this way, even the data migration between memories is omitted, which obviously improves the reduction of system processing steps and response speed.

In a specific implementation, after the first data is loaded into the memory database, the second data processing is performed immediately without time intervals. Obviously, this can be compared to the prior art in which data is stored in the disk. When performing the second data processing, it is necessary to not only save the process of writing data to the disk, but also include the process of reading data from the disk; obviously it is beneficial to improve the real-time performance of data processing and reduce the delay.

In addition, in step S130, memory data is used to store the first data, and some data processing mechanisms are used. When the server is down, it is easier to restore the data in the memory than to restore the disk data. Obviously, compared to the first Data is written on the disk, which can improve data security and system reliability.

Further, the step S110 may include: when the database log of the source system changes, acquiring the changed database log information; and analyzing the changed database log information in real time and converting the changed database log information The data format of the database log information forms data suitable for data stream processing.

Specifically, in step S111, data changes of the source system are mainly captured in real time and converted into a data format recognizable in the first data processing, such as csv data format. The step S110 may include: copying data from the source system, and performing data analysis on the copied data.

Real-time data replication can read the log change data of external database business acceptance through the data replication product GoldenGate, such as data data, work order data, historical data, etc. The GoldenGate is the name of an information processing product that replicates data.

Data parsing may include: after receiving the data output by data replication products such as GoldenGate through the Stream Application implemented in Java, converting and processing the message data accordingly, and outputting a file format that can be recognized in stream processing, such as csv files. The Java is a programming language; the Stream Application is the name of an application. The csv file is a file with a data format of csv.

The step S120 may include: performing data cleaning and data summary processing on the data change information in real time, so as to standardize and reduce the weight of the first data loaded into the memory database.

The data cleaning includes data filtering; the data filtering includes deleting irrelevant data, specifically, deleting data irrelevant to service requests in the data. For example, delete information such as text descriptions in log information.

The data clarity also includes converting data into uniform codes and data of the same weight and measure, reducing data conversion and weight and measure conversion in the subsequent second data process, so as to improve subsequent processing speed.

The real-time data summary processing of the data change information includes:

Carry out light summary processing and/or high summary processing on the data of the data cleaning;

Wherein, the light summary processing includes performing data summary processing at a first time interval;

The high-level summary processing package performs data summary processing at a second time interval;

The first time interval is not greater than the second time interval.

Data summary is mainly responsible for summarizing data on the basis of data cleaning, thereby reducing the amount of data loaded into the memory database and improving system performance. This module provides two functions: light summary and high summary. Light summary The aggregated time granularity can be at the second level, and the highly aggregated aggregated time granularity can be at the minute level; data loading mainly provides the function of loading highly aggregated data into the memory database, and the implementation method is mainly by calling the data loading application program of the memory database Programming interface API implementation. Usually, highly aggregated processing logic is more complex than lightly aggregated processing logic.

The step S140 may include:

Based on the business requirements of the data, real-time statistics, real-time analysis, real-time monitoring and real-time decision-making are performed on the first data in the memory database to form a processing result.

The business requirements may include business acceptance requirements, which are one of the above-mentioned data processing requirements.

The following explains real-time statistics, real-time analysis, real-time monitoring, and real-time decision-making in combination with specific application scenarios.

Real-time statistics include real-time business statistics such as business checkout and payment of the source system as a business system, and can perform full-volume, large-scale real-time calculations based on dimensions such as all salespersons and business halls, to meet the statistical needs of operation and maintenance to the greatest extent.

Real-time analysis can include horizontal comparative analysis and notification of various channels and business points within the same marketing period; real-time monitoring can effectively control risks, such as income risk monitoring, electronic channel transaction monitoring, salesperson transaction monitoring, promotion monitoring; through monitoring volatility, sensitive Information access monitoring, etc.

Real-time decision-making can include guiding decision-making and actions based on the analysis results and analysis of real-time analysis. For example, based on the effective analysis of the golden period of sales flow, guidance can be given to business hall resources such as personnel scheduling and other supporting tasks, such as reasonable time points for logistics and distribution. The arrangement provides a basis to improve the efficiency of the store and realize the reduction of staff and efficiency.

The method described in this embodiment is based on the above steps, in order to improve the reliability of the system and further improve the response rate of the real-time data processing system, the method further includes at least one of the following:

Perform abnormal monitoring on the real-time data processing of the data real-time processing system;

performing load balancing processing on the real-time data processing of the data real-time processing system;

Perform route adaptation processing on the real-time data processing of the data real-time processing system.

The abnormality monitoring may include monitoring the data processing in steps S110 to S140, and immediately perform abnormality elimination processing once an abnormality occurs, so as to ensure the timeliness of the data real-time processing system, so as to prevent any abnormality in the data processing link from causing data processing. Timeliness declines.

The load balancing processing of the real-time data processing of the data real-time processing system includes at least one of the following:

performing load balancing among service hosts performing the first data processing;

Perform load balancing among service hosts performing the second data processing.

In this embodiment, the real-time data processing system can be divided into an in-memory database and a processing host for processing data. The processing host can be used to acquire data, perform first data processing and second data processing. The service hosts used for data processing in this system may form host clusters. Each host cluster may include one or more service hosts. The load of each service host may be light or heavy. Through load balancing, data processing on a host with a heavy load can be transferred to a host with a light load for processing, which can improve the timeliness of data processing as a whole.

The routing adaptation processing of the real-time data processing of the data real-time processing system includes:

Establishing a routing adaptation relationship between the service host performing the second data processing and the specified service request.

Routing adaptation processing is usually for the second data processing, which is a pre-set route search process, which can simplify the query operation of data from the memory database in the subsequent second data processing process. By simplifying the data query, it can Improve the data processing rate of the real-time data processing system again, and reduce the delay of data processing.

Device example:

As shown in Figure 2, the present embodiment provides a real-time data processing system, the system includes:

A data acquisition module 110, configured to acquire source system data change information;

The first data processing module 120 is configured to perform first data processing on the data change information in real time according to a first preset processing strategy to form processed first data;

a data storage module 130, configured to load the first data into the memory database;

The second data processing module 140 is configured to perform second data processing on the first data in the internal memory database in real time according to a second preset processing strategy based on data processing requirements, and obtain a data processing result.

The acquisition module described in this embodiment may specifically be various types of communication interfaces, which are connected to the source system and capable of receiving data from the source system. The communication interface may include a wired or wireless communication interface, and the wired communication interface may include a cable interface or an optical cable interface; the wireless interface may include structures such as receiving antennas.

The specific structures of the first data processing module 120, the data storage module 130 and the second data processing module 140 may include a processor and a storage medium; executable codes are stored on the storage medium. The storage medium is connected to the processor through a communication interface inside the user equipment such as a bus. The processor can implement the functions of the processing unit 120 and the generating unit 130 by executing the executable code. The processor may be a processor or a processing chip with information processing functions such as a central processing unit CPU, a microprocessor MCU, a digital signal processor DSP, or a programmable array PLC.

In this embodiment, it may correspond to the service host in the system. The memory database may include memory integrated on each service host in the system.

The real-time data processing system described in this embodiment is a system opposite to the source system, so that when data processing is performed, the system resources of the source system will obviously not be occupied, thereby avoiding adverse effects on the business of the source system . In this embodiment, the first data is loaded in the memory database, which can reduce the number of reads and writes to the disk, thereby improving data processing efficiency and reducing delay.

Further described data acquisition module 110 includes:

The replication submodule is used to obtain the changed database log information when the database log of the source system changes;

The parsing sub-module is used for parsing the changed database log information in real time and converting the data format of the changed database log information to form data suitable for data stream processing.

The copy sub-module is mainly used to copy the data change information from the source database by using replication technologies such as GoldenGate, DataGuard, SharePlex or PAC, such as the change data of the business acceptance request from the database of the source system, such as data data and work order data , historical data, etc., and transmit the obtained information to the parsing submodule. In this embodiment, the data change information may generally be specified change information, and may not be arbitrary change information.

The parsing sub-module is mainly used to receive the information transmitted by the parent-child module by using applications such as Stream Application written in the Java language, and convert and process the information accordingly, and output the files that can be identified in the stream processing, specifically as file in csv file format.

Optionally, the first data processing module 120 is configured to perform data cleaning and data summary processing on the data change information in real time, so as to standardize and reduce the weight of the first data loaded into the memory database.

The first data processing module 120 described in this embodiment can also be called a real-time data processing module, and its main functions include data cleaning, data summarization, and data loading.

For the relevant description of the data cleaning and data summarization, please refer to the method embodiments, specifically, the first data processing module 120 is configured to perform light summary processing and/or high-level summary processing on the data of the data cleaning; Wherein, the light summary processing includes data summary processing at a first time interval; the high summary processing package performs data summary processing at a second time interval; and the first time interval is not greater than the second time interval.

In specific implementation, the data storage module is mainly used to use an in-memory database as a storage medium. On the one hand, it can avoid the impact of large data volume transaction data landing (writing to disk) on the analysis performance during data processing, and guarantees the processing The timeliness of the process; on the other hand, the high-availability mechanism of the memory database storage technology ensures rapid recovery and no data loss when the server is down, ensuring the reliability of the system.

Optionally, the second data processing module 140 is configured to perform real-time statistics, real-time analysis, real-time monitoring, and real-time decision-making on the first data in the memory database based on data business requirements to form processing results.

The second data processing module 140 described in this embodiment can also be called a real-time data application module, which is mainly used on the basis of an in-memory database to construct business scenarios based on business requirements, including real-time statistics, real-time analysis, real-time monitoring and real-time decision-making, etc. deal with. Relevant definitions of the real-time statistics, real-time analysis, real-time monitoring and real-time decision-making can be found in the method embodiments, and will not be repeated here.

In addition, as shown in Figure 2, the system described in this embodiment also includes a system management module;

The system management module 150 is configured to monitor abnormalities in the real-time data processing of the data real-time processing system; and/or perform load balancing processing on the real-time data processing of the data real-time processing system; and/or perform load balancing processing on the real-time data processing system; The real-time data processing of the data processing system performs routing adaptation processing.

The specific structure of the system management module 150 here may also correspond to the service host or a processor or processing chip in the service host. For the structure of the processor or the processing chip, reference may be made to the foregoing part, which will not be repeated here.

The system management module 150 may specifically be configured to perform load balancing among service hosts performing the first data processing; and/or perform load balancing among service hosts performing the second data processing.

The system management module 150 may also be configured to establish a routing adaptation relationship between the service host performing the second data processing and the specified service request.

Based on the above, this embodiment provides a real-time data processing system, which provides hardware support for the above-mentioned real-time data processing method, and is obviously used for data analysis and decision-making of the source system, without occupying resources of the source system, and for the normal operation of the source system. It has the advantages of small impact on operation and small delay in data analysis and decision-making.

Several specific application examples are provided below in conjunction with the above-mentioned embodiments:

Example one:

As shown in Figure 3, this example provides a specific method for obtaining the data change information, as follows:

Step 101: The database log of external business acceptance changes. This means that the database log of the above-mentioned source system has changed.

Step 102: capture by the data replication module. The data replication module here is the above-mentioned replication sub-module. The capture here is to obtain data change information, specifically, use the data replication product GoldenGate to capture the Redo Log of the database, obtain the changed business acceptance data, and capture the change data of the source database, track queues, data pump distribution, route compression encryption and In the process of delivering the target, the log information of the change of the business acceptance database is captured in real time with minimal system expenditure.

Step 103: Pump out the trail file, this step is automatically implemented by the data replication product.

Step 104: The log parsing module reads the trail file (trail file) pumped by the data replication product such as GoldenGate through the Stream Application implemented by Java, and performs corresponding parsing and processing on the trail file.

Step 105: Generate a data file in csv format to provide data input for the real-time processing module. The csv format is the above-mentioned data format suitable for data flow processing. The data stream processing is a technology for data processing. For details on how to implement data stream processing, please refer to the prior art, which will not be expanded here. In short, data stream processing has the advantages of small data processing delay and high data processing efficiency. high merit.

Step 106: The real-time data acquisition process ends, and the real-time processing module is delivered. The real-time processing module here is equivalent to the first data processing module described in the device embodiment.

Example two:

As shown in Figure 4, this example provides a specific method for the first data processing, which may include:

Step 1: The log parsing module in the real-time data acquisition module incrementally generates data files in csv format by parsing and processing the change logs of the source system database. The log parsing module here is detailed in the parsing sub-module above.

Step 2: The data cleaning module queries data cleaning, summarizing, and loading rule bases to obtain data cleaning rules.

Step 3: Data cleaning, summarizing, loading rule base, and returning the cleaning rules to be queried in the data cleaning module. The cleaning rule here is a part of the first preset processing strategy.

Step 4: According to the data cleaning rules, check the consistency of the csv business acceptance data incrementally generated by the log parsing module, process invalid and missing values, and clean the data. Stream processing technologies such as StreamBase can be used for data cleaning.

Step 5: The data aggregation module queries the data cleaning, aggregation, and loading rule base to obtain data aggregation rules, and the data aggregation rules are also one of the components of the first preset processing strategy.

Step 6: data cleaning, summarizing, loading rule base, and returning the summarizing rules to be queried in the data summarizing module.

Step 7: For the cleaned data, generate summary-level data according to the data summary rules, which can be divided into light summary-level data and highly summary-level data.

Step 8: The data loading module queries the data cleaning, summarizing, and loading rule base to obtain data loading rules. The data loading rule is also one of the components of the first preset processing strategy, and is used for loading data into the memory database.

Step 9: Data cleaning, summarization, and loading of the rule base, returning the loading rules to be queried in the data loading module.

Step 10: According to the data loading rules, load the data into the loading memory database in real time

Example three:

As shown in Figure 5, this example provides a system management method, which may include:

Step s1: Receive a data service request from a data service requester. Specifically, if the real-time monitoring interface is used to monitor and initiate a data service request, multiple data service requesters can send multiple data request services concurrently.

Step s2: Use a load balancer cluster to receive high-concurrency data service requests, perform service balancing on high-concurrency data request services, and hosts in the cluster all back up each other, realize high-concurrency data service request processing, and speed up service request response speed . The mutual backup mode of the hosts in the cluster can be two-by-two, or one backup for multiple.

Step s3: Distributing the service request after the route adaptation calculates the mod (the said determination of the mod is the calculation of the modulus). Specifically, after balancing the services with high concurrent requests through the load balancer cluster, the balanced request services are obtained through the mod method. The primary load balancer uses the primary data routing Mod (such as routing value 0-3, the notification rule configuration is implemented, the same below), and the standby data routing Mod (such as routing value 4-7, the same below). The standby load balancer distributes the main data routing Mod (4-7) and the standby data routing Mod (0-3) to the corresponding service hosts in the service and memory database cluster module, and adapts the corresponding service requests and service hosts .

Step s4: Configure the memory database for each service host, and cluster all the service hosts in a cluster mode. After the service request distributed by the routing adaptation module is passed the request authentication, dynamically bind the corresponding service host for special Service request processing, because each host uses a memory database, which greatly improves the access efficiency of data request services. At the same time, due to the multiple cluster mode, the number of data access request services is greatly increased, and real high concurrency is achieved. data real-time monitoring.

Step s5: Return the request processing result to the service requester. The request here refers to the above-mentioned service request. Specifically, it may include: returning the processing result of the data service request to the real-time monitoring page for display, and providing the processing result to each corresponding data service requester.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or May be integrated into another system, or some features may be ignored, or not implemented. In addition, the mutual coupling, or direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms. of.

The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units; Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention can be integrated into one processing module, or each unit can be used as a single unit, or two or more units can be integrated into one unit; the above-mentioned integration The unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

Those of ordinary skill in the art can understand that all or part of the steps for realizing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the Including the steps of the foregoing method embodiments; and the foregoing storage medium includes: a removable storage device, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc. A medium on which program code can be stored.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (13)

1. a Real-time Data Processing Method, it is characterised in that described method is applied to generating date system In system, described method includes:

Obtain source system data change information；

Preset process strategy according to first and in real time described data variation information is carried out the first data process, formed First data after process；

Described first data are loaded into memory database；

Based on data processing needs, preset according to second and process strategy in real time to the in described memory database One data carry out the second data process, it is thus achieved that data processed result.

Method the most according to claim 1, it is characterised in that

Described acquisition source system data change information, including:

When the database journal of described origin system changes, obtain the described database journal changed Information；

Described according to first preset process strategy in real time described data variation information is carried out the first data process, First data after formation process, including:

The described data changed described in described database journal information that real time parsing changes and conversion The data form of storehouse log information, forms the data being suitable for Data Stream Processing.

Method the most according to claim 1, it is characterised in that

Described according to first preset process strategy in real time described data variation information is carried out the first data process, First data after formation process, including:

In real time described data variation information is carried out data cleansing and data aggregation process, with standardization and light weight The first data that makeup is downloaded in described memory database.

Method the most according to claim 3, it is characterised in that

Described in real time described data variation information is carried out data summarization process, including:

Data to described data cleansing, carry out slight aggregation process and/or height aggregation process；

Wherein, described slight aggregation process includes carrying out data summarization process with very first time interval；

Described height aggregation process bag carries out data summarization process with the second time interval；

Interval of the described very first time is not more than described second time interval.

Method the most according to claim 1, it is characterised in that

Described based on data processing needs, preset according to second and process strategy in real time in described memory database The first data carry out the second data process, it is thus achieved that data processed result, including:

The first data in described memory database are carried out real-time statistics, reality by business demands based on data Time analyze, in real time monitoring and Real-time Decision, form result.

Method the most according to claim 1, it is characterised in that

Described method also include following at least one:

The generating date of described generating date system is carried out abnormal monitoring；

The generating date of described generating date system is carried out load balance process；

Carry out routeing adaptation processing to the generating date of described generating date system.

Method the most according to claim 6, it is characterised in that

The described generating date to described generating date system carries out load balance process, including with Descend at least one:

Load balancing is carried out between each service host carrying out described first data process；

Load balancing is carried out between each service host carrying out described second data process.

Method the most according to claim 6, it is characterised in that

The described generating date to described generating date system carries out routeing adaptation processing, including:

The route adaptation setting up the service host carrying out described second data process and the service request specified is closed System.

9. a generating date system, it is characterised in that described system includes:

Data acquisition module, is used for obtaining source system data change information；

First data processing module, processes strategy in real time to described data variation information for presetting according to first Carry out the first data process, the first data after formation process；

Data memory module, for being loaded into described memory database by described first data；

Second data processing module, for based on data processing needs, presets according to second and processes strategy in real time The first data in described memory database are carried out the second data process, it is thus achieved that data processed result.

System the most according to claim 9, it is characterised in that

Described data acquisition module, including:

Replicon module, for when the database journal of described origin system changes, acquisition changes Described database journal information；

Analyzing sub-module, the described database journal information changed for real time parsing and conversion are described to be sent out The data form of the described database journal information of changing, forms the data being suitable for Data Stream Processing.

11. systems according to claim 9, it is characterised in that

Described first data processing module, for carrying out data cleansing sum to described data variation information in real time According to aggregation process, the first data being loaded in described memory database with standardization and lightweight.

12. systems according to claim 9, it is characterised in that

Described second data processing module, for business demands based on data, in described memory database The first data carry out real-time statistics, analyze in real time, monitoring and Real-time Decision in real time, form result.

13. systems according to claim 9, it is characterised in that

Described system also includes system management module；

Described system management module, for carrying out different to the generating date of described generating date system Often monitoring；And/or, the generating date of described generating date system is carried out load balance process； And/or carry out routeing adaptation processing to the generating date of described generating date system.