CN105989163A - Data real-time processing method and system - Google Patents
Data real-time processing method and system Download PDFInfo
- Publication number
- CN105989163A CN105989163A CN201510095976.6A CN201510095976A CN105989163A CN 105989163 A CN105989163 A CN 105989163A CN 201510095976 A CN201510095976 A CN 201510095976A CN 105989163 A CN105989163 A CN 105989163A
- Authority
- CN
- China
- Prior art keywords
- data
- real time
- generating date
- time
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data real-time processing method and system. The method is applied to the data real-time processing system. The method comprises the steps of obtaining source system data change information; performing first data processing on the data change information in real time according to a first preset processing policy to form processed first data; loading the first data in a memory database; and performing second data processing on the first data in the memory database in real time according to a second preset processing policy based on a data processing demand to obtain a data processing result.
Description
Technical field
The present invention relates to the information processing technology of field of information processing, particularly relate to a kind of generating date side
Method and system.
Background technology
Along with the development of social informatization, data (concrete as to business management data) are worth and grow with each passing day, right
The promptness of operation information analysis requires the most constantly to promote, it is desirable to the data that can be accepted by real-time monitoring business,
Reach to analyze the purpose with Real-time Decision in real time.
Existing data analysis generally comprises two kinds:
The first: directly carry out statistical analysis in original operation system, operation system may be taken substantial amounts of
Resource, and then cause operation system normal operation is caused extreme influence, analyze promptness and be also difficult to ensure that.
The second: build data warehouse and be used for analyzing；Often in operation system idle from database table
Per diem, monthly extracted data, leave in data base, the analysis of complexity can be carried out, but this based on solely
The data that the vertical data base run is carried out process, and real-time is not enough, and cause that problem is found to have in various degree is stagnant
Rear property.
Current industry has some and is applicable to method of data synchronization, but the method for data synchronization of existing industry is basic
Can only meet the data syn-chronization in operation system to another system, it is impossible to support complicated data operation,
Cause cannot accomplishing service handling data are analyzed in real time, pinpoint the problems when carrying out occurring equally during corresponding decision
Ductility.
Therefore summary, it is provided that a kind of fast response time and the little data processing method of time delay, be prior art
Problem demanding prompt solution.
Summary of the invention
In view of this, embodiment of the present invention expectation provides a kind of Real-time Data Processing Method and system, with at least
Part solves the problem that in prior art, data processing delay is big.
For reaching above-mentioned purpose, the technical scheme is that and be achieved in that:
Embodiment of the present invention first aspect provides a kind of Real-time Data Processing Method, and described method is applied to data
In real time processing system, described method includes:
Obtain source system data change information；
Preset process strategy according to first and in real time described data variation information is carried out the first data process, formed
First data after process；
Described first data are loaded into memory database；
Based on data processing needs, preset according to second and process strategy in real time to the in described memory database
One data carry out the second data process, it is thus achieved that data processed result.
Preferably,
Described acquisition source system data change information, including:
When the database journal of described origin system changes, obtain the described database journal changed
Information；
The described data changed described in described database journal information that real time parsing changes and conversion
The data form of storehouse log information, forms the data being suitable for Data Stream Processing.
Preferably,
Described according to first preset process strategy in real time described data variation information is carried out the first data process,
First data after formation process, including:
In real time described data variation information is carried out data cleansing and data aggregation process, with standardization and light weight
The first data that makeup is downloaded in described memory database.
Preferably,
Described in real time described data variation information is carried out data summarization process, including:
Data to described data cleansing, carry out slight aggregation process and/or height aggregation process；
Wherein, described slight aggregation process includes carrying out data summarization process with very first time interval；
Described height aggregation process bag carries out data summarization process with the second time interval；
Interval of the described very first time is not more than described second time interval.
Preferably,
Described based on data processing needs, preset according to second and process strategy in real time in described memory database
The first data carry out the second data process, it is thus achieved that data processed result, including:
The first data in described memory database are carried out real-time statistics, reality by business demands based on data
Time analyze, in real time monitoring and Real-time Decision, form result.
Preferably,
Described method also include following at least one:
The generating date of described generating date system is carried out abnormal monitoring；
The generating date of described generating date system is carried out load balance process；
Carry out routeing adaptation processing to the generating date of described generating date system.
Preferably,
The described generating date to described generating date system carries out load balance process, including with
Descend at least one:
Load balancing is carried out between each service host carrying out described first data process；
Load balancing is carried out between each service host carrying out described second data process.
Preferably,
The described generating date to described generating date system carries out routeing adaptation processing, including:
The route adaptation setting up the service host carrying out described second data process and the service request specified is closed
System.
Embodiment of the present invention second aspect provides a kind of generating date system, and described system includes:
Data acquisition module, is used for obtaining source system data change information；
First data processing module, processes strategy in real time to described data variation information for presetting according to first
Carry out the first data process, the first data after formation process；
Data memory module, for being loaded into described memory database by described first data；
Second data processing module, for based on data processing needs, presets according to second and processes strategy in real time
The first data in described memory database are carried out the second data process, it is thus achieved that data processed result.
Preferably,
Described data acquisition module, including:
Replicon module, for when the database journal of described origin system changes, acquisition changes
Described database journal information；
Analyzing sub-module, the described database journal information changed for real time parsing and conversion are described to be sent out
The data form of the described database journal information of changing, forms the data being suitable for Data Stream Processing.
Preferably,
Described first data processing module, for carrying out data cleansing sum to described data variation information in real time
According to aggregation process, the first data being loaded in described memory database with standardization and lightweight.
Preferably,
Described second data processing module, for business demands based on data, in described memory database
The first data carry out real-time statistics, analyze in real time, monitoring and Real-time Decision in real time, form result.
Preferably,
Described system also includes system management module；
Described system management module, for carrying out different to the generating date of described generating date system
Often monitoring；And/or, the generating date of described generating date system is carried out load balance process；
And/or carry out routeing adaptation processing to the generating date of described generating date system.The present invention implements
Real-time Data Processing Method described in example, is applied to the generating date system independent of origin system, and base
Described generating date system is provided in described Real-time Data Processing Method.First, these data are located in real time
Reason system is independent of origin system, from the resource without taking origin system, thus properly functioning to origin system
Affect little.Secondly, the Real-time Data Processing Method described in the embodiment of the present invention and system, use internal storage data
Storehouse carries out the storage of data, relative to using disk system storage data in prior art, decreases data
Write disk and read from disk the shared time, and the method described in the present embodiment can use place in real time
The stream treatment technology of reason carries out above-mentioned data process, it is clear that data processing cycle is little, has time delay little and loud
Should fireballing advantage.
Accompanying drawing explanation
Fig. 1 is the schematic flow sheet of the Real-time Data Processing Method of the embodiment of the present invention；
Fig. 2 is the structural representation of the generating date system described in the embodiment of the present invention；
Fig. 3 is the schematic flow sheet of the source system data change information described in the embodiment of the present invention；
Fig. 4 is the schematic flow sheet that the first data described in the embodiment of the present invention process；
Fig. 5 is the system administration schematic flow sheet described in the embodiment of the present invention.
Detailed description of the invention
Below in conjunction with Figure of description and specific embodiment technical scheme done and further explain in detail
State.
Embodiment of the method:
As it is shown in figure 1, the present embodiment provides a kind of Real-time Data Processing Method, described method is applied to data
In real time processing system, described method includes:
Step S110: obtain source system data change information；
Step S120: preset process strategy according to first and in real time described data variation information is carried out the first data
Process, the first data after formation process；
Step S130: described first data are loaded into memory database；
Step S140: based on data processing needs, presets according to second and processes strategy in real time to described interior poke
The second data process is carried out, it is thus achieved that data processed result according to the first data in storehouse.
Described origin system can be various types of operation system, concrete as service system, charging
System etc..
The Real-time Data Processing Method of the present embodiment, is to be applied to count independent of the special of described origin system
According in the system processed, so in carrying out data handling procedure, it is clear that the resource of origin system will not be taken,
It is thus possible to avoid the operation of origin system is interfered, thus origin system is caused to perform the response of service request
The problems such as speed is slow.
The service handling change journal of the data variation information concretely origin system obtained in step s 110
The data such as data.
In step s 130, the first data after the first data process are loaded directly into memory database
In.Described memory database is the data base being made up of internal memory, follow-up when carrying out the second data and processing, directly
Connect and obtain data from memory database, the most a lot of relative to the speed of acquisition data on disk, reading data.
If generally data process also is to carry out in the internal memory of service host, data are moved from internal memory part
The speed moving on to another part is the fastest, or when carrying out the first data and processing, directly at internal memory
Carrying out in the internal memory that data base is corresponding, the Data Migration so connected between internal memory is omitted, it is clear that improves and subtracts
Few systematic processing steps and response speed.
When implementing, after described first data are loaded in described memory database, immediately without time
Between interval carry out described second data process, it is clear that so can relative in prior art by data store
In disk, when carrying out the second data and processing, also need to not only save write data into disk process,
The most also include the process that disk is read data；Being clearly advantageous to improve data and process real-time, minimizing is prolonged
Shi Xing.
Additionally, use internal storage data to store described first data in step s 130, utilize at some data
Reason mechanism, when server delays machine, relative to replying data in magnetic disk, the data in reply internal memory are the simplest,
Obviously relative to the first data are write on disk, it is possible to increase data can safety and the reliability of system.
Further, described step S110 comprises the steps that when the database journal of described origin system changes,
Obtain the described database journal information changed；And the described database journal that real time parsing changes
The data form of the described database journal information changed described in information and conversion, is formed and is suitable for data
The data that stream processes.
The most such as, in step S111, it is mainly the data variation of captured in real time origin system and changes into the first number
According to discernible data form such as csv data form in processing.Described step S110 comprises the steps that from origin system
Middle duplication data, and the data replicated are carried out the step of data parsing.
Real-time data replication can be by data replicated product GoldenGate reading external data storehouse service handling
Journal change data, such as data, work order data, historical data etc..Described GoldenGate is for replicating
The title of one information processing product of data.
Data parsing comprises the steps that the Stream Application application program by using Java to realize, and receives
After the data that data replicated product such as GoldenGate exports, message data is changed accordingly and processes,
The file format such as csv file that output stream can identify in processing.Described Java is a kind of program language；Described
Stream Application is the title of a kind of application program.Described csv file be data form be csv's
File.
Described step S120 comprises the steps that and in real time described data variation information is carried out data cleansing and data summarization
Process, the first data being loaded in described memory database with standardization and lightweight.
Described data cleansing includes data filtering；Described data filtering includes deleting unrelated data, the most such as
Delete data unrelated with service request in data.The most such as, the textual description etc. deleted in log information is believed
Breath.
Described clear data also includes converting the data into Unified coding and the data of same weights and measures, after minimizing
Continue and carry out data conversion and the conversion of weights and measures in second time data procedures, to improve subsequent treatment speed.
Described in real time described data variation information is carried out data summarization process, including:
Data to described data cleansing, carry out slight aggregation process and/or height aggregation process；
Wherein, described slight aggregation process includes carrying out data summarization process with very first time interval；
Described height aggregation process bag carries out data summarization process with the second time interval；
Interval of the described very first time is not more than described second time interval.
Data, for be mainly responsible on the basis of data cleansing, are carried out corresponding collection work by data summarization,
Thus reduce and be loaded into the data volume of memory database, promote systematic function, this module provide slightly collect and
Highly collecting two kinds of functions, slightly collecting the time granularity referring to collect can be second level, highly collects what finger collected
Time granularity can be a minute level；Data load main offer and height cohersive and integrated data are loaded into memory database
Function, it is achieved mode mainly loads application programming interface API by the data calling memory database
Realize.The process logic generally highly collected is increasingly complex relative to the process logic slightly collected.
Described step S140 comprises the steps that
The first data in described memory database are carried out real-time statistics, reality by business demands based on data
Time analyze, in real time monitoring and Real-time Decision, form result.
Described business demand can include service handling demand, is the one of above-mentioned data processing needs.
Below in conjunction with concrete application scenarios, real-time statistics, in real time analysis, in real time monitoring and Real-time Decision are entered
Row is explained.
Real-time statistics includes carrying on the business the origin system as operation system the real-time business such as checkout, payment
Statistics, and can be according to dimensions such as all shop assistants, business halls, the high-volume carrying out full dose calculates in real time,
The statistics needs meeting O&M of limits.
Analyze in real time and can include each channel, the lateral comparison analysis circular of agency in the same marketing time；Real
Time monitoring can effectively management and control risk, as Income Risks monitor, electronic channel transaction monitoring, shop assistant transaction
Monitoring, sales promotion monitors；By monitoring stability bandwidth, sensitive information access monitoring etc..
Real-time Decision can include can according to the analysis result analyzed in real time and analysis and guidance decision-making and action, as
On the basis of selling stream golden period effectively analyze, instruct arrange an order according to class and grade for business hall resource such as personnel, other
Support work, as logistics distribution time point reasonable arrangement provides foundation, promotes shop, Room efficiency, it is achieved cut payroll to improve efficiency.
Described in the present embodiment, method is on the basis of above-mentioned steps, in order to improve the reliability of system with further
Improve generating date system the speed of response, described method also include following at least one:
The generating date of described generating date system is carried out abnormal monitoring；
The generating date of described generating date system is carried out load balance process；
Carry out routeing adaptation processing to the generating date of described generating date system.
Described abnormal monitoring can include processing the data in step S110 to step S140 being monitored, and one
Occur extremely carrying out Abnormality remove process immediately, to ensure the promptness of generating date system, in order to avoid which
Exception in one data processing links causes data to process the phenomenon that promptness declines.
The described generating date to described generating date system carries out load balance process, including with
Descend at least one:
Load balancing is carried out between each service host carrying out described first data process；
Load balancing is carried out between each service host carrying out described second data process.
The most described generating date system can be divided into memory database and to data at
The process main frame of reason.Described process main frame can be used for obtaining data, carrying out the first data process and the second data
Process.The service host being used for carrying out data process within the system is formed with mainframe cluster.Each
Mainframe cluster potentially includes one or more service host.The load of each service host may have and gently have
Weight.By load balancing, the data on the main frame meeting weight can be processed and transfer on the main frame that load is light
Process, the promptness that data process can be improved the most on the whole.
The described generating date to described generating date system carries out routeing adaptation processing, including:
The route adaptation setting up the service host carrying out described second data process and the service request specified is closed
System.
For route adaptation processing is typically to process for the second data, it is that one presets route querying
Process, it is possible to simplify data from memory database in follow-up second data handling procedure inquiry operation,
By simplifying data query, it is possible to again improve the data processing rate of generating date system, reduce number
According to the time delay processed.
Apparatus embodiments:
As in figure 2 it is shown, the present embodiment provides a kind of generating date system, described system includes:
Data acquisition module 110, is used for obtaining source system data change information；
First data processing module 120, processes strategy in real time to described data variation for presetting according to first
Information carries out the first data process, the first data after formation process；
Data memory module 130, for being loaded into described memory database by described first data；
Second data processing module 140, for based on data processing needs, presets according to second and processes strategy
In real time the first data in described memory database are carried out the second data process, it is thus achieved that data processed result.
Acquisition module described in the present embodiment, the most various types of communication interfaces, this communication interface with
Set up between described origin system and have connection, it is possible to receive data from origin system.Described communication interface can include
Line or wireless communication interface, described wired communication interface can include cable interface or fiber optic cable interface；Described
Wave point can include the structures such as reception antenna.
First data processing module 120, data memory module 130 and the tool of the second data processing module 140
Body structure can include processor and storage medium；On described storage medium, storage has executable code.Described deposit
Storage media is connected by the communication interface within the subscriber equipmenies such as bus with described processor.Described processor leads to
Cross and perform described executable code and can realize processing unit 120 and the function of signal generating unit 130.Described place
Reason device can be with central processor CPU, Micro-processor MCV, digital signal processor DSP or battle array able to programme
Row PLC etc. has the processor of the information processing function or processes chip.
Can correspond to the service host in system in the present embodiment.Described memory database can include system
In internal memory integrated in each service host.
Generating date system described in the present embodiment, is system opposed with described origin system, so
When carrying out data and processing, it is clear that the system resource of origin system will not be taken such that it is able to avoid origin system
Business have undesirable effect.First data are loaded in memory database in the present embodiment, it is possible to reduce
Read-write number of times to disk, thus improve data-handling efficiency and reduce time delay.
The most described data acquisition module 110, including:
Replicon module, for when the database journal of described origin system changes, acquisition changes
Described database journal information；
Analyzing sub-module, the described database journal information changed for real time parsing and conversion are described to be sent out
The data form of the described database journal information of changing, forms the data being suitable for Data Stream Processing.
Replicon module is mainly used in using GoldenGate, DataGuard, SharePlex or PAC etc. multiple
Technology processed replicates described data variation information from source database, concrete as from the business of the data base of origin system
Accept the delta data of request, such as data, work order data, historical data etc., and the information that will obtain
It is transferred to analyzing sub-module.In the present embodiment, described data variation information generally can be the change letter specified
Breath, can be with non-arbitrary change information.
Described analyzing sub-module is mainly used in using the Stream Application etc. write by Java language to apply
Program receives the information of the transmission of father and son's module, and changes these information accordingly and process, defeated
Go out during stream processes the file that can identify, concrete such as the file of csv file format.
The most described first data processing module 120, in real time to described data variation information number
According to cleaning and data aggregation process, the first number being loaded in described memory database with standardization and lightweight
According to.
The first data processing module 120 described in the present embodiment, can be described as again real time data processing module, main
The function that can realize is wanted to include the functions such as data cleansing, data summarization and data loading.
The associated description of described data cleansing and data summarization may refer to embodiment of the method, such as, the most described
First data processing module 120, for data to described data cleansing, carry out slight aggregation process and/or
Highly aggregation process；Wherein, described slight aggregation process includes carrying out at data summarization with very first time interval
Reason；Described height aggregation process bag carries out data summarization process with the second time interval；Between the described very first time
Every the most described second time interval.
When implementing, described data memory module, it is mainly used in using memory database to be situated between as storage
Matter, on the one hand it can be avoided that data big data quantity transaction data in processing procedure lands (write magnetic dish) to dividing
The impact of analysis performance, has ensured the promptness of processing procedure；On the other hand the height of memory database memory technology
Available mechanism in turn ensure that to be recovered rapidly and data are not lost server delays machine when, had ensured system
Reliability.
Alternatively, described second data processing module 140, for business demands based on data, to described
The first data in memory database carry out real-time statistics, in real time analysis, in real time monitoring and Real-time Decision,
Form result.
The second data processing module 140 described in the present embodiment can be described as again real time data application module, mainly
On the basis of memory database, build business scenario based on business demand, including real-time statistics, in real time
Analyze, monitoring and Real-time Decision etc. process in real time.Described real-time statistics, analyze in real time, monitoring in real time and real
Time decision-making related definition may refer to embodiment of the method, be not repeated at this.
Additionally, as in figure 2 it is shown, the system described in the present embodiment also includes system management module；
Described system management module 150, for entering the generating date of described generating date system
Row abnormal monitoring；And/or, the generating date of described generating date system is carried out at load balancing
Reason；And/or carry out routeing adaptation processing to the generating date of described generating date system.
The concrete structure of system management module 150 herein may correspond in service host or service host
Processor or process chip etc..The structure of described processor or process chip may refer to preceding sections,
This is not repeated.
Described system management module 150, be particularly used in carry out described first data process respectively service master
Load balancing is carried out between machine；And/or, carry out between each service host carrying out described second data process
Load balancing.
Described system management module 150, it may also be used for set up and carry out the service host that described second data process
Route fitting relation with the service request specified.
Summary, present embodiments provides a kind of generating date system, for above-mentioned generating date
Method provides hardware support, it is clear that is used for the data analysis to origin system and decision-making, is not take up origin system
The advantage such as resource, and data analysis-decision system time delay little on the properly functioning impact of origin system are little.
Below in conjunction with above-described embodiment several concrete application examples of offer:
Example one:
As it is shown on figure 3, this example is for providing a kind of concrete grammar obtaining described data variation information, specifically
As follows:
Step 101: the database journal that external business accepts changes.The most i.e. refer to above-mentioned origin system
Database journal changes.
Step 102: data replication module captures.Data replication module herein is above-mentioned replicon module.
It is captured as obtaining data variation information described in herein, concrete as used data replicated product GoldenGate to catch
Obtain the Redo Log of data base, obtain the service handling data of change, by capture Source sink change data, tracking
Queue, data pump distribution, routing compaction encryption and the process of payment target, it is achieved utilize few system to open
, the log information of the change of captured in real time service handling data base.
Step 103: pumping out queue file (trail file), this step is automatically obtained by data replicated product.
Step 104: log analyzing module, the Stream Application application program realized by Java,
Read the queue file (trail file) that data replicated product such as GoldenGate pumps out, queue file is carried out phase
The parsing answered and process.
Step 105: generate the data file of csv form, provides data input for real-time processing module.Described
Csv form is the above-mentioned data form being suitable for Data Stream Processing.Described Data Stream Processing is to process data
A kind of technology, the concrete Data Stream Processing that how to realize may refer to prior art, the most reinflated at this
, Data Stream Processing has the data process advantage that time delay is little and data-handling efficiency is high in a word.
Step 106: Real time data acquisition flow process terminates, pays real-time processing module.Real-time process herein
Module is equivalent to the first data processing module described in apparatus embodiments.
Example two:
As shown in Figure 4, the concrete grammar that this example processes for providing a kind of first data, it may include:
Step 1: the log analyzing module in Real time data acquisition module, by resolving and the process of origin system
The change journal of data base, increment generates the data file of csv form.Log analyzing module herein is the most in detail
The above-mentioned analyzing sub-module of the chief of the Xiongnu in Acient China.
Step 2: data cleansing module polls data cleansing, collect, load rule base, clear to obtain data
Wash rule.
Step 3: data cleansing, collect, load rule base, returns required inquiry in data cleansing module
Cleaning rule.Cleaning rule herein is described first part presetting process strategy.
Step 4: according to data cleansing rule, log analyzing module increment is generated the service handling data of csv
Carry out consistency check, process invalid value and missing values, data are carried out, stream treatment technology can be used
As StreamBase carries out data cleansing.
Step 5: data summarization module polls data cleansing, collect, load rule base, converges obtaining data
GREV, described data summarization rule is also one of described first ingredient presetting process strategy.
Step 6: data cleansing, collect, load rule base, returns required inquiry in data summarization module
Collect rule.
Step 7: to the data after cleaning, by data summarization rule, generate and collect DBMS, can be divided into
Slightly collect DBMS and highly collect DBMS.
Step 8: the inquiry data cleansing of data loading module, collect, load rule base, obtains data loading
Rule.It also be described first to preset and process one of tactful ingredient that described data load rule, be used for
Memory database loading data.
Step 9: data cleansing, collect, load rule base, returns required inquiry in data loading module
Load rule.
Step 10: load rule according to data, by data in real time to loading in memory database
Example three:
As it is shown in figure 5, this example is for providing a kind of method for managing system, it may include:
Step s1: receive data service request from data service request side.It is concrete such as by real-time monitoring interface,
Data service request is initiated in monitoring, can be by the most multiple data request service in multiple data service request sides.
Step s2: use load equalizer cluster, receives high concurrent data service request, concurrent to height
Data request service carries out service equilibrium, and in cluster, main frame backs up mutually, it is achieved high concurrent data
Service request handling, accelerates service request response speed.The mutual backup mode of the main frame in cluster can be adopted
With the most standby, it is also possible to be 1 for many modes.
Step s3: route adaptation asks mod (described in seek mod be modulus) distribution service request afterwards.The most such as,
After the service of high concurrent request being equalized by load equalizer cluster, by logical for the request service after equilibrium
Cross and seek mod mode.Main load equalizer route Mod with master data, and (such as routing value 0-3, notification rule configures
Realize, lower with), standby data route Mod (such as routing value 4-7, lower with).Standby load equalizer is with master data
Route Mod (4-7), standby data route Mod (0-3), it is distributed in service and memory database cluster module
Corresponding with service main frame, services the adaptation of request and service host accordingly.
Step s4: every service host all configures memory database, simultaneously with trunking mode by all of clothes
Business main frame carries out cluster, the service request to route adaptation module distribution, after request certification is passed through, dynamically
Bind corresponding service host and carry out special service request handling, because every main frame all employs internal storage data
Storehouse, substantially increase the access efficiency of data request service, simultaneously because multiple stage trunking mode, greatly carries
The high quantity of data access request service, it is achieved that the data that real height is concurrent monitor in real time.
Step s5: return request result to service requester.Here request acute pyogenic infection of finger tip above-mentioned service request.
Specifically comprise the steps that the result to data service request, return to monitor the page in real time and represent, will
Result is supplied to the data service request side of each correspondence.
In several embodiments provided herein, it should be understood that disclosed equipment and method,
Can realize by another way.Apparatus embodiments described above is only schematically, such as,
The division of described unit, is only a kind of logic function and divides, and actual can have other division when realizing
Mode, such as: multiple unit or assembly can be in conjunction with, or are desirably integrated into another system, or some are special
Levy and can ignore, or do not perform.It addition, the coupling each other of shown or discussed each ingredient,
Or direct-coupling or communication connection can be the INDIRECT COUPLING by some interfaces, equipment or unit or logical
Letter connect, can be electrical, machinery or other form.
The above-mentioned unit illustrated as separating component can be or may not be physically separate, makees
The parts shown for unit can be or may not be physical location, i.e. may be located at a place,
Can also be distributed on multiple NE；Can select according to the actual needs therein partly or entirely
Unit realizes the purpose of the present embodiment scheme.
It addition, each functional unit in various embodiments of the present invention can be fully integrated into a processing module
In, it is also possible to it is that each unit is individually as a unit, it is also possible to two or more unit collection
Become in a unit；Above-mentioned integrated unit both can realize to use the form of hardware, it would however also be possible to employ
Hardware adds the form of SFU software functional unit and realizes.
One of ordinary skill in the art will appreciate that: realize all or part of step of said method embodiment
Can be completed by the hardware that programmed instruction is relevant, aforesaid program can be stored in a computer-readable
Taking in storage medium, this program upon execution, performs to include the step of said method embodiment；And it is aforementioned
Storage medium include: movable storage device, read only memory (ROM, Read-Only Memory),
Random access memory (RAM, Random Access Memory), magnetic disc or CD etc. are various
The medium of program code can be stored.
The above, the only detailed description of the invention of the present invention, but protection scope of the present invention is not limited to
In this, any those familiar with the art, can be easily in the technical scope that the invention discloses
Expect change or replace, all should contain within protection scope of the present invention.Therefore, the protection of the present invention
Scope should be as the criterion with described scope of the claims.
Claims (13)
1. a Real-time Data Processing Method, it is characterised in that described method is applied to generating date system
In system, described method includes:
Obtain source system data change information；
Preset process strategy according to first and in real time described data variation information is carried out the first data process, formed
First data after process；
Described first data are loaded into memory database；
Based on data processing needs, preset according to second and process strategy in real time to the in described memory database
One data carry out the second data process, it is thus achieved that data processed result.
Method the most according to claim 1, it is characterised in that
Described acquisition source system data change information, including:
When the database journal of described origin system changes, obtain the described database journal changed
Information；
Described according to first preset process strategy in real time described data variation information is carried out the first data process,
First data after formation process, including:
The described data changed described in described database journal information that real time parsing changes and conversion
The data form of storehouse log information, forms the data being suitable for Data Stream Processing.
Method the most according to claim 1, it is characterised in that
Described according to first preset process strategy in real time described data variation information is carried out the first data process,
First data after formation process, including:
In real time described data variation information is carried out data cleansing and data aggregation process, with standardization and light weight
The first data that makeup is downloaded in described memory database.
Method the most according to claim 3, it is characterised in that
Described in real time described data variation information is carried out data summarization process, including:
Data to described data cleansing, carry out slight aggregation process and/or height aggregation process；
Wherein, described slight aggregation process includes carrying out data summarization process with very first time interval；
Described height aggregation process bag carries out data summarization process with the second time interval；
Interval of the described very first time is not more than described second time interval.
Method the most according to claim 1, it is characterised in that
Described based on data processing needs, preset according to second and process strategy in real time in described memory database
The first data carry out the second data process, it is thus achieved that data processed result, including:
The first data in described memory database are carried out real-time statistics, reality by business demands based on data
Time analyze, in real time monitoring and Real-time Decision, form result.
Method the most according to claim 1, it is characterised in that
Described method also include following at least one:
The generating date of described generating date system is carried out abnormal monitoring；
The generating date of described generating date system is carried out load balance process；
Carry out routeing adaptation processing to the generating date of described generating date system.
Method the most according to claim 6, it is characterised in that
The described generating date to described generating date system carries out load balance process, including with
Descend at least one:
Load balancing is carried out between each service host carrying out described first data process；
Load balancing is carried out between each service host carrying out described second data process.
Method the most according to claim 6, it is characterised in that
The described generating date to described generating date system carries out routeing adaptation processing, including:
The route adaptation setting up the service host carrying out described second data process and the service request specified is closed
System.
9. a generating date system, it is characterised in that described system includes:
Data acquisition module, is used for obtaining source system data change information；
First data processing module, processes strategy in real time to described data variation information for presetting according to first
Carry out the first data process, the first data after formation process；
Data memory module, for being loaded into described memory database by described first data；
Second data processing module, for based on data processing needs, presets according to second and processes strategy in real time
The first data in described memory database are carried out the second data process, it is thus achieved that data processed result.
System the most according to claim 9, it is characterised in that
Described data acquisition module, including:
Replicon module, for when the database journal of described origin system changes, acquisition changes
Described database journal information；
Analyzing sub-module, the described database journal information changed for real time parsing and conversion are described to be sent out
The data form of the described database journal information of changing, forms the data being suitable for Data Stream Processing.
11. systems according to claim 9, it is characterised in that
Described first data processing module, for carrying out data cleansing sum to described data variation information in real time
According to aggregation process, the first data being loaded in described memory database with standardization and lightweight.
12. systems according to claim 9, it is characterised in that
Described second data processing module, for business demands based on data, in described memory database
The first data carry out real-time statistics, analyze in real time, monitoring and Real-time Decision in real time, form result.
13. systems according to claim 9, it is characterised in that
Described system also includes system management module；
Described system management module, for carrying out different to the generating date of described generating date system
Often monitoring；And/or, the generating date of described generating date system is carried out load balance process；
And/or carry out routeing adaptation processing to the generating date of described generating date system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510095976.6A CN105989163A (en) | 2015-03-04 | 2015-03-04 | Data real-time processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510095976.6A CN105989163A (en) | 2015-03-04 | 2015-03-04 | Data real-time processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105989163A true CN105989163A (en) | 2016-10-05 |
Family
ID=57039107
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510095976.6A Pending CN105989163A (en) | 2015-03-04 | 2015-03-04 | Data real-time processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105989163A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106681837A (en) * | 2016-12-29 | 2017-05-17 | 北京奇虎科技有限公司 | Data sheet based data eliminating method and device |
| CN106993060A (en) * | 2017-05-25 | 2017-07-28 | 北京仿真中心 | A kind of master data integrated approach based on service architecture |
| CN108446362A (en) * | 2018-03-13 | 2018-08-24 | 平安普惠企业管理有限公司 | Data cleansing processing method, device, computer equipment and storage medium |
| CN108874964A (en) * | 2018-06-07 | 2018-11-23 | 火烈鸟网络(广州)股份有限公司 | A kind of method and system in monitoring data library |
| CN109241033A (en) * | 2018-08-21 | 2019-01-18 | 北京京东尚科信息技术有限公司 | The method and apparatus for creating real-time data warehouse |
| CN110019269A (en) * | 2017-12-04 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Method, system and the terminal device of data check |
| CN110147391A (en) * | 2019-04-08 | 2019-08-20 | 顺丰速运有限公司 | Data handover method, system, equipment and storage medium |
| CN110162519A (en) * | 2019-04-17 | 2019-08-23 | 苏宁易购集团股份有限公司 | Data clearing method |
| CN111626716A (en) * | 2020-07-30 | 2020-09-04 | 浙江大学 | Agricultural test station teaching and scientific research online service system |
| CN113821556A (en) * | 2021-09-16 | 2021-12-21 | 江苏方天电力技术有限公司 | Data loading method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102033930A (en) * | 2010-12-17 | 2011-04-27 | 北京世纪互联工程技术服务有限公司 | Distributed memory database system |
| CN102279885A (en) * | 2011-08-16 | 2011-12-14 | 中兴通讯股份有限公司 | Method and device for operating data by memory database |
| CN102456076A (en) * | 2011-11-23 | 2012-05-16 | 北京安天电子设备有限公司 | Massive fragment data aggregation system and method |
| CN102916844A (en) * | 2012-11-22 | 2013-02-06 | 南京恩瑞特实业有限公司 | Mass data fusion and real-time monitoring system |
-
2015
- 2015-03-04 CN CN201510095976.6A patent/CN105989163A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102033930A (en) * | 2010-12-17 | 2011-04-27 | 北京世纪互联工程技术服务有限公司 | Distributed memory database system |
| CN102279885A (en) * | 2011-08-16 | 2011-12-14 | 中兴通讯股份有限公司 | Method and device for operating data by memory database |
| CN102456076A (en) * | 2011-11-23 | 2012-05-16 | 北京安天电子设备有限公司 | Massive fragment data aggregation system and method |
| CN102916844A (en) * | 2012-11-22 | 2013-02-06 | 南京恩瑞特实业有限公司 | Mass data fusion and real-time monitoring system |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106681837A (en) * | 2016-12-29 | 2017-05-17 | 北京奇虎科技有限公司 | Data sheet based data eliminating method and device |
| CN106681837B (en) * | 2016-12-29 | 2020-10-16 | 北京奇虎科技有限公司 | Data elimination method and device based on data table |
| CN106993060A (en) * | 2017-05-25 | 2017-07-28 | 北京仿真中心 | A kind of master data integrated approach based on service architecture |
| CN110019269A (en) * | 2017-12-04 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Method, system and the terminal device of data check |
| CN108446362A (en) * | 2018-03-13 | 2018-08-24 | 平安普惠企业管理有限公司 | Data cleansing processing method, device, computer equipment and storage medium |
| CN108874964A (en) * | 2018-06-07 | 2018-11-23 | 火烈鸟网络(广州)股份有限公司 | A kind of method and system in monitoring data library |
| CN109241033A (en) * | 2018-08-21 | 2019-01-18 | 北京京东尚科信息技术有限公司 | The method and apparatus for creating real-time data warehouse |
| CN110147391A (en) * | 2019-04-08 | 2019-08-20 | 顺丰速运有限公司 | Data handover method, system, equipment and storage medium |
| CN110162519A (en) * | 2019-04-17 | 2019-08-23 | 苏宁易购集团股份有限公司 | Data clearing method |
| WO2020211299A1 (en) * | 2019-04-17 | 2020-10-22 | 苏宁云计算有限公司 | Data cleansing method |
| CN111626716A (en) * | 2020-07-30 | 2020-09-04 | 浙江大学 | Agricultural test station teaching and scientific research online service system |
| CN113821556A (en) * | 2021-09-16 | 2021-12-21 | 江苏方天电力技术有限公司 | Data loading method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105989163A (en) | Data real-time processing method and system | |
| US20160055044A1 (en) | Fault analysis method, fault analysis system, and storage medium | |
| US9992269B1 (en) | Distributed complex event processing | |
| CN105426292A (en) | Game log real-time processing system and method | |
| CN109783512A (en) | Data processing method, device, computer equipment and storage medium | |
| CN107103064B (en) | Data statistical method and device | |
| US8005860B1 (en) | Object-level database performance management | |
| US11321155B2 (en) | Automatic resource dependency tracking and structure for maintenance of resource fault propagation | |
| US10217073B2 (en) | Monitoring transactions from distributed applications and using selective metrics | |
| US20080065588A1 (en) | Selectively Logging Query Data Based On Cost | |
| CN110175163A (en) | More library separation methods, system and medium based on business function intelligently parsing | |
| CN108650684A (en) | A kind of correlation rule determines method and device | |
| CN1293473C (en) | System process protection method | |
| CN109101296B (en) | Distributed automatic deduction method and system | |
| CN105653322A (en) | Operation and maintenance server and server event processing method | |
| CN105069029B (en) | A kind of real-time ETL system and method | |
| CN104636211A (en) | Information interaction method among software systems, and middleware system | |
| CN110196868A (en) | Based on distributed work order flow monitoring method | |
| CN110309206B (en) | Order information acquisition method and system | |
| US8285752B1 (en) | System and method for maintaining a plurality of summary levels in a single table | |
| CN109120706A (en) | Business scheduling method and system | |
| CN112631754A (en) | Data processing method, data processing device, storage medium and electronic device | |
| CN108304293A (en) | A kind of software systems monitoring method based on big data technology | |
| CN105630997A (en) | Data parallel processing method, device and equipment | |
| CN102930046B (en) | Data processing method, computing node and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161005 |
|
| RJ01 | Rejection of invention patent application after publication |