CN109426576A - Fault-tolerance processing method and fault-tolerant component - Google Patents
Fault-tolerance processing method and fault-tolerant component Download PDFInfo
- Publication number
- CN109426576A CN109426576A CN201710761219.7A CN201710761219A CN109426576A CN 109426576 A CN109426576 A CN 109426576A CN 201710761219 A CN201710761219 A CN 201710761219A CN 109426576 A CN109426576 A CN 109426576A
- Authority
- CN
- China
- Prior art keywords
- record
- rule
- fault
- abnormal
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0751—Error or fault detection not based on redundancy
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application discloses fault-tolerance processing method and fault-tolerant component, which is applied to extract in conversion load ETL operation process, which comprises obtains the business datum extracted from data source, the business datum includes multiple records；The business datum is checked, with record and normal record abnormal in the determination business datum；The abnormal record is repaired, successfully record will be repaired and the normal record is sent to downstream components.The scheme for implementing the application is able to solve to improve the robustness of ETL program, is reduced application program because abnormal caused by data quality problem, is reduced the human cost of data warehouse O&M.
Description
Technical field
The present invention relates to Data Integration field more particularly to fault-tolerance processing method and fault-tolerant components.
Background technique
IT application in enterprises is play an important role in the management operating of all trades and professions, and IT application in enterprise has become promotion
The important channel of enterprise core competence.In the database of IT application in enterprises, generally comprise a large amount of redundant data, need into
Row is made rational planning for and is handled, and the appearance of data warehouse, meets demand of the enterprise to planning and handling big data.Data
Warehouse system technology joined the function of Data Analysis Services in the system-based of legacy data library, and enterprise is allowed to obtain essence
True and comprehensive business data, provides effective information for the formulation and implementation of business decision.Data warehouse can usually divide
For data source, ETL module and data depot layer, ETL module is responsible for extraction, cleaning/conversion, load of data etc., to data into
Row integration, is the important link of data warehouse technology.
However, increasing with business event, data warehouse layer, which may correspond to operation, a large amount of application program, and even
Then a large amount of data source may frequently occur the exception as caused by data quality problem in this way, daily to put into largely
O&M manpower.
Summary of the invention
The embodiment of the invention discloses a kind of fault-tolerance processing method and fault-tolerant components, are able to solve to improve ETL module
Robustness, reduce application program because abnormal caused by data quality problem, reduce the human cost of data warehouse O&M.
In a first aspect, the method is applied to extract to convert and add the embodiment of the invention provides a kind of fault-tolerance processing method
Carry in ETL operation process, which comprises data from source side into data warehouse side transmission process, obtain from number
According to the business datum that source is extracted, the business datum includes multiple records；The business datum is checked, described in determination
Abnormal record and normal record in business datum；The abnormal record is repaired, will repair successfully record and
The normal record is sent to downstream components.
As can be seen that the embodiment of the present invention is in the ETL stage, just to the data extracted from data source, (such as the data are
Table) carry out fault-tolerant, it will be repaired by checking the abnormal record in determining data, and to abnormal record, and eliminate factor
According to exception caused by quality problems, then by downstream components, (component of cleaning function, the component of conversion function load function again
Can component etc.) final output is to data warehouse side.
Based in a first aspect, being obtained before the business datum that data source extracts, packet in the specific embodiment of the invention
It includes: configuring fault-tolerant rule, the fault-tolerant rule checks rule and reparation rule for determining；Wherein, the rule that checks is used for
Determine that the record belongs to normal standard, the reparation rule is for determining side used by repairing the abnormal record
Formula；The business datum is checked, specifically: based on the determining standard of rule that checks to the business datum
It is checked；The abnormal record is repaired, specifically: based on the determining mode of the reparation rule to described
Abnormal record is repaired, so that the abnormal record meets the standard.
The embodiment of the present invention is in the specific implementation, the configuration of fault-tolerant rule can be carried out according to specific business, for example, for one
A data source, data source generally include a variety of tables, each table is all related to a kind of specific business, then can be directed to each
Kind of table configures fault-tolerant rule as needed respectively, i.e., the fault-tolerant rule configured in the embodiment of the present invention be can customize, can
Extension.
Based in a first aspect, checking the standard pair determined by rule based on described in the specific embodiment of the invention
The business datum is checked, comprising: is checked in the business datum based on the determining standard of rule that checks
Whether the technical parameter of record is abnormal, and the technical parameter is the parameter of the IT technical aspect of the record in data, including data
Type, field length, accuracy value, time and date value, in digit strings at least one of；And rule are checked based on described
Whether the business relevant field that the standard then determined checks the record in the business datum is abnormal, the business related words
Section includes at least one in customer type, type of service, quantity on order.
Based in a first aspect, after being repaired to the abnormal record, also being wrapped in the specific embodiment of the invention
It includes:
(1) if to the abnormal record reparation success, successfully record will be repaired and be saved in alarm log；Such feelings
Under condition, aftermentioned successfully record and the normal record of repairing is sent to downstream components.
(2) if to the abnormal record repairing failure, the record of repairing failure is saved in refusal log, this feelings
Under condition, normal record is only sent to downstream components；The record of the repairing failure in the refusal log can be used for into
Pedestrian's work reparation, the record after manually repairing can replace and the data pair in data transmission procedure next time
The original abnormal data answered, to be output to downstream components.
It should be noted that can also be recorded respectively in alarm log and refusal log in the possible embodiment of the present invention
Each original value, field for abnormal field occur of saved record is corresponding to check regular code and timestamp
Etc. information.
As can be seen that after the checking and repair of the embodiment of the present invention, it can be effectively to defective in quality data
It is collected, repairs and saves, the data of repairing failure can also be imported again in the transmission next time after artificial repair,
Ensure the correctness of data.
Based in a first aspect, acquisition is described after the business datum that data source extracts in the specific embodiment of the invention
Method further include: be ranked up according to the multiple record of key-value pair, wherein each records a corresponding key assignments；And such as
The key assignments for the record that fruit discovery is currently checked and the key assignments of another record repeat, and remove the duplicate record of the key assignments.
For example, the different records in the data indicate different orders, then key assignments is order number, then, sequence
Deduplication module 303 can be ranked up the different records in the data according to order number, if the order number of record 1 and record 5
It is identical, then illustrating that the two orders repeat, it is possible to record 1 is deleted from the data (or be also possible to that 5 will be recorded
It deletes).
Second aspect, the embodiment of the invention provides a kind of fault-tolerant component, the fault-tolerant component is applied to extract to convert and add
It carries in ETL operation process, comprising: receiving module, for obtaining the business datum extracted from data source, the business datum includes
Multiple records；Check module, for being checked to the business datum, with record abnormal in the determination business datum and
Normal record；Repair module, for being repaired to the abnormal record；Sending module, for successfully note will to be repaired
Record and the normal record are sent to downstream components.
In the specific implementation, data integration and the integrated tool DataStage of IBM can be used, develop to obtain by C Plus Plus
The fault-tolerant component, that is to say, that the fault-tolerant component can be a kind of DataStage program assembly.
Based on second aspect, in the specific embodiment of the invention, the fault-tolerant component further includes configuration module, the configuration
Module checks rule and reparation rule for determining for configuring fault-tolerant rule, the fault-tolerant rule；Wherein, described to check rule
For determining that the record belongs to normal standard, the reparation rule is for determining used by repairing the abnormal record
Mode；The module that checks for being checked to the business datum, specifically: the module that checks is for based on the inspection
The standard that core rule determines checks the business datum；The repair module be used for the abnormal record into
Row is repaired, specifically: the mode that the repair module is used to determine based on the reparation rule is to the abnormal record
It is repaired, so that the abnormal record meets the standard.
Based on second aspect, in the specific embodiment of the invention, it is described check module for be based on described in check regular institute
The determining standard checks the business datum, comprising: the module that checks is for based on described to check rule true
Whether the technical parameter that the fixed standard checks the record in the business datum is abnormal, and the technical parameter includes data class
Type, field length, accuracy value, time and date value, in digit strings at least one of；The module that checks is also used to be based on
Whether the business relevant field that the standard for checking rule determination checks the record in the business datum is abnormal, described
Business relevant field include customer type, type of service, in quantity on order at least one of.
Based on second aspect, in the specific embodiment of the invention, the fault-tolerant component further includes log module, the log
Module includes alarm log and refusal log；
The log module is used for, if the abnormal record reparation success will be repaired successfully record and is saved in
Alarm log；If the record of repairing failure is saved in refusal log, the refusal to the abnormal record repairing failure
The record of the repairing failure in log can be used for manually being repaired.
Based on second aspect, in the specific embodiment of the invention, the fault-tolerant component further includes sequence deduplication module, described
Sequence deduplication module is used to be ranked up according to the multiple record of key-value pair, and, remove the duplicate record of the key assignments；Its
In, each records a corresponding key assignments.
The third aspect, the embodiment of the invention provides the fault-tolerant component of another ETL, the fault-tolerant component of the ETL includes checking mould
Block (and can be divided into IT checks module and business checks module), sequence deduplication module, repair module, log module and configuration mould
Block.In the specific implementation, data integration and the integrated tool DataStage of IBM can be used, develop to obtain the ETL by C Plus Plus
Fault-tolerant component, that is to say, that the fault-tolerant component of ETL can be a kind of DataStage program assembly.The industry extracted from data source
For data of being engaged in component fault-tolerant by ETL, the fault-tolerant rule that the fault-tolerant component of ETL is configured according to configuration module is checking module
(including IT checks module and business checks module) checks the business datum, as needed in sequence deduplication module to this
Business datum is ranked up duplicate removal processing, after determining that exception occurs in the business datum, will occur abnormal part in data
(such as abnormal record) is input to repair module and is repaired, if repaired successfully, will repair in successful data and data
Do not occur abnormal data (such as normal record) as normal data and is output to downstream components；In addition, will also repair successful
Data are saved to log module.If the data for not occurring abnormal in data are output to by repairing failure as normal data
Downstream components, in addition, the data of repairing failure are also saved in log module.Wherein, the downstream components can be for clear
The component washed/converted, or the component for load, in some instances it may even be possible to be a kind of data sending assembly.
Fourth aspect, the embodiment of the invention provides a kind of data warehouses, and the system comprises ETL modules, source system
System and DWI system, wherein source system is the integrated of database, that is to say, that source system may include thousands of data source,
Data source is stored with data.ETL module is a kind of data integration and integrated tool, for realizing data loading processing, ETL module
Including fault-tolerant component provided in an embodiment of the present invention, ETL operation process includes being extracted to the data of data source, being fault-tolerant, is clear
The processing such as wash/convert, load, and then by the Data Integration of data source to DWI system, DWI system is based on specific data knot
Decision data constructed by structure and application program analyzes library.
5th aspect, the embodiment of the invention provides it is a kind of store computer instruction readable non-volatile memory medium,
The readable non-volatile memory medium includes computer instruction, in which: the computer instruction is performed to realize first aspect
The method of description.
6th aspect, the embodiment of the invention provides a kind of computer program products, when computer program product is run on
When computer, it is performed the method to realize first aspect description.
As can be seen that in the embodiment of the present invention, during data are from data source to DWI, when the number of data source
When according to data quality problem occur, the fault-tolerant component of ETL (such as DataStage program assembly) provided in an embodiment of the present invention can be with
After data pick-up, fault-tolerant processing is carried out to data, checks data quality problem, and repair to data, to will repair
Data afterwards continue to be sent to downstream components.Implement the embodiment of the present invention, program can be reduced to a great extent because of the quality of data
Abnormal caused by problem, guarantee procedure operates normally, and realizes the configurability of fault-tolerant rule, customized property and expansible
Property, fault-tolerant component reaches code reuse, greatly reduces development amount.
Detailed description of the invention
Fig. 1 is a kind of structural schematic diagram for data warehouse that the prior art provides；
Fig. 2 is a kind of structural schematic diagram of data warehouse provided in an embodiment of the present invention；
Fig. 3 is a kind of structural schematic diagram of fault-tolerant component provided in an embodiment of the present invention；
Fig. 4 is a kind of schematic diagram of a scenario of fault-tolerant processing provided in an embodiment of the present invention；
Fig. 5 is the schematic diagram of a scenario of another fault-tolerant processing provided in an embodiment of the present invention；
Fig. 6 is a kind of flow diagram of fault-tolerance processing method provided in an embodiment of the present invention；
Fig. 7 is the flow diagram of another fault-tolerance processing method provided in an embodiment of the present invention；
Fig. 8 is the structural schematic diagram of another fault-tolerant component provided in an embodiment of the present invention；
Fig. 9 is a kind of structural schematic diagram of fault tolerance facility provided in an embodiment of the present invention.
Specific embodiment
In order to join to describe a kind of structural schematic diagram of data warehouse of the prior art first convenient for scheme understanding
See Fig. 1, which includes data pick-up, conversion and load (Extract, Transform, and Load, ETL) mould
Block 100, source system 101 and data warehouse integrate (Data Warehouse Integration, DWI) 105, wherein source system
101 be the integrated of database, that is to say, that source system 101 may include thousands of data source, and data source is stored with data.
ETL module 100 is a kind of data integration and integrated tool, and for realizing data loading processing, process includes to data source
Data carry out the processing such as extraction 102, cleaning/conversion 103, load 104, and then send data to DWI system 105, DWI system
105 be to analyze library based on decision data constructed by specific data structure and application program.In practical application, for one
Big DWI system 105 may run thousands of a application programs, and connect thousands of data source, thus may
Can frequently occur as caused by source system data quality problems it is abnormal, thus cause application program related service exception or in
It is disconnected.The solution of the prior art be usually be directed in different application programs to develop corresponding fault-tolerant respectively, however,
In face of thousands of application program, to be customized respectively for application program one by one, the code development of repeatability, exploitation
Workload is huge, needs to put into a large amount of O&M manpower.
In order to solve above-mentioned problems of the prior art, the embodiment of the invention provides a kind of fault-tolerance processing method with
And fault-tolerant component, the fault-tolerance processing method and fault-tolerant component can be applied in the ETL module of data warehouse, to improve
The robustness of ETL module, to reduce the probability that application program is abnormal, reduce the manpower of data warehouse O&M at
This.
Referring to fig. 2, Fig. 2 is a kind of structural schematic diagram of data warehouse provided in an embodiment of the present invention.The present invention is real
The data warehouse for applying example offer includes ETL module 200, source system 201 and DWI system 206, wherein for ETL module
200, which there is fault-tolerant component to take out to have fault tolerance in the data of the data source to source system
After taking 202, fault-tolerant 203 processing is first passed through, is eliminated because of data exception caused by data quality problem, then using cleaning/turn
204, load 205 is changed, to send data to DWI system 206.
A kind of fault-tolerant component provided in an embodiment of the present invention is described below in detail.It is provided in an embodiment of the present invention referring to Fig. 3
A kind of fault-tolerant component 300 of ETL, including check module (and can be divided into IT checks module 301 and business checks module 302), sequence
Deduplication module 303, repair module 304, log module 305 and configuration module 306.In the specific implementation, the data set of IBM can be used
At with integrated tool DataStage, develop to obtain the fault-tolerant component of the ETL by C Plus Plus, that is to say, that the ETL fault-tolerant group
Part can be a kind of DataStage program assembly.As shown in figure 3, data (such as the business relevant to business that data source extracts
Data) in component 300 fault-tolerant by ETL, the fault-tolerant rule that the fault-tolerant component 300 of ETL is configured according to configuration module 306,
Module (including IT checks module 301 and business checks module 302) is checked to check the business datum, optional embodiment
In, it is also possible to duplicate removal processing can be ranked up to the business datum in sequence deduplication module as needed, determine the business
After exception occur in data, the part (such as abnormal record) that will occur exception in data is input to repair module 304 and repairs,
If repaired successfully, it will repair in successful data and data and do not occur abnormal data (such as normal record) as normal
Data are output to downstream components；It saves in addition, will also repair successful data to log module 305.It, will if repairing failure
Do not occur abnormal data in data and be output to downstream components as normal data, in addition, also saving the data of repairing failure
To log module 305.Wherein, the downstream components can be the component for cleaning/converting 204, or for loading 205
Component, in some instances it may even be possible to be a kind of data sending assembly (normal data is sent to DWI).
Each functional module of the fault-tolerant component of the ETL is described as follows:
(1) configuration module 306: in the embodiment of the present invention, configuration module 306 is used to configure fault-tolerant rule, the fault-tolerant rule
Then rule and reparation rule are checked for determining；Wherein, rule is checked for specifying the normal standard of the business datum, is repaired
Rule is for mode used by specified restoration data；In the specific implementation, matching for fault-tolerant rule can be carried out according to specific business
It sets, for example, data source generally includes a variety of tables for a data source, each table is all related to a kind of specific business, that
Fault-tolerant rule can be configured as needed respectively for each table, i.e., configuration module 306 configures in the embodiment of the present invention
Fault-tolerant rule can customize, is expansible.
(2) IT checks module 301:IT and checks module 301 for checking to data progress IT technical aspect, specifically,
Can based on it is configured it is good check the standard specified by rule check data technical parameter it is whether abnormal, the technology ginseng
Number includes at least one in data type, field length, accuracy value, time and date value, character string, sky field；
Specifically, for the data source comprising a variety of tables, a table will include multiple records, each record includes more
A field, each field include some specific technical parameters, such as data type, field length, accuracy value, time and day
Time value, character string, sky field etc., the embodiment of the present invention in checking the standard specified by rule, specify how
The data type of sample, field length, accuracy value, time and date value, character string, sky field etc. be it is correct legal, that
, when the table inputted checks module 301 by IT, IT checks module 301 and judges in the table according to the standard configured
Whether the technical parameter of the section in each record is correct legal, if certain technical parameters are not inconsistent standardization, these technologies
Record where parameter is exactly abnormal data occur, if technical parameter all in a record complies with standard, i.e., should
It is recorded as normal data.
For example, it checks in standard specified by rule, the data type for recording the field A in A is value type, so
And the data of input are checked, it is found that the data type of the field A in the record A of the data is character string type, is not inconsistent
Standardization, so this record A is exactly abnormal data.
For example, recording the field B in B indicates the contract number of certain business, checks in standard specified by rule, word
The field length range of section B is 10-12, however is checked to the data of input, finds the section B in the record B of the data
Field length be 15, be not inconsistent standardization, thus this record B be exactly abnormal data.
For example, the field C for recording C indicates the amount of money, checks in standard specified by rule, records the essence of the field C of C
Angle value is 0.01, however is checked to the data of input, it is found that the accuracy value of the field C of the record C of the data is 0.001,
Standardization is not inconsistent, so this record C is exactly abnormal data.
For example, the field D for recording D indicates date value, checks in standard specified by rule, records the field D's of D
The range of date value is " being greater than on January 1st, 2002 ", however is checked to the data of input, finds the record D's of the data
Field D is " March 4 nineteen ninety ", is not inconsistent standardization, so this record D is exactly abnormal data.
For example, the field E for recording E indicates employee-number, checks in standard specified by rule, records the field E of E
It (is made of Arabic numerals) for digit strings, however the data of input is checked, find the record E's of the data
Field E is " 2019283774a ", is not inconsistent standardization, so this record E is exactly abnormal data.
For example, it checking in standard specified by rule, the field F for recording F is non-empty field, however to input
Data are checked, it is found that the field F of the record F of the data is sky, be not inconsistent standardization, so this record F is exactly abnormal number
According to.
It should be noted that above-described embodiment is only in the specific implementation, the embodiment of the present invention can also be other technologies parameter
It is merely illustrative, it is not construed as limitation of the invention.
(3) business checks module 302: business checks module 302 for checking in terms of data carry out business.As described in
Data are business datum relevant to business, and business checks module 302 and checks the mark specified by rule based on what is configured
Standard checks whether the business relevant field in the business datum is abnormal, and the business relevant field includes customer type, business
At least one of in type, quantity on order.
The embodiment of the present invention specifies the business related words in business datum in checking the standard specified by rule
It is correct legal how section (customer type, type of service, quantity on order etc.) is only, then, when the table inputted passes through
When business checks module 302, business checks module 302 and judges business in each record in the table according to the standard configured
Whether relevant field is correct legal, if certain business relevant fields are not inconsistent standardization, where these business relevant fields
Record be exactly abnormal data occur, if technical parameter all in a record complies with standard, i.e., the record is positive
Normal data.
For example, the business relevant field G for recording G indicates customer type, checks in standard specified by rule, records
The business relevant field G of G is external client, however is checked to the data of input, finds the business phase of the record G of the data
Pass field G is internal customer, is not inconsistent standardization, so this record G is exactly abnormal data.
For example, the business relevant field H for recording H indicates type of service, checks in standard specified by rule, records
The business relevant field H of H is domestic service, however is checked to the data of input, finds the business phase of the record H of the data
Pass field H is overseas operation, is not inconsistent standardization, so this record H is exactly abnormal data.
For example, the business relevant field I for recording I indicates quantity on order, checks in standard specified by rule, records
The value of the business relevant field I of I is not less than 0, however checks to the data of input, finds the business of the record I of the data
The value of relevant field I is -2, is not inconsistent standardization, so this record I is exactly abnormal data.
It should be noted that in the specific implementation, the embodiment of the present invention can also be other business relevant fields, above-mentioned reality
Applying example is only example, is not construed as limitation of the invention.
(4) the sequence fault-tolerant component 300 of deduplication module 303:ETL may also include sequence deduplication module 303, of the invention possible
Embodiment in, each of data record can correspond to a key assignments, and sequence deduplication module 303 can be used for according to key-value pair institute
Multiple records of the data of input are ranked up, and, if it find that the key assignments of the record currently checked and another record
Key assignments repeat, then remove the duplicate record of this key assignments；
For example, the different records in the data indicate different orders, then key assignments is order number, then, sequence
Deduplication module 303 can be ranked up the different records in the data according to order number, if the order number of record 1 and record 5
It is identical, then illustrating that the two orders repeat, it is possible to record 1 is deleted from the data (or be also possible to that 5 will be recorded
It deletes).
It should be noted that above-described embodiment is only example, it is not construed as limitation of the invention.
(5) repair module 304: repair module 304 is used for when determining that inputted data occur abnormal, based on being configured
The reparation rule specified mode data are repaired so that data fit checks standard specified in rule.Tool
It in body embodiment, is checked passing through, determines and occur abnormal record (i.e. abnormal data) in data and do not occur abnormal record
After (i.e. normal data), repair module 304 is used to repair the record for exception occur based on reparation rule.
For example, the field C of the record C in data indicates the amount of money, checks in standard specified by rule, records C's
The accuracy value of field C is 0.01, and the reparation rule of the field C for recording C is configured as accuracy value by rounding up
It is converted into 0.01.The accuracy value for checking the field C of the record C for the data that discovery inputs is 0.001, and record C is exception occur
It records, after the clearly record of appearance exception of repair module 304, the value of the field C of record C is rounded up, thus
So that the accuracy value of field C is converted into 0.01, repairs record C for normal data to realize.
For example, the business relevant field I of the record I in data indicates quantity on order, checks mark specified by rule
In standard, the value of the business relevant field I of I is recorded not less than 0, and the reparation rule of the business relevant field I for recording I is matched
It is set to and all negatives is rewritten as 0.The value for checking the business relevant field I of the record I of the data of discovery input is -2, records I
To there is abnormal record, after the clearly record of appearance exception of repair module 304, the value of business relevant field I is rewritten as
0, record I is repaired as normal data to realize.
For example, it checks in standard specified by rule, the data type of the field A in the record A of data is numerical value
Type, and the reparation rule for recording A is " if exception occurs in record A, record deletes A ".Check the number of discovery input
According to record A in the data type of field A be character string type, record A be the record for exception occur, and repair module 304 is bright
After the record of true appearance exception, record A is deleted.
It should be noted that in the specific implementation, the embodiment of the present invention can also be other reparation rules, here not further
One enumerates.Above-described embodiment is only example, is not construed as limitation of the invention.
(6) log module 305: log module 305 can be used for storing occurred abnormal data (may include repair it is successful
The record of record and repairing failure), for technical staff's analysis, repair.Log module 305 may include alarm log and refusal day
Will, specifically, being identified as and " accusing if successfully record duplication is a by the reparation to there is abnormal record reparation success
Alert record ", is saved in alarm log so that the reparation of the duplication successfully be recorded, another, which repairs successfully record, to continue
It is output to downstream components；If being that " refusal is remembered by the record identification of the repairing failure to there is abnormal record repairing failure
Record ", the record of the repairing failure cannot be output to downstream components, and the record of the repairing failure is saved in refusal log.
It should be noted that carrying out the original of repairing failure occur in repair process in abnormal data in the embodiment of the present invention
It is diversified because can be.
For example, recording the field B in B indicates the contract number of certain business, and configuration checks mark specified by rule
In standard, the field length range of field B is 10-12.Section B's in the record B of the data is found to the data validation of input
Field length is 15, however, the reparation rule of configuration module 306 and the field B being not configured in record B, so to record B's
Repairing failure.
For example, for the business relevant field of some records, if important ratio is larger, occur abnormal with regard to needing
It manually to repair, for example check in standard specified by rule, the business relevant field H for recording H is domestic service, checks discovery
The business relevant field H of the record H of the data of input is overseas operation, and configuration module 306 is to the business related words of record H
The reparation rule of section H configuration is to need manually to repair, so at this time to the repairing failure of record B.
, can be after technical staff repair for the abnormal data in refusal log in the embodiment of the present invention, it can be with
Downstream components are sent to again through fault-tolerant component when needed.
It should be noted that can also be recorded respectively in alarm log and refusal log in the possible embodiment of the present invention
Each original value, field for abnormal field occur of saved record is corresponding to check regular code and timestamp
Etc. information.
The fault-tolerant processing process of the embodiment of the present invention is exemplified below.
Referring to fig. 4, in an application scenarios, during ETL, the data extracted from data source are table 1, and table 1 includes
Multiple records: record 1, record 2, record 3 etc., wherein record 1 includes field 11, field 12, field 13 etc., 2 packet of record
Field 21, field 22, field 23 etc. are included, record 3 includes field 31, field 32, field 33 etc., record 1, record 2, record
3 etc. are input to fault-tolerant component to record the form of stream, and fault-tolerant component checks module according to the fault-tolerant rule configured, to note
Record stream is checked, and determines that record 1 is normal data by checking；Determine that record 2 is abnormal data, field 22 is that appearance is abnormal
Field；Determine that record 3 is abnormal data, field 31 is abnormal field occur.So, for checking normal record 1 then
The downstream components for being output to ETL module continue with.For checking abnormal record 2 and record 3, then carried out by repair module
It repairs.For record 2, according to the reparation rule of configuration, repair module repairs successfully the field 22 of record 2, then repairs
Record 2 can continue to output downstream components, meanwhile, the record 2 that this is repaired replicates portion, is saved in the announcement of log module
In alert log.For record 3, repair module then records 3 and is saved directly to refusal log, without defeated to 3 repairing failures are recorded
Downstream components are arrived out.
Referring to Fig. 5, in another application scenarios, technical staff carries out people to the field 31 of the record 3 in refusal log
Work reparation, the record 3 repaired.Later, during ETL, the data extracted from data source are table 1, and table 1 includes multiple
Record: record 1, record 2, record 3 etc., wherein record 1 includes field 11, field 12, field 13 etc., and record 2 includes word
Section 21, field 22, field 23 etc., record 3 include field 31, field 32, field 33 etc., record 1, record 2, record 3 etc.
Deng being input to fault-tolerant component in the form of recording stream, fault-tolerant component checks module according to the fault-tolerant rule configured, to record
Stream is checked, and determines that record 1 is normal data by checking；Determine that record 2 is abnormal data, field 22 is exception occur
Field；Determine that record 3 is abnormal data.So, for check normal record 1 be then output to the downstream components of ETL module after
Continuous processing.For there is abnormal record 2, according to the reparation rule of configuration, then repaired by field 22 of the repair module to record 2
Multiple success, the then record 2 repaired can continue to output downstream components, optionally, the duplication of record 2 one that this is repaired
Part, be saved in the alarm log of log module (if original existing record 2 repaired in alarm log, it can also not
With preservation again).For there is abnormal record 3, fault-tolerant component discovery exists in refusal log by manually repairing
Record 3, so, fault-tolerant component, which replaces the record repaired 3 in record stream, there is abnormal record 3, and this is repaired
Record 3 continue to output downstream components.
It should be noted that record 1, record 2 involved in above-mentioned Fig. 4 and Fig. 5, record 3 etc. are only used for indicating not
Same record, and non-recorded title；Related field 21, field 22, field 23, field 21, field 22, field 23, word
Section 31, field 32, field 33 etc. are only used for indicating different fields, rather than the title of field.
It should also be noted that, above-mentioned Fig. 4 and Fig. 5 are only example, it is not construed as limitation of the invention.
As can be seen that in the embodiment of the present invention, during data are from data source to DWI, when the number of data source
When according to data quality problem occur, the fault-tolerant component of ETL (such as DataStage program assembly) provided in an embodiment of the present invention can be with
After data pick-up, fault-tolerant processing is carried out to data, checks data quality problem, and repair to data, to will repair
Data afterwards continue to be sent to downstream components.Implement the embodiment of the present invention, program can be reduced to a great extent because of the quality of data
Abnormal caused by problem, guarantee procedure operates normally, and realizes the configurability of fault-tolerant rule, customized property and expansible
Property, fault-tolerant component reaches code reuse, greatly reduces development amount.
Referring to Fig. 6, it is based on data warehouse as described above and fault-tolerant component, the embodiment of the invention provides one kind
Fault-tolerance processing method, this method include but is not limited to following steps:
The business datum that S601, acquisition are extracted from the data source of source system.
Source system is the integrated of database, that is to say, that source system may include thousands of data source, and data source is deposited
Contain data (such as business datum relevant to business).The data source can be general database, such as Oracle, My
SQL and SQL Server etc..
S602, the business datum is checked, whether exception occurs with the determination business datum.
In specific embodiment, before checking to the business datum, it is pre-configured with fault-tolerant rule, the fault-tolerant rule
Rule and reparation rule are checked for determining；Wherein, the rule that checks is for specifying the normal standard of the business datum, institute
Reparation rule is stated for mode used by specified restoration data；Then, the standard specified by rule is checked based on described
The business datum is checked.
In specific embodiment, checks the standard specified by rule based on described the business datum is checked,
Include: based on it is described check the standard specified by rule check the business datum technical parameter it is whether abnormal, it is described
Technical parameter include data type, field length, accuracy value, time and date value, in digit strings at least one of；With
And check the standard specified by rule based on described whether check business relevant field in the business datum abnormal,
The business relevant field include customer type, type of service, in quantity on order at least one of.
It should be noted that can refer to Fig. 3 about the detailed content for configuring fault-tolerant rule and being checked to business datum
In relation to configuration module 306 and the associated description for checking module (IT checks module 301 and business checks module 302) in embodiment,
Which is not described herein again.
S603, when determining that the business datum occurs abnormal, the business datum is repaired.
Specifically, the mode that can be specified based on be pre-configured with reparation rule repairs the business datum, so that
Business datum, which meets, preconfigured checks standard specified by rule.
In specific embodiment, the business datum includes table, and the table includes multiple records, determines the table by checking
It is middle there is abnormal record and do not occur abnormal record after, can to occur in the table abnormal record be based on reparation rule into
Row is repaired.
It should be noted that can refer to related in Fig. 3 embodiment repair about the detailed content repaired to business datum
The associated description of multiple module 304, which is not described herein again.
S604, the business datum after reparation is sent to downstream components.
In the embodiment of the present invention, after being repaired to the record for occurring exception in the table based on reparation rule, if to institute
State and abnormal record reparation success occur, then by the reparation, successfully record is saved in alarm log, and by it is described repair at
Do not occur abnormal record in the record and business datum of function and is sent to the downstream components；If there is abnormal record to described
The record of the repairing failure is then saved in refusal log, and will not occur abnormal record in business datum by repairing failure
It is sent to the downstream components.
It should be noted that being saved in alarm log about successfully record will be repaired and saving the record of repairing failure
Detailed content to refusal log can refer to the associated description in relation to log module 305 in Fig. 3 embodiment, and which is not described herein again.
Referring to Fig. 7, it is based on data warehouse as described above and fault-tolerant component, the embodiment of the invention provides another
Kind fault-tolerance processing method, this method includes but is not limited to following steps:
The business datum that S701, acquisition are extracted from data source.
S702, the business datum is checked.
In specific embodiment, before checking to the business datum, it is pre-configured with fault-tolerant rule, the fault-tolerant rule
Rule and reparation rule are checked for determining；Wherein, the rule that checks is for specifying the normal standard of the business datum, institute
Reparation rule is stated for mode used by specified restoration data；Then, the standard specified by rule is checked based on described
The business datum is checked.
In specific embodiment, checks the standard specified by rule based on described the business datum is checked,
It include: to check whether extremely the standard specified by rule checks the technical parameter of the business datum based on described.
It should be noted that can refer to Fig. 3 about the detailed content for configuring fault-tolerant rule and being checked to business datum
In relation to configuration module 306 and the associated description for checking module (IT checks module 301 and business checks module 302) in embodiment,
Which is not described herein again.
S703, duplicate removal is ranked up to the business datum.
In specific embodiment, the business datum is to include, and the table includes multiple records, each record is one corresponding
Key assignments can be ranked up according to the multiple record of key-value pair；If it find that there are the duplicate records of key assignments in table
Remove the duplicate record of the key assignments.
It should be noted that can refer in Fig. 3 embodiment about the detailed content for being ranked up duplicate removal to business datum has
The associated description of sequence deduplication module 303 is closed, which is not described herein again.
It should also be noted that, the embodiment of the present invention in, step S703 can also after step S701, step S702 it
Preceding execution, the present invention do not limit its and execute sequence.
S704, judge whether business datum exception occurs.Specifically, can judge business datum according to the rule that checks of configuration
Whether there is exception, if there is exception, thens follow the steps S705；If not occurring exception, S707 is thened follow the steps.
S705, the business datum is repaired.
Specifically, the mode that can be specified based on be pre-configured with reparation rule repairs the business datum, so that
Business datum, which meets, preconfigured checks standard specified by rule.
In specific embodiment, the business datum includes table, and the table includes multiple records, determines the table by checking
It is middle there is abnormal record and do not occur abnormal record after, can to occur in the table abnormal record be based on reparation rule into
Row is repaired.
It should be noted that can refer to related in Fig. 3 embodiment repair about the detailed content repaired to business datum
The associated description of multiple module 304, which is not described herein again.
S706, judge whether success is repaired to the business datum.If repairing successfully, S707 and S708 are thened follow the steps；
If repairing failure thens follow the steps S709.
S707, the data after reparation, data without exception are sent to downstream components.
S708, the data after repairing are recorded to alarm log.
Specifically, by the reparation, successfully record is saved in announcement if there is abnormal record reparation success to described
Alert log.
It should be noted that can refer to Fig. 3 reality about the detailed content that successfully record is saved in alarm log will be repaired
The associated description in relation to log module 305 in example is applied, which is not described herein again.
S709, records abnormal data to refusing log.
Specifically, the record of the repairing failure is saved in and is refused if there is abnormal record repairing failure to described
Exhausted log, and do not occur abnormal record by described and be sent to the downstream components.
It should be noted that can refer to Fig. 3 reality about the detailed content that the record of repairing failure is saved in refusal log
The associated description in relation to log module 305 in example is applied, which is not described herein again.
It should be noted that having no inevitable sequencing between above-mentioned steps S707, S708 and S709.
It should also be noted that, the way of realization of above-mentioned Fig. 6 and Fig. 7 embodiment of the method can also be implemented with reference to Fig. 4 and Fig. 5
The associated description of example.
As can be seen that during ETL of the business datum from data source to DWI, working as data in the embodiment of the present invention
When data quality problem occur in the data in source, fault-tolerant processing can be carried out to data, the quality of data is checked and ask after data pick-up
Topic, and data are repaired, so that the data after reparation be continued to be sent to downstream components.Implement the embodiment of the present invention, it can
Program is reduced to a great extent because abnormal caused by data quality problem, guarantee procedure is operated normally, and realizes fault-tolerant
Configurability, customized property and scalability, the fault-tolerant component of rule reach code reuse, greatly reduce development amount.
Based on same inventive concept, another fault-tolerant component provided in an embodiment of the present invention is described below.Referring to Fig. 8,
Fault-tolerant component 800 provided in an embodiment of the present invention is applied to extract in conversion load ETL operation process, comprising:
Receiving module 801, for obtaining the business datum extracted from data source, the business datum includes multiple records；
Module 802 is checked, for checking to the business datum, with record abnormal in the determination business datum
With normal record；
Repair module 803, for being repaired to the abnormal record；
Sending module 804 is sent to downstream components for that will repair successfully record and the normal record.
In the specific implementation, data integration and the integrated tool DataStage of IBM can be used, develop to obtain by C Plus Plus
The fault-tolerant component, that is to say, that the fault-tolerant component can be a kind of DataStage program assembly.It is fault-tolerant in specific deployment
Component includes the group of component (such as the fault-tolerant component of the ETL is deployed in server end operator component) and client of server end
Part (the stage component that such as fault-tolerant component of the ETL is deployed in client).
It should be noted that those skilled in the art can by the detailed description of the fault-tolerant component in earlier figures 3-5 embodiment
To be clear from the implementation method for each functional module that fault-tolerant component 800 is included, so in order to illustrate the succinct of book, herein
No longer it is described in detail.
Based on same inventive concept, the embodiment of the invention provides a kind of fault tolerance facilities 900, referring to Fig. 9, the fault-tolerant dress
Setting 900 includes processor 901, memory 902 and transmitter 903 and receiver 904, the processor 901, memory 902
It is connected with transmitter 903 and receiver 904 (as being connected with each other by bus).
Memory 902 include but is not limited to be random access memory (Random Access Memory, RAM), it is read-only
Memory (Read-Only Memory, ROM), Erasable Programmable Read Only Memory EPROM (Erasable Programmable
Read OnlyMemory, EPROM) or portable read-only memory (Compact Disc Read-Only Memory, CD-
ROM), which can be used for storing program therefor code and data (such as storage alarm log and refusal log).
For receiving data, transmitter 903 is for emitting data for receiver 904.
Processor 901 can be one or more central processing units (Central Processing Unit, CPU), locate
In the case that reason device 1301 is a CPU, which can be monokaryon CPU, be also possible to multi-core CPU.
The processor 901 is for reading the program code stored in the memory 902, to realize in Fig. 3 embodiment
The function of modules.
In the above-described embodiments, it can be realized wholly or partly by software, hardware, firmware or any combination.
When implemented in software, it can realize in the form of a computer program product in whole or in part.The computer program
Product includes one or more computer instructions, when loading on computers and executing the computer program instructions, all or
It partly generates according to process or function described in the embodiment of the present invention.The computer can be general purpose computer, dedicated meter
Calculation machine, computer network or other programmable devices.The computer instruction is storable in computer readable storage medium, or
Person is transmitted from a computer readable storage medium to another computer readable storage medium, for example, the computer instruction
Wired (such as coaxial cable, optical fiber, digital subscriber can be passed through from a website, computer, server or data center
Line) or wirelessly (such as infrared, microwave etc.) mode is passed to another website, computer, server or data center
It is defeated.The computer readable storage medium can be any usable medium that computer can access, and be also possible to comprising one
Or the data storage devices such as integrated server, data center of multiple usable mediums.The usable medium can be magnetic medium
(such as floppy disk, hard disk, tape etc.), optical medium (such as DVD etc.) or semiconductor medium (such as solid state hard disk) etc..
In the above-described embodiments, it emphasizes particularly on different fields to the description of each embodiment, there is no the part being described in detail in some embodiment,
Reference can be made to the related descriptions of other embodiments.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
Those familiar with the art in the technical scope disclosed by the present invention, can easily think of the change or the replacement, and should all contain
Lid is within protection scope of the present invention.Therefore, protection scope of the present invention should be based on the protection scope of the described claims.
Claims (10)
1. a kind of fault-tolerance processing method, which is characterized in that the method is applied to extract in conversion load ETL operation process, institute
The method of stating includes:
The business datum extracted from data source is obtained, the business datum includes multiple records；
The business datum is checked, with record and normal record abnormal in the determination business datum；
The abnormal record is repaired, successfully record will be repaired and the normal record is sent to downstream components.
2. the method according to claim 1, wherein being wrapped obtaining before the business datum that data source extracts
It includes: configuring fault-tolerant rule, the fault-tolerant rule checks rule and reparation rule for determining；Wherein, the rule that checks is used for
Determine that the record belongs to normal standard, the reparation rule is for determining side used by repairing the abnormal record
Formula；
The business datum is checked, specifically: based on the determining standard of rule that checks to the business number
According to being checked；
The abnormal record is repaired, specifically: based on the determining mode of the reparation rule to the exception
Record repaired so that the abnormal record meets the standard.
3. according to the method described in claim 2, it is characterized in that, checking the standard determined by rule based on described in institute
Business datum is stated to be checked, comprising:
Whether the technical parameter for checking the record in the business datum based on the standard for checking rule determination is abnormal,
The technical parameter include data type, field length, accuracy value, time and date value, in digit strings at least one of；
And based on the business relevant field for checking the standard that rule determines and checking the record in the business datum
It is whether abnormal, the business relevant field include customer type, type of service, in quantity on order at least one of.
4. method according to any one of claims 1 to 3, which is characterized in that repaired to the abnormal record
Later, further includes:
If will repair successfully record to the abnormal record reparation success and be saved in alarm log；
If the record of repairing failure is saved in refusal log, the refusal log to the abnormal record repairing failure
In the repairing failure record for manually being repaired.
5. method according to any one of claims 1 to 4, which is characterized in that obtain the business datum extracted from data source
Later, the method also includes:
It is ranked up according to the multiple record of key-value pair, wherein each records a corresponding key assignments；And
Remove the duplicate record of the key assignments.
6. a kind of fault-tolerant component, which is characterized in that the fault-tolerant component is applied to extract in conversion load ETL operation process, packet
It includes:
Receiving module, for obtaining the business datum extracted from data source, the business datum includes multiple records；
Module is checked, for checking to the business datum, with record abnormal in the determination business datum and normally
Record；
Repair module, for being repaired to the abnormal record；
Sending module is sent to downstream components for that will repair successfully record and the normal record.
7. fault-tolerant component according to claim 6, which is characterized in that the fault-tolerant component further includes configuration module, described
Configuration module checks rule and reparation rule for determining for configuring fault-tolerant rule, the fault-tolerant rule；Wherein, described to check
Rule is for determining that the record belongs to normal standard, and the reparation rule is for determining that repairing the abnormal record is adopted
Mode；
The module that checks for being checked to the business datum, specifically: the module that checks is for based on the inspection
The standard that core rule determines checks the business datum；
The repair module is used to repair the abnormal record, specifically: the repair module is used for based on described
The mode that reparation rule determines repairs the abnormal record, so that the abnormal record meets the mark
It is quasi-.
8. fault-tolerant component according to claim 7, which is characterized in that it is described check module for be based on described in check rule
The identified standard checks the business datum, comprising:
The module that checks is for checking the record in the business datum based on the standard for checking rule determination
Whether technical parameter is abnormal, and the technical parameter includes data type, field length, accuracy value, time and date value, numeric word
At least one of in symbol string；
The module that checks is also used to check the record in the business datum based on the standard for checking rule determination
Business relevant field it is whether abnormal, the business relevant field include customer type, type of service, in quantity on order at least
One.
9. according to the described in any item fault-tolerant components of claim 6 to 8, which is characterized in that the fault-tolerant component further includes log
Module, the log module include alarm log and refusal log；
The log module is used for, if will repair successfully record to the abnormal record reparation success and be saved in alarm
Log；If the record of repairing failure is saved in refusal log, the refusal log to the abnormal record repairing failure
In the repairing failure record for manually being repaired.
10. according to the described in any item fault-tolerant components of claim 6 to 9, which is characterized in that the fault-tolerant component further includes sequence
Deduplication module, the sequence deduplication module are used to be ranked up according to the multiple record of key-value pair, and, remove the key assignments
Duplicate record；Wherein, each records a corresponding key assignments.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710761219.7A CN109426576B (en) | 2017-08-30 | 2017-08-30 | Fault-tolerant processing method and fault-tolerant assembly |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710761219.7A CN109426576B (en) | 2017-08-30 | 2017-08-30 | Fault-tolerant processing method and fault-tolerant assembly |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109426576A true CN109426576A (en) | 2019-03-05 |
| CN109426576B CN109426576B (en) | 2022-03-29 |
Family
ID=65503838
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710761219.7A Active CN109426576B (en) | 2017-08-30 | 2017-08-30 | Fault-tolerant processing method and fault-tolerant assembly |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109426576B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110795479A (en) * | 2019-10-08 | 2020-02-14 | 中国建设银行股份有限公司 | Method and device for distributed ETL scheduling based on data |
| CN114741577A (en) * | 2022-04-27 | 2022-07-12 | 广域铭岛数字科技有限公司 | Business data management method, system, electronic equipment and readable storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101533407A (en) * | 2009-04-10 | 2009-09-16 | 中国科学院软件研究所 | Method for detecting exceptional data in ETL flow |
| CN103473143A (en) * | 2012-06-07 | 2013-12-25 | 中兴通讯股份有限公司 | Fault correcting treatment, fault correcting method and device |
| CN103647804A (en) * | 2013-11-22 | 2014-03-19 | 华为技术有限公司 | Method for data processing of storage unit, device and system |
| CN103823869A (en) * | 2014-02-26 | 2014-05-28 | 中科怡海高新技术发展江苏股份公司 | Data extracting and predicting model establishing method for environment monitoring |
| CN104462082A (en) * | 2013-09-12 | 2015-03-25 | 深圳中科金证科技有限公司 | Data warehouse based medical data integration method and system |
| CN105389649A (en) * | 2015-10-21 | 2016-03-09 | 中国电力科学研究院 | Electric energy quality abnormal data detection and processing method |
| CN105787089A (en) * | 2016-03-15 | 2016-07-20 | 国家电网公司 | Power distribution network scheduling foundation data integrating method |
| US20160350390A1 (en) * | 2013-08-06 | 2016-12-01 | International Business Machines Corporation | Post-migration validation of etl jobs and exception management |
| CN107025224A (en) * | 2016-01-29 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of monitor task operation |
-
2017
- 2017-08-30 CN CN201710761219.7A patent/CN109426576B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101533407A (en) * | 2009-04-10 | 2009-09-16 | 中国科学院软件研究所 | Method for detecting exceptional data in ETL flow |
| CN103473143A (en) * | 2012-06-07 | 2013-12-25 | 中兴通讯股份有限公司 | Fault correcting treatment, fault correcting method and device |
| US20160350390A1 (en) * | 2013-08-06 | 2016-12-01 | International Business Machines Corporation | Post-migration validation of etl jobs and exception management |
| CN104462082A (en) * | 2013-09-12 | 2015-03-25 | 深圳中科金证科技有限公司 | Data warehouse based medical data integration method and system |
| CN103647804A (en) * | 2013-11-22 | 2014-03-19 | 华为技术有限公司 | Method for data processing of storage unit, device and system |
| CN103823869A (en) * | 2014-02-26 | 2014-05-28 | 中科怡海高新技术发展江苏股份公司 | Data extracting and predicting model establishing method for environment monitoring |
| CN105389649A (en) * | 2015-10-21 | 2016-03-09 | 中国电力科学研究院 | Electric energy quality abnormal data detection and processing method |
| CN107025224A (en) * | 2016-01-29 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of monitor task operation |
| CN105787089A (en) * | 2016-03-15 | 2016-07-20 | 国家电网公司 | Power distribution network scheduling foundation data integrating method |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110795479A (en) * | 2019-10-08 | 2020-02-14 | 中国建设银行股份有限公司 | Method and device for distributed ETL scheduling based on data |
| CN114741577A (en) * | 2022-04-27 | 2022-07-12 | 广域铭岛数字科技有限公司 | Business data management method, system, electronic equipment and readable storage medium |
| CN114741577B (en) * | 2022-04-27 | 2023-08-11 | 广域铭岛数字科技有限公司 | Service data management method, system, electronic equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109426576B (en) | 2022-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102236672B (en) | A kind of data lead-in method and device | |
| CN107506451A (en) | abnormal information monitoring method and device for data interaction | |
| US20170109657A1 (en) | Machine Learning-Based Model for Identifying Executions of a Business Process | |
| CN107818431B (en) | Method and system for providing order track data | |
| US20170109676A1 (en) | Generation of Candidate Sequences Using Links Between Nonconsecutively Performed Steps of a Business Process | |
| US20170109668A1 (en) | Model for Linking Between Nonconsecutively Performed Steps in a Business Process | |
| CN106656536A (en) | Method and device for processing service invocation information | |
| US20170109667A1 (en) | Automaton-Based Identification of Executions of a Business Process | |
| US20170109636A1 (en) | Crowd-Based Model for Identifying Executions of a Business Process | |
| US9152627B2 (en) | Automatic rule generation | |
| CN109241014B (en) | Data processing method and device and server | |
| US20200184026A1 (en) | Computing system simulation and testing environment | |
| US9384045B2 (en) | Intelligent inclusion/exclusion automation | |
| US20170109638A1 (en) | Ensemble-Based Identification of Executions of a Business Process | |
| CN114880405A (en) | Data lake-based data processing method and system | |
| CN106095678A (en) | Automatization's result inspection method of data bank service operation under windows platform | |
| CN113868498A (en) | Data storage method, electronic device, device and readable storage medium | |
| CN113326247A (en) | Cloud data migration method and device and electronic equipment | |
| US20080189150A1 (en) | Supply chain multi-dimensional serial containment process | |
| CN112506771A (en) | Message comparison method and device | |
| KR101253335B1 (en) | Method for constructing database based on warehouse and database system therof | |
| CN109426576A (en) | Fault-tolerance processing method and fault-tolerant component | |
| US20170109640A1 (en) | Generation of Candidate Sequences Using Crowd-Based Seeds of Commonly-Performed Steps of a Business Process | |
| US20170109637A1 (en) | Crowd-Based Model for Identifying Nonconsecutive Executions of a Business Process | |
| CN116719799A (en) | Environment-friendly data management method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |