CN104298773B - A kind of ETL operations automatically cut storehouse system and method - Google Patents
A kind of ETL operations automatically cut storehouse system and method Download PDFInfo
- Publication number
- CN104298773B CN104298773B CN201410601466.7A CN201410601466A CN104298773B CN 104298773 B CN104298773 B CN 104298773B CN 201410601466 A CN201410601466 A CN 201410601466A CN 104298773 B CN104298773 B CN 104298773B
- Authority
- CN
- China
- Prior art keywords
- etl
- error
- operations
- report
- storehouse
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Hardware Redundancy (AREA)
Abstract
The present invention relates to a kind of ETL operations automatically cut storehouse system and method, including each ETL operation execution situations of periodic scanning；Manual intervention information is pieced together out according to the job information for the ETL operations that report an error when discovery reports an error ETL operations；The data syn-chronization situation of periodic reinvestigation backup library, judge the working condition of backup library, generated if in data syn-chronization completion status and cut storehouse triggering command, the operation object for the ETL operations that report an error is switched into backup library by storage facility located at processing plant, Qie Ku is completed to change job configuration information according to manual intervention information, and ETL operations are performed in backup library；The present invention is after operation reports an error, automatically Job Operations object is switched to backup library by storage facility located at processing plant, ensure that operation is normally extracted, so as to ensure the stability of data pick-up, support the data promptness of down-stream system, save human cost, hence it is evident that improve operating efficiency, make system maintenance more hommization, intellectuality.
Description
Technical field
The present invention relates to field of computer technology, more particularly to a kind of ETL operations automatically cut storehouse system and method.
Background technology
ETL operations refer to distribution, data (such as relation data, panel data in heterogeneous data source using ETL instruments
File etc.) it is drawn into behind interim intermediate layer and is cleaned, change, integrate, finally it is loaded into data warehouse or Data Mart, into
For the basis of on-line analytical processing, data mining.
System important at present can all have storage facility located at processing plant and backup library (BCV storehouses), and backup library storage produces a certain moment
Data of taking pictures, be used as emergent.ETL instruments are all to be obtained immediately from storage facility located at processing plant the peak time for avoiding data manipulation mostly
Data of taking pictures, but in the present information highly developed epoch, the data manipulation of storage facility located at processing plant is quite frequent, when can cause to extract
Between longer ETL operations due to dirty mistake of reading the newspaper, so as to not take out data, only by being manually switched to backup library extraction number
According to.Which expends a large amount of manpowers, realizes that efficiency is low, and can not ensure the stability and reliability of operation.
The content of the invention
The technical problems to be solved by the invention are in view of the shortcomings of the prior art, there is provided storehouse is cut in a kind of ETL operations automatically
System and method, realize in ETL operations, database automatically switches to ensure that ETL operations smoothly complete.
The technical scheme that the present invention solves above-mentioned technical problem is as follows：Storehouse system, including production are cut in a kind of ETL operations automatically
Storehouse, backup library, ETL operation execution modules, ETL operations scan module, backup library monitoring module and ETL cut storehouse processing mould
Block；
The storage facility located at processing plant, it is used to provide data source for ETL operations；
The backup library, its data being used in schedule backup storage facility located at processing plant, when ETL operations mistake occur in the operation of storage facility located at processing plant
The ETL operations for mistaking to report an error provide data source；
The ETL operation execution modules, it is used for from storage facility located at processing plant, and data carry out ETL processing needed for extraction, in storage facility located at processing plant
Operation cut storehouse triggering command by the operation object for the ETL operations that report an error by giving birth to according to what ETL cut that storehouse processing module sends when reporting an error
Product storehouse is switched to backup library, and then the manual intervention information modification job configuration information of storehouse processing module transmission is cut according to ETL,
The ETL operations that report an error are performed in backup library；
The ETL operations scan module, it is used for the situation that periodic scanning ETL operation execution modules perform each ETL operations,
When discovery reports an error ETL operations, ETL is sent to according to the job information for the ETL operations that report an error generation manual intervention information and cuts storehouse
Processing module；
The backup library monitoring module, it is used to periodically verify the synchronous situation of backup library, and will backup
The status information in storehouse is sent to ETL and cuts storehouse processing module；
The ETL cuts storehouse processing module, and it is used for after the manual intervention information of ETL operations scan module transmission is received,
The state of backup library is judged according to backup library status information, when backup library is in data syn-chronization completion status, generation is cut storehouse and touched
Send instructions and be sent to ETL and cut storehouse processing module, after ETL operation execution modules complete to cut storehouse operation, manual intervention information is sent out
Give ETL operation execution modules.
The beneficial effects of the invention are as follows：The present invention is automatically switched Job Operations object by storage facility located at processing plant after operation reports an error
To backup library, ensure that operation is normally extracted, so as to ensure the stability of data pick-up, support the data of down-stream system timely
Property, save human cost, hence it is evident that improve operating efficiency, make system maintenance more hommization, intellectuality.
On the basis of above-mentioned technical proposal, the present invention can also do following improvement.
Further, the ETL cuts storehouse processing module and is additionally operable to insert this after ETL operation execution modules perform and cut storehouse operation
Before the manual intervention information of secondary operation, this is detected according to the ETL operation execution situations that report an error of ETL operations scan module acquisition and reported an error
ETL operations, which whether there is, successfully to be recorded, and production is switched back into by backup library if there is the operation object then by the ETL operations that report an error
Storehouse, if there is no then further detecting whether the ETL operations that report an error are currently running, waited if being currently running, until fortune
Row is completed to judge whether to run successfully again, and the operation object by the ETL operations that report an error if running successfully is switched back into by backup library
Storage facility located at processing plant, otherwise control ETL operation execution modules to remove already present temporary file and manual intervention information, insert this operation
Manual intervention information, and then ETL operation execution modules are performed according to this manual intervention information and reported an error ETL operations.
Using the beneficial effect of above-mentioned further scheme：Interim text is removed before the manual intervention of this operation is inserted
Part, it is to ensure extraction data accuracy, avoiding can be the temporary file of preceding subjob when being run again after failing because of operation
Also calculate, and then cause the generation of duplicate data；Manual intervention information before cleaning is to ensure that manual intervention is believed
The correctness of insertion is ceased, avoids triggering reporting an error for database unique constraints because repeatedly inserting identical manual intervention information.
Further, the ETL operations scan module is additionally operable to when scanning performs completion to the ETL operations that report an error in backup library
When, the notice that storehouse processing module sends the ETL operations processing that reports an error and complete is cut to ETL, the ETL cuts storehouse processing module according to ETL
The notice generation that scan module is sent cuts storehouse triggering command and is sent to ETL operation execution modules, and ETL operation execution modules are by ETL
The operation object of operation is by backup library switchback storage facility located at processing plant.
Further, the ETL operation execution modules internal memory contains allocation list, and allocation list phase is changed according to manual intervention information
Content is answered, is performed in backup library.
Further, the ETL operations scan module be detected simultaneously by it is more than one report an error ETL operations when, by respectively reporting an error
The storage facility located at processing plant generation job number list of ETL Job Operations, the respectively each ETL operations that report an error of processing.
The technical scheme that the present invention solves above-mentioned technical problem is as follows：Storehouse method is cut in a kind of ETL operations automatically, including as follows
Step：
Step 1, each ETL operation execution situations of periodic scanning；
Step 2, the ETL operations that report an error are judged whether, if there is performing step 3, otherwise return to step 1；
Step 3, manual intervention information is pieced together out according to the job information for the ETL operations that report an error；
Step 4, the data syn-chronization situation of periodic reinvestigation backup library, the working condition of backup library is judged, if being in data
Synchronously complete state and then perform step 5；Waited if data syn-chronization does not complete, until data syn-chronization is completed and then performs step
5；
Step 5, generation cuts storehouse triggering command and the operation object for the ETL operations that report an error is switched into backup library, Qie Ku by storage facility located at processing plant
Complete to change job configuration information according to manual intervention information, ETL operations are performed in backup library.
On the basis of above-mentioned technical proposal, the present invention can also do following improvement.
Further, also need to proceed as follows before performing the ETL that reports an error in backup library in step 5：
Step 51, obtain and report an error ETL operation execution situations, detect the ETL operations that report an error and whether there is and successfully record, if
In the presence of then execution step 55；If there is no then execution step 52；
Step 52, detect whether the ETL operations that report an error are currently running, waited if being currently running, until operation is completed
Judge whether to run again successfully, step 55 is performed if running successfully；Otherwise step 53 is performed；
Step 53, already present temporary file and manual intervention information are removed, the manual intervention for inserting this operation is carried out,
And then job configuration information is changed according to manual intervention information, the ETL operations that report an error are performed in backup library；
Step 54, judge whether the ETL operations that report an error run succeeded in backup library, step 55 is performed if success；It is no
Then return to step 53；
Step 55, the operation object for the ETL operations that report an error is switched back into storage facility located at processing plant by backup library, terminates flow.
Further, above-mentioned technical proposal is also included when scanning performs completion to the ETL operations that report an error in backup library, and generation is cut
Storehouse trigger command is by the operation object for the ETL operations that report an error by backup library switchback storage facility located at processing plant.
Further, the ETL operations that report an error are triggered according to manual intervention information modification allocation list corresponding contents to hold in backup library
OK.
Further, above-mentioned technical proposal also include when check simultaneously it is more than one report an error ETL operations when, by respectively reporting an error
The storage facility located at processing plant generation job number list of ETL Job Operations, the respectively each ETL operations that report an error of processing.
Brief description of the drawings
Fig. 1 is that storehouse system block diagram is cut in a kind of ETL operations of the present invention automatically；
Fig. 2 is that storehouse method flow diagram is cut in a kind of ETL operations of the present invention automatically.
In accompanying drawing, the list of parts representated by each label is as follows：
100th, storage facility located at processing plant, 200, backup library, 300, ETL operation execution modules, 400, ETL operation scan modules, 500, standby
Part storehouse monitoring module, 600, ETL cut storehouse processing module.
Embodiment
The principle and feature of the present invention are described below in conjunction with accompanying drawing, the given examples are served only to explain the present invention, and
It is non-to be used to limit the scope of the present invention.
As shown in figure 1, storehouse system, including storage facility located at processing plant 100, backup library 200, ETL Job executions are cut in a kind of ETL operations automatically
Module 300, ETL operations scan module 400, backup library monitoring module 500 and ETL cut storehouse processing module 600；The production
Storehouse 100, it is used to provide data source for ETL operations；The backup library 200, its number being used in schedule backup storage facility located at processing plant 100
According to when ETL operation of the ETL operations when the operation of storage facility located at processing plant 100 occurs wrong to report an error provides data source；The ETL operations
Execution module 300, it is used for from storage facility located at processing plant 100, and data carry out ETL processing needed for extraction, are reported an error in the operation of storage facility located at processing plant 100
When according to ETL cut storehouse processing module 600 transmission cut storehouse triggering command by the operation object for the ETL operations that report an error by storage facility located at processing plant 100
Backup library 200 is switched to, and then the manual intervention information modification job configuration information of the transmission of storehouse processing module 600 is cut according to ETL,
The ETL operations that report an error are performed in backup library 200；The ETL operations scan module 400, it is used for periodic scanning ETL Job executions
Module 300 performs the situation of each ETL operations, when discovery reports an error ETL operations, is given birth to according to the job information for the ETL operations that report an error
ETL, which is sent to, into manual intervention information cuts storehouse processing module 600；The backup library monitoring module 500, it is used for periodically right
The synchronous situation of backup library 200 is verified, and the status information of backup library 200 is sent into ETL and cuts storehouse processing module 600；
The ETL cuts storehouse processing module 600, and it is used for after the manual intervention information of the transmission of ETL operations scan module 400 is received, root
The state of backup library is judged according to the status information of backup library 200, when backup library is in data syn-chronization completion status, generation is cut storehouse and touched
Send instructions and be sent to ETL and cut storehouse processing module, after ETL operation execution modules complete to cut storehouse operation, manual intervention information is sent out
Give ET L operation execution modules 300.
The ETL cuts storehouse processing module 600 and is additionally operable to insert this after ETL operation execution modules 300 perform and cut storehouse operation
Before the manual intervention information of secondary operation, the ETL operation execution situations detection that reports an error obtained according to ETL operations scan module 400 is somebody's turn to do
The ETL operations that report an error, which whether there is, successfully to be recorded, and is brought back to life if there is then switching the operation object for the ETL operations that report an error by backup library
Product storehouse, if there is no then further detecting whether the ETL operations that report an error are currently running, waited if being currently running, until
Operation is completed to judge whether to run successfully again, and the operation object by the ETL operations that report an error if running successfully is switched by backup library
Retrogradation product storehouse, ETL operation execution modules 300 are otherwise controlled to remove already present temporary file and manual intervention information, insertion is originally
The manual intervention information of secondary operation, and then ETL operation execution modules 300 perform the ETL that reports an error according to this manual intervention information and made
Industry.
The ETL operations scan module 400 is additionally operable to when scanning performs completion to the ETL operations that report an error in backup library 200,
The notice that storehouse processing module 600 sends the ETL operations processing that reports an error and complete is cut to ETL, the ETL cuts the basis of storehouse processing module 600
The notice generation that ETL scan modules are sent cuts storehouse triggering command and is sent to ETL operation execution modules 300, ETL operation execution modules
300 by the operation object of ETL operations by the switchback storage facility located at processing plant 100 of backup library 200.
The internal memory of ETL operation execution modules 300 contains job information allocation list, is changed and configured according to manual intervention information
Table corresponding contents, are performed in backup library.The manual intervention information can customize the time for the ETL job runs that report an error, frequency
And number.
On the ETL operations scan module 400 simultaneously check it is more than one report an error ETL operations when, by the ETL that respectively reports an error
The storage facility located at processing plant generation job number list of Job Operations, the respectively each ETL operations that report an error of processing.
As shown in Fig. 2 storehouse method is cut in a kind of ETL operations automatically, comprise the following steps：
Step 1, each ETL operation execution situations of periodic scanning；
Step 2, the ETL operations that report an error are judged whether, if there is performing step 3, otherwise return to step 1；
Step 3, manual intervention information is pieced together out according to the job information for the ETL operations that report an error；
Step 4, the data syn-chronization situation of periodic reinvestigation backup library, the working condition of backup library is judged, if being in data
Synchronously complete state and then perform step 5；Waited if data syn-chronization does not complete, until data syn-chronization is completed and then performs step
5；
Step 5, generation cuts storehouse triggering command and the operation object for the ETL operations that report an error is switched into backup library, Qie Ku by storage facility located at processing plant
Complete to change job configuration information according to manual intervention information, ETL operations are performed in backup library.
Also need to proceed as follows before performing the ETL that reports an error in backup library in step 5：
Step 51, obtain and report an error ETL operation execution situations, detect the ETL operations that report an error and whether there is and successfully record, if
In the presence of then execution step 55；If there is no then execution step 52；
Step 52, detect whether the ETL operations that report an error are currently running, waited if being currently running, until operation is completed
Judge whether to run again successfully, step 55 is performed if running successfully；Otherwise step 53 is performed；
Step 53, already present temporary file and manual intervention information are removed, the manual intervention for inserting this operation is carried out,
And then job configuration information is changed according to manual intervention information, the ETL operations that report an error are performed in backup library；
Step 54, judge whether the ETL operations that report an error run succeeded in backup library, step 55 is performed if success；It is no
Then return to step 53；
Step 55, the operation object for the ETL operations that report an error is switched back into storage facility located at processing plant by backup library, terminates flow.
Examine to whether there is to look into again after Qie Ku and successfully record, to ensure that job state does not change during Qie Ku, because
To have handled the ETL operations that report an error by hand in view of possible someone, ETL job states are made to become successfully, so there is no need to again
Performed in backup database, therefore switchback storage facility located at processing plant again.
Above-mentioned technical proposal is also included when scanning performs completion to the ETL operations that report an error in backup library, and storehouse triggering is cut in generation
Order is by the operation object for the ETL operations that report an error by backup library switchback storage facility located at processing plant.
The ETL operations that report an error are triggered according to job configuration information in manual intervention information modification allocation list to hold in backup library
OK.Temporary file is generated during job run, constantly chases after the data extracted in database in data extraction process
It is added in temporary file, job run success, temporary file, which no longer increases, is automatically converted into formal file.The operation is matched somebody with somebody
Confidence, which ceases, include some information of job run, such as job number, database, extracts table name, generation filename, the time started,
End time etc..The manual intervention information can customize time, frequency and the number of ETL job runs.
Above-mentioned technical proposal also include when simultaneously check it is more than one report an error ETL operations when, by the ETL operations that respectively report an error
The storage facility located at processing plant generation job number list of operation, the respectively each ETL operations that report an error of processing.
The present invention, which can realize, can automatically detect the database met using needs, so that foundation connects automatically, and from number
The data required according to application is extracted in storehouse.
The technical program core includes three parts：1) the ETL job informations that report an error scan；2) the synchronous feelings of backup library (BCV storehouses)
Condition is verified；3) cut storehouse automatically and insert intervention information.
1) the ETL job informations that report an error scan
One crash time point, timing scan ETL operation performances, if do not had are evaluated according to the instantaneity of operation
Report an error ETL operations, then continues to scan on, and if the ETL operations that report an error, is risked according to the job information for the ETL operations that report an error artificial dry
Pre-information, for reforming ETL operations.
Wherein, manual intervention information simulates artificial forced service operation, is a trigger switch of ETL Job executions, people
Work intervenes information can be with one ETL operation of enforceable operation.As long as the self-defined job run of information can is intervened in insertion
Time, frequency and number etc..
2) backup library (BCV storehouses) synchronous situation is verified
Backup library is all to synchronize daily, but situation of the backup library there is also lock in time early or late, so insertion
First the synchronous situation of backup library is confirmed before manual intervention information, prevents from not reaching expected effect after database is switched
Fruit.The inspection is regular check, is checked at regular intervals once, is prepared to cut storehouse.
3) cut storehouse automatically and insert manual intervention information
After the instruction that backup library synchronously completes is obtained, the storage facility located at processing plant that related operation is extracted just is switched to backup library,
Ensure that the operation (is handled by hand really without success charge book and in the case of not being currently running in view of possible someone
The ETL operations that report an error, make ETL job states become successfully, so there is no need to performed in backup database again), insertion
Operation has been adjusted and has extracted data by manual intervention information again, and after job run, tracking is timed to job run situation, with
Ensure that operation can successfully terminate, after end by the operation object of ETL operations again again switchback storage facility located at processing plant (due to back up stock
In synchronous evening, compared to storage facility located at processing plant, the integrality of data is slightly worse, so preferentially extracted when extracting from storage facility located at processing plant).
Record includes the corresponding relation of storage facility located at processing plant and backup library in allocation list, for example A storehouses corresponding A storehouse BCV, B storehouse corresponds to B storehouses
BCV。
It is the difference because different operations is probably to be extracted from different storage facility located at processing plants by storehouse generation job number list
Storage facility located at processing plant correspond to different backup libraries, the step is will to be switched to different BCV storehouses to distinguish different work.
The foregoing is only presently preferred embodiments of the present invention, be not intended to limit the invention, it is all the present invention spirit and
Within principle, any modification, equivalent substitution and improvements made etc., it should be included in the scope of the protection.
Claims (10)
1. storehouse system is cut in a kind of ETL operations automatically, it is characterised in that including storage facility located at processing plant, backup library, ETL operation execution modules,
ETL operations scan module, backup library monitoring module and ETL cut storehouse processing module；
The storage facility located at processing plant, it is used to provide data source for ETL operations；
The backup library, its data being used in schedule backup storage facility located at processing plant, when ETL operations occur wrong in the operation of storage facility located at processing plant
ETL operations to report an error provide data source；
The ETL operation execution modules, it is used for from storage facility located at processing plant, and data carry out ETL processing needed for extraction, in the behaviour of storage facility located at processing plant
Storehouse triggering command is cut by the operation object for the ETL operations that report an error by storage facility located at processing plant according to what ETL cut that storehouse processing module sends when reporting an error
Backup library is switched to, and then the manual intervention information modification job configuration information of storehouse processing module transmission is cut according to ETL, is being backed up
The ETL operations that report an error are performed in storehouse；
The ETL operations scan module, it is used for the situation that periodic scanning ETL operation execution modules perform each ETL operations, works as hair
It is existing report an error ETL operations when, according to the job information for the ETL operations that report an error generate manual intervention information be sent to ETL cut storehouse processing
Module；
The backup library monitoring module, it is used to periodically verify the synchronous situation of backup library, and by backup library
Status information is sent to ETL and cuts storehouse processing module；
The ETL cuts storehouse processing module, and it is used for after the manual intervention information of ETL operations scan module transmission is received, according to
Backup library status information judges the state of backup library, and when backup library is in data syn-chronization completion status, generation is cut storehouse triggering and referred to
Order is sent to ETL and cuts storehouse processing module, and after ETL operation execution modules complete to cut storehouse operation, manual intervention information is sent to
ETL operation execution modules；
Wherein, time, frequency and the number of the self-defined message ETL job runs that report an error of manual intervention information；Job configuration information
Including job number, database, extract table name, generation filename, time started and end time.
2. storehouse system is cut in a kind of ETL operations according to claim 1 automatically, it is characterised in that the ETL cuts storehouse processing mould
Block is additionally operable to before the manual intervention information of this operation is inserted after storehouse operation is cut in the execution of ETL operation execution modules, is made according to ETL
The ETL operation execution situations that report an error that industry scan module obtains, which detect the ETL operations that report an error and whether there is, successfully to be recorded, if there is
Then the operation object by the ETL operations that report an error switches back into storage facility located at processing plant by backup library, if there is no then further detecting the ETL that reports an error
Whether operation is currently running, and is waited if being currently running, until operation complete judge whether run successfully again, if run into
Operation object of the work(then by the ETL operations that report an error switches back into storage facility located at processing plant by backup library, otherwise controls ETL operation execution modules to remove
Existing temporary file and manual intervention information, insert the manual intervention information of this operation, and then ETL operation execution module roots
The ETL operations that report an error are performed according to this manual intervention information.
3. storehouse system is cut in a kind of ETL operations according to claim 1 automatically, it is characterised in that the ETL operations scan mould
Block is additionally operable to, when scanning performs completion to the ETL operations that report an error in backup library, cut storehouse processing module to ETL and send the ETL works that report an error
The notice that industry processing is completed, the ETL cut the notice generation that storehouse processing module is sent according to ETL scan modules and cut storehouse triggering command
ETL operation execution modules are sent to, ETL operation execution modules are by the operation object of ETL operations by backup library switchback storage facility located at processing plant.
4. storehouse system is cut in a kind of ETL operations according to claim 1 automatically, it is characterised in that the ETL Job executions mould
Block internal memory contains allocation list, and allocation list corresponding contents are changed according to manual intervention information, and the ETL works that report an error are performed in backup library
Industry.
5. storehouse system is cut in a kind of ETL operations according to claim 1 automatically, it is characterised in that the ETL operations scan mould
Block be detected simultaneously by it is more than one report an error ETL operations when, by the ETL Job Operations that respectively report an error storage facility located at processing plant generation job number row
Table, the respectively each ETL operations that report an error of processing.
6. storehouse method is cut in a kind of ETL operations automatically, it is characterised in that is comprised the following steps：
Step 1, each ETL operation execution situations of periodic scanning；
Step 2, the ETL operations that report an error are judged whether, if there is performing step 3, otherwise return to step 1；
Step 3, manual intervention information is pieced together out according to the job information for the ETL operations that report an error；
Step 4, the data syn-chronization situation of periodic reinvestigation backup library, the working condition of backup library is judged, if being in data syn-chronization
Completion status then performs step 5；Waited if data syn-chronization does not complete, until data syn-chronization is completed and then performs step 5；
Step 5, generation cuts storehouse triggering command and the operation object for the ETL operations that report an error is switched into backup library by storage facility located at processing plant, cuts storehouse completion
Job configuration information is changed according to manual intervention information, ETL operations are performed in backup library；
Wherein, time, frequency and the number of the self-defined message ETL job runs that report an error of manual intervention information；Job configuration information
Including job number, database, extract table name, generation filename, time started and end time.
7. storehouse method is cut in a kind of ETL operations according to claim 6 automatically, it is characterised in that in step 5 in backup library
Execution reports an error and also needs to proceed as follows before ETL：
Step 51, obtain and report an error ETL operation execution situations, detect the ETL operations that report an error and whether there is and successfully record, if there is
Then perform step 55；If there is no then execution step 52；
Step 52, detect whether the ETL operations that report an error are currently running, waited if being currently running, until operation is completed to sentence again
It is disconnected whether to run success, perform step 55 if running successfully；Otherwise step 53 is performed；
Step 53, already present temporary file and manual intervention information are removed, the manual intervention for inserting this operation is carried out, and then
Job configuration information is changed according to manual intervention information, the ETL operations that report an error are performed in backup library；
Step 54, judge whether the ETL operations that report an error run succeeded in backup library, step 55 is performed if success；Otherwise return
Return step 53；
Step 55, the operation object for the ETL operations that report an error is switched back into storage facility located at processing plant by backup library, terminates flow.
8. storehouse method is cut in a kind of ETL operations according to claim 6 automatically, it is characterised in that is also included when scanning to report
When backup library performs completion, generation cuts storehouse trigger command and cuts the operation object for the ETL operations that report an error by backup library for wrong ETL operations
Retrogradation product storehouse.
9. storehouse method is cut in a kind of ETL operations according to claim 6 automatically, it is characterised in that according to manual intervention information
Modification allocation list corresponding contents trigger the ETL operations that report an error and performed in backup library.
10. storehouse method is cut in a kind of ETL operations according to claim 6 automatically, it is characterised in that also includes checking simultaneously
To it is more than one report an error ETL operations when, by the ETL Job Operations that respectively report an error storage facility located at processing plant generate job number list, handle respectively
Each ETL operations that report an error.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410601466.7A CN104298773B (en) | 2014-10-30 | 2014-10-30 | A kind of ETL operations automatically cut storehouse system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410601466.7A CN104298773B (en) | 2014-10-30 | 2014-10-30 | A kind of ETL operations automatically cut storehouse system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104298773A CN104298773A (en) | 2015-01-21 |
| CN104298773B true CN104298773B (en) | 2018-01-09 |
Family
ID=52318498
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410601466.7A Active CN104298773B (en) | 2014-10-30 | 2014-10-30 | A kind of ETL operations automatically cut storehouse system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104298773B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107358334B (en) * | 2017-05-25 | 2021-02-09 | 全球能源互联网研究院有限公司 | Data accuracy determination method, device, terminal and computer-readable storage medium |
| CN108710684B (en) * | 2018-05-21 | 2023-05-30 | 平安科技（深圳）有限公司 | ETL task data source switching method, system, computer equipment and storage medium |
| CN111127192B (en) * | 2019-12-28 | 2023-10-27 | 辽宁振兴银行股份有限公司 | Distributed framework-based proxy routing +Zuul gateway |
| CN113392078A (en) * | 2021-06-04 | 2021-09-14 | 上海浦东发展银行股份有限公司 | Method and device for monitoring operation change state, computer equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101075304A (en) * | 2006-05-18 | 2007-11-21 | 河北全通通信有限公司 | Method for constructing decision supporting system of telecommunication industry based on database |
| CN101587477A (en) * | 2008-05-23 | 2009-11-25 | 阿里巴巴集团控股有限公司 | Method and system for automatically maintaining ETL modules |
-
2014
- 2014-10-30 CN CN201410601466.7A patent/CN104298773B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101075304A (en) * | 2006-05-18 | 2007-11-21 | 河北全通通信有限公司 | Method for constructing decision supporting system of telecommunication industry based on database |
| CN101587477A (en) * | 2008-05-23 | 2009-11-25 | 阿里巴巴集团控股有限公司 | Method and system for automatically maintaining ETL modules |
Non-Patent Citations (2)
| Title |
|---|
| "An MAS-based And Fault-Tolerant Distributed ETL Workflow Engine";jinluan Huang等;《Proceedings of the 2012 IEEE 16th International Conference on Computer Supported Cooperative Work in Design》;20120525;第54-58页 * |
| "基于数据库镜像的高可用性数据采集应用系统";胡哗永;《中国优秀硕士学位论文全文数据库信息科技辑》;20130215(第2期);第I138-1241页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104298773A (en) | 2015-01-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101904786B1 (en) | Apparatus and method for replicating changed data in a source database management system to a target database management system in real time | |
| CN104298773B (en) | A kind of ETL operations automatically cut storehouse system and method | |
| CN101706795B (en) | Method for synchronizing data of database in active/standby server | |
| CN102968486B (en) | A kind of highly reliable file synchronisation method based on change journal | |
| CN110347746B (en) | Heterogeneous database synchronous data consistency checking method and device | |
| CN104252500B (en) | The fault repairing method and device of a kind of database management platform | |
| CN101414946B (en) | Method and medium server for remote data backup | |
| CN103699580A (en) | Database synchronization method and database synchronization device | |
| CN104252502A (en) | Method and device for carrying out data migration on database management platform | |
| CN109189860A (en) | A kind of active and standby increment synchronization method of MySQL based on Kubernetes system | |
| CN102368222A (en) | Online repairing method of multiple-copy storage system | |
| CN104360923A (en) | Monitoring method and monitoring system for batch application process | |
| CN104809200A (en) | Database synchronization method and device | |
| CN113987064A (en) | Data processing method, system and equipment | |
| CN105589797A (en) | Method for synchronous data time delay detection between master database and slave database | |
| CN102231161A (en) | Method for synchronously verifying and monitoring databases | |
| CN103158987A (en) | Medicine identification and handling method in medicine fast distribution system | |
| CN105589887A (en) | Data processing method for distributed file system and distributed file system | |
| CN103294704A (en) | File synchronous system and method | |
| CN103973727A (en) | Data synchronizing method and device | |
| CN110011853B (en) | Cross fault troubleshooting method and device for multiple platforms and clusters | |
| CN106682141B (en) | Data synchronization method based on service operation log | |
| CN103581262B (en) | A kind of master/slave data synchronous method, device and system | |
| CN102073523B (en) | Realize the method and device of software version synchronization | |
| CN104516953B (en) | A kind of black box subsystem for power dispatching automation magnanimity message |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |