CN104794124A - Intelligent implementation method and system for data missing supplementation - Google Patents
Intelligent implementation method and system for data missing supplementation Download PDFInfo
- Publication number
- CN104794124A CN104794124A CN201410025136.8A CN201410025136A CN104794124A CN 104794124 A CN104794124 A CN 104794124A CN 201410025136 A CN201410025136 A CN 201410025136A CN 104794124 A CN104794124 A CN 104794124A
- Authority
- CN
- China
- Prior art keywords
- task
- daily record
- interface
- interface document
- ftp
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an intelligent implementation method for data missing supplementation. The intelligent implementation method comprises the steps that an interface file of an interface corresponding to an uncompleted task is scanned, and an interface document which is newly set or retransmitted is loaded; matching is conducted on a preset task log and the interface file, if matching succeeds, the state of the task log is changed; if matching fails, the current task is skipped. The invention further discloses an intelligent implementation system for data missing supplementation.
Description
Technical field
The present invention relates to data traffic interfaces technology, particularly relate to a kind of intelligent realization method and system of shortage of data mending-leakage.
Background technology
Data are extracted, change and load (Extraction-Transformation-Loading, ETL) be data pick-up (Extract), cleaning (Cleaning), conversion (Transform), load the process of (Load), be the important step building data warehouse.User extracts required data from data source, through data cleansing, finally according to the data warehouse model pre-defined, by Data import in data warehouse.
Data warehouse is an independently data environment, needs to import to data warehouse by data from the data storage medium of Transaction Processing environment, external data source and off line by extraction process; Technically, ETL is mainly concerned with several aspects such as association, conversion, increment, scheduling and monitoring, and the data in data warehouse do not require and the real time data synchronization in online transaction processing system, so ETL can regularly carry out.
The mode that the characteristic that traditional E TL transmits according to interface adopts timing to drive carries out chain type computing, such as every night eight point processing interface A; When interface data lacks, scheduling cannot normally be carried out, and intelligence can not carry out follow-up work when interface data postpones or arrives after disappearance a couple of days, and follow-up mending-leakage need of work is manually intervened.Usually, need maintainer manually to carry out data mending-leakage operation, or some configure/status/switch of adjustment ETL, make it the interface data starting computing disappearance, bring trouble to ETL maintainer, also bring trouble to related system.
Summary of the invention
In view of this, the embodiment of the present invention expects the intelligent realization method and the system that provide a kind of shortage of data mending-leakage, can when no user is intervened, and intelligence completes the disappearance mending-leakage operation of all data.
For achieving the above object, technical scheme of the present invention is achieved in that
Embodiments provide a kind of intelligent realization method of shortage of data mending-leakage, described method comprises:
Scan the interface document of corresponding interface of not finishing the work, and download interface document that is newly-increased or that retransmit;
The task daily record generated in advance and interface document are mated, when determining that the match is successful, task status in the daily record of change task; When determining that it fails to match, skip current task.
In such scheme, described scanning do not finish the work corresponding interface interface document before, described method also comprises: read task daily record, determine not completing of task, and send scan request.
In such scheme, after the interface document that described download increases newly or retransmits, described method also comprises: in task daily record acceptance of the bid note file transfer protocol (FTP) (File Transfer Protocol, FTP) download phase for complete, mark wash phase is for carry out.
In such scheme, described mark wash phase is for after carrying out, and before the task daily record generated in advance and interface document being mated, described method also comprises: read task status in task daily record, and send interface document matching request.
The intelligence that the embodiment of the present invention additionally provides a kind of shortage of data mending-leakage realizes system, and described system comprises: FTP scanner and task daily record managing driver; Wherein,
Described FTP scanner, for scanning the interface document of corresponding interface of not finishing the work, and downloads interface document that is newly-increased or that retransmit;
Described task daily record managing driver, for the daily record of generating in advance of task and interface document are mated, when determining that the match is successful, change task stl status; When determining that it fails to match, skip current task.
In such scheme, described system also comprises: core controller, interface document storehouse, local file storehouse and storer; Wherein,
Described core controller, for reading task daily record, determines the task of not completing and task status, the operation of scheduling FTP scanner and task daily record managing driver;
Described interface document storehouse, for storing the interface document that the other side's interface message processor (IMP) receives;
Described local file storehouse, for storing the total interface file of local interface machine;
Described storer, for store tasks daily record.
In such scheme, described FTP scanner also for, in task daily record, mark FTP download phase for complete, mark wash phase for carry out.
The intelligent realization method of the shortage of data mending-leakage that the embodiment of the present invention provides, scans the interface document of corresponding interface of not finishing the work, and downloads interface document that is newly-increased or that retransmit; The task daily record generated in advance and interface document are mated, when determining that the match is successful, change task stl status; When determining that it fails to match, skip current task.So, the deficiency of existing ETL to the process of the abnormal conditions such as data delay, loss can be made up, promote the supportability to data accuracy, promptness, reduce the cost that ETL is safeguarded simultaneously, make value-added service comprehensive operation platform (Value-added Service General Operation Platform, VGOP) more effective data service can be supported, provide guide for data service advances for a long time stably.
Accompanying drawing explanation
Fig. 1 is the intelligent realization method schematic flow sheet of the embodiment of the present invention one shortage of data mending-leakage;
Fig. 2 is the intelligent realization method schematic flow sheet of the embodiment of the present invention two shortage of data mending-leakage;
Fig. 3 is that the intelligence of embodiment of the present invention shortage of data mending-leakage realizes system composition structural representation.
Embodiment
In embodiments of the present invention, scan the interface document of corresponding interface of not finishing the work, and download interface document that is newly-increased or that retransmit; The task daily record generated in advance and interface document are mated, when determining that the match is successful, change task stl status; When determining that it fails to match, skip current task.
Fig. 1 is the intelligent realization method schematic flow sheet of the embodiment of the present invention one shortage of data mending-leakage, and as shown in Figure 1, the intelligent realization method flow process of this shortage of data mending-leakage comprises:
Step 101: the interface document scanning corresponding interface of not finishing the work, and download interface document that is newly-increased or that retransmit;
Here, described scanning do not finish the work corresponding interface interface document before, described method also comprises: read task daily record, determine not completing of task, and send scan request; Wherein, the mission bit stream do not completed is comprised in described scan request;
The interface document that described download increases newly or retransmits is: by interface document attribute, as creation-time information etc. knows interface document that is newly-increased or that retransmit, then downloads interface document that is newly-increased or that retransmit;
After the interface document that described download increases newly or retransmits, described method also comprises: in task daily record, and mark FTP download phase is for complete, and mark wash phase is for carry out.
Step 102: the task daily record generated in advance and interface document are mated, when determining that the match is successful, task status in the daily record of change task; When determining that it fails to match, skip current task;
Here, the described task daily record generated in advance is: the task daily record generated according to the information that user is pre-configured, described pre-configured information can comprise task and fileinfo, as timing needs program and interface document name, interface document size, storage time, the preposition storage condition etc. of scheduling;
Described mark wash phase is for after carrying out, and before the task daily record generated in advance and interface document being mated, described method also comprises: read task status in task daily record, and send interface document matching request; Here, the task of needing to mate and corresponding interface information is comprised in task daily record in described interface document matching request;
Describedly coupling is carried out in the task daily record generated in advance and interface document comprise: according to the task daily record generated in advance, search the interface document of corresponding interface of not finishing the work, if find correct interface document, then the match is successful, if the interface document of correspondence does not exist or finds the interface document of mistake, then it fails to match;
In the daily record of described change task, task status comprises: the interface document matching result in the daily record of change task after FTP download phase is for the match is successful, and triggering following ETL treatment scheme.
Fig. 2 is the intelligent realization method schematic flow sheet of the embodiment of the present invention two shortage of data mending-leakage, and as shown in Figure 2, the intelligent realization method flow process of this shortage of data mending-leakage comprises:
Step 201: read task daily record, determine not completing of task, and send scan request;
Here, the mission bit stream do not completed is comprised in described scan request.
Step 202: the interface document scanning corresponding interface of not finishing the work, and download interface document that is newly-increased or that retransmit;
Here, the interface document that described download increases newly or retransmits is: by interface document attribute, as creation-time information etc. knows interface document that is newly-increased or that retransmit, then downloads interface document that is newly-increased or that retransmit.
Step 203: mark FTP download phase for complete in task daily record, mark wash phase is for carry out.
Step 204: read task status in task daily record, and send interface document matching request;
Here, read task status in task daily record be FTP download phase for complete, wash phase is when carrying out, transmission interface file matching request;
The task and corresponding interface information that need to mate is comprised in task daily record in described interface document matching request.
Step 205: the task daily record generated in advance and interface document are mated, and judge that whether coupling is successful, when determining that the match is successful, perform step 206; When determining that it fails to match, perform step 207;
Here, the described task daily record generated in advance is: the task daily record generated according to the information that user is pre-configured, described pre-configured information can comprise task and fileinfo, as timing needs program and interface document name, interface document size, storage time, the preposition storage condition etc. of scheduling;
Describedly coupling is carried out in the task daily record generated in advance and interface document comprise: the interface document searching corresponding interface of not finishing the work according to the task daily record generated in advance, if find correct interface document, then the match is successful, if the interface document of correspondence does not exist or finds the interface document of mistake, then it fails to match.
Step 206: task status in the daily record of change task, performs step 208;
In the daily record of described change task, task status comprises: the interface document matching result in the daily record of change task after FTP download phase is for the match is successful, and triggering following ETL treatment scheme.
Step 207: skip current task;
Here, skip current task described in and namely carry out next abortive interface document scanning, coupling flow process.
Step 208: terminate this treatment scheme.
Fig. 3 is that the intelligence of embodiment of the present invention shortage of data mending-leakage realizes system composition structural representation, and as shown in Figure 3, the intelligence of this shortage of data mending-leakage realizes system composition structure and comprises: FTP scanner 31 and task daily record managing driver 32; Wherein,
Described FTP scanner 31, for scanning the interface document of corresponding interface of not finishing the work, and downloads interface document that is newly-increased or that retransmit;
Described task daily record managing driver 32, for the daily record of generating in advance of task and interface document are mated, when determining that the match is successful, change task stl status; When determining that it fails to match, skip current task;
Here, the described task daily record generated in advance is: the task daily record generated according to the information that user is pre-configured, described pre-configured information can comprise task and fileinfo, as timing needs program and interface document name, interface document size, storage time, the preposition storage condition etc. of scheduling;
Describedly coupling is carried out in the task daily record generated in advance and interface document comprise: the interface document searching corresponding interface of not finishing the work according to the task daily record generated in advance, if find correct interface document, then the match is successful, if the interface document of correspondence does not exist or finds the interface document of mistake, then it fails to match;
In the daily record of described change task, task status comprises: the interface document matching result in the daily record of change task after FTP download phase is for the match is successful, and triggering following ETL treatment scheme.
Further, described system also comprises: core controller 33, interface document storehouse 34, local file storehouse 35 and storer 36; Wherein,
Described core controller 33, for reading task daily record, determines the task of not completing and task status, the operation of scheduling FTP scanner 31 and task daily record managing driver 32;
Described interface document storehouse 34, for storing the interface document that the other side's interface message processor (IMP) receives;
Described local file storehouse 35, for storing the total interface file of local interface machine;
Described storer 36, for store tasks daily record.
Further, described FTP scanner 31, also for marking FTP download phase for complete in task daily record, mark wash phase is for carry out;
Described core controller 33, also for the operation of other service and function in control ETL process.
The above, be only preferred embodiment of the present invention, be not intended to limit protection scope of the present invention.
Claims (7)
1. an intelligent realization method for shortage of data mending-leakage, is characterized in that, described method comprises:
Scan the interface document of corresponding interface of not finishing the work, and download interface document that is newly-increased or that retransmit;
The task daily record generated in advance and interface document are mated, when determining that the match is successful, task status in the daily record of change task; When determining that it fails to match, skip current task.
2. method according to claim 1, is characterized in that, described scanning do not finish the work corresponding interface interface document before, described method also comprises: read task daily record, determine not completing of task, and send scan request.
3. method according to claim 1, it is characterized in that, after the interface document that described download increases newly or retransmits, described method also comprises: in task daily record acceptance of the bid note file transfer protocol (FTP) FTP download phase for complete, mark wash phase for carry out.
4. method according to claim 3, it is characterized in that, described mark wash phase for after carrying out, before the task daily record generated in advance and interface document are mated, described method also comprises: read task status in task daily record, and send interface document matching request.
5. the intelligence of shortage of data mending-leakage realizes a system, it is characterized in that, described system comprises: FTP scanner and task daily record managing driver; Wherein,
Described FTP scanner, for scanning the interface document of corresponding interface of not finishing the work, and downloads interface document that is newly-increased or that retransmit;
Described task daily record managing driver, for the daily record of generating in advance of task and interface document are mated, when determining that the match is successful, change task stl status; When determining that it fails to match, skip current task.
6. system according to claim 5, it is characterized in that, described system also comprises: core controller, interface document storehouse, local file storehouse and storer; Wherein,
Described core controller, for reading task daily record, determines the task of not completing and task status, the operation of scheduling FTP scanner and task daily record managing driver;
Described interface document storehouse, for storing the interface document that the other side's interface message processor (IMP) receives;
Described local file storehouse, for storing the total interface file of local interface machine;
Described storer, for store tasks daily record.
7. system according to claim 5, is characterized in that, described FTP scanner also for, in task daily record, mark FTP download phase for complete, mark wash phase is for carry out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410025136.8A CN104794124A (en) | 2014-01-20 | 2014-01-20 | Intelligent implementation method and system for data missing supplementation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410025136.8A CN104794124A (en) | 2014-01-20 | 2014-01-20 | Intelligent implementation method and system for data missing supplementation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104794124A true CN104794124A (en) | 2015-07-22 |
Family
ID=53558919
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410025136.8A Pending CN104794124A (en) | 2014-01-20 | 2014-01-20 | Intelligent implementation method and system for data missing supplementation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104794124A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108197079A (en) * | 2016-12-08 | 2018-06-22 | 广东精点数据科技股份有限公司 | A kind of improved algorithm to missing values interpolation |
| CN113312357A (en) * | 2021-06-23 | 2021-08-27 | 中国农业银行股份有限公司 | Data loading method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101477543A (en) * | 2008-01-03 | 2009-07-08 | 埃森哲环球服务有限公司 | System and method for automating ETL application |
| CN102479113A (en) * | 2010-11-30 | 2012-05-30 | 中国移动通信集团黑龙江有限公司 | Abnormal self-adapting processing method and system |
| CN102637214A (en) * | 2012-04-28 | 2012-08-15 | 中国工商银行股份有限公司 | Method and system for synchronizing general data among database services |
| CN102902785A (en) * | 2012-09-29 | 2013-01-30 | 合一网络技术(北京)有限公司 | Webpage information acquisition system and method |
-
2014
- 2014-01-20 CN CN201410025136.8A patent/CN104794124A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101477543A (en) * | 2008-01-03 | 2009-07-08 | 埃森哲环球服务有限公司 | System and method for automating ETL application |
| US20090177671A1 (en) * | 2008-01-03 | 2009-07-09 | Accenture Global Services Gmbh | System and method for automating etl application |
| CN102479113A (en) * | 2010-11-30 | 2012-05-30 | 中国移动通信集团黑龙江有限公司 | Abnormal self-adapting processing method and system |
| CN102637214A (en) * | 2012-04-28 | 2012-08-15 | 中国工商银行股份有限公司 | Method and system for synchronizing general data among database services |
| CN102902785A (en) * | 2012-09-29 | 2013-01-30 | 合一网络技术(北京)有限公司 | Webpage information acquisition system and method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108197079A (en) * | 2016-12-08 | 2018-06-22 | 广东精点数据科技股份有限公司 | A kind of improved algorithm to missing values interpolation |
| CN113312357A (en) * | 2021-06-23 | 2021-08-27 | 中国农业银行股份有限公司 | Data loading method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102637214B (en) | Method and system for synchronizing general data among database services | |
| CN103095769B (en) | Across method of data synchronization and the system of machine room | |
| CN100428141C (en) | Programmable controller and communication unit, and methods of solving variables and of handing data | |
| CN105405442B (en) | voice recognition method, device and equipment | |
| CN104267701B (en) | The control method and device of smart home newly added equipment | |
| CN110990482A (en) | Data synchronization method and device between asynchronous databases | |
| CN104579752A (en) | Remote automatic upgrading system and remote automatic upgrading method | |
| CN103164232B (en) | Update method, system and the computer of intelligent terminal operation system | |
| CN105335316A (en) | Motor assembling line serial port server based on cloud computation | |
| CN110071855A (en) | Equipment linkage control method, device, system, gateway and storage medium | |
| CN104794124A (en) | Intelligent implementation method and system for data missing supplementation | |
| CN105808619A (en) | Task redoing method based on influence analysis, influence analysis calculation device and one-key reset device | |
| CN104978635A (en) | Production management information physical system and implementation method thereof | |
| CN107741965B (en) | Database synchronous processing method and device, computing equipment and computer storage medium | |
| CN105607606A (en) | Data acquisition device and data acquisition method based on double-mainboard framework | |
| CN113297218B (en) | Multi-system data interaction method, device and system | |
| CN105245571A (en) | Data acquisition method, apparatus and system | |
| CN104065613A (en) | Offline operation data synchronization method, system and device of application | |
| CN103810103A (en) | Method and system for debugging electronic device | |
| CN103488712A (en) | Automatized testing method and automatized testing system | |
| CN106850596B (en) | Data acquisition system and method | |
| CN105224327A (en) | A kind of embedded equipment variable exchanges transport address off-line distribution method | |
| CN105592097B (en) | A kind of client-based asynchronous interactive information approach | |
| CN110134086B (en) | Data acquisition and control device for realizing middleware function in digital workshop | |
| CN110716524B (en) | WEB-based ESOP system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150722 |
|
| RJ01 | Rejection of invention patent application after publication |