CN110245127A - A kind of data migration method based on Row control - Google Patents
A kind of data migration method based on Row control Download PDFInfo
- Publication number
- CN110245127A CN110245127A CN201910505341.7A CN201910505341A CN110245127A CN 110245127 A CN110245127 A CN 110245127A CN 201910505341 A CN201910505341 A CN 201910505341A CN 110245127 A CN110245127 A CN 110245127A
- Authority
- CN
- China
- Prior art keywords
- priority
- migration
- task
- migration task
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data migration method based on Row control, this method includes parsing priority configuration file, initialization task priority；The migration task of assigned priority is loaded, migration task is executed；Terminate all migration tasks.The present invention parses configuration file initialization task priority first, reloads the migration task for executing assigned priority；By thread management and migration task priority configuration, Row control is carried out to entire transition process, guarantees data correctness and integrality in transition process；Abnormal monitoring is carried out in entirely migration task execution simultaneously, loss of data is reduced convenient for subsequent to dealing of abnormal data in the position and abnormal data that accurate recording occurs extremely.
Description
Technical field
The invention belongs to database technology, thread control and technical field of data transmission, and in particular to one kind is based on process
The data migration method of control.
Background technique
System data amount and data class are constantly increasing with using, original data management mode or data management system
System has been unable to meet current actual use, needs to upgrade data management system.Data are cores in system structure
Historical data must be moved to new system by part, when system upgrade, guaranteed that the data of system continue and used.
But original system data are filled in lack of standardization, and content is imperfect, and data relationship is complicated, and storage form multiplicity, former data are deposited
Storage mode, data structure and new system database are inconsistent, carry out will appear various problems when Data Migration, such as:
(1) the storage mode multiplicity of source data, target database are not supported to be introduced directly into data, are integrated into uniform data lattice
Formula, heavy workload；
(2) feed data content is imperfect, lack of standardization, needs to import by specific cleaning；
(3) in source data, data conversion is complicated and processing step has very strong ordinal relation, and the mistake of processing sequence can shadow
Ring the correctness of data.
(4) in transition process, if encountering exception, it is difficult to find out wrong data in mass data, and because abnormal
Caused process flow stops being difficult restarting from the place of failure.
Existing Data Migration Tools are generally integrated by database management tools, do not provide the function of data cleansing.Separately
Outside, the Data Migration Tools kettle now to increase income can support multi-data source to move to the use of target database, but kettle work
Tool transition process is unable to control the priority of task execution, and abnormal conditions are unable to continuing for control flow when occurring, provide
Universal cleaning method limitation.
Summary of the invention
The main purpose of the present invention is to provide a kind of data migration methods based on Row control, it is intended to which solution migrated
Sequential control present in journey and the technical problems such as abnormal positioning and abnormality processing.
To achieve the above object, the present invention provides a kind of data migration method based on Row control, comprising the following steps:
S1, parsing priority configuration file, initialization task priority；
S2, the migration task for loading assigned priority execute migration task；
S3, terminate all migration tasks.
Preferably, the step S1 specifically:
Priority configuration file is read, the priority setting information in configuration file is carried out according to sequence from high to low
Sequence, and navigate to the first priority data.
Preferably, the step S2 specifically:
First using the first priority as current priority, migration task is executed according to the first priority, then according to excellent
First grade sequence successively executes migration task, until all migration task execution is completed.
Preferably, the step S2 specifically include it is following step by step:
Task execution number in S21, reading configuration file, loads and starts the migration of the same priority of specified number
Task, it is independent, parallel to execute migration task；
S22, the migration task that will exceed starting number are placed on Accreditation Waiting Area waiting；
S23, when execute complete starting a migration task when, judge Accreditation Waiting Area whether there is migration task；If so,
It loads and starts the migration task executed in Accreditation Waiting Area；If it is not, then continuing to execute the migration task in having been turned on；
S24, after the migration task execution of current priority, one is excellent in the presence of being searched whether according to priority orders
First grade；If so, using next priority as current priority, the migration task for loading current priority is continued to execute；If it is not,
Then carry out step S3.
Preferably, the step S3 specifically:
All data connections are closed, all cachings are emptied.
Preferably, the step S2 further includes the exception during monitoring migration task execution.
Preferably, the exception during the monitoring migration task execution specifically:
When being abnormal situation during migrating task execution, the reason of being abnormal situation and location information are recorded simultaneously
It saves；
It chooses whether to continue to execute migration task, when selecting to continue to execute migration task, does nothing, continue to hold
Row migration task, and migration task abnormity monitoring is carried out simultaneously；When selection does not continue to execute migration task, current priority is interrupted
In all tasks, whether reselection enter next priority；
When selection enters next priority, step S24 is continued to execute, and carries out migration task abnormity monitoring simultaneously；Selection
When not entering next priority, step S3 is carried out.
The beneficial effects of the present invention are: the present invention parses configuration file initialization task priority first, execution is reloaded
The migration task of assigned priority；By thread management and migration task priority configuration, realizes and entire transition process is carried out
Row control guarantees that the data of migration are correctly and complete；Abnormal monitoring is carried out in entirely migration task execution simultaneously, it is accurate to remember
The position occurred extremely and abnormal data are recorded, convenient for subsequent to dealing of abnormal data, reduces loss of data.
Detailed description of the invention
Fig. 1 is the data migration method flow diagram of the invention based on Row control.
Specific embodiment
In order to make the objectives, technical solutions, and advantages of the present invention clearer, with reference to the accompanying drawings and embodiments, right
The present invention is further elaborated.It should be appreciated that described herein, specific examples are only used to explain the present invention, not
For limiting the present invention.
As shown in Figure 1, being the data migration method flow diagram of the invention based on Row control；One kind being based on process
The data migration method of control, comprising the following steps:
S1, parsing priority configuration file, initialization task priority；
S2, the migration task for loading assigned priority execute migration task；
S3, terminate all migration tasks.
In an alternate embodiment of the present invention where, above-mentioned steps S1 parses priority configuration file, and initialization task is excellent
First grade, specifically:
Priority configuration file is read, the priority setting information in configuration file is carried out according to sequence from high to low
Sequence, and navigate to the first priority data.When the step for after the completion of, enter step S2.
In an alternate embodiment of the present invention where, above-mentioned steps S2 loads the task of assigned priority, executes migration and appoints
Business, specifically:
When executing migration task for the first time, using the first priority as current priority, migration is executed according to the first priority
Then task successively executes migration task according to priority orders, until all migration task execution is completed.
Above-mentioned steps S2 specifically include it is following step by step:
Task execution number in S21, reading configuration file, loads and starts the migration of the same priority of specified number
Task independently, is performed in parallel migration task；
S22, when having, when exceeding the migration task for starting number, the migration task that will exceed starting number is placed on Accreditation Waiting Area
Etc. pending；
S23, when execute complete starting a migration task when, judge Accreditation Waiting Area whether there is migration task；If waiting
There are migration tasks in area, then load and start the migration task executed in Accreditation Waiting Area；If migration task is not present in Accreditation Waiting Area,
Then continue to execute the migration task in having been turned on.Migration task in current priority all executes completion, enters step S24；
S24, after the migration task execution of current priority, one is excellent in the presence of being searched whether according to priority orders
First grade；Next priority if it exists then obtains next priority, and using next priority as current priority, load is current excellent
The migration task of first grade continues to execute；Next priority if it does not exist, then carrying out step S3 terminates migration task.
Also monitoring migrates the abnormal conditions generation during task execution to the present invention simultaneously during executing migration task,
Specifically:
When being abnormal situation during migrating task execution, the reason of being abnormal situation and location information are recorded simultaneously
It is saved in file or exception database；
Choose whether to continue to execute migration task, when selection continues to execute migration task, without other any operations, after
It is continuous to execute migration task, and migration task abnormity monitoring is carried out simultaneously；When selection does not continue to execute migration task, interrupt current excellent
Whether all tasks in first grade, reselection enter next priority；
When selection enters next priority, step S24 is continued to execute, and carries out migration task abnormity monitoring simultaneously；Selection
When not entering next priority, step S3 is carried out, all migration task executions terminate.
In an alternate embodiment of the present invention where, above-mentioned steps S3 terminates all migration tasks, specifically:
All data connections are closed, all cachings are emptied.
Those of ordinary skill in the art will understand that the embodiments described herein, which is to help reader, understands this hair
Bright principle, it should be understood that protection scope of the present invention is not limited to such specific embodiments and embodiments.This field
Those of ordinary skill disclosed the technical disclosures can make according to the present invention and various not depart from the other each of essence of the invention
The specific variations and combinations of kind, these variations and combinations are still within the scope of the present invention.
Claims (7)
1. a kind of data migration method based on Row control, which comprises the following steps:
S1, parsing priority configuration file, initialization task priority；
S2, the migration task for loading assigned priority execute migration task；
S3, terminate all migration tasks.
2. as described in claim 1 based on the data migration method of Row control, which is characterized in that the step S1 is specific
Are as follows:
Priority configuration file is read, the priority setting information in configuration file is arranged according to sequence from high to low
Sequence, and navigate to the first priority data.
3. as claimed in claim 2 based on the data migration method of Row control, which is characterized in that the step S2 is specific
Are as follows:
First using the first priority as current priority, migration task is executed according to the first priority, then according to priority
Sequence successively executes migration task, until all migration task execution is completed.
4. as claimed in claim 3 based on the data migration method of Row control, which is characterized in that the step S2 is specifically wrapped
Include it is following step by step:
Task execution number in S21, reading configuration file, the migration for loading and starting the same priority of specified number are appointed
Business, it is independent, parallel to execute migration task；
S22, the migration task that will exceed starting number are placed on Accreditation Waiting Area waiting；
S23, when execute complete starting a migration task when, judge Accreditation Waiting Area whether there is migration task；If so, load
And start the migration task executed in Accreditation Waiting Area；If it is not, then continuing to execute the migration task in having been turned on；
S24, after the migration task execution of current priority, one is preferential in the presence of being searched whether according to priority orders
Grade；If so, using next priority as current priority, the migration task for loading current priority is continued to execute；If it is not, then
Carry out step S3.
5. as claimed in claim 4 based on the data migration method of Row control, which is characterized in that the step S3 is specific
Are as follows:
All data connections are closed, all cachings are emptied.
6. the data migration method a method as claimed in any one of claims 1 to 5 based on Row control, which is characterized in that the step S2
It further include the exception during monitoring migration task execution.
7. as claimed in claim 6 based on the data migration method of Row control, which is characterized in that the monitoring migration task
Exception in implementation procedure specifically:
When being abnormal situation during migrating task execution, records the reason of being abnormal situation and location information and protect
It deposits；
It chooses whether to continue to execute migration task, when selecting to continue to execute migration task, does nothing, continue to execute and move
Shifting task, and migration task abnormity monitoring is carried out simultaneously；When selection does not continue to execute migration task, interrupt in current priority
Whether all tasks, reselection enter next priority；
When selection enters next priority, step S24 is continued to execute, and carries out migration task abnormity monitoring simultaneously；Selection not into
When entering next priority, step S3 is carried out.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910505341.7A CN110245127A (en) | 2019-06-12 | 2019-06-12 | A kind of data migration method based on Row control |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910505341.7A CN110245127A (en) | 2019-06-12 | 2019-06-12 | A kind of data migration method based on Row control |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110245127A true CN110245127A (en) | 2019-09-17 |
Family
ID=67886740
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910505341.7A Pending CN110245127A (en) | 2019-06-12 | 2019-06-12 | A kind of data migration method based on Row control |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110245127A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021217942A1 (en) * | 2020-04-27 | 2021-11-04 | 山东省科学院海洋仪器仪表研究所 | Border-gateway-based method and system for migrating task of underwater ad hoc network |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101383751A (en) * | 2008-07-24 | 2009-03-11 | 四川长虹电器股份有限公司 | Network downloading method |
| US20120159336A1 (en) * | 2009-03-27 | 2012-06-21 | Michael Roy Norwood | Electronic list priority management system and method of using same |
| CN103019838A (en) * | 2012-11-14 | 2013-04-03 | 清华大学 | Multi-DSP (Digital Signal Processor) platform based distributed type real-time multiple task operating system |
| CN103617493A (en) * | 2013-11-27 | 2014-03-05 | 北京国电通网络技术有限公司 | Method and device for task processing |
| CN103842968A (en) * | 2013-11-22 | 2014-06-04 | 华为技术有限公司 | Migration method, computer and device of stored data |
| CN104199738A (en) * | 2014-08-11 | 2014-12-10 | 苏州阔地网络科技有限公司 | Multi-data processing equipment cooperative work method and system |
| CN105045832A (en) * | 2015-06-30 | 2015-11-11 | 北京奇艺世纪科技有限公司 | Data acquisition method and apparatus |
| CN106990924A (en) * | 2017-04-07 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of data migration method and system |
| CN107241205A (en) * | 2016-03-28 | 2017-10-10 | 阿里巴巴集团控股有限公司 | abnormality monitoring method and device |
| CN108446180A (en) * | 2018-03-23 | 2018-08-24 | 南京航空航天大学 | A kind of data center dynamic method for scheduling task based on Data Migration |
| CN108874518A (en) * | 2018-05-21 | 2018-11-23 | 福建省数字福建云计算运营有限公司 | A kind of method for scheduling task and terminal |
-
2019
- 2019-06-12 CN CN201910505341.7A patent/CN110245127A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101383751A (en) * | 2008-07-24 | 2009-03-11 | 四川长虹电器股份有限公司 | Network downloading method |
| US20120159336A1 (en) * | 2009-03-27 | 2012-06-21 | Michael Roy Norwood | Electronic list priority management system and method of using same |
| CN103019838A (en) * | 2012-11-14 | 2013-04-03 | 清华大学 | Multi-DSP (Digital Signal Processor) platform based distributed type real-time multiple task operating system |
| CN103842968A (en) * | 2013-11-22 | 2014-06-04 | 华为技术有限公司 | Migration method, computer and device of stored data |
| CN103617493A (en) * | 2013-11-27 | 2014-03-05 | 北京国电通网络技术有限公司 | Method and device for task processing |
| CN104199738A (en) * | 2014-08-11 | 2014-12-10 | 苏州阔地网络科技有限公司 | Multi-data processing equipment cooperative work method and system |
| CN105045832A (en) * | 2015-06-30 | 2015-11-11 | 北京奇艺世纪科技有限公司 | Data acquisition method and apparatus |
| CN107241205A (en) * | 2016-03-28 | 2017-10-10 | 阿里巴巴集团控股有限公司 | abnormality monitoring method and device |
| CN106990924A (en) * | 2017-04-07 | 2017-07-28 | 广东浪潮大数据研究有限公司 | A kind of data migration method and system |
| CN108446180A (en) * | 2018-03-23 | 2018-08-24 | 南京航空航天大学 | A kind of data center dynamic method for scheduling task based on Data Migration |
| CN108874518A (en) * | 2018-05-21 | 2018-11-23 | 福建省数字福建云计算运营有限公司 | A kind of method for scheduling task and terminal |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021217942A1 (en) * | 2020-04-27 | 2021-11-04 | 山东省科学院海洋仪器仪表研究所 | Border-gateway-based method and system for migrating task of underwater ad hoc network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7139629B2 (en) | Planning and scheduling for failure recovery system and method | |
| US4727494A (en) | Computerized robot control system with scheduling feature | |
| CN100583782C (en) | Software upgrade method and device | |
| CN110245127A (en) | A kind of data migration method based on Row control | |
| EP2722774B1 (en) | Long term archiving of data in a MES system | |
| US6757634B2 (en) | Methods and structure for maintaining state information to resume automated test processing after reset | |
| EP0320876A2 (en) | Fault information collection processing system | |
| CN100576182C (en) | The real-time monitoring system of computer documents and method | |
| JPH05151013A (en) | System for performance prediction and performance diagnosis | |
| KR100510065B1 (en) | Method for automatically operating overlay equipment used in manufacturing semiconductor | |
| US7457986B2 (en) | Apparatus and method for using variable end state delay to optimize JTAG transactions | |
| CN113868027B (en) | Data snapshot method and device | |
| CN112257906B (en) | Imaging satellite autonomous task planning driving method based on state management | |
| CN109298887B (en) | Program control instruction autonomous execution system and method based on callback function | |
| JP2857381B2 (en) | Dynamic simulator | |
| UA56310C2 (en) | Method for maintenance and repair of computers and the system for realization of the method | |
| JPS62271151A (en) | Automatic testing system for computer main body | |
| JPH05143422A (en) | Updated journal managing system | |
| JP2658699B2 (en) | Method of managing device under test of input / output device test program | |
| CN118093565A (en) | Data cleaning method, device and medium based on redis | |
| CN115685729A (en) | Serial operation method based on data acquisition | |
| JPS59111501A (en) | Sequence controller | |
| JPH04175830A (en) | Rerun control system | |
| JPH07248810A (en) | Numerical controller | |
| CN116560884A (en) | Data development task exception handling method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190917 |