CN106503018A - Carry out the method and system of historical data migration - Google Patents
Carry out the method and system of historical data migration Download PDFInfo
- Publication number
- CN106503018A CN106503018A CN201510565997.XA CN201510565997A CN106503018A CN 106503018 A CN106503018 A CN 106503018A CN 201510565997 A CN201510565997 A CN 201510565997A CN 106503018 A CN106503018 A CN 106503018A
- Authority
- CN
- China
- Prior art keywords
- message
- service
- migration
- triggering message
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
The invention discloses carrying out the method and system of historical data migration, wherein, the method includes:Obtain the triggering message that the manufacturing side is generated in real time;Parse to triggering message, obtain type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business identifies corresponding business datum from storage facility located at processing plant, business datum is moved to by history library using the migration logic for obtaining.The present invention program can reduce migration pressure, improve transport efficiency.
Description
Technical field
The present invention relates to data processing technique, more particularly to the method and system for carrying out historical data migration.
Background technology
At present, during each business carries out data processing, substantial amounts of business datum can be generated.For example, net is being carried out
During upper data message is interacted, the service interaction data being related to need to be stored.
Read-write efficiency is caused to decline to solve the problems, such as data base's factor data amount excessive, it will usually to create one
Storage facility located at processing plant and history library, storage facility located at processing plant are stored to the business datum for producing in real time.Then periodically storage facility located at processing plant
Historical data is moved on history library, improves the performance of storage facility located at processing plant with this;Specifically, counted using intervalometer
When, timer time is reached, batch operation is carried out, the batch data in storage facility located at processing plant is moved to history library;Thus, all
Phase property ground is executed.
There is following defect in the existing migration scheme:Periodically carry out, the time concentrates, to storage facility located at processing plant and history library
Cause very big migration pressure;Also, the continuous growth with business, it is right that periodic migration pattern can not meet
The requirement of transport efficiency.
Content of the invention
The invention provides a kind of method for carrying out historical data migration, the method can reduce migration pressure, and raising is moved
Move efficiency.
The invention provides a kind of system for carrying out historical data migration, the system can reduce migration pressure, and raising is moved
Move efficiency.
A kind of method for carrying out historical data migration, the method include:
Obtain the triggering message that the manufacturing side is generated in real time;
Parse to triggering message, obtain type of service and service identification;
Migration logic corresponding with type of service is obtained, inquiry business identifies corresponding business datum from storage facility located at processing plant, adopts
Business datum is moved to history library with the migration logic for obtaining.
Preferably, the generation of the triggering message includes:
The manufacturing side is in process of service execution, knowing how business datum meets outdated conditions, then generates triggering message.
Preferably, the method also includes:
Receive and subscribe to request, the subscription request bag contains type of service;
Generate the subscription information with regard to type of service.
Preferably, after described pair of triggering message is parsed, the method also includes:
The type of service for whether having triggering message package to contain in querying information of subscription, if it is, execute the acquisition and industry
Service type corresponding migration logic the step of;Otherwise, terminate flow process.
Preferably, the method also includes:
The Data Migration of storage facility located at processing plant to history library is monitored, if migration failure, is executed again described using obtaining
The step of business datum is moved to history library by the migration logic for taking.
A kind of system for carrying out historical data migration, the system include message control center, storage facility located at processing plant and history library, institute
Stating message control center includes message manager and consumption terminal;
The message manager, obtains the triggering message that the manufacturing side is generated in real time, is sent to the consumption terminal;
The consumption terminal, receives the triggering message from the message manager, parses to triggering message, obtains
Type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business mark is right from storage facility located at processing plant
The business datum for inquiring is moved to history library using the migration logic for obtaining by the business datum that answers;
The storage facility located at processing plant, stores business datum to be migrated;
The history library, for the data that storage carries out migration operation.
Preferably, the system also includes the manufacturing side, in process of service execution, knowing how business datum meet expired
Condition, then generate triggering message, and triggering message is sent to the message manager.
Preferably, the consumption terminal, also sends to the message manager and subscribes to request, and the subscription request bag contains industry
Service type;
The message manager, receives and subscribes to request, generates the subscription information with regard to type of service.
Preferably, the message manager, before triggering message is sent to the consumption terminal, goes back querying information of subscription
In whether have the triggering type of service that contains of message package, if it is, triggering message is sent to the consumption terminal;Otherwise,
Do not send.
Preferably, the consumption terminal, is monitored to the Data Migration of storage facility located at processing plant to history library, if migration failure,
Then failure response is returned to the message manager;
The message manager, after reception failure response, sends triggering message to the consumption terminal again.
Preferably, the manufacturing side is production cluster, and the message end is message trunking.
From such scheme as can be seen that in the present invention, obtaining the triggering message that the manufacturing side is generated in real time;To triggering message
Parsed, obtained type of service and service identification;Migration logic corresponding with type of service is obtained, from storage facility located at processing plant
Inquiry business identifies corresponding business datum, and business datum is moved to history library using the migration logic for obtaining.This
The bright mode for employing real-time migration, and the concrete scheme migrated with reference to type of service and service identification is given,
So as to, reduce and very big migration pressure is caused to storage facility located at processing plant and history library, also improve transport efficiency, meeting needs
Ask.
Description of the drawings
Fig. 1 is the method indicative flowchart that the present invention carries out historical data migration;
Fig. 2 is the method flow diagram example that the present invention carries out historical data migration;
Fig. 3 is the system structure diagram that the present invention carries out historical data migration.
Specific embodiment
For making the object, technical solutions and advantages of the present invention become more apparent, with reference to embodiment and accompanying drawing,
The present invention is described in more detail.
The timed periodic of existing employing carries out the scheme of batch migration to historical data, and existence time is concentrated, migration pressure
Power is big, the low defect of efficiency.In order to solve the technical problem, the invention provides the mode of real-time migration, and be given
The concrete scheme that migrated with reference to type of service and service identification.
Referring to Fig. 1, it is the method indicative flowchart for carrying out historical data migration of the invention, which comprises the following steps:
Step 101, obtains the triggering message that the manufacturing side is generated in real time.
During online data message interaction is carried out, the service interaction data being related to need to be stored, the manufacturing side
Storage facility located at processing plant is arrived in the storage of corresponding business datum.In the present invention, in order to carry out real-time migration, the manufacturing side is needed to enter industry
In the storage of business data, generate triggering message in real time, report message control center, the triggering message package contains service class
Type and service identification, type of service illustrate the data type of business datum, and service identification can be specially business datum
Numbering, can inquire corresponding business datum according to service identification.
Further, if the triggering message of synchronization reception is more, message can be triggered and is stored in message queue, while
Extract triggering message from message queue one by one to be executed.
Step 102, parses to triggering message, obtains type of service and service identification.
Step 103, obtains migration logic corresponding with type of service, and from storage facility located at processing plant, inquiry business mark is corresponding
Business datum is moved to history library using the migration logic for obtaining by business datum.
The business datum of different service types, has each different storage logics, correspondingly, using corresponding
Migration logic is migrated to corresponding business datum.
In the present invention, the triggering message that the manufacturing side is generated in real time is obtained, then, parse to triggering message, obtain
Type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business mark is right from storage facility located at processing plant
Business datum is moved to history library using the migration logic for obtaining by the business datum that answers.Present invention employs and move in real time
The mode of shifting, and the concrete scheme migrated with reference to type of service and service identification is given, so as to it is right to reduce
Storage facility located at processing plant and history library cause very big migration pressure, also improve transport efficiency, meet demand.
In implementing, the manufacturing side can report triggering message when trigger condition is met, and the trigger condition can be as needed
Arrange;When for example, storing finishing service data;For another example, triggering message is reported when business datum meets outdated conditions,
The outdated conditions can be concrete:The manufacturing side knows that certain job order is disposed in process of service execution, just can carry out
Trigger reporting for message.
Further, in the present invention, also the business datum for migrating can be limited, specifically, first record needs to limit
Business information, including:
Receive and subscribe to request, the subscription request bag contains type of service;
Generate the subscription information with regard to type of service.
Correspondingly, in the flow process of Fig. 1, after described in step 102, pair triggering message is parsed, the method also includes:
The type of service for whether having triggering message package to contain in querying information of subscription, if it is, execute the acquisition and industry
Service type corresponding migration logic the step of;Otherwise, terminate flow process.
In the existing scheme for carrying out historical data migration, it is assumed that timing migration time be daily 2:00 AM to 3 points,
When business datum sharp increase, ultimately result in daily historical data and cannot migrate completely, cause some Data Migration failures;
For this situation, existing scheme can not be retried automatically, need manual intervention.Also, when business datum increases, make
Into migration pressure big when, cause Data Migration failure, cause migration failure;For this situation, existing scheme is also required to
Manual intervention, is migrated again.
In consideration of it, the invention provides the scheme for being migrated again automatically, specifically:
The Data Migration of storage facility located at processing plant to history library is monitored, if migration failure, is executed again described using obtaining
The step of business datum is moved to history library by the migration logic for taking.
When implementing, the Data Migration to storage facility located at processing plant to history library is monitored can be realized in the following way:
For each triggering message, migration response is all fed back, if the response with regard to storage facility located at processing plant and history library is migration failure,
Then start and migrate again, if the response with regard to storage facility located at processing plant and history library is to migrate successfully, terminate flow process.
With reference to the flow process of Fig. 2, the method that the present invention carries out historical data migration is illustrated, which includes
Following steps:
Step 201, the manufacturing side knowing how business datum meets outdated conditions, are then generated in process of service execution
Triggering message, real-time report is to message control center.
Specifically, triggering message can be formed by the encapsulation of the message of XML format.XML is exactly extensible markup language,
It is cross-platform in Internet environment, depends on the technology of content, is the strong of currently processed structured document information
Instrument.
One instantiation of following triggering message for XML format:
Wherein, businessType represents type of service in type of service, this example and is represented with code, is 1, businessId
Business numbering, that is, service identification is represented, service identification is represented with code in this example, is 12345.
Step 202, message control center obtain the triggering message that the manufacturing side is generated in real time, are stored in message queue.
Step 203, message control center extract triggering message from message queue, parse to triggering message, obtain
To type of service and service identification.
Message control center receives the triggering message of XML format, first by JDOM parsing triggering message.JDOM
It is a kind of unique Java tool kits using XML (a subset under standard generalized markup language), can be to XML
The message of form is parsed.
In this example, also the business datum for migrating can be limited, specifically, first record the business information that need to be limited,
Including:
Receive and subscribe to request, the subscription request bag contains type of service;
Generate the subscription information with regard to type of service.
Step 204, the type of service for whether having triggering message package to contain in message control center querying information of subscription, if
It is, then execution step 205;Otherwise, terminate flow process.
Step 205, message control center obtain migration logic corresponding with type of service, inquiry business from storage facility located at processing plant
Corresponding business datum is identified, business datum is moved to by history library using the migration logic for obtaining.
Step 206, message control center are monitored to the Data Migration of storage facility located at processing plant to history library, if migration failure,
Execution step 205 is then returned.
Specifically, if storage facility located at processing plant breaks down in the data migration process of history library, can be on message control center
Report failure response.
Referring to Fig. 3, it is system structure diagram that the present invention carries out historical data migration, the system includes message control
Center, storage facility located at processing plant and history library, the message control center include message manager and consumption terminal;
The message manager, obtains the triggering message that the manufacturing side is generated in real time, is sent to the consumption terminal;
The consumption terminal, receives the triggering message from the message manager, parses to triggering message, obtains
Type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business mark is right from storage facility located at processing plant
The business datum for inquiring is moved to history library using the migration logic for obtaining by the business datum that answers;
The storage facility located at processing plant, stores business datum to be migrated;
The history library, for the data that storage carries out migration operation.
Further, if the triggering message of synchronization reception is more, message manager can trigger message and be stored in message
Queue, while extract triggering message from message queue one by one executed.
In implementing, the manufacturing side can report triggering message when trigger condition is met, and the trigger condition can be as needed
Arrange;When for example, storing finishing service data;For another example, triggering message is reported when business datum meets outdated conditions,
The outdated conditions can be concrete:The manufacturing side knows that certain job order is disposed in process of service execution, just can carry out
Trigger reporting for message.
It is preferred that the system also includes the manufacturing side, in process of service execution, knowing how business datum meet expired
Condition, then generate triggering message, and triggering message is sent to the message manager.
It is preferred that the consumption terminal, also sends to the message manager and subscribe to request, the subscription request bag contains industry
Service type;
The message manager, receives and subscribes to request, generates the subscription information with regard to type of service.
It is preferred that before triggering message is sent to the consumption terminal, going back
Comprising type of service, if it is, by triggering message be sent to the consumption terminal;Otherwise, do not send.
In the existing scheme for carrying out historical data migration, it is assumed that timing migration time be daily 2:00 AM to 3 points,
When business datum sharp increase, ultimately result in daily historical data and cannot migrate completely, cause some Data Migration failures;
For this situation, existing scheme can not be retried automatically, need manual intervention.Also, when business datum increases, make
Into migration pressure big when, cause Data Migration failure, cause migration failure;For this situation, existing scheme is also required to
Manual intervention, is migrated again.
In consideration of it, the invention provides the scheme for being migrated again automatically, specifically:
The consumption terminal, is monitored to the Data Migration of storage facility located at processing plant to history library, if migration failure, to described
Message manager returns failure response;
The message manager, after reception failure response, sends triggering message to the consumption terminal again.
It is preferred that the manufacturing side is production cluster, it is made up of multiple manufacturing sides, the message end is message trunking,
It is made up of multiple message ends.
Using the present invention program, the time of migration history data disperses, and wall scroll is processed, to data base's no pressure.
So, migration time is unrestricted, can migrate historical data, it is ensured that the performance of storage facility located at processing plant within the most fast time.And
And, after failure, can retry automatically, save manual maintenance cost.
Presently preferred embodiments of the present invention is the foregoing is only, not in order to limit the present invention, all in essence of the invention
Within god and principle, any modification, equivalent substitution and improvements that is done etc. should be included in the scope of protection of the invention
Within.
Claims (11)
1. a kind of method for carrying out historical data migration, it is characterised in that the method includes:
Obtain the triggering message that the manufacturing side is generated in real time;
Parse to triggering message, obtain type of service and service identification;
Migration logic corresponding with type of service is obtained, inquiry business identifies corresponding business datum from storage facility located at processing plant, adopts
Business datum is moved to history library with the migration logic for obtaining.
2. the method for claim 1, it is characterised in that the generation of the triggering message includes:
The manufacturing side is in process of service execution, knowing how business datum meets outdated conditions, then generates triggering message.
3. the method for claim 1, it is characterised in that the method also includes:
Receive and subscribe to request, the subscription request bag contains type of service;
Generate the subscription information with regard to type of service.
4. method as claimed in claim 3, it is characterised in that after described pair of triggering message is parsed, the method
Also include:
The type of service for whether having triggering message package to contain in querying information of subscription, if it is, execute the acquisition and industry
Service type corresponding migration logic the step of;Otherwise, terminate flow process.
5. the method as any one of Claims 1-4, it is characterised in that the method also includes:
The Data Migration of storage facility located at processing plant to history library is monitored, if migration failure, is executed again described using obtaining
The step of business datum is moved to history library by the migration logic for taking.
6. a kind of system for carrying out historical data migration, it is characterised in that the system includes message control center, production
Storehouse and history library, the message control center include message manager and consumption terminal;
The message manager, obtains the triggering message that the manufacturing side is generated in real time, is sent to the consumption terminal;
The consumption terminal, receives the triggering message from the message manager, parses to triggering message, obtains
Type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business mark is right from storage facility located at processing plant
The business datum for inquiring is moved to history library using the migration logic for obtaining by the business datum that answers;
The storage facility located at processing plant, stores business datum to be migrated;
The history library, for the data that storage carries out migration operation.
7. system as claimed in claim 6, it is characterised in that the system also includes the manufacturing side, executes in business
Cheng Zhong, knowing how business datum meets outdated conditions, then generates triggering message, triggering message is sent to described disappearing
Breath manager.
8. system as claimed in claim 6, it is characterised in that the consumption terminal, also sends out to the message manager
Subscription request, the subscription request bag is sent to contain type of service;
The message manager, receives and subscribes to request, generates the subscription information with regard to type of service.
9. system as claimed in claim 8, it is characterised in that the message manager, is sending out to the consumption terminal
Before sending triggering message, the type of service for whether having triggering message package to contain in querying information of subscription is gone back, if it is, will
Triggering message is sent to the consumption terminal;Otherwise, do not send.
10. system as claimed in claim 6, it is characterised in that the consumption terminal, arrives history library to storage facility located at processing plant
Data Migration is monitored, if migration failure, returns failure response to the message manager;
The message manager, after reception failure response, sends triggering message to the consumption terminal again.
11. systems as described in claim 6-11, it is characterised in that the manufacturing side is production cluster, described disappears
Breath end is message trunking.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510565997.XA CN106503018A (en) | 2015-09-08 | 2015-09-08 | Carry out the method and system of historical data migration |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201510565997.XA CN106503018A (en) | 2015-09-08 | 2015-09-08 | Carry out the method and system of historical data migration |
Publications (1)
Publication Number | Publication Date |
---|---|
CN106503018A true CN106503018A (en) | 2017-03-15 |
Family
ID=58286711
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201510565997.XA Pending CN106503018A (en) | 2015-09-08 | 2015-09-08 | Carry out the method and system of historical data migration |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN106503018A (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109614386A (en) * | 2018-09-28 | 2019-04-12 | 阿里巴巴集团控股有限公司 | Data processing method, device, server and computer readable storage medium |
CN110765069A (en) * | 2019-09-29 | 2020-02-07 | 北京淇瑀信息科技有限公司 | Data migration method, device and system |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101794419A (en) * | 2010-04-13 | 2010-08-04 | 中兴通讯股份有限公司 | Method and device for automatically implementing business flow |
CN103617161A (en) * | 2013-09-26 | 2014-03-05 | 金蝶软件(中国)有限公司 | Data storing method and device for SAAS mode |
CN104065685A (en) * | 2013-03-22 | 2014-09-24 | 中国银联股份有限公司 | Data migration method in cloud computing environment-oriented layered storage system |
CN104536994A (en) * | 2014-12-11 | 2015-04-22 | 北京京东尚科信息技术有限公司 | Universal data migration method and device |
CN104572867A (en) * | 2014-12-18 | 2015-04-29 | 上海新炬网络信息技术有限公司 | Data processing method based on Oracle |
-
2015
- 2015-09-08 CN CN201510565997.XA patent/CN106503018A/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101794419A (en) * | 2010-04-13 | 2010-08-04 | 中兴通讯股份有限公司 | Method and device for automatically implementing business flow |
CN104065685A (en) * | 2013-03-22 | 2014-09-24 | 中国银联股份有限公司 | Data migration method in cloud computing environment-oriented layered storage system |
CN103617161A (en) * | 2013-09-26 | 2014-03-05 | 金蝶软件(中国)有限公司 | Data storing method and device for SAAS mode |
CN104536994A (en) * | 2014-12-11 | 2015-04-22 | 北京京东尚科信息技术有限公司 | Universal data migration method and device |
CN104572867A (en) * | 2014-12-18 | 2015-04-29 | 上海新炬网络信息技术有限公司 | Data processing method based on Oracle |
Non-Patent Citations (4)
Title |
---|
侯松著: "《高并发Oracle数据库系统的架构与设计》", 30 November 2014, 机械工业出版社 * |
刘洋著: "《信息存储技术原理分析》", 31 December 2014, 经济管理出版社 * |
杨铁军主编: "《产业专利分析报告 第34册 高端存储》", 30 June 2015, 知识产权出版社 * |
王占京编: "《VPN网络技术与业务应用》", 31 May 2012, 国防工业出版社 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109614386A (en) * | 2018-09-28 | 2019-04-12 | 阿里巴巴集团控股有限公司 | Data processing method, device, server and computer readable storage medium |
CN109614386B (en) * | 2018-09-28 | 2023-09-22 | 创新先进技术有限公司 | Data processing method, device, server and computer readable storage medium |
CN110765069A (en) * | 2019-09-29 | 2020-02-07 | 北京淇瑀信息科技有限公司 | Data migration method, device and system |
CN110765069B (en) * | 2019-09-29 | 2024-05-17 | 北京淇瑀信息科技有限公司 | Data migration method, device and system |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107766132B (en) | Multi-task scheduling method, application server and computer readable storage medium | |
US10169709B2 (en) | Avoiding incompatibility between data and computing processes to enhance computer performance | |
CN108055264B (en) | Scheduling apparatus and method for push streaming server, and computer-readable storage medium | |
CN106101021B (en) | A kind of real-time execution dispatching method of Ethernet system functional block | |
CN105635283A (en) | Organization and management and using method and system for cloud manufacturing service | |
CN105049256B (en) | A kind of general self defined interface message realization method and system | |
CN109040183A (en) | Node information acquisition method, device, equipment and computer readable storage medium | |
CN103019803B (en) | A kind of virtual environment system processing grid node for remote sensing satellite ground data | |
CN105868021A (en) | Task allocation method and apparatus | |
CN108989358A (en) | One kind being based on ICP/IP protocol frame operation data acquisition methods | |
CN102891768A (en) | Method and network element for network management | |
CN107179977B (en) | Database Full-automatic monitoring system based on mpm models | |
CN101562606A (en) | Open type data calling system and method | |
CN110297944A (en) | Distributed XML data processing method and system | |
CN105589410A (en) | Workshop monitoring and management system based on Android platform | |
CN106503018A (en) | Carry out the method and system of historical data migration | |
CN112650545A (en) | Configuration management system, method and storage medium | |
CN104410511A (en) | Server management method and system | |
CN103677975A (en) | System and method for having access to internal objects of application systems | |
CN102148702B (en) | Method for managing network by utilizing network configuration protocol | |
CN101692737B (en) | Light weight data synchronization system and method | |
CN109672731A (en) | A kind of distributed node information monitoring method, system and application | |
KR101736382B1 (en) | Ems server and log data management method thereof | |
CN107122246B (en) | Intelligent numerical simulation operation management and feedback method | |
CN115242660B (en) | Heterogeneous computing power federal system based on centralization, networking and execution method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
RJ01 | Rejection of invention patent application after publication | ||
RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170315 |