CN106503018A - Carry out the method and system of historical data migration - Google Patents

Carry out the method and system of historical data migration Download PDF

Info

Publication number
CN106503018A
CN106503018A CN201510565997.XA CN201510565997A CN106503018A CN 106503018 A CN106503018 A CN 106503018A CN 201510565997 A CN201510565997 A CN 201510565997A CN 106503018 A CN106503018 A CN 106503018A
Authority
CN
China
Prior art keywords
message
service
migration
triggering message
type
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201510565997.XA
Other languages
Chinese (zh)
Inventor
张子敬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Priority to CN201510565997.XA priority Critical patent/CN106503018A/en
Publication of CN106503018A publication Critical patent/CN106503018A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21Design, administration or maintenance of databases
    • G06F16/214Database migration support

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Testing And Monitoring For Control Systems (AREA)

Abstract

The invention discloses carrying out the method and system of historical data migration, wherein, the method includes:Obtain the triggering message that the manufacturing side is generated in real time;Parse to triggering message, obtain type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business identifies corresponding business datum from storage facility located at processing plant, business datum is moved to by history library using the migration logic for obtaining.The present invention program can reduce migration pressure, improve transport efficiency.

Description

Carry out the method and system of historical data migration
Technical field
The present invention relates to data processing technique, more particularly to the method and system for carrying out historical data migration.
Background technology
At present, during each business carries out data processing, substantial amounts of business datum can be generated.For example, net is being carried out During upper data message is interacted, the service interaction data being related to need to be stored.
Read-write efficiency is caused to decline to solve the problems, such as data base's factor data amount excessive, it will usually to create one Storage facility located at processing plant and history library, storage facility located at processing plant are stored to the business datum for producing in real time.Then periodically storage facility located at processing plant Historical data is moved on history library, improves the performance of storage facility located at processing plant with this;Specifically, counted using intervalometer When, timer time is reached, batch operation is carried out, the batch data in storage facility located at processing plant is moved to history library;Thus, all Phase property ground is executed.
There is following defect in the existing migration scheme:Periodically carry out, the time concentrates, to storage facility located at processing plant and history library Cause very big migration pressure;Also, the continuous growth with business, it is right that periodic migration pattern can not meet The requirement of transport efficiency.
Content of the invention
The invention provides a kind of method for carrying out historical data migration, the method can reduce migration pressure, and raising is moved Move efficiency.
The invention provides a kind of system for carrying out historical data migration, the system can reduce migration pressure, and raising is moved Move efficiency.
A kind of method for carrying out historical data migration, the method include:
Obtain the triggering message that the manufacturing side is generated in real time;
Parse to triggering message, obtain type of service and service identification;
Migration logic corresponding with type of service is obtained, inquiry business identifies corresponding business datum from storage facility located at processing plant, adopts Business datum is moved to history library with the migration logic for obtaining.
Preferably, the generation of the triggering message includes:
The manufacturing side is in process of service execution, knowing how business datum meets outdated conditions, then generates triggering message.
Preferably, the method also includes:
Receive and subscribe to request, the subscription request bag contains type of service;
Generate the subscription information with regard to type of service.
Preferably, after described pair of triggering message is parsed, the method also includes:
The type of service for whether having triggering message package to contain in querying information of subscription, if it is, execute the acquisition and industry Service type corresponding migration logic the step of;Otherwise, terminate flow process.
Preferably, the method also includes:
The Data Migration of storage facility located at processing plant to history library is monitored, if migration failure, is executed again described using obtaining The step of business datum is moved to history library by the migration logic for taking.
A kind of system for carrying out historical data migration, the system include message control center, storage facility located at processing plant and history library, institute Stating message control center includes message manager and consumption terminal;
The message manager, obtains the triggering message that the manufacturing side is generated in real time, is sent to the consumption terminal;
The consumption terminal, receives the triggering message from the message manager, parses to triggering message, obtains Type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business mark is right from storage facility located at processing plant The business datum for inquiring is moved to history library using the migration logic for obtaining by the business datum that answers;
The storage facility located at processing plant, stores business datum to be migrated;
The history library, for the data that storage carries out migration operation.
Preferably, the system also includes the manufacturing side, in process of service execution, knowing how business datum meet expired Condition, then generate triggering message, and triggering message is sent to the message manager.
Preferably, the consumption terminal, also sends to the message manager and subscribes to request, and the subscription request bag contains industry Service type;
The message manager, receives and subscribes to request, generates the subscription information with regard to type of service.
Preferably, the message manager, before triggering message is sent to the consumption terminal, goes back querying information of subscription In whether have the triggering type of service that contains of message package, if it is, triggering message is sent to the consumption terminal;Otherwise, Do not send.
Preferably, the consumption terminal, is monitored to the Data Migration of storage facility located at processing plant to history library, if migration failure, Then failure response is returned to the message manager;
The message manager, after reception failure response, sends triggering message to the consumption terminal again.
Preferably, the manufacturing side is production cluster, and the message end is message trunking.
From such scheme as can be seen that in the present invention, obtaining the triggering message that the manufacturing side is generated in real time;To triggering message Parsed, obtained type of service and service identification;Migration logic corresponding with type of service is obtained, from storage facility located at processing plant Inquiry business identifies corresponding business datum, and business datum is moved to history library using the migration logic for obtaining.This The bright mode for employing real-time migration, and the concrete scheme migrated with reference to type of service and service identification is given, So as to, reduce and very big migration pressure is caused to storage facility located at processing plant and history library, also improve transport efficiency, meeting needs Ask.
Description of the drawings
Fig. 1 is the method indicative flowchart that the present invention carries out historical data migration;
Fig. 2 is the method flow diagram example that the present invention carries out historical data migration;
Fig. 3 is the system structure diagram that the present invention carries out historical data migration.
Specific embodiment
For making the object, technical solutions and advantages of the present invention become more apparent, with reference to embodiment and accompanying drawing, The present invention is described in more detail.
The timed periodic of existing employing carries out the scheme of batch migration to historical data, and existence time is concentrated, migration pressure Power is big, the low defect of efficiency.In order to solve the technical problem, the invention provides the mode of real-time migration, and be given The concrete scheme that migrated with reference to type of service and service identification.
Referring to Fig. 1, it is the method indicative flowchart for carrying out historical data migration of the invention, which comprises the following steps:
Step 101, obtains the triggering message that the manufacturing side is generated in real time.
During online data message interaction is carried out, the service interaction data being related to need to be stored, the manufacturing side Storage facility located at processing plant is arrived in the storage of corresponding business datum.In the present invention, in order to carry out real-time migration, the manufacturing side is needed to enter industry In the storage of business data, generate triggering message in real time, report message control center, the triggering message package contains service class Type and service identification, type of service illustrate the data type of business datum, and service identification can be specially business datum Numbering, can inquire corresponding business datum according to service identification.
Further, if the triggering message of synchronization reception is more, message can be triggered and is stored in message queue, while Extract triggering message from message queue one by one to be executed.
Step 102, parses to triggering message, obtains type of service and service identification.
Step 103, obtains migration logic corresponding with type of service, and from storage facility located at processing plant, inquiry business mark is corresponding Business datum is moved to history library using the migration logic for obtaining by business datum.
The business datum of different service types, has each different storage logics, correspondingly, using corresponding Migration logic is migrated to corresponding business datum.
In the present invention, the triggering message that the manufacturing side is generated in real time is obtained, then, parse to triggering message, obtain Type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business mark is right from storage facility located at processing plant Business datum is moved to history library using the migration logic for obtaining by the business datum that answers.Present invention employs and move in real time The mode of shifting, and the concrete scheme migrated with reference to type of service and service identification is given, so as to it is right to reduce Storage facility located at processing plant and history library cause very big migration pressure, also improve transport efficiency, meet demand.
In implementing, the manufacturing side can report triggering message when trigger condition is met, and the trigger condition can be as needed Arrange;When for example, storing finishing service data;For another example, triggering message is reported when business datum meets outdated conditions, The outdated conditions can be concrete:The manufacturing side knows that certain job order is disposed in process of service execution, just can carry out Trigger reporting for message.
Further, in the present invention, also the business datum for migrating can be limited, specifically, first record needs to limit Business information, including:
Receive and subscribe to request, the subscription request bag contains type of service;
Generate the subscription information with regard to type of service.
Correspondingly, in the flow process of Fig. 1, after described in step 102, pair triggering message is parsed, the method also includes:
The type of service for whether having triggering message package to contain in querying information of subscription, if it is, execute the acquisition and industry Service type corresponding migration logic the step of;Otherwise, terminate flow process.
In the existing scheme for carrying out historical data migration, it is assumed that timing migration time be daily 2:00 AM to 3 points, When business datum sharp increase, ultimately result in daily historical data and cannot migrate completely, cause some Data Migration failures; For this situation, existing scheme can not be retried automatically, need manual intervention.Also, when business datum increases, make Into migration pressure big when, cause Data Migration failure, cause migration failure;For this situation, existing scheme is also required to Manual intervention, is migrated again.
In consideration of it, the invention provides the scheme for being migrated again automatically, specifically:
The Data Migration of storage facility located at processing plant to history library is monitored, if migration failure, is executed again described using obtaining The step of business datum is moved to history library by the migration logic for taking.
When implementing, the Data Migration to storage facility located at processing plant to history library is monitored can be realized in the following way: For each triggering message, migration response is all fed back, if the response with regard to storage facility located at processing plant and history library is migration failure, Then start and migrate again, if the response with regard to storage facility located at processing plant and history library is to migrate successfully, terminate flow process.
With reference to the flow process of Fig. 2, the method that the present invention carries out historical data migration is illustrated, which includes Following steps:
Step 201, the manufacturing side knowing how business datum meets outdated conditions, are then generated in process of service execution Triggering message, real-time report is to message control center.
Specifically, triggering message can be formed by the encapsulation of the message of XML format.XML is exactly extensible markup language, It is cross-platform in Internet environment, depends on the technology of content, is the strong of currently processed structured document information Instrument.
One instantiation of following triggering message for XML format:
Wherein, businessType represents type of service in type of service, this example and is represented with code, is 1, businessId Business numbering, that is, service identification is represented, service identification is represented with code in this example, is 12345.
Step 202, message control center obtain the triggering message that the manufacturing side is generated in real time, are stored in message queue.
Step 203, message control center extract triggering message from message queue, parse to triggering message, obtain To type of service and service identification.
Message control center receives the triggering message of XML format, first by JDOM parsing triggering message.JDOM It is a kind of unique Java tool kits using XML (a subset under standard generalized markup language), can be to XML The message of form is parsed.
In this example, also the business datum for migrating can be limited, specifically, first record the business information that need to be limited, Including:
Receive and subscribe to request, the subscription request bag contains type of service;
Generate the subscription information with regard to type of service.
Step 204, the type of service for whether having triggering message package to contain in message control center querying information of subscription, if It is, then execution step 205;Otherwise, terminate flow process.
Step 205, message control center obtain migration logic corresponding with type of service, inquiry business from storage facility located at processing plant Corresponding business datum is identified, business datum is moved to by history library using the migration logic for obtaining.
Step 206, message control center are monitored to the Data Migration of storage facility located at processing plant to history library, if migration failure, Execution step 205 is then returned.
Specifically, if storage facility located at processing plant breaks down in the data migration process of history library, can be on message control center Report failure response.
Referring to Fig. 3, it is system structure diagram that the present invention carries out historical data migration, the system includes message control Center, storage facility located at processing plant and history library, the message control center include message manager and consumption terminal;
The message manager, obtains the triggering message that the manufacturing side is generated in real time, is sent to the consumption terminal;
The consumption terminal, receives the triggering message from the message manager, parses to triggering message, obtains Type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business mark is right from storage facility located at processing plant The business datum for inquiring is moved to history library using the migration logic for obtaining by the business datum that answers;
The storage facility located at processing plant, stores business datum to be migrated;
The history library, for the data that storage carries out migration operation.
Further, if the triggering message of synchronization reception is more, message manager can trigger message and be stored in message Queue, while extract triggering message from message queue one by one executed.
In implementing, the manufacturing side can report triggering message when trigger condition is met, and the trigger condition can be as needed Arrange;When for example, storing finishing service data;For another example, triggering message is reported when business datum meets outdated conditions, The outdated conditions can be concrete:The manufacturing side knows that certain job order is disposed in process of service execution, just can carry out Trigger reporting for message.
It is preferred that the system also includes the manufacturing side, in process of service execution, knowing how business datum meet expired Condition, then generate triggering message, and triggering message is sent to the message manager.
It is preferred that the consumption terminal, also sends to the message manager and subscribe to request, the subscription request bag contains industry Service type;
The message manager, receives and subscribes to request, generates the subscription information with regard to type of service.
It is preferred that before triggering message is sent to the consumption terminal, going back Comprising type of service, if it is, by triggering message be sent to the consumption terminal;Otherwise, do not send.
In the existing scheme for carrying out historical data migration, it is assumed that timing migration time be daily 2:00 AM to 3 points, When business datum sharp increase, ultimately result in daily historical data and cannot migrate completely, cause some Data Migration failures; For this situation, existing scheme can not be retried automatically, need manual intervention.Also, when business datum increases, make Into migration pressure big when, cause Data Migration failure, cause migration failure;For this situation, existing scheme is also required to Manual intervention, is migrated again.
In consideration of it, the invention provides the scheme for being migrated again automatically, specifically:
The consumption terminal, is monitored to the Data Migration of storage facility located at processing plant to history library, if migration failure, to described Message manager returns failure response;
The message manager, after reception failure response, sends triggering message to the consumption terminal again.
It is preferred that the manufacturing side is production cluster, it is made up of multiple manufacturing sides, the message end is message trunking, It is made up of multiple message ends.
Using the present invention program, the time of migration history data disperses, and wall scroll is processed, to data base's no pressure. So, migration time is unrestricted, can migrate historical data, it is ensured that the performance of storage facility located at processing plant within the most fast time.And And, after failure, can retry automatically, save manual maintenance cost.
Presently preferred embodiments of the present invention is the foregoing is only, not in order to limit the present invention, all in essence of the invention Within god and principle, any modification, equivalent substitution and improvements that is done etc. should be included in the scope of protection of the invention Within.

Claims (11)

1. a kind of method for carrying out historical data migration, it is characterised in that the method includes:
Obtain the triggering message that the manufacturing side is generated in real time;
Parse to triggering message, obtain type of service and service identification;
Migration logic corresponding with type of service is obtained, inquiry business identifies corresponding business datum from storage facility located at processing plant, adopts Business datum is moved to history library with the migration logic for obtaining.
2. the method for claim 1, it is characterised in that the generation of the triggering message includes:
The manufacturing side is in process of service execution, knowing how business datum meets outdated conditions, then generates triggering message.
3. the method for claim 1, it is characterised in that the method also includes:
Receive and subscribe to request, the subscription request bag contains type of service;
Generate the subscription information with regard to type of service.
4. method as claimed in claim 3, it is characterised in that after described pair of triggering message is parsed, the method Also include:
The type of service for whether having triggering message package to contain in querying information of subscription, if it is, execute the acquisition and industry Service type corresponding migration logic the step of;Otherwise, terminate flow process.
5. the method as any one of Claims 1-4, it is characterised in that the method also includes:
The Data Migration of storage facility located at processing plant to history library is monitored, if migration failure, is executed again described using obtaining The step of business datum is moved to history library by the migration logic for taking.
6. a kind of system for carrying out historical data migration, it is characterised in that the system includes message control center, production Storehouse and history library, the message control center include message manager and consumption terminal;
The message manager, obtains the triggering message that the manufacturing side is generated in real time, is sent to the consumption terminal;
The consumption terminal, receives the triggering message from the message manager, parses to triggering message, obtains Type of service and service identification;Migration logic corresponding with type of service is obtained, inquiry business mark is right from storage facility located at processing plant The business datum for inquiring is moved to history library using the migration logic for obtaining by the business datum that answers;
The storage facility located at processing plant, stores business datum to be migrated;
The history library, for the data that storage carries out migration operation.
7. system as claimed in claim 6, it is characterised in that the system also includes the manufacturing side, executes in business Cheng Zhong, knowing how business datum meets outdated conditions, then generates triggering message, triggering message is sent to described disappearing Breath manager.
8. system as claimed in claim 6, it is characterised in that the consumption terminal, also sends out to the message manager Subscription request, the subscription request bag is sent to contain type of service;
The message manager, receives and subscribes to request, generates the subscription information with regard to type of service.
9. system as claimed in claim 8, it is characterised in that the message manager, is sending out to the consumption terminal Before sending triggering message, the type of service for whether having triggering message package to contain in querying information of subscription is gone back, if it is, will Triggering message is sent to the consumption terminal;Otherwise, do not send.
10. system as claimed in claim 6, it is characterised in that the consumption terminal, arrives history library to storage facility located at processing plant Data Migration is monitored, if migration failure, returns failure response to the message manager;
The message manager, after reception failure response, sends triggering message to the consumption terminal again.
11. systems as described in claim 6-11, it is characterised in that the manufacturing side is production cluster, described disappears Breath end is message trunking.
CN201510565997.XA 2015-09-08 2015-09-08 Carry out the method and system of historical data migration Pending CN106503018A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201510565997.XA CN106503018A (en) 2015-09-08 2015-09-08 Carry out the method and system of historical data migration

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201510565997.XA CN106503018A (en) 2015-09-08 2015-09-08 Carry out the method and system of historical data migration

Publications (1)

Publication Number Publication Date
CN106503018A true CN106503018A (en) 2017-03-15

Family

ID=58286711

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201510565997.XA Pending CN106503018A (en) 2015-09-08 2015-09-08 Carry out the method and system of historical data migration

Country Status (1)

Country Link
CN (1) CN106503018A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109614386A (en) * 2018-09-28 2019-04-12 阿里巴巴集团控股有限公司 Data processing method, device, server and computer readable storage medium
CN110765069A (en) * 2019-09-29 2020-02-07 北京淇瑀信息科技有限公司 Data migration method, device and system

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101794419A (en) * 2010-04-13 2010-08-04 中兴通讯股份有限公司 Method and device for automatically implementing business flow
CN103617161A (en) * 2013-09-26 2014-03-05 金蝶软件(中国)有限公司 Data storing method and device for SAAS mode
CN104065685A (en) * 2013-03-22 2014-09-24 中国银联股份有限公司 Data migration method in cloud computing environment-oriented layered storage system
CN104536994A (en) * 2014-12-11 2015-04-22 北京京东尚科信息技术有限公司 Universal data migration method and device
CN104572867A (en) * 2014-12-18 2015-04-29 上海新炬网络信息技术有限公司 Data processing method based on Oracle

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101794419A (en) * 2010-04-13 2010-08-04 中兴通讯股份有限公司 Method and device for automatically implementing business flow
CN104065685A (en) * 2013-03-22 2014-09-24 中国银联股份有限公司 Data migration method in cloud computing environment-oriented layered storage system
CN103617161A (en) * 2013-09-26 2014-03-05 金蝶软件(中国)有限公司 Data storing method and device for SAAS mode
CN104536994A (en) * 2014-12-11 2015-04-22 北京京东尚科信息技术有限公司 Universal data migration method and device
CN104572867A (en) * 2014-12-18 2015-04-29 上海新炬网络信息技术有限公司 Data processing method based on Oracle

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
侯松著: "《高并发Oracle数据库系统的架构与设计》", 30 November 2014, 机械工业出版社 *
刘洋著: "《信息存储技术原理分析》", 31 December 2014, 经济管理出版社 *
杨铁军主编: "《产业专利分析报告 第34册 高端存储》", 30 June 2015, 知识产权出版社 *
王占京编: "《VPN网络技术与业务应用》", 31 May 2012, 国防工业出版社 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109614386A (en) * 2018-09-28 2019-04-12 阿里巴巴集团控股有限公司 Data processing method, device, server and computer readable storage medium
CN109614386B (en) * 2018-09-28 2023-09-22 创新先进技术有限公司 Data processing method, device, server and computer readable storage medium
CN110765069A (en) * 2019-09-29 2020-02-07 北京淇瑀信息科技有限公司 Data migration method, device and system
CN110765069B (en) * 2019-09-29 2024-05-17 北京淇瑀信息科技有限公司 Data migration method, device and system

Similar Documents

Publication Publication Date Title
CN107766132B (en) Multi-task scheduling method, application server and computer readable storage medium
US10169709B2 (en) Avoiding incompatibility between data and computing processes to enhance computer performance
CN108055264B (en) Scheduling apparatus and method for push streaming server, and computer-readable storage medium
CN106101021B (en) A kind of real-time execution dispatching method of Ethernet system functional block
CN105635283A (en) Organization and management and using method and system for cloud manufacturing service
CN105049256B (en) A kind of general self defined interface message realization method and system
CN109040183A (en) Node information acquisition method, device, equipment and computer readable storage medium
CN103019803B (en) A kind of virtual environment system processing grid node for remote sensing satellite ground data
CN105868021A (en) Task allocation method and apparatus
CN108989358A (en) One kind being based on ICP/IP protocol frame operation data acquisition methods
CN102891768A (en) Method and network element for network management
CN107179977B (en) Database Full-automatic monitoring system based on mpm models
CN101562606A (en) Open type data calling system and method
CN110297944A (en) Distributed XML data processing method and system
CN105589410A (en) Workshop monitoring and management system based on Android platform
CN106503018A (en) Carry out the method and system of historical data migration
CN112650545A (en) Configuration management system, method and storage medium
CN104410511A (en) Server management method and system
CN103677975A (en) System and method for having access to internal objects of application systems
CN102148702B (en) Method for managing network by utilizing network configuration protocol
CN101692737B (en) Light weight data synchronization system and method
CN109672731A (en) A kind of distributed node information monitoring method, system and application
KR101736382B1 (en) Ems server and log data management method thereof
CN107122246B (en) Intelligent numerical simulation operation management and feedback method
CN115242660B (en) Heterogeneous computing power federal system based on centralization, networking and execution method

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20170315