CN116303341A

CN116303341A - Data migration method and device

Info

Publication number: CN116303341A
Application number: CN202310039654.4A
Authority: CN
Inventors: 何智鸿; 郑广彬; 廖阳春
Original assignee: Guangzhou Zhufeng Network Technology Co ltd
Current assignee: Guangzhou Zhufeng Network Technology Co ltd
Priority date: 2023-01-12
Filing date: 2023-01-12
Publication date: 2023-06-23

Abstract

The invention provides a data migration method, which comprises the following steps: responding to the triggered of the target program, and acquiring the input migration parameters; according to the migration parameters, matching and triggering corresponding policy parent classes, and executing corresponding migration policies; each policy parent corresponds to one of the migration policies; triggering a corresponding migration subclass according to the migration parameters and the selected policy parent class, wherein the migration subclass is inherited from the policy parent class; according to the migration parameters, the triggered migration subclasses acquire target data from a source service system in batches according to the migration strategy; and the migration subclass writes the target data into a target database in batches according to the migration strategy. A data migration apparatus is also provided. The invention aims to provide a big data migration method which can be compatible with a plurality of service systems and is convenient to expand and maintain.

Description

Data migration method and device

Technical Field

The present invention relates to data processing, and more particularly, to a data migration method and apparatus.

Background

There is a need for each business system to synchronize orders and order related information to the business system for statistical reconciliation, billing, and some data adjustment and processing from a risk and audit reducing perspective. Just because the data of multiple business systems need to be collected in the financial system at the same time every day, the difficulty and complexity of data synchronization of the multiple business systems are accompanied, and finally, the code complexity and future maintainability of managing various migration methods are also involved.

And with the development of respective business, the order quantity is also larger and larger, and especially millions of data quantity generated by the card system of the internet of things in each month can easily cause the breakdown of the application or the downtime of the database server if the data quantity is improperly migrated.

At present, a cloud platform is used to provide a cross-database migration synchronization technology in many fields, and although data support conversion from Oracle to Mysql data sources, some processing needs to be further performed during order data migration, for example, adjustment and supplementation of target data, merging of multiple source table data, and the like. When the table structure is different from the target table structure, the scheme cannot be well applied. This approach is not used, especially if the database of the business system is not deployed on a cloud service.

The technology used in the current industry is different, the traditional method is to call the interfaces of each service system to acquire the required data, but each service system needs to put the data interface outside, which is not beneficial to maintenance and expansion, and when the data volume is huge, the interface is used to meet the required requirement, and the normal operation of the service system is easily affected. It is contemplated to do data migration in a migration target system with multiple data source configurations in respective corresponding business modules or micro-services on the migration system. However, the migration modes are different, the service A only needs full migration, the service B needs timing T-N migration, and the service C needs timing T-N migration, so that the redundancy degree of codes in each module is higher when a set of micro services are respectively realized. And if a migration project needs to be newly expanded or a new service type is accessed, a migration scheme needs to be rewritten for the target service system correspondingly, and the maintenance cost rises with the increase of the redundancy degree of the codes. For hundreds of thousands or even millions of data generated on the same day, the migration volume cannot be dynamically regulated, so that the service of a timer is down, other normal migration plans are influenced, and the data which are lack of migration need to be monitored every other day and manually triggered again.

Disclosure of Invention

In view of the above-described drawbacks of the prior art, an object of the present invention is to provide a big data migration method compatible with a plurality of service systems, and convenient for expansion and maintenance.

The application scheme provides a data migration method, which comprises the following steps:

responding to the triggered of the target program, and acquiring the input migration parameters;

according to the migration parameters, matching and triggering corresponding policy parent classes, and executing corresponding migration policies; each policy parent corresponds to one of the migration policies;

triggering a corresponding migration subclass according to the migration parameter, wherein the migration subclass is inherited from the policy parent class;

according to the migration parameters, the triggered migration subclasses acquire target data from a source service system in batches according to the migration strategy;

and the migration subclass writes the target data into a target database in batches according to the migration strategy.

Further, the migration parameters include a time range, a source service system identifier, a source data table name, and a migration version number.

Further, the matching the corresponding migration policy according to the migration parameter includes: and traversing all migration subclasses preset with migration annotations by using the source service system identification and the source data table name in the migration parameters, and matching and selecting the migration strategy in the annotations on the corresponding migration subclasses.

Further, the migration policy includes a time slicing policy and a full-scale batching policy.

Further, the matching and triggering the corresponding policy parent class according to the migration parameter includes: traversing preset migration notes by using the source service system identifier and the source data table name in the migration parameters, and matching and selecting the corresponding policy parent class; the migration annotation comprises a source service system identifier, a source data table name, and a corresponding policy parent class and migration subclass.

Further, if the time slicing strategy is executed, inquiring the data volume of the target data according to the time range, and judging whether the data volume exceeds a threshold value or not; and if so, carrying out batch processing on the target data according to time halving segmentation, and circulating the judgment and halving segmentation until each batch of target data after batch processing does not exceed a threshold value.

Further, if the full-batch strategy is executed, performing paging calculation according to a set threshold value from 0 in the time range condition, performing data query and migration according to the paging number range, and circulating the paging calculation and migration until no data can be found in the calculated paging condition.

Further, the method further comprises the following steps: the step of adding the migration subclass is as follows: configuring the migration annotation, and adding the migration subclass under the appropriate policy parent class; and the migration subclass rewrites the data acquisition method and the data writing method of the policy parent class inherited by the migration subclass.

Further, after the target data is obtained, the target data is packaged into a soft reference object, and then the soft reference object is written into the target database.

There is also provided a data migration apparatus comprising:

the parameter acquisition component is used for responding to the triggered target program and acquiring the transmitted migration parameters;

migration annotation, according to the migration parameters, matching and triggering corresponding policy parent classes and migration subclasses;

the policy parent class executes a corresponding migration policy; each policy parent corresponds to one of the migration policies;

the migration subclass acquires target data from a source service system according to the migration parameters, and writes the target data into a target database in batches according to the migration strategy; the migration subclass inherits from the policy parent class.

Further, the method further comprises the following steps: the component newly-adding module is used for newly adding the migration component by configuring the migration annotation and newly adding the migration subclass under the proper policy parent class; and the migration subclass rewrites the data acquisition method and the data writing method of the policy parent class inherited by the migration subclass.

The improvement of the present application brings the following advantages: the method unifies the data migration inlets of a plurality of source service systems, and realizes the identity of the migration inlets; the plug-in processing is carried out on the migration subclasses, so that the expandability is improved, the coupling degree of migration functions is reduced, and the code redundancy is reduced; the same migration flow is used, so that the research and development cost and the learning cost are saved, and the maintainability of codes is greatly improved.

Drawings

Fig. 1 is a schematic system architecture diagram of a data migration method according to an embodiment of the present application.

Fig. 2 is a schematic step flow diagram of a data migration method according to an embodiment of the present application.

Fig. 3 is a schematic structural flow diagram of a data migration method according to an embodiment of the present application.

Fig. 4 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present application.

Detailed Description

Other advantages and effects of the present invention will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present invention with reference to specific examples. The invention may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present invention.

Fig. 1 is a schematic diagram of a system architecture of a data migration method according to the present invention, and as shown in fig. 1, the system architecture includes a plurality of source service systems 100, a migration platform 200, and a target database 300.

The source service systems may be a plurality of databases, servers, service platforms and the like storing target data, and each source service system may store a plurality of data tables with different data structures. The target program set in the migration platform can interface and access a plurality of different source business systems and acquire data thereof, and then write target data into a target database. The migration platform can be a local platform or a cloud platform; either a single server or a cluster of servers. The target database is the destination of the data migration for storing the migrated data, such as the financial system described in the background. The target database may be independent of the migration platform or may be integral with the migration platform.

The method and the system aim to realize the application are to meet the requirements of target data of a plurality of different business systems, target programs in a migration platform migrate the data to a target database through a unified migration entrance by acquiring the incoming migration parameters and then starting corresponding migration components (policy parent class, migration subclass and the like).

Next, from the perspective of a migration platform, an application of a data migration method is described, as shown in fig. 2 and 3, fig. 2 shows a flow chart of steps of the data migration method provided in an embodiment of the present application, and fig. 3 shows a flow chart of a structure of the data migration method provided in an embodiment of the present application (a policy class in the figure is a generic name of a policy parent class), where the method may include the following steps:

s100: in response to the triggering of the target program, the incoming migration parameters are acquired.

Specifically, the triggering modes of the target program comprise timer triggering, manual triggering and third party interface triggering.

The manual trigger is a time range selected by an operator for data migration. Timer triggering can trigger the target program at a certain time point of each day to perform migration tasks. Timer triggering migration can be divided into T-N (T means the date of the day, N means the first N days of the day) migration and whole month migration of the last month, and the date before the N days of the day is calculated through the T-N, so that a time range needing to be migrated is obtained. Some target data is ductile in existence, the data state of the day is not necessarily already final, e.g., some order data may take days to know whether the order was successful or failed. In order to ensure the stability of the target data, it is necessary to ensure that the target data are all in a final state. Then a time slicing strategy of T-N is configured on the corresponding migration note to ensure that the target data is in the final state.

Specifically, the migration parameters include a time range, a source service system identifier, a source data table name, and a migration version number. The source service system identifier manages the corresponding relation between the identifier code and the service name by the enumeration class. The migration version number is a means for distinguishing the same source service system, the same source database table name, different migration modes and data processing modes. For example, version A migration may operate on data and version B migration may operate on data as a subtraction. The range of migration and the manner of migration can be confirmed by the above four dimensions of the migration parameters, including the migration time.

S200: and matching and triggering the corresponding policy parent class according to the migration parameters, and executing the corresponding migration policy. Each policy parent corresponds to a migration policy.

In particular, migration policies include time slicing policies and full-scale batch policies.

And inquiring the data volume of the source service system according to the time range, and judging whether the data volume can cause the migration platform to exceed the maximum migration volume. The maximum migration amount means that the migration platform can migrate data at a normal speed at the same time. If the data volume to be migrated at the same time is too large, the migration speed of the migration platform is seriously affected, and even blocking and downtime are caused. Therefore, it is necessary to control the amount of data to be migrated at the same time within a proper range. The maximum migration volume is used as a judging threshold value to be configured in a configuration file of a migration platform, and when a migration strategy is executed, the target data can be batched according to the data volume of the target data and the threshold value, wherein the time slicing strategy of batching according to time and the full-volume batching strategy of batching according to the data volume are included. And the migration strategy is used for batching the target data with larger data volume, so that when the migration subclass migrates the data, the queried data does not exceed the maximum migration volume, and the memory consumption of the migration platform during migration is ensured to be within a certain range. If the target data is more than the threshold value, halving calculation is carried out according to the time granularity or the data quantity, and the judgment and halving segmentation are circulated until each batch of target data after batch does not exceed the threshold value.

Full batch strategy: when the target data does not have a time dimension or a large amount of data in a very short time, and the data volume cannot be reduced by time slicing, a full-batch strategy can be selected to be used, and a strategy parent class corresponding to the full-batch strategy is configured in the migration annotation, so that the migration component continuously migrates the data according to the data volume batch mode when migrating the data.

Specifically, the source service system identifier and the source data table name are used for traversing the preset migration annotation, and the corresponding policy parent class is selected in a matching mode. The migration annotation comprises a source service system identification, a source data table name, and a corresponding policy parent class and migration subclass. For example, when an order of the internet of things card needs to be migrated, a time range, an internet of things card system identifier, an order form name and a corresponding migration version number need to be migrated are transmitted. Then the migration annotation can be traversed according to the information, and the migration policy configured in the migration annotation (for example, using a full-scale batch policy) and the corresponding policy parent class and migration child class can be found, so as to trigger the corresponding policy parent class to execute the migration policy.

In this example, when executing the full batch policy abstract class, according to the maximum migration volume on the configuration file of the migration platform, the paging start parameter and the paging end parameter in the current loop batch are calculated, and when calling the query method of the subclass, the paging parameter is transmitted and the batch migration is performed.

For another example, if the card table data of the card system of the internet of things sets a time slicing strategy in the migration annotation, then the strategy method of slicing time migration is executed with respect to the start time and the end time (time range).

S300: triggering corresponding migration subclasses according to the migration parameters; the migration subclass inherits from the policy parent class.

S400: and according to the migration parameters, the triggered migration subclasses acquire target data from the source business system in batches according to the migration strategy.

And the migration strategy is used for batching the target data with larger data volume, so that when the migration subclass migrates the data, the queried data does not exceed the maximum migration volume, and the memory consumption of the migration platform during migration is ensured to be within a certain range.

Specifically, after the target data is acquired, the target data is packaged as a soft reference object. The object reference clearing space is released automatically in time when the system memory is insufficient, so that program breakdown is avoided. For a soft-referenced object, when the system memory space is sufficient, it will not be reclaimed by the system, and the program can use the object as well; when the system memory space is insufficient, the system may reclaim it.

S500: the migration subclass writes the target data into the target database in batches according to the migration policy.

The method provided by the embodiment unifies the entry (unified access migration platform) of each different service system for data migration, and realizes the uniformity of migration functions; because only one interface is provided, the migration operations such as manual triggering and the like can be conveniently and uniformly carried out on the platform.

The method carries out plug-in processing on the migration class of each service system (only the corresponding migration annotation and migration subclass are required to be configured), improves expandability, reduces the workload of relevant service development and access, reduces the redundancy of codes, and reduces the strong dependence of a migration platform and the migration class.

As the unified use interface is realized and the same flow is used, the research and development cost and the learning cost are saved, and the maintainability of codes is greatly improved.

Through a series of memory optimization measures: the fine granularity time cutting method, the batch migration method, the data soft reference measure and the like reduce the resource consumption and greatly improve the system stability.

The method greatly reduces the dependence on each service system and reduces the influence on a migration platform, the service system is not perceived on the migration process, and the migration platform can directly acquire data from a data source only by acquiring the data source connection of the service system.

As one specific example, two of the existing A, B data, which are also millions of levels, need to be synchronized to the migration platform.

The data of the data source A is characterized by standard order data, the creation time of the order is provided, the time distribution is uniform, but the order data is ductile, the state of the order on the same day is not necessarily the final state, and whether the order is successful or failed can be known only on the same day or the last day. Then a time slicing strategy of T-N is configured on the corresponding migration note, setting N to 4 days. The timer calculates the date and time period of T-4 when migrating after acquiring the information of the migration annotation, namely acquiring the data of the whole day of the first four days of the time of the day _。 This N is the time that guarantees that orders will be in the final state, and that data that was migrated all day before N days each day will ensure that order data that was migrated today is already in the final state. However, assuming that three million pieces of data exist in the day, the maximum migration amount of the migration platform (i.e. the threshold value is exceeded) is exceeded, and the data in the day cannot be all queried out at a time. Because the excessive data volume can overwhelm the migration platform server and also affect the source database performance, the source business system is affected accordingly. When the policy parent class detects that the data amount of the target data exceeds a set threshold, the target data is batched by shortening the migration time by half. For example: the data of the time period of 00:00:00-23:59:59 is migrated originally, the time period of 00:00:00-11:59:59 is returned after the calculation of the halving algorithm, whether the time period exceeds a threshold value is searched again in a recursion mode, and if the time period exceeds the threshold value, the halving time is calculated continuously in the recursion mode; if the data migration time is not longer than the predetermined time, the target data is migrated, and then the data migration is performed for the period of 12:00:00-23:59:59.

Assume again that: data B is on the order of millions as data a, but is characterized by the fact that the target data does not have a time field that can be conditioned, or that millions of data are already in the range of a few seconds, then the use of a time slicing strategy is no longer applicable. Then the migration policy with migration annotation being full batch is configured, the policy parent class executes the full batch policy according to the configuration of the migration annotation, and batches the target data according to the data volume until the complete migration.

As an embodiment, further comprising:

s300: the steps of adding the migration subclass are as follows: configuring migration notes, and adding migration subclasses under a proper policy parent class; the migration subclass rewrites the data acquisition method and the data writing method of the inherited policy parent class.

When the migration platform needs to expand a new migration item (namely, migrating certain specific data through a given time range, a source service system identifier, a source data table name and a migration strategy), only one migration subclass corresponding to the migration item needs to be added in a target program of the migration platform. And expanding a migration subclass under the selected policy parent class, wherein the migration subclass inherits from the policy parent class, and rewrites the data acquisition method and the data writing method of the inherited policy parent class, so that the function of migrating data according to the migration policy corresponding to the policy parent class can be realized. Then adding migration annotation on the subclass and configuring parameters corresponding to the migration item for scanning and triggering, for example, configuring source service system identification and source data table name corresponding to the migration item so as to define the service and data table range migrated by the subclass; configuring a migration strategy identifier corresponding to the migration item to trigger a corresponding strategy parent class, and adopting a proper migration strategy to perform data migration; if there is a data delay migration (T-N migration) requirement, configuring delay days corresponding to the migration item to assist the timer in calculating the migration time range parameters in a targeted manner when executing the migration.

According to the method, the migration subclasses are subjected to componentization and plug-in processing, when a new migration project is expanded, only the new migration subclasses are required to be configured on the migration platform, the expandability is improved, and the workload of relevant service development and access is greatly reduced. Meanwhile, the new migration subclass only needs to configure migration annotation on the subclass, select a proper policy parent class according to a migration policy, and then rewrite a data acquisition and data storage method of the parent class by using the migration subclass, so that a set of service codes specially connected with a new migration project does not need to be rewritten, the development code quantity and code redundancy are greatly reduced, and the coupling between a migration program and the migration subclass is also reduced.

As a specific example, assuming that a refund statement (i.e. a migration item) is to be newly migrated to the card system of the internet of things, a technician determines that the data size of the statement to be migrated within a certain time range is not very large, and has a uniform transaction time field, so that the policy of time slicing is suitable for use. Then a migration subclass is created under the policy parent class corresponding to the time slicing policy and a migration annotation is configured for the migration subclass. The migration annotation information comprises: the system identification of the internet of things card, the name of the refund list, the time slicing strategy and the T-N value of the time slicing strategy (N is set to be 1, namely, the data of the whole day before 1 day of the day, and T is the date of the day). The migration subclass inherits from the corresponding policy parent class, and the method of inquiring source data and the method of storing data of the policy parent class are rewritten, so that the migration subclass for migrating the new table can be completed. The coupling degree between the migration logic part and the realization part is reduced, and the realization of the future expansion of the part is facilitated, so that the integration and development of the migration logic part are simple and quick, and the maintainability of the system is improved.

The above is a typical strategy of using time to change space, however, in practical use, the time consumption is not slower than that of the conventional method, because the database has performance problems in terms of processing large-batch data, different data sources, when the data throughput is too large, the situation that the data source server occupies large resources, the processing data volume is large, the response is slow, the connection is overtime, and the like can occur, and the batch policy is used, so that the stability of the source data source server and the migration platform can be ensured, and the migration performance and efficiency can be ensured.

As one embodiment, the method of interrupt is also included in the loop migration process, if the user needs to interrupt the migration task in progress, an interrupt interface is called, and the interface stores an interrupt flag in a database or middleware (such as Redis). When the target program detects that the interrupt mark for the migration task exists when the target program is in target data, the rest migration is abandoned and directly exits, the migration is interrupted, and meaningless data migration and resource consumption caused by some possible misoperation are stopped. The flag can be initiated by an external third party interface, can be a parameter input by a page of the platform, and can also extend other interrupt methods. In this embodiment, whether the Redis contains the flag is detected, and the detection can be implemented by other tools. However, the migration process cannot be influenced by the exception of the middleware or the database, for example, when the request fails due to various problems during the migration process, the exception is wrongly thrown out, so that the whole migration is forced to be interrupted and exit. When encoding, various abnormal conditions need to be considered completely, then the exception is captured and processed, and then migration is continuously executed, so that the stability can be ensured.

As one embodiment, there is a logging mechanism that enables the entire migration flow to be stored and provided for review in a database from a start request to each round of migration, to a log record of complete migration completion.

In an embodiment of the present application, as shown in fig. 4, fig. 4 shows a schematic structural diagram of a data migration apparatus provided in this embodiment, including:

the parameter acquisition component 1 responds to the triggered target program to acquire the incoming migration parameters;

migration annotation 2, according to migration parameters, matching and triggering corresponding policy parent classes and migration subclasses;

the policy parent class 3 executes a corresponding migration policy; each policy parent corresponds to a migration policy;

the migration subclass 4 acquires target data from the source service system according to migration parameters, and writes the target data into a target database in batches according to migration strategies; the migration subclass inherits from the policy parent class.

As an embodiment, further comprising: the component newly-added module is used for newly adding migration subclasses under a proper policy parent class to newly add migration components by configuring migration notes; the migration subclass rewrites the data acquisition method and the data writing method of the inherited policy parent class.

The present invention is not limited to the above-mentioned embodiments, and any changes or substitutions that can be easily understood by those skilled in the art within the technical scope of the present invention are intended to be included in the scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method of data migration, comprising:

2. The method of claim 1, wherein the migration parameters include a time range, a source service system identification, a source data table name, and a migration version number.

3. The method of claim 2, wherein said matching and triggering the corresponding policy parent class according to the migration parameters comprises: traversing all migration subclasses in the migration annotation by using the source service system identifier and the source data table name in the migration parameters, and matching and selecting the corresponding migration subclasses; the migration annotation comprises a source service system identifier, a source data table name, a corresponding strategy identifier and migration delay days, wherein the delay days enable a timer to calculate a time range of migration required by the migration subclass when the timer executes migration.

4. A data migration method according to claim 3, wherein said migration policies include a time slicing policy and a full-scale batching policy.

5. The method according to claim 4, wherein if the time slicing strategy is executed, determining whether the data amount exceeds a threshold value according to the data amount of the target data queried in the time range; and if so, carrying out batch processing on the target data according to time halving segmentation, and circulating the judgment and halving segmentation until each batch of target data after batch processing does not exceed a threshold value.

6. The method according to claim 4, wherein if the full-batch policy is executed, performing a paging calculation according to a set threshold value from 0 in the time range condition, performing data inquiry and migration according to the paging number range, and circulating the paging calculation and migration until no data is found in the calculated paging condition.

7. A method of data migration according to claim 3, further comprising: the step of adding the migration subclass is as follows: configuring the migration annotation, and adding the migration subclass under the appropriate policy parent class; and the migration subclass rewrites the data acquisition method and the data writing method of the policy parent class inherited by the migration subclass.

8. A data migration method according to claim 1, wherein after the target data is obtained, the target data is packaged as a soft reference object and then written into the target database.

9. A data migration apparatus, comprising:

10. The data migration apparatus of claim 9, further comprising: the component newly-adding module is used for newly adding the migration component by configuring the migration annotation and newly adding the migration subclass under the proper policy parent class; and the migration subclass rewrites the data acquisition method and the data writing method of the policy parent class inherited by the migration subclass.