CN116955319A - Service data migration method, device, equipment and storage medium - Google Patents

Service data migration method, device, equipment and storage medium Download PDF

Info

Publication number
CN116955319A
CN116955319A CN202310942703.5A CN202310942703A CN116955319A CN 116955319 A CN116955319 A CN 116955319A CN 202310942703 A CN202310942703 A CN 202310942703A CN 116955319 A CN116955319 A CN 116955319A
Authority
CN
China
Prior art keywords
data
transplanted
migration
service
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310942703.5A
Other languages
Chinese (zh)
Inventor
练志宏
骆栋鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Merchants Bank Co Ltd
Original Assignee
China Merchants Bank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Merchants Bank Co Ltd filed Critical China Merchants Bank Co Ltd
Priority to CN202310942703.5A priority Critical patent/CN116955319A/en
Publication of CN116955319A publication Critical patent/CN116955319A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/11—File system administration, e.g. details of archiving or snapshots
    • G06F16/119—Details of migration of file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2365—Ensuring data consistency and integrity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/242—Query formulation
    • G06F16/2433—Query languages

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to the technical field of data processing, and discloses a service data migration method, a device, equipment and a storage medium, wherein the method comprises the following steps: when receiving a service migration requirement of a user, generating service data to be transplanted according to the service migration requirement; transplanting and distributing the service data to be transplanted according to the system overhead condition to obtain a transplanting detail file and a temporary file to be transplanted, which correspond to the service data to be transplanted; and migrating the temporary file to be transplanted to a target system, so that the target system writes the temporary file to be transplanted into the database file when the verification result of the temporary file to be transplanted accords with the data uniqueness. According to the method, the data and the partition processing are automatically uploaded by distributing the migration detail file and the temporary file to be migrated, the data uniqueness is verified when the target system receives the data, and then the data is written into the database file, so that the data security and the integrity are ensured, the shutdown state is avoided for migrating the data, and the data migration efficiency is improved.

Description

Service data migration method, device, equipment and storage medium
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a service data migration method, device, equipment, and storage medium.
Background
With the deep development of cloud computing and the Internet, enterprise digital transformation is rapidly popularized, and a large number of enterprises make different degrees of customized development on the basis of the existing system, and the demands of system updating are brought along with the demands. With the inherent complexity of the system, the updating is not easy, as the technology of big data is developed faster and faster, the data can be the highest value product of clients, and the updating of the product tends to generate the requirement for data migration.
In the process of data migration of a system, the conventional data migration server assignment and service personnel verification functions need to be stopped for a plurality of hours or days, and data loss and migration failure are caused by tool loss and failure of migration verification tests. Service data migration is characterized by low migration efficiency and long service interruption time. How to cross-platform, safe, efficient, stable and shorten the service interruption time length becomes a problem to be solved for enterprise production data migration.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The invention mainly aims to provide a service data migration method, device, equipment and storage medium, and aims to solve the technical problem of low migration efficiency caused by shutdown, migration verification test and the like in the traditional system migration data process.
In order to achieve the above object, the present invention provides a service data migration method, which includes the following steps:
when receiving a service migration requirement of a user, generating service data to be transplanted according to the service migration requirement;
transplanting and distributing the service data to be transplanted according to the system overhead condition to obtain a transplanting detail file and a temporary file to be transplanted, which correspond to the service data to be transplanted;
and migrating the temporary file to be transplanted to a target system according to the transplanting detail file, so that the target system writes the temporary file to be transplanted into a database file when the verification result of the temporary file to be transplanted accords with the data uniqueness.
Optionally, the overhead condition includes a data priority and a storage condition;
the transplanting allocation is carried out on the service data to be transplanted according to the system overhead condition, and a transplanting detail file and a temporary file to be transplanted corresponding to the service data to be transplanted are obtained, which comprises the following steps:
transplanting and distributing the service data to be transplanted according to the storage condition of the system overhead condition to obtain a temporary file to be transplanted corresponding to the service data to be transplanted;
And extracting the temporary file to be transplanted according to the data priority of the system overhead condition and the preset service condition to obtain a transplanting detail file corresponding to the service data to be transplanted.
Optionally, when receiving a service migration requirement of a user, generating service data to be transplanted according to the service migration requirement further includes:
carrying out parameter configuration on the service data to be transplanted according to the service migration requirement to obtain a black-and-white list corresponding to the service data to be transplanted in the transplanting range;
checking the system state of the receiving system in the black-and-white list, and judging whether the checking result accords with a preset health state or not;
when the checking result accords with the preset health state, making rules for the business data to be transplanted according to the business migration requirement, and obtaining a transplanting data checking rule;
correspondingly, the extracting the temporary file to be transplanted according to the data priority and the preset service condition of the overhead condition, after obtaining the transplanting detail file corresponding to the service data to be transplanted, further includes:
checking the temporary file to be transplanted according to the data transplanting checking rule, and judging whether a checking result accords with data consistency;
And when the verification result accords with the data consistency, executing the step of migrating the temporary file to be transplanted to a target system according to the transplanting detail file.
Optionally, when the inspection result meets the preset health state, making rules on the service data to be transplanted according to the service migration requirement to obtain a verification rule of the transplanted data, including:
when the checking result accords with the preset health state, extracting the data scale of the business data to be transplanted;
traversing a preset migration strategy library according to the service migration requirements and the data scale to obtain a migration strategy corresponding to the service data to be migrated;
SQL statement configuration is carried out on the business data to be transplanted according to the business migration requirement, and a transplanted data verification rule is obtained;
correspondingly, the transplanting and distributing the to-be-transplanted service data according to the storage condition of the system overhead condition to obtain a temporary to-be-transplanted file corresponding to the to-be-transplanted service data, which comprises the following steps:
and carrying out transplanting distribution on the service data to be transplanted according to the storage condition and the strategy to be transplanted to obtain a temporary file to be transplanted corresponding to the service data to be transplanted.
Optionally, before the temporary file to be transplanted is migrated to the target system according to the migration detail file, the method further includes:
judging the current resource utilization rate of a current system and the current resource utilization rate of the receiving system according to the system state, and judging whether the current resource utilization rate is lower than a preset threshold value or not;
correspondingly, when the verification result accords with the data consistency, the step of executing the migration of the temporary file to be transplanted to a target system according to the transplanting detail file comprises the following steps:
and when the verification result accords with the data consistency and the current resource utilization rate is lower than the preset threshold value, executing the step of migrating the temporary file to be transplanted to a target system according to the transplanting detail file.
Optionally, when the verification result of the temporary file to be transplanted accords with the data uniqueness, the target system writes the temporary file to be transplanted into a database file, including:
the target system reads the number of the files to be transferred of the transplanting detail, compares the number of the files to be transferred with the number of the files to be transferred of the temporary files to be transplanted, and judges whether the comparison results are consistent;
When the comparison results are consistent, the target system converts the temporary file to be transplanted into data to be put in storage;
the target system performs data verification on the data to be put in storage and judges whether a verification result accords with data uniqueness;
and when the verification result accords with the data uniqueness, the target system writes the data to be put in storage into a database file.
Optionally, after the migration of the temporary file to be migrated to the target system according to the migration detail file, the method further includes:
when data to be put in storage sent by the target system is received, carrying out protocol comparison on the data to be put in storage through a DAP temporary table, and judging whether the data to be put in storage is consistent with the temporary file to be transplanted or not according to a comparison result;
when the data to be put in storage is inconsistent with the temporary file to be transplanted, updating the temporary file to be transplanted through a data processing tool to obtain an updated temporary file to be transplanted;
and migrating the updated temporary file to be transplanted to the target system until the data to be put in storage is consistent with the updated temporary file to be transplanted, and stopping migrating the temporary file to be transplanted to the target system according to the transplanting detail file.
In addition, to achieve the above object, the present invention further provides a service data migration apparatus, which includes:
the data generation module is used for generating service data to be transplanted according to the service migration requirement when the service migration requirement of the user is received;
the data distribution module is used for carrying out transplanting distribution on the service data to be transplanted according to the system overhead condition to obtain a transplanting detail file and a temporary file to be transplanted, which correspond to the service data to be transplanted;
and the data migration module is used for migrating the temporary file to be migrated to a target system according to the migration detail file, so that the target system writes the temporary file to be migrated into a database file when the verification result of the temporary file to be migrated accords with the data uniqueness.
In addition, to achieve the above object, the present invention also proposes a service data migration apparatus, the apparatus comprising: a memory, a processor and a service data migration program stored on the memory and executable on the processor, the service data migration program being configured to implement the steps of the service data migration method as described above.
In addition, in order to achieve the above object, the present invention also proposes a storage medium having stored thereon a service data migration program which, when executed by a processor, implements the steps of the service data migration method as described above.
When receiving a service migration requirement of a user, generating service data to be transplanted according to the service migration requirement; then transplanting and distributing the service data to be transplanted according to the system overhead condition to obtain a transplanting detail file and a temporary file to be transplanted, which correspond to the service data to be transplanted; and finally, the temporary file to be transplanted is migrated to a target system according to the transplanting detail file, so that the target system writes the temporary file to be transplanted into a database file when the verification result of the temporary file to be transplanted accords with the data uniqueness. According to the method, the data and the partition processing are automatically uploaded by distributing the migration detail file and the temporary file to be migrated, the data uniqueness is verified when the target system receives the data, and then the data is written into the database file, so that the data security and the integrity are ensured, shutdown for data migration is avoided, and the data migration efficiency is improved.
Drawings
FIG. 1 is a schematic structural diagram of a service data migration device of a hardware running environment according to an embodiment of the present invention;
FIG. 2 is a flowchart of a first embodiment of a business data migration method according to the present invention;
FIG. 3 is a flowchart of a second embodiment of a business data migration method according to the present invention;
Fig. 4 is a flow chart of real-time migration of a large amount of data to cloud service data in a second embodiment of the service data migration method of the present invention;
fig. 5 is a block diagram of a first embodiment of a service data migration apparatus according to the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a service data migration device of a hardware running environment according to an embodiment of the present invention.
As shown in fig. 1, the service data migration apparatus may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (Wi-Fi) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
Those skilled in the art will appreciate that the architecture shown in fig. 1 is not limiting of the business data migration apparatus and may include more or fewer components than shown, or may combine certain components, or may be a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a service data migration program may be included in the memory 1005 as one type of storage medium.
In the service data migration apparatus shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the service data migration device of the present invention may be disposed in the service data migration device, where the service data migration device invokes a service data migration program stored in the memory 1005 through the processor 1001, and executes the service data migration method provided by the embodiment of the present invention.
An embodiment of the present invention provides a service data migration method, referring to fig. 2, fig. 2 is a schematic flow chart of a first embodiment of the service data migration method of the present invention.
In this embodiment, the service data migration method includes the following steps:
Step S10: and when receiving a service migration requirement of a user, generating service data to be transplanted according to the service migration requirement.
It should be noted that, the execution body of the method of the present embodiment may be a computing service device, such as a personal computer, a server, etc., with data distribution, data migration, and data processing, or may be another electronic device, such as the above-mentioned service data migration device, capable of implementing the same or similar functions, which is not limited in this embodiment. Here, the present embodiment and the following embodiments will be specifically described with reference to the above-described service data migration apparatus (abbreviated as data migration apparatus).
It can be understood that the service migration requirement is a requirement for preparing to migrate the service data according to the service scene requirement, such as a requirement for migrating the data by customers such as branches, customer numbers, service numbers, migration of different batches, etc. The service data to be transplanted is data to be transplanted, which is extracted from a database according to service migration requirements.
Step S20: and carrying out transplanting distribution on the service data to be transplanted according to the system overhead condition to obtain a transplanting detail file and a temporary file to be transplanted, which correspond to the service data to be transplanted.
It should be noted that the overhead condition is the resource consumption and the additional burden involved in executing the task in the computer system, such as computing resources, memory occupation, disk space, network bandwidth, data priority, and the like.
It can be understood that the migration detail file is a file containing a specific migration detail in the data migration process of the service data to be migrated, for example, the migration process includes: single batch migration, multi-batch migration, migration time, size of each migration file, and the like. The temporary file to be transplanted is a temporary data file obtained by distributing the business data to be transplanted according to the system overhead condition.
In a specific implementation, when receiving a service migration requirement of a user, data migration equipment extracts and generates service data to be transplanted from a local database according to the service migration requirement; and then, carrying out data distribution on the service data to be transplanted according to the system overhead condition to obtain temporary files to be transplanted and distributed transplanting detail files.
Step S30: and migrating the temporary file to be transplanted to a target system according to the transplanting detail file, so that the target system writes the temporary file to be transplanted into a database file when the verification result of the temporary file to be transplanted accords with the data uniqueness.
It should be noted that, the data uniqueness refers to whether the data in the temporary file to be transplanted is repeated with the data in the target system, and whether the data is repeatedly migrated exists. When the verification result accords with the data uniqueness, the data migration is successful, and the target system writes the temporary file to be transplanted into the database.
Furthermore, during data migration, the data in the database can be formatted and organized into a data format required by the Java Spring framework, such as data statistics, standardization, redundant data removal, data classification, integration and other processes, so that the data migration efficiency is improved. Meanwhile, according to the transplanting detail file, the data can be automatically uploaded, indexed, partitioned and verified and then enter the database, so that the safety and the integrity of the data are ensured, and meanwhile, the input of high-efficiency data is ensured.
In a specific implementation, after the temporary file to be transplanted and the transplanting detail file are obtained, the data migration device may migrate the temporary file to be transplanted to the target system according to the transplanting detail file through the TCP protocol, and if the file transmission fails, the data file transmission step may be repeatedly executed. When the target system receives the temporary file to be transplanted, the target system can perform data verification on the temporary file to be transplanted, such as the uniqueness of the data, whether the data has conflict, whether logic has errors and the like; and the target system writes the temporary file to be transplanted into the database.
When receiving a service migration requirement of a user, the data migration equipment extracts and generates service data to be transplanted from a local database according to the service migration requirement; and then, carrying out data distribution on the service data to be transplanted according to the system overhead condition to obtain temporary files to be transplanted and distributed transplanting detail files. And then the temporary file to be transplanted can be migrated to the target system according to the migration detail file through the TCP protocol, and if the file transmission fails, the data file transmission step can be repeatedly executed. When the target system receives the temporary file to be transplanted, the target system can perform data verification on the temporary file to be transplanted, such as the uniqueness of the data, whether the data has conflict, whether logic has errors and the like; and the target system writes the temporary file to be transplanted into the database. According to the embodiment, the data and the partition processing are automatically uploaded by distributing the migration detail file and the temporary file to be migrated, the database file is written after the uniqueness of the data is verified when the target system receives the data, the data security and the integrity are ensured, shutdown for data migration is avoided, and the data migration efficiency is improved.
Referring to fig. 3, fig. 3 is a flow chart of a second embodiment of the service data migration method according to the present invention.
Based on the above-described first embodiment, in this embodiment, in consideration of the partition processing of data, the overhead condition includes the data priority and the storage condition, and the step S20 includes:
step S21: and carrying out transplanting distribution on the service data to be transplanted according to the storage condition of the system overhead condition to obtain a temporary file to be transplanted corresponding to the service data to be transplanted.
It should be noted that, the data priority is a migration priority corresponding to a previous connection of data. Storage is the case when storage is involved in executing tasks in a computer system, such as memory usage, disk space, etc.
Step S22: and extracting the temporary file to be transplanted according to the data priority of the system overhead condition and the preset service condition to obtain a transplanting detail file corresponding to the service data to be transplanted.
It should be noted that the preset service condition is a condition for preparing to migrate the service data according to the service scenario requirement, for example, the service scenario, the service data requirement, and the like.
In a specific implementation, the data migration device performs data distribution on service data to be transplanted according to the storage condition of the overhead condition, for example, temporary files to be transplanted are built in batches when the memory occupation is low; and then extracting the temporary files to be transplanted according to the data priority of the overhead condition and preset service conditions (such as service data requirements), and determining the transplanting detail files (such as transplanting flow, sequencing and the like) of the temporary files to be transplanted. Therefore, in the process of automatically performing data migration, operations such as indexing, partitioning processing, data verification and the like are performed, and the high-efficiency data input is ensured while the safety and the integrity of the data are ensured.
Further, in consideration of the security of the data migration process, the embodiment further includes, after step S10: carrying out parameter configuration on the service data to be transplanted according to the service migration requirement to obtain a black-and-white list corresponding to the service data to be transplanted in the transplanting range; checking the system state of the receiving system in the black-and-white list, and judging whether the checking result accords with a preset health state or not; when the checking result accords with the preset health state, making rules for the business data to be transplanted according to the business migration requirement, and obtaining a transplanting data checking rule;
correspondingly, after the step S22, the method further includes: checking the temporary file to be transplanted according to the data transplanting checking rule, and judging whether a checking result accords with data consistency; and when the verification result accords with the data consistency, executing the step of migrating the temporary file to be transplanted to a target system according to the transplanting detail file.
It should be noted that, the black-and-white list is a list of data migration targets, for example, may be configured by parameters, for example: the black and white lists of the dimensions of the whole row, the branch row, the client number, the service number and the like control the maintenance data migration.
It can be understood that the preset health state is a working state of each system in the data migration process, for example, whether the network communication state of the two parties, the CPU usage rate, the storage usage rate and the like are normal or not, and when the system state of the receiving system is in the preset health state, the data migration preparation can be performed.
It should be noted that the migration data verification rule is a rule of a data migration process, for example, a communication or data coding form.
In a specific implementation, the data migration equipment can firstly perform parameter configuration on service data to be transplanted according to service migration requirements to obtain a black-and-white list corresponding to a data migration range; and then checking the system state of the receiving system in the black-and-white list, and when the system state of the receiving system is in a preset health state, performing data migration preparation. Then making rules for the service data to be transplanted according to service migration requirements to obtain a transplanted data verification rule; correspondingly, after obtaining the migration detail file corresponding to the service data to be migrated, the temporary file to be migrated can be further verified according to a migration data verification rule, and when the verification result accords with the data consistency, the temporary file to be migrated can be migrated to a target system. Therefore, the safety and reliability of the data transmission and warehousing process are ensured through multi-level security measures and a fault recovery mechanism.
Further, in this embodiment, when the inspection result meets the preset health state, rule formulation is performed on the service data to be transplanted according to the service migration requirement, so as to obtain a verification rule of the transplanted data, including: when the checking result accords with the preset health state, extracting the data scale of the business data to be transplanted; traversing a preset migration strategy library according to the service migration requirements and the data scale to obtain a migration strategy corresponding to the service data to be migrated; SQL statement configuration is carried out on the business data to be transplanted according to the business migration requirement, and a transplanted data verification rule is obtained;
correspondingly, the step S21 includes: and carrying out transplanting distribution on the service data to be transplanted according to the storage condition and the strategy to be transplanted to obtain a temporary file to be transplanted corresponding to the service data to be transplanted.
It should be noted that the data size is the size of the service data to be transplanted, for example, the size of the transmission data, the size of the single batch or multiple batches of the data. The preset migration policy library is a migration policy preset in the data migration device, and can directly query a proper migration policy in the preset migration policy library according to service migration requirements and data size, so that the efficiency of data migration is improved.
In a specific implementation, when the receiving system accords with a preset health state, extracting the data scale of service data to be transplanted; traversing a preset migration strategy library according to service migration requirements and data scale, and determining a strategy to be migrated corresponding to the service data to be migrated. And the migration data verification rule can directly carry out SQL statement configuration according to the service migration requirement, and the standardization of management and operation of the relational database is carried out. Correspondingly, the data distribution can be carried out on the service data to be transplanted according to the storage condition and the strategy to be transplanted to obtain the corresponding temporary file to be transplanted, so that the proper transplanting strategy can be directly inquired in the preset transplanting strategy library, and the efficiency of data migration is improved.
Further, considering the working state of the system during data migration, the embodiment further includes, before step S30: judging the current resource utilization rate of a current system and the current resource utilization rate of the receiving system according to the system state, and judging whether the current resource utilization rate is lower than a preset threshold value or not;
correspondingly, when the verification result accords with the data consistency, the step of executing the migration of the temporary file to be transplanted to a target system according to the transplanting detail file comprises the following steps: and when the verification result accords with the data consistency and the current resource utilization rate is lower than the preset threshold value, executing the step of migrating the temporary file to be transplanted to a target system according to the transplanting detail file.
It should be noted that, the preset threshold is a threshold set according to the resource usage rate of the system, for example, the resource usage rate of the system is low at night, and at this time, the smoothness and the migration efficiency of data migration can be improved by performing data migration.
In a specific implementation, before data migration, the data migration device can judge whether the current resource utilization rate of the current system and the receiving system is lower than a preset threshold value according to the system state; and when the data accords with the data consistency and the current resource utilization rate is lower than the preset threshold value, the temporary file to be transplanted can be migrated to the target system according to the transplanting detail file. Therefore, when the resource utilization rate is low, the data safety and the integrity are ensured, the shutdown is avoided for transplanting the data, and the smoothness and the migration efficiency of data migration are improved.
Further, in order to obtain the benefit or reason of the right to issue, in this embodiment, when the verification result of the temporary file to be transplanted accords with the data uniqueness, the target system writes the temporary file to be transplanted into the database file, which includes: the target system reads the number of the files to be transferred of the transplanting detail, compares the number of the files to be transferred with the number of the files to be transferred of the temporary files to be transplanted, and judges whether the comparison results are consistent; when the comparison results are consistent, the target system converts the temporary file to be transplanted into data to be put in storage; the target system performs data verification on the data to be put in storage and judges whether a verification result accords with data uniqueness; and when the verification result accords with the data uniqueness, the target system writes the data to be put in storage into a database file.
In a specific implementation, after the target system receives the data, the migration data may be verified. For example, the number of transmission data in the migration detail file is read, and the number of transmission data is used for comparison with the number of data in the temporary migration file. And when the comparison results are consistent, converting the temporary file to be transplanted into data to be put in storage, and checking the uniqueness of the data to be put in storage and the existing data. And if the data uniqueness is met, writing the data to be put into storage into a database file. Thereby ensuring the safety and reliability of the data transmission and warehousing process.
Further, in consideration of the data format change in the data migration process, the embodiment further includes, after step S30: when data to be put in storage sent by the target system is received, carrying out protocol comparison on the data to be put in storage through a DAP temporary table, and judging whether the data to be put in storage is consistent with the temporary file to be transplanted or not according to a comparison result; when the data to be put in storage is inconsistent with the temporary file to be transplanted, updating the temporary file to be transplanted through a data processing tool to obtain an updated temporary file to be transplanted; and migrating the updated temporary file to be transplanted to the target system until the data to be put in storage is consistent with the updated temporary file to be transplanted, and stopping migrating the temporary file to be transplanted to the target system according to the transplanting detail file.
Note that, the DAP (Data Analysis Platform) temporary table is a table structure for temporarily storing data in the data analysis platform. In data analysis, it is often necessary to extract data from different data sources, perform operations such as cleaning, converting, and calculating, and then store the results in a table for subsequent analysis and processing. Because the protocols of the data migration process are inconsistent, special symbols or uncommon words may change, and therefore, verification and updating are needed.
In a specific implementation, when data to be stored sent by a target system is received, protocol comparison can be performed on the data to be stored through a DAP temporary table, when the data to be stored is inconsistent with a temporary file to be transplanted, the temporary file to be transplanted can be updated through a data processing tool, the updated temporary file to be transplanted is migrated into the target system, and the data migration operation is successfully completed until the data to be stored is consistent with the updated temporary file to be transplanted. Therefore, according to a multi-level fault recovery mechanism, data faults are prevented, and the reliability of data transmission and warehousing processes is ensured.
For easy understanding, the migration of mass data to cloud service data in real time is described, but the scheme is not limited. Referring to fig. 4, fig. 4 is a flow chart illustrating real-time migration of mass data to cloud service data in a second embodiment of the service data migration method according to the present invention. As shown in fig. 4, the data migration process is as follows:
Step 1: and (5) transplanting a black-and-white list. And customizing a transplanting range according to the service transplanting requirement to configure a black-and-white list. Configuration is performed by a parameter configuration module, for example: and controlling and maintaining black and white lists with full lines, branch lines, client numbers, service numbers and the like as dimensions.
Step 2: checking the health status. Checking the health state of the transplanting system of the requesting party and the receiving party. Such as checking requests, accepting both-party network connectivity status, CPU usage, storage usage, and running migration operations when both-party system resource usage is low.
Step 3: and (5) making a transplanting configuration. And (3) formulating single-batch or multi-batch transplanting configuration and customizing transplanting data verification rules. For example, the number of transplanting batches and the number of single transplanting strokes are configured by parameters according to the size of the transmission data and the transplanting strategy of the single batch or multiple batches of strokes. And simultaneously configuring SQL sentences through parameters, and configuring data verification rules meeting service requirements.
Step 4: generating data to be transplanted, generating service data to be transplanted according to service requirements, checking the data to be transplanted, and repeatedly executing if the step fails. For example, according to the system overhead condition, data meeting the service condition is extracted and written into the temporary file to be transplanted, a data checking step is executed, the data check statement configured in the step 3 is executed, and the data consistency of the extracted file and the temporary data file is checked.
Step 5: the migration data is sent and may be repeatedly performed if this step fails. The temporary file can be transferred and transplanted through TCP protocol, and the work number of data operation is sent according to the data quantity of the temporary file. If the file transfer fails, the data file transfer step may be repeatedly performed.
Step 6: checking data, checking the transplanted data after the new system receives the data, checking consistency of the compared data, searching failure reasons, and repeating the steps. For example, the transmission data number in the migration detail file is read, and the transmission data number is used for comparing with the temporary migration file. If the data check fails, searching for a failure reason, and repeatedly executing the data checking step or the data transmission step.
Step 7: and (5) warehousing the transplanted data, and warehousing the transplanted data into a database file of the new system. At this time, reading the SQL statement configured in the step 3, and checking the uniqueness of the data to be put in storage and the existing data. And if the conflict data exist, analyzing the reasons of the conflict data. For example, the duplicate data is removed, and the data after the duplicate removal is put into a corresponding database file in a new system.
Step 8: and (5) verifying the validity of the service, and randomly receiving the service to perform the validity verification. Service personnel distribute service requests to record service results in the new and old systems respectively, and compare whether the data results of the new and old systems are consistent.
The data migration device performs data distribution on service data to be transplanted according to the storage condition of the system overhead condition, for example, temporary files to be transplanted are built in batches when the memory occupation is low; and then extracting the temporary files to be transplanted according to the data priority of the overhead condition and preset service conditions (such as service data requirements), and determining the transplanting detail files (such as transplanting flow, sequencing and the like) of the temporary files to be transplanted. Therefore, in the process of automatically performing data migration, operations such as indexing, partitioning processing, data verification and the like are performed, and the high-efficiency data input is ensured while the safety and the integrity of the data are ensured. Further, the data migration device can firstly perform parameter configuration on service data to be migrated according to service migration requirements to obtain a black-and-white list corresponding to the data migration range; and then checking the system state of the receiving system in the black-and-white list, and when the system state of the receiving system is in a preset health state, performing data migration preparation. Then making rules for the service data to be transplanted according to service migration requirements to obtain a transplanted data verification rule; correspondingly, after obtaining the migration detail file corresponding to the service data to be migrated, the temporary file to be migrated can be further verified according to a migration data verification rule, and when the verification result accords with the data consistency, the temporary file to be migrated can be migrated to a target system. Therefore, the safety and reliability of the data transmission and warehousing process are ensured through multi-level security measures and a fault recovery mechanism. Further, when the receiving system accords with the preset health state, the data scale of the business data to be transplanted is extracted; traversing a preset migration strategy library according to service migration requirements and data scale, and determining a strategy to be migrated corresponding to the service data to be migrated. And the migration data verification rule can directly carry out SQL statement configuration according to the service migration requirement, and the standardization of management and operation of the relational database is carried out. Correspondingly, the data distribution can be carried out on the service data to be transplanted according to the storage condition and the strategy to be transplanted to obtain the corresponding temporary file to be transplanted, so that the proper transplanting strategy can be directly inquired in the preset transplanting strategy library, and the efficiency of data migration is improved. Meanwhile, before data migration, the data migration equipment can judge whether the current resource utilization rate of the current system and the current resource utilization rate of the receiving system are lower than a preset threshold value according to the system state; and when the data accords with the data consistency and the current resource utilization rate is lower than the preset threshold value, the temporary file to be transplanted can be migrated to the target system according to the transplanting detail file. Therefore, when the resource utilization rate is low, the data safety and the integrity are ensured, the shutdown is avoided for transplanting the data, and the smoothness and the migration efficiency of data migration are improved.
In addition, the embodiment of the invention also provides a storage medium, wherein the storage medium is stored with a service data migration program, and the service data migration program realizes the steps of the service data migration method when being executed by a processor.
Referring to fig. 5, fig. 5 is a block diagram illustrating a first embodiment of a service data migration apparatus according to the present invention.
As shown in fig. 5, the service data migration apparatus provided in the embodiment of the present invention includes:
the data generation module 501 is configured to generate service data to be transplanted according to a service migration requirement when receiving the service migration requirement of a user;
the data distribution module 502 is configured to perform migration distribution on the service data to be migrated according to a system overhead condition, so as to obtain a migration detail file and a temporary file to be migrated, which correspond to the service data to be migrated;
and the data migration module 503 is configured to migrate the temporary file to be migrated to a target system according to the migration detail file, so that the target system writes the temporary file to be migrated into a database file when the verification result of the temporary file to be migrated meets the data uniqueness.
When receiving a service migration requirement of a user, the data migration equipment extracts and generates service data to be transplanted from a local database according to the service migration requirement; and then, carrying out data distribution on the service data to be transplanted according to the system overhead condition to obtain temporary files to be transplanted and distributed transplanting detail files. And then the temporary file to be transplanted can be migrated to the target system according to the migration detail file through the TCP protocol, and if the file transmission fails, the data file transmission step can be repeatedly executed. When the target system receives the temporary file to be transplanted, the target system can perform data verification on the temporary file to be transplanted, such as the uniqueness of the data, whether the data has conflict, whether logic has errors and the like; and the target system writes the temporary file to be transplanted into the database. According to the embodiment, the data and the partition processing are automatically uploaded by distributing the migration detail file and the temporary file to be migrated, the database file is written after the uniqueness of the data is verified when the target system receives the data, the data security and the integrity are ensured, shutdown for data migration is avoided, and the data migration efficiency is improved.
Based on the above-mentioned first embodiment of the service data migration apparatus of the present invention, a second embodiment of the service data migration apparatus of the present invention is proposed.
In this embodiment, the overhead condition includes a data priority and a storage condition; the data distribution module 502 is further configured to perform migration distribution on the service data to be migrated according to the storage condition of the overhead condition, so as to obtain a temporary file to be migrated corresponding to the service data to be migrated; and extracting the temporary file to be transplanted according to the data priority of the system overhead condition and the preset service condition to obtain a transplanting detail file corresponding to the service data to be transplanted.
Further, the service data migration device further includes a rule making module 504, where the rule making module 504 is configured to perform parameter configuration on the service data to be migrated according to the service migration requirement, and obtain a black-and-white list corresponding to the service data to be migrated within the migration range; checking the system state of the receiving system in the black-and-white list, and judging whether the checking result accords with a preset health state or not; when the checking result accords with the preset health state, making rules for the business data to be transplanted according to the business migration requirement, and obtaining a transplanting data checking rule;
Correspondingly, the service data migration device further comprises a rule checking module 505, wherein the rule checking module 505 is used for checking the temporary file to be transplanted according to the data transplanting checking rule, and judging whether a checking result accords with data consistency; and when the verification result accords with the data consistency, executing the operation of the data migration module 503.
Further, the rule making module 504 is further configured to extract a data size of the service data to be transplanted when the inspection result meets the preset health state; traversing a preset migration strategy library according to the service migration requirements and the data scale to obtain a migration strategy corresponding to the service data to be migrated; SQL statement configuration is carried out on the business data to be transplanted according to the business migration requirement, and a transplanted data verification rule is obtained;
correspondingly, the data distribution module 502 is further configured to perform migration distribution on the service data to be migrated according to the storage condition and the policy to be migrated, so as to obtain a temporary file to be migrated corresponding to the service data to be migrated.
Further, the service data migration apparatus further includes a threshold value judging module 506, where the threshold value judging module 506 judges, according to the system state, a current resource usage rate of a current system and the receiving system, and judges whether the current resource usage rate is lower than a preset threshold value;
Correspondingly, the rule checking module 505 is further configured to execute the operation of the data migration module 503 when the check result meets the data consistency and the current resource usage rate is lower than the preset threshold.
Further, after the data migration module 503, further includes: the target system reads the number of the files to be transferred of the transplanting detail, compares the number of the files to be transferred with the number of the files to be transferred of the temporary files to be transplanted, and judges whether the comparison results are consistent; when the comparison results are consistent, the target system converts the temporary file to be transplanted into data to be put in storage; the target system performs data verification on the data to be put in storage and judges whether a verification result accords with data uniqueness; and when the verification result accords with the data uniqueness, the target system writes the data to be put in storage into a database file.
Further, the service data migration device further includes a data update module 507, where the data update module 507 is configured to, when receiving data to be put in storage sent by the target system, perform protocol comparison on the data to be put in storage through a DAP temporary table, and determine whether the data to be put in storage is consistent with the temporary file to be transplanted according to a comparison result; when the data to be put in storage is inconsistent with the temporary file to be transplanted, updating the temporary file to be transplanted through a data processing tool to obtain an updated temporary file to be transplanted; and migrating the updated temporary file to be transplanted to the target system until the data to be put in storage is consistent with the updated temporary file to be transplanted, and stopping migrating the temporary file to be transplanted to the target system according to the transplanting detail file.
Other embodiments or specific implementation manners of the service data migration apparatus of the present invention may refer to the above method embodiments, and are not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. read-only memory/random-access memory, magnetic disk, optical disk), comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

Claims (10)

1. The service data migration method is characterized by comprising the following steps:
when receiving a service migration requirement of a user, generating service data to be transplanted according to the service migration requirement;
transplanting and distributing the service data to be transplanted according to the system overhead condition to obtain a transplanting detail file and a temporary file to be transplanted, which correspond to the service data to be transplanted;
and migrating the temporary file to be transplanted to a target system according to the transplanting detail file, so that the target system writes the temporary file to be transplanted into a database file when the verification result of the temporary file to be transplanted accords with the data uniqueness.
2. The traffic data migration method of claim 1, wherein the overhead conditions include data priority and storage conditions;
The transplanting allocation is carried out on the service data to be transplanted according to the system overhead condition, and a transplanting detail file and a temporary file to be transplanted corresponding to the service data to be transplanted are obtained, which comprises the following steps:
transplanting and distributing the service data to be transplanted according to the storage condition of the system overhead condition to obtain a temporary file to be transplanted corresponding to the service data to be transplanted;
and extracting the temporary file to be transplanted according to the data priority of the system overhead condition and the preset service condition to obtain a transplanting detail file corresponding to the service data to be transplanted.
3. The service data migration method as claimed in claim 2, wherein after generating the service data to be migrated according to the service migration requirement when the service migration requirement of the user is received, the method further comprises:
carrying out parameter configuration on the service data to be transplanted according to the service migration requirement to obtain a black-and-white list corresponding to the service data to be transplanted in the transplanting range;
checking the system state of the receiving system in the black-and-white list, and judging whether the checking result accords with a preset health state or not;
when the checking result accords with the preset health state, making rules for the business data to be transplanted according to the business migration requirement, and obtaining a transplanting data checking rule;
Correspondingly, the extracting the temporary file to be transplanted according to the data priority and the preset service condition of the overhead condition, after obtaining the transplanting detail file corresponding to the service data to be transplanted, further includes:
checking the temporary file to be transplanted according to the data transplanting checking rule, and judging whether a checking result accords with data consistency;
and when the verification result accords with the data consistency, executing the step of migrating the temporary file to be transplanted to a target system according to the transplanting detail file.
4. The service data migration method of claim 3, wherein when the inspection result meets the preset health state, making a rule for the service data to be migrated according to the service migration requirement, and obtaining a migration data verification rule includes:
when the checking result accords with the preset health state, extracting the data scale of the business data to be transplanted;
traversing a preset migration strategy library according to the service migration requirements and the data scale to obtain a migration strategy corresponding to the service data to be migrated;
SQL statement configuration is carried out on the business data to be transplanted according to the business migration requirement, and a transplanted data verification rule is obtained;
correspondingly, the transplanting and distributing the to-be-transplanted service data according to the storage condition of the system overhead condition to obtain a temporary to-be-transplanted file corresponding to the to-be-transplanted service data, which comprises the following steps:
and carrying out transplanting distribution on the service data to be transplanted according to the storage condition and the strategy to be transplanted to obtain a temporary file to be transplanted corresponding to the service data to be transplanted.
5. The business data migration method of claim 4, wherein before the migration of the temporary file to be migrated to the target system according to the migration specification file, the method further comprises:
judging the current resource utilization rate of a current system and the current resource utilization rate of the receiving system according to the system state, and judging whether the current resource utilization rate is lower than a preset threshold value or not;
correspondingly, when the verification result accords with the data consistency, the step of executing the migration of the temporary file to be transplanted to a target system according to the transplanting detail file comprises the following steps:
and when the verification result accords with the data consistency and the current resource utilization rate is lower than the preset threshold value, executing the step of migrating the temporary file to be transplanted to a target system according to the transplanting detail file.
6. The service data migration method of claim 5, wherein the target system writes the temporary file to be migrated into a database file when the verification result of the temporary file to be migrated conforms to data uniqueness, comprising:
the target system reads the number of the files to be transferred of the transplanting detail, compares the number of the files to be transferred with the number of the files to be transferred of the temporary files to be transplanted, and judges whether the comparison results are consistent;
when the comparison results are consistent, the target system converts the temporary file to be transplanted into data to be put in storage;
the target system performs data verification on the data to be put in storage and judges whether a verification result accords with data uniqueness;
and when the verification result accords with the data uniqueness, the target system writes the data to be put in storage into a database file.
7. The business data migration method of claim 6, wherein after the migration of the temporary file to be migrated to the target system according to the migration specification file, further comprising:
when data to be put in storage sent by the target system is received, carrying out protocol comparison on the data to be put in storage through a DAP temporary table, and judging whether the data to be put in storage is consistent with the temporary file to be transplanted or not according to a comparison result;
When the data to be put in storage is inconsistent with the temporary file to be transplanted, updating the temporary file to be transplanted through a data processing tool to obtain an updated temporary file to be transplanted;
and migrating the updated temporary file to be transplanted to the target system until the data to be put in storage is consistent with the updated temporary file to be transplanted, and stopping migrating the temporary file to be transplanted to the target system according to the transplanting detail file.
8. A traffic data migration apparatus, the apparatus comprising:
the data generation module is used for generating service data to be transplanted according to the service migration requirement when the service migration requirement of the user is received;
the data distribution module is used for carrying out transplanting distribution on the service data to be transplanted according to the system overhead condition to obtain a transplanting detail file and a temporary file to be transplanted, which correspond to the service data to be transplanted;
and the data migration module is used for migrating the temporary file to be migrated to a target system according to the migration detail file, so that the target system writes the temporary file to be migrated into a database file when the verification result of the temporary file to be migrated accords with the data uniqueness.
9. A business data migration apparatus, the apparatus comprising: a memory, a processor and a service data migration program stored on the memory and executable on the processor, the service data migration program being configured to implement the steps of the service data migration method of any one of claims 1 to 7.
10. A storage medium having stored thereon a service data migration program which, when executed by a processor, implements the steps of the service data migration method according to any one of claims 1 to 7.
CN202310942703.5A 2023-07-28 2023-07-28 Service data migration method, device, equipment and storage medium Pending CN116955319A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310942703.5A CN116955319A (en) 2023-07-28 2023-07-28 Service data migration method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310942703.5A CN116955319A (en) 2023-07-28 2023-07-28 Service data migration method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116955319A true CN116955319A (en) 2023-10-27

Family

ID=88460152

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310942703.5A Pending CN116955319A (en) 2023-07-28 2023-07-28 Service data migration method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116955319A (en)

Similar Documents

Publication Publication Date Title
US20070198988A1 (en) Multiple application integration
CN113536081B (en) Data center data management method and system based on artificial intelligence
CN112087334A (en) Alarm root cause analysis method, electronic device and storage medium
CN111026602A (en) Health inspection scheduling management method and device of cloud platform and electronic equipment
CN112182089B (en) Report generation method, device and equipment based on data warehouse model
CN112631686A (en) Data processing method, data processing device, computer equipment and storage medium
EP4122163A1 (en) Causality determination of upgrade regressions via comparisons of telemetry data
JP2018060570A (en) Reference data segmentation from single to multiple tables
CN111737227A (en) Data modification method and system
CN112860706A (en) Service processing method, device, equipment and storage medium
CN112506771A (en) Message comparison method and device
CN112559525B (en) Data checking system, method, device and server
CN115604343A (en) Data transmission method, system, electronic equipment and storage medium
CN116955319A (en) Service data migration method, device, equipment and storage medium
CN115344633A (en) Data processing method, device, equipment and storage medium
CN112749197B (en) Data fragment refreshing method, device, equipment and storage medium
CN114064678A (en) Event data processing method and device and terminal equipment
CN111930800B (en) Enterprise big data analysis and processing system based on cloud computing
CN114900531B (en) Data synchronization method, device and system
CN115794609A (en) Script sharing method and device, electronic equipment and storage medium
CN117992422A (en) Data transmission method, device and computer readable storage medium
CN114185896A (en) Data processing method and device, electronic equipment and storage medium
CN116820927A (en) Method and device for processing exception of batch operation
CN116383284A (en) Data access method, device, equipment and storage medium
CN117170924A (en) Method, system, electronic equipment and medium for processing clearing task abnormality

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination