CN115639972B - Data migration method and device, electronic equipment and storage medium - Google Patents
Data migration method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN115639972B CN115639972B CN202211660378.5A CN202211660378A CN115639972B CN 115639972 B CN115639972 B CN 115639972B CN 202211660378 A CN202211660378 A CN 202211660378A CN 115639972 B CN115639972 B CN 115639972B
- Authority
- CN
- China
- Prior art keywords
- data
- storage system
- target data
- verification result
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data migration method, a data migration device, electronic equipment and a storage medium, and relates to the technical field of data processing, wherein the method comprises the following steps: acquiring target data, wherein the target data is obtained by performing format conversion on source data exported from the first storage system based on a second data format corresponding to the second storage system; constructing a third storage system based on the second storage system, and importing target data into the third storage system, wherein the data structures of the second storage system and the third storage system are the same; performing data verification on the target data to obtain a data verification result, wherein the data verification result comprises an integrity verification result and a security verification result; and under the condition that the data verification result is determined to be verification pass, the second storage system is replaced by a third storage system, so that the technical problem that how to avoid the influence on the normal operation of the IT application due to errors in the process of carrying out format conversion on the source data in the prior art is solved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data migration method and apparatus, an electronic device, and a storage medium.
Background
Data of the IT application is stored in a specific storage system, and when version updating, equipment updating or data backup is carried out on the IT application, the data in the original storage system needs to be migrated to a new storage system.
Because there are often many differences between the original storage system and the new storage system, it is necessary to convert the source data exported from the original storage system into data conforming to the format of the new storage system, and import the data after format conversion into the new storage system. Once an error occurs in the process of converting the source data into data conforming to the new storage system format, unpredictable risks are brought to the IT application.
Therefore, a technical problem to be solved by those skilled in the relevant art needs to be solved, how to avoid errors occurring in the process of converting the format of the source data, and how to influence the normal operation of the IT application.
Disclosure of Invention
The invention provides a data migration method, a data migration device, electronic equipment and a storage medium, which are used for solving the technical problem that how to avoid the influence on the normal operation of IT application caused by errors in the process of carrying out format conversion on source data in the prior art.
The invention provides a data migration method, which comprises the following steps:
acquiring target data, wherein the target data is obtained by performing format conversion on source data exported from the first storage system based on a second data format corresponding to the second storage system;
constructing a third storage system based on the second storage system, and importing the target data into the third storage system, wherein the data structures of the second storage system and the third storage system are the same;
performing data verification on the target data to obtain a data verification result, wherein the data verification result comprises an integrity verification result and a security verification result;
and replacing the second storage system with the third storage system under the condition that the data verification result is determined to be passed through verification.
According to the data migration method provided by the present invention, the performing data verification on the target data to obtain a data verification result includes:
performing integrity verification on the target data based on a data verification list corresponding to the first storage system to obtain an integrity verification result of the target data;
performing security verification on the target data based on the data constraint information and the data structure information of the third storage system to obtain a security verification result of the target data;
and determining a data verification result of the target data based on the integrity verification result and the safety verification result.
According to a data migration method provided by the present invention, the performing integrity check on the target data based on the data checking list corresponding to the first storage system includes:
detecting whether a first-level file name of the data item to be checked exists in the target data or not aiming at each data item to be checked in the data checking list;
under the condition that the primary file name of the data item to be checked exists in the target data, detecting whether the secondary file name of each object to be checked under the data item to be checked exists in the target data;
and under the condition that the secondary file name of the object to be checked exists in the target data, detecting whether the tertiary file name of each file to be checked corresponding to the object to be checked exists in the target data.
According to a data migration method provided by the present invention, the performing security check on the target data based on the data constraint information and the data structure information of the third storage system to obtain a security check result of the target data includes:
verifying the target data based on the data constraint information of the third storage system to obtain a constraint verification result; at least one data constraint item of the third storage system is contained in the data constraint information;
verifying the target data based on the data structure information of the third storage system to obtain a structure verification result; the data structure information comprises a data corresponding relation between a first storage area and a second storage area in the third storage system;
and determining a safety verification result of the target data based on the constraint verification result and the structure verification result.
According to a data migration method provided by the present invention, replacing the second storage system with the third storage system includes:
under the condition that the second storage system and the third storage system are databases, disconnecting the communication connection between the second storage system and the first storage system, and establishing the communication connection between the third storage system and the first storage system;
and modifying the data export path of the first storage system into a third storage path of the third storage system under the condition that the second storage system and the third storage system are storage files.
According to a data migration method provided by the present invention, the obtaining of the target data includes:
acquiring source data exported from a first storage system, a first data format corresponding to the first storage system and a second data format corresponding to a second storage system;
acquiring format difference data corresponding to the first data format and the second data format;
and carrying out format conversion on the source data based on the format difference data to obtain target data after format conversion.
According to a data migration method provided by the present invention, the importing the target data into the third storage system includes:
under the condition that the format conversion of the source data is successfully completed, detecting whether abnormal data exists in target data after the format conversion;
and under the condition that abnormal data do not exist in the target data, importing the target data into the third storage system.
According to a data migration method provided by the present invention, the method further comprises:
and under the condition that the format conversion of the source data fails or abnormal data exists in the target data, acquiring a format conversion log, and reporting the abnormality based on the format conversion log.
According to a data migration method provided by the invention, the method further comprises the following steps:
detecting whether the target data is successfully imported into the third storage system;
and under the condition that the target data is not successfully imported into the third storage system, acquiring a data import log, and reporting an exception based on the data import log.
The present invention also provides a data migration apparatus, comprising:
the data acquisition module is used for acquiring target data, and the target data is obtained by carrying out format conversion on source data exported from the first storage system based on a second data format corresponding to the second storage system;
the data import module is used for constructing a third storage system based on the second storage system and importing the target data into the third storage system, and the data structures of the second storage system and the third storage system are the same;
the data verification module is used for performing data verification on the target data to obtain a data verification result, and the data verification result comprises an integrity verification result and a security verification result;
and the system replacing module is used for replacing the second storage system with the third storage system under the condition that the data verification result is determined to be passed through verification.
The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data migration method as described in any of the above when executing the program.
The present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data migration method as described in any of the above.
According to the data migration method, the data migration device, the electronic equipment and the storage medium, the target data after format conversion is imported into the third storage system with the same data structure as the second storage system, and integrity verification and safety verification are carried out on the imported target data based on the third storage system so as to detect whether format conversion errors exist in the imported target data; and under the condition that the data verification result is confirmed to be passed, namely no error occurs in the process of carrying out format conversion on the source data, replacing the second storage system with a third storage system to carry out data storage based on the replaced storage system so as to avoid the influence of the error occurring in the process of carrying out format conversion on the source data on the normal operation of the IT application, and the technical problem that how to avoid the influence of the error occurring in the process of carrying out format conversion on the source data on the normal operation of the IT application in the prior art is solved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a flow chart of a data migration method according to an embodiment of the present invention;
FIG. 2 is a second flowchart illustrating a data migration method according to an embodiment of the present invention;
FIG. 3 is a third flowchart illustrating a data migration method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of one embodiment of a data collation list;
FIG. 5 is a second schematic diagram of a data check list according to an embodiment of the present invention;
FIG. 6 is a fourth flowchart illustrating a data migration method according to an embodiment of the present invention;
FIG. 7 is a fifth flowchart illustrating a data migration method according to an embodiment of the present invention;
FIG. 8 is a sixth flowchart illustrating a data migration method according to an embodiment of the present invention;
FIG. 9 is a seventh flowchart illustrating a data migration method according to an embodiment of the present invention;
FIG. 10 is a schematic structural diagram of a data migration apparatus according to an embodiment of the present invention;
fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The data migration method of the present invention is described below in conjunction with FIGS. 1-9. As shown in fig. 1, the present invention provides a data migration method, including:
The data format of the source data is a first data format corresponding to the first storage system. The data format of the target data is a second data format corresponding to the second storage system. The first storage system is an original storage system, and the second storage system is a new storage system.
And 102, constructing a third storage system based on the second storage system, and importing the target data into the third storage system, wherein the data structures of the second storage system and the third storage system are the same.
The third storage system is a backup storage system of the second storage system, and the third storage system and the second storage system have the same data structure.
In one embodiment, a create data packet of a second storage system is obtained; based on the created data package, a third storage system is constructed.
And 103, performing data verification on the target data to obtain a data verification result, wherein the data verification result comprises an integrity verification result and a security verification result.
Further, data verification is performed on the target data based on the data verification list corresponding to the first storage system and the data constraint information and the data structure information of the third storage system, so that a data verification result is obtained. Wherein the data constraints and data structures of the third storage system need to conform to the data structures of the second storage system.
And 104, replacing the second storage system with a third storage system under the condition that the data verification result is determined to be verification passing.
Further, under the condition that the data verification result is determined to be that the verification fails, the second storage system is continuously used for data storage.
In the foregoing steps 101 to 104, the target data after format conversion is imported into a third storage system having the same data structure as the second storage system, and integrity check and security check are performed on the imported target data based on the third storage system to detect whether the imported target data has a format conversion error, and since the whole data check process is performed in the third storage system, normal operation of the first storage system and the second storage system, that is, normal operation of the IT application is not affected; and under the condition that the data verification result is confirmed to be passed, namely no error occurs in the process of carrying out format conversion on the source data, replacing the second storage system with a third storage system to carry out data storage based on the replaced storage system so as to avoid the influence of the error occurring in the process of carrying out format conversion on the source data on the normal operation of the IT application, and the technical problem that how to avoid the influence of the error occurring in the process of carrying out format conversion on the source data on the normal operation of the IT application in the prior art is solved.
In one embodiment, as shown in fig. 2, the step 103 includes steps 201 to 203, wherein:
The data checking list represents a data checking list of the source data derived from the first storage system, and includes at least one to-be-checked data item, which is used for checking whether a file corresponding to each to-be-checked data item in the source data exists in the target data after format conversion. Each data item to be checked represents a file corresponding to one type of data, for example, in the case where the data item to be checked is a user data item, the user data item represents a file corresponding to user data.
And 202, performing security verification on the target data based on the data constraint information and the data structure information of the third storage system to obtain a security verification result of the target data.
The data constraint information comprises at least one data constraint item of the third storage system, and the data constraint item is a fixed constraint item of the third storage system. For example, the data constraint item is a non-overlapping constraint or a data bit constraint, and the non-overlapping constraint refers to a constraint that the user name or the device ID do not overlap. The data bit constraint includes a constraint that the data bits of the telephone number are preset values.
And step 203, determining a data verification result of the target data based on the integrity verification result and the security verification result.
In one embodiment, when both the integrity check result and the security check result are determined to pass the check, determining that the data check result is a check pass; and under the condition that any item of the integrity check result and the safety check result is determined to be not passed, determining the data check result to be not passed.
In the above steps 201 to 203, integrity check is performed on the target data based on the data check list corresponding to the first storage system, so as to avoid a situation that some data is lost in the target data after format conversion, thereby ensuring data integrity of the target data after format conversion; and based on the data constraint information and the data structure information of the third storage system, performing security check on the target data to ensure that the target data after format conversion conforms to the data constraint and the data structure of the third storage system.
In one embodiment, as shown in fig. 3, the step 201 includes steps 301 to 303, wherein:
As shown in fig. 4, the data collation list indicates the primary file name of at least one kind of data item to be collated, the secondary file name of each object to be collated under each kind of data item to be collated, and the tertiary file name of each file to be collated corresponding to each object to be collated. The data items to be collated include user data items, project data items, operation data items, and the like.
In one embodiment, the data item to be checked is taken as a user data item for explanation, the user data item includes the second-level file name of each user-corresponding file, and each user-corresponding file includes the third-level file name of each file to be checked. The files to be checked comprise configuration files, authority files and log files. The configuration file comprises configuration information of a user, the authority file comprises authority information of the user, and the log file comprises log information of the user.
As shown in fig. 5, the user data item includes the secondary file names of the files corresponding to user 1, user 2, and user 3. The user 1 corresponds to the third-level file names of the file to be collated 1 and the file to be collated 2 contained under the file. The user 2 corresponds to the third-level file names of the file to be collated 3 and the file to be collated 4 contained under the file. The user 3 corresponds to the third-level file names of the file to be collated 5 and the file to be collated 6 contained under the file.
It should be noted that whether the first-level file name of the data item to be checked exists in the target data is detected to determine whether the file corresponding to the data item to be checked exists in the target data, that is, specific data in the file corresponding to the data item to be checked does not need to be detected, and only the first-level file name of the data item to be checked exists needs to be checked, so that the data checking flow is simplified while the data checking function is realized, the data checking efficiency is improved, and the data checking efficiency of the target data is improved.
Further, in the case that the first-level file name of the data item to be checked does not exist in the target data, since the first-level directory does not exist, the second-level directory and the third-level directory do not exist, and therefore, the detection of each object to be checked under the data item to be checked is no longer performed, and the step 301 is skipped to, so as to perform the detection of the next data item to be checked.
It should be noted that, whether a file corresponding to each object to be checked under the data item to be checked exists in the target data is determined by detecting whether a secondary file name of each object to be checked under the data item to be checked exists in the target data, that is, specific data in the file corresponding to each object to be checked does not need to be detected, and only whether a secondary file name of each object to be checked exists needs to be checked, so that the data checking function is realized, and meanwhile, the data checking flow is simplified, so that the data checking efficiency is improved, and further, the data checking efficiency of the target data is improved.
Step 303, under the condition that the secondary file name of the object to be checked exists in the target data, detecting whether the tertiary file name of each file to be checked corresponding to the object to be checked exists in the target data.
Further, under the condition that the second-level file name of the object to be checked does not exist in the target data, since the second-level directory does not exist, the third-level directory does not exist, and therefore, the file to be checked corresponding to the object to be checked is not detected any more, but the process jumps to the step 302 to detect the next object to be checked.
It should be noted that whether the third-level file name of each to-be-checked file corresponding to the to-be-checked object exists in the target data is detected, so as to determine whether each to-be-checked file corresponding to the to-be-checked object exists in the target data, that is, specific data in each to-be-checked file corresponding to the to-be-checked object does not need to be detected, and only whether the third-level file name of each to-be-checked file exists needs to be checked, so that the data checking function is realized, and meanwhile, the data checking flow is simplified, so that the data checking efficiency is improved, and further, the data checking efficiency of the target data is improved.
In one embodiment, as shown in fig. 6, the step 202 includes steps 401 to 403, wherein:
The data constraint item is a fixed constraint item of the third storage system, and comprises a non-overlapping constraint and a data bit constraint, wherein the non-overlapping constraint refers to a constraint that user names or device IDs do not overlap, and is used for detecting whether repeated user names or device IDs exist in target data. The data bit constraint includes a constraint that a data bit of the telephone number is a preset value, and is used for detecting whether the data bit of specific data in the target data conforms to a data bit constraint rule.
For example, whether the number of digits of the telephone number is a preset value. For example, if the number of bits of a character string representing time is equal to the bit number constraint value of the time character string, and if the number of bits of the character string is not equal to the bit number constraint value of the time character string, it is determined that the character string cannot be used to represent time, and thus an error exists.
Wherein the data structure information is updated with the addition of the user data, and the data structure information includes data corresponding relations between the storage regions, for example, the data structure information includes data corresponding relations between the first storage region and the second storage region,
in one embodiment, the first storage area represents a user data storage area for storing all user data of all users. The storage mode of the first storage area is a 2-system storage mode, for example, a binary number is used to represent the authority information of the user, for example 10101, each binary bit in the binary number represents a user authority, where "0" represents that the user does not have the user authority corresponding to the binary bit, and "1" represents that the user has the user authority corresponding to the binary bit.
In one embodiment, the second storage area represents a visualization storage area for storing a portion of the user data of the individual user. The storage mode of the first storage area is a visual storage mode, for example, "user 1 has user right 1, user right 2 and user right 3".
And step 403, determining a security check result of the target data based on the constraint check result and the structure check result.
Specifically, when it is determined that both the constraint verification result and the structure verification result pass the verification, it is determined that the security verification result passes the verification. And under the condition that any one of the constraint verification result and the structure verification result is determined to be failed in verification, determining that the security verification result is failed in verification.
In one embodiment, the data migration method provided by the present invention further includes: and under the condition that the data verification result is detected to be that the verification is failed, acquiring a data verification log, and reporting the abnormal condition based on the data verification log, wherein the data verification log comprises an integrity verification log and a security verification log, and the first storage system and the second storage system of the original edition are continuously used, namely the original edition IT application is continuously used without influencing the normal operation of the IT application.
Further, under the condition that the cause of the abnormality is eliminated, the data migration methods provided by the above embodiments are repeatedly executed to migrate the source data of the first storage system to the new second storage system, which is the replaced third storage system. In the embodiment, when the data verification result is detected to be that the verification fails, the data verification log is acquired, abnormal reporting is performed based on the data verification log to prompt technical staff to investigate the reason of the abnormality, and then under the condition that the reason of the abnormality is eliminated, the data migration methods provided by the embodiment are repeatedly executed to realize the migration of the source data of the first storage system to the new second storage system.
In one embodiment, as shown in fig. 7, the step 104 includes steps 501 to 502, where:
For example, the second storage system and the third storage system serve two mysql services with the same database structure, and the mysql services are mysql databases.
For example, the second storage system and the third storage system are two files or two heaps of files having the same data structure.
Specifically, the data export path of the first storage system is modified from the second storage path of the second storage system to the third storage path of the third storage system.
In one embodiment, as shown in fig. 8, the step 101 includes steps 601 to 603, where:
In the above steps 601 to 603, by obtaining the format difference data corresponding to the first data format and the second data format, and performing format conversion on the source data based on the format difference data, that is, only the place with the difference needs to be subjected to format conversion, the data processing amount in the format conversion process is effectively reduced on the basis of realizing the format conversion function, so that the format conversion efficiency is improved, and the data migration efficiency is further improved.
In one embodiment, as shown in fig. 9, the step 102 includes steps 701 to 702, where:
step 701, detecting whether abnormal data exists in the target data after format conversion under the condition that the format conversion of the source data is successfully completed.
Step 702, when it is detected that the abnormal data does not exist in the target data, importing the target data into a third storage system.
In the above steps 701 to 702, when it is detected that the format conversion of the source data is successfully completed, whether abnormal data exists in the target data after the format conversion is detected, and when it is detected that the abnormal data does not exist in the target data, the target data is imported into the third storage system, so as to ensure the data quality of the target data imported into the third storage system, thereby avoiding that the subsequent data migration process is affected due to the abnormality of the target data imported into the third storage system, and further ensuring the smooth proceeding of the subsequent data migration process.
In one embodiment, the data migration method provided by the present invention further includes: and under the condition that the format conversion of the source data fails or abnormal data exists in the target data, acquiring a format conversion log, reporting the abnormality based on the format conversion log, and continuously using the first storage system and the second storage system of the original edition, namely continuously using the original edition IT application without influencing the normal operation of the IT application.
Further, under the condition that the cause of the abnormality is eliminated, the data migration methods provided by the above embodiments are repeatedly executed to migrate the source data of the first storage system to the new second storage system, which is the replaced third storage system.
In the embodiment, when failure of format conversion of the source data is detected or abnormal data exists in the target data, the format conversion log is acquired, and abnormal reporting is performed on the basis of the format conversion log so as to prompt a technician to investigate the reason of the abnormality, and further, under the condition of eliminating the reason of the abnormality, the data migration methods provided by the embodiment are repeatedly executed so as to realize migration of the source data of the first storage system to the new second storage system.
In one embodiment, the data migration method provided by the present invention further includes: and detecting whether the target data is successfully imported into the third storage system, acquiring a data import log under the condition that the target data is unsuccessfully imported into the third storage system, reporting the abnormality based on the data import log, and continuously using the first storage system and the second storage system of the original edition, namely continuously using the original edition IT application without influencing the normal operation of the IT application.
Further, under the condition that the reason for the abnormality is eliminated, the data migration methods provided by the above embodiments are repeatedly executed to migrate the source data of the first storage system to the new second storage system, where the new second storage system is the replaced third storage system.
In the embodiment, the data import log is acquired when it is detected that the target data is not successfully imported into the third storage system, and abnormal reporting is performed based on the data import log to prompt a technician to investigate the reason of the abnormality, so that each data migration method provided by the embodiment is repeatedly executed when the reason of the abnormality is eliminated, so that the source data of the first storage system is migrated to the new second storage system.
In addition, according to the data migration method provided by the invention, through troubleshooting errors occurring in the process of converting the source data into the data conforming to the new storage system format, unpredictable risks are avoided from being brought to IT application, and the unpredictable risks can comprise abnormal combination, abnormal deletion, abnormal splitting, permission loss, abnormal configuration transformation, white list loss, data loss and the like of the data.
The data migration apparatus provided by the present invention is described below, and the data migration apparatus described below and the data migration method described above may be referred to correspondingly.
As shown in fig. 10, the present invention provides a data migration apparatus, and a data migration apparatus 100 includes:
the data obtaining module 101 is configured to obtain target data, where the target data is obtained by performing format conversion on source data derived in the first storage system based on a second data format corresponding to the second storage system.
And the data importing module 102 is configured to construct a third storage system based on the second storage system, and import the target data into the third storage system, where the data structures of the second storage system and the third storage system are the same.
The data verification module 103 is configured to perform data verification on the target data to obtain a data verification result, where the data verification result includes an integrity verification result and a security verification result.
And the system replacing module 104 is configured to replace the second storage system with a third storage system when the data verification result is determined to be verification passed.
In one embodiment, the data verification module 103 includes a first verification unit, a second verification unit, and a result determination unit, wherein:
and the first verification unit is used for carrying out integrity verification on the target data based on the data verification list corresponding to the first storage system to obtain an integrity verification result of the target data.
And the second checking unit is used for carrying out security checking on the target data based on the data constraint information and the data structure information of the third storage system to obtain a security checking result of the target data.
And the result determining unit is used for determining the data verification result of the target data based on the integrity verification result and the safety verification result.
In one embodiment, the first checking unit is further configured to detect, for each to-be-checked data item in the data checking list, whether a primary file name of the to-be-checked data item exists in the target data; under the condition that the primary file name of the data item to be checked exists in the target data, detecting whether the secondary file name of each object to be checked under the data item to be checked exists in the target data; and under the condition that the secondary file name of the object to be checked exists in the target data, detecting whether the tertiary file name of each file to be checked corresponding to the object to be checked exists in the target data.
In one embodiment, the second verifying unit is further configured to verify the target data based on the data constraint information of the third storage system to obtain a constraint verification result; the data constraint information comprises at least one data constraint item of the third storage system; verifying the target data based on the data structure information of the third storage system to obtain a structure verification result; the data structure information comprises a data corresponding relation between a first storage area and a second storage area in the third storage system; and determining a safety verification result of the target data based on the constraint verification result and the structure verification result.
In an embodiment, the result determining unit is further configured to determine that the data verification result is a verification pass when it is determined that both the integrity verification result and the security verification result are verification pass; and under the condition that any item of the integrity check result and the safety check result is determined to be not passed, determining the data check result to be not passed.
In an embodiment, the data migration apparatus 100 further includes an anomaly monitoring module, configured to, when it is detected that the data verification result is that the verification fails, obtain a data verification log, and report an anomaly based on the data verification log, where the data verification log includes an integrity verification log and a security verification log.
In one embodiment, the system replacement module 104 is further configured to disconnect the communication connection between the second storage system and the first storage system and create a communication connection between the third storage system and the first storage system when the second storage system and the third storage system are databases; and modifying the data export path of the first storage system into a third storage path of a third storage system under the condition that the second storage system and the third storage system are used for storing files.
In an embodiment, the data obtaining module 101 is further configured to obtain source data derived in the first storage system, a first data format corresponding to the first storage system, and a second data format corresponding to the second storage system; acquiring format difference data corresponding to a first data format and a second data format; and carrying out format conversion on the source data based on the format difference data to obtain target data after format conversion.
In an embodiment, the data importing module 102 is further configured to, in a case that it is detected that format conversion of the source data is successfully completed, detect whether abnormal data exists in the target data after format conversion; and when detecting that abnormal data does not exist in the target data, importing the target data into a third storage system.
In an embodiment, the anomaly monitoring module is further configured to, when it is detected that format conversion of the source data fails or abnormal data exists in the target data, obtain a format conversion log, and report an anomaly based on the format conversion log.
In one embodiment, the anomaly monitoring module is further configured to detect whether the target data is successfully imported into the third storage system; and under the condition that the target data is detected to be imported into the third storage system unsuccessfully, acquiring a data import log, and reporting the exception based on the data import log.
Fig. 11 illustrates a physical structure diagram of an electronic device, and as shown in fig. 11, the electronic device may include: a processor (processor) 810, a communication Interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication Interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logic instructions in the memory 830 to perform the data migration methods provided by the methods described above, including: acquiring target data, wherein the target data is obtained by performing format conversion on source data exported from the first storage system based on a second data format corresponding to the second storage system; constructing a third storage system based on the second storage system, and importing target data into the third storage system, wherein the data structures of the second storage system and the third storage system are the same; performing data verification on the target data to obtain a data verification result, wherein the data verification result comprises an integrity verification result and a security verification result; and replacing the second storage system with a third storage system under the condition that the data verification result is determined to be verification passing.
In addition, the logic instructions in the memory 830 may be implemented in software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to execute the data migration method provided by the above methods, the method including: acquiring target data, wherein the target data is obtained by performing format conversion on source data exported from the first storage system based on a second data format corresponding to the second storage system; constructing a third storage system based on the second storage system, and importing target data into the third storage system, wherein the data structures of the second storage system and the third storage system are the same; performing data verification on the target data to obtain a data verification result, wherein the data verification result comprises an integrity verification result and a security verification result; and replacing the second storage system with a third storage system under the condition that the data verification result is determined to be verification passing.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods of the various embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (11)
1. A method of data migration, comprising:
acquiring target data, wherein the target data is obtained by carrying out format conversion on source data exported from the first storage system based on a second data format corresponding to the second storage system;
constructing a third storage system based on the second storage system, and importing the target data into the third storage system, wherein the data structures of the second storage system and the third storage system are the same;
performing data verification on the target data to obtain a data verification result, wherein the data verification result comprises an integrity verification result and a security verification result;
replacing the second storage system with the third storage system when the data verification result is determined to be verification passing;
wherein, the data verification of the target data to obtain a data verification result includes:
performing integrity verification on the target data based on a data verification list corresponding to the first storage system to obtain an integrity verification result of the target data;
performing security verification on the target data based on the data constraint information and the data structure information of the third storage system to obtain a security verification result of the target data;
and determining a data verification result of the target data based on the integrity verification result and the safety verification result.
2. The data migration method according to claim 1, wherein the performing integrity check on the target data based on the data check list corresponding to the first storage system includes:
detecting whether a first-level file name of the data item to be checked exists in the target data or not aiming at each data item to be checked in the data checking list;
under the condition that the primary file name of the data item to be checked exists in the target data, detecting whether the secondary file name of each object to be checked under the data item to be checked exists in the target data;
and under the condition that the secondary file name of the object to be checked exists in the target data, detecting whether the tertiary file name of each file to be checked corresponding to the object to be checked exists in the target data.
3. The data migration method according to claim 1, wherein the performing security check on the target data based on the data constraint information and the data structure information of the third storage system to obtain a security check result of the target data includes:
verifying the target data based on the data constraint information of the third storage system to obtain a constraint verification result; at least one data constraint item of the third storage system is contained in the data constraint information;
verifying the target data based on the data structure information of the third storage system to obtain a structure verification result; the data structure information comprises a data corresponding relation between a first storage area and a second storage area in the third storage system;
and determining a safety verification result of the target data based on the constraint verification result and the structure verification result.
4. The data migration method according to any one of claims 1 to 3, wherein the replacing the second storage system with the third storage system includes:
under the condition that the second storage system and the third storage system are databases, disconnecting the communication connection between the second storage system and the first storage system, and establishing the communication connection between the third storage system and the first storage system;
and modifying the data export path of the first storage system into a third storage path of the third storage system under the condition that the second storage system and the third storage system are storage files.
5. The data migration method according to any one of claims 1 to 3, wherein the acquiring target data includes:
acquiring source data exported from a first storage system, a first data format corresponding to the first storage system and a second data format corresponding to a second storage system;
acquiring format difference data corresponding to the first data format and the second data format;
and carrying out format conversion on the source data based on the format difference data to obtain target data after format conversion.
6. The data migration method according to any one of claims 1 to 3, wherein the importing the target data into the third storage system includes:
under the condition that the format conversion of the source data is successfully completed, detecting whether abnormal data exist in the target data after the format conversion;
and when detecting that abnormal data does not exist in the target data, importing the target data into the third storage system.
7. The data migration method according to claim 6, further comprising:
and under the condition that the format conversion of the source data fails or abnormal data exists in the target data, acquiring a format conversion log, and reporting the abnormality based on the format conversion log.
8. The data migration method of claim 7, wherein the method further comprises:
detecting whether the target data is imported into the third storage system successfully;
and under the condition that the target data is not successfully imported into the third storage system, acquiring a data import log, and reporting an exception based on the data import log.
9. A data migration apparatus, comprising:
the data acquisition module is used for acquiring target data, and the target data is obtained by carrying out format conversion on source data exported from the first storage system based on a second data format corresponding to the second storage system;
the data import module is used for constructing a third storage system based on the second storage system and importing the target data into the third storage system, and the data structures of the second storage system and the third storage system are the same;
the data verification module is used for performing data verification on the target data to obtain a data verification result, and the data verification result comprises an integrity verification result and a security verification result;
the system replacing module is used for replacing the second storage system with the third storage system under the condition that the data verification result is determined to be passed through verification;
the data checking module comprises a first checking unit, a second checking unit and a result determining unit, wherein:
the first checking unit is used for carrying out integrity checking on the target data based on the data checking list corresponding to the first storage system to obtain an integrity checking result of the target data;
the second checking unit is used for carrying out security checking on the target data based on the data constraint information and the data structure information of the third storage system to obtain a security checking result of the target data;
and the result determining unit is used for determining the data verification result of the target data based on the integrity verification result and the safety verification result.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data migration method of any of claims 1 to 8 when executing the program.
11. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the data migration method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211660378.5A CN115639972B (en) | 2022-12-23 | 2022-12-23 | Data migration method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211660378.5A CN115639972B (en) | 2022-12-23 | 2022-12-23 | Data migration method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115639972A CN115639972A (en) | 2023-01-24 |
| CN115639972B true CN115639972B (en) | 2023-04-07 |
Family
ID=84949921
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211660378.5A Active CN115639972B (en) | 2022-12-23 | 2022-12-23 | Data migration method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115639972B (en) |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10198447B2 (en) * | 2015-09-14 | 2019-02-05 | Komprise Inc. | Electronic file migration system and various methods of transparent data migration management |
| CN113672597A (en) * | 2021-09-03 | 2021-11-19 | 中国银行股份有限公司 | Database cross-platform migration method, device, system and equipment |
| CN115033551A (en) * | 2021-12-30 | 2022-09-09 | 天翼云科技有限公司 | Database migration method and device, electronic equipment and storage medium |
| CN114372043A (en) * | 2022-01-14 | 2022-04-19 | 中国农业银行股份有限公司 | Data migration method and device, electronic equipment and storage medium |
| CN115129704A (en) * | 2022-07-22 | 2022-09-30 | 中国平安人寿保险股份有限公司 | Migration data verification method and system, electronic device and storage medium |
-
2022
- 2022-12-23 CN CN202211660378.5A patent/CN115639972B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN115639972A (en) | 2023-01-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110088744B (en) | Database maintenance method and system | |
| CN106327140B (en) | Method and device for monitoring data modification | |
| CN103440460A (en) | Application system change validation method and system | |
| CN109582504A (en) | A kind of data reconstruction method and device for apple equipment | |
| US20220405184A1 (en) | Method, electronic device, and computer program product for data processing | |
| CN107451007B (en) | Verification method and system of block chain for error correction aiming at added blocks | |
| CN114116170A (en) | Timed task execution method and device, computer equipment and storage medium | |
| CN115639972B (en) | Data migration method and device, electronic equipment and storage medium | |
| CN112256532A (en) | Test interface generation method and device, computer equipment and readable storage medium | |
| CN113051180A (en) | Test task monitoring method, device, equipment and storage medium | |
| CN107451011B (en) | Error correction method and system for block chain of parallel added blocks | |
| CN116506340A (en) | Flow link testing method and device, electronic equipment and storage medium | |
| CN111737351A (en) | Transaction management method and device for distributed management system | |
| CN111222181A (en) | AI model supervision method, system, server and storage medium | |
| CN109426576B (en) | Fault-tolerant processing method and fault-tolerant assembly | |
| CN113378239B (en) | Data content right confirming method and system | |
| CN113704123A (en) | Interface test method, device, equipment and storage medium | |
| CN110647771B (en) | Mysql database storage integrity verification protection method and device | |
| CN110807037B (en) | Data modification method and device, electronic equipment and storage medium | |
| WO2017125001A1 (en) | Protection group processing method and device | |
| US20220398143A1 (en) | Network monitoring apparatus, method, and program | |
| US8811195B2 (en) | Revenue assurances proactive testing method and system | |
| CN115776508A (en) | Buried point checking method and system | |
| CN117667643A (en) | Evolution method and device of test case set, computing device cluster and storage medium | |
| CN117632733A (en) | Verification method and device for verifying accounting system and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |