CN113392091A - Distributed cluster data migration method and device - Google Patents

Distributed cluster data migration method and device Download PDF

Info

Publication number
CN113392091A
CN113392091A CN202110739626.4A CN202110739626A CN113392091A CN 113392091 A CN113392091 A CN 113392091A CN 202110739626 A CN202110739626 A CN 202110739626A CN 113392091 A CN113392091 A CN 113392091A
Authority
CN
China
Prior art keywords
data
migrated
deformation
preset
transfer machine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110739626.4A
Other languages
Chinese (zh)
Inventor
金童
张世瑛
赵吉昆
梁晔华
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Industrial and Commercial Bank of China Ltd ICBC
Original Assignee
Industrial and Commercial Bank of China Ltd ICBC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Industrial and Commercial Bank of China Ltd ICBC filed Critical Industrial and Commercial Bank of China Ltd ICBC
Priority to CN202110739626.4A priority Critical patent/CN113392091A/en
Publication of CN113392091A publication Critical patent/CN113392091A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/21—Indexing scheme relating to G06F21/00 and subgroups addressing additional information or applications relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/2107—File encryption

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Bioethics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the application provides a distributed cluster data migration method and a distributed cluster data migration device, which relate to the field of big data processing and can also be used in the field of finance, and the method comprises the following steps: performing data deformation on data to be migrated in a source database according to a preset data deformation rule, and extracting the data to be migrated after the data deformation into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset appearance; extracting data in the target table to the appearance, and generating a transfer machine data file so that a scheduling server can transfer the transfer machine data file to a corresponding target database according to directory parameters in the transfer machine; the data migration method and the data migration device can effectively improve data migration efficiency and save system and human resources.

Description

Distributed cluster data migration method and device
Technical Field
The application relates to the field of distributed technology, can also be used in the field of big data processing or finance, and particularly relates to a distributed cluster data migration method and device.
Background
Under the background of big data distributed cluster data migration, the following difficulties exist in the data migration scene of mass data import/export:
first, data migration involves production and test environment interactions, requiring modification applications, resulting in a long process.
Second, the conventional ORACLE database uses dump method, and uses client utility EXP and IMP. Based on production control requirements, data also needs to be transformed before being exported. The rough flow is "data deformation" → "dump export file" → "transfer" → "dump import file". People need to pay attention to the dump process when the dump file is stored, and the error report is found and solved in time, so that the labor is consumed; the exported files are stored in the scheduling server, and occupy storage resources to influence the operation of other programs; the file transfer rate is low.
Thirdly, when data migration is performed on the traditional distributed database, import \ export tasks need to be issued from the data management node to the data node, and processing results are received, so that the task pressure of the management node is huge.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides a distributed cluster data migration method and device, which can effectively improve the data migration efficiency and save the system and human resources.
In order to solve at least one of the above problems, the present application provides the following technical solutions:
in a first aspect, the present application provides a distributed cluster data migration method, including:
performing data deformation on data to be migrated in a source database according to a preset data deformation rule, and extracting the data to be migrated after the data deformation into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset appearance;
and extracting the data in the target table to the appearance, and generating a transfer machine data file so that the scheduling server can transfer the transfer machine data file to a corresponding target database according to the directory parameters in the transfer machine.
Further, before the data to be migrated in the source database is subjected to data transformation according to a preset data transformation rule, the method further includes:
and according to the position information of the data to be migrated and the specific data separator defined in the preset appearance, performing data to be migrated identification operation on the source database to obtain the data to be migrated.
Further, the data deformation of the data to be migrated in the source database according to the preset data deformation rule further includes:
and determining a to-be-deformed table and a corresponding to-be-deformed field which need to be subjected to data deformation in the to-be-migrated data according to a preset data deformation table.
Further, after the determining, according to a preset data transformation table, a to-be-transformed table and a corresponding to-be-transformed field, which need to undergo data transformation, in the to-be-migrated data, the method includes:
and determining a deformation function corresponding to each field to be deformed according to a preset deformation algorithm table and carrying out encryption deformation operation.
In a second aspect, the present application provides a distributed cluster data migration apparatus, including:
the data transformation module is used for carrying out data transformation on data to be migrated in the source database according to a preset data transformation rule and extracting the data to be migrated after the data transformation into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset table;
and the data migration module is used for extracting the data in the target table to the appearance and generating a transfer machine data file so that the scheduling server migrates the transfer machine data file to a corresponding target database according to the directory parameters in the transfer machine.
Further, still include:
and the data to be migrated identification unit is used for carrying out data to be migrated identification operation on the source database according to the position information of the data to be migrated and the specific data separator defined in the preset appearance to obtain the data to be migrated.
Further, still include:
and the to-be-deformed field determining unit is used for determining the to-be-deformed table and the corresponding to-be-deformed field which need to be subjected to data deformation in the to-be-migrated data according to a preset data deformation table.
Further, still include:
and the deformation function determining unit is used for determining the deformation function corresponding to each field to be deformed according to a preset deformation algorithm table and carrying out encryption deformation operation.
In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the distributed cluster data migration method when executing the program.
In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the distributed cluster data migration method described.
According to the technical scheme, the target table corresponding to the data to be migrated which is subjected to data deformation in the source database is mapped to the corresponding transfer machine through the data transmission port and the IP address defined in the appearance, so that the scheduling server migrates the data file of the transfer machine to the corresponding target database according to the directory parameter in the transfer machine, secondary transmission of the data to be migrated from the table is avoided, the data migration efficiency can be effectively improved, and the system and the human resources are saved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a distributed cluster data migration method in an embodiment of the present application;
fig. 2 is a structural diagram of a distributed cluster data migration apparatus in an embodiment of the present application;
fig. 3 is a schematic diagram illustrating a distributed cluster data migration process in an embodiment of the present application;
FIG. 4 is a schematic diagram of a distributed cluster data migration system according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In view of the problems in the prior art: firstly, data migration relates to interaction between production and test environments, and modification application needs to be provided, so that the process is long; second, the conventional ORACLE database uses dump method, and uses client utility EXP and IMP. Based on production control requirements, data also needs to be transformed before being exported. The rough flow is "data deformation" → "dump export file" → "transfer" → "dump import file". People need to pay attention to the dump process when the dump file is stored, and the error report is found and solved in time, so that the labor is consumed; the exported files are stored in the scheduling server, and occupy storage resources to influence the operation of other programs; the file transmission rate is low; thirdly, when data migration is performed on a traditional distributed database, import/export tasks need to be issued from data management nodes to data nodes, and processing results are received, so that the task pressure of the management nodes is huge.
In order to effectively improve data migration efficiency and save system and human resources, the present application provides an embodiment of a distributed cluster data migration method, and referring to fig. 1, the distributed cluster data migration method specifically includes the following contents:
step S101: data deformation is carried out on data to be migrated in a source database according to a preset data deformation rule, and the data to be migrated after the data deformation is extracted into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset appearance.
Optionally, the data transformation is actually a transformation encryption method, and is to add a transformation function to the sensitive field in the data to be migrated.
Specifically, due to the sensitivity of production data, in order to guarantee data safety and reduce risks brought to a production library by data extraction to the maximum extent, a dbbackup library can be separately classified in the production database for testing and storing extracted target data. In addition, the data transformation table data _ trans _ joband the transformation algorithm table data _ trans _ arith can be designed separately.
Wherein, what fields in which tables in the database need to be encrypted and transformed is recorded in the data transformation table data _ trans _ joba. Recorded in the morphing algorithm table data _ trans _ arith are which morphing functions the relevant sensitive fields need to apply.
After the data transformation, the obtained data file can be extracted into a target table of the source database, for example, a dbback up database of a production database, so that all sensitive data are encrypted and transformed when the target data are extracted from an original dwpdata database into the dbback up database and are mapped to a directory of a transit machine at the same time through the cooperative work of the data transformation table data _ trans _ joband the transformation algorithm table data _ trans _ arith.
Optionally, the application may start a data service tool GDS and create an appearance in advance, where a data file corresponding to the appearance maps a target table in the dbback up library to the transfer machine through an ip and a port of a data transmission service file defined in the appearance, and data in a field where a transformation function is put on is transformed according to an algorithm criterion to generate a transfer machine data file.
It is understood that the GDS is a distributed framework data service tool, provides data import \ export functions, and is deployed on a data server. The data server is located outside the database system and connected with the system through a network. In the process of data parallel import, the data in the data file is directly imported/exported to the data node through the appearance, so that the operation of 'deforming and extracting' the data to be migrated can be realized, and the data migration efficiency is improved.
The outer surface stores information such as the position, file format, storage position, coding format, separators among data and the like of the data source file, and can help a data service tool to identify the data source file.
Therefore, after the data is extracted into files, the data migration can be realized only by configuring the relevant information of the central switching machine on the scheduling server of the target data.
Step S102: and extracting the data in the target table to the appearance, and generating a transfer machine data file so that the scheduling server can transfer the transfer machine data file to a corresponding target database according to the directory parameters in the transfer machine.
Optionally, the method of the present application is preferably applied to a production environment, and may also be applied to a test environment, where in the test environment, the present application starts a data service tool, creates a table according to the read table field information, executes insert operation to the table through a service table, and places the data file in the transfer machine to a database.
As can be seen from the above description, in the distributed cluster data migration method provided in this embodiment of the present application, the target table corresponding to the data to be migrated that has undergone data transformation in the source database can be mapped to the corresponding transfer machine through the data transmission port and the IP address defined in the external table, so that the scheduling server migrates the data file of the transfer machine to the corresponding target database according to the directory parameter in the transfer machine, thereby avoiding secondary transmission of the data to be migrated falling from the table, effectively improving the data migration efficiency, and saving the system and human resources.
In order to accurately identify data to be migrated from the source database, in an embodiment of the distributed cluster data migration method according to the present application, before the step S101, the following may be further included:
and according to the position information of the data to be migrated and the specific data separator defined in the preset appearance, performing data to be migrated identification operation on the source database to obtain the data to be migrated.
The appearance stores information such as the position, file format, storage position, coding format, and separator between data of the data source file, and can help the data service tool to identify the data source file and obtain the data to be migrated.
In order to accurately determine data that needs to be subjected to data transformation in the data to be migrated, in an embodiment of the distributed cluster data migration method of the present application, the step S102 may further specifically include the following:
and determining a to-be-deformed table and a corresponding to-be-deformed field which need to be subjected to data deformation in the to-be-migrated data according to a preset data deformation table.
In order to accurately determine a specific transformation function of data transformation, in an embodiment of the distributed cluster data migration method of the present application, the step S102 may further specifically include the following steps:
and determining a deformation function corresponding to each field to be deformed according to a preset deformation algorithm table and carrying out encryption deformation operation.
Optionally, which fields in which tables in the database need to be encrypted and transformed are recorded in the data transformation table data _ trans _ joba. Recorded in the morphing algorithm table data _ trans _ arith are which morphing functions the relevant sensitive fields need to apply.
After the data transformation, the obtained data file can be extracted into a target table of the source database, for example, a dbback up database of a production database, so that all sensitive data are encrypted and transformed when the target data are extracted from an original dwpdata database into the dbback up database and are mapped to a directory of a transit machine at the same time through the cooperative work of the data transformation table data _ trans _ joband the transformation algorithm table data _ trans _ arith.
In order to effectively improve data migration efficiency and save system and human resources, the present application provides an embodiment of a distributed cluster data migration apparatus for implementing all or part of contents of the distributed cluster data migration method, and referring to fig. 2, the distributed cluster data migration apparatus specifically includes the following contents:
the data transformation module 10 is configured to perform data transformation on data to be migrated in the source database according to a preset data transformation rule, and extract the data to be migrated after the data transformation into a target table of the source database, where the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset table.
And the data migration module 20 is configured to extract the data in the target table to the external table, and generate a transfer machine data file, so that the scheduling server migrates the transfer machine data file to a corresponding target database according to the directory parameter in the transfer machine.
As can be seen from the above description, the distributed cluster data migration apparatus provided in this embodiment of the present application can map, through the data transmission port and the IP address defined in the external appearance, the target table corresponding to the data to be migrated that has undergone data transformation in the source database to the corresponding transfer machine, so that the scheduling server migrates the data file of the transfer machine to the corresponding target database according to the directory parameter in the transfer machine, thereby avoiding secondary transmission of the data to be migrated falling from the table, and being capable of effectively improving data migration efficiency and saving system and human resources.
In order to accurately identify data to be migrated from a source database, in an embodiment of the distributed cluster data migration apparatus of the present application, the following contents are further specifically included:
and the data to be migrated identification unit is used for carrying out data to be migrated identification operation on the source database according to the position information of the data to be migrated and the specific data separator defined in the preset appearance to obtain the data to be migrated.
In order to accurately determine data that needs to be subjected to data transformation in the data to be migrated, in an embodiment of the distributed cluster data migration apparatus of the present application, the following is further specifically included:
and the to-be-deformed field determining unit is used for determining the to-be-deformed table and the corresponding to-be-deformed field which need to be subjected to data deformation in the to-be-migrated data according to a preset data deformation table.
In order to accurately determine a specific transformation function of data transformation, in an embodiment of the distributed cluster data migration apparatus of the present application, the following contents are further specifically included:
and the deformation function determining unit is used for determining the deformation function corresponding to each field to be deformed according to a preset deformation algorithm table and carrying out encryption deformation operation.
To further illustrate the present solution, the present application further provides a specific application example of implementing the distributed cluster data migration method by using the distributed cluster data migration apparatus, which is shown in fig. 3 and fig. 4, and specifically includes the following contents:
step 1, in a production environment, deforming the data in the target table, backing up the deformed data to a dbback up library of a production database, and meanwhile, sleeving a sensitive field with a deformation function.
And step 2, starting a data service tool, creating an appearance, mapping a target table in the dbback up library to a transit machine through defining an ip and a port of a data transmission service file in the appearance by a data file corresponding to the appearance, and deforming data under a field sleeved with a deformation function according to an algorithm criterion to generate a data file.
And 3, starting a data service tool in the test environment, creating a surface according to the read surface field information, executing insert operation to the surface through the service table, and landing the data file in the transfer machine to the database.
The GDS is a distributed framework data service tool, provides a data import/export function, and is deployed on a data server. The data server is located outside the database system and connected with the system through a network. In the process of data parallel import, the data in the data file is directly imported/exported to the data node through the appearance.
Specifically, the appearance stores information such as the position, file format, storage position, coding format, and separators between data of the data source file, and the function of the information is to help the data service tool identify the data source file.
Specifically, the transfer machine is a transfer directory of data files, and data files derived from the production dbback up library can be stored in a target directory by configuring gds parameter-d (directory), and the examples are as follows:
./gds_C80-d/approot1/mpp/mppedw-p 73.16.161.143:23356-H 0.0.0.0/0-l/approot1/mpp/mppedw/gds_exp6.txt-D-t 4-S 30720GB
./gds_C80-d/approot1/mpp/mppedw-p 73.16.161.143:23357-H 0.0.0.0/0-l/approot1/mpp/mppedw/gds_exp7.txt-D-t 4-S 30720GB
./gds_C80-d/approot1/mpp/mppedw-p 73.16.161.143:23358-H 0.0.0.0/0-l/approot1/mpp/mppedw/gds_exp8.txt-D-t 4-S 30720GB
in the above statements, gds is configured on 73.16.161.143 transfer machine, 3 ports 23356, 23357 and 23358 are available at present, data files are generated in the directory of/aproot 1/mpp/mppedw, and the running log of gds is viewed at/aproot 1/mpp/mppedw.
Similarly, after the data is extracted into the file, only the relevant information of the transfer machine needs to be configured on the current scheduling server of the data, so that the data migration can be realized.
It can be understood that due to the sensitivity of the production data, in order to ensure the data security and reduce the risk brought by data extraction to the production library to the maximum extent, the production database is separately classified into a dbbackup database for testing and storing the extracted target data. In addition, a data transformation table data _ trans _ joband a transformation algorithm table data _ trans _ arith are designed.
Specifically, which fields in which tables in the database need to be encrypted and transformed are recorded in the data transformation table data _ trans _ joba.
Specifically, what transformation functions are needed to be applied to the relevant sensitive fields are recorded in the transformation algorithm table data _ trans _ arith, and the following takes the table dwpdata.t03_ MTG _ prog _ INFO _ H as an example:
DWPDATA. T03_ MTG _ PROJ _ INFO _ H records in data _ trans _ joba that HOUSE _ SELL _ LICS _ ORG field (d _ col _ name) needs to be deformed by algorithm 1(arith _ id), and it can be known that the deformation function with arith _ id of 1 in data _ trans _ arith is TransName (trim (< param0 >).
Therefore, the data transformation table data _ trans _ joband the transformation algorithm table data _ trans _ arith work cooperatively, and when target data are extracted from an original dwwpdata base to a dbback up base and are mapped to a transit directory at the same time, all sensitive data are encrypted and transformed.
As can be seen from the above, compared with the conventional dump method of the orcalcle database, the GDS supports high concurrency, and can increase the transmission rate by about 60%; the data source is directly connected with the data nodes, and the data files are directly landed on the database through an appearance mechanism, so that secondary transmission of data landing from the table is avoided; the data file is stored in the transfer machine, so that the performance of the scheduling server is prevented from being influenced; and extra manual monitoring is not needed in the data migration process, so that the human resources are saved.
In terms of hardware, in order to effectively improve data migration efficiency and save system and human resources, the present application provides an embodiment of an electronic device for implementing all or part of contents in the distributed cluster data migration method, where the electronic device specifically includes the following contents:
a processor (processor), a memory (memory), a communication Interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission between the distributed cluster data migration device and relevant equipment such as a core service system, a user terminal and a relevant database; the logic controller may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the logic controller may be implemented with reference to the embodiment of the distributed cluster data migration method in the embodiment and the embodiment of the distributed cluster data migration apparatus, which are incorporated herein, and repeated details are not repeated here.
It is understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), an in-vehicle device, a smart wearable device, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..
In practical applications, part of the distributed cluster data migration method may be executed on the electronic device side as described in the above, or all operations may be completed in the client device. The selection may be specifically performed according to the processing capability of the client device, the limitation of the user usage scenario, and the like. This is not a limitation of the present application. The client device may further include a processor if all operations are performed in the client device.
The client device may have a communication module (i.e., a communication unit), and may be communicatively connected to a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.
Fig. 5 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 5, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this FIG. 5 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In an embodiment, the distributed cluster data migration method functions may be integrated into the central processor 9100. The central processor 9100 may be configured to control as follows:
step S101: data deformation is carried out on data to be migrated in a source database according to a preset data deformation rule, and the data to be migrated after the data deformation is extracted into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset appearance.
Step S102: and extracting the data in the target table to the appearance, and generating a transfer machine data file so that the scheduling server can transfer the transfer machine data file to a corresponding target database according to the directory parameters in the transfer machine.
As can be seen from the above description, in the electronic device provided in the embodiment of the present application, the target table corresponding to the data to be migrated that has undergone data transformation in the source database is mapped to the corresponding transfer machine through the data transmission port and the IP address defined in the external table, so that the scheduling server migrates the data file of the transfer machine to the corresponding target database according to the directory parameter in the transfer machine, thereby avoiding secondary transmission of the data to be migrated falling from the table, effectively improving data migration efficiency, and saving system and human resources.
In another embodiment, the distributed cluster data migration apparatus may be configured separately from the central processor 9100, for example, the distributed cluster data migration apparatus may be configured as a chip connected to the central processor 9100, and the functions of the distributed cluster data migration method are implemented by the control of the central processor.
As shown in fig. 5, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 5; further, the electronic device 9600 may further include components not shown in fig. 5, which may be referred to in the art.
As shown in fig. 5, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.
The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.
The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.
The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.
An embodiment of the present application further provides a computer-readable storage medium capable of implementing all the steps in the distributed cluster data migration method with a server or a client as an execution subject in the foregoing embodiments, where the computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the computer program implements all the steps in the distributed cluster data migration method with a server or a client as an execution subject in the foregoing embodiments, for example, when the processor executes the computer program, the processor implements the following steps:
step S101: data deformation is carried out on data to be migrated in a source database according to a preset data deformation rule, and the data to be migrated after the data deformation is extracted into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset appearance.
Step S102: and extracting the data in the target table to the appearance, and generating a transfer machine data file so that the scheduling server can transfer the transfer machine data file to a corresponding target database according to the directory parameters in the transfer machine.
As can be seen from the above description, in the computer-readable storage medium provided in this embodiment of the present application, the target table corresponding to the data to be migrated that has undergone data transformation in the source database is mapped to the corresponding transfer machine through the data transmission port and the IP address defined in the external table, so that the scheduling server migrates the data file of the transfer machine to the corresponding target database according to the directory parameter in the transfer machine, thereby avoiding secondary transmission of the data to be migrated falling from the table, effectively improving the data migration efficiency, and saving the system and human resources.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims (10)

1. A distributed cluster data migration method, the method comprising:
performing data deformation on data to be migrated in a source database according to a preset data deformation rule, and extracting the data to be migrated after the data deformation into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset appearance;
and extracting the data in the target table to the appearance, and generating a transfer machine data file so that the scheduling server can transfer the transfer machine data file to a corresponding target database according to the directory parameters in the transfer machine.
2. The distributed cluster data migration method according to claim 1, wherein before performing data transformation on the data to be migrated in the source database according to a preset data transformation rule, the method further comprises:
and according to the position information of the data to be migrated and the specific data separator defined in the preset appearance, performing data to be migrated identification operation on the source database to obtain the data to be migrated.
3. The distributed cluster data migration method according to claim 1, wherein the data to be migrated in the source database is subjected to data transformation according to a preset data transformation rule, and further comprising:
and determining a to-be-deformed table and a corresponding to-be-deformed field which need to be subjected to data deformation in the to-be-migrated data according to a preset data deformation table.
4. The distributed cluster data migration method according to claim 2, wherein after determining, according to a preset data transformation table, a to-be-transformed table and a corresponding to-be-transformed field that need data transformation in the to-be-migrated data, the method includes:
and determining a deformation function corresponding to each field to be deformed according to a preset deformation algorithm table and carrying out encryption deformation operation.
5. A distributed cluster data migration apparatus, comprising:
the data transformation module is used for carrying out data transformation on data to be migrated in the source database according to a preset data transformation rule and extracting the data to be migrated after the data transformation into a target table of the source database, wherein the target table is mapped to a corresponding transfer machine through a data transmission port and an IP address defined in a preset table;
and the data migration module is used for extracting the data in the target table to the appearance and generating a transfer machine data file so that the scheduling server migrates the transfer machine data file to a corresponding target database according to the directory parameters in the transfer machine.
6. The distributed cluster data migration apparatus of claim 5, further comprising:
and the data to be migrated identification unit is used for carrying out data to be migrated identification operation on the source database according to the position information of the data to be migrated and the specific data separator defined in the preset appearance to obtain the data to be migrated.
7. The distributed cluster data migration apparatus of claim 5, further comprising:
and the to-be-deformed field determining unit is used for determining the to-be-deformed table and the corresponding to-be-deformed field which need to be subjected to data deformation in the to-be-migrated data according to a preset data deformation table.
8. The distributed cluster data migration apparatus of claim 7, further comprising:
and the deformation function determining unit is used for determining the deformation function corresponding to each field to be deformed according to a preset deformation algorithm table and carrying out encryption deformation operation.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the distributed cluster data migration method of any of claims 1 to 4 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the distributed cluster data migration method according to any one of claims 1 to 4.
CN202110739626.4A 2021-06-30 2021-06-30 Distributed cluster data migration method and device Pending CN113392091A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110739626.4A CN113392091A (en) 2021-06-30 2021-06-30 Distributed cluster data migration method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110739626.4A CN113392091A (en) 2021-06-30 2021-06-30 Distributed cluster data migration method and device

Publications (1)

Publication Number Publication Date
CN113392091A true CN113392091A (en) 2021-09-14

Family

ID=77624709

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110739626.4A Pending CN113392091A (en) 2021-06-30 2021-06-30 Distributed cluster data migration method and device

Country Status (1)

Country Link
CN (1) CN113392091A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115277840A (en) * 2022-03-18 2022-11-01 中国建设银行股份有限公司 Data migration method and device, electronic equipment and computer readable medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115277840A (en) * 2022-03-18 2022-11-01 中国建设银行股份有限公司 Data migration method and device, electronic equipment and computer readable medium
CN115277840B (en) * 2022-03-18 2024-04-23 中国建设银行股份有限公司 Data migration method, device, electronic equipment and computer readable medium

Similar Documents

Publication Publication Date Title
CN108388988A (en) Synergetic office work method, system and computer readable storage medium based on block chain
CN112463634B (en) Software testing method and device under micro-service architecture
US9451423B2 (en) Method and apparatus for recording information during a call
CN111679790A (en) Remote software development storage space distribution method and device
CN111464352A (en) Call link data processing method and device
CN112953908A (en) Network isolation configuration method, device and system
CN113392091A (en) Distributed cluster data migration method and device
CN111367561B (en) Remote development method and device for software program
CN112396511A (en) Distributed wind control variable data processing method, device and system
CN115495519A (en) Report data processing method and device
CN111880896B (en) Method and device for rapidly restoring container and running state data
CN113110829B (en) Multi-UI component library data processing method and device
CN113791984B (en) Automatic interface testing method and device
CN115421976A (en) Remote disaster recovery data processing method and device
CN113986879A (en) Service data migration method and related device
CN112905464B (en) Application running environment data processing method and device
CN114416212A (en) Mobile terminal application program data resource processing method and device
CN113569289A (en) Data desensitization method and device
CN112766827A (en) Automatic processing method and device for asset inventory data
CN113613061A (en) Checkpoint template generation method, checkpoint template generation device, checkpoint template generation equipment and storage medium
CN112102058A (en) Management system parameter configuration method and device
CN104317670A (en) Android system trace forensic restorer
CN113342501B (en) System fault processing method and device
CN118113273A (en) Rendering engine plug-in data processing method and device
CN115658076A (en) Program relevance analysis method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination