CN117131018A

CN117131018A - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN117131018A
Application number: CN202311147202.4A
Authority: CN
Inventors: 周明
Original assignee: CCB Finetech Co Ltd
Current assignee: CCB Finetech Co Ltd
Priority date: 2023-09-06
Filing date: 2023-09-06
Publication date: 2023-11-28

Abstract

The disclosure provides a data processing method, a data processing device, electronic equipment and a storage medium, which can be applied to the technical field of cloud computing and the technical field of finance. The method comprises the following steps: in response to receiving a data migration request regarding a first database associated with the cloud service platform, establishing a migration path with the first database; acquiring a data set related to the first database by utilizing a migration path, wherein the data set comprises an analysis data set and a storage data set, the analysis data set represents an analysis result of data to be migrated in the first database, and the storage data set represents a set of data to be migrated in the first database; determining creation information related to the data to be migrated according to the analysis data set; according to the creation information, migrating the data to be migrated in the storage data set to a second database; and determining a migration result of the data to be migrated according to the migrated data in the second database and the analysis data set.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of cloud computing technology and the field of financial technology, and in particular, to a data processing method, apparatus, electronic device, storage medium, and program product.

Background

With the rapid development of cloud service technology, more and more users select cloud servers. The cloud is a platform which is intelligently used as a carrier, and a new mode of using resources including computing resources, storage resources, application software, services, networks and the like as required is provided on a large scale. In the process of using the cloud service, migration of data related to the cloud service is avoided.

In the process of implementing the disclosed concept, the inventor finds that at least the following problems exist in the related art: in the related art, a method for migrating data related to cloud service may cause data deletion and inconsistency, so that accuracy of the data is reduced, and the migrated data cannot be used.

Disclosure of Invention

In view of the foregoing, the present disclosure provides a data processing method, apparatus, electronic device, storage medium, and program product.

A first aspect of the present disclosure provides a data processing method, including:

establishing a migration path between a cloud service platform and a first database in response to receiving a data migration request about the first database;

acquiring a data set related to the first database by using the migration path, wherein the data set comprises an analysis data set and a storage data set, the analysis data set represents an analysis result of data to be migrated in the first database, and the storage data set represents a set of data to be migrated in the first database;

Determining creation information related to the data to be migrated according to the analysis data set;

migrating the data to be migrated in the storage data set to a second database according to the creation information; and

and determining a migration result of the data to be migrated according to the migrated data in the second database and the analysis data set.

According to an embodiment of the present disclosure, the migrating the data to be migrated in the storage data set to a second database according to the creation information includes:

under the condition that a creating user exists in the data to be migrated in the storage data set according to the creating information, migrating the data to be migrated in the storage data set to the second database according to account information related to the creating user; and

and under the condition that the creation user does not exist in the data to be migrated in the storage data set according to the creation information, migrating the data to be migrated in the storage data set to the second database according to account information of a new user.

According to an embodiment of the present disclosure, the acquiring, by using the migration path, a data set related to the first database includes:

Determining data migration time according to the data migration request;

determining a service sub-database set according to the service sub-databases in the first database; and

and acquiring a data set related to the first database by utilizing the migration path according to the data migration time and the service sub-database set.

According to an embodiment of the present disclosure, the service sub-database set includes I service sub-databases, I being an integer greater than or equal to 1;

the acquiring, according to the service sub-database set, the data set related to the first database by using the migration path includes:

and repeatedly executing the following operations according to the service sub-database set until a preset condition is met:

under the condition that the ith service sub-database in the service sub-database set is determined to exist, determining an ith analysis data subset and an ith storage data subset which correspond to the ith service sub-database, wherein I is a positive integer smaller than I;

determining an ith data subset corresponding to the ith service sub-database according to the ith analysis data subset corresponding to the ith service sub-database and the ith storage data subset, wherein i=i+1;

Under the condition that I is determined to be larger than I, determining that a preset condition is met; and

and determining a data set related to the first database according to the I data subsets under the condition that the preset condition is met.

According to an embodiment of the present disclosure, the determining, in the case where it is determined that an ith service sub-database in the service sub-database set exists, an ith analysis data subset and the ith storage data subset corresponding to the ith service sub-database includes:

determining a database object type corresponding to the ith service sub-database, the number of database objects corresponding to the database object type and the ith creation information of the ith sub-data to be migrated corresponding to the ith service sub-database under the condition that the ith service sub-database in the service sub-database set is determined to exist;

determining an ith analysis data subset related to the ith service sub-database according to the type of the database object corresponding to the ith service sub-database, the number of database objects corresponding to the type of the database object and the ith creation information of the ith sub-data to be migrated corresponding to the ith service sub-database;

Determining an ith storage data subset corresponding to the ith service sub-database according to the ith sub-data to be migrated corresponding to the ith service sub-database; and

determining an ith data set associated with said first database based on said ith analysis data subset and said ith storage data subset associated with said ith service sub-database.

According to an embodiment of the present disclosure, the determining, according to the migrated data in the second database and the analysis data set, a migration result of the data to be migrated includes:

generating a migration log corresponding to the migrated data according to the migration process of the data to be migrated; and

and determining the migration result of the data to be migrated according to the migration log and the analysis data set.

According to an embodiment of the present disclosure, the above data processing method further includes:

uploading the data set to a storage space associated with the second database;

wherein, according to the creation information, migrating the data to be migrated in the storage data set to a second database, and further comprising:

and according to the creation information, migrating the data to be migrated in the storage data set from the storage space to a second database.

A second aspect of the present disclosure provides a data processing apparatus comprising:

the cloud service platform comprises a building module, a first database and a second database, wherein the building module is used for building a migration path between the cloud service platform and the first database in response to receiving a data migration request about the first database related to the cloud service platform;

the acquisition module is used for acquiring a data set related to the first database by using the migration path, wherein the data set comprises an analysis data set and a storage data set, the analysis data set represents an analysis result of data to be migrated in the first database, and the storage data set represents a set of data to be migrated in the first database;

the first determining module is used for determining creation information related to the data to be migrated according to the analysis data set;

the migration module is used for migrating the data to be migrated in the storage data set to a second database according to the creation information;

and the second determining module is used for determining the migration result of the data to be migrated according to the migrated data in the second database and the analysis data set.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method described above.

A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described method.

A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above method.

According to the data processing method, the device, the electronic equipment, the storage medium and the program product provided by the disclosure, a migration path between a cloud service platform and a first database is established by responding to the received data migration request of the first database; acquiring a data set related to the first database by utilizing the migration path; determining creation information related to the data to be migrated according to the analysis data set; then, according to the creation information, the data to be migrated in the storage data set is migrated to a second database; and finally, determining the migration result of the data to be migrated according to the migrated data in the second database and the analysis data set. The method has the advantages that the data to be migrated in the storage data set is migrated to the second data base according to the creation information, the problem that the data cannot be used due to different authorities is avoided, the migration result of the data to be migrated can be determined according to the migrated data and the analysis data set in the second data base, the migrated data can be checked timely, the problem in the migrated data is found, the problem that the data cannot be used due to data missing and inconsistency is avoided, manual checking is not needed, and the accuracy and efficiency of the migrated data are improved.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a data processing method, apparatus, device, medium and program product according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a data processing method according to an embodiment of the disclosure;

FIG. 3 schematically illustrates a flow chart of yet another data processing method according to an embodiment of the disclosure;

FIG. 4 schematically illustrates a flow chart of yet another data processing method according to an embodiment of the disclosure;

FIG. 5 schematically illustrates a flow chart of yet another data processing method according to an embodiment of the disclosure;

FIG. 6 schematically illustrates a block diagram of a data processing apparatus according to an embodiment of the present disclosure; and

fig. 7 schematically illustrates a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

In the technical scheme of the disclosure, the related data (such as including but not limited to personal information of a user) are collected, stored, used, processed, transmitted, provided, disclosed, applied and the like, all conform to the regulations of related laws and regulations, necessary security measures are adopted, and the public welcome is not violated.

In the process of implementing the present disclosure, it is found that, in a method of manually performing data migration, a person who needs to perform the data migration often has better database expertise, but the technical levels of different persons are inconsistent, and in the process of performing the data migration, data deletion and inconsistency may be caused by irregular and inconsistent operations, so that the migrated data cannot be used, the accuracy of data migration is reduced, and migration or repair needs to be performed again, which results in longer migration time and even affects the service.

To this end, an embodiment of the present disclosure provides a data processing method, including: in response to receiving a data migration request regarding a first database associated with the cloud service platform, establishing a migration path with the first database; acquiring a data set related to the first database by utilizing a migration path, wherein the data set comprises an analysis data set and a storage data set, the analysis data set represents an analysis result of data to be migrated in the first database, and the storage data set represents a set of data to be migrated in the first database; determining creation information related to the data to be migrated according to the analysis data set; according to the creation information, migrating the data to be migrated in the storage data set to a second database; and determining a migration result of the data to be migrated according to the migrated data in the second database and the analysis data set.

Fig. 1 schematically illustrates an application scenario diagram for a data processing method according to an embodiment of the present disclosure.

As shown in fig. 1, an application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium used to provide a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 through the network 104 using at least one of the first terminal device 101, the second terminal device 102, the third terminal device 103, to receive or send messages, etc. Various communication client applications, such as a shopping class application, a web browser application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only) may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103.

The first terminal device 101, the second terminal device 102, the third terminal device 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.

For example, a migration path may be established with a first database associated with a cloud service platform by server 105 in response to receiving a data migration request with the first database; acquiring a data set related to the first database by utilizing the migration path; determining creation information related to the data to be migrated according to the analysis data set; then, according to the creation information, the data to be migrated in the storage data set is migrated to a second database; and finally, determining the migration result of the data to be migrated according to the migrated data in the second database and the analysis data set.

It should be noted that the data processing method provided in the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may be generally provided in the server 105. The data processing method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The data processing method of the disclosed embodiment will be described in detail below with reference to fig. 2 to 6 based on the scenario described in fig. 1.

Fig. 2 schematically illustrates a flow chart of a data processing method according to an embodiment of the present disclosure.

As shown in fig. 2, the data processing method of this embodiment includes operations S210 to S250.

In operation S210, in response to receiving a data migration request with respect to a first database related to a cloud service platform, a migration path with the first database is established.

According to embodiments of the present disclosure, a cloud service platform may be a platform provided by internet-based related service augmentation, usage, and delivery modes, which generally involves processing computing data through a cloud service, which may be a dynamic, easily-expanding virtualized resource provided through the internet.

According to embodiments of the present disclosure, the first database may characterize a database that migrates data to be migrated. For example, the first database may be a relational database management system (My Structured Query Language, mySQL) database.

According to an embodiment of the present disclosure, the migration request may include path establishment information related to the first database, where the path establishment information may include a database name, a database user name, a database password, and the like. A migration path with the first data may be established according to the path establishment information.

In operation S220, a data set associated with the first database is acquired using the migration path.

According to an embodiment of the present disclosure, the dataset includes an analysis dataset and a storage dataset.

According to the embodiment of the disclosure, the analysis data set may represent an analysis result of data to be migrated in the first database, and the analysis result may include a result of analyzing a data type of the data to be migrated in the first database and may further include a result of analyzing creation information of the data to be migrated in the first database.

According to embodiments of the present disclosure, the stored data set may characterize a set of data to be migrated in the first database.

According to embodiments of the present disclosure, the dataset may not include built-in database tables in the first database. The built-in database table may be a database that is automatically created by the database at the time of creating an instance for internal management and operational use of the database. For example, where the first database is a MySQL database, the built-in database table of the MySQL database may include: information_ schema, mysql, performance _ schema, sys, world, sysdb, etc., and thus the dataset may not include built-in database tables such as information_ schema, mysql, performance _ schema, sys, world, sysdb, etc.

In operation S230, creation information related to data to be migrated is determined from the analysis dataset.

According to embodiments of the present disclosure, the creation information may characterize information in the first database related to the creator of the data to be migrated. For example, the creation information may be information such as a user name and a password of a creation user of the data to be migrated, and may also be authority information set by a creator of the data to be migrated.

In operation S240, the data to be migrated in the storage data set is migrated to the second database according to the creation information.

According to embodiments of the present disclosure, the second database may characterize the database that migrated into the data to be migrated.

According to the embodiment of the disclosure, the user name and the password of the user can be created according to the creation information, and the data to be migrated in the storage data set can be migrated to the second database. Corresponding rights can also be set for the data migrated to the second database according to the rights information set by the creator. For example, the data to be migrated is created by the creating user Zhang Sanchuang in a first database and created by the creating user Zhang three in a second database.

In operation S250, a migration result of the data to be migrated is determined according to the migrated data and the analysis data set in the second database.

According to the embodiment of the disclosure, whether the migration result of the migration data is normal can be determined by comparing the migrated data in the second database with the analysis data set.

According to the embodiment of the disclosure, when the data after migration is determined to be consistent with the analysis result in the analysis data set, the migration result of the data to be migrated is determined to represent normal migration, and when the data after migration is determined to be inconsistent with the analysis result in the analysis data set, the migration result of the data to be migrated is determined to represent abnormal migration. Under the condition that the migration result represents the migration abnormality, notification information can be generated and sent to related operation and maintenance users for timely processing.

According to an embodiment of the present disclosure, in response to receiving a data migration request regarding a first database associated with a cloud service platform, a migration path is established with the first database; acquiring a data set related to the first database by utilizing the migration path; determining creation information related to the data to be migrated according to the analysis data set; then, according to the creation information, the data to be migrated in the storage data set is migrated to a second database; and finally, determining the migration result of the data to be migrated according to the migrated data in the second database and the analysis data set. The method has the advantages that the data to be migrated in the storage data set is migrated to the second data base according to the creation information, the problem that the data cannot be used due to different authorities is avoided, the migration result of the data to be migrated can be determined according to the migrated data and the analysis data set in the second data base, the migrated data can be checked timely, the problem in the migrated data is found, the problem that the data cannot be used due to data missing and inconsistency is avoided, manual checking is not needed, and the accuracy and efficiency of the migrated data are improved.

According to an embodiment of the present disclosure, migrating data to be migrated in a storage dataset to a second database according to creation information, including: under the condition that the creating user exists in the data to be migrated in the storage data set according to the creating information, migrating the data to be migrated in the storage data set to a second database according to account information related to the creating user; and under the condition that the creating user does not exist in the data to be migrated in the storage data set according to the creating information, migrating the data to be migrated in the storage data set to the second database according to the account information of the newly-built user.

According to an embodiment of the present disclosure, in a case where it is determined that there is a creation user with data to be migrated in a storage data set according to creation information, account information related to the creation user may be determined according to the creation information, and the account information may be a user name and a password of the creation user. And migrating the data to be migrated in the storage data set to the second database by creating a user name and a password of the user. The creation information of the data to be migrated may include a plurality of creation users. For example, the data to be migrated may be created by both creating user three and creating user Lifour. The data to be migrated is migrated to the second database by the account information related to the creating user, so that the authority of the creating user in the first database and the authority of the creating user in the second database can be kept consistent.

According to the embodiment of the disclosure, in the case that it is determined that the creating user does not exist in the data to be migrated in the storage data set according to the creating information, the data to be migrated in the storage data set can be migrated to the second database through the user name and the password of the newly created user.

According to the embodiment of the disclosure, the data to be migrated is migrated to the second database by creating the account information related to the user or the account information of the newly-built user, so that the authority of the migrated data in the second database is ensured to be consistent with the authority of the migrated data in the first database, the problem that the data cannot be used due to different authorities is avoided, and the accuracy and the safety of data migration are improved.

According to an embodiment of the present disclosure, acquiring a data set related to a first database using a migration path includes: determining data migration time according to the data migration request; determining a service sub-database set according to the service sub-databases in the first database; and acquiring the data set related to the first database by utilizing the migration path according to the data migration time and the service sub-database set.

According to embodiments of the present disclosure, database consistency reads may be concurrent read operations in a database in which the read data should be consistent. In particular, when one transaction is performing a read operation, if another transaction is modifying the same data (which may be, for example, a write operation), the read operation should retrieve the latest version of data that has been committed, rather than read the partially modified or old version of data. Read consistency may ensure accuracy and reliability of data, that is, data obtained based on database consistency may be database data that a database has committed based on a point in time, excluding data that committed after that point in time. Therefore, the data migration time is determined according to the data migration request, and data inconsistency before and after migration can be avoided.

For example, the time point of receiving the data migration request is 3 points, the time point of starting the migration is 4 points, and in the case that the data in the first database changes from 3 points to 4 points, the data to be migrated only includes the data before 3 points.

According to embodiments of the present disclosure, the global transaction identification (Global Transaction Identifier, GTID) may be an identification of MySQL database set to prevent data synchronization inconsistencies, commonly used for master-slave synchronization. And the process of migrating the data to be migrated in the first database to the second database does not comprise global transaction identification.

According to embodiments of the present disclosure, the business database may include a plurality of business sub-databases, such as a transfer sub-database, a query balance sub-database, a query bill sub-database, and the like.

According to the embodiment of the disclosure, the data consistency can be ensured by acquiring the data set related to the first database by utilizing the migration path according to the data migration time and the service sub-database set, so that the correctness and the integrity of the data are improved.

Fig. 3 schematically shows a flow chart of a further data processing method according to the present disclosure.

As shown in FIG. 3, the method 300 includes operations S310-S360.

In operation S310, it is determined whether the service sub-database set is empty. In the case where it is determined that the service sub-data set is empty, operation S360 is performed, and in the case where it is determined that the service sub-data set is not empty, operation S320 is performed. The set of business sub-databases may include I business sub-databases, I being an integer greater than or equal to 1.

In operation S320, it is determined whether an ith service sub-database exists, I being a positive integer less than I. In case it is determined that the ith service sub-database exists, operation S330 is performed, and in case it is determined that the ith service sub-database does not exist, operation S350 is performed.

In operation S330, an i-th analysis data subset and an i-th storage data subset corresponding to the i-th service sub-database are determined.

According to embodiments of the present disclosure, the analysis data subset may include an object type of the data to be migrated in the business sub-database, a number corresponding to the object type. The analysis data subset may be named according to the name of the object type.

According to embodiments of the present disclosure, the subset of stored data may include sub-data to be migrated in the business sub-database, which may be tables, indexes, triggers, events, stored procedures, and functions. The subset of stored data may be named according to the name of the business sub-database.

In operation S340, an i-th subset of data corresponding to the i-th sub-database of services is determined from the i-th subset of analysis data and the i-th subset of storage data corresponding to the i-th sub-database of services.

In operation S350, let i=i+1, determine whether I is greater than I. In the case where it is determined that I is greater than I, operation S360 is performed, and in the case where it is determined that I is not greater than I, operation S310 is performed.

According to an embodiment of the present disclosure, the preset condition is satisfied in the case where I is greater than I.

In operation S360, in case it is determined that the preset condition is satisfied, a data set related to the first database is determined according to the I data subsets.

According to the embodiment of the disclosure, the data to be migrated in the service sub-databases can be sequentially obtained by traversing the I service sub-databases in the service sub-sets, so that the data sets are generated according to the classification of the service sub-databases, and the uniformity and normalization of the data are improved.

Fig. 4 schematically shows a flow chart of yet another data processing method according to the present disclosure.

As shown in fig. 4, the method 400 includes operations S410 to S440.

In operation S410, in case it is determined that an ith service sub-database of the service sub-database set exists, a database object type corresponding to the ith service sub-database, the number of database objects corresponding to the database object type, and ith creation information of an ith sub-data to be migrated corresponding to the ith service sub-database are determined.

According to embodiments of the present disclosure, database object types may be table types, index types, trigger types, event types, stored procedure types, and function types; the number of database objects corresponding to the database object types may be, for example, 10 table types, 15 index types, 20 trigger types, 25 event types, and the like.

According to the embodiment of the disclosure, the i-th creation information of the i-th sub data to be migrated corresponding to the i-th service sub database may be creation information of the i-th sub data to be migrated corresponding to the i-th service sub database, for example, the i-th sub data to be migrated corresponding to the i-th service sub database is created for creating the user's name and password, and thus the i-th creation information of the i-th sub data to be migrated corresponding to the i-th service sub database may be the account name and password of the user's name and password.

In operation S420, an ith analysis data subset related to the ith service sub-database is determined according to the database object type corresponding to the ith service sub-database, the number of database objects corresponding to the database object type, and the ith creation information of the ith sub-data to be migrated corresponding to the ith service sub-database.

According to the embodiment of the disclosure, the ith database can be detected by detecting scripts, and the database object type corresponding to the ith service sub-database, the number of the database objects corresponding to the database object type and the ith creation information of the ith sub-data to be migrated corresponding to the ith service sub-database are determined.

According to the embodiment of the disclosure, a file can be built according to the object type name corresponding to the ith sub-data to be migrated related to the ith service sub-database, the database object type corresponding to the ith service sub-database, the number of database objects corresponding to the database object type and the ith creation information of the ith sub-data to be migrated corresponding to the ith service sub-database are placed in the file, and the file is compressed to obtain the ith analysis data subset.

In operation S430, an ith stored data subset corresponding to the ith service sub-database is determined according to the ith sub-data to be migrated corresponding to the ith service sub-database.

According to the embodiment of the disclosure, the ith sub-data to be migrated corresponding to the ith sub-database can be obtained according to the ith sub-database, and the ith sub-data to be migrated corresponding to the ith sub-database is compressed to obtain the ith storage data subset corresponding to the ith sub-database.

In operation S440, an ith data set associated with the first database is determined based on the ith analysis data subset and the ith storage data subset associated with the ith service sub-database.

According to the embodiment of the disclosure, under the condition that the ith service sub-database exists, the ith analysis data subset can be determined by analyzing the ith sub-data to be migrated, and the ith storage data subset is determined according to the ith sub-data to be migrated, so that each service sub-database can be traversed, data omission is avoided, and the integrity of the data is further improved.

According to an embodiment of the present disclosure, determining a migration result of data to be migrated according to migrated data and an analysis data set in a second database includes: generating a migration log corresponding to the migrated data according to the migration process of the data to be migrated; and determining a migration result of the data to be migrated according to the migration log and the analysis data set.

According to an embodiment of the present disclosure, the migration log may record an object type of the migrated data in the second database, a number of database objects corresponding to the database object type, for example, 10 table types, 15 index types, 20 trigger types, and 25 event types may be recorded in the migration log.

According to the embodiment of the disclosure, by comparing the object types of the database in the migration log with the number of database objects corresponding to the database object types, it can be determined that the number of event types is inconsistent, and the migration result of the data to be migrated is abnormal.

According to the embodiment of the disclosure, since the analysis data set is a record of data to be migrated, comparing the analysis data set with the migration log can determine whether the migrated data is abnormal. For example, in the case that 10 record table types, 15 index types, 20 trigger types and 25 event types are recorded in the migration log, and 10 record table types, 15 index types, 20 trigger types and 25 event types are recorded in the analysis data set, the migration log is consistent with the analysis data set, and it is determined that the data after migration is normal. Under the condition that the number of record table types in the migration log is 10 and the number of record table types in the analysis data set is 20, the migration log is inconsistent with the analysis data set, the data after migration is determined to be abnormal, and notification information can be timely generated and sent to relevant operation and maintenance users for timely processing.

According to the embodiment of the disclosure, when the data in the second database is detected in the data migration process, the data to be migrated can be compared with the data in the second database, and when the data to be migrated is consistent with the data in the second database, three processing mode options including coverage, skipping and exiting can be provided. The coverage refers to directly covering the data which are the same as the data to be migrated and exist in the second database; skipping refers to skipping over the same data present in the second database as the data to be migrated; exit refers to not performing an migrate database operation for the data to be migrated.

According to the embodiment of the disclosure, the migration result can be determined in time through the migration log and the analysis data set, the migration can be completed under the condition that the migration result represents the normal migration, and the notification information can be generated under the condition that the migration result represents the abnormal migration and sent to the relevant operation and maintenance user for timely processing, so that the accuracy of data migration is improved.

According to an embodiment of the present disclosure, the data processing method may further comprise uploading the data set into a storage space associated with the second database; according to the creation information, migrating the data to be migrated in the storage data set to a second database, and further comprising: and migrating the data to be migrated in the storage data set to the second database from the storage space according to the creation information.

According to the embodiment of the disclosure, if the server performing data migration cannot establish connection with the first database and the second database at the same time, the data to be migrated needs to be transferred to the second database through data transfer, and the storage space can be set to store the data to be migrated. Before data migration, the environment of the server performing data migration may be detected, it may be determined whether the server is provided with a tool for performing data migration, in the case where it is determined that the server is provided with a tool for performing data migration, the parameters connected to the first database or the second database may be checked, and in the case where it is determined that there is no problem in the result of checking the parameters connected to the first database or the second database, data migration may be started.

According to the embodiment of the disclosure, the data to be migrated in the storage data set can be compressed and packaged to obtain the compressed file of the storage data set. The size of the compressed file can be set, if the size of the compressed file exceeds 16G, the compressed file is split and compressed according to 16G, so that the problem that the uploading speed is too slow or the uploading fails due to the fact that the file is uploaded at one time is too large is avoided.

Because the first database can comprise a plurality of service sub-databases, under the condition of migrating the data to be migrated to the second database, the data to be migrated in each service sub-database in the service sub-database set can be sequentially imported to the second database according to the service sub-database set.

According to embodiments of the present disclosure, a compressed file storing a data set may be uploaded into a storage space. Types of upload support include: direct connection, object storage services (Object Storage Service, OBS), shared network attached storage (Network Attached Storage, NAS), and file transfer protocol servers (File Transfer Protocol Server, FTP servers). Wherein, direct connection refers to directly copying the compressed file storing the data set to the storage space according to the user name and the password of the operating user of data processing; the OBS refers to that compressed files of a stored data set can be directly uploaded into a storage space after configuration; the shared NAS refers to copying compressed files storing data sets to a mount point of the NAS on a server performing a data processing method. The FTP server refers to uploading a compressed file storing a data set to the server by a user name and a password of an operating user of the data processing method.

According to the embodiment of the disclosure, the method can support limiting the network transmission speed during file uploading, and avoid network paralysis caused by excessive network resources occupied in the data migration process.

According to the embodiment of the disclosure, in the case that the data to be migrated of the storage data set is uploaded to the storage space, the second database may be connected, in the case that the data to be migrated of the storage data set is downloaded from the storage space, and the data to be migrated is migrated to the second database. Types of file download support for storing data to be migrated of a data set include: object store (Object Storage Service, OSS), shared NAS and FTP servers. The OSS refers to directly downloading a file to a second database through an object link of data to be migrated. The shared NAS refers to copying the compressed files of the stored data set to a local mounting point; FTP server: it means that the compressed file storing the data set is downloaded to the server executing the data processing method by means of the user name and password of the operating user of the data processing method.

According to the embodiment of the disclosure, the network transmission speed is limited by supporting when the file is downloaded, and network paralysis caused by excessive network resources occupied in the data migration process is avoided.

According to the embodiment of the disclosure, multiple transmission mode selections are provided in the file uploading and downloading processes so as to adapt to different network environments and control the transmission rate.

According to the embodiment of the disclosure, the data to be migrated is transferred through the storage space, so that the data to be migrated can be transferred under the condition that the first database and the second database cannot be simultaneously established, and the flexibility of data migration is improved.

For a better understanding of embodiments of the present disclosure, a data processing method of embodiments of the present disclosure is described below in conjunction with fig. 5.

Fig. 5 schematically shows a flow chart of yet another data processing method according to the present disclosure.

As shown in fig. 5, the method includes operations S501 to S515.

In operation S501, in response to receiving a data migration request regarding a first database associated with a cloud service platform, a migration path is established with the first database.

In operation S502, a data migration time is determined according to a data migration request.

In operation S503, a set of service sub-databases is determined according to the service sub-databases in the first database.

In operation S504, it is determined whether the service sub-database set is empty. In the case where it is determined that the service sub data set is empty, operation S509 is performed, and in the case where it is determined that the service sub data set is not empty, operation S505 is performed.

In operation S505, it is determined whether an ith service sub-database exists, I being a positive integer less than I. In the case where it is determined that the ith service sub-database exists, operation S506 is performed, and in the case where it is determined that the ith service sub-database does not exist, operation S508 is performed.

In operation S506, an i-th analysis data subset and an i-th storage data subset corresponding to the i-th service sub-database are determined.

In operation S507, an i-th subset of data corresponding to the i-th sub-database of services is determined from the i-th subset of analysis data and the i-th subset of storage data corresponding to the i-th sub-database of services.

In operation S508, let i=i+1, it is determined whether I is greater than I. In the case where it is determined that I is greater than I, operation S509 is performed, and in the case where it is determined that I is not greater than I, operation S504 is performed.

In operation S509, in case it is determined that the preset condition is satisfied, a data set related to the first database is determined according to the I data subsets and the data migration time.

Uploading the data set into a storage space associated with a second database in operation S510;

in operation S511, in the case of establishing a connection with the second database, the data set is downloaded from the storage space.

In operation S512, it is determined whether there is a creating user for the data to be migrated in the data set. In the case where it is determined that the creation user exists for the data to be migrated in the data set, operation S513 is performed, and in the case where it is determined that the creation user does not exist for the data to be migrated in the data set, operation S514 is performed.

In operation S513, the data to be migrated in the storage data set is migrated to the second database according to the account information related to the creating user.

In operation S514, the data to be migrated in the storage data set is migrated to the second database according to the account information of the newly created user.

In operation S515, a migration result of the data to be migrated is determined according to the migrated data and the analysis data set in the second database.

Operations S501 to S515 may refer to descriptions of other embodiments of the present disclosure, and are not described herein.

Based on the data processing method, the disclosure also provides a data processing device. The device will be described in detail below in connection with fig. 6.

Fig. 6 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

As shown in fig. 6, the data processing apparatus 600 of this embodiment includes a setup module 610, an acquisition module 620, a first determination module 630, a migration module 640, and a second determination module 650.

The establishing module 610 is configured to establish a migration path with a first database in response to receiving a data migration request regarding the first database associated with the cloud service platform. In an embodiment, the establishing module 610 may be configured to perform the operation S210 described above, which is not described herein.

The obtaining module 620 is configured to obtain a data set related to the first database using the migration path, where the data set includes an analysis data set and a storage data set, the analysis data set represents an analysis result of data to be migrated in the first database, and the storage data set represents a set of data to be migrated in the first database. In an embodiment, the obtaining module 620 may be configured to perform the operation S220 described above, which is not described herein.

The first determining module 630 is configured to determine, according to the analysis dataset, creation information related to the data to be migrated. In an embodiment, the first determining module 630 may be configured to perform the operation S230 described above, which is not described herein.

The migration module 640 is configured to migrate data to be migrated in the storage dataset to the second database according to the creation information. In an embodiment, the migration module 640 may be configured to perform the operation S240 described above, which is not described herein.

The second determining module 650 is configured to determine a migration result of the data to be migrated according to the migrated data and the analysis data set in the second database. In an embodiment, the second determining module 650 may be configured to perform the operation S250 described above, which is not described herein.

According to an embodiment of the present disclosure, the migration module 640 for migrating data to be migrated in a storage data set to a second database according to creation information includes:

and the first migration submodule is used for migrating the data to be migrated in the storage data set to the second database according to account information related to the creating user under the condition that the creating user exists in the data to be migrated in the storage data set according to the creating information.

And the second migration submodule is used for migrating the data to be migrated in the storage data set to the second database according to the account information of the newly-built user under the condition that the creation user does not exist in the data to be migrated in the storage data set according to the creation information.

According to an embodiment of the present disclosure, the acquiring module 620 for acquiring a data set related to a first database using a migration path includes:

the first acquisition sub-module is used for determining data migration time according to the data migration request.

And the second acquisition sub-module is used for determining a service sub-database set according to the service sub-databases in the first database.

And the third acquisition sub-module is used for acquiring the data set related to the first database by utilizing the migration path according to the data migration time and the service sub-database set.

According to an embodiment of the present disclosure, a third obtaining submodule for obtaining a data set related to a first database by using a migration path according to a data migration time and a service sub-database set includes:

the first acquisition unit is used for determining an ith analysis data subset and an ith storage data subset corresponding to the ith service sub-database under the condition that the ith service sub-database in the service sub-database set is determined to exist, wherein I is a positive integer smaller than I.

And the second acquisition unit is used for determining an ith data subset corresponding to the ith service sub-database according to the ith analysis data subset and the ith storage data subset corresponding to the ith service sub-database, so that i=i+1.

And the third acquisition unit is used for determining that the preset condition is met under the condition that I is determined to be larger than I.

And the fourth acquisition unit is used for determining a data set related to the first database according to the I data subsets under the condition that the preset condition is met.

According to an embodiment of the present disclosure, in a case where it is determined that an ith service sub-database in the service sub-database set exists, determining an ith analysis data subset and an ith storage data subset corresponding to the ith service sub-database, where I is a positive integer smaller than I, the first obtaining unit includes:

the first obtaining subunit is configured to determine, when it is determined that an ith service sub-database in the service sub-database set exists, a database object type corresponding to the ith service sub-database, a number of database objects corresponding to the database object type, and ith creation information of an ith sub-data to be migrated corresponding to the ith service sub-database.

And the second acquisition subunit is used for determining an ith analysis data subset related to the ith service sub-database according to the database object type corresponding to the ith service sub-database, the number of database objects corresponding to the database object type and the ith creation information of the ith sub-data to be migrated corresponding to the ith service sub-database.

And the third acquisition subunit is used for determining an ith storage data subset corresponding to the ith service sub-database according to the ith sub-data to be migrated corresponding to the ith service sub-database.

A fourth acquisition subunit, configured to determine an ith data set associated with the first database according to the ith analysis data subset and the ith storage data subset associated with the ith service sub-database.

According to an embodiment of the present disclosure, the second determining module 650 for determining a migration result of the data to be migrated according to the migrated data and the analysis data set in the second database includes:

the first determining submodule is used for generating a migration log corresponding to the migrated data according to the migration process of the data to be migrated; and

and the second determining submodule is used for determining the migration result of the data to be migrated according to the migration log and the analysis data set.

According to an embodiment of the present disclosure, the apparatus 600 further includes:

the uploading module is used for uploading the data set to a storage space related to the second database;

wherein, the migration module 640 for migrating the data to be migrated in the storage data set to the second database according to the creation information further includes:

and the third migration submodule is used for migrating the data to be migrated in the storage data set from the storage space to a second database according to the creation information.

Any of the plurality of modules of the setup module 610, the acquisition module 620, the first determination module 630, the migration module 640, and the second determination module 650 may be combined in one module to be implemented, or any of the plurality of modules may be split into a plurality of modules, according to embodiments of the present disclosure. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the setup module 610, the acquisition module 620, the first determination module 630, the migration module 640, and the second determination module 650 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging the circuitry, or in any one of or a suitable combination of any of the three implementations of software, hardware, and firmware. Alternatively, at least one of the setup module 610, the acquisition module 620, the first determination module 630, the migration module 640, and the second determination module 650 may be at least partially implemented as computer program modules, which when executed, may perform the respective functions.

As shown in fig. 7, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. The processor 701 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 701 may also include on-board memory for caching purposes. The processor 701 may comprise a single processing unit or a plurality of processing units for performing different actions of the method flows according to embodiments of the disclosure.

In the RAM 703, various programs and data necessary for the operation of the electronic apparatus 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. The processor 701 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 702 and/or the RAM 703. Note that the program may be stored in one or more memories other than the ROM 702 and the RAM 703. The processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the present disclosure, the electronic device 700 may further include an input/output (I/O) interface 705, the input/output (I/O) interface 705 also being connected to the bus 704. The electronic device 700 may also include one or more of the following components connected to an input/output (I/O) interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to an input/output (I/O) interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 702 and/or RAM 703 and/or one or more memories other than ROM 702 and RAM 703 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code means for causing a computer system to carry out the data processing methods provided by the embodiments of the present disclosure when the computer program product is run on the computer system.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 701. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed over a network medium in the form of signals, downloaded and installed via the communication section 709, and/or installed from the removable medium 711. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 701. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. A data processing method, comprising:

responsive to receiving a data migration request regarding a first database associated with a cloud service platform, establishing a migration path with the first database;

Acquiring a data set related to the first database by utilizing the migration path, wherein the data set comprises an analysis data set and a storage data set, the analysis data set represents an analysis result of data to be migrated in the first database, and the storage data set represents a set of data to be migrated in the first database;

according to the creation information, migrating the data to be migrated in the storage data set to a second database; and

2. The method of claim 1, wherein the migrating the data to be migrated in the stored dataset to a second database according to the creation information comprises:

And under the condition that the creating user does not exist in the data to be migrated in the storage data set according to the creating information, migrating the data to be migrated in the storage data set to the second database according to account information of a new user.

3. The method of claim 1, wherein the utilizing the migration path to obtain the data set associated with the first database comprises:

determining data migration time according to the data migration request;

4. The method of claim 3, wherein the set of business sub-databases comprises I business sub-databases, I being an integer greater than or equal to 1;

the obtaining, according to the service sub-database set, the data set related to the first database by using the migration path includes:

Under the condition that the ith service sub-database in the service sub-database set is determined to exist, determining an ith analysis data subset and an ith storage data subset corresponding to the ith service sub-database, wherein I is a positive integer smaller than I;

determining an ith data subset corresponding to the ith service sub-database according to the ith analysis data subset and the ith storage data subset corresponding to the ith service sub-database, and enabling i=i+1;

and in the case that the preset condition is met, determining a data set related to the first database according to the I data subsets.

5. The method of claim 4, wherein the determining the ith analysis data subset and the ith storage data subset corresponding to the ith service sub-database in the determining that the ith service sub-database in the set of service sub-databases exists comprises:

determining a database object type corresponding to an ith service sub-database, the number of database objects corresponding to the database object type and the ith creation information of the ith sub-data to be migrated corresponding to the ith service sub-database under the condition that the ith service sub-database in the service sub-database set is determined to exist;

Determining an ith analysis data subset related to the ith service sub-database according to a database object type corresponding to the ith service sub-database, the number of database objects corresponding to the database object type and the ith creation information of the ith sub-data to be migrated corresponding to the ith service sub-database;

determining an ith data set associated with the first database based on the ith analysis data subset and the ith storage data subset associated with the ith business sub-database.

6. A method according to any one of claims 1 to 3, wherein said determining a migration result of said data to be migrated from said migrated data in said second database and said analysis dataset comprises:

and determining a migration result of the data to be migrated according to the migration log and the analysis data set.

7. A method according to any one of claims 1 to 3, further comprising:

uploading the data set into a storage space associated with the second database;

wherein the migration of the data to be migrated in the storage data set to a second database according to the creation information further includes:

8. A data processing apparatus comprising:

an acquisition module, configured to acquire a data set related to the first database by using the migration path, where the data set includes an analysis data set and a storage data set, the analysis data set represents an analysis result of data to be migrated in the first database, and the storage data set represents a set of data to be migrated in the first database;

The migration module is used for migrating the data to be migrated in the storage data set to a second database according to the creation information; and

9. An electronic device, comprising:

one or more processors;

a memory for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-7.

10. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-7.

11. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.