CN115982133A

CN115982133A - Data processing method and device

Info

Publication number: CN115982133A
Application number: CN202310049417.6A
Authority: CN
Inventors: 高煜东
Original assignee: Petal Cloud Technology Co Ltd
Current assignee: Petal Cloud Technology Co Ltd
Priority date: 2023-02-01
Filing date: 2023-02-01
Publication date: 2023-04-18

Abstract

The application discloses a data processing method and a data processing device, which relate to the field of communication; the method comprises the following steps: receiving metadata information from a message queue server, wherein the metadata information is a data identifier corresponding to first incremental data which is newly added when an application server executes a service; under the condition that the metadata information exists in the preset data conversion configuration information, first incremental data corresponding to the metadata information are obtained from a server of a source database; converting the first incremental data into second incremental data of a server suitable for at least one type of target database according to preset data conversion configuration information, wherein the source database and the target database are heterogeneous databases; and second incremental data corresponding to the server of the target database is sent to the server of at least one type of target database, so that the success of data migration among different types of databases can be guaranteed.

Description

Data processing method and device

Technical Field

The present application relates to the field of communications, and in particular, to a data processing method and apparatus.

Background

With the continuous development of databases, many enterprises provide cloud services. Cloud services refer to services that place infrastructures such as various databases (e.g., relational, non-relational, newSQL) on a cloud server, allowing users to apply for use of the databases in an individual or organization form. Compared with the method that users build the database more quickly, the database is higher in safety and stability, the database can be applied according to needs along with the service scale, and different users can share the database, so that the cost is saved.

When the database is arranged on the cloud server, the migration of the data in the database is the most important thing. The existing scheme for exporting data from a source database and importing the data into a target database provided by a database engine is limited by the type difference of the databases, and migration failure is easily caused.

Disclosure of Invention

The application provides a data processing method and device, which can guarantee successful data migration among different types of databases.

In order to achieve the purpose, the technical scheme is as follows:

in a first aspect, a data processing method is provided, where the method is applied to a first device, and includes: receiving metadata information from a message queue server, wherein the metadata information is a data identifier corresponding to first incremental data which is newly added when an application server executes a service; under the condition that the metadata information exists in the preset data conversion configuration information, first incremental data corresponding to the metadata information are obtained from a server of a source database; presetting data conversion configuration information for data conversion between a server of a source database and a server of at least one type of target database; converting the first incremental data into second incremental data of a server suitable for at least one type of target database according to preset data conversion configuration information, wherein the source database and the target database are heterogeneous databases; and sending second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

In the first aspect, the message queue server records data identification information metadata information corresponding to newly added first incremental data when the application server executes a service, and based on the metadata information, it can be determined which first incremental data need to be migrated, so that data omission is not caused even if online data migration is performed, data consistency between the metadata database and the target database can be ensured, and online data migration can be further achieved. In addition, because online migration can be performed, compared with a data migration scheme in which data is migrated after being accumulated to a certain degree at present, how much first incremental data is generated, how much first incremental data is migrated without accumulation, and the data migration volume during data migration is differentiated, so that the data volume of data to be migrated in unit time is small, the import and export time is also short, the probability of migration failure caused by network jitter or other abnormalities is reduced, and the success rate of data migration is further improved. And the preset data conversion configuration information is used for converting the first incremental data field and the second incremental data field between the source database and the target database, so that the aim of migrating the data in the source database to a plurality of target databases can be achieved, and the success of heterogeneous data migration is guaranteed (the source database and the target database are heterogeneous databases).

In one possible implementation, the method further comprises:

sending a first request for requesting preset data conversion configuration information to second equipment; and receiving preset data conversion configuration information from the second equipment.

In the implementation, the first device sends a first request to the second device to request the preset data conversion configuration information, and the second device sends the preset data conversion configuration information to the first device after receiving the first request, so that the preset data conversion configuration information can be provided for the first device in time.

In one possible implementation, the method further comprises:

and calling preset data conversion configuration information from the local cache.

In the implementation, the first device calls the preset data conversion configuration information from the local cache, and the efficiency of calling the preset data conversion configuration information is high.

In one possible implementation, prior to receiving the metadata information from the message queue server, the method further comprises:

acquiring first stock data stored by a server of a source database from the server of the source database; converting the first stock data into second stock data of the server suitable for at least one type of target database according to preset data conversion configuration information; and sending second stock data corresponding to the server of the target database to the server of the at least one type of target database.

In the implementation, the first device acquires the first stock data from the server of the source database, converts the first stock data into second stock data of the server suitable for the target database according to the preset data conversion configuration information, and sends the second stock data to the server of the target database, so that migration of the first stock data in the source database is realized. Because the preset data conversion configuration information may include the conversion relationship between the first field and the second field between the source database and the multiple types of target databases, the purpose of migrating stock data in one source database to multiple target databases and the purpose of migrating heterogeneous data (the source database and the target databases are heterogeneous databases) can be achieved, and the success of data migration is further guaranteed. And moreover, the migration of the first stock data is performed firstly, and then the migration of the first incremental data is performed, so that the data stored by the server of the source database can be ensured to be consistent with the data stored by the server of the target database.

In one possible implementation, the preset data conversion configuration information includes a conversion relationship between a first field corresponding to data in the source database and a second field corresponding to data in the target database; the information represented by the first field and the second field has the same meaning, and the field structures are different; converting the first incremental data into second incremental data suitable for a server of at least one type of target database according to preset data conversion configuration information, wherein the method comprises the following steps: and converting the first field in the first incremental data into a second field according to the conversion relation in the preset data conversion configuration information to obtain second incremental data of the server suitable for at least one type of target database.

In this implementation, the preset data conversion configuration information may include a conversion relationship between a first field and a second field between the source database and the multiple types of target databases, so as to achieve a purpose of migrating data in one source database to multiple target databases and a purpose of migrating heterogeneous data.

In one possible implementation, the second device includes one second device or a plurality of second devices, and one second device corresponds to one target database; or, one second device corresponds to a plurality of target databases.

In this implementation, one or more second devices are designed to be able to provide preset data conversion configuration information to the first device.

In a second aspect, a data processing method is provided, where the method is applied to an application server, and includes:

under the condition that first incremental data are generated in the service execution process, sending the first incremental data to a server of a source database and sending metadata information corresponding to the first incremental data to a message queue server; wherein the metadata information is for: under the condition that the metadata information exists in the preset data conversion configuration information, after the first device acquires the metadata information from the message queue server, the first device converts the first incremental data into second incremental data suitable for the server of the at least one type of target database according to the preset data conversion configuration information, and sends the second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

In the second aspect, the application server sends the metadata information of the data identification information corresponding to the newly added first incremental data when the application server executes the service to the message queue server, and based on the metadata information, it can be determined which first incremental data need to be migrated, so that data omission is not caused even if online data migration is performed on the data, the data consistency of the metadata database and the target database can be ensured, and further online data migration can be realized. In addition, because online migration can be performed, compared with a data migration scheme in which data is migrated after being accumulated to a certain degree at present, how much first incremental data is generated, how much first incremental data is migrated without accumulation, and the data migration volume during data migration is differentiated, so that the data volume of data to be migrated in unit time is small, the import and export time is also short, the probability of migration failure caused by network jitter or other abnormalities is reduced, and the success rate of data migration is further improved. And the preset data conversion configuration information is used for converting the first incremental data field and the second incremental data field between the source database and the target database, so that the aim of migrating the data in the source database to a plurality of target databases can be achieved, and the success of heterogeneous data migration is guaranteed (the source database and the target database are heterogeneous databases).

In one possible implementation, the method further comprises:

under the condition that a server of a source database fails, sending a data acquisition request to a server of any one target database; the data acquisition request is used for requesting second incremental data; second delta data is received from the server of the target database.

In this implementation, in the case of a failure of the server of the source database, the application server sends an inventory data acquisition request to the server of any one of the target databases to acquire the second inventory data, so as to ensure that the service of the application server can be normally executed.

In a third aspect, a data processing method is provided, where the method is applied to a message queue server, and includes:

receiving metadata information from an application server, wherein the metadata information is a data identifier corresponding to first incremental data which is newly added when the application server executes a service; transmitting metadata information to the first device; wherein the metadata information is for: and under the condition that the metadata information exists in the preset data conversion configuration information, the first device converts the first incremental data into second incremental data suitable for the servers of the at least one type of target database according to the preset data conversion configuration information, and sends the second incremental data corresponding to the servers of the target database to the servers of the at least one type of target database.

In the third aspect, the message queue server receives the metadata information of the data identification information corresponding to the newly added first incremental data when the application server executes the service, and based on the metadata information, it can determine which first incremental data to be migrated are, so that data omission is not caused even if online data migration is performed on the data, the data consistency between the metadata database and the target database can be ensured, and further online data migration can be realized. In addition, because online migration can be performed, compared with a data migration scheme in which data is migrated after being accumulated to a certain degree at present, how much first incremental data is generated, how much first incremental data is migrated without accumulation, and the data migration volume during data migration is differentiated, so that the data volume of data to be migrated in unit time is small, the import and export time is also short, the probability of migration failure caused by network jitter or other abnormalities is reduced, and the success rate of data migration is further improved. Moreover, the conversion of the first incremental data field and the second incremental data between the source database and the target database is performed through the preset data conversion configuration information, so that the purpose of migrating data in one source database to a plurality of target databases can be achieved, and the success of heterogeneous data migration is guaranteed (the source database and the target database are heterogeneous databases).

In one possible implementation, the first device includes one first device or a plurality of first devices, and one first device corresponds to one target database; or, one first device corresponds to a plurality of target databases.

In the implementation, a plurality of first devices are designed to simultaneously execute the migration of the first incremental data, so that the data migration efficiency can be improved.

In a fourth aspect, a data processing apparatus is provided, which may be a first device or a chip or a system on a chip in the first device. The data processing apparatus may implement the functions performed by the first device in the first aspect or the possible implementations of the first aspect, and the functions may be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The data processing apparatus includes: the receiving and sending module is used for receiving metadata information from the message queue server, wherein the metadata information is a data identifier corresponding to first incremental data which is newly added when the application server executes the service; the receiving and sending module is further used for acquiring first incremental data corresponding to the metadata information from a server of the source database under the condition that the metadata information exists in the preset data conversion configuration information; presetting data conversion configuration information for data conversion between a server of a source database and a server of at least one type of target database; the processing module is used for converting the first incremental data into second incremental data of a server suitable for at least one type of target database according to preset data conversion configuration information, and the source database and the target database are heterogeneous databases; and the transceiver module is further used for sending second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

In one possible implementation, the transceiver module is further configured to: sending a first request to second equipment, wherein the first request is used for requesting preset data conversion configuration information; and receiving preset data conversion configuration information from the second device.

In one possible implementation, the processing module is further configured to: and calling preset data conversion configuration information from the local cache.

In a possible implementation, the transceiver module is further configured to, before receiving the metadata information from the message queue server, obtain, from the server of the source database, first stock data stored by the server of the source database; the processing module is also used for converting the first stock data into second stock data of the server suitable for at least one type of target database according to the preset data conversion configuration information; and the transceiver module is also used for transmitting second stock data corresponding to the server of the target database to the server of the at least one type of target database.

In one possible implementation, the preset data conversion configuration information includes a conversion relationship between a first field corresponding to data in the source database and a second field corresponding to data in the target database; the information represented by the first field and the second field has the same meaning, and the field structures are different; and the processing module is specifically used for converting the first field in the first incremental data into the second field according to the conversion relation in the preset data conversion configuration information to obtain second incremental data of the server suitable for at least one type of target database.

In a fifth aspect, a data processing apparatus is provided, which may be an application server or a chip or a system on chip in an application server. The data processing apparatus may implement the functions performed by the application server in the second aspect or the possible implementation of the second aspect, and the functions may be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The data processing apparatus includes: the receiving and sending module is used for sending the first incremental data to a server of a source database and sending metadata information corresponding to the first incremental data to the message queue server under the condition that the first incremental data is generated in the service execution process; wherein the metadata information is for: under the condition that the metadata information exists in the preset data conversion configuration information, after the first device acquires the metadata information from the message queue server, the first device converts the first incremental data into second incremental data suitable for the server of the at least one type of target database according to the preset data conversion configuration information, and sends the second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

In one possible implementation, the transceiver module is further configured to: under the condition that a server of a source database fails, sending a data acquisition request to a server of any one target database; the data acquisition request is used for requesting second incremental data; and receiving second incremental data from the server of the target database.

In a sixth aspect, a data processing apparatus is provided, which may be a message queue server or a chip or a system on chip in a message queue server. The data processing apparatus may implement the functions performed by the message queue server in the first aspect or the possible implementations of the first aspect, and the functions may be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The data processing apparatus includes: the receiving and sending module is used for receiving metadata information from the application server, wherein the metadata information is a data identifier corresponding to first incremental data which is newly added when the application server executes a service; the receiving and sending module is also used for sending metadata information to the first equipment; wherein the metadata information is for: and under the condition that the metadata information exists in the preset data conversion configuration information, the first device converts the first incremental data into second incremental data suitable for the servers of the at least one type of target database according to the preset data conversion configuration information, and sends the second incremental data corresponding to the servers of the target database to the servers of the at least one type of target database.

In a seventh aspect, a data processing apparatus is provided, where the data processing apparatus includes a processor and a transceiver, and the processor and the transceiver are configured to support the data processing apparatus to perform the methods of the first aspect, the second aspect, and the third aspect.

In an eighth aspect, a computer-readable storage medium is provided, which stores computer instructions that, when executed, perform the method of the first, second and third aspects.

In a ninth aspect, there is provided a data processing system comprising: the system comprises an application server, a message queue server, a first device and a second device; the application server is configured to: under the condition that first incremental data are generated in the service execution process, sending the first incremental data newly added when the application server executes the service to a server of a source database, and sending metadata information corresponding to the first incremental data to a message queue server; the metadata information is a data identifier corresponding to newly added first incremental data when the application server executes the service; a message queue server to: receiving metadata information from an application server; and transmitting the metadata information to the first device; a first device to: receiving metadata information from a message queue server; under the condition that the metadata information exists in the preset data conversion configuration information, first incremental data corresponding to the metadata information are obtained from a server of a source database; presetting data conversion configuration information for data conversion between a server of a source database and a server of at least one type of target database; presetting data conversion configuration information for sending to the first equipment by the second equipment; converting the first incremental data into second incremental data of a server suitable for at least one type of target database according to preset data conversion configuration information, wherein the source database and the target database are heterogeneous databases; and sending second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

In a tenth aspect, the present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of the first, second and third aspects described above.

In an eleventh aspect, the present application provides a chip comprising a processor and a transceiver for enabling a data processing apparatus to perform the methods of the first, second and third aspects.

In this application, the beneficial effects described in the fourth aspect to the eleventh aspect may be analyzed with reference to the beneficial effects of the first aspect to the third aspect, and are not described herein again.

Drawings

Fig. 1 is a first schematic structural diagram of a data processing system according to an embodiment of the present application;

fig. 2 is a schematic structural diagram of a data processing system according to an embodiment of the present application;

fig. 3 is a first flowchart illustrating a data processing method according to an embodiment of the present application;

fig. 4 is a schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 5 is a schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 6 is a schematic flowchart of a data processing method according to a fourth embodiment of the present disclosure;

fig. 7 is a schematic flowchart of a data processing method according to an embodiment of the present application;

fig. 8 is a flowchart illustrating a sixth data processing method according to an embodiment of the present application;

fig. 9 is a seventh flowchart illustrating a data processing method according to an embodiment of the present application;

fig. 10 is a first schematic structural diagram of a data processing apparatus according to an embodiment of the present application;

fig. 11 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application.

Detailed Description

The network architecture and the service scenario described in the embodiment of the present application are for more clearly illustrating the technical solution of the embodiment of the present application, and do not form a limitation on the technical solution provided in the embodiment of the present application, and it can be known by a person of ordinary skill in the art that the technical solution provided in the embodiment of the present application is also applicable to similar technical problems with the evolution of the network architecture and the occurrence of a new service scenario.

The terms referred to in the present application will first be explained below:

heterogeneous databases (heterogeneoous databank): database management systems (DBMS) different databases. For example, the types of the dbms may be the same, and the specific dbms model may be different. For example, oracle and SQL Server, which are both relational database systems, are heterogeneous databases. Or databases with different types of database management systems, such as Oracle in a relational database system and Cassandra in a non-relational database (NoSQL) system, which are also heterogeneous databases. Alternatively, it can be a database of different data models, such as relational, schema, hierarchical, network, object-oriented, different databases. In the embodiment of the application, the source database and the target database are heterogeneous databases.

Database Migration (Database Migration): migrating data in a source database to a target database; the online migration of the database comprises two processes of stock data migration and incremental data migration.

Stock Data Migration (Full Data Migration): by the time a data migration task begins, the full amount of data already in the source database.

Incremental Data Migration (Incremental Data Migration): after a data migration task begins, the program modifies (including newly writing, updating, deleting, etc.) the data in the source database.

Online Migration (Zero-Downtime Database Migration): namely, the writing of new data corresponding to the service is not stopped, and the service can be continuously provided to the outside in the process of database migration.

Topic/Topic (Topic): in the message queue, different message pipelines are distinguished.

In the current data migration scheme, after data stored in a source database is accumulated to a certain extent by a export command (e.g., dump, copy) of the source database, the data in the source database is exported to a disk of a server and stored as a file (e.g., in a csv format, in an sql format, etc.). And then, the data in the file is imported into the target database by executing an import command (for example, source) of the Source database. In the process of exporting and importing data, in order to ensure the data consistency between the source database and the target database, the source database must stop writing the new database, and must stop serving the external database, so that online migration cannot be achieved. Moreover, in the current data migration scheme, migration is usually completed one-to-one, that is, one source database is migrated to one target database, and most of the migration schemes are homogeneous database migration and cannot be applied to scenarios of heterogeneous data migration and one-to-many database migration.

In addition, if the migrated data has some special characters, it is easy to fail in importing the target database. If the migrated data volume is large (for example, in a non-relational database with a large part of data storage volume, there may be tens of T or even hundreds of T data), the import and export time of the data is also long, and once there is network jitter or other abnormality, the data needs to be restarted, which easily results in data migration failure. If the source database is heterogeneous with the target database, the export file of the source database may not be directly imported into the target database, and heterogeneous migration cannot be completed.

As can be seen, current data migration schemes are prone to data migration failures.

In order to solve the above technical problem, an embodiment of the present application provides a data processing method, and the method provided by the embodiment of the present application is described below with reference to the accompanying drawings of the specification.

The data processing method provided by the embodiment of the present application is described below by taking the data processing system shown in fig. 1 as an example.

Fig. 1 is a first schematic diagram of a data processing system according to an embodiment of the present application, and as shown in fig. 1, the data processing system may include the following four nodes:

the device comprises an application server, a message queue server, a first device and a second device. The first device and the second device may be computers, servers or other electronic devices having data processing capabilities. The server may be a single server, or may be a server cluster formed by a plurality of servers. In some embodiments, the server cluster may also be a distributed cluster.

Specifically, the application server establishes a communication connection with the server of the source database, and the application server also establishes a communication connection with the message queue server. Based on the communication connection, the application server sends the first incremental data to the server of the source database and sends the metadata information corresponding to the first incremental data to the message queue server under the condition that the application server generates the first incremental data in the service execution process. The application server may install a DB Migrate Software Development Kit (SDK) and perform the above steps through the DB Migrate SDK.

The message queue server establishes a communication connection with the application server and also with the first device. The message queue server is provided with a message queue, the message queue belongs to a component with short-time message storage, the message queue has the characteristic of order in the same partition, and received data are managed in order through Topic. Illustratively, for a scenario where the number of first devices is multiple, one Topic corresponds to one first device. Taking the example of two first devices, the message queue may maintain a pair of Topic a and a pair of Topic B, where one first device is subscribed to Topic a and the other device is subscribed to Topic B. The message queue, which may also be referred to as message middleware, receives the metadata information from the application server and sends the metadata information to the first device. In the embodiment of the application, only the metadata information corresponding to the first incremental data is written into the message queue server, so that the storage capacity of the message queue is prevented from being insufficient due to the fact that all the first incremental data are written.

The first device establishes communication connection with the server of the source database, establishes communication connection with the server of the target database, and establishes communication connection with the message queue server. Receiving metadata information from the message queue server based on the communication connection, and acquiring first incremental data corresponding to the metadata information from a server of the source database when the metadata information exists in preset data conversion configuration information (acquired from the second device), for example, acquiring the first incremental data in a SELECT query manner in a Distributed Resource Scheduler (DRS); then converting the first incremental data into second incremental data suitable for a server of at least one type of target database according to preset data conversion configuration information; and sending second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

And the second equipment is configured with preset data conversion configuration information between the heterogeneous databases, and the preset data conversion configuration information records the conversion relationship between fields with the same meaning but different field structures between the heterogeneous databases. And providing the preset data conversion configuration information to the first device. The first device and the second device may be integrated in the same electronic device, or may be respectively disposed in different electronic devices.

Further, the number of the first devices may be one or more. One first device may correspond to one target database, and may also correspond to a plurality of target databases; the number of the second devices may be one or more. One second device may correspond to one target database, and may also correspond to a plurality of target databases. As shown in fig. 2, a second schematic diagram of the data processing system in a scenario where the first device and the second device are different electronic devices is shown, where one first device corresponds to one target database and is provided with a plurality of target databases in the scenario.

The scheme provided by the embodiment of the application is mainly introduced from the perspective of system composition. The scheme provided by the embodiment of the present application is described below from the perspective of interaction between various nodes in a data processing system.

Fig. 3 shows a first flowchart of a data processing method according to an embodiment of the present application. As shown in fig. 3, the method may include the steps of:

s301, under the condition that the application server generates the first incremental data in the service execution process, the application server sends the first incremental data newly added when the application server executes the service to the server of the source database. Accordingly, the server of the source database receives the first incremental data.

In reference to the description of the data processing system in the embodiment of the present application, the application server establishes a communication connection with the server of the source database. Based on the communication connection, the application server and the source database may transmit the first incremental data. The first incremental data is data generated by the application server on line in the service execution process. For example, the first incremental data may include four fields, i.e., employee id, employee name, employee tag, employee _ label, and employee gender, as shown in table 1.

TABLE 1

S302, the application server sends the metadata information corresponding to the first incremental data to the message queue server. Accordingly, the message queue server receives the metadata information.

In reference to the description of the data processing system in the embodiment of the present application, the application server establishes a communication connection with the message queue server. Based on the communication connection, the application server and the message queue server may transmit metadata information corresponding to the first incremental data. For example, the metadata information is a data identifier corresponding to first incremental data added when the application server executes the service, and has a function of indicating the first incremental data. For example, the metadata information may be a set of information including a library name (database), a table name (table), an operation (operation), and a primary key (primary) corresponding to the first incremental data. Specifically, the metadata information may be corresponding to the first incremental data shown in table 1: library name: "company _ system", "table name": applied layer "," operation ": INSERT", "primary key": 1025".

S303, the message queue server sends the metadata information to the first device. Accordingly, the first device receives metadata information from the message queue server.

The first device establishes communication connection with the message queue server and can transmit metadata information. The mode in which the message queue server sends the metadata information to the first device can be flexibly set. For example, the mode in which the message queue server sends the metadata information to the first device may be a subscription mode, i.e., once the message queue server receives the metadata information, the metadata information is sent to the first device. Or, when the message queue server receives the metadata information and meets a certain condition (for example, the information amount of the accumulated metadata information reaches a certain threshold or the time length of the accumulated metadata information reaches a certain threshold), the message queue server sends the metadata information to the first device.

S304, the first device acquires first incremental data corresponding to the metadata information from the server of the source database under the condition that the metadata information exists in the preset data conversion configuration information.

And communication connection is established between the first equipment and the server of the source database. Based on the communication connection, the first device can interact with the server of the source database for the first incremental data.

The preset data conversion configuration information is used for data conversion between the server of the source database and the server of at least one type of target database. For example, the preset data conversion configuration information includes a conversion relationship between a first field corresponding to data in the source database and a second field corresponding to data in at least one type of target database, where the first field and the second field have the same information meaning and different field structures. The data model of the database includes field structures in the database. For a source database, the data model includes a first field; for the target database, the data model includes a second field. For example, taking the source database as MySQL and the target database as Cassandra as an example, the preset data transformation configuration information may be as shown in table 2. The first field may be company system and the corresponding second field is user system; the first field may be employee, and the corresponding second field is user; the first field may be an employee id, the corresponding second field a user id, etc.

The first field can comprise special characters, and data migration failure caused by the special characters can be overcome by converting the first field into the second field, so that data migration success is guaranteed.

TABLE 2

In another example, where the source database is MySQL and the target databases are two, cassandra and openGauss respectively, the preset data transformation configuration information can be as shown in table 3.

TABLE 3

S305, the first device converts the first incremental data into second incremental data suitable for the server of at least one type of target database according to the preset data conversion configuration information.

The conversion of the first incremental data into the second incremental data of the server suitable for at least several types of target databases depends on the data migration scenario applied in the embodiment of the present application. Illustratively, if the data of a source database needs to be converted into a plurality of target databases in a data migration scenario, converting the first incremental data into second incremental data suitable for servers of multiple types of target databases; if the data of the source database needs to be converted into one target database in the data migration scene, the first incremental data is converted into second incremental data of a server suitable for one type of target database.

Specifically, S305 may include: and converting the first field in the first incremental data into a second field according to the conversion relation in the preset data conversion configuration information to obtain second incremental data of the server suitable for at least one type of target database.

For example, for the first incremental data shown in table 1, the target database is Cassandra, and the transformed second incremental data may include: user (user _ id, user _ name, user _ label, user _ generator) VALUES ('1025', 'wangwu', 'mad cool dazzle', 'male').

S306, the first device sends second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

A communication connection is established between the first device and the server of the target database, and based on the communication connection, after the second incremental data is converted in S305, the second incremental data can be sent to the server of the corresponding target database. And the server of the target database receives the second incremental data, namely, the second incremental data can be read to realize storage.

In the embodiment of the application, when the application server generates the first incremental data in the service execution process, the application server sends the first incremental data to the server of the source database, and sends metadata information corresponding to the first incremental data to the message queue server. The message queue server then sends the metadata information to the first device. And the first device acquires first incremental data corresponding to the metadata information from the server of the source database under the condition that the metadata information exists in the preset data conversion configuration information. And the first equipment converts the first incremental data into second incremental data suitable for the server of at least one type of target database according to the preset data conversion configuration information. And finally, the first equipment sends second incremental data corresponding to the server of the target database to the servers of at least one type of target database, so that the purposes of heterogeneous database migration, source database splitting, source database backup and the like are achieved.

The message queue server records data identification information metadata information corresponding to newly added first incremental data when the application server executes a service, and based on the metadata information, the first incremental data to be migrated can be determined, so that data omission can not be caused even if online data migration is performed, the data consistency of a metadata database and a target database can be ensured, and the online data migration can be further realized. In addition, because online migration can be performed, compared with a data migration scheme in which data is migrated after being accumulated to a certain degree at present, how much first incremental data is generated, how much first incremental data is migrated without accumulation, and the data migration volume during data migration is differentiated, so that the data volume of data to be migrated in unit time is small, the import and export time is also short, the probability of migration failure caused by network jitter or other abnormalities is reduced, and the success rate of data migration is further improved. Moreover, the preset data conversion configuration information may include a conversion relationship between a first field and a second field between the source database and the multiple types of target databases, so that the purpose of migrating data in one source database to multiple target databases and the purpose of migrating heterogeneous data (the source database and the target databases are heterogeneous databases) can be achieved. Moreover, the conversion relationship between the first field and the second field can also include the conversion relationship of the special characters, so that the situation that the target database cannot read the special characters is avoided, and the successful data migration is further ensured.

In an embodiment, the number of the first devices may be set to be multiple, as shown in fig. 4, fig. 4 shows a flowchart of a second method for processing data provided in the embodiment of the present application, and then the multiple first devices respectively execute S303 to S306, and the specific process may refer to the description in fig. 3 and is not described again.

In the embodiment of the application, a plurality of first devices are designed to simultaneously execute the migration of the first incremental data, so that the data migration efficiency can be improved.

In S305, a description is given below of a way in which the first device may obtain the preset data conversion configuration information:

in an embodiment, the first device locally caches preset data conversion configuration information, as shown in fig. 5, where fig. 5 shows a third flowchart of a data processing method provided in this embodiment, and the method may further include:

and S510, calling preset data conversion configuration information from a local cache by the first equipment.

In the embodiment of the application, the first device calls the preset data conversion configuration information from the local cache, and the efficiency of calling the preset data conversion configuration information is high.

In an embodiment, the first device does not have preset data conversion configuration information in the local cache, as shown in fig. 6, fig. 6 shows a fourth flowchart of the data processing method provided in the embodiment of the present application, and the method may further include:

s610, the first device sends a first request to the second device, and correspondingly, the second device receives the first request.

The first device and the second device can transmit a first request based on the communication connection, wherein the first request is used for requesting preset data conversion configuration information.

S620, the second device sends the configuration information of the preset data conversion to the first device in response to the first request, and correspondingly, the first device receives the configuration information of the preset data conversion from the second device.

After receiving the preset data conversion configuration information, the first device may store the preset data conversion configuration information in a local cache, so as to apply the preset data conversion configuration information in the application scenario corresponding to fig. 4.

In the embodiment of the application, the first device sends the first request to the second device to request the preset data conversion configuration information, and the second device responds to the first request after receiving the first request and sends the preset data conversion configuration information to the first device, so that the preset data conversion configuration information can be provided for the first device in time.

After S306 is completed, the second incremental data in the server of the target database may be used in scenarios such as database disaster recovery, heterogeneous database recovery, and the like, and accordingly, in an embodiment, as shown in fig. 7, fig. 7 shows a fifth flowchart of the data processing method provided in the embodiment of the present application, and the method may further include:

s710, the application server sends a data acquisition request to a server of any target database under the condition that the server of the source database fails. Correspondingly, the server of the target database receives the data acquisition request.

And the data acquisition request is used for requesting second incremental data.

S720, the server of the target database receives and responds to the data acquisition request, second incremental data is sent to the application server, and correspondingly, the application server receives the second incremental data from the server of the target database.

After receiving the second incremental data, the application server may use the second incremental data in scenarios such as database disaster tolerance, heterogeneous database recovery, and the like.

In the embodiment of the application, under the condition that the server of the source database fails, the application server sends a data acquisition request to the server of any one target database to acquire the second incremental data, so that the service of the application server can be normally executed.

The migration process of the first incremental data generated by the application server during the service execution process from the source database to the target database is described above. In the actual application, the source database further stores the first stock data generated by the application server in the service history execution process, and the migration process of the first stock data is described below.

In an embodiment, as shown in fig. 8, fig. 8 is a flowchart illustrating a sixth method for processing data provided in an embodiment of the present application, before receiving metadata information from a message queue server, the method may further include:

s810, the first device acquires the first stock data stored by the server of the source database from the server of the source database.

The first device may send an inventory data request to a server of the source database to obtain the first inventory data. For example, as shown in table 4, the source database includes two pieces of first stock data:

TABLE 4

S820, the first device converts the first stock quantity data into second stock quantity data of the server suitable for at least one type of target database according to preset data conversion configuration information.

With reference to the introduction of S304, the preset data conversion configuration information includes a conversion relationship between a first field corresponding to data in the source database and a second field corresponding to data in at least one type of target database. The first inventory data can be converted into second inventory data of the server suitable for at least one type of target database based on the preset data conversion configuration information.

For example, for the first inventory data shown in table 4, the converted second inventory data includes:

user _ id, user _ name, user _ label, user _ generator) VALUES ('1023', 'zhang', 'high big power', 'male');

user (user _ id, user _ name, user _ label, user _ generator) VALUES ('1024', 'lie four', 'engjun, unrestrained', 'man').

In another example, for the first inventory data shown in table 4, the converted second inventory data may further include:

station _ id, station _ name, station _ generator) VALUES ('1023', 'zhang', 'male');

stab _ int _ organization _ system _ staff (staff _ id, staff _ name, staff _ generator) VALUES ('1024', 'lie four', 'engjun, unrestrained', 'male').

S830, the first device sends second stock data corresponding to the server of the target database to the server of the at least one type of target database.

In the embodiment of the application, the first device acquires the first stock data from the server of the source database, converts the first stock data into the second stock data of the server suitable for the target database according to the preset data conversion configuration information, and sends the second stock data to the server of the target database, so that migration of the first stock data in the source database is realized. Because the preset data conversion configuration information can include the conversion relationship between the first field and the second field between the source database and the multiple types of target databases, the purpose of migrating the stock data in one source database to multiple target databases and the purpose of migrating heterogeneous data (the source database and the target databases are heterogeneous databases) can be achieved, and the success of data migration is guaranteed. Moreover, the migration of the first stock data is performed first, and then the migration of the first incremental data is performed, so that the data stored by the server of the source database and the data stored by the server of the target database can be ensured to be consistent.

In an embodiment, after S830 is completed, the second inventory data in the server of the target database may be used in scenarios such as database disaster recovery, heterogeneous database recovery, and the like, and accordingly, in an embodiment, as shown in fig. 9, fig. 9 shows a seventh flowchart of the data processing method provided in the embodiment of the present application, and the method may further include:

s910, the application server sends a stock data acquisition request to a server of any one target database under the condition that the server of the source database fails. Correspondingly, the server of the target database receives the stock data acquisition request.

Wherein the stock data acquisition request is for requesting second stock data.

And S920, the server of the target database receives and responds to the stock data acquisition request, and sends second stock data to the application server, and correspondingly, the application server receives the second stock data from the server of the target database.

After receiving the second inventory data, the application server may use the second inventory data in scenarios such as database disaster recovery, heterogeneous database recovery, and the like.

In the embodiment of the application, under the condition that the server of the source database fails, the application server sends the stock data acquisition request to the server of any one target database to acquire the second stock data, so that the service of the application server can be normally executed.

The above-mentioned scheme provided by the embodiments of the present application is mainly introduced from the perspective of interaction between the nodes. It will be appreciated that each node, in order to carry out the above-described functions, comprises corresponding hardware structures and/or software modules for performing the respective functions. Those skilled in the art will readily appreciate that the methods of the embodiments of the present application can be implemented in hardware, software, or a combination of hardware and computer software, in conjunction with the exemplary algorithm steps described in connection with the embodiments disclosed herein. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiment of the present application, functional modules may be divided for each node according to the above method example, for example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. It should be noted that, in the embodiment of the present application, the division of the module is schematic, and is only one logic function division, and another division manner may be available in actual implementation.

In a specific implementation, each node shown in the present application is as follows: the first device may adopt the composition shown in fig. 10 or include the components shown in fig. 10. Fig. 10 is a first schematic structural diagram of a data processing apparatus according to an embodiment of the present application, where the data processing apparatus has a function of the first device according to the embodiment of the present application, the data processing apparatus may be the first device or a chip or a system on chip in the first device.

As shown in fig. 10, the data processing apparatus may include a processor 101, a communication line 102, a transceiver 103, and a memory 104. The processor 101, the memory 104 and the transceiver 103 may be connected via a communication line 102. In one example, processor 101 may include one or more CPUs, such as CPU0 and CPU1 in fig. 10.

As an alternative implementation, the data processing apparatus comprises a plurality of processors, for example, a processor 107 may be included in addition to the processor 101 in fig. 10.

The processor 101 may be a Central Processing Unit (CPU), a general purpose processor Network (NP), a Digital Signal Processor (DSP), a microprocessor, a microcontroller, a Programmable Logic Device (PLD), or any combination thereof. The processor 101 may also be other devices with processing functions, such as a circuit, a device, a software module, or the like.

A communication line 102 for transmitting information between the respective components included in the data processing apparatus.

A transceiver 103 for communicating with other devices or other communication networks. The other communication network may be an ethernet, a Radio Access Network (RAN), a Wireless Local Area Network (WLAN), or the like. The transceiver 103 may be an interface circuit, a pin, a radio frequency module, a transceiver, or any device capable of enabling communication.

Further, the data processing apparatus may further include a memory 104. A memory 104 for storing instructions. Wherein the instructions may be a computer program.

The memory 104 may be a read-only memory (ROM) or other types of static storage devices that can store static information and/or instructions, a Random Access Memory (RAM) or other types of dynamic storage devices that can store information and/or instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD ROM), or other optical disc storage, magnetic disc storage media or other magnetic storage devices, and the optical disc storage includes a compact disc, a laser disc, an optical disc, a digital versatile disc, a blu-ray disc, and the like.

It should be noted that the memory 104 may exist independently from the processor 101, or may be integrated with the processor 101. The memory 104 may be used for storing instructions or program code or some data or the like. The memory 104 may be located inside the data processing apparatus or outside the data processing apparatus, without limitation. The processor 101, when executing the instructions stored in the memory 104, may implement the methods provided by the embodiments of the present application.

As an alternative implementation, the data processing apparatus further comprises an output device 105 and an input device 106. Illustratively, the input device 106 is a keyboard, mouse, microphone, joystick, or the like, and the output device 105 is a display screen, speaker (microphone), or the like.

It should be noted that the data processing apparatus may be a desktop computer, a portable computer, a network server, a mobile phone, a tablet computer, a wireless terminal, an embedded device, a chip system or a device with a similar structure as that in fig. 10. Further, the constituent configuration shown in fig. 10 does not constitute a limitation of the data processing apparatus, and the data processing apparatus may include more or less components than those shown in fig. 10, or combine some components, or a different arrangement of components, in addition to the components shown in fig. 10.

In the embodiment of the present application, the chip system may be formed by a chip, and may also include a chip and other discrete devices.

Fig. 11 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application, where the data processing apparatus is applied to a first device. Each module in the apparatus shown in fig. 11 has the function of implementing the corresponding steps in fig. 3-9, and can achieve the corresponding technical effects. The beneficial effects corresponding to the steps executed by each module can refer to the descriptions of the steps corresponding to fig. 3 to 9, and are not described again. The data processing apparatus may be the first device or a chip or system on a chip in the first device. The function can be realized by hardware, and can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above functions. The data processing apparatus includes:

the receiving and sending module 111 is configured to receive metadata information from the message queue server, where the metadata information is a data identifier corresponding to first incremental data that is newly added when the application server executes a service; the transceiving module 111 is further configured to, when the metadata information exists in the preset data conversion configuration information, obtain first incremental data corresponding to the metadata information from a server of the source database; presetting data conversion configuration information for data conversion between a server of a source database and a server of at least one type of target database; the processing module 112 is configured to convert the first incremental data into second incremental data of a server suitable for at least one type of target database according to preset data conversion configuration information, where the source database and the target database are heterogeneous databases; the transceiver module 111 is further configured to send second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

In one embodiment, the transceiver module 111 is further configured to: sending a first request to second equipment, wherein the first request is used for requesting preset data conversion configuration information; and receiving preset data conversion configuration information from the second device.

In one embodiment, the processing module 112 is further configured to: and calling preset data conversion configuration information from the local cache.

In an embodiment, the transceiving module 111 is further configured to, before receiving the metadata information from the message queue server, obtain, from the server of the source database, first inventory data stored by the server of the source database; the processing module 112 is further configured to convert the first stock data into second stock data of the server suitable for the at least one type of target database according to the preset data conversion configuration information; the transceiver module 111 is further configured to send second inventory data corresponding to the server of the target database to the server of the at least one type of target database.

In one embodiment, the preset data conversion configuration information includes a conversion relationship between a first field corresponding to data in the source database and a second field corresponding to data in the target database; the information represented by the first field and the second field has the same meaning, and the field structures are different; the processing module 112 is specifically configured to convert the first field in the first incremental data into the second field according to the conversion relationship in the preset data conversion configuration information, so as to obtain second incremental data of the server that is applicable to at least one type of target database.

In one embodiment, the second device includes a second device or a plurality of second devices, one second device corresponding to one target database; or one second device corresponds to a plurality of target databases.

The embodiment of the application also provides another data processing device, and the data processing device is applied to an application server. Each module in the device has the function of realizing the corresponding steps in the figures 3-9 and can achieve the corresponding technical effect. The beneficial effects corresponding to the steps executed by each module can refer to the descriptions of the steps corresponding to fig. 3 to 9, and are not described again. The data processing device may be an application server or a chip or system on a chip in an application server. The function can be realized by hardware, and can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above functions. The data processing apparatus includes:

the receiving and sending module is used for sending the first incremental data to a server of a source database and sending metadata information corresponding to the first incremental data to the message queue server under the condition that the first incremental data is generated in the service execution process; wherein the metadata information is for: under the condition that the metadata information exists in the preset data conversion configuration information, after the first device acquires the metadata information from the message queue server, the first device converts the first incremental data into second incremental data suitable for the server of the at least one type of target database according to the preset data conversion configuration information, and sends the second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

In one embodiment, the transceiver module is further configured to: under the condition that a server of a source database fails, sending a data acquisition request to a server of any one target database; the data acquisition request is used for requesting second incremental data; and receiving second incremental data from the server of the target database.

The embodiment of the application also provides another data processing device, and the data processing device is applied to a message queue server. Each module in the device has the function of realizing the corresponding steps in the figures 3-9 and can achieve the corresponding technical effect. The beneficial effects corresponding to the steps executed by each module can refer to the descriptions of the steps corresponding to fig. 3 to 9, and are not described again. The data processing device may be a message queue server or a chip or a system on chip in a message queue server. The function can be realized by hardware, and can also be realized by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. The data processing apparatus includes:

the receiving and sending module is used for receiving metadata information from the application server, wherein the metadata information is a data identifier corresponding to first incremental data which is newly added when the application server executes a service; the receiving and sending module is also used for sending metadata information to the first equipment; wherein the metadata information is for: and under the condition that the metadata information exists in the preset data conversion configuration information, the first device converts the first incremental data into second incremental data suitable for the servers of the at least one type of target database according to the preset data conversion configuration information, and sends the second incremental data corresponding to the servers of the target database to the servers of the at least one type of target database.

The embodiment of the application also provides a computer readable storage medium. All or part of the processes in the above method embodiments may be performed by relevant hardware instructed by a computer program, which may be stored in the above computer-readable storage medium, and when executed, may include the processes in the above method embodiments. The computer readable storage medium may be the terminal device of any of the foregoing embodiments, such as: including internal storage units of the data transmitting end and/or the data receiving end, such as a hard disk or a memory of the terminal device. The computer readable storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) card, a flash memory card (flash card), and the like, which are provided on the terminal device. Further, the computer-readable storage medium may include both an internal storage unit and an external storage device of the terminal apparatus. The computer-readable storage medium stores the computer program and other programs and data required by the terminal device. The above-described computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

The embodiment of the application also provides a computer instruction. All or part of the flow of the above method embodiments may be performed by computer instructions to instruct relevant hardware (such as a computer, a processor, a network device, a terminal, and the like). The program may be stored in the computer-readable storage medium described above.

The embodiment of the application also provides a chip system. The chip system may be formed by a chip, and may also include a chip and other discrete devices, without limitation. The chip system includes a processor and a transceiver, and all or part of the processes in the method embodiments may be completed by the chip system, for example, the chip system may be used to implement the functions executed by each node in the method embodiments, or implement the functions executed by each node in the method embodiments.

In a possible design, the chip system further includes a memory, where the memory is used to store program instructions and/or data, and when the chip system runs, the processor executes the program instructions stored in the memory, so that the chip system executes the functions executed by the nodes in the method embodiment or executes the functions executed by the nodes in the method embodiment.

In the embodiments of the present application, the processor may be a general-purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic blocks disclosed in the embodiments of the present application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in a processor.

In the embodiment of the present application, the memory may be a nonvolatile memory, such as a Hard Disk Drive (HDD) or a solid-state drive (SSD), and may also be a volatile memory, for example, a random-access memory (RAM). The memory is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function for storing instructions and/or data.

It should be noted that the terms "first" and "second" and the like in the description, claims and drawings of the present application are used for distinguishing different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

It should be understood that in the embodiments of the present application, "at least one" means one or more, "a plurality" means two or more, "at least two" means two or three and more, "and/or" is used to describe the association relationship of the associated objects, which means that there may be three relationships, for example, "a and/or B" may mean: only A, only B and both A and B are present, wherein A and B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of single item(s) or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural. It should be understood that in the embodiment of the present application, "B corresponding to a" means that B is associated with a. For example, B may be determined from A. It should also be understood that determining B from a does not mean determining B from a alone, but may also be determined from a and/or other information. In addition, the term "connect" in the embodiment of the present application refers to various connection manners, such as direct connection or indirect connection, to implement communication between devices, and this is not limited in this embodiment of the present application.

The "transmission" appearing in the embodiments of the present application refers to a bidirectional transmission, including actions of transmission and/or reception, unless otherwise specified. Specifically, "transmission" in the embodiment of the present application includes transmission of data, reception of data, or both transmission of data and reception of data. Alternatively, the data transmission herein includes uplink and/or downlink data transmission. The data may include channels and/or signals, uplink data transmission, i.e., uplink channel and/or uplink signal transmission, and downlink data transmission, i.e., downlink channel and/or downlink signal transmission. The expression "network" and "system" in the embodiments of the present application is the same concept, and a data processing system is a communication network.

Through the above description of the embodiments, it is clear to those skilled in the art that, for convenience and simplicity of description, the foregoing division of the functional modules is merely used as an example, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device may be divided into different functional modules to complete all or part of the above described functions.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical functional division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another device, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may be one physical unit or a plurality of physical units, that is, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application may be substantially or partially implemented in the form of software products, which are stored in a storage medium and include instructions for causing a device, such as: the method can be a single chip, a chip, or a processor (processor) for executing all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

The above description is only an embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present disclosure should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A data processing method, applied to a first device, comprising:

receiving metadata information from a message queue server; the metadata information is a data identifier corresponding to newly added first incremental data when the application server executes the service;

under the condition that the metadata information exists in preset data conversion configuration information, acquiring the first incremental data corresponding to the metadata information from a server of a source database; the preset data conversion configuration information is used for data conversion between the server of the source database and the server of at least one type of target database;

converting the first incremental data into second incremental data suitable for a server of the at least one type of target database according to the preset data conversion configuration information, wherein the source database and the target database are heterogeneous databases;

and sending the second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

2. The data processing method of claim 1, wherein the method further comprises:

sending a first request to a second device, wherein the first request is used for requesting the preset data conversion configuration information;

receiving the preset data conversion configuration information from the second device.

3. The data processing method of claim 2, wherein the method further comprises:

and calling the preset data conversion configuration information from a local cache.

4. The data processing method of claim 1, wherein prior to receiving the metadata information from the message queue server, the method further comprises:

acquiring first stock data stored by a server of a source database from the server of the source database;

converting the first stock data into second stock data of a server suitable for the at least one type of target database according to the preset data conversion configuration information;

and sending the second stock data corresponding to the server of the target database to the server of the at least one type of target database.

5. The data processing method according to any one of claims 1 to 4, wherein the preset data conversion configuration information includes a conversion relationship between a first field corresponding to data in the source database and a second field corresponding to data in the target database; the information represented by the first field and the second field has the same meaning, and the field structures are different;

the converting the first incremental data into second incremental data suitable for the server of the at least one type of target database according to the preset data conversion configuration information includes:

and converting a first field in the first incremental data into a second field according to the conversion relation in the preset data conversion configuration information to obtain second incremental data of the server suitable for the at least one type of target database.

6. The data processing method of claim 3, wherein the second device comprises one second device or a plurality of second devices, one of the second devices corresponding to one of the target databases; or, one second device corresponds to a plurality of target databases.

7. A data processing method is applied to an application server and comprises the following steps:

under the condition that first incremental data are generated in the service execution process, sending the first incremental data to a server of a source database and sending metadata information corresponding to the first incremental data to a message queue server; wherein the metadata information is to:

and under the condition that the metadata information exists in preset data conversion configuration information, after acquiring the metadata information from the message queue server, the first device converts the first incremental data into second incremental data suitable for a server of at least one type of target database according to the preset data conversion configuration information, and sends the second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

8. The data processing method of claim 7, wherein the method further comprises:

under the condition that the server of the source database fails, sending a data acquisition request to any server of the target database; the data acquisition request is used for requesting the second incremental data;

receiving the second delta data from a server of the target database.

9. A data processing method is applied to a message queue server and comprises the following steps:

receiving metadata information from an application server; the metadata information is a data identifier corresponding to newly added first incremental data when the application server executes the service;

sending the metadata information to a first device; wherein the metadata information is used to: and under the condition that the metadata information exists in preset data conversion configuration information, the first device converts the first incremental data into second incremental data suitable for a server of at least one type of target database according to the preset data conversion configuration information, and sends the second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

10. The data processing method of claim 9, wherein the first device comprises one first device or a plurality of first devices, one first device corresponding to one target database; or, one first device corresponds to a plurality of target databases.

11. A data processing apparatus, applied to a first device, comprising:

the receiving and sending module is used for receiving metadata information from the message queue server; the metadata information is a data identifier corresponding to newly added first incremental data when the application server executes the service;

the transceiving module is further configured to acquire the first incremental data corresponding to the metadata information from a server of a source database when the metadata information exists in preset data conversion configuration information; the preset data conversion configuration information is used for data conversion between the server of the source database and the servers of at least one type of target database;

the processing module is used for converting the first incremental data into second incremental data suitable for a server of the at least one type of target database according to the preset data conversion configuration information, and the source database and the target database are heterogeneous databases;

the transceiver module is further configured to send the second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

12. The data processing apparatus of claim 11, wherein the transceiver module is further configured to:

sending a first request to a second device, wherein the first request is used for requesting the preset data conversion configuration information; and receiving the preset data conversion configuration information from the second device.

13. The data processing apparatus of claim 12, wherein the processing module is further configured to:

14. The data processing apparatus according to claim 11, wherein the transceiver module is further configured to, before receiving the metadata information from the message queue server, obtain, from the server of the source database, first inventory data stored by the server of the source database;

the processing module is further configured to convert the first stock data into second stock data of a server suitable for the at least one type of target database according to the preset data conversion configuration information;

the transceiver module is further configured to send the second inventory data corresponding to the server of the target database to the server of the at least one type of target database.

15. The data processing apparatus according to any of claims 11 to 14, wherein the preset data conversion configuration information includes a conversion relationship between a first field corresponding to data in the source database and a second field corresponding to data in the target database; the information represented by the first field and the second field has the same meaning, and the field structures are different;

the processing module is specifically configured to convert a first field in the first incremental data into a second field according to the conversion relationship in the preset data conversion configuration information, so as to obtain second incremental data of a server applicable to the at least one type of target database.

16. The data processing apparatus of claim 13, wherein the second device comprises one second device or a plurality of second devices, one of the second devices corresponding to one of the target databases; or, one second device corresponds to a plurality of target databases.

17. A data processing apparatus, applied to an application server, comprising:

the receiving and sending module is used for sending the first incremental data to a server of a source database and sending metadata information corresponding to the first incremental data to a message queue server under the condition that the first incremental data is generated in the service execution process; wherein the metadata information is used to:

18. The data processing apparatus of claim 17, wherein the transceiver module is further configured to:

under the condition that the server of the source database fails, sending a data acquisition request to any server of the target database; the data acquisition request is used for requesting the second incremental data; and receiving the second delta data from the server of the target database.

19. A data processing apparatus, applied to a message queue server, comprising:

the receiving and sending module is used for receiving the metadata information from the application server; the metadata information is a data identifier corresponding to newly added first incremental data when the application server executes the service;

the transceiver module is further configured to send the metadata information to the first device; wherein the metadata information is to: and under the condition that the metadata information exists in preset data conversion configuration information, the first device converts the first incremental data into second incremental data suitable for a server of at least one type of target database according to the preset data conversion configuration information, and sends the second incremental data corresponding to the server of the target database to the server of the at least one type of target database.

20. The data processing apparatus of claim 19, wherein the first device comprises one first device or a plurality of first devices, one of the first devices corresponding to one of the target databases; or, one first device corresponds to a plurality of target databases.

21. A data processing apparatus, characterized in that the data processing apparatus comprises a processor and a transceiver for enabling the data processing apparatus to perform the method according to any of claims 1-10.

22. A computer-readable storage medium storing computer instructions which, when executed, perform the method of any one of claims 1-10.

23. A data processing system, characterized in that the data processing system comprises: the system comprises an application server, a message queue server, a first device and a second device;

the application server is configured to: under the condition that first incremental data are generated in the service execution process, sending the first incremental data added when the application server executes the service to a server of a source database, and sending metadata information corresponding to the first incremental data to a message queue server; the metadata information is a data identifier corresponding to newly added first incremental data when the application server executes the service;

the message queue server is configured to: receiving the metadata information from the application server; and sending the metadata information to the first device;

the first device to: receiving metadata information from a message queue server; under the condition that the metadata information exists in preset data conversion configuration information, acquiring the first incremental data corresponding to the metadata information from a server of a source database; the preset data conversion configuration information is used for data conversion between the server of the source database and the server of at least one type of target database; the preset data conversion configuration information is sent to the first equipment by the second equipment; converting the first incremental data into second incremental data suitable for a server of the at least one type of target database according to the preset data conversion configuration information, wherein the source database and the target database are heterogeneous databases; and sending the second incremental data corresponding to the server of the target database to the server of the at least one type of target database.