CN113672692A

CN113672692A - Data processing method, data processing device, computer equipment and storage medium

Info

Publication number: CN113672692A
Application number: CN202111239027.2A
Authority: CN
Inventors: 钟子宏
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Cloud Computing Beijing Co Ltd
Priority date: 2021-10-25
Filing date: 2021-10-25
Publication date: 2021-11-19
Anticipated expiration: 2041-10-25
Also published as: CN113672692B

Abstract

The application discloses a data processing method, a data processing device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a first updating parameter of a source database at a first moment; the source database comprises M pieces of storage data at a first moment, wherein M is a positive integer; the first updating parameters comprise updating parameters corresponding to the M pieces of storage data at a first moment; acquiring a second updating parameter of the source database at a second moment; the source database comprises N pieces of storage data at a second moment, wherein N is a positive integer; the second updating parameter comprises updating parameters corresponding to the N pieces of storage data at a second moment; the first time is earlier than the second time; if the first updating parameter is inconsistent with the second updating parameter, determining target storage data obtained by updating based on M pieces of storage data from the N pieces of storage data; and synchronizing the target storage data partition to the target database. By the method and the device, the vehicle-mounted scene can be applied to the vehicle-mounted scene, and the efficiency of synchronizing the data in the source database can be improved.

Description

Data processing method, data processing device, computer equipment and storage medium

Technical Field

The present application relates to the field of data processing technologies, and in particular, to a data processing method and apparatus, a computer device, and a storage medium.

Background

Data synchronization refers to backup operation between different storage devices, or between a terminal and a terminal, or between a terminal and a server, so that data integrity and uniformity are maintained.

In an existing application, data synchronization can be performed on data in a source database, and the data in the source database can be updated continuously, so that each time data in the source database is synchronized, a full amount of data in the source database needs to be synchronized, which results in low efficiency of synchronizing the data in the source database.

Disclosure of Invention

The application provides a data processing method, a data processing device, computer equipment and a storage medium, which can improve the efficiency of synchronizing data in a source database.

One aspect of the present application provides a data processing method, including:

acquiring a first updating parameter of a source database at a first moment; the source database comprises M pieces of storage data at a first moment, wherein M is a positive integer; the first updating parameters comprise updating parameters corresponding to the M pieces of storage data at a first moment;

acquiring a second updating parameter of the source database at a second moment; the source database comprises N pieces of storage data at a second moment, wherein N is a positive integer; the second updating parameter comprises updating parameters corresponding to the N pieces of storage data at a second moment; the first time is earlier than the second time;

if the first updating parameter is inconsistent with the second updating parameter, determining target storage data obtained by updating based on M pieces of storage data from the N pieces of storage data;

and synchronizing the target storage data partition to the target database.

One aspect of the present application provides a data processing apparatus, including:

the acquisition module is used for acquiring a first updating parameter of a source database at a first moment; the source database comprises M pieces of storage data at a first moment, wherein M is a positive integer; the first updating parameters comprise updating parameters corresponding to the M pieces of storage data at a first moment;

the acquisition module is used for acquiring a second updating parameter of the source database at a second moment; the source database comprises N pieces of storage data at a second moment, wherein N is a positive integer; the second updating parameter comprises updating parameters corresponding to the N pieces of storage data at a second moment; the first time is earlier than the second time;

the determining module is used for determining target storage data obtained by updating based on the M pieces of storage data from the N pieces of storage data if the first updating parameter is inconsistent with the second updating parameter;

and the synchronization module is used for synchronizing the target storage data partition to the target database.

Optionally, each piece of storage data in the N pieces of storage data and the M pieces of storage data has a corresponding data identifier, and the updated storage data is the same as the data identifier corresponding to the storage data before updating; the updating parameters respectively corresponding to the M pieces of storage data contained in the first updating parameter are updating timestamps respectively corresponding to the M pieces of storage data at a first moment; the update parameters respectively corresponding to the N pieces of storage data included in the second update parameter are update timestamps respectively corresponding to the N pieces of storage data at the second time.

Optionally, any one of the M pieces of storage data is represented as the ith piece of storage data, i is a positive integer smaller than or equal to M, any one of the N pieces of storage data is represented as the jth piece of storage data, and j is a positive integer smaller than or equal to N;

if the first update parameter is inconsistent with the second update parameter, the determining module determines a mode of target storage data updated based on the M pieces of storage data from the N pieces of storage data, including:

comparing the update time stamp corresponding to the ith storage data at the first moment with the update time stamp corresponding to the jth storage data at the second moment;

if the ith piece of storage data and the jth piece of storage data have the same data identification, and the update time stamp corresponding to the ith piece of storage data at the first moment is not consistent with the update time stamp corresponding to the jth piece of storage data at the second moment by comparison, determining the jth piece of storage data as target storage data;

and determining the data identifier of the jth piece of storage data as a target data identifier, and if the data identifier of the jth piece of storage data does not contain the update time stamp of the storage data with the target data identifier at the first moment in the M pieces of storage data, determining the jth piece of storage data as the target storage data.

Optionally, the manner of synchronizing the target storage data partition to the target database by the synchronization module includes:

adding a partition identifier to target storage data;

and synchronizing the target storage data added with the partition identifications into the target database.

Optionally, the method for adding the partition identifier to the target storage data by the synchronization module includes:

acquiring a timestamp corresponding to the second moment;

and determining the time stamp corresponding to the second moment as the partition identification of the target storage data.

Optionally, the apparatus is further configured to:

acquiring a data query request sent by a data client; the data query request carries a partition identifier;

inquiring target storage data added with the partition identification according to the data inquiry request;

and returning the inquired target storage data to the data client so that the data client displays the target storage data on a client page.

Optionally, the source database includes a plurality of sub-databases;

the method for synchronizing the target storage data partition to the target database by the synchronization module comprises the following steps:

generating a target database, and synchronizing target storage data partitions to the target database; alternatively, the first and second electrodes may be,

and selecting a target database from the plurality of sub-databases, and synchronizing target storage data to the target database.

Optionally, the manner in which the synchronization module generates the target database includes:

acquiring a target storage address in a distributed storage cluster; the distributed storage cluster comprises a plurality of storage clusters; each storage cluster is provided with a corresponding storage address;

determining a storage cluster indicated by a target storage address in the plurality of storage clusters as a target storage cluster;

a target database is generated in the target storage cluster.

Optionally, the number of the target storage data is multiple;

classifying target storage data with the same data structure in a plurality of pieces of target storage data to obtain L storage data sets; l is a positive integer; a storage data set comprises at least one target storage data;

respectively carrying out data merging on target storage data in each storage data set to obtain a merged data table corresponding to each storage data set;

and synchronizing the merged data table partition corresponding to each storage data set to the target database.

Optionally, the target storage data with the same field structure has the same data structure;

the method for classifying the target storage data with the same data structure in the plurality of pieces of target storage data by the synchronization module to obtain L storage data sets includes:

identifying a field structure of each piece of target storage data in the plurality of pieces of target storage data;

and classifying the target storage data with the same field structure in the plurality of pieces of target storage data into the same storage data set to obtain L storage data sets.

Optionally, the apparatus is further configured to:

acquiring a data synchronization period aiming at a source database;

determining a second moment according to the data synchronization period and the first moment; the first time and the second time are separated by a data synchronization period.

An aspect of the application provides a computer device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the method of an aspect of the application.

An aspect of the application provides a computer-readable storage medium having stored thereon a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method of the above-mentioned aspect.

According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various alternatives of the above aspect and the like.

The method includes the steps that a first updating parameter of a source database at a first moment is obtained; the source database comprises M pieces of storage data at a first moment, wherein M is a positive integer; the first updating parameters comprise updating parameters corresponding to the M pieces of storage data at a first moment; acquiring a second updating parameter of the source database at a second moment; the source database comprises N pieces of storage data at a second moment, wherein N is a positive integer; the second updating parameter comprises updating parameters corresponding to the N pieces of storage data at a second moment; the first time is earlier than the second time; if the first updating parameter is inconsistent with the second updating parameter, determining target storage data obtained by updating based on M pieces of storage data from the N pieces of storage data; and synchronizing the target storage data partition to the target database. Therefore, the method provided by the application can acquire the incremental data (such as target storage data) in the source database through the first updating parameter and the second updating parameter, the incremental data is the updated data, and only the incremental data needs to be synchronized, so that the data synchronization efficiency is high. In addition, the accuracy of synchronizing the storage data is also improved by performing partition synchronization (e.g., synchronization according to time partitions) on the target storage data.

Drawings

In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

Fig. 1 is a schematic structural diagram of a network architecture according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a data synchronization scenario provided herein;

FIG. 3 is a schematic flow chart diagram of a data processing method provided herein;

FIG. 4 is a schematic diagram of a scenario for determining target storage data provided herein;

FIG. 5 is a schematic diagram of a scenario for determining target storage data provided herein;

FIG. 6 is a schematic diagram of a scenario of a data query provided herein;

FIG. 7 is a schematic flow chart diagram of a data processing method provided herein;

FIG. 8 is a schematic diagram of a data classification scenario provided herein;

FIG. 9 is a block diagram of a data synchronization framework provided herein;

FIG. 10 is a schematic diagram of a data processing apparatus provided in the present application;

fig. 11 is a schematic structural diagram of a computer device provided in the present application.

Detailed Description

The technical solutions in the present application will be described clearly and completely with reference to the accompanying drawings in the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

The present application relates to the field of block chaining. The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism and an encryption algorithm. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product services layer, and an application services layer. The Block chain comprises a series of blocks (blocks) which are mutually connected according to the generated chronological order, once a new Block is added into the Block chain, the new Block cannot be removed, and the recorded data submitted by the nodes in the Block chain system are recorded in the blocks. According to the method and the device, the finally obtained target storage data can be synchronized to the block chain nodes, so that the safety of the target storage data is guaranteed.

The application also relates to cloud technology. The Cloud Technology (Cloud Technology) is a hosting Technology for unifying series resources such as hardware, software, network and the like in a wide area network or a local area network to realize calculation, storage, processing and sharing of data.

The cloud technology is based on the general names of network technology, information technology, integration technology, management platform technology, application technology and the like applied in the cloud computing business model, can form a resource pool, is used as required, and is flexible and convenient. Cloud computing technology will become an important support. Background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture-like websites and more web portals. With the high development and application of the internet industry, each article may have its own identification mark and needs to be transmitted to a background system for logic processing, data in different levels are processed separately, and various industrial data need strong system background support and can only be realized through cloud computing.

The cloud technology referred to in the present application mainly refers to that target storage data can be synchronized through "cloud", for example, target storage data can be stored in a distributed manner through "cloud".

First, a description will be given of a related art concept to which the present application relates:

(1) data synchronization: and backup operation between different storage devices or between the terminal and between the terminal and the server is carried out, so that the integrity and the uniformity of the data are kept.

(2) And (3) updating data: the data updating is realized by deleting, modifying and reinserting the new data item or record and the old data item or record corresponding to the new data item or record in the data file or database.

(3) Data partitioning: a physical database design technique is mainly aimed at reducing the total amount of data read and write in a specific SQL operation (a structured query operation) so as to shorten the response time.

(4) Distributed storage: a data storage technology uses disk space on each machine in an enterprise through a network, and forms a virtual storage device by using scattered storage resources, and data are scattered and stored in each corner of the enterprise.

Referring to fig. 1, fig. 1 is a schematic structural diagram of a network architecture according to an embodiment of the present disclosure. As shown in fig. 1, the network architecture may include a server 200 and a terminal device cluster, and the terminal device cluster may include one or more terminal devices, where the number of terminal devices is not limited herein. As shown in fig. 1, the plurality of terminal devices may specifically include a terminal device 100a, a terminal device 101a, terminal devices 102a, …, and a terminal device 103 a; as shown in fig. 1, the terminal device 100a, the terminal device 101a, the terminal devices 102a, …, and the terminal device 103a may all be in network connection with the server 200, so that each terminal device may perform data interaction with the server 200 through the network connection.

The server 200 shown in fig. 1 may be an independent physical server, may also be a server cluster or a distributed system formed by a plurality of physical servers, and may also be a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a network service, cloud communication, a middleware service, a domain name service, a security service, a CDN, and a big data and artificial intelligence platform. The terminal device may be: the intelligent terminal comprises intelligent terminals such as a smart phone, a tablet computer, a notebook computer, a desktop computer, an intelligent television and a vehicle-mounted terminal. The following takes communication between the terminal device 100a and the server 200 as an example, and a detailed description of an embodiment of the present application is made.

Referring to fig. 2, fig. 2 is a schematic view of a data synchronization scenario provided in the present application. As shown in fig. 2, the source database at the first time may include M storage data, where the M storage data may be storage 1 to storage data M, and the source database at the first time may have a first update parameter, where the first update parameter may include an update parameter corresponding to each storage data in the M storage data, and specifically may include an update parameter M corresponding to update parameter 1 corresponding to storage data 1 to storage data M.

The source database at the second time may include N storage data, where the N storage data may be storage 1 to storage data N, and the source database at the second time may have a second update parameter, where the second update parameter may include an update parameter corresponding to each storage data in the N storage data, and specifically may include update parameters N corresponding to update parameters 1 corresponding to storage data 1 to storage data N.

Wherein the first time is earlier than the second time.

Therefore, the server 200 may compare the first update parameter with the second update parameter, and if the first update parameter is inconsistent with the second update parameter, the server 200 may determine the storage data updated between the first time and the second time from among the N storage data at the second time, and use the storage data as the target storage data. The target storage data is the storage data updated on the basis of the M storage data. The specific process of determining the target storage data from the N storage data by comparing the first update parameter and the second update parameter may also be referred to the following description in the corresponding embodiment of fig. 3.

Then, the server 200 may synchronize the target storage data partition to the target database, where the target storage data partition synchronized to the target database carries a partition identifier, where the partition identifier may be an identifier of a time partition, and for example, the partition identifier may be a timestamp corresponding to the second time.

The terminal device 100a may include a data client, and a background server of the data client may be the server 200, so that the terminal device 100a may generate a data query request in the data client according to a user operation, where the data query request may carry the partition identifier, the terminal device 100a may send the data query request to the server 200, and further, the server 200 may query the synchronized target storage data according to the partition identifier in the data query request, and return the queried target storage data to the terminal device 100a, and the terminal device 100a may display the target storage data in a client page of the data client for a user to view.

By adopting the method provided by the application, the incremental data (namely the updated storage data, such as the target storage data) in the source database can be synchronized at each data synchronization time (such as the second time), and the full amount of storage data in the source database does not need to be synchronized, so that the efficiency of synchronizing the storage data is improved.

Referring to fig. 3, fig. 3 is a schematic flow chart of a data processing method provided in the present application. The execution subject in the embodiment of the present application may be one computer device or a computer device cluster formed by a plurality of computer devices. The computer equipment can be a server or terminal equipment. Therefore, the execution subject in the embodiment of the present application may be a server, or may be a terminal device, or may be a server and a terminal device, and the execution subject in the embodiment of the present application is described as an example of the server. As shown in fig. 3, the method may include:

step S101, acquiring a first updating parameter of a source database at a first moment; the source database comprises M pieces of storage data at a first moment, wherein M is a positive integer; the first update parameter includes update parameters corresponding to the M pieces of storage data at the first time.

Optionally, the source database contains all data that can be synchronized, i.e. the source database is used to provide data for synchronization. The data in the source database may be continuously updated (updated in real time), and the source database may include the final state data of each data, in other words, the data included in the source database is an overlay update, and after updating a data, only the updated result of the data is retained, and the result before updating the data is not retained. The updating of the data in the source database may refer to operations such as deleting, modifying, adding and the like to the data in the source database, and therefore, it can be understood that the data synchronized in the source database may be the modified or added data included in the source database, and the deleted data in the source database may not be synchronized. The synchronization of the data may be understood as backup storage of the data.

For example, if the source database contains data a at a first time, the data a is updated at a second time, and the updated data a can be denoted as data b, then the data b overwrites the data a in the source database, that is, the source database contains the data b at the second time, but does not contain the data a.

Note that, the data in the source database is usually data that is not partitioned.

Therefore, the server may obtain the update parameter of the source database at the first time, and the update parameter of the source database at the first time may be referred to as a first update parameter.

Optionally, the source database may be determined according to an actual service scenario, for example, data contained in the source database may be user data or may be system data, and the data contained in the source database may be referred to as storage data. The source database may include M pieces of storage data at the first time, and a specific value of M may be determined according to an actual application scenario, which is not limited to this. The first time may be a historical time prior to the current time.

Therefore, it can be understood that the first update parameter may include an update parameter corresponding to each piece of the M pieces of storage data at the first time.

Optionally, the update parameter corresponding to each piece of storage data in the M pieces of storage data at the first time may be an update timestamp corresponding to each piece of storage data in the M pieces of storage data at the first time, and the update timestamp may be a timestamp corresponding to a time when the storage data is updated last until the first time. Alternatively, the update parameter of each storage data may be other than the update time stamp, as long as the parameter can represent that the storage data is updated. The method is not limited, and may be specifically set according to actual service requirements.

Step S102, acquiring a second updating parameter of the source database at a second moment; the source database comprises N pieces of storage data at a second moment, wherein N is a positive integer; the second updating parameter comprises updating parameters corresponding to the N pieces of storage data at a second moment; the first time is earlier than the second time.

Optionally, the server may further obtain the update parameter of the source database at the second time, and the update parameter of the source database at the second time may be referred to as a second update parameter. The second time is later than the first time, that is, the first time is earlier than the second time, and the second time may be a current time, that is, the second time may be a latest time at which data synchronization is currently required.

Optionally, the server may obtain a data synchronization period for the source database, where the data synchronization period may be set according to an actual service scenario. The data synchronization period may be a period for detecting whether the storage data in the source database is updated and which storage data are updated, and further may perform data synchronization on the storage data that are updated.

Therefore, the server can determine the second time according to the first time and the data synchronization period, and the first time and the second time can be separated by one data synchronization period. For example, the data synchronization period may be 1 day, so that if the first time is the starting time of day 1, the second time may be the starting time of day 2.

Similarly, the source database may include N pieces of stored data at the second time, and the specific value of N may be determined according to the actual application scenario, which is not limited to this. The values of N and M may be the same or different.

Therefore, it can be understood that the second update parameter may include an update parameter corresponding to each of the N pieces of storage data at the second time.

Optionally, the update parameter corresponding to each piece of the N pieces of storage data at the second time may be an update timestamp corresponding to each piece of the storage data at the second time in the N pieces of storage data (it is understood that the storage data may be updated at any time between the first time and the second time), and the update timestamp may be a timestamp corresponding to a time when the storage data is updated last until the second time. Alternatively, the update parameter of each storage data may be other than the update time stamp, as long as the parameter can represent that the storage data is updated. The method is not limited, and may be specifically set according to actual service requirements.

Therefore, the N pieces of storage data may include storage data that is not updated in the M pieces of storage data, storage data obtained by modifying part or all of the M pieces of storage data, or storage data that is newly added on the basis of the M pieces of storage data.

Step S103, if the first update parameter is inconsistent with the second update parameter, determining target storage data updated based on the M pieces of storage data from the N pieces of storage data.

Optionally, the server may compare the first update parameter with the second update parameter, and if the first update parameter is inconsistent with the second update parameter, the server may determine, from the N pieces of storage data, storage data updated based on the M pieces of storage data, and may refer to the storage data as target storage data. In other words, the target storage data may include storage data obtained by modifying some or all of the M pieces of storage data, or storage data newly added on the basis of the M pieces of storage data.

The target storage data belongs to incremental data of the source database from a first time to a second time.

The specific process of determining the target storage data from the N pieces of storage data according to the comparison between the first update parameter and the second update parameter may be:

here, the update parameter of each piece of stored data is taken as an example of an update time stamp at a corresponding time (e.g., a first time or a second time).

Each piece of the N pieces of storage data and each piece of the M pieces of storage data has a corresponding data identifier, and the updated storage data is the same as the data identifier corresponding to the storage data before updating. For example, N pieces of storage data include storage data a, M pieces of storage data include storage data b, and storage data b is obtained by updating storage data a, then storage data b has the same data identifier as storage data a, and a piece of storage data may have a data identifier for identifying the corresponding storage data.

Any one of the M pieces of storage data may be represented as the ith piece of storage data, i is a positive integer smaller than or equal to M, and any one of the N pieces of storage data may be represented as the jth piece of storage data, j is a positive integer smaller than or equal to N.

Therefore, the server can compare the update time stamp corresponding to the ith piece of stored data in the first update parameter at the first moment with the update time stamp corresponding to the jth piece of stored data in the second update parameter at the second moment; if the ith piece of storage data and the jth piece of storage data have the same data identifier, and the server compares that the update timestamp of the ith piece of storage data corresponding to the first time is inconsistent with the update timestamp of the jth piece of storage data corresponding to the second time, where the inconsistency may be that the update timestamp of the jth piece of storage data corresponding to the second time is later than the update timestamp of the ith piece of storage data corresponding to the first time, it may be determined that the jth piece of storage data is obtained by updating the ith piece of storage data, and the jth piece of storage data may be used as the target storage data.

Referring to fig. 4, fig. 4 is a schematic view of a scenario for determining target storage data according to the present application. As shown in fig. 4, any piece of storage data in the source database at the first time may be represented as an ith piece of storage data, any piece of storage data in the source database at the second time may be represented as a jth piece of storage data, an update parameter corresponding to the ith piece of storage data at the first time may be an update timestamp 1, and an update parameter corresponding to the jth piece of storage data at the second time may be an update timestamp 2. Therefore, if the ith piece of storage data and the jth piece of storage data have the same data identifier y, and the update timestamp 2 is located after the update timestamp 1 on the time axis, that is, the update timestamp 2 is later than the update timestamp 1, it indicates that the jth piece of storage data is obtained by modifying the ith piece of storage data, and therefore, the jth piece of storage data at this time can be used as the target storage data.

More, the data identifier of the jth piece of storage data may be referred to as a target data identifier, and if the server compares that the M pieces of storage data do not include the update time stamp of the storage data with the target data identifier at the first time, that is, the M pieces of storage data do not include the storage data with the target data identifier, the jth piece of storage data may be considered as storage data newly added above the M pieces of storage data, and at this time, the jth piece of storage data may also be used as the target storage data.

Referring to fig. 5, fig. 5 is a schematic view of a scenario for determining target storage data according to the present application. As shown in fig. 5, any piece of storage data in the source database at the first time may be represented as an ith piece of storage data, any piece of storage data in the source database at the second time may be represented as a jth piece of storage data, a data identifier of the ith piece of storage data may be a data identifier x, and a data identifier of the jth piece of storage data may be a data identifier y. If the source database at the first time does not have the storage data with the data identifier y, the jth piece of storage data is the storage data newly added on the basis of the M pieces of storage data, and at this time, the jth piece of storage data may also be used as the target storage data.

In addition, if the ith piece of storage data and the jth piece of storage data have the same data identifier, and the server compares that the update time stamp corresponding to the ith piece of storage data at the first time is consistent with the update time stamp corresponding to the jth piece of storage data at the second time, it indicates that the jth piece of storage data is the ith piece of storage data, and the ith piece of storage data is not updated between the first time and the second time, so that the jth piece of storage data is not the target storage data at this time.

Optionally, the source database may further include a plurality of sub-databases, the first update parameter may further include data volumes respectively corresponding to the sub-databases at the first time, one sub-database may correspond to one data volume, and the data volume corresponding to the first time of one sub-database is the number of storage data included in the sub-database at the first time. Therefore, the second update parameter may further include data volumes respectively corresponding to the sub-databases at the second time, and a data volume corresponding to one sub-database at the second time is the number of pieces of storage data included in the sub-database at the second time.

Therefore, for some sub-databases whose data volume is increased but not decreased (i.e. the data volume in the sub-database is increased continuously, even if part of the stored data in the sub-database is deleted, the data volume of the deleted part of the stored data is still included in the data volume of the sub-database), the data volume of the sub-database at the second time can be directly compared with the data volume of the sub-database at the first time, i.e. whether the stored data in the sub-database is updated or not can be determined, by comparing the data volumes of the sub-databases at different times (e.g. the first time and the second time), the sub-database with obviously updated data can be quickly located (e.g. the sub-database with the data volume at the second time larger than the data volume at the first time, the stored data in the sub-database is obviously updated), and further, by comparing the stored data in the sub-databases at different times (e.g. the first time and the second time) respectively Moment) corresponding to the update time stamp (i.e., the update parameter), it is possible to quickly determine which specific storage data in the sub-database is updated, and further, the storage data updated in the sub-database is used as the target storage data.

It can be understood that, when the data volume of a certain sub-database at the second time is equal to the data volume of the certain sub-database at the first time, whether each storage parameter is updated (if yes, whether modification occurs) can be further determined by continuing to use the update time stamps respectively corresponding to each storage data in the sub-database at the second time and the first time.

Specifically, what parameters the first update parameter and the second update parameter include may be determined according to an actual application scenario, as long as whether the stored data in the source database is updated between the first time and the second time and which storage parameters are updated can be determined by the first update parameter and the second update parameter.

And step S104, synchronizing the target storage data partition to the target database.

Optionally, the partition synchronization for the target storage data may be performed by adding a partition identifier (partition ID) to the target storage data. Optionally, the partition identifier may be a current synchronization timestamp of the target storage data, for example, the synchronization timestamp may be a timestamp corresponding to the second time. Alternatively, optionally, the partition identifier may also be another identifier, such as a storage cluster identifier, and the partition identifier may be specifically determined according to an actual service scenario, which is not limited to this.

Specifically, the server may add a partition identifier to the target storage data, and then synchronize the target storage data added with the partition identifier to the target database. The method for the server to obtain the partition identifier of the target storage data may be as follows: the server may obtain a timestamp corresponding to the second time, and may use the timestamp corresponding to the second time as the partition identifier of the target storage data.

The server can be a background server of the data client, and the data client can be any client which can inquire the stored data synchronized by the server. Subsequently, the server can also obtain a data query request sent by the data client, the data query request can carry the partition identifier, the server can query the target storage data added with the partition identifier according to the data query request, then the queried target storage data is returned to the data client, and the data client can display the target storage data on a client page for a user to check.

Optionally, the source database may include a plurality of sub-databases, and the specific number of the plurality of sub-databases may be determined according to an actual application scenario, which is not limited to this.

Therefore, the target database for storing the target storage data in a partitioned manner may be newly generated by the server, or may be selected from the plurality of sub-databases, and if the target database is selected from the plurality of sub-databases, the target database may be any selected sub-database from the plurality of sub-databases.

Optionally, if the target database is regenerated by the server, the method for generating the target database may be: the server may obtain a target storage address in a distributed storage cluster, where the distributed storage cluster may include multiple storage clusters, each storage cluster has a corresponding storage address, and the target storage address may be selected according to a corresponding data synchronization policy, for example, the target storage address may be an address of a storage cluster with a smaller detected storage pressure.

Therefore, the server may use the storage cluster indicated by the target storage address of the plurality of storage clusters in the distributed storage cluster as the target storage cluster, and the storage address of the target storage cluster may be the target storage address.

Thus, the server may generate the target database in the target storage cluster.

It can be known from the foregoing that, since the data in the source database can be updated in real time, each stored data in the source database corresponds to the current latest update timestamp, that is, the second update timestamp can be directly obtained in real time. After the update data (e.g., the target storage data) at the current time (e.g., the second time) is synchronized, the latest update parameter of the target storage data may be recorded, that is, the first update parameter may be pre-recorded by the server, and after each data synchronization occurs, the server may update the recorded first update parameter by using the latest update parameter (e.g., the latest update timestamp) of the synchronized data (e.g., the target storage data), so that the updated first update parameter may be used as the first update parameter when the data is synchronized next time. Or, the server may directly re-use the current second update parameter as the first update parameter, and re-use the current second time as the first time, so as to perform data synchronization on the incremental data in the source database at the next time (e.g., at the next cycle node).

By adopting the method provided by the application, when the stored data in the source database is synchronized, only incremental data (such as the target stored data) in the source database can be synchronized without synchronizing all the stored data in the source database, so that the accuracy and efficiency of synchronizing the stored data are improved.

Moreover, when the application synchronizes the storage data, the non-partitioned storage data (e.g., the target storage data in the source database) may be synchronized into the partitioned storage data (e.g., the target storage data to which the partition identifier is added), and if the partition identifier may be a timestamp (e.g., a timestamp corresponding to the second time) when the data synchronization occurs, the data is synchronized according to the time partition, and a time of the data synchronization may be understood as a time partition, and a time of the data synchronization may synchronize the storage data updated at the time, so finally, the server may synchronize the storage data updated by each time partition (e.g., each time of the data synchronization), and a time of the data synchronization (e.g., the second time) may correspond to one time partition, so that the storage data updated by synchronizing the backup at each time partition may subsequently query the update history of each storage data For example, what data is the data of the time partition updated from the data of a certain time partition to the next time partition and what data is the data of the time partition updated from the data of the next time partition, and the storage data is stored in the partitions, so that a subsequent user can quickly inquire the storage data synchronized with the corresponding time partition according to the partition identification, and the data inquiry is more convenient and accurate.

In addition, when the target storage data is stored, a distributed storage mode can be adopted for storage (for example, the target storage data of each period node of each data synchronization period is subjected to distributed storage by adopting a distributed storage cluster, and the target storage data of each period node can be stored in different storage clusters in the distributed storage cluster), so that the use reasonability of the storage resources is improved.

Referring to fig. 6, fig. 6 is a schematic view of a scenario of data query provided in the present application. As shown in fig. 6, it is assumed that the stored data in the source database can be synchronized in 3 consecutive time partitions, where the 3 time partitions may sequentially include time partition 1, time partition 2, and time partition 3, and one time partition is a partition corresponding to the corresponding data synchronization time.

As shown in fig. 6, several storage data may be synchronized in time partition 1, and the several storage data may include storage data c 1; at time partition 2, several storage data may be synchronized, which may include storage data c 2; several storage data may be synchronized in time partition 3, which may include storage data c 3. The stored data c1, the stored data c2 and the stored data c3 are different from each other (indicating that the update time stamps corresponding to the stored data c1, the stored data c2 and the stored data c3 are different from each other), but the stored data c1, the stored data c2 and the stored data c3 have the same data identifier x, so it can be known that the stored data c2 is obtained by modifying the stored data c1, and the stored data c3 is obtained by modifying the stored data c 2. Therefore, by synchronizing the storage data updated by the respective time partitions, the process of data change can be queried. As here, the data change status of the storage data with data ID x that can be queried is modified from storage data c1 to storage data c2 to storage data c 3. And related data personnel can know the change condition of each stored data conveniently.

Referring to fig. 7, fig. 7 is a schematic flow chart of a data processing method provided in the present application. The execution subject in the embodiment of the present application may be one computer device or a computer device cluster formed by a plurality of computer devices. The computer equipment can be a server or terminal equipment. Therefore, the execution subject in the embodiment of the present application may be a server, or may be a terminal device, or may be a server and a terminal device, and the execution subject in the embodiment of the present application is described as an example of the server. As shown in fig. 7, the method may include:

step S201, acquiring a first update parameter of a source database at a first time; the source database comprises M pieces of storage data at a first moment, wherein M is a positive integer; the first update parameter includes update parameters corresponding to the M pieces of storage data at the first time.

Step S202, acquiring a second updating parameter of the source database at a second moment; the source database comprises N pieces of storage data at a second moment, wherein N is a positive integer; the second updating parameter comprises updating parameters corresponding to the N pieces of storage data at a second moment; the first time is earlier than the second time.

In step S203, if the first update parameter is inconsistent with the second update parameter, target storage data updated based on the M pieces of storage data is determined from the N pieces of storage data.

Optionally, the detailed description of steps S201 to S203 may refer to the detailed description of steps S101 to S103 in the embodiment corresponding to fig. 3.

Step S204, classifying the target storage data with the same data structure in the plurality of pieces of target storage data to obtain L storage data sets; a storage data set contains at least one piece of target storage data.

Alternatively, there may be a plurality of target storage data, and target storage data having the same field structure may be regarded as having the same data structure, where target storage data having the same field structure may refer to target storage data having the same field type.

Therefore, the server can identify the field structure of each target storage data, and can group (i.e., classify) the target storage data having the same field structure into the same storage data set, and further group the target storage data into L storage data sets, where L is a positive integer, a specific value of L is determined according to an actual application scenario, and one storage data set includes at least one target storage data. At least one target storage data included in one storage data set has the same data structure.

Referring to fig. 8, fig. 8 is a schematic view of a data classification scenario provided in the present application. As shown in fig. 8, the plurality of target storage data may include target storage data 1 to target storage data n, the server may identify a field structure of each target storage data, classify the target storage data having the same field structure into the same storage data set, and finally classify to obtain L storage data sets, where one storage data set may include at least one target storage data, and the data structures of the target storage data included in one storage data set are the same.

Step S205, respectively performing data merging on the target storage data in each storage data set to obtain a merged data table corresponding to each storage data set.

Optionally, the server may perform data merging (i.e., performing table merging) on the target storage data in each storage data set to obtain a merged data table corresponding to each storage data set, where one storage data set corresponds to one merged data table, and one merged data table corresponding to one storage data set is obtained by performing table merging on the target storage data included in the storage data set.

And step S206, synchronizing the merged data table partition corresponding to each storage data set to the target database.

Optionally, the server may synchronize the merged data table partition corresponding to each storage data set to the target database, for example, the server may add a partition identifier to the merged data table corresponding to each storage data set, and further synchronously store each merged data table added with the partition identifier to the target database.

In the method and the device, the target storage data with the same data structure are subjected to partitioning synchronization after being combined, the synchronized storage data can be classified, and subsequently, all the synchronized storage data containing the field can be inquired through a certain specified field, so that the efficiency and the accuracy of data inquiry are improved.

Referring to fig. 9, fig. 9 is a schematic diagram of a data synchronization framework provided in the present application. The application can provide 7 data processing modules to realize the synchronization of the stored data, and the 7 data processing modules can comprise a main database module, a data monitoring module, a data updating module, a single-database table data updating module, a multi-database table data updating module, a data synchronization module and a target database module.

Specifically, the master database module may be configured to provide a master database, where the master database is the source database, and the master database module may also be referred to as a data source module, which is an initial data source for data synchronization, and includes a database and data tables under each database (such as the above sub-databases), and is used for operations such as storage and modification of the database/table. The master database module typically stores the contained databases/tables in a distributed fashion across multiple data clusters by type, purpose, function, etc. For database tables with large storage amount, the database tables are stored into a plurality of clusters in a sub-database and sub-table mode, so that the pressure of data storage is relieved. The main database is mainly a database without partitions, such as mysql (a relational database management system) database, Oracle (a relational database management system) database, and the like. The master database module primarily outputs raw data, i.e. the storage data comprised by the master database.

The master database module may input the storage data in the source database to the data monitoring module, and the data monitoring module may be configured to monitor a data update condition of the master database according to the input storage data, and when a data update occurs (for example, there is storage data updated between a first time and a second time), the data monitoring module may store a time of data change, count and store the number (for example, the number) of the change data (and the update data, that is, the storage data updated, such as the target storage data), the database where the change data is stored, the database where the data table is stored, and the cluster id where the change data is located (for example, an identifier of a storage cluster), and a cluster address (for example, a storage address of the storage cluster).

The data update module may be configured to input data (e.g., the N storage data) of the master database during the t +1 period (e.g., the t +1 th period node). The data monitoring module can automatically compare with the changed data record (which can contain the first updating parameter) in the t-th period, and if the changed data record is changed, the data updating module can be informed to extract corresponding changed data (such as extraction target stored data) from the N stored data according to the changed data time (such as an updating time stamp), the cluster address where the changed data is located, the database where the changed data is located, the cluster address where the data table is located, the log and other parameters. The extraction method can be that the change data of the partial time interval is extracted according to the time interval formed from the maximum change time stored in the period t to the change time appearing in the period t + 1.

The single database table data updating module can be used for inputting the change data extracted by the data updating module, if the change data is from the single database table, the change data with the same data structure does not exist among the extracted change data, and all the change data are directly sent to the data synchronization module.

The multi-database table data updating module can be used for inputting the change data extracted by the data updating module, and if the change data is from a plurality of database tables, the change data with the same data structure can be merged to generate a corresponding data table (such as the merged data table), and then the merged data table is sent to the data synchronization module.

The data synchronization module may add a partition identifier to the obtained changed data (e.g., a single changed data or a data table generated by combining changed data), generate a target database in the target storage cluster, and synchronize the changed data added with the partition identifier to the target database.

The target database module may include a target database synchronized with changed data carrying partition identifiers, and may return update parameters (such as a current latest update timestamp) for the changed data to the data monitoring module, and may also return parameters of the changed data counted and stored by the diversity groups, such as a database in which the changed data is stored, a database table, a cluster id (an identifier of a storage cluster), a cluster address, and a log, to the data monitoring module, so that the data monitoring module records the latest parameters of the changed data, and may continue to synchronize the changed data in the source database next time according to the latest parameters of the changed data and recorded relevant parameters of other stored data that are not updated.

By adopting the method provided by the application, the data updating synchronization and the table merging problem of a plurality of data tables of a plurality of databases can be realized, and the data synchronization and data analysis work efficiency of data personnel can be effectively improved.

The method and the device can be suitable for incremental synchronization among single databases such as mysql, Oracle, hive (a data warehouse tool), hbase (a distributed and column-oriented open source database) and the like, and can also realize cross-database type incremental synchronization from the mysql database to the Oracle database, the Oracle database to the hive database, or from the mysql database to the hive database and the like. The synchronization scenarios for the data of the database are more extensive.

Referring to fig. 10, fig. 10 is a schematic structural diagram of a data processing apparatus provided in the present application. The data processing apparatus may be a computer program (including program code) running on a computer device, for example, the data processing apparatus is an application software, and the data processing apparatus may be configured to execute corresponding steps in the methods provided by the embodiments of the present application. As shown in fig. 10, the data processing apparatus 1 may include: an acquisition module 11, a determination module 12 and a synchronization module 13.

The obtaining module 11 is configured to obtain a first update parameter of a source database at a first time; the source database comprises M pieces of storage data at a first moment, wherein M is a positive integer; the first updating parameters comprise updating parameters corresponding to the M pieces of storage data at a first moment;

the obtaining module 11 is configured to obtain a second update parameter of the source database at a second time; the source database comprises N pieces of storage data at a second moment, wherein N is a positive integer; the second updating parameter comprises updating parameters corresponding to the N pieces of storage data at a second moment; the first time is earlier than the second time;

a determining module 12, configured to determine, if the first update parameter is inconsistent with the second update parameter, target storage data updated based on the M pieces of storage data from the N pieces of storage data;

and the synchronization module 13 is configured to synchronize the target storage data partition into the target database.

if the first update parameter is inconsistent with the second update parameter, the determining module 12 determines, from the N pieces of storage data, a manner of updating the target storage data based on the M pieces of storage data, where the manner includes:

Optionally, the manner for the synchronization module 13 to synchronize the target storage data partition to the target database includes:

adding a partition identifier to target storage data;

Optionally, the manner in which the synchronization module 13 adds the partition identifier to the target storage data includes:

acquiring a timestamp corresponding to the second moment;

Optionally, the apparatus 1 is further configured to:

Optionally, the source database includes a plurality of sub-databases;

the way for the synchronization module 13 to synchronize the target storage data partition into the target database includes:

Optionally, the manner in which the synchronization module 13 generates the target database includes:

a target database is generated in the target storage cluster.

Optionally, the number of the target storage data is multiple;

the method for classifying the target storage data having the same data structure in the plurality of pieces of target storage data by the synchronization module 13 to obtain L storage data sets includes:

Optionally, the apparatus 1 is further configured to:

acquiring a data synchronization period aiming at a source database;

According to an embodiment of the present application, the steps involved in the data processing method shown in fig. 3 may be performed by respective modules in the data processing apparatus 1 shown in fig. 10. For example, step S101 shown in fig. 3 may be performed by the obtaining module 11 in fig. 10, and step S102 shown in fig. 3 may be performed by the obtaining module 11 in fig. 10; step S103 shown in fig. 3 may be performed by the determination module 12 in fig. 10, and step S104 shown in fig. 3 may be performed by the synchronization module 13 in fig. 10.

The method includes the steps that a first updating parameter of a source database at a first moment is obtained; the source database comprises M pieces of storage data at a first moment, wherein M is a positive integer; the first updating parameters comprise updating parameters corresponding to the M pieces of storage data at a first moment; acquiring a second updating parameter of the source database at a second moment; the source database comprises N pieces of storage data at a second moment, wherein N is a positive integer; the second updating parameter comprises updating parameters corresponding to the N pieces of storage data at a second moment; the first time is earlier than the second time; if the first updating parameter is inconsistent with the second updating parameter, determining target storage data obtained by updating based on M pieces of storage data from the N pieces of storage data; and synchronizing the target storage data partition to the target database. Therefore, the device provided by the application can acquire the incremental data (such as target storage data) in the source database through the first update parameter and the second update parameter, wherein the incremental data is the data which is updated, and further only the incremental data needs to be synchronized, so that the data synchronization efficiency is high. In addition, the accuracy of synchronizing the storage data is also improved by performing partition synchronization (e.g., synchronization according to time partitions) on the target storage data.

According to an embodiment of the present application, each module in the data processing apparatus 1 shown in fig. 10 may be respectively or entirely combined into one or several units to form the unit, or some unit(s) therein may be further split into multiple sub-units with smaller functions, which may implement the same operation without affecting implementation of technical effects of the embodiment of the present application. The modules are divided based on logic functions, and in practical application, the functions of one module can be realized by a plurality of units, or the functions of a plurality of modules can be realized by one unit. In other embodiments of the present application, the data processing apparatus 1 may also include other units, and in practical applications, the functions may also be implemented by being assisted by other units, and may be implemented by cooperation of a plurality of units.

According to an embodiment of the present application, the data processing apparatus 1 as shown in fig. 10 may be constructed by running a computer program (including program codes) capable of executing the steps involved in the corresponding method as shown in fig. 3 on a general-purpose computer device such as a computer including a processing element such as a Central Processing Unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), and a storage element, and implementing the data processing method of the embodiment of the present application. The computer program may be recorded on a computer-readable recording medium, for example, and loaded into and executed by the computing apparatus via the computer-readable recording medium.

Referring to fig. 11, fig. 11 is a schematic structural diagram of a computer device provided in the present application. As shown in fig. 11, the computer device 1000 may include: the processor 1001, the network interface 1004, and the memory 1005, and the computer device 1000 may further include: a user interface 1003, and at least one communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display) and a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a standard wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 1005 may optionally be at least one memory device located remotely from the processor 1001. As shown in fig. 11, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a device control application program.

In the computer device 1000 shown in fig. 11, the network interface 1004 may provide a network communication function; the user interface 1003 is an interface for providing a user with input; and the processor 1001 may be used to invoke a device control application stored in the memory 1005 to implement:

and synchronizing the target storage data partition to the target database.

In a possible implementation, any one of the M pieces of storage data is represented as the ith piece of storage data, i is a positive integer smaller than or equal to M, any one of the N pieces of storage data is represented as the jth piece of storage data, j is a positive integer smaller than or equal to N, and the processor 1001 may be further configured to call a device control application program stored in the memory 1005 to implement:

In one possible implementation, the processor 1001 may also be configured to invoke a device control application stored in the memory 1005 to implement:

adding a partition identifier to target storage data;

acquiring a timestamp corresponding to the second moment;

In one possible embodiment, the source database contains a plurality of sub-databases; the processor 1001 may also be used to invoke a device control application stored in the memory 1005 to implement:

a target database is generated in the target storage cluster.

In one possible embodiment, the number of pieces of target storage data is plural; the processor 1001 may also be used to invoke a device control application stored in the memory 1005 to implement:

In one possible embodiment, target storage data having the same field structure have the same data structure; the processor 1001 may also be used to invoke a device control application stored in the memory 1005 to implement:

acquiring a data synchronization period aiming at a source database;

It should be understood that the computer device 1000 described in this embodiment of the present application may perform the description of the data processing method in the embodiment corresponding to fig. 3, and may also perform the description of the data processing apparatus 1 in the embodiment corresponding to fig. 10, which is not described herein again. In addition, the beneficial effects of the same method are not described in detail.

Further, here, it is to be noted that: the present application further provides a computer-readable storage medium, and the computer-readable storage medium stores the aforementioned computer program executed by the data processing apparatus 1, and the computer program includes program instructions, and when the processor executes the program instructions, the description of the data processing method in the embodiment corresponding to fig. 3 can be performed, so that details are not repeated here. In addition, the beneficial effects of the same method are not described in detail. For technical details not disclosed in the embodiments of the computer storage medium referred to in the present application, reference is made to the description of the embodiments of the method of the present application.

By way of example, the program instructions described above may be executed on one computer device, or on multiple computer devices located at one site, or distributed across multiple sites and interconnected by a communication network, which may comprise a blockchain network.

The computer readable storage medium may be the data processing apparatus provided in any of the foregoing embodiments or an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The computer readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) card, a flash card (flash card), and the like, provided on the computer device. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the computer device. The computer-readable storage medium is used for storing the computer program and other programs and data required by the computer device. The computer readable storage medium may also be used to temporarily store data that has been output or is to be output.

A computer program product or computer program is provided that includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and executes the computer instruction, so that the computer device performs the description of the data processing method in the embodiment corresponding to fig. 3, which is described above, and therefore, the description thereof will not be repeated here. In addition, the beneficial effects of the same method are not described in detail. For technical details not disclosed in embodiments of the computer-readable storage medium referred to in the present application, reference is made to the description of embodiments of the method of the present application.

The terms "first," "second," and the like in the description and in the claims and drawings of the embodiments of the present application are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "comprises" and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or apparatus that comprises a list of steps or elements is not limited to the listed steps or modules, but may alternatively include other steps or modules not listed or inherent to such process, method, apparatus, product, or apparatus.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The method and the related apparatus provided by the embodiments of the present application are described with reference to the flowchart and/or the structural diagram of the method provided by the embodiments of the present application, and each flow and/or block of the flowchart and/or the structural diagram of the method, and the combination of the flow and/or block in the flowchart and/or the block diagram can be specifically implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block or blocks of the block diagram. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block or blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block or blocks.

The above disclosure is only for the purpose of illustrating the preferred embodiments of the present application and is not to be construed as limiting the scope of the present application, so that the present application is not limited thereto, and all equivalent variations and modifications can be made to the present application.

Claims

1. A method of data processing, the method comprising:

acquiring a first updating parameter of a source database at a first moment; the source database comprises M pieces of storage data at the first moment, wherein M is a positive integer; the first updating parameters comprise updating parameters respectively corresponding to the M pieces of storage data at the first moment;

acquiring a second updating parameter of the source database at a second moment; the source database comprises N pieces of storage data at the second moment, wherein N is a positive integer; the second updating parameters comprise updating parameters respectively corresponding to the N pieces of storage data at the second moment; the first time is earlier than the second time;

if the first updating parameter is inconsistent with the second updating parameter, determining target storage data updated based on the M pieces of storage data from the N pieces of storage data;

and synchronizing the target storage data partition to a target database.

2. The method according to claim 1, wherein each of the N pieces of storage data and the M pieces of storage data has a corresponding data identifier, and the updated storage data is the same as the data identifier corresponding to the storage data before updating; the update parameters respectively corresponding to the M pieces of storage data included in the first update parameter are update timestamps respectively corresponding to the M pieces of storage data at the first time; the update parameters respectively corresponding to the N pieces of storage data included in the second update parameter are update timestamps respectively corresponding to the N pieces of storage data at the second time.

3. The method according to claim 2, wherein any one of the M pieces of storage data is represented as the ith piece of storage data, i is a positive integer smaller than or equal to M, any one of the N pieces of storage data is represented as the jth piece of storage data, and j is a positive integer smaller than or equal to N;

if the first update parameter is inconsistent with the second update parameter, determining target storage data updated based on the M pieces of storage data from the N pieces of storage data, including:

if the ith piece of storage data and the jth piece of storage data have the same data identification, and the update time stamp corresponding to the ith piece of storage data at the first moment is not consistent with the update time stamp corresponding to the jth piece of storage data at the second moment by comparison, determining the jth piece of storage data as the target storage data;

and determining the data identifier of the jth piece of storage data as a target data identifier, and if the data identifier of the jth piece of storage data does not contain the update time stamp of the storage data with the target data identifier at the first moment, determining the jth piece of storage data as the target storage data.

4. The method of claim 1, wherein synchronizing the target stored data partition into a target database comprises:

adding a partition identification to the target storage data;

synchronizing the target storage data added with the partition identification into the target database.

5. The method of claim 4, wherein the adding a partition identification to the target storage data comprises:

acquiring a timestamp corresponding to the second moment;

and determining the timestamp corresponding to the second moment as the partition identification of the target storage data.

6. The method of claim 4, further comprising:

acquiring a data query request sent by a data client; the data query request carries the partition identification;

inquiring the target storage data added with the partition identification according to the data inquiry request;

7. The method of claim 1, wherein the source database comprises a plurality of sub-databases;

the synchronizing the target storage data partition into a target database includes:

generating the target database, and synchronizing the target storage data partition to the target database; alternatively, the first and second electrodes may be,

and selecting the target database from the plurality of sub-databases, and synchronizing the target storage data to the target database.

8. The method of claim 7, wherein the generating the target database comprises:

determining a storage cluster indicated by the target storage address in the plurality of storage clusters as a target storage cluster;

generating the target database in the target storage cluster.

9. The method of claim 1, wherein the target storage data is plural in number;

10. The method of claim 9, wherein target storage data having the same field structure have the same data structure;

the classifying the target storage data with the same data structure in the plurality of pieces of target storage data to obtain L storage data sets includes:

identifying a field structure of each of the plurality of pieces of target storage data;

and classifying the target storage data with the same field structure in the target storage data into the same storage data set to obtain the L storage data sets.

11. The method of claim 1, further comprising:

acquiring a data synchronization period aiming at a source database;

determining the second moment according to the data synchronization period and the first moment; the first time and the second time are separated by the data synchronization period.

12. A data processing apparatus, characterized in that the apparatus comprises:

the acquisition module is used for acquiring a first updating parameter of a source database at a first moment; the source database comprises M pieces of storage data at the first moment, wherein M is a positive integer; the first updating parameters comprise updating parameters respectively corresponding to the M pieces of storage data at the first moment;

the obtaining module is configured to obtain a second update parameter of the source database at a second time; the source database comprises N pieces of storage data at the second moment, wherein N is a positive integer; the second updating parameters comprise updating parameters respectively corresponding to the N pieces of storage data at the second moment; the first time is earlier than the second time;

a determining module, configured to determine, if the first update parameter is inconsistent with the second update parameter, target storage data updated based on the M pieces of storage data from the N pieces of storage data;

and the synchronization module is used for synchronizing the target storage data partition to a target database.

13. A computer program product comprising computer programs/instructions which, when executed by a processor, carry out the steps of the method of any one of claims 1 to 11.

14. A computer arrangement comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to carry out the steps of the method according to any one of claims 1-11.

15. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program adapted to be loaded by a processor and to perform the method of any of claims 1-11.