CN113656427B

CN113656427B - Data synchronization method and related equipment

Info

Publication number: CN113656427B
Application number: CN202111218913.7A
Authority: CN
Inventors: 赵孔胜; 呼美玲; 王树青; 舒攀; 陈凯丰; 何志敏
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2021-10-20
Filing date: 2021-10-20
Publication date: 2022-02-08
Anticipated expiration: 2041-10-20
Also published as: CN113656427A

Abstract

The disclosure provides a data synchronization method and related equipment, and belongs to the technical field of computers. The method comprises the following steps: acquiring data change information of a source system, wherein the source system comprises at least two source objects; determining a target source object with data change in a source system from at least two source objects of the source system according to data change information of the source system; determining a target object corresponding to a target source object in a source system in a target system, wherein the target system comprises at least one target object, the at least one target object comprises the target object, and the target object comprises one or more of at least two source objects; converting a target source field with data change in a target source object into a target destination field in a target destination object; and writing the target destination field into a target source object in the destination system. By utilizing the scheme provided by the embodiment of the disclosure, the efficiency of data synchronization can be improved.

Description

Data synchronization method and related equipment

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data synchronization method and apparatus, a computer-readable storage medium, an electronic device, and a computer program product.

Background

The internet background service system has the characteristics of complex function, long link, fast iteration and fast data scale increase, one part of input data of a user is generally consumed and processed by a plurality of downstream systems, and due to different related services and different data models, source data input by the user needs to be subjected to various processing conversions and stored as required, and at the moment, source data distribution synchronization is required. In addition, data synchronization is also needed in scenes such as data backup, read-write separation, new and old system switching and the like.

Therefore, the requirement of data synchronization is very common, and for a business system, on the premise of ensuring the functional integrity and the final data consistency, low delay is a key index for evaluating data synchronization service.

However, in the data synchronization scheme in the related art, the data synchronization link is complicated and long, and needs to be executed serially following the service invocation relationship, so that the data synchronization delay is uncontrollable, and the efficiency of data synchronization is affected.

It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure.

Disclosure of Invention

The embodiment of the disclosure provides a data synchronization method and device, a computer readable storage medium, an electronic device and a computer program product, which can improve the efficiency of data synchronization.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.

The embodiment of the disclosure provides a data synchronization method, which includes: acquiring data change information of a source system, wherein the source system comprises at least two source objects; determining a target source object with data change in the source system from at least two source objects of the source system according to the data change information of the source system; determining a target destination object corresponding to a target source object in a source system in a destination system, wherein the destination system comprises at least one destination object, the at least one destination object comprises the target destination object, and the target destination object comprises one or more of the at least two source objects; converting a target source field with data change in the target source object into a target destination field in the target destination object; and writing the target destination field into the target source object in the destination system.

The disclosed embodiment provides a data synchronization device, which comprises: the system comprises a data change information acquisition unit, a data change information acquisition unit and a data change information acquisition unit, wherein the data change information acquisition unit is used for acquiring data change information of a source system, and the source system comprises at least two source objects; a data change object determining unit, configured to determine, according to data change information of the source system, a target source object in which data change occurs in the source system from among at least two source objects of the source system; a target destination object determining unit, configured to determine a target destination object corresponding to a target source object in a source system in the destination system, where the destination system includes at least one destination object, the at least one destination object includes the target destination object, and the target destination object includes one or more of the at least two source objects; a target object field conversion unit, configured to convert a target source field in the target source object, where data change occurs, into a target destination field in the target destination object; and the target destination field writing unit is used for writing the target destination field into the target source object in the destination system.

The disclosed embodiments provide a computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the data synchronization method as in the above embodiments.

An embodiment of the present disclosure provides an electronic device, including: at least one processor; a storage device configured to store at least one program that, when executed by at least one processor, causes the at least one processor to implement the data synchronization method as in the above embodiments.

According to an aspect of the application, a computer program product or computer program is provided, comprising computer instructions, the computer instructions being stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various alternative implementations of the embodiments described above.

In the technical solutions provided by some embodiments of the present disclosure, when a source system and a destination system perform data synchronization, the source system and the destination system are disassembled to an object hierarchy, an object in the source system is referred to as a source object, an object in the destination system is referred to as a destination object, and a corresponding relationship between each source object and each destination object is determined, so that when data change information of the source system is acquired, a target source object with data change in the source system is determined from at least two source objects of the source system according to the data change information of the source system, then a target destination object in the destination system corresponding to the target source object in the source system is determined according to the corresponding relationship between each source object and each destination object, and a target source field with data change in the target source object is converted into a target field in the target destination object, and writing the target destination field into the target source object in the destination system, thereby realizing field conversion of an object level between the source system and the destination system, and thus, when the target destination object comprises a plurality of source objects in at least two source objects, realizing parallel calculation and parallel writing of data mapping conversion, simplifying a data synchronization process and improving data synchronization efficiency.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.

Fig. 1 is a schematic view of an application scenario of data synchronization provided in an embodiment of the present disclosure.

Fig. 2 schematically shows a flow chart of a data synchronization method according to an embodiment of the present disclosure.

Fig. 3 is a diagram illustrating a data synchronization scheme in the related art.

FIG. 4 schematically shows a schematic diagram of a data synchronization system according to an embodiment of the present disclosure.

Fig. 5 schematically shows a schematic diagram of a data synchronization method according to an embodiment of the present disclosure.

Fig. 6 schematically shows a schematic diagram of a data synchronization method according to a further embodiment of the present disclosure.

Fig. 7 schematically shows a schematic diagram of a data synchronization method according to another embodiment of the present disclosure.

Fig. 8 schematically shows a block diagram of a data synchronization apparatus according to an embodiment of the present disclosure.

FIG. 9 shows a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus, a repetitive description thereof will be omitted.

The described features, structures, or characteristics of the disclosure may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the subject matter of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and the like. In other instances, well-known methods, devices, implementations, or operations have not been shown or described in detail to avoid obscuring aspects of the disclosure.

The drawings are merely schematic illustrations of the present disclosure, in which the same reference numerals denote the same or similar parts, and thus, a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in at least one hardware module or integrated circuit, or in different networks and/or processor means and/or microcontroller means.

The flow charts shown in the drawings are merely illustrative and do not necessarily include all of the contents and steps, nor do they necessarily have to be performed in the order described. For example, some steps may be decomposed, and some steps may be combined or partially combined, so that the actual execution sequence may be changed according to the actual situation.

In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element/component/etc.; the terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements/components/etc. other than the listed elements/components/etc.; the terms "first," "second," and "third," etc. are used merely as labels, and are not limiting on the number of their objects.

In the embodiment of the present application, please refer to fig. 1, and fig. 1 is a schematic view of an application scenario of data synchronization provided in the embodiment of the present disclosure. The method and the apparatus provided by the embodiment of the application can be implemented by the computer device 102, the computer device 102 can provide a data synchronization service, and the computer device 102 can perform data interaction with the computer device 101 and the computer device 103. The computer device 102 may obtain data change information of a source system from the computer device 101, the computer device 101 may include the source system, the source system may include at least two source objects, and the computer device 102 may determine, according to the data change information of the source system, a target source object in the source system, where data change occurs, from the at least two source objects of the source system; determining a target object corresponding to a target source object in a source system in a target system, wherein the target system comprises at least one target object, the at least one target object comprises the target object, and the target object comprises one or more of at least two source objects; the target source field in the target source object, in which the data change occurs, is converted into a target destination field in the target destination object, and the computer device 102 can write the target destination field into the target source object in the destination system in the computer system 103, so as to implement synchronization of the data in the source system in the computer device 101 to the destination system in the computer device 103.

The computer device 101, the computer device 103, and/or the computer device 102 may be a terminal device, a server, or a system composed of a server and a terminal device, and are not limited herein.

The above-mentioned terminal device may be an electronic device, including but not limited to a mobile phone, a tablet computer, a desktop computer, a notebook computer, a palm computer, a vehicle-mounted terminal, an Augmented Reality/Virtual Reality (AR/VR) device, a helmet display, a smart television, a wearable device, a smart speaker, a digital camera, a camera, and other Mobile Internet Devices (MID) with network access capability, or a terminal device in a scene of train, ship, flight, and the like.

The above-mentioned server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, Network service, cloud communication, middleware service, domain name service, security service, vehicle-road cooperation, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.

As shown in fig. 2, a data synchronization method provided by an embodiment of the present disclosure may include the following steps.

In step S210, data change information of a source system is obtained, where the source system includes at least two source objects.

The source system in the embodiment of the present disclosure is a system that stores data, changes data, records data change information, and synchronizes the data change information to the destination system, and the destination system is a system that synchronizes data in the source system according to the data change information of the source system. In some embodiments, the source system and the destination system may both be databases, i.e., the data synchronization at this time is a database-to-database data synchronization. In other embodiments, the source system may be a database and the destination system may be a memory, i.e. the data synchronization in this case is data synchronization between the database and the memory. The present disclosure is not limited to the type of source system and destination system, and may be any two systems that require data synchronization. In the following description, the source system is taken as a service a database, and the destination system is taken as a service B database, which are used as examples, and service a and service B may refer to any two different service systems.

In an exemplary embodiment, obtaining data change information of the source system may include: subscribing to a binary log file (binlog, shorthand for binary log) topic (topic) of a publish-subscribe messaging system; and receiving a notification message published by the publish-subscribe message system, where the notification message may carry a binary log file, and the binary log file may include data change information of the source system.

Wherein the publish-subscribe message system may be configured to publish the binary log file obtained from a synchronized incremental data tool.

In the following description, the publish-subscribe messaging system is exemplified by Pulsar, but the present disclosure is not limited thereto, and any other suitable publish-subscribe messaging system may be adopted, such as any one of Kafka and RabbitMQ (a message-oriented middleware), etc.

A synchronization delta data tool may be used to receive the binary log file from the source system and send the binary log file to the publish-subscribe message system.

In the embodiment of the present disclosure, taking the service a database as MySQL (a relational database management system) as an example, the synchronous incremental data tool may receive binlog from MySQL in real time based on the principle of primary and secondary MySQL replication, and analyze the binlog into structured data to discover writing of source data.

In the following description, the incremental synchronization data tool is exemplified by using Canal (which is interpreted as a water channel/pipe/trench, and may provide incremental data subscription and consumption based on MySQL database incremental log parsing), but the present disclosure is not limited thereto, and for example, the incremental synchronization data tool may use any system capable of implementing MySQL binlog real-time synchronization function.

The method comprises the steps that the client masquerades the client into a MySQL Slave (namely, a MySQL backup or MySQL Slave library, at the moment, a business A database can be called as a MySQL Master (Master MySQL or MySQL Master library)), the interactive protocol of the client simulation MySQL Slave sends a dump request based on the dump protocol to the MySQL Master, the MySQL Master receives the dump request sent by the client, starts to push a binlog to the client, then the client analyzes the binlog and sends the binlog to the pulser, and therefore data change in the business A database can be monitored. However, in the embodiment of the present disclosure, monitoring whether there is data change in the service a database is not limited to the above-mentioned exemplary manner, and in other embodiments, when there is data change in the service a, in addition to writing to the local database (i.e., the service a database), it may also double-write data to change to a file system or a running water change table, and by polling data in the file system or the running water change table, when it is found that there is new service data written, it is monitored that there is data change in the system a.

In the above embodiment, the system B senses that the system a has data change, which is exemplified by using a publish/subscribe mode, that is, when new data is written in the service a database, the new data is sent to topic according to a general format, and then the system B subscribes to the binlog topic in an MQ (Message Queue, here, Pulsar is used as an example) to obtain binlog data after Message parsing, thereby implementing that the system B senses that the system a has data change. However, the present disclosure is not limited thereto, and in other embodiments, the polling mode may also be used to implement that system B perceives that system a has data change, for example, system B always actively queries the latest position of binlog, and system B perceives that system B has new data written into it immediately.

In the embodiment of the present disclosure, each field in the source system is divided into corresponding objects, each field in the destination system is divided into corresponding objects, an object in the source system is referred to as a source object, and an object in the destination system is referred to as a destination object.

In step S220, a target source object with data change in the source system is determined from at least two source objects in the source system according to the data change information of the source system.

In an exemplary embodiment, the data change information may include a change object name (e.g., may be the name of a data table in which the data change occurred, denoted as tableName, opType) and a change object type.

In the embodiment of the present disclosure, each field in the source system is stored in the corresponding data table in the source system, and therefore, each data table in the source system may be divided into corresponding source objects, that is, fields belonging to the same source object may be stored in the same data table, and fields belonging to different source objects may be stored in different data tables, so that the name of the changed object may be, for example, the name of the data table in which the data change occurs, which is denoted by tableme, and the type of the changed object may be, for example, the type of the change operation performed on the fields in the data table denoted by tableme, which is denoted by opType, which may be, for example, any one or more of deleted data, inserted data, modified data, and the like.

In an exemplary embodiment, determining a target source object of the source system, in which a data change occurs, from among at least two source objects of the source system according to data change information of the source system may include: inquiring and acquiring a key (key) of a target source object change observer (observer) according to the change object name and the change object type; and determining a target source object change observer with data change in the source system from at least two source objects of the source system according to the key of the target source object change observer.

In the embodiment of the disclosure, a corresponding source object change observer may be set for each source object, a key may be set for each source object change observer, and the source object change observer and the corresponding key thereof are stored in association, so that when a change object name and a change object type are obtained, the key of the target source object change observer may be obtained by querying, and then the associated target source object change observer may be obtained by querying according to the key of the target source object change observer.

In step S230, a target destination object corresponding to a target source object in the source system in a destination system is determined, where the destination system includes at least one destination object, the at least one destination object includes the target destination object, and the target destination object includes one or more of the at least two source objects.

The plurality of embodiments of the present disclosure includes two or more.

In the embodiment of the present disclosure, a conversion mapping relationship between a destination object in a destination system and a source object in a source system may be stored, that is, when a target source object change observer monitors that data under a target source object is changed, a target source field in which the data change has occurred may be converted into a target destination field in which destination object (that is, a target destination object is determined) according to the conversion mapping relationship.

In step S240, the target source field of the target source object in which the data change occurs is converted into the target destination field of the target destination object.

In the embodiment of the present disclosure, when it is monitored that data change occurs in two or more source objects (i.e., the target destination object includes a plurality of source objects), converting the target source field in which the data change occurs in each source object in the target destination object into the target destination field in the corresponding target destination object may be performed in parallel.

In the embodiment of the present disclosure, when the target destination object includes one source object, the target destination object may include one or more destination objects, that is, a target source field in one source object in which data change occurs may be converted into a target destination field in a corresponding one target destination object; for example, a part of the target source fields may be converted into target destination fields in a certain target object, and another part of the target source fields may be converted into target destination fields in another target object.

In an exemplary embodiment, converting the target source field of the target source object, where the data change occurs, into the target destination field of the target destination object may include: and informing the target source object change observer to call a field conversion function of the target source object so as to convert the target source field with data change in the target source object into the target destination field in the target destination object.

In this embodiment of the present disclosure, the field conversion function of the target source object may automatically convert each target source field in which data change occurs into each target destination field in the corresponding target destination object, may map some target source fields one by one to corresponding target destination fields, and may map multiple target source fields therein to one of the target destination fields, which is not limited in this disclosure.

In an exemplary embodiment, the method may further include: respectively setting a source object change observer for each source object in the source system; each source object change observer can be used for monitoring whether data in the corresponding source object is changed or not, and calling a field conversion function of the corresponding source object when the data in the corresponding source object is monitored to be changed so as to convert a source field of the corresponding source object, in which the data is changed, into a corresponding destination field in a corresponding destination object in the destination system; the source object change observer includes the target source object change observer.

In an exemplary embodiment, the source system may store advertisement service data therein, the at least two source objects may include a schedule object (Campaign), an advertisement object (AdGroup), a Creative object (Creative), and an advertisement Creative object (Ad Creative) in the source system, and the source object change observer may include a schedule update observer (CampaignUpdateObserver), an advertisement update observer (adgroupwupdateobserver), a Creative insertion observer (Creative insertion observer), and an advertisement Creative update observer (adatupdateobserver).

In the following description, the example is given by taking the example of storing advertisement service data in the source system, but the present disclosure is not limited thereto, and the solution provided by the embodiment of the present disclosure may be applied to any type of service data, and when the type of the service data stored in the source system changes, corresponding source object, destination object, and source object change observer may be defined. For the same service data type, if the divided source object types are different, corresponding source objects, destination objects and source object change observers can be defined.

In an exemplary embodiment, the setting of the source object change observer for each source object in the source system may include: setting the plan update observer for a plan object in the source system; setting the advertisement updating observer for the advertisement object in the source system; setting the creative insertion viewer for the creative object in the source system; and setting the advertisement creative update observer for the advertisement creative object in the source system.

In an exemplary embodiment, notifying the target source object change observer to call a field conversion function of the target source object to perform conversion of the target source field of the target source object in which a data change occurs into the target destination field of the target destination object may include: when the target destination objects include a plan object, an advertisement object, a creative object, and an advertisement creative object in the source system, determining that the target source object change observer includes the plan update observer, the advertisement update observer, the creative insertion observer, and the advertisement creative update observer, and performing the following in parallel: informing the plan update observer to call a field conversion function of a plan object in the source system so as to convert the target source field of the plan object in the source system, in which data change occurs, into the target destination field in the target destination object; informing the advertisement update observer to call a field conversion function of an advertisement object in the source system so as to convert the target source field with data change in the advertisement object in the source system into the target destination field in the target destination object; notifying the creative insertion observer to call a field conversion function of the creative object in the source system to convert the target source field of the creative object in the source system with data change into the target destination field of the target destination object; notifying the advertising creative update observer to invoke a field conversion function of the advertising creative object in the source system to convert the target source field of the advertising creative object in the source system with the data change into the target destination field of the target destination object.

In an exemplary embodiment, the at least one destination object may include a plan object, an advertisement object, a creative object, and a material object in the destination system. Determining a target destination object in a destination system corresponding to a target source object in the source system may include: when the target object comprises a planning object, an advertisement object, a creative object and an advertisement creative object in the source system, determining that the planning object in the target system is the target object of the planning object in the source system, the advertisement object in the target system is the target object of the advertisement object in the source system, the creative object in the target system is the target object of the creative object in the source system, and the material object in the target system is the target object of the advertisement creative object in the source system.

The field conversion functions of the source object may include campaignOnChange (a planned object field conversion function), adGroupOnChange (an ad object field conversion function), creativeOnChange (a creative object field conversion function), and adCreativeOnChange (an ad creative object field conversion function), and assuming that a planning object in the source system corresponds to a planning object in the destination system, an advertisement object in the source system corresponds to an advertisement object in the destination system, a creative object in the source system corresponds to a creative object in the destination system, an advertisement creative object in the source system corresponds to a material object in the destination system, when the plan update observer monitors that the plan object in the source system has data change, the plan update observer executes a campaignOnChange () function to automatically convert a target source field of the plan object in the source system, which has data change, into a corresponding target destination field of the plan object in the destination system; when the advertisement updating observer monitors that the advertisement object in the source system has data change, the advertisement updating observer executes an adGroupOnChange () function, and automatically converts a target source field of the advertisement object in the source system, which has data change, into a corresponding target field in the advertisement object in the target system; when a creative insertion observer monitors that a creative object in a source system has data change, the creative insertion observer executes a creationonChange () function and automatically converts a target source field of the creative object in the source system, which has the data change, into a corresponding target destination field in a creative object in a destination system; when the advertisement creative update observer monitors that the advertisement creative object in the source system has data change, the advertisement creative update observer executes an add creationonchange () function to automatically convert the target source field of the advertisement creative object in the source system, which has data change, into the corresponding target destination field in the material object in the destination system.

In an exemplary embodiment, notifying the plan update observer to call a field conversion function of a plan object in the source system to convert the target source field in the plan object in the source system where the data change occurs into the target destination field in the target destination object may include: and informing the plan updating observer to call a field conversion function of the plan object in the source system so as to map the target source fields with data change in the plan object in the source system to the target destination fields in the plan object in the destination system one by one. However, the present disclosure does not limit the mapping between the target source field of the plan object in the source system where the data change occurs and the target destination field of the plan object in the destination system, and may determine the mapping according to actual needs, for example, may perform a many-to-one mapping.

Wherein notifying the creative insertion observer to invoke a field conversion function of the creative object in the source system to convert the target source field of the creative object in the source system with data change into the target destination field in the target destination object may include: and informing the creative insertion observer to call a field conversion function of the creative object in the source system so as to map target source fields with data change in the creative object in the source system to target destination fields in the creative object in the destination system one by one. However, the present disclosure does not limit the mapping between the target source field of the creative object in the source system where the data change occurs and the target destination field of the creative object in the destination system, and the mapping may be determined according to actual needs, for example, the mapping may be performed in a many-to-one manner.

Notifying the advertisement update observer to call a field conversion function of the advertisement object in the source system to convert the target source field in the advertisement object in the source system, where data change occurs, into the target destination field in the target destination object, may include: and informing the advertisement updating observer to call a field conversion function of the advertisement object in the source system so as to map a plurality of target source fields with data change in the advertisement object in the source system to one target destination field in the advertisement object in the destination system.

It is to be understood that, a part of the target source fields of the advertisement object in the source system where data change occurs may be mapped to the target destination fields of the advertisement object in the destination system in a many-to-one manner, and another part of the target source fields may be mapped to the target destination fields of the advertisement object in the destination system in a one-to-one manner, which is not limited by the present disclosure. Alternatively, the target source fields of the advertisement object in the source system, in which data change occurs, may be mapped to the target destination fields of the advertisement object in the destination system one-to-one.

In an exemplary embodiment, the at least one destination object may include an advertisement object and a material object in the destination system.

Determining a target destination object in a destination system corresponding to a target source object in the source system may include: when the target object comprises the advertising creative object in the source system, determining the advertising object and the material object in the target system as the target object of the advertising creative object in the source system.

Wherein notifying the advertising creative update observer to invoke a field conversion function of the advertising creative object in the source system to convert the target source field of the advertising creative object in the source system with the data change into the target destination field of the target destination object may include: notifying the advertising creative update observer to call a field conversion function of the advertising creative object in the source system to perform conversion of a portion of the target source field of the advertising creative object in the source system, where the data change occurs, into a target destination field in the material object in the destination system, and to convert another portion of the target source field into a target destination field in the advertising object in the destination system.

In some embodiments, at least some of the source objects may be mapped with destination objects in a staggered manner, for example, a portion of the target source fields in a source object may be mapped to one of the destination objects, another portion of the target source fields in the source object may be mapped to another destination object, yet another portion of the target source fields in the source object may be mapped to yet another destination object, and so on. The method is not limited in the disclosure, and can be set according to actual requirements.

In step S250, the target destination field is written into the target source object in the destination system.

In an exemplary embodiment, writing the target destination field into the target source object in the destination system may include: mapping a plurality of target source fields in an advertising creative object in the source system to a target destination field in a material object in the destination system for writing into the destination system.

In an exemplary embodiment, an advertising creative object in the source system can include creative elements and elements, the advertising creative object can be used to store style specification information for advertising material, the creative elements can be used to store an identification of the advertising material, and the elements can be used to store content for the advertising material.

Wherein mapping a plurality of target source fields in an ad creative object in the source system to a target destination field in a material object in the destination system for writing into the destination system may comprise: and writing the content of the advertisement material stored in the element into a material object of the destination system.

In an exemplary embodiment, an advertising creative object in the source system includes a plurality of elements, and a summary field is present on a material object of the destination system.

Wherein mapping a plurality of target source fields in an ad creative object in the source system to a target destination field in a material object in the destination system for writing into the destination system may comprise: and mapping the data stored in the plurality of elements to the summarizing field, wherein the summarizing field summarizes the data stored in the plurality of elements in a structure with a preset format so as to write the data into the material object of the destination system.

In an exemplary embodiment, the method may further include: recording the mapping relation between the identification of each target source object in the source system and the identification of the corresponding target object in the target system; and recording the data relation before and after the target object in the target system is changed, and storing the data relation into an object change log of the target system.

In the data synchronization method provided by the embodiment of the present disclosure, when a source system and a destination system perform data synchronization, the source system and the destination system are disassembled to an object hierarchy, objects in the source system are referred to as source objects, objects in the destination system are referred to as destination objects, and a corresponding relationship between each source object and each destination object is determined, so that when data change information of the source system is acquired, a target source object with data change in the source system is determined from at least two source objects of the source system according to the data change information of the source system, then a target destination object corresponding to the target source object in the source system is determined according to the corresponding relationship between each source object and each destination object, and a target source field with data change in the target source object is converted into a target destination field in the target destination object, and writing the target destination field into the target source object in the destination system, thereby realizing field conversion of an object level between the source system and the destination system, and thus, when the target destination object comprises a plurality of source objects in at least two source objects, realizing parallel calculation and parallel writing of data mapping conversion, simplifying a data synchronization process and improving data synchronization efficiency.

The method provided by the embodiment of the present disclosure is illustrated with reference to fig. 3 to 7, but the present disclosure is not limited thereto.

Synchronizing data from system a to system B is roughly divided into three steps: (1) the monitoring system A has data change; (2) the system B senses that the system A has data change; (3) and the system B converts the source data in the system A into model data of own business for storage. The efficiency of each link affects the overall delay of the final data synchronization.

And (3) converting the source data in the system A into model data of own service by the system B for storage, wherein generally speaking, the storage formats of different services for the data are different, especially for a system with complex service logic, the binlog data after analysis cannot directly fall into the system B, and a series of complex conversion processing is carried out. Fig. 3 shows a schematic flow chart of a data synchronization scheme in the related art, when a consumer (system B) consumes binlog data from the service a database 301 stored in the MQ 302 in its own synchronization program (e.g., the service B synchronization task 303 in fig. 3), stores it in the service B database 306, and can synchronize data already stored in the system a database 301 to the system B database 306 through a complete creation/update path of the service B service layer 305 in a manner of using a service interface call to assemble source data into a system B service interface through the service B access layer 304.

The related art scheme of fig. 3 has the following disadvantages in practice:

1) the scheme shown in fig. 3 is severely coupled with the system B in service, and the number of modules designed upstream and downstream is large, so that the data synchronization is easily affected by other system fluctuation.

2) When the scheme shown in fig. 3 is in a problem, two sides are required to be matched and positioned, and the communication cost is high and the efficiency is low.

3) When the scheme shown in fig. 3 supports new functions, a plurality of modules are changed, cross-system joint debugging serial test is needed, and the development period is long.

The scheme shown in fig. 3 directly multiplexes the service interface of the system B by using interface-level mapping, but the strong dependence on the system B and the complex and lengthy data synchronization link cause uncontrollable data synchronization delay and affect the efficiency of data synchronization.

In the following, the scheme provided by the embodiment of the present disclosure is exemplified by taking the source system and the destination system as databases, that is, the source system is referred to as a source database, and the destination system is referred to as a destination database. The embodiment of the present disclosure provides a parallel data synchronization scheme capable of implementing a DB (database) model mapping. And performing object-level mapping on the source DB and the destination DB, and calculating the field conversion relation between corresponding objects of the source DB and the destination DB to finish model translation.

Specifically, a monitoring rule is configured at each object level, for example, a source object change observer is configured for each source object, a message sent by, for example, pulser is monitored, it is detected which source object has data update/data change, a field corresponding to the source object is triggered to be converted, and a source object change observer corresponding to the source object performs general DB model conversion calculation, so as to ensure decoupling of object mapping. Furthermore, when the object is written, a plurality of source DB objects can be assembled into one target DB object to be stored, so that no change dependency among the objects of the target DB can be ensured when the DB object is mapped in a wrong layer, parallel data writing is realized, and the efficiency of data synchronization is further improved.

The scheme provided by the embodiment of the disclosure can be applied to an advertisement delivery system, and can realize efficient synchronization of advertisement service data when cross-department storage of advertisement service data and switching between a new system and an old system are performed, so that a user can create at one end and play at multiple ends, and user experience is improved. In particular, the synchronization delay is significantly reduced, enabling advertisements to be exposed more quickly and with better delivery results.

In a typical cross-department cooperative scene, the data synchronization scheme based on the DB model mapping in the embodiment of the disclosure simplifies the synchronization link, is decoupled from the advertisement putting service, and improves the stability of the synchronization function.

As shown in fig. 4, the data synchronization system may include a service a database 401 (i.e., a source system, which may also be referred to as a source database or a source DB model), a synchronization incremental data tool 402 (e.g., Canal), a publish-subscribe message system 403 (e.g., Pulsar), a data synchronization service 404, and a service B database 405 (i.e., a destination system, which may also be referred to as a destination database or a destination DB model).

The service a database 401, as the MySQL Master, records the data change information of the service a database 401 into a binary log file binlog. Compared with the related art, the data synchronization system provided in the embodiment of fig. 4 adds the data synchronization service 404 independent of the service a database 401 and the service B database 405, and the data synchronization service 404 may be implemented by the data synchronization apparatus provided in the embodiment of the present disclosure, and the data synchronization apparatus may be disposed on the computer device 102, for example.

As shown in fig. 4, the synchronized incremental data tool 402 disguises itself as a MySQL Slave, receives binlog from the service a database 401 in real time and then writes to the database tables. In order to support a plurality of databases downstream, binlog data from the incremental synchronization data tool 402 is published through a publish-subscribe message system 403 (e.g. Pulsar), then the data synchronization service 404 subscribes to the binlog topic, and data of the service a database 401 is synchronized into the service B database 405 in parallel through data conversion based on DB model mapping.

Model map decomposition is described below.

Taking advertisement service data as an example, in the application scenario, the service a database 401 stores advertisement service data, and therefore may also be referred to as a source advertisement database, and the data synchronization service 404 is configured to translate data stored in the source advertisement database into a destination advertisement database (i.e., the service B database 405). Although the advertisement delivery systems on both sides have certain differences in delivery details and DB storage structures, the exposed advertisement structures are consistent and are all composed of four-layer structures of plans, advertisements, creatives and materials. Thus, upon data synchronization, mapping may be performed according to such peer-to-peer relationship, as shown in fig. 5 in particular. Based on the definite disassembly rule, the conversion mapping relation between the field of the source advertisement database and the field of the target advertisement database is calculated in a programmed mode, and the object stored in the source advertisement database is translated to the object corresponding to the target advertisement database.

In the disclosed embodiment, it is assumed that each advertisement traffic data includes plans, advertisements, creatives, and materials.

The plan may also be referred to as a promotion plan, and creating a promotion plan may include, for example: selecting a promotion plan type; selecting a corresponding promotion target; setting a planned daily budget; set a promotion plan name (i.e., a plan set name), and so on. The selected promotion plan type can be, for example, a display advertisement plan or a search advertisement plan, the display advertisement plan can be a regular display type advertisement, and the search advertisement plan can be a browser keyword search advertisement. The corresponding promotion target can be selected according to marketing purpose, for example, a certain plan is to promote a certain APP (application). The planned daily budget is set for controlling the maximum of the sum of the daily consumption of all advertisements under the promotion plan, e.g. information about what the total budget is for promoting this or this type of APP.

Placing an advertisement may include selecting an advertisement format, selecting targeted details, selecting an advertisement presentation location, selecting a targeted demographic for placement, setting an advertisement placement time, bids and daily budget, and setting an advertisement title (setting a title for an advertisement), among others. The advertisement form can be divided into a conventional display advertisement and a commodity advertisement. The regular display advertisement corresponds to a regular placement scene. The commodity advertisement is suitable for scenes with mass commodity putting requirements, and batch putting and dynamic creative generation are performed around commodities. The selection target details may be set at an advertisement level, for example, for different promotion targets such as merchandise promotion, application promotion, mini-game promotion, store promotion, and the like. The selection of the advertisement display position is to select the traffic on which the advertisement is expected to be displayed, and can be selected in an automatic position or a designated position mode. The automatic position is that the system selects a position with better performance for putting according to the advertisement setting. The designated slot is the flow which is selected to be released by self. The target population to which the advertisement is to be delivered is selected, and the population labels with various dimensions such as demographic attributes, user behaviors, user-defined population, equipment orientation and the like are provided. Setting the advertisement placement time refers to the advertisement being presented to the user within the date and time period of placement. Bids and daily budgets can affect the number of times an advertisement is exposed.

Setting the creative may include selecting a presentation style of the advertisement on the traffic, filling in content required by the corresponding style, and may be a creative of the advertisement made based on materials provided by the advertiser, such as an advertisement style, a landing page address, video, audio, and the like. Creativity is a combined application of materials. The material is information related to the product to be advertised, information related to the advertiser, and/or the like.

As shown in fig. 5, based on the structure of the advertisement service data, the source DB model 510 may include a schedule (Campaign, promotion schedule/advertisement schedule) object 511, an advertisement (ad group) object 512, a Creative (Creative) object 513, and an advertisement Creative (ad Creative) object 514. Destination DB model 520 may include a plan (Campiagn) object 521, an ad (AdGroup) object 522, a Creative object 523, and a Material (Material) object 524.

In the FIG. 5 embodiment, there may be only one advertisement in AdGroup. The Creative object 513 may include an identifier of the Campaign object 511 (denoted as Campaign ID (abbreviation for index)), an identifier of the created object 514 (denoted as created ID), and an identifier of the created object 512 (denoted as created ID), and for a specific piece of advertisement service data, the created object 513 is generated after the Campaign object 511, the created object 512, and the created object 514 are ready.

Referring to the fig. 5 embodiment, the Campaign object 511 in the source DB model 510 corresponds to the Campaign object 521 in the destination DB model 520, i.e. when data conversion is performed, each field in the Campaign object 511 is converted into a corresponding field in the Campaign object 521. The specific fields in the Campaign object 511 may be different from the fields in the Campaign object 521, but there may be corresponding translation relationships between the different fields.

With continued reference to the FIG. 5 embodiment, various fields in the plan object 511 may have a one-to-one mapping relationship with various fields in the plan object 521, such as converting field A in the plan object 511 to field B in the plan object 521.

Referring to the FIG. 5 embodiment, ad object 512 corresponds to ad object 522. Further, the fields in ad object 512 may be in a many-to-one mapping relationship with the fields in ad object 522.

For example, there are multiple statuses on multiple fields of FSysStatus (playback system status), FApprovalStatus (playback approval status), fusersstatus (playback user status), fneedapvalstatus (playback approval required status), fdeletstatus (playback deletion status) on the AdGroup object 512 in the source DB model 510 for indicating whether the advertisement is playable, and there is only one status field on the AdGroup object 522 of the destination DB model 520 for indicating whether the advertisement is playable, for example, when the status field is normal, the advertisement is playable. In field conversion, multiple fields such as FSysStatus, FApprovalStatus, FUserStatus, fneedaprovalstatus, fdeletstatus, etc. on the AdGroup object 512 may be mapped to status fields on the AdGroup object 522.

Referring to the FIG. 5 embodiment, creative object 513 corresponds to creative object 523. Further, the fields in the creative object 513 may have a one-to-one mapping relationship with the fields in the creative object 523.

Referring to the FIG. 5 embodiment, the advertisement creative object 514 corresponds to a material object 524. Further, the fields in the advertising creative object 514 may be in a many-to-one mapping relationship with the fields in the material object 524.

For example, the additive Object 514 in the source DB model 510 stores a plurality of elements (Element 5142), such as video (video), title (title), and end page (end _ page), and the Material Object 524 of the destination DB model 520 has a summary field, such as a crinklin field, which summarizes the Material information to be played in a JSON (JavaScript Object Notation, a kind of illustration of a predetermined format), and the crinklin field of the Material Object 524 of the destination DB model 520 is exemplified as follows:

[

{

"card_info": {

"title": "123"，

"desc": [

"456"，

"789"

]，

"description": "123"，

"end _ page _ desc", where to go to see the book,

"end_page_type": 1，

}，

"short_video": {

"width": 960，

"height": 540，

"media_id": 123

}，

}

referring to the FIG. 5 embodiment, the ad creative object 514 may further include a creative element5141 and an element 5142.

The CreativeElement5141 may be, for example, an ID of a picture or a video in the material, and the element 5142 may be, for example, specific content of the picture or the video in the material, such as a URL (Uniform Resource Locator). The advertising creative object 514 is information of the style, specification and the like of the material, which is called as the style specification information of the material, but does not include the content of the specific picture or video, i.e. the advertising creative object 514, the creative element5141 and the element 5142 are in a layer-by-layer calling relationship. At the time of conversion, the ULR in element 5142 may be stored directly into Material object 524, possibly without including the ID of the picture or video.

With continued reference to FIG. 5, the advertising creative object 514 may also be hierarchically mapped to the advertising object 522. That is, fields in AdCretive object 514 in source DB model 510 may be mapped to fields in both AdGroup object 522 and Material object 524 in destination DB model 520.

For example, for a commercial advertisement, the information of the commercial in source DB model 510 is stored at the level of an AdCretive object 514, and destination DB model 520 also stores the level of an AdGroup object 522, so that the information stored on the AdCretive object 514, such as, for example, dynamic AdProduceId (Commodity ID), and dynamic ProduceAdCategory (Commodity Domain _ ID), is translated to the AdGroup object 522 of destination DB model 520, i.e., the AdCretive object 514 in source DB model 510 maps a part of the fields to the Material object 524 of destination DB model 520 and a part to the AdGroup object 522 of destination DB model 520.

The design of the data synchronization module, i.e. the specific implementation of the new synchronization link based on the DB model mapping, is then introduced. As shown in fig. 6, after subscribing to the stream change, after configuring a listening rule at each Object level, that is, after listening to an Object change (DI Object Changed, DI may be a message sent by pulser and informs that there is a data change in the data synchronization service), osbserver (which is a processing function and calls a corresponding field conversion function to perform conversion on fields in a corresponding source Object when the message informs that there is a data update of the corresponding source Object) of each Object (here, each source Object) is used as an observer to pay attention to the change message of specific data. When data is written into the source DB model 510, the corresponding oberver performs the conversion calculation of the DB model on the source object with the change, and then the data is updated to the destination DB model 520. Therefore, the mapping logic of each object is decoupled and is not correlated, and the responsibility of each object is clear, so that the maintenance is convenient.

As shown in fig. 6, for example, a plan update observer (CampaignUpdateObserver) 601 is provided for the plan object 511, an ad update observer (adgroupwateobserver) 602 is provided for the ad object 512, a Creative insertion observer (Creative inserter observer) 603 is provided for the Creative object 513, an ad Creative update observer (Creative update observer) 604 is provided for the ad Creative object 514, the CampaignUpdateObserver 601 is made to monitor whether the Campaign object 511 in the source DB model 510 has data updates, the Creative update observer 603 is made to monitor whether the Creative object 513 in the source DB model 510 has new insertions, the Creative update 604 is made to monitor whether the Creative object 513 in the source DB model 510 has data updates, and the Creative update observer object 604 is made to monitor whether the Creative object 510 has data updates in the source DB model 510, and the Creative update observer object 602 is made to monitor whether the Creative object 510 has data updates in the source DB model 510.

Each oberver has a corresponding field transfer function (), which can also be denoted as OnChange (). In the embodiment of the present disclosure, a field conversion function corresponding to CampaignUpdateObserver 601 is represented as campaignOnChange 701; the field conversion function corresponding to AdGroupUpdateObserver 602 is denoted as AdGroupOnChange 702; the field conversion function corresponding to the creativeInsertObserver 603 is denoted as creativeOnChange 703; the corresponding field transfer function of the AdCreationUpdateObserver 604 is denoted as an AdCreationOnChange 704.

For example, when CampaignUpdateObserver 601 listens that Campaign object 511 in source DB model 510 has data to update, CampaignUpdateObserver 601 executes campaignOnChange 701, converting Campaign object 511 in source DB model 510 into Campaign object 521 in destination DB model 520.

When AdGroupUpdateObserver 602 monitors that there is data update in AdGroup object 512 in source DB model 510, AdGroupUpdateObserver 602 executes AdGroupOnChange 702 to convert AdGroup object 512 in source DB model 510 into AdGroup object 522 in destination DB model 520.

When creativeInsertObserver 603 listens that a Creative object 513 in source DB model 510 has data to update, creativeInsertObserver 603 executes creativeOnChange 703, converting the Creative object 513 in source DB model 510 into a Creative object 523 in destination DB model 520.

When AdCreationUpdateObserver 604 listens that an AdCreationve object 514 in source DB model 510 has data to update, AdCreationUpdateObserver 604 executes an AdCreationve OnChange 704, converting the AdCreationve object 514 in source DB model 510 into a Material object 524 and an AdGroup object 522 in destination DB model 520.

The implementation procedure of the observer is illustrated below.

func notifyChangeOneInvoice(ctx context.Context，

invoice *qzap_invoice_pb.DataInvoice_EntityInvoice，

advertiser *client.CsAdvertiser，

mpClient *client.MpClient，traceID string) (objChangeLogs []client.ObjectChangeLog，err error) {

topParams: = topParamBox { }// variable declaration

opType := qzap_pb.Mutate_Operator(invoice.GetOpcode())

tableName := invoice.GetTblname()

key = change Key { tableName, opType }// according to change object name and type, obtain key of inquiring oberver

// store the correspondence between keys and specific servers

var changeObservers = map[changeKey]ChangeObserver{

{"Tbl_Creative_"，qzap_pb.Mutate_ADD}: &creativeInsertObserver{}，

{"Tbl_Creative_"，qzap_pb.Mutate_SET}: &creativeUpdateObserver{}，

{"Tbl_AdCreative_"，qzap_pb.Mutate_SET}: &adCreativeUpdateObserver{}，

{"Tbl_Campaign_"，qzap_pb.Mutate_SET}: &campaignUpdateObserver{}，

{"Tbl_Campaign_"，qzap_pb.Mutate_DELETE}: &campaignUpdateObserver{}，

{"Tbl_AdTarget_"，qzap_pb.Mutate_SET}: &adTargetUpdateObserver{}，

{"Tbl_AdTarget_"，qzap_pb.Mutate_DELETE}: &adTargetUpdateObserver{}，

{"Tbl_AdGroup_"，qzap_pb.Mutate_SET}: &adGroupUpdateObserver{}，

{"Tbl_AdGroup_"，qzap_pb.Mutate_DELETE}: &adGroupUpdateObserver{}，

{"Tbl_Product_"，qzap_pb.Mutate_SET}: &productUpdateObserver{}，

{"Tbl_SubordinateProduct_"，qzap_pb.Mutate_SET}: &subProductUpdateObserver{}，

}

observer, ok = changeobserver [ key ]// obtain specified change observer from key

log.DebugContextf(ctx，"key: %v，observer: %#v，ok: %v"，key，observer，ok)

if !ok {

return nil，nil

}

Each observer defines its own OnChange function, and the observers of specific changed objects are informed to execute the OnChange function and execute the mapping calculation logic

changeLogs，err := observer.OnChange(ctx，invoice，key，advertiser，mpClient，topParams)

if err != nil {

return nil，errors.Wrapf(err，"data notify failed key: %#v，invoice: %s"，key，invoice.String())

}

return changeLogs，err

}

The following describes a manner in which embodiments of the present disclosure implement intra-object series and inter-object parallel.

In the data synchronization process, two data dimensions are selected: a listening dimension and a writing dimension.

For the listening dimension, all that is needed is to discover changes to the data in source DB model 510 and translate into a corresponding destination object in destination DB model 520 for destination DB model 520 writing. Therefore, the source object of the source DB model 510 is used as a listening dimension here as a trigger condition for data synchronization.

For the write dimension, rather than splitting one destination object in destination DB model 520 into multiple source object writes in source DB model 510, multiple source objects in source DB model 510 may be selected for storage at write time, such that changes to source objects in source DB model 510 are completely decoupled at DB map level errors.

The following data synchronization process based on the DB model mapping is described by taking the synchronization of a complete piece of advertisement service data as an example.

As shown in fig. 7, when the Creative insertetworver 603 monitors that data change occurs in the Creative object 513, it may be determined that the piece of advertisement service data is complete, which indicates that data change occurs in the Creative object 511 when the campaignupdatebyerver 601 monitors that data change occurs in the Campaign object 511 before, data change occurs in the adgroupupdatebyerver 602 monitors that data change occurs in the AdGroup object 512, and data change occurs in the Creative object 514 when the Creative updatebyserver 604 monitors that data change occurs in the Creative object 514, at this time, it may start to write each source object in the source DB model 510 into a destination object in the destination DB model 520.

The campigingupdateobserver 601 monitors that data change occurs in the campiging object 511, the adgrouppdateobserver 602 monitors that data change occurs in the AdGroup object 512, the Creative observer 603 monitors that data change occurs in the Creative object 513, and the AdCreative updatebyobserver 604 monitors that data change occurs in the Creative object 514, which may be at least partially overlapped in time or completely simultaneous, that is, the campiging object 511, the AdGroup object 512, the Creative object 513, and the Creative object 514 may be updated simultaneously.

For example, as shown in fig. 7, assume that CampaignUpdateObserver 601 executes campaignOnChange 701 first, while campaignOnChange 701 has not been executed, AdGroupUpdateObserver 602 starts executing adGroupOnChange 702, while campaignOnChange 701 and adGroupOnChange 702 have not been executed, adcreataiuppateobserver 604 may start executing adcreationonchange 704, while addresinsertochange 704 may have not been executed, and creativeinspertolserver 603 may start executing creationonchange 703. In the embodiment of fig. 7, this situation may be referred to as inter-object parallelism, but is not limited thereto, and in the embodiment of the present disclosure, execution of the OnChange function corresponding to each object between the respective object servers is completely independent, and there is no dependency relationship between the precedence order of the respective object servers.

With continued reference to FIG. 7, each source object has a plurality of fields (referred to as source fields for distinction), the correspondence between the various fields in the source DB and the destination DB is known in advance and is automatically transformed by corresponding field transformation functions, and the transformations between the various fields within an object may be serialized during execution, referred to as intra-object serialization.

As shown in fig. 7, intra-object serialization may include the following steps.

In step S710, object conversion is performed.

For example, campaignOnChange 701 sequentially converts each source field in Campaign object 511 in the source DB to each destination field in Campaign object 521 in the destination DB.

The adgrouponchanges 702 in turn convert each source field in the AdGroup object 512 in the source DB to each destination field in the AdGroup object 522 in the destination DB.

creationchange703 in turn converts each source field in the Creative object 513 in the source DB into each destination field in the Creative object 523 in the destination DB.

The add creation onchange 704 converts each source field in the add object 514 in the source DB into each destination field in the Material object 524 and the add group object 522 in the destination DB in turn.

In step S720, the destination database is written.

I.e., writing each target destination field after the conversion is completed into the destination DB.

In step S730, the ID association is recorded.

In the embodiment of the present disclosure, the IDs are self-added in each DB, and the ID association relationship is a mapping relationship between the IDs in the corresponding source objects in the record source DB and the IDs in the corresponding destination objects in the destination DB.

In step S740, an object change log of the destination database is collected.

The object change log of the destination DB records the data relationship before and after the change of the destination object in the destination DB.

Based on the processes of the mapping splitting, the selection of the data writing dimension, the conversion mapping of the fields and the writing of the destination database, the embodiments of the present disclosure can achieve the serial in the objects and the parallel between the objects. That is, the Campaign object 511, the regroup object 512, the progressive object 514, and the progressive object 513 in the source DB can perform data preparation and data writing without being aware of each other, which can greatly improve the synchronization efficiency.

The method provided by the embodiment of the present disclosure, on one hand, can implement a parallel data synchronization scheme based on DB model mapping, perform translation based on DB model, calculate a conversion mapping relationship between fields of a source object in a source DB and a destination object in a destination DB, and translate the source object stored in the source DB to the destination DB. The object level mapping does not depend on a system of the service B, is decoupled among the objects, can perform conversion in parallel, does not need to follow a service calling relationship, namely for example, does not need to wait for the conversion of fields in the Campiagn object 511 before the conversion of each field in the AdGroup object 512, and can perform the conversion of each source object in parallel, thereby overcoming the problem of long calling link and realizing low delay of data synchronization. The source DB and the target DB are subjected to model mapping and are disassembled to the object level, parallel calculation and parallel writing of data mapping conversion are realized, and the service logic is moved upwards, so that the data synchronization process is simplified, and the problem of low efficiency of data synchronization in cross-department cooperation is solved. On the other hand, data mapping is directly performed through decoupling with the target system, so that data synchronization failure, backlog and other abnormalities possibly caused by target system fluctuation are avoided, and the stability of a data synchronization function is improved. Compared with the old synchronization realization based on interface mapping, the old synchronization link needs one person to maintain at the access layer of the data synchronization service and the advertisement service before, and the maintenance number can be reduced by adopting the scheme provided by the embodiment of the disclosure. Meanwhile, the online development period of the new function can be shortened, the time for writing into the target advertisement library is shortened, and the writing performance is improved.

Fig. 8 schematically shows a block diagram of a data synchronization apparatus according to an embodiment of the present disclosure. The data synchronization apparatus 800 in the embodiment of fig. 8 may include: a data change information acquisition unit 810, a data change object determination unit 820, a target destination object determination unit 830, a target object field conversion unit 840, and a target destination field writing unit 850.

The data change information acquiring unit 810 may be configured to acquire data change information of a source system, where the source system includes at least two source objects.

The data change object determination unit 820 may be configured to determine a target source object of the source system, where a data change occurs, from at least two source objects of the source system according to the data change information of the source system.

The target destination object determining unit 830 may be configured to determine a target destination object corresponding to a target source object in the source system in a destination system, where the destination system includes at least one destination object, the at least one destination object includes the target destination object, and the target destination object includes one or more of the at least two source objects.

The target object field conversion unit 840 may be configured to convert a target source field of the target source object, where a data change occurs, into a target destination field of the target destination object.

A destination field write unit 850 may be used to write the destination field into the destination source object in the destination system.

In an exemplary embodiment, the data change information may include a change object name and a change object type. The data change object determining unit 820 may include: the target source object change observer key query unit may be configured to query and obtain a key of a target source object change observer according to the change object name and the change object type; the target source object change observer determining unit may be configured to determine, from at least two source objects of the source system, a target source object change observer having data changed in the source system, based on a key of the target source object change observer.

The target object field conversion unit 840 may include: a field conversion function execution unit, configured to notify the target source object change observer to call a field conversion function of the target source object, so as to perform conversion of the target source field in the target source object, where a data change occurs, into the target destination field in the target destination object.

In an exemplary embodiment, the data synchronization apparatus 800 may further include: the source object change observer setting unit may be configured to set a source object change observer for each source object in the source system. Each source object change observer can be used to monitor whether the data in the corresponding source object is changed, and when the data in the corresponding source object is monitored to be changed, a field conversion function of the corresponding source object is called to execute conversion of the source field of the corresponding source object, in which the data is changed, into the corresponding destination field in the corresponding destination object in the destination system. The source object change observer may comprise the target source object change observer.

In an exemplary embodiment, the source system may store advertisement traffic data therein, the at least two source objects may include a plan object, an advertisement object, a creative object, and an advertisement creative object in the source system, and the source object alteration viewer may include a plan update viewer, an advertisement update viewer, a creative insertion viewer, and an advertisement creative update viewer.

Wherein the source object change observer setting unit may include: a plan update observer setting unit operable to set the plan update observer for a plan object in the source system; an advertisement update observer setting unit operable to set the advertisement update observer for the advertisement object in the source system; a creative insertion viewer setting unit, which may be configured to set the creative insertion viewer for the creative object in the source system; and the advertisement creative update observer setting unit can be used for setting the advertisement creative update observer for the advertisement creative object in the source system.

In an exemplary embodiment, the field conversion function execution unit may include: a target source object change observer determination unit operable to determine that the target source object change observer includes the planned update observer, the advertisement update observer, the creative insertion observer, and the advertisement creative update observer when the target destination object includes a planned object, an advertisement object, a creative object, and an advertisement creative object in the source system, and to perform the following operations in parallel: informing the plan update observer to call a field conversion function of a plan object in the source system so as to convert the target source field of the plan object in the source system, in which data change occurs, into the target destination field in the target destination object; informing the advertisement update observer to call a field conversion function of an advertisement object in the source system so as to convert the target source field with data change in the advertisement object in the source system into the target destination field in the target destination object; notifying the creative insertion observer to call a field conversion function of the creative object in the source system to convert the target source field of the creative object in the source system with data change into the target destination field of the target destination object; notifying the advertising creative update observer to invoke a field conversion function of the advertising creative object in the source system to convert the target source field of the advertising creative object in the source system with the data change into the target destination field of the target destination object.

In an exemplary embodiment, the at least one destination object may include a plan object, an advertisement object, a creative object, and a material object in the destination system.

The target destination object determining unit 830 may include: and the target object mapping unit may be configured to determine that the planned object in the target system is a target object of the planned object in the source system, the advertisement object in the target system is a target object of the advertisement object in the source system, the creative object in the target system is a target object of the creative object in the source system, and the material object in the target system is a target object of the advertising creative object in the source system, when the target object includes the planned object, the advertising object, the creative object and the advertising creative object in the source system.

In an exemplary embodiment, the target source object change observer determination unit may be further operable to: and informing the plan updating observer to call a field conversion function of the plan object in the source system so as to map the target source fields with data change in the plan object in the source system to the target destination fields in the plan object in the destination system one by one.

Wherein the target source object change observer determining unit may be further configured to: and informing the creative insertion observer to call a field conversion function of the creative object in the source system so as to map target source fields with data change in the creative object in the source system to target destination fields in the creative object in the destination system one by one.

Wherein the target source object change observer determining unit may be further configured to: and informing the advertisement updating observer to call a field conversion function of the advertisement object in the source system so as to map a plurality of target source fields with data change in the advertisement object in the source system to one target destination field in the advertisement object in the destination system.

The target destination object determining unit 830 may include: an advertising creative target object determination unit, which may be used to determine that the advertising object and the material object in the target system are the target objects of the advertising creative object in the source system when the target objects include the advertising creative object in the source system.

Wherein the target source object change observer determining unit may be further configured to: notifying the advertising creative update observer to call a field conversion function of the advertising creative object in the source system to perform conversion of a portion of the target source field of the advertising creative object in the source system, where the data change occurs, into a target destination field in the material object in the destination system, and to convert another portion of the target source field into a target destination field in the advertising object in the destination system.

In an exemplary embodiment, the target destination field writing unit 850 may include: an advertising creative object many-to-one mapping unit may be used to map a plurality of target source fields in an advertising creative object in the source system to one target destination field in a material object in the destination system for writing into the destination system.

In an exemplary embodiment, an advertising creative object in the source system can include creative elements and elements, the advertising creative object can be used to store style specification information for advertising material, the creative elements can be used to store an identification of the advertising material, and the elements can be used to store content for the advertising material. Wherein the ad creative object many-to-one mapping unit may be operable to: and writing the content of the advertisement material stored in the element into a material object of the destination system.

In an exemplary embodiment, an advertising creative object in the source system may include multiple elements, and a summary field may exist on a material object of the destination system. Wherein the ad creative object many-to-one mapping unit may be operable to: and mapping the data stored in the plurality of elements to the summarizing field, wherein the summarizing field summarizes the data stored in the plurality of elements in a structure with a preset format so as to write the data into the material object of the destination system.

In an exemplary embodiment, the data synchronization apparatus 800 may further include: the object identifier mapping relationship recording unit may be configured to record a mapping relationship between an identifier of each target source object in the source system and an identifier of a corresponding target destination object in the destination system; and an object change log recording unit, configured to record a data relationship between the target object and the target object in the target system before and after the target object is changed, and store the data relationship in an object change log of the target system.

In an exemplary embodiment, the data change information acquiring unit 810 may include: the subscription unit of the subject of the binary log file can be used for subscribing the subject of the binary log file of the publishing and subscribing message system; the notification message receiving unit may be configured to receive a notification message published by the publish-subscribe message system, where the notification message carries a binary log file, and the binary log file includes data change information of the source system.

The synchronization delta data tool may be configured to receive the binary log file from the source system and send the binary log file to the publish-subscribe message system.

Other contents of the data synchronization device of the embodiment of the present disclosure may refer to the above-described embodiments.

It should be noted that although in the above detailed description several units of the device for action execution are mentioned, this division is not mandatory. Indeed, the features and functions of two or more units described above may be embodied in one unit, in accordance with embodiments of the present disclosure. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.

Reference is now made to fig. 9, which illustrates a schematic diagram of an electronic device suitable for use in implementing embodiments of the present application. The electronic device shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.

Referring to fig. 9, an electronic device provided in an embodiment of the present disclosure may include: a processor 901, a communication interface 902, a memory 903, and a communication bus 904.

Wherein the processor 901, the communication interface 902 and the memory 903 are in communication with each other via a communication bus 904.

Alternatively, the communication interface 902 may be an interface of a communication module, such as an interface of a GSM (Global System for Mobile communications) module. The processor 901 is used to execute programs. The memory 903 is used for storing programs. The program may comprise a computer program comprising computer operating instructions. Wherein, can include in the procedure: and (5) a program of the video client.

The processor 901 may be a central processing unit CPU, or an application Specific Integrated circuit asic, or one or more Integrated circuits configured to implement embodiments of the present disclosure.

The memory 903 may include a Random Access Memory (RAM) memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

Among them, the procedure can be specifically used for: acquiring data change information of a source system, wherein the source system comprises at least two source objects; determining a target source object with data change in the source system from at least two source objects of the source system according to the data change information of the source system; determining a target destination object corresponding to a target source object in a source system in a destination system, wherein the destination system comprises at least one destination object, the at least one destination object comprises the target destination object, and the target destination object comprises one or more of the at least two source objects; converting a target source field with data change in the target source object into a target destination field in the target destination object; and writing the target destination field into the target source object in the destination system.

It is to be understood that any number of elements in the drawings of the present disclosure are by way of example and not by way of limitation, and any nomenclature is used for differentiation only and not by way of limitation.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method of data synchronization, comprising:

acquiring data change information of a source system, wherein the source system comprises at least two source objects, and the data change information comprises a change object name and a change object type;

determining a target source object with data change in the source system from at least two source objects of the source system according to the data change information of the source system;

determining a target destination object corresponding to a target source object in a source system in a destination system, wherein the destination system comprises at least one destination object, the at least one destination object comprises the target destination object, and the target destination object comprises one or more of the at least two source objects;

converting a target source field with data change in the target source object into a target destination field in the target destination object;

writing the target destination field into the target source object in the destination system;

determining a target source object with data change in the source system from at least two source objects of the source system according to the data change information of the source system, including:

inquiring and acquiring a key of a target source object change observer according to the change object name and the change object type;

determining a target source object change observer with data change in the source system from at least two source objects of the source system according to the key of the target source object change observer;

converting a target source field in the target source object, in which data change occurs, into a target destination field in the target destination object, including:

and informing the target source object change observer to call a field conversion function of the target source object so as to convert the target source field with data change in the target source object into the target destination field in the target destination object.

2. The method of claim 1, further comprising:

respectively setting a source object change observer for each source object in the source system;

each source object change observer is used for monitoring whether data in the corresponding source object are changed or not, and calling a field conversion function of the corresponding source object when the data in the corresponding source object are monitored to be changed so as to convert a source field of the corresponding source object, in which the data are changed, into a corresponding destination field in a corresponding destination object in the destination system;

the source object change observer includes the target source object change observer.

3. The method of claim 2, wherein the source system stores advertisement traffic data therein, the at least two source objects include a plan object, an advertisement object, a creative object, and an advertisement creative object in the source system, and the source object alteration viewer includes a plan update viewer, an advertisement update viewer, a creative insertion viewer, and an advertisement creative update viewer;

wherein, respectively setting a source object change observer for each source object in the source system comprises:

setting the plan update observer for a plan object in the source system;

setting the advertisement updating observer for the advertisement object in the source system;

setting the creative insertion viewer for the creative object in the source system;

and setting the advertisement creative update observer for the advertisement creative object in the source system.

4. The method of claim 3, wherein notifying the target source object change observer to invoke a field conversion function of the target source object to perform the conversion of the target source field of the target source object where the data change occurs into the target destination field of the target destination object comprises:

when the target destination objects include a plan object, an advertisement object, a creative object, and an advertisement creative object in the source system, determining that the target source object change observer includes the plan update observer, the advertisement update observer, the creative insertion observer, and the advertisement creative update observer, and performing the following in parallel:

informing the plan update observer to call a field conversion function of a plan object in the source system so as to convert the target source field of the plan object in the source system, in which data change occurs, into the target destination field in the target destination object;

informing the advertisement update observer to call a field conversion function of an advertisement object in the source system so as to convert the target source field with data change in the advertisement object in the source system into the target destination field in the target destination object;

notifying the creative insertion observer to call a field conversion function of the creative object in the source system to convert the target source field of the creative object in the source system with data change into the target destination field of the target destination object;

notifying the advertising creative update observer to invoke a field conversion function of the advertising creative object in the source system to convert the target source field of the advertising creative object in the source system with the data change into the target destination field of the target destination object.

5. The method of claim 4, wherein the at least one destination object includes a plan object, an advertisement object, a creative object, and a material object in the destination system;

determining a target destination object in a destination system corresponding to a target source object in a source system, including:

when the target object comprises a planning object, an advertisement object, a creative object and an advertisement creative object in the source system, determining that the planning object in the target system is the target object of the planning object in the source system, the advertisement object in the target system is the target object of the advertisement object in the source system, the creative object in the target system is the target object of the creative object in the source system, and the material object in the target system is the target object of the advertisement creative object in the source system.

6. The method of claim 5, wherein notifying the plan update observer to invoke a field transformation function of a plan object in the source system to transform the target source field in the plan object in the source system where the data change occurred to the target destination field in the target destination object comprises:

informing the plan updating observer to call a field conversion function of a plan object in the source system so as to map target source fields with data change in the plan object in the source system to target destination fields in the plan object in the destination system one by one;

wherein notifying the creative insertion viewer to invoke a field transformation function of the creative object in the source system to transform the target source field of the creative object in the source system with a data change to the target destination field in the target destination object comprises:

informing the creative insertion observer to call a field conversion function of the creative object in the source system so as to map target source fields with data change in the creative object in the source system to target destination fields in the creative object in the destination system one by one;

wherein notifying the advertisement update observer to invoke a field conversion function of the advertisement object in the source system to convert the target source field in the advertisement object in the source system, in which the data change occurs, into the target destination field in the target destination object, comprises:

and informing the advertisement updating observer to call a field conversion function of the advertisement object in the source system so as to map a plurality of target source fields with data change in the advertisement object in the source system to one target destination field in the advertisement object in the destination system.

7. The method of claim 4, wherein the at least one target object comprises an advertisement object and a material object in the target system;

when the target object comprises the advertising creative object in the source system, determining the advertising object and the material object in the target system as the target object of the advertising creative object in the source system;

wherein notifying the advertising creative update observer to invoke a field conversion function of the advertising creative object in the source system to convert the target source field in the advertising creative object in the source system with the data change into the target destination field in the target destination object comprises:

notifying the advertising creative update observer to call a field conversion function of the advertising creative object in the source system to perform conversion of a portion of the target source field of the advertising creative object in the source system, where the data change occurs, into a target destination field in the material object in the destination system, and to convert another portion of the target source field into a target destination field in the advertising object in the destination system.

8. The method of claim 7, wherein writing the target destination field into the target source object in the destination system comprises:

mapping a plurality of target source fields in an advertising creative object in the source system to a target destination field in a material object in the destination system for writing into the destination system.

9. The method of claim 8 wherein the advertising creative object in the source system includes a creative element and an element, the advertising creative object for storing style specification information for advertising material, the creative element for storing an identification of the advertising material, the element for storing content of the advertising material;

wherein mapping a plurality of target source fields in an ad creative object in the source system to a target destination field in a material object in the destination system for writing into the destination system comprises:

and writing the content of the advertisement material stored in the element into a material object of the destination system.

10. The method of claim 8 wherein the advertising creative object in the source system includes a plurality of elements and wherein a summary field exists on the material object of the destination system;

and mapping the data stored in the plurality of elements to the summarizing field, wherein the summarizing field summarizes the data stored in the plurality of elements in a structure with a preset format so as to write the data into the material object of the destination system.

11. The method of claim 1, further comprising:

recording the mapping relation between the identification of each target source object in the source system and the identification of the corresponding target object in the target system;

and recording the data relation before and after the target object in the target system is changed, and storing the data relation into an object change log of the target system.

12. The method of claim 1, wherein obtaining data change information for a source system comprises:

subscribing to a binary log file topic of a publish-subscribe message system;

receiving a notification message published by the publish-subscribe message system, wherein the notification message carries a binary log file, and the binary log file comprises data change information of the source system;

the publish-subscribe message system is used for publishing the binary log file acquired from the synchronous incremental data tool;

the synchronous increment data tool is used for receiving the binary log file from the source system and sending the binary log file to the publishing and subscribing message system.

13. A data synchronization apparatus, comprising:

the data change information acquisition unit is used for acquiring data change information of a source system, wherein the source system comprises at least two source objects, and the data change information comprises a change object name and a change object type;

a data change object determining unit, configured to determine, according to data change information of the source system, a target source object in which data change occurs in the source system from among at least two source objects of the source system;

a target destination object determining unit, configured to determine a target destination object corresponding to a target source object in a source system in the destination system, where the destination system includes at least one destination object, the at least one destination object includes the target destination object, and the target destination object includes one or more of the at least two source objects;

a target object field conversion unit, configured to convert a target source field in the target source object, where data change occurs, into a target destination field in the target destination object;

a target destination field writing unit, configured to write the target destination field into the target source object in the destination system;

wherein the data change object determination unit includes:

the target source object change observer key query unit is used for querying and obtaining the key of the target source object change observer according to the change object name and the change object type;

a target source object change observer determining unit, configured to determine, from at least two source objects of the source system, a target source object change observer with data change in the source system according to a key of the target source object change observer;

wherein the target object field conversion unit includes:

a field conversion function execution unit, configured to notify the target source object change observer to call a field conversion function of the target source object, so as to perform conversion from the target source field in the target source object, where a data change occurs, to the target destination field in the target destination object.

14. The apparatus of claim 13, further comprising:

a source object change observer setting unit configured to set a source object change observer for each source object in the source system, respectively;

15. The apparatus of claim 14, wherein the source system stores advertisement traffic data therein, the at least two source objects include a plan object, an advertisement object, a creative object, and an advertisement creative object in the source system, and the source object change observer includes a plan update observer, an advertisement update observer, a creative insertion observer, and an advertisement creative update observer;

wherein the source object change observer setting unit includes:

a plan update observer setting unit configured to set the plan update observer for a plan object in the source system;

an advertisement update observer setting unit for setting the advertisement update observer for the advertisement object in the source system;

the creative insertion observer setting unit is used for setting the creative insertion observer for the creative object in the source system;

and the advertisement creative update observer setting unit is used for setting the advertisement creative update observer for the advertisement creative object in the source system.

16. The apparatus of claim 15, wherein the field transfer function execution unit comprises:

a target source object change observer determination unit for determining that the target source object change observer includes the planned update observer, the advertisement update observer, the creative insertion observer, and the advertisement creative update observer when the target destination object includes a planned object, an advertisement object, a creative object, and an advertisement creative object in the source system, and performing the following operations in parallel:

17. The apparatus of claim 16, wherein the at least one destination object comprises a plan object, an advertisement object, a creative object, and a material object in the destination system;

wherein the target object determination unit includes:

and the target object mapping unit is used for determining that the planning object in the target system is the target object of the planning object in the source system, the advertising object in the target system is the target object of the advertising object in the source system, the creative object in the target system is the target object of the creative object in the source system, and the material object in the target system is the target object of the advertising creative object in the source system when the target object comprises the planning object, the advertising object, the creative object and the advertising creative object in the source system.

18. The apparatus of claim 17, wherein the target source object alteration observer determination unit is further configured to: informing the plan updating observer to call a field conversion function of a plan object in the source system so as to map target source fields with data change in the plan object in the source system to target destination fields in the plan object in the destination system one by one;

wherein the target source object alteration observer determination unit is further configured to: informing the creative insertion observer to call a field conversion function of the creative object in the source system so as to map target source fields with data change in the creative object in the source system to target destination fields in the creative object in the destination system one by one;

wherein the target source object alteration observer determination unit is further configured to: and informing the advertisement updating observer to call a field conversion function of the advertisement object in the source system so as to map a plurality of target source fields with data change in the advertisement object in the source system to one target destination field in the advertisement object in the destination system.

19. The apparatus of claim 16, wherein the at least one destination object comprises an advertisement object and a material object in the destination system;

wherein the target object determination unit includes:

an advertising creative target object determination unit, configured to determine that the advertising object and the material object in the target system are target objects of the advertising creative object in the source system when the target object includes the advertising creative object in the source system;

wherein the target source object change observer determination unit is further configured to: notifying the advertising creative update observer to call a field conversion function of the advertising creative object in the source system to perform conversion of a portion of the target source field of the advertising creative object in the source system, where the data change occurs, into a target destination field in the material object in the destination system, and to convert another portion of the target source field into a target destination field in the advertising object in the destination system.

20. The apparatus of claim 19, wherein the target destination field write unit comprises:

and the advertising creative object many-to-one mapping unit is used for mapping a plurality of target source fields in the advertising creative object in the source system into one target destination field in the material object in the destination system so as to write the target destination system.

21. The apparatus of claim 20 wherein the advertising creative object in the source system includes a creative element and an element, the advertising creative object for storing style specification information for advertising material, the creative element for storing an identification of the advertising material, the element for storing content of the advertising material;

wherein the ad creative object many-to-one mapping unit is to: and writing the content of the advertisement material stored in the element into a material object of the destination system.

22. The apparatus of claim 20 wherein the advertising creative object in the source system includes a plurality of elements and wherein a summary field exists on the material object of the destination system;

wherein the ad creative object many-to-one mapping unit is used for: and mapping the data stored in the plurality of elements to the summarizing field, wherein the summarizing field summarizes the data stored in the plurality of elements in a structure with a preset format so as to write the data into the material object of the destination system.

23. The apparatus of claim 13, further comprising:

the object identifier mapping relation recording unit is used for recording the mapping relation between the identifier of each target source object in the source system and the identifier of the corresponding target object in the target system;

and the object change log recording unit is used for recording the data relation before and after the target object in the target system is changed and storing the data relation into an object change log of the target system.

24. The apparatus according to claim 13, wherein the data change information acquiring unit includes:

the subscription unit of the subject of the binary log file is used for subscribing the subject of the binary log file of the publishing and subscribing message system;

a notification message receiving unit, configured to receive a notification message published by the publish-subscribe message system, where the notification message carries a binary log file, and the binary log file includes data change information of the source system;

25. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 12.

26. An electronic device, comprising:

at least one processor;

a storage device configured to store at least one program that, when executed by the at least one processor, causes the at least one processor to implement the method of any one of claims 1 to 12.