CN109165200B - Data synchronization method and device, computing equipment and computer storage medium - Google Patents
Data synchronization method and device, computing equipment and computer storage medium Download PDFInfo
- Publication number
- CN109165200B CN109165200B CN201810912653.5A CN201810912653A CN109165200B CN 109165200 B CN109165200 B CN 109165200B CN 201810912653 A CN201810912653 A CN 201810912653A CN 109165200 B CN109165200 B CN 109165200B
- Authority
- CN
- China
- Prior art keywords
- data
- customization
- rule
- source data
- customized
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data synchronization method, a data synchronization device, a computing device and a computer storage medium. The method comprises the following steps: receiving a data customization request, and determining a data customization rule corresponding to the data customization request; according to the data customization rule, customizing the initial source data from the source end to obtain customized source data; and synchronizing the customized source data to a target end. According to the scheme of the invention, the initial source data is customized according to the data customization rule, and the customized source data after the customization processing is synchronized to the target end, so that the customized source data synchronized to the target end meets the requirements of a user on the content or format of the data, and the subsequent use and processing of the synchronized customized source data are facilitated.
Description
Technical Field
The invention relates to the technical field of data synchronization, in particular to a data synchronization method, a data synchronization device, a computing device and a computer storage medium.
Background
In the information age of the present, a large amount of business data is generated all the time; meanwhile, with the generation of a large amount of service data, the storage and processing of the data face huge challenges. To facilitate subsequent use or processing of these services, service data needs to be synchronized from the initial message queue to another message queue or file system.
In the prior art, the synchronization is usually achieved by using a data synchronization function provided by a synchronization framework such as script, flash, and the like. However, such a conventional synchronization scheme can only copy the service data from the initial message queue to another message queue or a file system, and the service data after synchronization is difficult to meet the user requirement, which is not favorable for the user to further use the service data.
Disclosure of Invention
In view of the above, the present invention has been made to provide a data synchronization method, apparatus, computing device and computer storage medium that overcome or at least partially address the above-mentioned problems.
According to an aspect of the present invention, there is provided a data synchronization method, including:
receiving a data customization request, and determining a data customization rule corresponding to the data customization request;
according to the data customization rule, customizing the initial source data from the source end to obtain customized source data;
and synchronizing the customized source data to a target end.
According to another aspect of the present invention, there is provided a data synchronization apparatus including:
the determining module is suitable for receiving a data customizing request and determining a data customizing rule corresponding to the data customizing request;
the customizing module is suitable for customizing the initial source data from the source end according to the data customizing rule to obtain the customizing source data;
and the synchronization module is suitable for synchronizing the customized source data to the target end.
According to yet another aspect of the present invention, there is provided a computing device comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the data synchronization method.
According to still another aspect of the present invention, there is provided a computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the data synchronization method.
According to the data synchronization method, the data synchronization device, the computing equipment and the computer storage medium, a data customization request is received, and a data customization rule corresponding to the data customization request is determined; according to the data customization rule, customizing the initial source data from the source end to obtain customized source data; and synchronizing the customized source data to a target end. According to the scheme of the invention, the initial source data is customized according to the data customization rule, and the customized source data after the customization processing is synchronized to the target end, so that the customized source data synchronized to the target end meets the requirements of a user on the content or format of the data, and the subsequent use and processing of the synchronized customized source data are facilitated.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 shows a flow diagram of a data synchronization method according to one embodiment of the invention;
FIG. 2 shows a flow diagram of a data synchronization method according to another embodiment of the invention;
FIG. 3 shows a functional block diagram of a data synchronization apparatus according to one embodiment of the present invention;
FIG. 4 shows a schematic structural diagram of a computing device according to an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Fig. 1 shows a flow diagram of a data synchronization method according to an embodiment of the invention. As shown in fig. 1, the method includes:
step S101, receiving a data customization request, and determining a data customization rule corresponding to the data customization request.
A data customization request is received prior to performing data synchronization. The invention does not limit the specific way of triggering the data customization request, and the technical personnel in the field can flexibly set the data customization request according to the actual requirement. Optionally, the data customization request may be triggered by a data customization interface, or may be triggered by sending a trigger message. And, there may be one or more types of data customization requests, and different types of data customization requests correspond to different data customization rules. The data customization rules include rules for adjusting the data format of the initial source data and/or rules for processing the data content of the initial source data.
Step S102, according to the data customization rule, the initial source data from the source end is customized to obtain the customized source data.
And according to a data customization rule corresponding to the data customization request, customizing the initial source data from the source end to obtain the customized source data meeting the data format requirement and/or the field content requirement.
Step S103, synchronizing the customization source data to the target end.
And after the customization source data is obtained, synchronizing the customization source data to the target end. In the present invention, the synchronization mode is not specifically limited, and in the specific implementation, a person skilled in the art can determine the synchronization mode according to actual conditions.
According to the data synchronization method provided by the embodiment, a data customization request is received, and a data customization rule corresponding to the data customization request is determined; according to the data customization rule, customizing the initial source data from the source end to obtain customized source data; and synchronizing the customized source data to a target end. According to the scheme of the embodiment, the initial source data is customized according to the data customization rule, and the customized source data after the customization processing is synchronized to the target end, so that the customized source data synchronized to the target end meets the requirements of a user on the content or format of the data, and the synchronized customized source data is convenient to use and process subsequently.
Fig. 2 shows a flow chart of a data synchronization method according to another embodiment of the present invention. The method can be used for synchronization of streaming data and/or batch data. As shown in fig. 2, the method includes:
step S201, a plurality of data customizing interfaces respectively corresponding to different types of data customizing requests are preset.
In particular, a data customization interface corresponding to different types of data customization requests is arranged, so that a user is provided with a convenient entry triggering the data customization request. The user can trigger data customization requests corresponding to different data customization interfaces through selection operation. The type of the data customization request corresponding to the data customization interface includes but is not limited to at least one of the following: filtering requests, translation requests, aggregation requests, and/or offload requests.
Step S202, when a data customization request is received, a resource queue is applied to the resource cluster for data synchronization of initial source data from a source end.
Specifically, when a data customization request is received, according to requirements of data synchronization operations (including a customization processing process and a synchronization process), for example, according to the data volume of initial source data and the operation amount of the customization processing, a resource queue is applied to a resource cluster, that is, a corresponding resource for data synchronization can be allocated, wherein the resource cluster is a predetermined resource pool available for data synchronization; and then performing customization processing and synchronization processing operation on the initial source data in the allocated resource queue. According to the method for applying for the resource queue, a special application of a single machine and a cluster building are not needed when a data customization request is received every time, and resource waste is avoided.
In this embodiment, the initial source data from the source is streaming data and/or batch data.
Step S203, setting a bandwidth limit for performing data synchronization on the initial source data according to the priority of the service of the initial source data source.
Specifically, the priority of data synchronization of different services is preset; when a data customization request is received, determining a service to which initial source data belongs; and limiting the network bandwidth of data synchronization according to the priority of the service. The higher the priority of the service is, the wider the set limited bandwidth is, and meanwhile, the setting of the limited bandwidth also needs to consider the normal execution of other data synchronization jobs, and allocate reasonable bandwidth resources for each data synchronization job. Compared with the prior art in which the bandwidth is limited by the mode of setting the bandwidth, the mode of setting the bandwidth is characterized in that the mode of setting the bandwidth is started first and then occupies as many resources as possible, so that not only can the service data of the service with high priority be completed quickly and synchronously, but also the bandwidth requirements of other data synchronization operations can be met.
Step S204, receiving a data customization request triggered by a user through the data customization interface, and determining a data customization rule corresponding to the data customization request.
Specifically, a data customization request triggered by a user through a data customization interface is received, and the type of the data customization request is determined according to the interface type contained in the data customization request; and determining a data customization rule corresponding to the data customization request according to the type of the data customization request. Optionally, the identifier of the data customization interface, the type identifier of the data customization request, and the data customization rule are stored according to a corresponding relationship, and when the data customization request is received, the type of the data customization request and the data customization rule corresponding to the storage can be obtained by querying the interface type included in the data customization request.
So far, the preset data customization interfaces correspond to preset data customization rules of the platform. In other embodiments of the present invention, in order to meet more data customization requirements of a user, a preset custom interface is further provided, and when a data customization request triggered by the user through the preset custom interface is received, a custom rule corresponding to the data customization request is obtained, and the custom rule is determined as the data customization rule corresponding to the data customization request. Furthermore, in order to simplify the code of user-defined rules set by a user, a public function library is preset, and each public function in the public function library is used for calling through a user-defined interface; the custom rule is determined from the common function by determining the common function corresponding to the data customization request. The user may select a common function from the common function library to set the custom rule. Optionally, providing commonly used functions in a common function library to reduce the number of corresponding function codes written by a user; and/or parameter setting examples of the functions are provided in a common function library, and errors in code writing are avoided.
In other embodiments of the present invention, in consideration of the universality of the data customization rule, after the data customization request is received, the initial source data from the source end is preprocessed, so that the content of the preprocessed initial source data or the content of the preprocessed initial source data meets a certain specification, and the customization processing according to the data customization rule is facilitated. In these embodiments, one skilled in the art can flexibly determine the preprocessing mode according to the rule content of the data customization rule. Optionally, a specific pretreatment method is as follows: the initial source data is processed according to the preset sequence of each field, so as to obtain normalized data matched with the preset sequence of each field, and further facilitate the use of the data customization rules, for example, the initial source data is processed according to the sequence of "year", "month" and "day", so as to facilitate the use of the aggregation rules.
Step S205, according to the data customization rule, customizing the initial source data from the source end to obtain the customized source data; and synchronizing the customized source data to the target end.
Specifically, different data customization rules correspond to different customization processing procedures. The following describes the customization process and synchronization process of this embodiment with several typical data customization rules:
when the data customization rule is a distribution rule, dividing the initial source data into at least two groups of data according to the distribution rule. Specifically, the splitting rule includes a basis for splitting, and optionally, a basis for a difference in field content of the designated field, that is, data with the same field content of the designated field is divided into the same group. When there is a need to store the initial source data in a classified manner, by specifying the reference field of the classification, the initial source data can be classified according to the field content of the reference field, and the initial source data is divided into at least two groups of data. Then at least two groups of data are respectively synchronized to the corresponding target ends. For example, the initial source data is divided according to the field content of the gender field, so that two groups of data with the gender of male and the gender of female can be obtained, and the two groups of data are respectively synchronized to different target terminals, so that the subsequent statistical analysis can be conveniently performed on the data with different genders.
And when the data customization rule is a filtering rule, filtering the first field of the initial source data according to the filtering rule to obtain the customization source data with the field content of the first field filtered. Specifically, when there is a need to filter out field content of a first field in the initial source data, the first field and the field content thereof in the initial source data are deleted by specifying the filtered first field, and then the customized source data can be obtained. And then, synchronizing the customized source data with the field content of the first field filtered out to the target end.
And when the data customization rule is a conversion rule, converting the second field of the initial source data according to the conversion rule to obtain the customized source data of which the field content format of the second field is converted into a preset format. Specifically, when there is a need to adjust the data format in the initial source data, for example, when character case conversion is performed, a second field that needs to be format-adjusted is specified, and the format of the field content of the second field in the initial source data is converted into a preset format set in the conversion rule. Then, the field content format of the second field is converted into the customized source data with the preset format, and the customized source data is synchronized to the target end
And when the data customization rule is an aggregation rule, performing aggregation processing on at least two third fields of the initial source data according to the aggregation rule to obtain the customized source data after the field contents of the at least two third fields are aggregated. Specifically, the aggregation rule includes a specific form of aggregation, and optionally, the field contents of the at least two third fields are aggregated end to end according to the field order of the at least two third fields. When there is an aggregation demand, for example, the field contents of the three fields of "year", "month", and "day" are aggregated together, the third field for performing aggregation processing is specified, and the aggregation of the field contents is performed in the aggregation form specified in the aggregation rule. And then, synchronizing the customized source data after the field contents of the at least two third fields are aggregated to the target end.
It should be noted that, the foregoing has been described with only a few typical data customization rules, but the present invention is not limited to this, and in the specific implementation, a person skilled in the art may flexibly set different data customization rules according to actual data usage requirements, or a user may also flexibly customize data customization rules according to business processing requirements.
In addition, in other embodiments of the present invention, the accuracy of the synchronized data may be ensured by the following steps;
step one, according to a received synchronous trigger instruction for starting data synchronous operation, inquiring pre-recorded synchronous state information corresponding to the data synchronous operation; wherein the synchronization triggering instruction for starting the data synchronization operation comprises: a restart instruction, a resume instruction, and/or a resume instruction; optionally, the synchronization status information includes: the offset information of the source end, the identification of the stored file of the target end, the creation time of the stored file of the target end, the identification of the current written file of the target end, the creation time of the current written file of the target end and/or the written data volume of the current written file of the target end.
Determining offset information of a source end and write-in information of a target end according to the synchronization state information, wherein the offset information of the source end represents the time of correspondingly recording the synchronization state information, the source end completes synchronization progress, and the offset information of the source end can be used as a basis for dividing position boundaries of source data which are synchronized in the source end and source data to be synchronized; the written information of the target end represents the time for correspondingly recording the synchronous state information, the writing progress and the storage progress of the target end; the storage progress of the target end is consistent with the progress of the source end completing synchronization, in other words, the data stored by the target end corresponds to the source data of which the source end has completed synchronization.
And step three, synchronizing data from the source end to the target end according to the offset information of the source end and the write-in information of the target end, and updating the synchronization state information corresponding to the data synchronization operation in the synchronization process. Specifically, determining a position to be synchronized corresponding to source data which is not synchronized by a source end according to the offset information of the source end; starting to acquire source data to be synchronized from the position to be synchronized of a source end; judging whether the target end has source data which is not synchronized according to the write-in information of the target end; and if the source data which are not synchronized do not exist in the target end, synchronizing the source data to be synchronized to the target end. If the target end has source data which is not synchronized, covering the source data which is not synchronized by using the source data to be synchronized; and/or if the target end has source data which are not synchronized, removing the source data with the data volume equal to that of the source data which are not synchronized from the source data to be synchronized, and synchronizing the removed source data to be synchronized to the target end.
The data synchronization method in the above embodiment of the present invention may be implemented by improving based on an existing data processing framework, and optionally implemented based on a flink framework. However, the present invention is not limited to this, and in the implementation, those skilled in the art may implement the solution of the above embodiment through a completely new data processing framework.
According to the data synchronization method provided by the embodiment, when a data customization request triggered by a user through a data customization interface is received, corresponding data customization rules are determined according to different types of the data customization interfaces, and accordingly, a customization processing mode can be accurately determined and customization processing is carried out; and providing a user-defined interface so that a user can set personalized user-defined rules according to the self service logic and perform corresponding customization processing on initial source data; and when a data customization request is received, a resource queue is applied to the resource cluster, the purpose of saving resources is achieved, the limited bandwidth for carrying out data synchronization on initial source data is set according to the priority of the service of the initial source data source, and the service data of the service with high priority can be quickly and synchronously completed. By using the scheme of the embodiment, the initial source data is customized according to the data customization rule, and the customized source data after the customization processing is synchronized to the target end, so that the customized source data synchronized to the target end meets the requirements of the user on the content or format of the data, and the subsequent use and processing of the synchronized customized source data are facilitated.
Fig. 3 shows a functional block diagram of a data synchronization apparatus according to an embodiment of the present invention. As shown in fig. 3, the apparatus includes: a determining module 301, a customizing module 302 and a synchronizing module 303; optionally, the apparatus further comprises: a first setting module 304, a second setting module 305, an application module 306, and a restriction module 307
A determining module 301, adapted to receive a data customization request, and determine a data customization rule corresponding to the data customization request;
the customizing module 302 is adapted to perform customizing processing on the initial source data from the source end according to the data customizing rule to obtain customized source data;
a synchronization module 303, adapted to synchronize the customized source data to the target.
In an alternative embodiment, the apparatus further comprises: a first setting module 304 adapted to preset a plurality of data customizing interfaces respectively corresponding to different types of data customizing requests;
the determination module 301 is further adapted to: receiving a data customization request triggered by a user through the data customization interface, and determining the type of the data customization request according to the interface type contained in the data customization request;
and determining a data customization rule corresponding to the data customization request according to the type of the data customization request.
The type of the data customization request corresponding to the data customization interface comprises at least one of the following types: filtering requests, translation requests, aggregation requests, and/or offload requests.
In an alternative embodiment, the determining module 301 is further adapted to:
receiving a data customization request triggered by a user through a preset custom interface, acquiring a custom rule corresponding to the data customization request, and determining the custom rule as the data customization rule corresponding to the data customization request.
In an alternative embodiment, the apparatus further comprises:
a second setting module 305, adapted to preset a public function library, where each public function in the public function library is used for calling through the custom interface;
the determination module 301 is further adapted to: and determining a public function corresponding to the data customization request, and determining the self-defining rule according to the public function.
In an optional embodiment, when the data customization rule is a split rule, the customization module 302 is further adapted to: dividing the initial source data into at least two groups of data according to the shunting rule;
the synchronization module 303 is further adapted to: and synchronizing the at least two groups of data to corresponding target ends respectively.
In an alternative embodiment, when the data customization rule is a filter rule, the customization module 302 is further adapted to:
filtering the field content of the first field of the initial source data according to the filtering rule to obtain customized source data with the field content of the first field filtered;
the synchronization module 303 is further adapted to: and synchronizing the customized source data with the field content of the first field filtered out to a target end.
In an alternative embodiment, when the data customization rule is a transformation rule, the customization module 302 is further adapted to:
converting the field content format of the second field of the initial source data according to the conversion rule to obtain customized source data of which the second field is converted into a preset format;
the synchronization module 303 is further adapted to: and synchronizing the customized source data after converting the field content format of the second field into a preset format to a target end.
In an alternative embodiment, when the data customization rule is an aggregation rule, the customization module 302 is further adapted to:
according to the aggregation rule, the field contents of at least two third fields of the initial source data are aggregated to obtain customized source data after the field contents of the at least two third fields are aggregated;
the synchronization module 303 is further adapted to: and synchronizing the customized source data after the field contents of the at least two third fields are aggregated to the target end.
In an optional embodiment, the initial source data from the source end is streaming data and/or batch data.
In an alternative embodiment, the apparatus further comprises:
an applying module 306 is adapted to apply for a resource queue to the resource cluster for data synchronization of the initial source data from the source end.
In an alternative embodiment, the apparatus further comprises:
the limiting module 307 is adapted to set a bandwidth limit for performing data synchronization on the initial source data according to the priority of the service of the initial source data source.
The specific structure and operation principle of each module described above may refer to the description of the corresponding step in the method embodiment, and are not described herein again.
The embodiment of the application provides a non-volatile computer storage medium, wherein at least one executable instruction is stored in the computer storage medium, and the computer executable instruction can execute the data synchronization method in any method embodiment.
Fig. 4 is a schematic structural diagram of a computing device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the computing device.
As shown in fig. 4, the computing device may include: a processor (processor)402, a Communications Interface 404, a memory 406, and a Communications bus 408.
Wherein:
the processor 402, communication interface 404, and memory 406 communicate with each other via a communication bus 408.
A communication interface 404 for communicating with network elements of other devices, such as clients or other servers.
The processor 402 is configured to execute the program 410, and may specifically perform the relevant steps in the above-described data synchronization method embodiment.
In particular, program 410 may include program code comprising computer operating instructions.
The processor 402 may be a central processing unit CPU or an application Specific Integrated circuit asic or one or more Integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 406 for storing a program 410. Memory 406 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 410 may specifically be configured to cause the processor 402 to perform the following operations:
receiving a data customization request, and determining a data customization rule corresponding to the data customization request;
according to the data customization rule, customizing the initial source data from the source end to obtain customized source data;
and synchronizing the customized source data to a target end.
In an alternative embodiment, the program 410 may be further specifically configured to cause the processor 402 to perform the following operations: presetting a plurality of data customization interfaces respectively corresponding to different types of data customization requests;
receiving a data customization request triggered by a user through the data customization interface, and determining the type of the data customization request according to the interface type contained in the data customization request;
and determining a data customization rule corresponding to the data customization request according to the type of the data customization request.
In an optional embodiment, the type of the data customization request corresponding to the data customization interface includes at least one of: filtering requests, translation requests, aggregation requests, and/or offload requests.
In an alternative embodiment, the program 410 may be further specifically configured to cause the processor 402 to perform the following operations:
receiving a data customization request triggered by a user through a preset custom interface, acquiring a custom rule corresponding to the data customization request, and determining the custom rule as the data customization rule corresponding to the data customization request.
In an alternative embodiment, the program 410 may be further specifically configured to cause the processor 402 to perform the following operations:
presetting a public function library, wherein each public function in the public function library is used for calling through the user-defined interface;
and determining a public function corresponding to the data customization request, and determining the self-defining rule according to the public function.
In an optional implementation manner, when the data customization rule is a split rule, the program 410 may be further specifically configured to cause the processor 402 to perform the following operations:
dividing the initial source data into at least two groups of data according to the shunting rule;
and synchronizing the at least two groups of data to corresponding target ends respectively.
In an optional implementation manner, when the data customization rule is a filtering rule, the program 410 may be further specifically configured to cause the processor 402 to perform the following operations:
filtering the field content of the first field of the initial source data according to the filtering rule to obtain customized source data with the field content of the first field filtered;
and synchronizing the customized source data with the field content of the first field filtered out to a target end.
In an optional implementation manner, when the data customization rule is a conversion rule, the program 410 may be specifically further configured to cause the processor 402 to perform the following operations:
converting the field content format of the second field of the initial source data according to the conversion rule to obtain customized source data after the field content format of the second field is converted into a preset format;
and synchronizing the customized source data after converting the field content format of the second field into a preset format to a target end.
In an alternative embodiment, when the data customization rule is an aggregation rule, the program 410 may be further specifically configured to cause the processor 402 to:
according to the aggregation rule, the field contents of at least two third fields of the initial source data are aggregated to obtain customized source data after the field contents of the at least two third fields are aggregated;
the step of synchronizing the customized source data to the target specifically includes: and synchronizing the customized source data after the field contents of the at least two third fields are aggregated to the target end.
In an alternative embodiment, the initial source data from the source end is streaming data and/or batch data.
In an alternative embodiment, the program 410 may be further specifically configured to cause the processor 402 to perform the following operations:
and applying for a resource queue from the resource cluster for data synchronization of the initial source data from the source end.
In an alternative embodiment, the program 410 may be further specifically configured to cause the processor 402 to perform the following operations:
and setting the limited bandwidth for carrying out data synchronization on the initial source data according to the priority of the service of the initial source data source.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a data synchronization apparatus according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
Claims (18)
1. A method of data synchronization, comprising:
receiving a data customization request, and determining a data customization rule corresponding to the data customization request;
according to the data customization rule, customizing the initial source data from the source end to obtain customized source data;
synchronizing the customized source data to a target end;
before the step of receiving the data customization request, the method further includes: presetting a plurality of data customization interfaces respectively corresponding to different types of data customization requests;
the step of receiving a data customization request and determining a data customization rule corresponding to the data customization request specifically includes:
receiving a data customization request triggered by a user through the data customization interface, applying for a resource queue from a resource cluster when the data customization request is received so as to perform data synchronization on initial source data from a source end, and determining the type of the data customization request according to an interface type contained in the data customization request;
determining a data customization rule corresponding to the data customization request according to the type of the data customization request;
before the step of customizing the initial source data from the source end, the method further includes:
setting a limited bandwidth for carrying out data synchronization on initial source data according to the priority of the service of the initial source data source;
wherein the initial source data is streaming data and/or batch data.
2. The method of claim 1, wherein the type of data customization request corresponding to the data customization interface comprises at least one of: filtering requests, translation requests, aggregation requests, and/or offload requests.
3. The method according to claim 1, wherein the step of receiving a data customization request and determining a data customization rule corresponding to the data customization request specifically comprises:
receiving a data customization request triggered by a user through a preset custom interface, acquiring a custom rule corresponding to the data customization request, and determining the custom rule as the data customization rule corresponding to the data customization request.
4. The method of claim 3, wherein the step of receiving a data customization request triggered by a user through a preset custom interface is preceded by the step of: presetting a public function library, wherein each public function in the public function library is used for calling through the user-defined interface;
the step of obtaining the customized rule corresponding to the data customization request specifically includes: and determining a public function corresponding to the data customization request, and determining the self-defining rule according to the public function.
5. The method according to any one of claims 1 to 4, wherein, when the data customizing rule is a forking rule, the step of performing the customizing process on the initial source data from the source end to obtain the customized source data specifically includes:
dividing the initial source data into at least two groups of data according to the shunting rule;
the step of synchronizing the customized source data to the target specifically includes: and synchronizing the at least two groups of data to corresponding target ends respectively.
6. The method according to any one of claims 1 to 4, wherein, when the data customizing rule is a filtering rule, the step of performing the customizing process on the initial source data from the source end to obtain the customized source data specifically includes:
filtering the first field of the initial source data according to the filtering rule to obtain customized source data with the field content of the first field filtered;
the step of synchronizing the customized source data to the target specifically includes: and synchronizing the customized source data with the field content of the first field filtered out to a target end.
7. The method according to any one of claims 1 to 4, wherein, when the data customizing rule is a conversion rule, the step of performing the customizing process on the initial source data from the source end to obtain the customized source data specifically includes:
converting the second field of the initial source data according to the conversion rule to obtain customized source data of which the field content format of the second field is converted into a preset format;
the step of synchronizing the customized source data to the target specifically includes: and synchronizing the customized source data after converting the field content format of the second field into a preset format to a target end.
8. The method according to any one of claims 1 to 4, wherein, when the data customization rule is an aggregation rule, the step of performing customization processing on the initial source data from the source end to obtain the customized source data specifically includes:
performing aggregation processing on at least two third fields of the initial source data according to the aggregation rule to obtain customized source data after the field contents of the at least two third fields are aggregated;
the step of synchronizing the customized source data to the target specifically includes: and synchronizing the customized source data after the field contents of the at least two third fields are aggregated to the target end.
9. A data synchronization apparatus, comprising:
the determining module is suitable for receiving a data customizing request and determining a data customizing rule corresponding to the data customizing request;
the customizing module is suitable for customizing the initial source data from the source end according to the data customizing rule to obtain the customizing source data;
the synchronization module is suitable for synchronizing the customization source data to a target end;
the device further comprises:
the first setting module is suitable for presetting a plurality of data customization interfaces respectively corresponding to different types of data customization requests;
the determination module is further adapted to: receiving a data customization request triggered by a user through the data customization interface, applying for a resource queue from a resource cluster when the data customization request is received so as to perform data synchronization on initial source data from a source end, and determining the type of the data customization request according to an interface type contained in the data customization request;
determining a data customization rule corresponding to the data customization request according to the type of the data customization request;
the determining module is used for setting the bandwidth limitation for carrying out data synchronization on the initial source data according to the priority of the service of the initial source data source;
wherein the initial source data is streaming data and/or batch data.
10. The apparatus of claim 9, wherein the type of data customization request corresponding to the data customization interface comprises at least one of: filtering requests, translation requests, aggregation requests, and/or offload requests.
11. The apparatus of claim 9, wherein the determination module is further adapted to:
receiving a data customization request triggered by a user through a preset custom interface, acquiring a custom rule corresponding to the data customization request, and determining the custom rule as the data customization rule corresponding to the data customization request.
12. The apparatus of claim 11, wherein the apparatus further comprises:
the second setting module is suitable for presetting a public function library, and each public function in the public function library is used for calling through the user-defined interface;
the determination module is further adapted to: and determining a public function corresponding to the data customization request, and determining the self-defining rule according to the public function.
13. The apparatus of any of claims 9-12, wherein when the data customization rule is a forking rule, the customization module is further adapted to: dividing the initial source data into at least two groups of data according to the shunting rule;
the synchronization module is further adapted to: and synchronizing the at least two groups of data to corresponding target ends respectively.
14. The apparatus of any of claims 9-12, wherein when the data customization rule is a filtering rule, the customization module is further adapted to:
filtering the first field of the initial source data according to the filtering rule to obtain customized source data with the field content of the first field filtered;
the synchronization module is further adapted to: and synchronizing the customized source data with the field content of the first field filtered out to a target end.
15. The apparatus of any of claims 9-12, wherein when the data customization rule is a transformation rule, the customization module is further adapted to:
converting the second field of the initial source data according to the conversion rule to obtain customized source data of which the field content format of the second field is converted into a preset format;
the synchronization module is further adapted to: and synchronizing the customized source data after converting the field content format of the second field into a preset format to a target end.
16. The apparatus of any of claims 9-12, wherein when the data customization rule is an aggregation rule, the customization module is further adapted to:
according to the aggregation rule, the field contents of at least two third fields of the initial source data are aggregated to obtain customized source data after the field contents of the at least two third fields are aggregated;
the synchronization module is further adapted to: and synchronizing the customized source data after the field contents of the at least two third fields are aggregated to the target end.
17. A computing device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the data synchronization method according to any one of claims 1-8.
18. A computer storage medium having stored therein at least one executable instruction that causes a processor to perform operations corresponding to the data synchronization method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810912653.5A CN109165200B (en) | 2018-08-10 | 2018-08-10 | Data synchronization method and device, computing equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810912653.5A CN109165200B (en) | 2018-08-10 | 2018-08-10 | Data synchronization method and device, computing equipment and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109165200A CN109165200A (en) | 2019-01-08 |
| CN109165200B true CN109165200B (en) | 2022-04-01 |
Family
ID=64895618
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810912653.5A Active CN109165200B (en) | 2018-08-10 | 2018-08-10 | Data synchronization method and device, computing equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109165200B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101047549A (en) * | 2006-04-10 | 2007-10-03 | 华为技术有限公司 | Data synchronous system and its method |
| CN102541942A (en) * | 2010-12-31 | 2012-07-04 | 中国银联股份有限公司 | Data bulk transfer system and method thereof |
| CN102752294A (en) * | 2012-06-19 | 2012-10-24 | 中国科学院计算技术研究所 | Method and system for synchronizing data of multiple terminals on basis of equipment capacity |
| CN103914526A (en) * | 2014-03-27 | 2014-07-09 | 神华集团有限责任公司 | Interface method and interface device for SAP ERP (enterprise resource planning) system and ORACLE ERP system |
| CN106557497A (en) * | 2015-09-25 | 2017-04-05 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization and device |
| CN108121757A (en) * | 2017-11-10 | 2018-06-05 | 广州优视网络科技有限公司 | A kind of method of data synchronization, device, system, computing device and storage medium |
-
2018
- 2018-08-10 CN CN201810912653.5A patent/CN109165200B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101047549A (en) * | 2006-04-10 | 2007-10-03 | 华为技术有限公司 | Data synchronous system and its method |
| CN102541942A (en) * | 2010-12-31 | 2012-07-04 | 中国银联股份有限公司 | Data bulk transfer system and method thereof |
| CN102752294A (en) * | 2012-06-19 | 2012-10-24 | 中国科学院计算技术研究所 | Method and system for synchronizing data of multiple terminals on basis of equipment capacity |
| CN103914526A (en) * | 2014-03-27 | 2014-07-09 | 神华集团有限责任公司 | Interface method and interface device for SAP ERP (enterprise resource planning) system and ORACLE ERP system |
| CN106557497A (en) * | 2015-09-25 | 2017-04-05 | 阿里巴巴集团控股有限公司 | A kind of method of data synchronization and device |
| CN108121757A (en) * | 2017-11-10 | 2018-06-05 | 广州优视网络科技有限公司 | A kind of method of data synchronization, device, system, computing device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109165200A (en) | 2019-01-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108614726B (en) | Virtual machine creation method and device | |
| WO2019201039A1 (en) | Method and system for updating application and application server | |
| CN110247984B (en) | Service processing method, device and storage medium | |
| CN114244717B (en) | Configuration method and device of virtual network card resources, computer equipment and medium | |
| CN113918232B (en) | Algorithm service calling method, device, server and storage medium | |
| CN114416352A (en) | Computing resource allocation method and device, electronic equipment and storage medium | |
| CN111193622B (en) | Asymmetric configuration management method, device and equipment and readable storage medium | |
| CN114979286B (en) | Access control method, device, equipment and computer storage medium for container service | |
| CN108897858B (en) | Distributed cluster index fragmentation evaluation method and device and electronic equipment | |
| CN113760242A (en) | Data processing method, device, server and medium | |
| CN111126604B (en) | Model training method, device, server and storage medium | |
| CN109165200B (en) | Data synchronization method and device, computing equipment and computer storage medium | |
| CN112003930A (en) | Task allocation method, device, equipment and storage medium | |
| CN111767126A (en) | System and method for distributed batch processing | |
| CN114070889B (en) | Configuration method, traffic forwarding device, storage medium, and program product | |
| US10528625B2 (en) | Client cloud synchronizer | |
| CN114448990A (en) | Resource scheduling method, device and equipment based on fusion CDN | |
| CN114564210A (en) | Copy deployment method, device, system, electronic equipment and storage medium | |
| CN113467823A (en) | Configuration information acquisition method, device, system and storage medium | |
| WO2015196844A1 (en) | Task assignment method, apparatus, system and server | |
| CN112632422A (en) | Intelligent image cutting method and device, electronic equipment and storage medium | |
| CN107967549B (en) | Multi-process task processing device and method | |
| CN111507447A (en) | Label identification method and device | |
| CN111026613A (en) | Log processing method and device | |
| CN112395081A (en) | Resource online automatic recovery method, system, server and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |