CN114840277A - Data synchronization method, device, equipment and medium - Google Patents
Data synchronization method, device, equipment and medium Download PDFInfo
- Publication number
- CN114840277A CN114840277A CN202210502756.0A CN202210502756A CN114840277A CN 114840277 A CN114840277 A CN 114840277A CN 202210502756 A CN202210502756 A CN 202210502756A CN 114840277 A CN114840277 A CN 114840277A
- Authority
- CN
- China
- Prior art keywords
- target
- configuration information
- interface
- preset
- synchronized
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4488—Object-oriented
- G06F9/449—Object-oriented method invocation or resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/36—Software reuse
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data synchronization method, a device, equipment and a medium, which relate to the technical field of computers and comprise the following steps: determining a target message queue by using target configuration information configured through a preset synchronous task configuration platform, and reading source data to be synchronized from the target message queue; judging whether a message queue needs to be redistributed to the source data to be synchronized based on the target configuration information, and judging whether two-party interface calling needs to be carried out based on the target configuration information if the message queue needs to be redistributed to the source data to be synchronized; if the two-party interface is judged to be required to be called, determining the corresponding two-party interface type and the field type mapping relation corresponding to the source data to be synchronized; and storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation. Through the scheme, the cost can be reduced and the efficiency can be improved when the data synchronization task is completed.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data synchronization method, apparatus, device, and medium.
Background
When the current backup file system (i.e. dump) completes each data synchronization task (i.e. dump task) in each service line, developers corresponding to the service line need to compile corresponding codes for each data synchronization task, i.e. each data synchronization task needs one corresponding code, so that when the number of data synchronization tasks is large, the developers need to compile codes in a large amount of time, and the cost is increased; when different data synchronization tasks need to call the same interface, corresponding codes still need to be written each time so as to write source data into a target data source of an es (elastic search), thereby causing writing of repeated codes and reducing efficiency.
In summary, how to reduce the cost and improve the efficiency when completing the data synchronization task is a problem to be solved in the field.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a data synchronization method, apparatus, device and medium, which can reduce the cost and improve the efficiency when completing the data synchronization task. The specific scheme is as follows:
in a first aspect, the present application discloses a data synchronization method, including:
determining a target message queue by using target configuration information configured through a preset synchronous task configuration platform, and reading source data to be synchronized from the target message queue;
judging whether a message queue needs to be redistributed to the source data to be synchronized based on the target configuration information, and judging whether two-party interface calling needs to be carried out based on the target configuration information if the message queue needs to be redistributed to the source data to be synchronized;
if the two-party interface is judged to be required to be called, determining the corresponding two-party interface type and the field type mapping relation corresponding to the source data to be synchronized;
and storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation.
Optionally, before determining the target message queue by using the target configuration information configured by the preset synchronous task configuration platform, the method further includes:
packaging the general logic of the source data to be synchronized based on the acquired target data synchronization command through the preset synchronization task configuration platform to obtain a packaged general logic;
and extracting a target configurable item corresponding to the target data synchronization command from the encapsulated general logic through the preset synchronization task configuration platform, and configuring the target configurable item based on the target data synchronization command to obtain target configuration information.
Optionally, the extracting, by the preset synchronization task configuration platform, the target configurable item corresponding to the target data synchronization command from the encapsulated general logic, and configuring the target configurable item based on the target data synchronization command to obtain target configuration information includes:
if the target data synchronization command read by the preset synchronization task configuration platform comprises a preset service arrangement configuration command, extracting an interface configurable item in the encapsulated general logic by the preset synchronization task configuration platform;
and configuring the interface configurable item through the preset synchronous task configuration platform based on the preset service arrangement configuration command to obtain interface configuration information.
Optionally, the configuring, by the preset synchronous task configuration platform, the interface configurable item based on the preset service orchestration configuration command to obtain interface configuration information includes:
and configuring any one or more interface configurable items in the type of a two-party interface, an interface address, an interface version, an interface alias, interface dependency information, interface access parameter information, an arrangement script of the interface access parameter information and an interface calling sequence based on the preset service arrangement configuration command through the preset synchronous task configuration platform to obtain interface configuration information.
Optionally, the determining whether to perform two-party interface calling based on the target configuration information includes:
and judging whether to call a two-party interface or not based on the interface configuration information in the target configuration information.
Optionally, the determining a target message queue by using target configuration information configured by a preset synchronization task configuration platform, and reading source data to be synchronized from the target message queue includes:
reading basic configuration information in target configuration information configured by a preset synchronous task configuration platform, and determining a target message queue by using the basic configuration information; the basic configuration information comprises any one or more configuration information of target task name information, target task type information, target service level information, target responsible person information, target ES cluster information and target ES index information;
if the target message queue determined by the basic configuration information is a preset public message queue, reading source data to be synchronized from the preset public message queue; and if the target message queue determined by using the basic configuration information is a preset independent message queue, reading the source data to be synchronized from the preset independent message queue.
Optionally, before reading the basic configuration information in the target configuration information configured by the preset synchronous task configuration platform, the method further includes:
and reading a target data synchronization command through the preset synchronization task configuration platform to obtain a basic information configuration command, and extracting a basic information configurable item corresponding to the basic information configuration command from the packaged general logic through the preset synchronization task configuration platform.
In a second aspect, the present application discloses a data synchronization apparatus, comprising:
the source data reading module is used for determining a target message queue by using target configuration information configured by a preset synchronous task configuration platform and reading source data to be synchronized from the target message queue;
the judging module is used for judging whether the source data to be synchronized carries out redelivery processing based on the target configuration information, and if not, judging whether two-party interface calling is carried out based on the target configuration information;
the information determining module is used for determining the type of the two-party interface and the field type mapping relation corresponding to the source data to be synchronized if the two-party interface is judged to be called;
and the storage module is used for storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation.
In a third aspect, the present application discloses an electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the data synchronization method disclosed in the foregoing.
In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program realizes the steps of the data synchronization method disclosed in the foregoing when being executed by a processor.
Therefore, the method comprises the steps that a target message queue is determined by utilizing target configuration information configured through a preset synchronization task configuration platform, and source data to be synchronized are read from the target message queue; judging whether a message queue needs to be redistributed to the source data to be synchronized based on the target configuration information, and judging whether two-party interface calling needs to be carried out based on the target configuration information if the message queue needs to be redistributed to the source data to be synchronized; if the two-party interface is judged to be required to be called, determining the corresponding two-party interface type and the field type mapping relation corresponding to the source data to be synchronized; and storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation. Therefore, the preset synchronous task configuration platform is introduced, and the preset synchronous task configuration platform is used for configuring to obtain the target configuration information, so that the workload required by code compiling when the preset synchronous task configuration platform is configured is greatly reduced compared with the workload required by the prior art for completing data synchronous code compiling, and the cost is further reduced; and subsequently, based on the target configuration information, a target message queue can be determined so as to read the source data to be synchronized from the target message queue, whether the source data to be synchronized is redistributed to the message queue can be judged, whether the source data to be synchronized needs to be called by a two-party interface and a field type mapping relation corresponding to the source data to be synchronized can be obtained, and then the source data to be synchronized is stored to a target data source based on the field type mapping relation, wherein code writing work is basically not needed, the redundant code condition is reduced, and the data synchronization efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of a data synchronization method disclosed herein;
FIG. 2 is a flow chart of a particular data synchronization method disclosed herein;
FIG. 3 is a flow chart of a particular data synchronization method disclosed herein;
FIG. 4 is a flow chart of a particular data synchronization method disclosed herein;
FIG. 5 is a flow chart of a particular data synchronization method disclosed herein;
FIG. 6 is a schematic diagram of a data synchronization apparatus according to the present disclosure;
fig. 7 is a block diagram of an electronic device disclosed in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
When the current backup file system (i.e. dump) completes each data synchronization task (i.e. dump task) in each service line, developers corresponding to the service line need to compile corresponding codes for each data synchronization task, i.e. each data synchronization task needs one corresponding code, so that when the number of data synchronization tasks is large, the developers need to compile codes in a large amount of time, and the cost is increased; when different data synchronization tasks need to call the same interface, corresponding codes still need to be written each time so as to write source data into a target data source of an es (elastic search), thereby causing writing of repeated codes and reducing efficiency.
Therefore, the data synchronization scheme is correspondingly provided, so that the cost can be reduced and the efficiency can be improved when the data synchronization task is completed.
Referring to fig. 1, an embodiment of the present application discloses a data synchronization method, including:
step S11: and determining a target message queue by using target configuration information configured by a preset synchronous task configuration platform, and reading source data to be synchronized from the target message queue.
In this embodiment, the preset synchronization task configuration platform configures each configurable item of the source data to be synchronized based on the obtained target data synchronization command to obtain target configuration information, for example, a specific data synchronization method flowchart shown in fig. 2, and may configure the basic information configurable item based on a preset basic information configuration command in the target data synchronization command to obtain basic configuration information, where the basic information configurable item includes any one or more configurable items of a target task name, a target task type, a target service level, a target principal, target ES cluster information, and target ES index information, and the target task type may be an incremental dump or a full dump; configuring interface configurable items based on a preset service arrangement configuration command in a target data synchronization command to obtain interface configuration information, wherein the interface configurable items comprise any one or more configurable items of a two-party interface type, an interface address, an interface version, an interface alias, interface dependency information, interface entry parameter information, an arrangement script of the interface entry parameter information and an interface calling sequence, and the two-party interface type can be rpc (Remote Procedure Call) or http (Hyper Text Transfer Protocol); configuring a trigger configurable item based on a trigger rule configuration command in a target data synchronization command to obtain trigger configuration information, where the trigger configurable item may include any one or more configurable items of a data source type, topic, group Id, an alias, and whether a Message Queue is reallocated, and the data source type may be canal or MQ (Message Queue); configuring resource configurable items based on a resource control configuration command in a target data synchronization command to obtain resource configuration information, wherein the resource configurable items comprise any one or more configurable items of a target thread number, a target exception policy and retry times, and the target exception policy can be retry failure or discard failure; and configuring a mapping configurable item based on a result mapping configuration command in the target data synchronization command to obtain mapping configuration information, wherein the mapping configurable item comprises ES index mapping information, a field type mapping relation and a processing script corresponding to the field type mapping relation. When the system starts the data synchronization service, the system reads the target configuration information obtained after the configuration of the preset synchronization task configuration platform, and can determine a target message queue so as to read the source data to be synchronized from the target message queue.
Step S12: and judging whether a message queue needs to be redistributed to the source data to be synchronized based on the target configuration information, and if not, judging whether two-party interface calling needs to be carried out based on the target configuration information.
In this embodiment, it may be determined whether to reallocate a message queue for source data to be synchronized based on trigger configuration information in the target configuration information, and if it is determined that the message queue needs to be reallocated for the source data to be synchronized, the reallocated message queue is determined based on the trigger configuration information, for example, when it is determined that the reallocated message queue is a preset independent message queue, the source data to be synchronized is reallocated to the preset independent message queue, so as to ensure that resources are independent when the subsequent source data to be synchronized performs a data synchronization task. When it is determined that the message queue does not need to be reallocated to the source data to be synchronized, whether two-party interface calling is needed or not may be determined based on interface configuration information in the target configuration information, where the interface configuration information may be obtained by configuring, by a preset synchronization task configuration platform, the interface configurable item based on a preset service arrangement configuration command.
Step S13: and if the two-party interface call is judged to be needed, determining the corresponding two-party interface type and the field type mapping relation corresponding to the source data to be synchronized.
In this embodiment, if it is determined that the two-party interface call needs to be performed, determining a corresponding two-party interface type, where it may be understood that target configuration information including interface configuration information is obtained by configuring, by a preset synchronous task configuration platform, the two-party interface type in the interface configurable item based on a preset service orchestration configuration command, and when the system reads the interface configuration information in the target configuration information, determining the two-party interface type, for example, the determined two-party interface type is http; and when the system reads the mapping configuration information in the target configuration information, acquiring a field type mapping relation corresponding to the source data to be synchronized.
Step S14: and storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation.
In this embodiment, the source data to be synchronized is stored in the corresponding target data source in the ES based on the two-party interface type and the field type mapping relationship, for example, the determined two-party interface type is rpc, the rpc interface may be called in a dubbo generalized calling manner, so as to perform secondary processing on the source data to be synchronized, and the processed source data to be synchronized is stored in the corresponding target data source in the ES based on the field type mapping relationship.
Therefore, the method comprises the steps that a target message queue is determined by utilizing target configuration information configured through a preset synchronization task configuration platform, and source data to be synchronized are read from the target message queue; judging whether a message queue needs to be redistributed to the source data to be synchronized based on the target configuration information, and judging whether two-party interface calling needs to be carried out based on the target configuration information if the message queue needs to be redistributed to the source data to be synchronized; if the two-party interface is judged to be required to be called, determining the corresponding two-party interface type and the field type mapping relation corresponding to the source data to be synchronized; and storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation. Therefore, the preset synchronous task configuration platform is introduced, and the preset synchronous task configuration platform is used for configuring to obtain the target configuration information, so that the workload required by code compiling when the preset synchronous task configuration platform is configured is greatly reduced compared with the workload required by the prior art for completing data synchronous code compiling, and the cost is further reduced; and then, the source data to be synchronized is stored to the target data source based on the field type mapping relation, wherein code writing work is basically not needed, the redundant code condition is reduced, and the data synchronization efficiency is improved.
Referring to fig. 3, an embodiment of the present application discloses a specific data synchronization method, including:
step S21: reading basic configuration information in target configuration information configured by a preset synchronous task configuration platform, and determining a target message queue by using the basic configuration information; the basic configuration information comprises any one or more configuration information of target task name information, target task type information, target service level information, target responsible person information, target ES cluster information and target ES index information.
In this embodiment, before reading the basic configuration information in the target configuration information configured by the preset synchronous task configuration platform, the method further includes: and reading a target data synchronization command through the preset synchronization task configuration platform to obtain a basic information configuration command, and extracting a basic information configurable item corresponding to the basic information configuration command from the packaged general logic through the preset synchronization task configuration platform. It can be understood that, if the basic information configuration command includes the start time of the data synchronization task, the preset synchronization task configuration platform may configure the start time configurable item to obtain the time configuration information, so that a subsequent system starts the data synchronization task based on the time configuration information, and controllability of the data synchronization task time is achieved.
Step S22: if the target message queue determined by the basic configuration information is a preset public message queue, reading source data to be synchronized from the preset public message queue; and if the target message queue determined by using the basic configuration information is a preset independent message queue, reading the source data to be synchronized from the preset independent message queue.
In this embodiment, as shown in a specific flow chart of a data synchronization method shown in fig. 4, if a basic information configuration command in a target data synchronization command read by a preset synchronization task configuration platform shows that source data 1 to be synchronized is in a high level, that is, needs to be processed preferentially, or a processing link corresponding to the source data 1 to be synchronized is a core link, the preset synchronization task configuration platform may perform corresponding configuration on a target service level configurable item, so as to obtain target service level information representing that the source data 1 to be synchronized needs to be read from a preset independent message queue, thereby avoiding a situation that resources cannot be guaranteed when the dump task is processed, and when the system reads the target service level information in the target configuration information, the source data 1 to be synchronized is read from the preset independent message queue; it can be understood that, if the basic information configuration command of the source data N to be synchronized is characterized by a common level, the preset synchronization task configuration platform may configure the target service level configurable item as the common level, so that the system reads the source data N to be synchronized from the preset common message queue.
Step S23: and judging whether a message queue needs to be redistributed to the source data to be synchronized based on the target configuration information, and if not, judging whether two-party interface calling needs to be carried out based on the target configuration information.
In this embodiment, if it is determined that the message queue needs to be reallocated for the source data to be synchronized, the source data to be synchronized may be reallocated to the preset common message queue by default, or the reallocated message queue may be determined based on the basic configuration information in the target configuration information, for example, when the reallocated message queue is determined to be the preset independent message queue by the basic configuration information, the preset independent message queue receives data corresponding to the source data to be synchronized. It should be noted that, corresponding target configuration information may also be obtained by configuring any one or more configurable items of the required thread pool, MQ consumer client, task context, and execution node through a preset synchronization task configuration platform, so as to redistribute the source data to be synchronized to the corresponding message queue by using the target configuration information.
Step S24: and if the two-party interface call is judged to be needed, determining the corresponding two-party interface type and the field type mapping relation corresponding to the source data to be synchronized.
Step S25: and storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation.
Therefore, basic configuration information is obtained by configuring the basic information configurable item through the preset synchronization task configuration platform, when a subsequent system reads the basic configuration information, a target message queue can be determined by using the basic configuration information, and then data synchronization tasks of different levels can be in different message queues, so that independent resources of high-level data synchronization tasks can be guaranteed, task completion quality is improved, code writing work is basically not needed in the task execution process of the system, cost is reduced, and efficiency is improved.
Referring to fig. 5, an embodiment of the present application discloses a specific data synchronization method, including:
step S31: packaging the general logic of the source data to be synchronized based on the acquired target data synchronization command through a preset synchronization task configuration platform to obtain a packaged general logic; and extracting a target configurable item corresponding to the target data synchronization command from the encapsulated general logic through the preset synchronization task configuration platform, and configuring the target configurable item based on the target data synchronization command to obtain target configuration information.
In this embodiment, it should be noted that, in the data synchronization process, the preset synchronization task configuration platform may perform visual configuration on target configuration information required when a subsequent system executes a data synchronization task, for example, address information and a data structure of source data to be synchronized; determining whether the configuration of independent resources is needed or not according to the priority of the service link; the configuration corresponding to the two-party interface involved in the subsequent data assembling process; the field type mapping relation between the assembled data and the target data source, and the like.
In this embodiment, the extracting, by the preset sync task configuration platform, the target configurable item corresponding to the target data sync command from the encapsulated general logic, and configuring, based on the target data sync command, the target configurable item to obtain target configuration information includes: if the target data synchronization command read by the preset synchronization task configuration platform comprises a preset service arrangement configuration command, extracting an interface configurable item in the encapsulated general logic by the preset synchronization task configuration platform; and configuring the interface configurable item through the preset synchronous task configuration platform based on the preset service arrangement configuration command to obtain interface configuration information.
In this embodiment, the configuring, by the preset synchronous task configuration platform, the interface configurable item based on the preset service arrangement configuration command to obtain interface configuration information includes: and configuring any one or more interface configurable items in the interface type, the interface address, the interface version, the interface alias, the interface dependency information, the interface access parameter information editing script and the interface calling sequence of the two-party interface by the preset synchronous task configuration platform based on the preset service editing configuration command to obtain interface configuration information.
Step S32: and reading the target configuration information, determining a target message queue by using the target configuration information, and reading source data to be synchronized from the target message queue.
Step S33: and judging whether a message queue needs to be redistributed to the source data to be synchronized based on the target configuration information, and if not, judging whether two-party interface calling needs to be carried out based on the target configuration information.
In this embodiment, the determining whether to call a two-party interface based on the target configuration information specifically includes: and judging whether to call a two-party interface or not based on the interface configuration information in the target configuration information. It can be understood that the preset synchronization task configuration platform configures the interface configurable item based on the preset service arrangement configuration command to obtain interface configuration information, and when the preset service arrangement configuration command represents that the source data to be synchronized needs to be subjected to secondary processing, the preset synchronization task configuration platform configures the interface configurable item to obtain the interface configuration information representing that two-party interface calling needs to be performed. It should be noted that, when the preset synchronous task configuration platform configures the interface configurable item based on the preset service arranging configuration command, the request parameters and response result data required by the subsequent two-party interface call can also be arranged by using the JsonPath technology, wherein when the request parameters are arranged, the current interface is based on the interface request sequence and the preset service arranging configuration command, and the JsonPath technology is used for extracting the data cached in the task context and then assembling the data into the request parameters when the next interface is called; and when the response data are arranged, assembling all the call result data in the task context into a big json object for use when the data to be synchronized are written into a target data source subsequently.
Step S34: and if the two-party interface call is judged to be needed, determining the corresponding two-party interface type and the field type mapping relation corresponding to the source data to be synchronized.
In this embodiment, if it is determined that two-party interface calling is required, determining a corresponding two-party interface type, for example, when the two-party interface type is an http interface, any one or more of a calling address, a port, and a request method of the http interface may be obtained through target configuration information, then request parameter data is constructed, calling is performed in an http live calling manner, and calling result data is stored in a corresponding task context after the calling is completed; when the two-party interface type is an rpc interface, ReferenceConfig (client call configuration) required by generalized call can be constructed through version, class name, method name and grouping information, an actual call ip Address (Internet Protocol Address) and a port (computer port) of dubbo service of an invoker object inside the ReferenceConfig are obtained in a java reflection mode, a parameter type of the dubbo service is obtained according to the ip Address and the port by using a dubbo telnet technology so as to construct a genericservobject required by the dubbo generalized call mode, request parameter data is constructed, rpc interface call can be further carried out by using the dubbo generalized call mode, and the call interface data is stored into a task context after the call is completed.
Step S35: and storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the mapping relation between the two-party interface type and the field type.
In this embodiment, for example, if the determined type of the two-party interface is an rpc interface, an rpc interface is called to obtain assembled big json, before storing source data to be synchronized to a corresponding target data source based on a field type mapping relationship, the method further includes performing one-to-one mapping processing on index fields in big json and ES index information by using a preset synchronization task configuration platform according to a json path technology and a Groovy technology to obtain the field type mapping relationship, where the json path technology is mainly used to extract a value of the big json, and the Groovy technology performs corresponding processing on fields that need to be specially processed, for example, time data formatting and data type conversion operations are performed, and then the fields are assembled into a data format required by an ES index, so as to store the source data to be synchronized to the corresponding target data source.
Therefore, the method and the system have the advantages that the related configuration of whether the two-party interface is required to be called is completed by the preset synchronization task configuration platform, the system only needs to complete the corresponding data synchronization task based on the target configuration information, the burden of the system is reduced, and the system is enabled to run more stably.
Referring to fig. 6, an embodiment of the present application discloses a data synchronization apparatus, including:
the source data reading module 11 is configured to determine a target message queue by using target configuration information configured by a preset synchronization task configuration platform, and read source data to be synchronized from the target message queue;
a judging module 12, configured to judge whether the source data to be synchronized performs redelivery processing based on the target configuration information, and if not, judge whether to perform interface call of two parties based on the target configuration information;
an information determining module 13, configured to determine a type of the second-party interface and a field type mapping relationship corresponding to the source data to be synchronized if it is determined that the second-party interface is called;
and the storage module 14 is configured to store the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relationship.
Therefore, the method comprises the steps that a target message queue is determined by utilizing target configuration information configured through a preset synchronization task configuration platform, and source data to be synchronized are read from the target message queue; judging whether a message queue needs to be redistributed to the source data to be synchronized based on the target configuration information, and judging whether two-party interface calling needs to be carried out based on the target configuration information if the message queue needs to be redistributed to the source data to be synchronized; if the two-party interface is judged to be required to be called, determining the corresponding two-party interface type and the field type mapping relation corresponding to the source data to be synchronized; and storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation. Therefore, the preset synchronous task configuration platform is introduced, and the preset synchronous task configuration platform is used for configuring to obtain the target configuration information, so that the workload required by code compiling when the preset synchronous task configuration platform is configured is greatly reduced compared with the workload required by the prior art for completing data synchronous code compiling, and the cost is further reduced; and subsequently, based on the target configuration information, a target message queue can be determined so as to read the source data to be synchronized from the target message queue, whether the source data to be synchronized is redistributed to the message queue can be judged, whether the source data to be synchronized needs to be called by a two-party interface and a field type mapping relation corresponding to the source data to be synchronized can be obtained, and then the source data to be synchronized is stored to a target data source based on the field type mapping relation, wherein code writing work is basically not needed, the redundant code condition is reduced, and the data synchronization efficiency is improved.
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application. The method specifically comprises the following steps: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input output interface 25, and a communication bus 26. Wherein, the memory 22 is used for storing a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the data synchronization method executed by the electronic device disclosed in any of the foregoing embodiments.
In this embodiment, the power supply 23 is configured to provide a working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and an external device, and a communication protocol followed by the communication interface is any communication protocol applicable to the technical solution of the present application, and is not specifically limited herein; the input/output interface 25 is configured to obtain external input data or output data to the outside, and a specific interface type thereof may be selected according to specific application requirements, which is not specifically limited herein.
The processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 21 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 21 may further include an AI (Artificial Intelligence) processor for processing a calculation operation related to machine learning.
In addition, the storage 22 is used as a carrier for storing resources, and may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., the resources stored thereon include an operating system 221, a computer program 222, data 223, etc., and the storage may be a transient storage or a permanent storage.
The operating system 221 is used for managing and controlling hardware devices and computer programs 222 on the electronic device, so as to implement operations and processing of the mass data 223 in the memory 22 by the processor 21, and may be Windows, Unix, Linux, or the like. The computer program 222 may further include a computer program that can be used to perform other specific tasks in addition to the computer program that can be used to perform the data synchronization method performed by the electronic device disclosed in any of the foregoing embodiments. The data 223 may include data received by the electronic device and transmitted from an external device, or may include data collected by the input/output interface 25 itself.
Further, an embodiment of the present application further discloses a computer-readable storage medium, in which a computer program is stored, and when the computer program is loaded and executed by a processor, the method steps executed in the data synchronization process disclosed in any of the foregoing embodiments are implemented.
Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above detailed description is provided for a data synchronization method, apparatus, device and medium provided by the present invention, and a specific example is applied in the present document to explain the principle and the implementation of the present invention, and the description of the above embodiment is only used to help understanding the method and the core idea of the present invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (10)
1. A method of data synchronization, comprising:
determining a target message queue by using target configuration information configured through a preset synchronization task configuration platform, and reading source data to be synchronized from the target message queue;
judging whether a message queue needs to be redistributed to the source data to be synchronized based on the target configuration information, and if not, judging whether two-party interface calling needs to be carried out based on the target configuration information;
if the two-party interface is judged to be required to be called, determining the corresponding two-party interface type and the field type mapping relation corresponding to the source data to be synchronized;
and storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation.
2. The data synchronization method according to claim 1, wherein before determining the target message queue using the target configuration information configured by the preset sync task configuration platform, the method further comprises:
packaging the general logic of the source data to be synchronized based on the acquired target data synchronization command through the preset synchronization task configuration platform to obtain a packaged general logic;
and extracting a target configurable item corresponding to the target data synchronization command from the encapsulated general logic through the preset synchronization task configuration platform, and configuring the target configurable item based on the target data synchronization command to obtain target configuration information.
3. The data synchronization method according to claim 2, wherein the extracting, by the preset synchronization task configuration platform, the target configurable item corresponding to the target data synchronization command from the encapsulated general logic, and configuring, based on the target data synchronization command, the target configurable item to obtain target configuration information includes:
if the target data synchronization command read by the preset synchronization task configuration platform comprises a preset service arrangement configuration command, extracting an interface configurable item in the encapsulated general logic by the preset synchronization task configuration platform;
and configuring the interface configurable item through the preset synchronous task configuration platform based on the preset service arrangement configuration command to obtain interface configuration information.
4. The data synchronization method according to claim 3, wherein the configuring, by the preset synchronization task configuration platform, the interface configurable item based on the preset service orchestration configuration command to obtain interface configuration information comprises:
and configuring any one or more interface configurable items in the interface type, the interface address, the interface version, the interface alias, the interface dependency information, the interface access parameter information editing script and the interface calling sequence of the two-party interface by the preset synchronous task configuration platform based on the preset service editing configuration command to obtain interface configuration information.
5. The data synchronization method of claim 3, wherein the determining whether to make a two-party interface call based on the target configuration information comprises:
and judging whether to call a two-party interface or not based on the interface configuration information in the target configuration information.
6. The data synchronization method according to claim 2, wherein the determining a target message queue by using target configuration information configured by a preset synchronization task configuration platform, and reading source data to be synchronized from the target message queue comprises:
reading basic configuration information in target configuration information configured by a preset synchronous task configuration platform, and determining a target message queue by using the basic configuration information; the basic configuration information comprises any one or more configuration information of target task name information, target task type information, target service level information, target responsible person information, target ES cluster information and target ES index information;
if the target message queue determined by the basic configuration information is a preset public message queue, reading source data to be synchronized from the preset public message queue; and if the target message queue determined by using the basic configuration information is a preset independent message queue, reading the source data to be synchronized from the preset independent message queue.
7. The data synchronization method according to claim 6, wherein before reading the basic configuration information in the target configuration information configured by the preset synchronization task configuration platform, the method further comprises:
and reading a target data synchronization command through the preset synchronization task configuration platform to obtain a basic information configuration command, and extracting a basic information configurable item corresponding to the basic information configuration command from the packaged general logic through the preset synchronization task configuration platform.
8. A data synchronization apparatus, comprising:
the source data reading module is used for determining a target message queue by using target configuration information configured by a preset synchronous task configuration platform and reading source data to be synchronized from the target message queue;
the judging module is used for judging whether the source data to be synchronized carries out redelivery processing based on the target configuration information, and if not, judging whether two-party interface calling is carried out based on the target configuration information;
the information determining module is used for determining the type of the two-party interface and the field type mapping relation corresponding to the source data to be synchronized if the two-party interface is judged to be called;
and the storage module is used for storing the source data to be synchronized to a target data source corresponding to the source data to be synchronized based on the two-party interface type and the field type mapping relation.
9. An electronic device, comprising:
a memory for storing a computer program;
a processor for executing the computer program to carry out the steps of the data synchronization method according to any one of claims 1 to 7.
10. A computer-readable storage medium for storing a computer program; wherein the computer program realizes the steps of the data synchronization method according to any one of claims 1 to 7 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210502756.0A CN114840277A (en) | 2022-05-10 | 2022-05-10 | Data synchronization method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210502756.0A CN114840277A (en) | 2022-05-10 | 2022-05-10 | Data synchronization method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114840277A true CN114840277A (en) | 2022-08-02 |
Family
ID=82569141
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210502756.0A Pending CN114840277A (en) | 2022-05-10 | 2022-05-10 | Data synchronization method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114840277A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118568181A (en) * | 2024-07-31 | 2024-08-30 | 北京科杰科技有限公司 | Cross-environment data synchronization method and device |
-
2022
- 2022-05-10 CN CN202210502756.0A patent/CN114840277A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118568181A (en) * | 2024-07-31 | 2024-08-30 | 北京科杰科技有限公司 | Cross-environment data synchronization method and device |
| CN118568181B (en) * | 2024-07-31 | 2024-09-27 | 北京科杰科技有限公司 | Cross-environment data synchronization method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111078315B (en) | Microservice arranging and executing method and system, architecture, equipment and storage medium | |
| CN110532020B (en) | Data processing method, device and system for micro-service arrangement | |
| US10430735B2 (en) | Designer tool for managing cloud computing services | |
| US10015264B2 (en) | Generalized proxy architecture to provide remote access to an application framework | |
| CN110716748B (en) | Service processing method, device, computer readable medium and electronic equipment | |
| US20200410031A1 (en) | Systems and methods for cloud computing | |
| CN108196915B (en) | Code processing method and device based on application container engine and storage medium | |
| CN110442396B (en) | Application program starting method and device, storage medium and electronic equipment | |
| EP3113020A1 (en) | Data processing device and method for processing serial tasks | |
| CN117075930B (en) | Computing framework management system | |
| US20140067886A1 (en) | Information processing apparatus, method of outputting log, and recording medium | |
| US20170083387A1 (en) | High-performance computing framework for cloud computing environments | |
| US8904346B1 (en) | Method and system for automated load testing of web applications | |
| CN114840277A (en) | Data synchronization method, device, equipment and medium | |
| CN112235132A (en) | Method, device, medium and server for dynamically configuring service | |
| US20210263736A1 (en) | Semantic functional wrappers of services | |
| WO2024103840A1 (en) | Mirror image file making method and apparatus, device, and nonvolatile readable storage medium | |
| CN109408057B (en) | Method, device, medium and computing equipment for automatically generating codes | |
| CN116432624A (en) | Micro-service API document generation method, device, equipment and storage medium | |
| CN110599112A (en) | Method and device for developing and maintaining network page | |
| CN115756996A (en) | Chip testing method, device and equipment and readable storage medium | |
| CN115510064A (en) | ES alarm data backfilling method, device, equipment and medium | |
| CN113064987B (en) | Data processing method, apparatus, electronic device, medium, and program product | |
| CN113835904A (en) | Remote procedure call control method, device, equipment and storage medium | |
| CN111290818B (en) | Operation method for creating application program and operation method for application program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |