CN116737764A - Method and device for data synchronization, electronic equipment and storage medium - Google Patents
Method and device for data synchronization, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116737764A CN116737764A CN202210210603.9A CN202210210603A CN116737764A CN 116737764 A CN116737764 A CN 116737764A CN 202210210603 A CN202210210603 A CN 202210210603A CN 116737764 A CN116737764 A CN 116737764A
- Authority
- CN
- China
- Prior art keywords
- metadata
- data
- cache pool
- node
- storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000004891 communication Methods 0.000 claims abstract description 35
- 238000004590 computer program Methods 0.000 claims description 12
- 230000007246 mechanism Effects 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 7
- 238000005516 engineering process Methods 0.000 abstract description 6
- 230000015556 catabolic process Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 9
- 238000012545 processing Methods 0.000 description 6
- 238000013461 design Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
- G06F16/24539—Query rewriting; Transformation using cached or materialised query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24573—Query processing with adaptation to user needs using data annotations, e.g. user-defined metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application provides a data synchronization method and device, electronic equipment and storage medium, wherein the method comprises the following steps: establishing long connection communication with a main node of a target cluster, and acquiring metadata in the main node, wherein the metadata are used for representing associated information of a plurality of data types; storing the metadata in a cache pool, wherein the cache pool is used for synchronizing the metadata in the master node; and under the condition that the indication information of the access metadata is received, accessing the cache pool to obtain the data to be queried. The method and the device solve the problems that data among nodes of metadata are not completely consistent and a main node has breakdown when the data volume is overlarge in the related technology.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a method and apparatus for data synchronization, an electronic device, and a storage medium.
Background
The metadata node is the basic data, so that the strong consistency of the data needs to be ensured, otherwise, serious problems occur in the data processing of the data node (namely, the data node in fig. 1).
Therefore, only the master node of the meta cluster provides the read-write service to the outside, and other nodes exist as copies of the master node for high availability, for example, when meta1 in fig. 1 is used as the master node, then meta2 and meta3 are both copies of meta1, so that the situation that the load of the master node is too large is caused by bad design.
Currently, there are generally two schemes in all designs, scheme 1: regardless of the performance issues of the master node, the master node provides all read-write services, as in fig. 2. The problems brought are: when the metadata volume is too large or the data node queries too frequently, the service of the main node is suspended; scheme 2: all meta nodes, whether master or slave, provide the query function as in fig. 3. The problems brought are: since the metadata is successfully written only by most nodes, when the network delay exists, the data of the master node is inconsistent with the data of the slave nodes, and the data cannot be queried when the data of the data nodes reads the nodes.
Therefore, in the related art, there are problems that data among nodes of metadata is not completely consistent and a master node crashes when the data volume is too large.
Disclosure of Invention
The application provides a data synchronization method and device, a storage medium and electronic equipment, which at least solve the problems that data among nodes of metadata are not completely consistent and a main node has breakdown when the data volume is overlarge in the related technology.
According to an aspect of an embodiment of the present application, there is provided a method of data synchronization, the method including:
establishing long connection communication with a main node of a target cluster, and acquiring metadata in the main node, wherein the metadata are used for representing associated information of a plurality of data types;
storing the metadata in a cache pool, wherein the cache pool is used for synchronizing the metadata in the master node;
and under the condition that the indication information for accessing the metadata is received, accessing the cache pool to obtain the data to be queried.
According to another aspect of the embodiment of the present application, there is also provided an apparatus for data synchronization, including:
the first acquisition unit is used for establishing long-connection communication with a main node of a target cluster and acquiring metadata in the main node, wherein the metadata are used for representing associated information of a plurality of data types;
a storage unit, configured to store the metadata in a cache pool, where the cache pool is used to synchronize metadata in the master node;
and the access unit is used for accessing the cache pool under the condition of receiving the indication information for accessing the metadata to obtain the data to be queried.
Optionally, the first acquisition unit includes:
and the connection module is used for establishing long connection communication with the master node by utilizing a long polling mechanism, wherein the long polling mechanism is used for acquiring the metadata in real time.
Optionally, the storage unit includes:
the sending module is used for sending request information to the master node;
the acquisition module is used for acquiring feedback information, wherein the feedback information is information obtained after responding to the request information;
and the storage module is used for storing the feedback information in the cache pool as the metadata.
Optionally, the storage module includes:
an obtaining subunit, configured to obtain, when it is monitored that a similarity between the metadata and initial metadata is lower than a first threshold, the latest data corresponding to the metadata, where the initial metadata is data stored in the cache pool, the latest data is data corresponding to when a time point of obtaining the feedback information is a maximum value in a timestamp, and the first threshold is a maximum value of similarity between metadata;
and the replacing subunit is used for replacing the initial metadata by the latest data and storing the latest data in the cache pool.
Optionally, the apparatus further comprises:
a second obtaining unit, configured to obtain a local memory location before the metadata is stored in the cache pool;
and the setting unit is used for setting the cache pool at the local memory position.
Optionally, the access unit comprises:
the determining module is used for determining characters to be queried according to the request information;
the matching module is used for matching a plurality of candidate data sets from the cache pool, wherein the candidate data sets comprise reference characters with similarity between the reference characters and the characters being larger than a second threshold value, and the second threshold value is used for indicating the maximum value of the similarity between the characters;
the sorting module is used for sorting the reference characters from large to small according to the similarity value, and selecting a target character from the reference characters, wherein the target character is the character corresponding to the maximum similarity value;
and the module is used for taking the target character as the data to be queried.
Optionally, the setting unit includes:
the first setting module is used for setting the cache pools at each local storage position respectively;
or alternatively, the process may be performed,
the second setting module is used for setting a public area at a plurality of local storage positions, wherein the public area is an area which each data node is allowed to access;
and the third setting module is used for setting the cache pool in the public area.
According to still another aspect of the embodiments of the present application, there is provided an electronic device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus; wherein the memory is used for storing a computer program; a processor for performing the method steps of any of the embodiments described above by running the computer program stored on the memory.
According to a further aspect of the embodiments of the present application there is also provided a computer readable storage medium having stored therein a computer program, wherein the computer program is arranged to perform the method steps of any of the embodiments described above when run.
In the embodiment of the application, through establishing long connection communication with a main node of a target cluster, metadata in the main node is obtained, wherein the metadata are used for representing associated information of a plurality of data types; storing the metadata in a cache pool, wherein the cache pool is used for synchronizing the metadata in the master node; and under the condition that the indication information for accessing the metadata is received, accessing the cache pool to obtain the data to be queried. According to the embodiment of the application, the cache pool is arranged in the data node, the metadata is stored in the cache pool, and the data in the cache pool is consistent with the metadata in the main node, so that the real-time consistency of the data is ensured, the accessed pressure of the main node is reduced, and the problems that the data among the nodes of the metadata are not completely consistent and the main node is crashed when the data volume is overlarge in the related technology are solved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the application or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to a person skilled in the art that other drawings can be obtained from these drawings without inventive effort.
FIG. 1 is a diagram of a prior art time series database architecture;
FIG. 2 is a schematic diagram of a meta master node providing all read-write services;
FIG. 3 is a diagram of an architecture in which all meta nodes provide query functions;
FIG. 4 is a flow chart of an alternative method of data synchronization according to an embodiment of the application;
FIG. 5 is an architecture diagram of an alternative data synchronization in accordance with an embodiment of the present application;
FIG. 6 is a block diagram of an alternative apparatus for data synchronization in accordance with an embodiment of the present application;
fig. 7 is a block diagram of an alternative electronic device in accordance with an embodiment of the present application.
Detailed Description
In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
When the existing meta cluster provides the read-write service to the outside, there are two general designs: the performance problem of the main node is not considered, and the main node provides all read-write services; all meta nodes, whether master or slave, provide the query function. This presents a problem in that the master node service may hang up when the amount of metadata is too large or the data node queries too frequently, and when there is a network delay, there may be an inconsistency between the master node and the slave node data, resulting in errors in the data node reading data.
To solve the above-mentioned problems, an embodiment of the present application provides a method for data synchronization, as shown in fig. 4, fig. 4 is a schematic flow diagram of an alternative method for data synchronization according to an embodiment of the present application, where the method flow is applied to a data node server side, and the method flow may include the following steps:
step S401, establishing long connection communication with a main node of a target cluster, and acquiring metadata in the main node, wherein the metadata are used for representing associated information of a plurality of data types;
step S402, storing the metadata in a cache pool, wherein the cache pool is used for synchronizing the metadata in the master node;
step S403, accessing the cache pool to obtain the data to be queried under the condition that the indication information of the access metadata is received.
Optionally, in the embodiment of the present application, the data node is used as an execution body, and a long polling mechanism long polling technology may be used to establish long connection communication with a master node of the target cluster meta cluster, so that the data node obtains metadata of the master node. It is understood that the master node is a storage form that stores metadata in the database; metadata is associated information for characterizing a plurality of data types, such as user information, node information, data distribution information, and the like; the data node (i.e. data node) is used for storing real data having a corresponding relation with metadata in the main node, for example, the metadata includes user information: age, then the data held within the data node may be 24.
Meanwhile, the long polling mechanism is that the client side immediately hangs up after initiating a request to the server side, and the server side does not actively push information to the client side until the data of the server side is updated. In the period, the client will not send redundant requests, the server will not process, only the basic connection information is reserved, and after the server has update and pushes to the client, the client processes the update and then re-initiates the next round of requests.
After the metadata in the master node is obtained, the metadata is stored in a cache pool arranged at the memory of the data node, so that the data recorded in the cache pool are synchronized with the metadata.
When the data node needs to access the metadata, the data node server side receives an indication message, and at the moment, the data node accesses the cache pool to obtain the data to be queried.
In the embodiment of the application, the metadata in the master node is acquired by establishing long-connection communication with the master node of the target cluster, wherein the metadata is used for representing the associated information of a plurality of data types; storing the metadata in a cache pool, wherein the cache pool is used for synchronizing the metadata in the master node; and under the condition that the indication information of the access metadata is received, accessing the cache pool to obtain the data to be queried. According to the embodiment of the application, the cache pool is arranged in the data node, the metadata is stored in the cache pool, and the data in the cache pool is consistent with the metadata in the main node, so that the real-time consistency of the data is ensured, the accessed pressure of the main node is reduced, and the problems that the data among the nodes of the metadata are not completely consistent and the main node is crashed when the data volume is overlarge in the related technology are solved.
As an alternative embodiment, storing the metadata in the cache pool comprises:
sending request information to a master node;
acquiring feedback information, wherein the feedback information is information obtained after responding to the request information;
the feedback information is stored as metadata in a cache pool.
Optionally, when the data node and the master node store long connection requests, the data node needs to access metadata of the master node, send request information to the master node, the master node receives the request information and responds to the request information, returns feedback information, sends the feedback information to the data node, and the data node receives the feedback information and stores the obtained feedback information in a cache pool.
In the embodiment of the application, the metadata of the time sequence database is ensured to be consistent, and the efficiency of acquiring the metadata by the data nodes is improved through the set cache pool.
As an alternative embodiment, storing the feedback information as metadata in the cache pool comprises:
under the condition that the similarity between the metadata and the initial metadata is lower than a first threshold value, acquiring the latest data corresponding to the metadata, wherein the initial metadata is data stored in a cache pool, the latest data is data corresponding to the time point of obtaining feedback information is the maximum value in a time stamp, and the first threshold value is the maximum value for determining the similarity between the metadata;
the initial metadata is replaced with the latest data and the latest data is stored in the cache pool.
Optionally, in the long-connection communication between the data node and the master node, when the master node data changes, for example, when the similarity between the current metadata and the initial metadata stored in the cache pool is monitored to be lower than a first threshold (for example, 99%), the current metadata is determined to change, at this time, the data, corresponding to the feedback information corresponding to the metadata, at the maximum value position in the timestamp at the corresponding moment of time, is obtained as the latest data, that is, the latest obtained metadata is taken as the replacement data, the initial metadata is replaced, and the latest data is stored in the cache pool.
In the embodiment of the application, the metadata in the cache pool is updated in real time, so that the data in the cache pool and the metadata of the main node are always consistent, and the strong consistency of the metadata is ensured.
As an alternative embodiment, before storing the metadata in the cache pool, the method comprises:
acquiring a local memory location;
and setting a cache pool at the local memory location.
Alternatively, as shown in fig. 5, a cache pool is provided at a local memory location of the data node, so that the number of data nodes is the same as the number of cache pools, see fig. 5.
Alternatively, in order to save resources for setting the cache pool, a common area may be set at the local storage locations of the plurality of data nodes, where the common area is an area that is accessible to each data node, and then the cache pool is set in the common area.
In the embodiment of the application, when the data node needs to access the metadata, the metadata can be directly obtained from the local cache pool, so that the pressure of the master node is reduced, the upper processing limit of the master node is improved, and the access efficiency is improved.
As an alternative embodiment, accessing the cache pool, the obtaining the data to be queried includes:
determining characters to be queried according to the request information;
matching a plurality of candidate data sets from the cache pool, wherein the candidate data sets contain reference characters with similarity between characters being larger than a second threshold value, and the second threshold value is used for indicating the maximum value of the similarity between the characters, such as 80%;
sorting the reference characters according to the similarity value from large to small, and selecting a target character from the reference characters, wherein the target character is the character corresponding to the maximum similarity value;
and taking the target character as the data to be queried.
Optionally, when accessing the cache pool, the data node generally matches a plurality of candidate data sets from the cache pool in a traversal manner according to the character to be queried determined by the request information, where the embodiment of the application sets a second threshold as a screening condition to obtain the candidate data sets. It will be appreciated that the candidate data set should include reference characters having a similarity with the characters greater than a second threshold, and the second threshold is also a maximum value set to indicate the similarity between the characters, and when the similarity is greater than the second threshold, the obtained reference characters are described as being similar to the characters to be queried.
And the number of the obtained reference characters is a plurality, the similarity degree values between the characters to be queried obtained by each reference character are arranged according to the sequence from the large value to the small value, the reference character with the largest similarity degree is selected, namely the character which is ordered at the forefront is used as the target character, and then the target character is used as the data to be queried and fed back to the data node.
In the embodiment of the application, the target character with the highest similarity meeting the second threshold value and the character similarity to be queried of the data node is found by comparing the data in the cache pool, so that the aim of quickly querying is fulfilled, and the effect of improving the access efficiency is achieved.
It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.
From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM (Read-Only Memory)/RAM (Random Access Memory), magnetic disk, optical disk), comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method of the embodiments of the present application.
According to another aspect of the embodiment of the present application, there is also provided a data synchronization apparatus for implementing the above-mentioned data synchronization method. Fig. 6 is a block diagram of an alternative apparatus for data synchronization according to an embodiment of the present application, as shown in fig. 6, the apparatus may include:
a first obtaining unit 601, configured to establish long-connection communication with a master node of a target cluster, and obtain metadata in the master node, where the metadata is used to characterize association information of multiple data types;
a storage unit 602, configured to store metadata in a cache pool, where the cache pool is used to synchronize metadata in a master node;
and the accessing unit 603 is configured to access the cache pool to obtain data to be queried when receiving the indication information of accessing the metadata.
It should be noted that, the first obtaining unit 601 in this embodiment may be used to perform the above-mentioned step S401, the storage unit 602 in this embodiment may be used to perform the above-mentioned step S402, and the access unit 603 in this embodiment may be used to perform the above-mentioned step S203.
Through the module, the embodiment of the application sets the cache pool in the data node, and stores the metadata in the cache pool, and the data in the cache pool is consistent with the metadata in the main node, so that the real-time consistency of the data is ensured, the accessed pressure of the main node is reduced, and the problems that the data among the nodes of the metadata are not completely consistent and the main node has breakdown when the data volume is overlarge in the related technology are solved.
As an alternative embodiment, the first acquisition unit comprises:
and the connection module is used for establishing long connection communication with the master node by utilizing a long polling mechanism, wherein the long polling mechanism is used for acquiring the metadata in real time.
As an alternative embodiment, the storage unit includes:
the sending module is used for sending request information to the master node;
the acquisition module is used for acquiring feedback information, wherein the feedback information is information obtained after the response request information;
and the storage module is used for storing the feedback information in the cache pool as metadata.
As an alternative embodiment, the memory module includes:
the acquisition subunit is configured to acquire, when it is monitored that a similarity between metadata and initial metadata is lower than a first threshold, latest data corresponding to the metadata, where the initial metadata is data stored in the cache pool, the latest data is data corresponding to when a time point of obtaining feedback information is a maximum value in a timestamp, and the first threshold is a maximum value of similarity between metadata;
and the replacing subunit is used for replacing the initial metadata by the latest data and storing the latest data in the cache pool.
As an alternative embodiment, the apparatus further comprises:
a second obtaining unit, configured to obtain a local memory location before storing the metadata in the cache pool;
and the setting unit is used for setting a cache pool at the local memory position.
As an alternative embodiment, the access unit comprises:
the determining module is used for determining characters to be queried according to the request information;
the matching module is used for matching a plurality of candidate data sets from the cache pool, wherein the candidate data sets comprise reference characters with similarity between the reference characters and characters being larger than a second threshold value, and the second threshold value is used for indicating the maximum value of the similarity between the characters;
the sequencing module is used for sequencing the reference characters from large to small according to the similarity value, and selecting target characters from the reference characters, wherein the target characters are the characters corresponding to the maximum similarity value;
and the module is used for taking the target character as data to be queried.
As an alternative embodiment, the setting unit comprises:
the first setting module is used for setting a cache pool at each local storage position respectively;
or alternatively, the process may be performed,
the second setting module is used for setting a public area at a plurality of local storage positions, wherein the public area is an area which each data node is allowed to access;
and the third setting module is used for setting a cache pool in the public area.
According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, which may be a server, a terminal, or a combination thereof, for implementing the above-mentioned method of data synchronization.
Fig. 7 is a block diagram of an alternative electronic device, according to an embodiment of the application, as shown in fig. 7, comprising a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702 and the memory 703 communicate with each other via the communication bus 704, wherein,
a memory 703 for storing a computer program;
the processor 701 is configured to execute the computer program stored in the memory 703, and implement the following steps:
establishing long connection communication with a main node of a target cluster, and acquiring metadata in the main node, wherein the metadata are used for representing associated information of a plurality of data types;
storing the metadata in a cache pool, wherein the cache pool is used for synchronizing the metadata in the master node;
and under the condition that the indication information of the access metadata is received, accessing the cache pool to obtain the data to be queried.
Alternatively, in the present embodiment, the above-described communication bus may be a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus, or an EISA (Extended Industry Standard Architecture ) bus, or the like. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, only one thick line is shown in fig. 7, but not only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The memory may include RAM or may include non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
As an example, as shown in fig. 7, the memory 703 may include, but is not limited to, a first acquiring unit 601, a storage unit 602, and an access unit 603 in the apparatus including the data synchronization. In addition, other module units in the above-mentioned data synchronization device may be included, but are not limited to, and are not described in detail in this example.
The processor may be a general purpose processor and may include, but is not limited to: CPU (Central Processing Unit ), NP (Network Processor, network processor), etc.; but also DSP (Digital Signal Processing, digital signal processor), ASIC (Application Specific Integrated Circuit ), FPGA (Field-Programmable Gate Array, field programmable gate array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.
In addition, the electronic device further includes: and the display is used for displaying the result of the data synchronization.
Alternatively, specific examples in this embodiment may refer to examples described in the foregoing embodiments, and this embodiment is not described herein.
It will be understood by those skilled in the art that the structure shown in fig. 7 is only illustrative, and the device implementing the above data synchronization method may be a terminal device, and the terminal device may be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, a palmtop computer, a mobile internet device (Mobile Internet Devices, MID), a PAD, etc. Fig. 7 is not limited to the structure of the electronic device described above. For example, the terminal device may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in fig. 7, or have a different configuration than shown in fig. 7.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program for instructing a terminal device to execute in association with hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: flash disk, ROM, RAM, magnetic or optical disk, etc.
According to yet another aspect of an embodiment of the present application, there is also provided a storage medium. Alternatively, in the present embodiment, the storage medium described above may be used for program codes of a method of performing data synchronization.
Alternatively, in this embodiment, the storage medium may be located on at least one network device of the plurality of network devices in the network shown in the above embodiment.
Alternatively, in the present embodiment, the storage medium is configured to store program code for performing the steps of:
establishing long connection communication with a main node of a target cluster, and acquiring metadata in the main node, wherein the metadata are used for representing associated information of a plurality of data types;
storing the metadata in a cache pool, wherein the cache pool is used for synchronizing the metadata in the master node;
and under the condition that the indication information of the access metadata is received, accessing the cache pool to obtain the data to be queried.
Alternatively, specific examples in the present embodiment may refer to examples described in the above embodiments, which are not described in detail in the present embodiment.
Alternatively, in the present embodiment, the storage medium may include, but is not limited to: various media capable of storing program codes, such as a U disk, ROM, RAM, a mobile hard disk, a magnetic disk or an optical disk.
According to yet another aspect of embodiments of the present application, there is also provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium; the computer instructions are read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, cause the computer device to perform the method steps of data synchronization in any of the embodiments described above.
The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
The integrated units in the above embodiments may be stored in the above-described computer-readable storage medium if implemented in the form of software functional units and sold or used as separate products. Based on such understanding, the technical solution of the present application may be embodied essentially or partly in the form of a software product, or all or part of the technical solution, which is stored in a storage medium, and includes several instructions for causing one or more computer devices (which may be personal computers, servers or network devices, etc.) to perform all or part of the steps of the method for data synchronization of the various embodiments of the present application.
In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In several embodiments provided by the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and are merely a logical functional division, and there may be other manners of dividing the apparatus in actual implementation, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in the present embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application, which are intended to be comprehended within the scope of the present application.
Claims (10)
1. A method of data synchronization, the method comprising:
establishing long connection communication with a main node of a target cluster, and acquiring metadata in the main node, wherein the metadata are used for representing associated information of a plurality of data types;
storing the metadata in a cache pool, wherein the cache pool is used for synchronizing the metadata in the master node;
and under the condition that the indication information for accessing the metadata is received, accessing the cache pool to obtain the data to be queried.
2. The method of claim 1, wherein the establishing long-connection communication with the master node of the target cluster comprises:
and establishing long-connection communication with the master node by using a long-polling mechanism, wherein the long-polling mechanism is used for acquiring the metadata in real time.
3. The method of claim 1, wherein storing the metadata in a cache pool comprises:
transmitting request information to the master node;
acquiring feedback information, wherein the feedback information is information obtained after responding to the request information;
and storing the feedback information as the metadata in the cache pool.
4. The method of claim 3, wherein the storing the feedback information as the metadata in the cache pool comprises:
under the condition that the similarity between the metadata and the initial metadata is lower than a first threshold value, acquiring the latest data corresponding to the metadata, wherein the initial metadata is data stored in the cache pool, the latest data is data corresponding to the time point of obtaining the feedback information is the maximum value in the timestamp, and the first threshold value is the maximum value of similarity among the metadata;
and replacing the initial metadata by the latest data, and storing the latest data in the cache pool.
5. The method of claim 1, wherein prior to said storing the metadata in the cache pool, the method comprises:
acquiring a local memory location;
and setting the cache pool at the local memory location.
6. The method of claim 3, wherein accessing the cache pool to obtain the data to be queried comprises:
determining the character to be queried according to the request information;
matching a plurality of candidate data sets from the cache pool, wherein the candidate data sets comprise reference characters with similarity between the reference characters and the characters being larger than a second threshold value, and the second threshold value is used for indicating the maximum value of the similarity between the characters;
sorting the reference characters according to the similarity value from large to small, and selecting a target character from the reference characters, wherein the target character is the character corresponding to the maximum similarity value;
and taking the target character as the data to be queried.
7. The method of claim 5, wherein the setting the cache pool at the local memory location comprises:
setting the cache pools at each local storage position respectively;
or alternatively, the process may be performed,
setting a common area at a plurality of local storage locations, wherein the common area is an area which each data node is allowed to access;
and setting the cache pool in the public area.
8. An apparatus for data synchronization, the apparatus comprising:
the first acquisition unit is used for establishing long-connection communication with a main node of a target cluster and acquiring metadata in the main node, wherein the metadata are used for representing associated information of a plurality of data types;
a storage unit, configured to store the metadata in a cache pool, where the cache pool is used to synchronize metadata in the master node;
and the access unit is used for accessing the cache pool under the condition of receiving the indication information for accessing the metadata to obtain the data to be queried.
9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, characterized in that,
the memory is used for storing a computer program;
the processor is configured to perform the method steps of any of claims 1 to 7 by running the computer program stored on the memory.
10. A computer-readable storage medium, characterized in that the storage medium has stored therein a computer program, wherein the computer program, when executed by a processor, implements the method steps of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210210603.9A CN116737764A (en) | 2022-03-04 | 2022-03-04 | Method and device for data synchronization, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210210603.9A CN116737764A (en) | 2022-03-04 | 2022-03-04 | Method and device for data synchronization, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116737764A true CN116737764A (en) | 2023-09-12 |
Family
ID=87915659
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210210603.9A Pending CN116737764A (en) | 2022-03-04 | 2022-03-04 | Method and device for data synchronization, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116737764A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116991635A (en) * | 2023-09-26 | 2023-11-03 | 武汉吧哒科技股份有限公司 | Data synchronization method and data synchronization device |
-
2022
- 2022-03-04 CN CN202210210603.9A patent/CN116737764A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116991635A (en) * | 2023-09-26 | 2023-11-03 | 武汉吧哒科技股份有限公司 | Data synchronization method and data synchronization device |
| CN116991635B (en) * | 2023-09-26 | 2024-01-19 | 武汉吧哒科技股份有限公司 | Data synchronization method and data synchronization device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106878473B (en) | Message processing method, server cluster and system | |
| CN111555963B (en) | Message pushing method and device, electronic equipment and storage medium | |
| CN111291079A (en) | Data query method and device | |
| CN104246767A (en) | Telemetry system for a cloud synchronization system | |
| CN107181686B (en) | Method, device and system for synchronizing routing table | |
| CN107147748B (en) | File uploading method and device | |
| CN109981569B (en) | Network system access method, device, computer equipment and readable storage medium | |
| CN111177165A (en) | Method, device and equipment for detecting data consistency | |
| CN111049928A (en) | Data synchronization method, system, electronic device and computer readable storage medium | |
| CN113010549A (en) | Data processing method based on remote multi-active system, related equipment and storage medium | |
| CN116737764A (en) | Method and device for data synchronization, electronic equipment and storage medium | |
| CN107025257B (en) | Transaction processing method and device | |
| US11128622B2 (en) | Method for processing data request and system therefor, access device, and storage device | |
| CN112148206A (en) | Data reading and writing method and device, electronic equipment and medium | |
| CN108038127B (en) | Data synchronization method, device, terminal equipment and storage medium | |
| CN112749172A (en) | Data synchronization method and system between cache and database | |
| CN108647139B (en) | System test method, device, storage medium and electronic device | |
| CN110765125A (en) | Data storage method and device | |
| CN111586438B (en) | Method, device and system for processing service data | |
| CN110362535B (en) | File management method, device and system | |
| CN113419901A (en) | Data disaster recovery method and device and server | |
| CN107679093B (en) | Data query method and device | |
| CN115004662A (en) | Data synchronization method, data synchronization device, data storage system and computer readable medium | |
| CN108694209B (en) | Distributed index method based on object and client | |
| CN110837454A (en) | Interface monitoring processing method and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |