CN110543512B - Information synchronization method, device and system - Google Patents
Information synchronization method, device and system Download PDFInfo
- Publication number
- CN110543512B CN110543512B CN201810532857.6A CN201810532857A CN110543512B CN 110543512 B CN110543512 B CN 110543512B CN 201810532857 A CN201810532857 A CN 201810532857A CN 110543512 B CN110543512 B CN 110543512B
- Authority
- CN
- China
- Prior art keywords
- data
- instance
- updated
- document
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012544 monitoring process Methods 0.000 claims description 80
- 230000007246 mechanism Effects 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 10
- 230000001960 triggered effect Effects 0.000 claims description 8
- 230000002159 abnormal effect Effects 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 11
- 238000012545 processing Methods 0.000 description 5
- 230000009466 transformation Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
Images
Abstract
The embodiment of the application shows a data synchronization method, a device and a system, and the technical scheme shown in the embodiment of the application establishes a main instance and a standby instance at a Go end, monitors the operation state of the main instance in real time, and directly calls the standby instance as a connection medium between a query database and a Java end if any operation abnormal condition occurs in the main instance, so that the synchronization of data in an Elasticissearch search engine and data in the query database is realized. According to the technical scheme shown in the embodiment of the application, the problem that the query database and the Java end are in a cut-off state due to the fact that the Go end is down is solved through switching among the examples, and the synchronization between the data in the Elasticissearch search engine and the data in the query database is ensured at any time.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data synchronization method, apparatus, and system.
Background
Nowadays, automobiles have gone into thousands of households, and industries related to automobiles are also developed vigorously. For some automotive industries, there is a need to manage, process, and manage large and complex vehicle information. For example, the vehicle information management platform displays the latest data of the vehicle to the user in real time to ensure that the user acquires accurate vehicle information, but some vehicle data are in a real-time update state, for example: the maintenance record of the vehicle is in a real-time updating state along with the use of the vehicle by a user, so that the vehicle information management platform updates the data in real time to ensure that the data shown by the platform is accurate.
Fig. 1 shows a vehicle information management platform in the prior art, which generally includes: the system comprises a query database, a Go end and a Java end, wherein the query database is used for acquiring vehicle information in real time and storing the vehicle information, an example is arranged in the Go end and is used for monitoring the query database and receiving a binlog log of the query database, the binlog log comprises historical data and real-time data, and the historical data and the real-time data are compared to determine and update items and send the update items to the Java end; an Elasticissearch search engine is arranged in the Java end to complete the statistics of the vehicle information, and the Java end traverses the query database by receiving the update items to realize the synchronization of the data in the Elasticissearch engine and the data in the query database.
According to the vehicle information management platform shown in the prior art, the Go end is equivalent to a medium for connecting the query database and the Java end, the query database is communicated with the Java end through the Go end, and therefore synchronization between data in the Elasticissearch search engine and data in the query database is guaranteed.
Disclosure of Invention
The invention aims to provide a data synchronization method, a device and a system, which are used for solving the technical problem that when a Go end of a vehicle information management platform shown in the prior art is down, a query database and a Java end are in a cut-off state, and the synchronization of data in an Elasticissearch search engine and data in the query database cannot be realized.
A first aspect of an embodiment of the present application shows a data synchronization method, where the method includes:
establishing an instance set to contact a zookeeper, the instance set comprising: a primary instance, and, a standby instance;
and monitoring the running condition of the main instance, and calling the standby instance as a connecting medium between the query database and the Java end if the main instance does not run normally, so as to realize the synchronization of the data in the Elasticissearch search engine and the data in the query database.
Optionally, the step of monitoring the operation status of the main instance and calling the standby instance as a connection medium for querying the database and the Java side if the main instance is not operating normally comprises:
judging whether the main instance establishes contact with the zookeeper through a global monitoring mechanism;
if the connection is not established successfully, calling the standby instance as a connection medium for inquiring the database and the Java end;
if the contact is successfully established, establishing a temporary node on the zookeeper according to the operation logic of the main instance;
sequentially traversing the temporary nodes according to the operation logic of the main example;
determining a problem node according to the operation condition of the temporary node;
and calling a standby instance as a connection medium between the query database and the Java terminal, and traversing standby nodes in the standby instance, wherein the standby nodes are nodes which are logically corresponding to the problem nodes in operation.
Optionally, the step of monitoring the operation status of the main instance, and if the main instance does not operate normally, calling the standby instance as a connection medium for querying the database and the Java side includes:
judging whether the main instance establishes contact with the zookeeper through a global monitoring mechanism;
if the connection is not established successfully, calling the standby instance as a connection medium for inquiring the database and the Java end;
if the contact is successfully established, establishing a temporary node on the zookeeper according to the operation logic of the main instance;
sequentially traversing the temporary nodes according to the operation logic of the main example;
receiving a monitoring event of a temporary node;
judging whether the event recorded by the monitoring event is a node change event or not;
if yes, informing the zookeeper to acquire the monitoring event of the standby instance;
and calling the standby instance service with the minimum sequence number corresponding to the monitoring event of the standby instance as a connection medium between the query database and the Java terminal.
Optionally, the step of synchronizing data in the Elasticsearch search engine with data in the query database includes:
determining an update item, the update item comprising: updating the column names, updating the data, and updating the column names to be the column names corresponding to the updated data;
determining a target index according to the updated column name and the updated data;
traversing the query database based on the target index, and searching for an updated document ID;
and sending the updated document ID to the Java terminal, searching updated data by the Java terminal based on the updated document ID, and updating the index and the data corresponding to the Elasticissearch search engine by using the updated data.
Optionally, the step of traversing the query database based on the target index to search for updated document IDs includes:
determining sub-target indexes of the target index according to data service logic in the query database;
determining a target index according to the target index, and determining sub-target node documents according to the sub-target index;
and generating an update document ID according to the target node document and the sub-target node documents.
Optionally, the step of traversing the query database based on the target index to search for updated document IDs includes:
determining a local document ID according to the target index;
the document ID containing the local document ID is determined as the updated document ID.
Optionally, the step of determining the update item includes:
acquiring monitoring data, wherein the monitoring data comprises: historical data, and, real-time data;
and comparing the historical data with the real-time data input gorouine to determine an updating item.
Optionally, the step of comparing the historical data with the real-time data input gorouine to determine the updated item includes:
comparing the historical data with the real-time data input gorouine, and determining an item to be updated according to a comparison result;
judging whether the item to be updated triggers the updating operation of the Elasticissearch;
if so, determining the item to be updated as an updated item;
and if not, deleting the item to be updated.
A second aspect of embodiments of the present application shows a data synchronization apparatus, including:
the establishing unit is used for establishing an instance set to be in contact with the zookeeper, and the instance set comprises: a primary instance, and, a standby instance;
and the monitoring unit is used for monitoring the running condition of the main example, and calling the standby example as a connecting medium between the query database and the Java terminal if the main example does not run normally, so as to realize the synchronization of the data in the Elasticissearch search engine and the data in the query database.
Optionally, the listening unit includes:
the first main example judging unit is used for judging whether the main example establishes contact with the zookeeper through a global monitoring mechanism;
the first standby instance calling unit is used for calling the standby instance as a connecting medium for inquiring the database and the Java terminal if the contact is not successfully established;
the first temporary node establishing unit is used for establishing a temporary node on the zookeeper according to the operation logic of the main example if the contact is successfully established;
the first temporary node traversing unit is used for sequentially traversing the temporary nodes according to the operation logic of the main example;
the problem node determining unit is used for determining a problem node according to the running condition of the temporary node;
and the first standby instance calling unit is used for calling the standby instance as a connecting medium of the query database and the Java end and traversing the standby nodes in the standby instance, wherein the standby nodes are nodes which are logically corresponding to the problem nodes in operation.
Optionally, the listening unit includes:
the second main example judging unit is used for judging whether the main example establishes contact with the zookeeper through a global monitoring mechanism;
the second standby instance calling unit is used for calling the standby instance as a connecting medium for inquiring the database and the Java end if the contact is not established successfully;
the second temporary node establishing unit is used for establishing a temporary node on the zookeeper according to the operation logic of the main example if the contact is successfully established;
the second temporary node traversing unit sequentially traverses the temporary nodes according to the operation logic of the main example;
a monitoring event receiving unit, configured to receive a monitoring event of a temporary node;
a monitoring event judging unit, configured to judge whether an event recorded by the monitoring event is a node change event;
a notification unit, configured to notify the zookeeper to acquire a monitoring event of the standby instance if the zookeeper is the standby instance;
and the second standby instance calling unit is used for calling the standby instance service with the minimum sequence number corresponding to the monitoring event of the standby instance as a connection medium between the query database and the Java end.
Optionally, the listening unit includes:
an update item determination unit configured to determine an update item, the update item including: updating the column names, updating the data, and updating the column names to be the column names corresponding to the updated data;
the index determining unit is used for determining a target index according to the updated column name and the updated data;
the searching unit is used for traversing the query database and searching the updated document ID based on the target index;
and the updating unit is used for sending the updated document ID to the Java end, searching updated data by the Java end based on the updated document ID, and updating the index and the data corresponding to the Elasticissearch search engine by using the updated data.
Optionally, the search unit includes:
a sub-target index determining unit, configured to determine a sub-target index of the target index according to the data service logic in the query database;
the target document determining unit is used for determining a target index according to the target index and determining sub-target node documents according to the sub-target index;
and the update document ID generating unit is used for generating an update document ID according to the target node document and the sub-target node documents.
Optionally, the search unit includes:
a local document ID determining unit, configured to determine a local document ID according to the target index;
an update document ID determination unit for determining a document ID containing the local document ID as an update document ID.
Optionally, the update item determination unit includes:
an obtaining unit configured to obtain monitoring data, where the monitoring data includes: historical data, and, real-time data;
and the first updating item determining unit is used for comparing the historical data with the real-time data input gorouine to determine an updating item.
Optionally, the first update item determination unit includes:
the comparison unit is used for comparing the historical data with the real-time data input gorouine and determining an item to be updated according to a comparison result;
the judging unit is used for judging whether the item to be updated triggers the updating operation of the Elasticissearch;
a second update item determination unit, configured to determine, if triggered, that the item to be updated is an update item;
and the deleting unit is used for deleting the item to be updated if the item is not triggered.
A third aspect of an embodiment of the present application shows a data synchronization system, where the system includes: the system comprises an application platform server and a query database connected with the application platform server, wherein the query database is arranged in the application platform server or is arranged independently, and the application platform server is connected with a Java end through the Internet;
the Java terminal is used for searching updated data based on the updated document ID, and updating the index and the data corresponding to the Elasticissearch search engine by using the updated data.
The application platform server is used for realizing the method shown in the embodiment of the application;
and the query database is used for storing related data.
A fourth aspect of embodiments of the present application shows a server, where the server includes:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the methods of the embodiments of the present application.
As can be seen from the foregoing technical solutions, embodiments of the present application illustrate a data synchronization method, apparatus, and system, where the method includes: establishing an instance set to contact a zookeeper, the instance set comprising: a primary instance, and, a standby instance; and monitoring the running condition of the main example, and calling the standby example as a connecting medium between the query database and the Java terminal if the main example does not run normally, so as to realize the synchronization of the data in the Elasticissearch search engine and the data in the query database. According to the technical scheme, the main instance and the standby instance are established at the Go end, the operation state of the main instance is monitored in real time, if any operation abnormal condition occurs in the main instance, the standby instance is directly called to serve as a connecting medium of the query database and the Java end, and the synchronization of data in an Elasticissearch search engine and data in the query database is achieved. According to the technical scheme shown in the embodiment of the application, the problem that the query database and the Java end are in a cut-off state due to the fact that the Go end is down is solved through switching among the examples, and the synchronization between the data in the Elasticissearch search engine and the data in the query database is ensured at any time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required in the embodiments will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a block diagram showing a structure of a vehicle information management platform according to the related art;
FIG. 2 is a block diagram of an information synchronization system in accordance with a preferred embodiment;
FIG. 3 illustrates a flow diagram of a method for synchronizing information in accordance with a preferred embodiment;
FIG. 4 is a detailed flowchart of step S102, shown in accordance with a preferred embodiment;
FIG. 5 is a detailed flowchart of step S10215 shown in accordance with a preferred embodiment;
FIG. 6 is a detailed flowchart of step S102 shown in accordance with yet another preferred embodiment;
FIG. 7 is a detailed flowchart of step S10223, shown in accordance with a preferred embodiment;
FIG. 8 is a detailed flowchart of step S10223 shown in accordance with yet another preferred embodiment;
fig. 9 is a detailed flowchart of step S10221 shown in accordance with still another preferred embodiment;
fig. 10 is a detailed flowchart illustrating step S102212 according to still another preferred embodiment;
FIG. 11 is a block diagram of an information synchronization apparatus in accordance with a preferred embodiment;
FIG. 12 is a block diagram illustrating the structure of a snoop unit in accordance with a preferred embodiment;
FIG. 13 is a block diagram illustrating the structure of a problem node determination unit in accordance with a preferred embodiment;
FIG. 14 is a block diagram illustrating the structure of a listening unit according to yet another preferred embodiment;
FIG. 15 is a block diagram of a search unit in accordance with a preferred embodiment;
FIG. 16 is a block diagram illustrating a structure of a search unit in accordance with yet another preferred embodiment;
FIG. 17 is a block diagram illustrating the structure of an update item determination unit in accordance with a preferred embodiment;
FIG. 18 is a block diagram illustrating a first update item determination unit in accordance with a preferred embodiment;
FIG. 19 illustrates a server in accordance with a preferred embodiment;
FIG. 20 is a JSON data structure of a typical tree diagram.
Detailed Description
The technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
In the embodiment of the application, MQ: the full name MessageQueue, chinese name message queue, is a communication method for application programs. Applications communicate by writing and retrieving queued data (messages) for the application without requiring a dedicated connection to link them. Message passing refers to the communication between programs by sending data in a message, rather than communicating with each other through direct calls, which are often used for techniques such as remote procedure calls.
And the ElasticSearch is a search server based on Lucene. It provides a distributed multi-user capable full-text search engine based on restful web interface. The Elasticsearch was developed in Java and published as open source under the Apache licensing terms, and is currently a popular enterprise-level search engine.
MySqlBinlog, namely a binary log of a MYSQL database, which is also called a binary log, records SQL sentences for data change or potential data change, and stores the SQL sentences in a disk in a binary form;
zookeeper is a distributed and open source distributed application program coordination service, is an open source implementation of Chubby of Google, and is an important component of Hadoop and Hbase. The software is used for providing a consistency service for distributed application, and the provided functions comprise: configuration maintenance, domain name service, distributed synchronization, group service, etc.
Example 1:
referring to fig. 2, a first aspect of the embodiments of the present application shows a data synchronization system, which includes: the system comprises an application platform server 31 and a query database 32 connected with the application platform server 31, wherein the query database 32 is arranged inside the application platform server 31 or is arranged independently, and the application platform server 31 is connected with a Java terminal 33 through the Internet;
the Java terminal 33 is configured to search for updated data based on the updated document ID, and update the index and the data corresponding to the Elasticsearch engine with the updated data.
The application platform server 31 is configured to implement the method shown in the embodiment of the present application;
for establishing an instance set in contact with a zookeeper, the instance set comprising: a primary instance, and, a standby instance;
first, link the zookeeper with an instance set, wherein the instance set comprises: a primary instance, and, a standby instance;
the main example is firstly used as a connecting medium of a query database and a Java end, and the synchronization of data in an elastic search engine and data in the query database is ensured;
and monitoring the running condition of the main instance, and calling the standby instance as a connecting medium between the query database and the Java end if the main instance does not run normally, so as to realize the synchronization of the data in the Elasticissearch search engine and the data in the query database.
If the main instance occurs, for example, the main instance is disconnected from the query database, or the main instance internal program occurs, the Java terminal and the query database cannot synchronize data, and at this time, the embodiment of the application shows that the application platform server calls the standby instance as a connection medium between the query database and the Java terminal through zopeeper, so as to synchronize data in the Elasticsearch engine and data in the query database.
The method comprises the steps that a main instance or a standby instance is used for monitoring a binlog log, the binlog log comprises historical data and real-time data, when the real-time data is different from the historical data, an updating item is generated, a target index corresponding to the updating item is determined, updating document IDs of all updating documents are determined through the target index, the updating document IDs are packaged into messages and sent to a Java end, the messages are received by the Java end, the messages are analyzed by the Java end, the latest data are searched from a query database according to the updating document IDs, indexes corresponding to the latest data updating elastic search and documents are updated according to the latest data, according to the technical scheme, the binlog log is monitored, when the database is updated, a program monitoring mechanism is actively notified, according to the changed columns, whether the indexes are affected or not is judged, the documents in the indexes need to be updated, the corresponding documents are updated if needed, and synchronization of the data in an elastic search engine and the data in the query database is achieved.
And the query database is used for storing related data.
The embodiment of the application shows a system, a main instance and a standby instance are established at an application platform server (Go end), the running state of the main instance is monitored in real time, if any running abnormal condition occurs in the main instance, the standby instance is directly called as a connecting medium of a query database and a Java end, and synchronization of data in an Elasticissearch search engine and data in the query database is achieved. According to the technical scheme shown in the embodiment of the application, the problem that the query database and the Java end are in a cut-off state due to the fact that the Go end is down is solved through switching among the examples, and the synchronization between the data in the Elasticissearch search engine and the data in the query database is ensured at any time.
Example 2:
a second aspect of the embodiment of the present application shows a data synchronization method, please refer to fig. 3, where the method includes:
s101, establishing an instance set to be in contact with a zookeeper, wherein the instance set comprises: a primary instance, and, a standby instance;
the main example is firstly used as a connecting medium of a query database and a Java end, and the synchronization of data in an elastic search engine and data in the query database is ensured;
the main instance and the standby instance have the same running program and are used for monitoring data change in the query database and sending the update document ID corresponding to the changed data to the Java end, the Java end realizes the synchronization of the data in the Elasticissearch search engine and the data in the query database according to the update document ID.
S102, monitoring the operation condition of the main instance, and calling the standby instance as a connection medium between the query database and the Java end if the main instance does not operate normally, so as to realize the synchronization of the data in the Elasticissearch search engine and the data in the query database.
In an actual operation process, an operation condition, for example, disconnection between the main instance and the query database, or an internal program of the main instance, may occur, and therefore the Java end and the query database cannot achieve data synchronization.
The embodiment of the application shows a method, a main instance and a standby instance are established at a Go end, the operation state of the main instance is monitored in real time, if any operation abnormal condition occurs in the main instance, the standby instance is directly called as a connecting medium of a query database and a Java end, and the synchronization of data in an Elasticissearch search engine and data in the query database is realized. According to the technical scheme shown in the embodiment of the application, the problem that the query database and the Java end are in a cut-off state when the Go end is down is avoided through switching among the instances, and the synchronization between the data in the Elasticissearch search engine and the data in the query database is ensured at any time.
Example 3:
according to the method, the main example and the standby example are used for monitoring and inquiring the transformation of data in the database in real time, the transformation of the data comprises the deletion of a document, the inquiry of the document and the updating of the document, a set of monitoring program is built in the main example, and the monitoring program is used for monitoring and inquiring the transformation of the data in the database in real time. The monitoring program runs according to a certain running process.
Usually, some operation conditions are encountered during the monitoring of the main instance, and at this time, if the standby instance is called, monitoring is started, which undoubtedly prolongs the monitoring time, prolongs the waiting time of the application platform server, reduces the bandwidth of the system, and reduces the utilization rate of resources.
In order to solve the above problem, an embodiment of the present application illustrates a method for calling a standby instance, and specifically, refer to fig. 4:
embodiment 3 has similar steps as embodiment 2, and the only difference is that step S102 in the technical solution shown in embodiment 2 includes the following steps:
s10211, judging whether the main instance establishes contact with the zookeeper through a global monitoring mechanism;
firstly, judging whether the main example is in contact with the zookeeper, wherein the judging step of whether the main example is in contact with the zookeeper includes the step that the zookeeper successfully detects the operation condition of the main example, for example, 10 steps exist in one operation program of the main example, and the zookeeper can successfully monitor the operation of the main program to the step.
The method also comprises the steps that the main instance is successfully connected with the query database, and the main instance is successfully connected with the Java terminal.
S10212 if the connection is not established successfully, calling a standby instance as a connection medium for querying the database and the Java terminal;
and if the connection is not established for the first time, the process is not started successfully, the user restarts, and if the connection is disconnected in the running process, the standby instance is informed to establish contact with the zookeeper. The standby instance will monitor for notifications and the standby instance will self-start after being notified.
If the standby instance and the zookeeper can not successfully establish contact, then the next standby instance is called continuously.
S10213, if the connection is successfully established, establishing a temporary node on the zookeeper according to the operation logic of the main instance;
for example, an operation program of the main instance has 12 steps, zookeeper monitors the operation condition of the main instance in real time, big establishes a temporary node, for example, a json data structure shown in fig. 20, and explains a construction method of the temporary node by taking json shown in fig. 20 as an example of the operation program;
for example, zookeeper monitors that the main instance runs to step B, constructs a temporary node B at zookeeper, monitors that the main instance runs to step C, constructs a temporary node C at zookeeper, and so on until the main instance runs abnormally or the main instance finishes running.
S10214 traversing the temporary nodes in sequence according to the operation logic of the main example; the running logic of the main instance is the traversing logic of the temporary node.
S10215, determining a problem node according to the operation condition of the temporary node;
and if the main instance occurs when the main instance runs to the step C, the operation is abnormal, and the temporary node C is determined to be the problem node.
S10216 calls the standby instance as a connection medium for querying the database and the Java end, and traverses the standby nodes in the standby instance; the standby node is a node corresponding to the problem node in operation logic.
Usually, the standby instance and the main instance have the same operation program, so that when a problem node is determined, the program step corresponding to the problem node can be quickly determined in the standby instance based on the step sequence of the problem node, and then the step program corresponding to the standby instance is sequentially executed according to the operation logic by the step program corresponding to the problem node. Until the standby instance runs abnormally or the standby instance finishes running.
According to the method shown in the embodiment of the application, under the condition that the main instance has the operation condition, the standby nodes in the standby instances are found according to the operation logic, and then the standby nodes are sequentially operated downwards until the standby instances are operated. The technical scheme shown in the embodiment of the application shortens the time for monitoring and querying data by the examples (including the main example and/or the standby example), shortens the waiting time of the application platform server, and improves the bandwidth of the system and the utilization rate of resources.
Example 4:
referring to fig. 5, embodiment 4 has similar steps to embodiment 2, and the only difference is that step S102 in the technical solution shown in embodiment 2 includes the following steps:
s10231, judging whether the main instance establishes contact with the zookeeper through a global monitoring mechanism;
firstly, judging whether the main example is in contact with the zookeeper, wherein the judging step of whether the main example is in contact with the zookeeper includes the step that the zookeeper successfully detects the operation condition of the main example, for example, 10 steps exist in one operation program of the main example, and the zookeeper can successfully monitor the operation of the main program to the step.
The method also comprises the steps that the main instance is successfully connected with the query database, and the main instance is successfully connected with the Java end.
S10232, if the contact is not established successfully, calling a standby example as a connection medium for querying the database and the Java end;
and if the connection is not established for the first time, the process is not successfully started, the user restarts, and if the connection is disconnected in the running process, the standby instance is informed to establish contact with the zookeeper. The standby instance will monitor for notifications and the standby instance will self-start after being notified.
If the standby instance and the zookeeper can not successfully establish contact, then the next standby instance is called continuously.
S10233, if the contact is successfully established, establishing a temporary node on the zookeeper according to the operation logic of the main example;
for example, an operation program of the main instance has 12 steps, zookeeper monitors the operation condition of the main instance in real time, big establishes a temporary node, for example, a json data structure shown in fig. 20, and explains a construction method of the temporary node by taking json shown in fig. 20 as an example of the operation program;
for example, zookeeper monitors that the main instance runs to step B, constructs a temporary node B at zookeeper, monitors that the main instance runs to step C, constructs a temporary node C at zookeeper, and so on until the main instance runs abnormally or the main instance finishes running.
S10234, sequentially traversing the temporary nodes according to the operation logic of the main example;
s10235, receiving a monitoring event of the temporary node;
each temporary node corresponds to a monitoring event, the monitoring event is used for recording whether the temporary node normally operates or not.
S10236, judging whether the event recorded by the monitoring event is a node change event;
if the node runs abnormally, the event is displayed as a node change event,
s10237, if yes, notifying the zookeeper to acquire a monitoring event of the standby example;
the zookeeper is informed to acquire a monitoring event of the standby example, wherein the monitoring event is also used for recording a sequence number corresponding to each standby device application example, for example, the sequence number corresponding to the standby example 1 is-002, the sequence number corresponding to the standby example 2 is-003, and the sequence number corresponding to the standby example 3 is-004;
s10238 calls the standby instance service with the minimum sequence number corresponding to the monitoring event of the standby instance as a connection medium of the query database and the Java end.
The Zookeeper calls the standby instance service with the minimum sequence number corresponding to the monitoring event of the standby instance as a connection medium for inquiring the database and the Java end, namely calls the standby instance 1 as a connection medium for inquiring the database and the Java end.
According to the technical scheme shown in the embodiment of the application, the sequence number of each temporary node is recorded by monitoring events, and when the main node has problems, the standby instance with the minimum sequence number is called as a connection medium for inquiring the database and the Java terminal. According to the technical scheme, the main instance has the problem that the standby instance is designated through the serial number, so that the standby instance can be guaranteed to be orderly called.
Example 5:
embodiment 5 has similar steps to the technical solutions shown in embodiments 2 to 4, and the only difference is that step S102 in the technical solutions shown in embodiments 2 to 4 includes the following steps, specifically, please refer to fig. 6:
s10221 determines an update item, the update item including: updating the column names, updating the data, and updating the column names to the column names corresponding to the updated data;
the updating item is historical data and data inconsistent with the real-time data;
for example, the following steps are carried out:
historical data:
| VIN code | Colour(s) | Age of vehicle |
| 123 | Red colour | 1 year |
| 124 | |
2 years old |
| 125 | Red colour | 3 years old |
Real-time data:
| VIN code | Colour(s) | Age of vehicle |
| 123 | Red colour | 4 years old |
| 124 | |
2 years old |
| 125 | Red colour | 3 years old |
As can be seen, for the vehicle with the VIN code 123, the real-time data and the historical data are different, and at this time, the definition of the update item is: updating column names: vehicle age, update data: and 4 years later.
S10222 determining a target index according to the updated column name and the updated data;
it should be noted that, in the solution shown in the embodiment of the present application, an index is set for each document, and the document ID plays a role of unique identification.
For example: the VIN code 123 is red in color, and the index of the document corresponding to the vehicle age of 4 years is index 1;
VIN code 124 with red color, and index 2 of the document corresponding to the vehicle age of 2 years;
VIN code 125, the color is red, and the index of the document with the age of 3 years is index 3;
wherein, index 1 is the target index.
S10223, traversing the query database based on the target index, and searching for an updated document ID;
it is noted that, in the method shown in the embodiment of the present application, the data in the query database is stored in a json structure; the data are linked with each other through a 'parent-child' relationship, and if one data is changed, the associated data and the child data of the data may be changed.
By way of example:
when the age of the vehicle changes, the document IDs are correspondingly updated and comprise the ID of the document 1, the ID of the document 2 and the ID of the document 3.
In practical application, the updated document ID may be searched out according to the association between the document corresponding to the target index and the rest of documents, or the updated document ID may be searched out according to the "parent-child" relationship between the documents.
S10224 sends the ID of the updated document to the Java terminal, and the Java terminal searches the updated data based on the ID of the updated document, and updates the index and the data corresponding to the Elasticissearch engine by the updated data.
The method comprises the steps of monitoring a binlog log, wherein the binlog log comprises historical data and real-time data, generating an update item when the real-time data is different from the historical data, determining a target index corresponding to the update item, determining update document IDs of all update documents through the target index, packaging the update document IDs into messages, sending the messages to a Java end, receiving the messages by the Java end, analyzing the messages, searching out latest data from a query database according to the update document IDs, updating indexes corresponding to the latest data updating electronic search and documents according to the technical scheme shown in the embodiment of the application.
Example 6:
in order to further ensure synchronization between data in the Elasticsearch search engine and data in the query database, an embodiment of the present application shows a generation method for updating a document ID, and specifically, please refer to fig. 7;
the technical solution shown in embodiment 6 has similar steps with the technical solution shown in embodiment 5, and the only difference is that step S10223 in the technical solution shown in embodiment 5 includes the following steps:
s1022311, determining a sub-target index of the target index according to the data service logic in the query database;
the data in the query database exists in the form of statements.
The method disclosed by the embodiment of the self application, the data in the query database is stored in a json structure; the data are linked with each other through a 'parent-child' relationship, and if one data is changed, the associated data and the child data of the data may be changed.
For example: the documents corresponding to the target index Audi are: document 1, document 2, document 3, and, document 4;
red, documents corresponding to Audi are document 1-1 and document 2-1; the document 1-1 may be the same document as the document 1, or may be a document in which some contents are added on the basis of the document 1; the document 2-1 may be the same document as the document 2, or may be a document added with some content on the basis of the document 2;
at this time, according to the data service logic in the query database, it is found that the index "red audi" is a sub-target index of the target index "audi".
S1022312, according to the target index, determining a target index, and according to the sub-target index, determining sub-target node documents;
then, document 1, document 2, document 3, and document 4 are target documents;
the corresponding document 1-1, document 2-1, is a sub-target document.
S1022313 generates an update document ID according to the target node document and the sub-target node documents.
According to the technical scheme shown in the embodiment of the application, the updated document and the sub-document corresponding to the updated document are determined by inquiring the data service logic in the database, so that all the changed documents are determined, and then the IDs of all the changed documents are determined.
Example 7:
in order to further ensure synchronization between data in the Elasticsearch search engine and data in the query database, an embodiment of the present application shows a generation method for updating a document ID, and specifically, please refer to fig. 8; the technical solution shown in embodiment 7 has similar steps to the technical solution shown in embodiment 5, and the only difference is that step S10223 in the technical solution shown in embodiment 5 includes the following steps:
s1022321, determining a local document ID according to the target index;
in the process of constructing the index, the index and the ID of the document have some corresponding relations:
for example, for one index: red, audi. Wherein, the ID corresponding to red is 235, and the ID corresponding to Audi is 456;
red, local ID for audi is: 8230 \\ 8230;, 235 \ 8230; \ 8230456; \ 8230; or 8230, 8230456 8230, 8230235 8230, and 8230;
s1022322 determines the document ID including the local document ID as the update document ID.
The method shown in the embodiment of the application determines all updated document IDs according to the local document IDs, searches the IDs of all changed documents, avoids traversing and querying the database for multiple times, shortens the traversing time, and further ensures the real-time performance of the Elasticissearch search engine data shown in the embodiment of the application.
Example 8:
in order to further shorten the rate of acquiring the update item and ensure the real-time performance of the method shown in the embodiment of the present application, the embodiment of the present application shows a method for comparing historical data with real-time data, and specifically, please refer to fig. 9;
embodiment 8 has similar steps with the solution shown in embodiment 5, the only difference is that step S10221 in the solution shown in embodiment 5 comprises the following steps:
s102211 acquires monitoring data, the monitoring data including: historical data, and, real-time data;
the detection data shown in the embodiment of the application is a binlog log of a row mode.
The binlog log of the row schema typically has one value for each column and one or more values for each document.
S102212 compares the historical data with the real-time data input gorouine to determine an update item.
The gorouine can regard it as a local scheduler, a plurality of gorouines simultaneously wait for comparing historical data with the real-time data, and historical data 1 is compared with the real-time data 1 through the gorouine-1; when the historical data 2 needs to be compared with the real-time data 2, the gorouine-2 can be directly used for comparison, and so on, in the comparison process, the comparison of the previous data does not need to be waited for, so that the data processing efficiency is improved, the acquisition rate of the updated item is shortened, and the real-time performance of the Elasticissearch engine data shown in the embodiment of the application is ensured.
Example 9:
in order to reduce the data processing amount of the application platform service, the embodiment of the present application shows a method for filtering an update item, specifically, please refer to fig. 10;
embodiment 10 has similar steps with the solution shown in embodiment 9, and the only difference is that step S102212 in the solution shown in embodiment 9 includes the following steps:
s1022121, comparing the historical data with the real-time data input gorouine, and determining an item to be updated according to a comparison result;
not all the items to be updated affect the index and data corresponding to the Elasticsearch engine, but the real-time data and the historical data corresponding to the updated items are acquired simultaneously in the data acquisition process.
Before the process of continuing downward processing of the item to be updated, the embodiment of the application shows a method, wherein the item to be updated is judged in advance, the item to be updated is judged, and whether the updating operation of an elastic search is triggered is judged;
s1022122 judges whether the item to be updated triggers the updating operation of the Elasticissearch;
specifically, the determination method is as follows:
an Elasticissearch search engine comprising an Elasticissearch index, and, a document; these indexes and/or documents are the update items that do not trigger the update operation of the Elasticsearch.
The method shown in the embodiment of the application comprises the steps of firstly determining an index to be updated corresponding to an item to be updated, then determining a related index and a sub-index of the index to be updated, judging whether the related index of the index to be updated and the sub-index are Elasticissearch indexes, and if the related index is the Elasticissearch index, determining that the item to be updated triggers the updating operation of the Elasticissearch.
S1022123, if triggered, determining the item to be updated as an updated item;
if the associated index of the index to be updated and the sub-index are not the Elasticissearch index, the item to be updated is not considered to trigger the updating operation of the Elasticissearch.
S1022124 deletes the item to be updated if not triggered.
The deleting o update item comprises: ignoring, processing logic ends directly.
Therefore, the technical side shown in the embodiment of the application filters some useless items to be updated in advance, and directly deletes the useless items to be updated, so that the data processing amount of an application platform is reduced, the bandwidth of a system and the utilization rate of resources are improved, and the real-time performance of the Elasticissearch search engine data shown in the embodiment of the application is ensured.
Example 10:
a third aspect of the embodiments of the present application shows a data synchronization apparatus, please refer to fig. 11, where the apparatus includes:
an establishing unit 21, configured to establish a contact between an instance set and a zookeeper, where the instance set includes: a primary instance, and, a standby instance;
and the monitoring unit 22 is used for monitoring the operation condition of the main instance, and calling the standby instance as a connection medium between the query database and the Java end if the main instance does not operate normally, so as to realize the synchronization of the data in the Elasticissearch search engine and the data in the query database.
Example 11:
referring to fig. 12, in the technical solution shown in embodiment 10, the listening unit 22 includes:
a first main instance determining unit 2211, configured to determine, through a global monitoring mechanism, whether the main instance establishes a contact with a zookeeper;
a first standby instance calling unit 2212, configured to, if the association is not successfully established, call the standby instance as a connection medium between the query database and the Java side;
a first temporary node establishing unit 2213, configured to establish a temporary node on the zookeeper according to the operation logic of the main instance if the connection is successfully established;
a first temporary node traversal unit 2214 configured to sequentially traverse temporary nodes according to the operation logic of the main instance;
a problem node determining unit 2215, configured to determine a problem node according to the operating condition of the temporary node;
a first standby instance calling unit 2216, configured to call a standby instance as a connection medium between the query database and the Java side, and traverse a standby node in the standby instance, where the standby node is a node logically corresponding to the problem node in operation.
Example 12:
please refer to fig. 13;
the listening unit 22 includes:
a second main instance determining unit 2231, configured to determine, through a global monitoring mechanism, whether the main instance establishes a contact with the zookeeper;
a second standby instance invoking unit 2232, configured to, if the association is not successfully established, invoke the standby instance as a connection medium between the query database and the Java side;
a second temporary node establishing unit 2233, for establishing, if the contact is successfully established, according to the running logic of the master instance,
establishing a temporary node on a zookeeper;
a second temporary node traversal unit 2234, sequentially traversing the temporary nodes according to the operation logic of the master instance;
a snoop event receiving unit 2235, configured to receive a snoop event of the temporary node;
a snoop event determining unit 2236, configured to determine whether an event recorded by the snoop event is a node change event;
a notifying unit 2237, configured to notify the zookeeper to acquire a listening event of the standby instance if the monitored event is positive;
a second standby instance invoking unit 2238, configured to invoke a standby instance service with a minimum sequence number corresponding to the listening event of the standby instance as a connection medium between the query database and the Java side. Example 13:
referring to fig. 14, the listening unit 22 includes:
an update item determination unit 2221 configured to determine an update item, which includes: updating the column names, updating the data, and updating the column names to be the column names corresponding to the updated data;
an index determining unit 2222 configured to determine a target index according to the updated column name and the update data;
a searching unit 2223, configured to search for an updated document ID by traversing the query database based on the target index;
an updating unit 2224, configured to send the update document ID to the Java side, where the Java side searches for update data based on the update document ID, and updates the index and data corresponding to the Elasticsearch engine with the update data.
In embodiment 14, referring to fig. 15, the search unit 2223 includes:
a sub-goal index determining unit 222311, configured to determine a sub-goal index of the goal index according to the data service logic in the query database;
a target document determining unit 222312, configured to determine a target index according to the target index, and determine sub-target node documents according to the sub-target index;
an update document ID generating unit 222313 configured to generate an update document ID according to the target node document and the sub-target node documents.
Example 15:
referring to fig. 16, the search unit 2223 includes:
a local document ID determining unit 222321 configured to determine a local document ID according to the target index;
an updated document ID determination unit 222322 operable to determine the document ID containing the local document ID as the updated document ID.
Example 16:
referring to fig. 17, the update item determination unit 2221 includes:
an obtaining unit 22211, configured to obtain monitoring data, where the monitoring data includes: historical data, and, real-time data;
a first update item determining unit 22212, configured to compare the historical data with the real-time data input gorouine, and determine an update item.
Example 17:
referring to fig. 18, the first update item determination unit 22212 includes:
a comparing unit 222121, configured to compare the historical data with the real-time data input gorouine, and determine an item to be updated according to a comparison result;
a determining unit 222122, configured to determine whether the item to be updated triggers an updating operation of an Elasticsearch;
a second update item determining unit 22213, configured to determine that the item to be updated is an update item if the trigger occurs;
a delete unit 222124, configured to delete the item to be updated if not triggered.
Example 18:
in a fourth aspect of the embodiments of the present application, a server is shown, and please refer to fig. 19 including:
one or more processors 41;
a memory 42 for storing one or more programs;
when executed by the one or more processors 41, cause the one or more processors 41 to implement the methods of the embodiments of the present application.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (13)
1. A method for synchronizing data, the method comprising:
establishing an instance set to contact a zookeeper, the instance set comprising: a primary instance, and, a standby instance;
monitoring the running state of the main instance, and calling the standby instance as a connecting medium between the query database and the Java end if the main instance runs abnormally to realize the synchronization of the data in the Elasticissearch search engine and the data in the query database;
the step of monitoring the operation condition of the main instance, and calling the standby instance as a connection medium for inquiring the database and the Java end if the main instance does not operate normally comprises the following steps:
judging whether the main instance establishes contact with the zookeeper through a global monitoring mechanism;
if the connection is not established successfully, calling the standby instance as a connection medium for inquiring the database and the Java end;
if the contact is successfully established, establishing a temporary node on the zookeeper according to the operation logic of the main instance;
the step of establishing the temporary node on the zookeeper according to the operation logic of the main instance comprises the following steps: monitoring that the main instance runs to a first step by the zookeeper, constructing a first temporary node on the zookeeper, monitoring that the main instance runs to a second step by the zookeeper, constructing a second temporary node on the zookeeper, and so on until the main instance runs abnormally or the main instance finishes running;
sequentially traversing the temporary nodes according to the operation logic of the main example; the operation logic of the main example is the traversal logic of the temporary node;
receiving a monitoring event of a temporary node;
judging whether the event recorded by the monitoring event is a node change event or not;
if yes, informing the zookeeper to acquire the monitoring event of the standby instance;
and calling the backup instance service with the minimum sequence number corresponding to the monitoring event of the backup instance as a connection medium between the query database and the Java terminal, and traversing the backup nodes in the backup instance, wherein the backup nodes are nodes corresponding to the problem nodes in the operation logic.
2. The method of claim 1, wherein the step of synchronizing data in the Elasticsearch search engine with data in the query database comprises:
determining an update item, the update item comprising: updating the column names, updating the data, and updating the column names to be the column names corresponding to the updated data;
determining a target index according to the updated column name and the updated data;
traversing the query database based on the target index, and searching for an updated document ID;
and sending the updated document ID to the Java end, searching updated data by the Java end based on the updated document ID, and updating the index and the data corresponding to the Elasticissearch search engine by using the updated data.
3. The method of claim 2, wherein the step of searching the updated document ID based on the target index, traversing the query database, comprises:
determining sub-target indexes of the target index according to data service logic in the query database;
determining a target index according to the target index, and determining sub-target node documents according to the sub-target index;
and generating an update document ID according to the target node document and the sub-target node documents.
4. The method of claim 2, wherein the step of searching the updated document ID based on the target index, traversing the query database, comprises:
determining a local document ID according to the target index;
the document ID containing the local document ID is determined as the updated document ID.
5. The method of claim 2, wherein the step of determining an update item comprises:
acquiring monitoring data, wherein the monitoring data comprises: historical data, and, real-time data;
and comparing the historical data with the real-time data input gorouine to determine an updating item.
6. The method of claim 5, wherein comparing the historical data to the real-time data input gorouine determines updated terms comprising:
comparing the historical data with the real-time data input gorouine, and determining an item to be updated according to a comparison result;
judging whether the item to be updated triggers the updating operation of the Elasticissearch;
if the trigger is carried out, determining the item to be updated as an updated item;
and if not, deleting the item to be updated.
7. A data synchronization apparatus, the apparatus comprising:
the establishing unit is used for establishing an instance set to be in contact with the zookeeper, and the instance set comprises: a primary instance, and, a standby instance;
the monitoring unit is used for monitoring the running condition of the main example, and calling the standby example as a connecting medium between the query database and the Java terminal if the main example runs abnormally so as to realize the synchronization of the data in the Elasticissearch search engine and the data in the query database;
the listening unit includes:
the first main example judging unit is used for judging whether the main example establishes contact with the zookeeper through a global monitoring mechanism;
the first standby instance calling unit is used for calling the standby instance as a connecting medium for inquiring the database and the Java terminal if the contact is not successfully established;
the first temporary node establishing unit is used for establishing a temporary node on the zookeeper according to the operation logic of the main example if the contact is successfully established;
the step of establishing the temporary node on the zookeeper according to the operation logic of the main instance comprises the following steps: monitoring that the main instance runs to a first step by the zookeeper, constructing a first temporary node on the zookeeper, monitoring that the main instance runs to a second step by the zookeeper, constructing a second temporary node on the zookeeper, and so on until the main instance runs abnormally or the main instance finishes running;
the first temporary node traversing unit is used for sequentially traversing the temporary nodes according to the operation logic of the main example; the operation logic of the main example is the traversal logic of the temporary node;
a monitoring event receiving unit, configured to receive a monitoring event of a temporary node;
a monitoring event judging unit, configured to judge whether an event recorded by the monitoring event is a node change event;
a notification unit, configured to notify the zookeeper to acquire a monitoring event of the standby instance if the zookeeper is the standby instance;
and the first standby instance calling unit is used for calling the standby instance service with the minimum sequence number corresponding to the monitoring event of the standby instance as a connection medium between the query database and the Java end, and traversing the standby nodes in the standby instance, wherein the standby nodes are nodes which are logically corresponding to the problem nodes in operation.
8. The apparatus of claim 7, wherein the listening unit comprises:
an update item determination unit configured to determine an update item, the update item including: updating the column names, updating the data, and updating the column names to be the column names corresponding to the updated data;
the index determining unit is used for determining a target index according to the updated column name and the updated data;
the searching unit is used for traversing the query database and searching the updated document ID based on the target index;
and the updating unit is used for sending the updated document ID to the Java terminal, the Java terminal searches for updated data based on the updated document ID, and updates the index corresponding to the Elasticissearch search engine and the data by using the updated data.
9. The apparatus of claim 8, wherein the search unit comprises:
a sub-target index determining unit, configured to determine a sub-target index of the target index according to the data service logic in the query database;
the target document determining unit is used for determining a target index according to the target index and determining sub-target node documents according to the sub-target index;
and the update document ID generating unit is used for generating an update document ID according to the target node document and the sub-target node documents.
10. The apparatus of claim 8, wherein the search unit comprises:
a local document ID determining unit, configured to determine a local document ID according to the target index;
an update document ID determination unit for determining a document ID containing the local document ID as an update document ID.
11. The apparatus of claim 8, wherein the update item determination unit comprises:
an obtaining unit configured to obtain monitoring data, where the monitoring data includes: historical data, and, real-time data;
and the first updating item determining unit is used for comparing the historical data with the real-time data input gorouine to determine an updating item.
12. The apparatus of claim 11, wherein the first update item determination unit comprises:
the comparison unit is used for comparing the historical data with the real-time data input gorouine and determining an item to be updated according to a comparison result;
the judging unit is used for judging whether the item to be updated triggers the updating operation of the Elasticissearch;
a second update item determination unit, configured to determine, if triggered, that the item to be updated is an update item;
and the deleting unit is used for deleting the item to be updated if the item is not triggered.
13. A data synchronization system, the system comprising: the system comprises an application platform server and a query database connected with the application platform server, wherein the query database is arranged in the application platform server or is arranged independently, and the application platform server is connected with a Java end through the Internet;
the Java terminal is used for searching updated data based on the updated document ID, and updating the index corresponding to the Elasticissearch search engine and the data by using the updated data;
the application platform server for implementing the method of any one of claims 1-6;
and the query database is used for storing related data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810532857.6A CN110543512B (en) | 2018-05-29 | 2018-05-29 | Information synchronization method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810532857.6A CN110543512B (en) | 2018-05-29 | 2018-05-29 | Information synchronization method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110543512A CN110543512A (en) | 2019-12-06 |
| CN110543512B true CN110543512B (en) | 2022-10-04 |
Family
ID=68701581
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810532857.6A Active CN110543512B (en) | 2018-05-29 | 2018-05-29 | Information synchronization method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110543512B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112437146B (en) * | 2020-11-18 | 2022-10-14 | 青岛海尔科技有限公司 | Equipment state synchronization method, device and system |
| CN112100249B (en) * | 2020-11-20 | 2021-03-30 | 北京药联健康科技有限公司 | Data mining method and system |
| CN114564485A (en) * | 2022-04-28 | 2022-05-31 | 深圳竹云科技股份有限公司 | User data processing method based on Elastic Search |
| CN115495634B (en) * | 2022-11-17 | 2023-04-07 | 北京滴普科技有限公司 | Method and system for capturing change data based on Elasticissearch plug-in |
| CN116467284A (en) * | 2023-04-24 | 2023-07-21 | 上海济语智能科技有限公司 | Non-inductive dynamic switching system based on multiple data sources |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1836211A (en) * | 2003-08-14 | 2006-09-20 | 甲骨文国际公司 | Automatic and dynamic provisioning of databases |
Family Cites Families (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10467686B2 (en) * | 2007-02-09 | 2019-11-05 | Xcira, Inc. | Integrated auctioning environment platform |
| CN101977124B (en) * | 2010-11-05 | 2013-09-04 | 山东中创软件工程股份有限公司 | Service clustering method and system based on ZooKeeper technology |
| US10121169B2 (en) * | 2015-09-16 | 2018-11-06 | Amobee, Inc. | Table level distributed database system for big data storage and query |
| CN105468727A (en) * | 2015-11-20 | 2016-04-06 | 国家电网公司 | Zookeeper based method for realizing MySQL strong-consistency copy |
| CN106936862B (en) * | 2015-12-29 | 2019-11-01 | 优信拍（北京）信息科技有限公司 | Method of data synchronization, apparatus and system |
| CN106021070A (en) * | 2016-04-29 | 2016-10-12 | 乐视控股（北京）有限公司 | Method and device for server cluster monitoring |
| CN107783975B (en) * | 2016-08-24 | 2021-02-26 | 北京京东尚科信息技术有限公司 | Method and device for synchronous processing of distributed databases |
| CN106375342A (en) * | 2016-10-21 | 2017-02-01 | 用友网络科技股份有限公司 | Zookeeper-technology-based system cluster method and system |
| CN106850260A (en) * | 2016-12-23 | 2017-06-13 | 曙光云计算技术有限公司 | A kind of dispositions method and device of virtual resources management platform |
| CN106843930A (en) * | 2016-12-23 | 2017-06-13 | 江苏途致信息科技有限公司 | Streaming dynamic configuration more new architecture and method based on zookeeper |
| CN106980699B (en) * | 2017-04-14 | 2020-02-14 | 中国科学院深圳先进技术研究院 | Data processing platform and system |
-
2018
- 2018-05-29 CN CN201810532857.6A patent/CN110543512B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1836211A (en) * | 2003-08-14 | 2006-09-20 | 甲骨文国际公司 | Automatic and dynamic provisioning of databases |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110543512A (en) | 2019-12-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110543512B (en) | Information synchronization method, device and system | |
| US11768848B1 (en) | Retrieving, modifying, and depositing shared search configuration into a shared data store | |
| US11768811B1 (en) | Managing user data in a multitenant deployment | |
| US11586692B2 (en) | Streaming data processing | |
| US11748394B1 (en) | Using indexers from multiple systems | |
| CN110609844B (en) | Data updating method, device and system | |
| CN110609865B (en) | Information synchronization method, device and system | |
| CN111078504A (en) | Distributed call chain tracking method and device, computer equipment and storage medium | |
| CA2896865C (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| CN112445626B (en) | Data processing method and device based on message middleware | |
| CN110362455B (en) | Data processing method and data processing device | |
| EP2984551A1 (en) | Creating global aggregated namespaces for storage management | |
| CN111859132A (en) | Data processing method and device, intelligent equipment and storage medium | |
| US11841834B2 (en) | Method and apparatus for efficient synchronization of search heads in a cluster using digests | |
| CN112506870B (en) | Data warehouse increment updating method and device and computer equipment | |
| CN110858194A (en) | Method and device for expanding database | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| US20220044144A1 (en) | Real time model cascades and derived feature hierarchy | |
| CN112948498A (en) | Method and device for generating global identification of distributed system | |
| CN114860782B (en) | Data query method, device, equipment and medium | |
| CN111984723A (en) | Data synchronization method and device and terminal equipment | |
| CN109739883B (en) | Method and device for improving data query performance and electronic equipment | |
| CN114138778A (en) | Data processing system, method, device, equipment and computer readable medium | |
| CN113779017A (en) | Method and apparatus for data asset management | |
| CN113535768A (en) | Production monitoring method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231128 Address after: 230012 In the factory building of Anhui Guogou Energy Co., Ltd., 100 meters east of the intersection of Guanjing Road and Luban Road in Xinzhan District, Hefei City, Anhui Province Patentee after: Hefei Youquan Information Technology Co.,Ltd. Address before: 100102 room 323701, building 5, yard 1, Futong East Street, Chaoyang District, Beijing Patentee before: YOUXINPAI (BEIJING) INFORMATION TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |