CN110928885B - Method and device for updating data of Mysql database to Es database - Google Patents
Method and device for updating data of Mysql database to Es database Download PDFInfo
- Publication number
- CN110928885B CN110928885B CN201811028161.6A CN201811028161A CN110928885B CN 110928885 B CN110928885 B CN 110928885B CN 201811028161 A CN201811028161 A CN 201811028161A CN 110928885 B CN110928885 B CN 110928885B
- Authority
- CN
- China
- Prior art keywords
- data
- database
- message
- type
- topic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a device for updating data of a Mysql database to an Es database, wherein the method comprises the following steps: and creating a local message list based on the Mysql database, and writing a data update record corresponding to the data update event into the local message list when the Mysql database generates the data update event. And then, the data update record in the local message list is read in real time by utilizing a first type of resident process started in advance, and the read data update record is issued to the Topic corresponding to the message queue. If a pre-started second type resident process is adopted to monitor the Topic message focused by the Es database in the message queue, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message. Therefore, the embodiment of the invention writes the data update record in the Mysql database into the message list positioned in the Mysql database, and then writes the complete update data into the Es database, so that the data finally updated to the Es database is the high available complete data.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a device for updating data of a Mysql database to an Es database.
Background
In general, the list of the item/page requires much information, and if the information is directly read from the Mysql database, many data tables need to be searched together, so that the searching speed of the information is slower, and therefore, indexing the data in the database into the Es (electric Search server) database can be considered.
However, in the prior art, for example, for update action of Mysql database, updated data is usually written directly into a message queue, and a resident process will consume the message immediately once it monitors the message written into the message queue, so that it is easy for updated data to be written into the message queue incompletely, and the resident process will update incomplete or unreliable or unusable data into Es database, thereby affecting accuracy of data update. If the complete updated data is to be written into Es database, compensation mechanism is needed to compensate, so that the real-time performance of data update is poor.
Disclosure of Invention
In view of the foregoing, the present invention has been made to provide a method and apparatus for updating data of a Mysql database to an Es database that overcomes or at least partially solves the foregoing problems.
According to an aspect of the present invention, there is provided a method for updating data of a Mysql database to an Es database, comprising:
creating a local message list based on the Mysql database;
when the Mysql database generates a data update event, writing a data update record corresponding to the data update event into the local message list;
reading data update records in the local message list in real time by using a first type of resident process started in advance, and issuing the read data update records to Topic corresponding to the message queue;
if a pre-started second type resident process is adopted to monitor the Topic message focused by the Es database in the message queue, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message.
Optionally, when the Mysql database generates a data update event, writing a data update record corresponding to the data update event into the local message list, including:
adding a hook function into a data access object Dao of the Mysql database;
when a Mysql database generates a data update event, writing a data update record corresponding to the data update event into the hook function;
and writing the data update record into the local message list by using the hook function.
Optionally, the message queue includes a message queue implemented based on a storage system Redis or a distributed publish-subscribe message system kafka.
Optionally, the method further comprises:
if a plurality of Es databases pay attention to the corresponding Topic messages, respectively starting a second type resident process for each Es database;
and monitoring the Topic messages focused by any Es database in the message queue by a second resident process corresponding to any Es database.
Optionally, the data update event includes at least one of: adding data, deleting data, modifying data.
Optionally, updating the Es database based on the acquired Topic message includes:
analyzing the type of the acquired Topic message;
triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
Optionally, the type of the Topic message is associated with the data update event,
if the data update event is adding data, the type of the Topic message is an adding type;
if the data update event is deleting data, the type of the Topic message is a deleting type;
and if the data update event is modification data, the type of the Topic message is modification type.
Optionally, triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event, including:
if the type of the Topic message is an adding type, triggering a data adding event, and adding data corresponding to the Topic message in the Es database;
if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting the data corresponding to the Topic message in the Es database;
and if the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
Optionally, the method further comprises:
detecting whether data update exists in the Mysql database according to a preset time interval;
if yes, the updated data in the preset time interval is compensated to the Es database.
Optionally, the method further comprises:
detecting whether an editing operation is received on a page where a server attribute list is located, wherein the server list comprises attribute information related to a server, and the attribute information related to the server is stored in the Mysql database;
if yes, calling a real-time updating interface preset on the Es database to update the data corresponding to the editing operation in the Es database.
Optionally, the attribute information related to the server includes at least one of:
server IP address, server configuration parameters, server brand, server model.
According to another aspect of the present invention, there is also provided an apparatus for updating data of a Mysql database to an Es database, including:
the creation module is suitable for creating a local message list based on the Mysql database;
the writing module is suitable for writing a data update record corresponding to a data update event into the local message list when the Mysql database generates the data update event;
the reading module is suitable for reading the data update record in the local message list in real time by utilizing a first type of resident process started in advance, and issuing the read data update record to the Topic corresponding to the message queue;
and the updating module is suitable for acquiring the monitored Topic message if the Topic message focused by the Es database is monitored in the message queue by adopting the second type resident process started in advance, and updating the Es database based on the acquired Topic message.
Optionally, the writing module is further adapted to:
adding a hook function into a data access object Dao of the Mysql database;
when a Mysql database generates a data update event, writing a data update record corresponding to the data update event into the hook function;
and writing the data update record into the local message list by using the hook function.
Optionally, the message queue includes a message queue implemented based on a storage system Redis or a distributed publish-subscribe message system kafka.
Optionally, the apparatus further comprises:
the starting module is suitable for starting the second type resident process for each Es database if a plurality of Es databases pay attention to the corresponding Topic messages;
and the monitoring module is suitable for monitoring the Topic messages focused by any Es database in the message queue by a second type resident process corresponding to the Es database.
Optionally, the data update event includes at least one of: adding data, deleting data, modifying data.
Optionally, the update module is further adapted to:
analyzing the type of the acquired Topic message;
triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
Optionally, the type of the Topic message is associated with the data update event,
if the data update event is adding data, the type of the Topic message is an adding type;
if the data update event is deleting data, the type of the Topic message is a deleting type;
and if the data update event is modification data, the type of the Topic message is modification type.
Optionally, the update module is further adapted to:
if the type of the Topic message is an adding type, triggering a data adding event, and adding data corresponding to the Topic message in the Es database;
if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting the data corresponding to the Topic message in the Es database;
and if the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
Optionally, the apparatus further comprises:
the detection module is suitable for detecting whether data update exists in the Mysql database according to a preset time interval;
and the compensation module is suitable for compensating the updated data in the preset time interval into the Es database if the detection module detects that the data update exists in the Mysql database according to the preset time interval.
Optionally, the detection module is further adapted to detect whether an editing operation is received on a page where a server attribute list is located, where the server list includes attribute information related to the server, and the attribute information related to the server is stored in the Mysql database;
and the updating module is further adapted to, if the detecting module detects that the page where the server attribute list is located receives the editing operation, call a real-time updating interface preset on the Es database to update the data corresponding to the editing operation in the Es database.
Optionally, the attribute information related to the server includes at least one of:
server IP address, server configuration parameters, server brand, server model.
According to a further aspect of the invention there is also provided a computer storage medium storing computer program code which, when run on a computing device, causes the computing device to perform a method of updating data of a Mysql database as described in any of the embodiments above to an Es database.
According to yet another aspect of the present invention, there is also provided a computing device including: a processor; a memory storing computer program code; the computer program code, when executed by the processor, causes the computing device to perform the method of updating data of a Mysql database to an Es database as described in any of the embodiments above.
In the embodiment of the invention, firstly, a local message list is created based on a Mysql database, and when the Mysql database generates a data update event, a data update record corresponding to the data update event is written into the local message list. And then, the data update record in the local message list is read in real time by utilizing a first type of resident process started in advance, and the read data update record is issued to the Topic corresponding to the message queue. If a pre-started second type resident process is adopted to monitor the Topic message focused by the Es database in the message queue, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message. Therefore, the embodiment of the invention does not directly write the data update record in the Mysql database into the message queue, but writes the data update record in the Mysql database into the newly created message list local to the Mysql database, so that after the complete Mysql database update data is recorded in the local message list, the complete update data is written into the Es database, and the data finally updated to the Es database is high available complete data. The implementation process of updating the data in the Mysql database to the Es database is simple and efficient, and the real-time property and accuracy of the data updated to the Es database are effectively ensured. Further, when a user queries the data list, the user does not need to directly search a plurality of data tables from the Mysql database in a combined way, but can directly search the required data in the Es database, so that the data query efficiency is improved, and the data query time is reduced.
The foregoing description is only an overview of the present invention, and is intended to be implemented in accordance with the teachings of the present invention in order that the same may be more clearly understood and to make the same and other objects, features and advantages of the present invention more readily apparent.
The above, as well as additional objectives, advantages, and features of the present invention will become apparent to those skilled in the art from the following detailed description of a specific embodiment of the present invention when read in conjunction with the accompanying drawings.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to designate like parts throughout the figures. In the drawings:
FIG. 1 shows a flow diagram of a method of updating data of a Mysql database to an Es database according to one embodiment of the invention;
FIG. 2 shows a schematic diagram of the publishing of data of a Mysql database into a message queue according to one embodiment of the invention;
FIG. 3 is a diagram illustrating the updating of a message in a message queue to an Es database according to one embodiment of the invention;
FIG. 4 shows a schematic diagram of the structure of an apparatus for updating data of a Mysql database to an Es database according to one embodiment of the invention; and
fig. 5 shows a schematic structural diagram of an apparatus for updating data of Mysql database to Es database according to another embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In order to solve the technical problems, the embodiment of the invention provides a method for updating data of a Mysql database to an Es database. FIG. 1 shows a flow diagram of a method for updating data of a Mysql database to an Es database according to one embodiment of the invention. Referring to fig. 1, the method includes at least steps S102 to S108.
Step S102, a local message list is created based on the Mysql database.
Step S104, when the Mysql database generates a data update event, writing a data update record corresponding to the data update event into the local message list.
Step S106, the data update record in the local message list is read in real time by utilizing the first type resident process started in advance, and the read data update record is issued to the Topic corresponding to the message queue.
In this step, since the Mysql database includes a plurality of data tables, and each data table corresponds to one Topic, when the data in the data table in the Mysql database is updated, the first resident process issues the read updated data to the Topic corresponding to the message queue. For example, the Mysql database includes a device table, and then the name of the topic corresponding to the device table may be cmdb.
Step S108, if the Topic message focused by the Es database is monitored in the message queue by adopting the second type resident process started in advance, the monitored Topic message is obtained, and the Es database is updated based on the obtained Topic message.
In the embodiment of the invention, the data update record in the Mysql database is not directly written into the message queue, but is written into the newly created message list which is positioned in the local of the Mysql database, so that after the complete Mysql database update data is recorded in the local message list, the complete update data is written into the Es database, and the data which is finally updated into the Es database is high-availability complete data. The implementation process of updating the data in the Mysql database to the Es database is simple and efficient, and the real-time property and accuracy of the data updated to the Es database are effectively ensured. Through actual operation, the scheme of the invention can update the update data generated by the Mysql database to the Es database within 30ms, thereby greatly improving the update speed of the data in the Es database. Further, when a user queries the data list, the user does not need to directly search a plurality of data tables from the Mysql database in a combined way, but can directly search the required data in the Es database, so that the data query efficiency is improved, and the data query time is reduced.
Referring to step S104 above, in an embodiment of the present invention, when writing the data update record corresponding to the data update event into the local message list, the data update record may also be written into the local message list through the hook function. Specifically, first, a hook function is added to the data access object Dao (Data Access Object ) of the Mysql database. Then, when the Mysql database generates a data update event, writing a data update record corresponding to the data update event into the hook function, and further writing the data update record into the local message list public_list by using the hook function.
Referring to step S106 above, in one embodiment of the present invention, the message queue may comprise a message queue implemented based on a key-value store Redis or a distributed publish-subscribe message system kafka. For example, if the message queue is implemented based on the storage system dis, it employs a pubsub (Publish/Subscribe) function of the storage system dis. Of course, the message queue may be implemented based on other systems, and the type of the message queue in the embodiment of the present invention is not specifically limited.
Referring to step S108, in an embodiment of the present invention, if there are a plurality of Es databases focusing on the corresponding Topic message, the second type resident processes may be started for each Es database, and then the second type resident processes corresponding to any one Es database may monitor the Topic message focused on the Es database in the message queue. When any second resident process monitors the Topic message concerned by the corresponding Es database in the message queue, the corresponding Es database can be updated based on the acquired Topic message.
In an embodiment of the present invention, the data update event may include at least one of adding data, deleting data, and modifying data. In step S108, when updating the Es database based on the acquired Topic message, a corresponding parsing operation may be performed on the acquired Topic message, that is, the type of the acquired Topic message is analyzed, and then a corresponding data processing event is triggered according to the type of the analyzed Topic message, so that data corresponding to the Topic message in the Es database is processed based on the data processing event, so as to update the Es database.
In this embodiment, the type of the Topic message is related to the data update event, for example, if the data update event is adding data, the type of the Topic message is an adding type. If the data update event is deleting data, the type of the Topic message is a delete type. If the data update event is modification data, the type of the Topic message is a modification type.
In an alternative embodiment of the present invention, a corresponding data processing event is triggered according to a type of the Topic message, and when data corresponding to the Topic message in the Es database is processed based on the data processing event, if the type of the Topic message is an addition type, a data adding event is triggered, and data corresponding to the Topic message is added in the Es database. If the type of the Topic message is a deletion type, triggering a data deletion event, and deleting the data corresponding to the Topic message in the Es database. If the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
In order to more clearly embody the scheme of the present invention, a detailed description will be given of the process of updating the data of the Mysql database to the Es database according to the present invention by using a specific embodiment.
First, the process of publishing the data of the Mysql database to the message queue will be described.
Referring to fig. 2, in this embodiment, a hook function is added to a data access object Dao of the Mysql database in advance, when the Mysql database generates a data update event (the data update event shown in fig. 2 is an add data add), a data update record corresponding to the data update event is written into the hook function first, and then the hook function is used to write the data update record into a local message list publish_list. As already described above, the Mysql database contains a plurality of tables, each corresponding to a Topic, so in the embodiment shown in FIG. 2, it is assumed that the data list of the added data in the Mysql database corresponds to Topic-01. And then, a pre-started first type resident process is adopted to read the data update record from the local message list public_list in real time, and the first type resident process publishes the data update record to a message queue realized based on a key-value storage system Redis.
In this embodiment, the Model shown in fig. 2 refers to a service logic layer, through which corresponding service processing logic can be set, and for example, the set service processing logic may include adding data add, modifying data update, and deleting data delete. The Mysql database generates data update events according to corresponding service processing logic preset in the service logic layer.
Then, a procedure is introduced how to update Es database with the data update record in the message queue.
Referring to fig. 3, the data in the Topic corresponding to the message queue includes the data update record in the Mysql database. The foregoing has described the hearing of the Topic messages of interest to the Es database in the message queue through a pre-started second type of resident process. In this embodiment, a second type of resident process is initiated by a Console Console, which may be a terminal server. The second class resident process monitors the Topic message focused by the Es database in a message queue realized based on the storage system Redis through the data access object EDao. For example, the Es database may be concerned with at least one Topic message in Topic-01, topic-02, etc. When the second resident process monitors the Topic message concerned by the Es database, the second resident process acquires the corresponding Topic message, triggers the EDao-event to process the Topic message in the event, and updates the Es database.
In an embodiment of the present invention, in order to ensure that the update data in the Mysql database can be accurately updated to the Es database, the embodiment of the present invention may further provide a compensation mechanism based on the above method, where the compensation mechanism may adjust the refresh data in increments according to the needs of the user. Specifically, whether data update exists in the Mysql database is detected according to a preset time interval, and if yes, the updated data in the preset time interval is compensated to the Es database. In this embodiment, the preset time interval may be any time interval preset by 5 minutes, 6 minutes, 7 minutes, etc., which is not particularly limited in the embodiment of the present invention.
For example, if the preset time interval is 5 minutes, a new resident process may be started first, so as to detect whether updated data exists in the Mysql database once every 5 minutes by using the new resident process, and when the updated data exists in the Mysql database at a certain moment, the updated data within 5 minutes before the current moment is acquired from the Mysql database, and the acquired updated data is compensated to the Es database. The compensation mechanism can be used as a guarantee operation to avoid the situation that the data in the Mysql database cannot be updated timely and accurately due to other reasons (such as that the updated data is not successfully written into a local message list or the data update record is not successfully issued into the corresponding Topic of the message queue, etc.).
In another embodiment of the present invention, it may also be detected whether the page on which the server attribute list is located receives an editing operation (such as an editing operation of a user on the page), where the server list includes attribute information related to the server, and the attribute information related to the server is stored in the Mysql database. If yes, calling a real-time updating interface preset on the Es database to update the data corresponding to the editing operation in the Es database. The checking process of the page is performed in real time, that is, the data information generated by the editing operation is updated to the Es database in real time for the editing operation received by the page, and the process approximates to 0 delay. Of course, the data information generated during the editing operation on the page is also stored in the Mysql database.
In this embodiment, the server-related attribute information may include at least one of a server IP address, a server configuration parameter, a server brand, a server model number, related responsible party information of the server, a manufacturer of the server, and the like.
Based on the same inventive concept, the embodiment of the present invention further provides a device for updating data of the Mysql database to the Es database, and fig. 4 shows a schematic structural diagram of a device for updating data of the Mysql database to the Es database according to one embodiment of the present invention. Referring to fig. 4, an apparatus 400 for updating data of mysql database to Es database includes at least a creation module 410, a writing module 420, a reading module 430, and an updating module 440.
The function of each component or device of the apparatus 400 for updating data of Mysql database to Es database according to the embodiment of the present invention will be described, and the connection relationship between each part will be described:
a creation module 410 adapted to create a local message list based on the Mysql database;
a writing module 420, coupled to the creating module 410, adapted to write a data update record corresponding to a data update event into the local message list when the Mysql database generates the data update event;
the reading module 430 is coupled to the writing module 420, and is adapted to read the data update record in the local message list in real time by using a first type of resident process started in advance, and issue the read data update record to the Topic corresponding to the message queue;
the updating module 440, coupled to the reading module 430, is adapted to acquire the monitored Topic message if the Topic message focused by the Es database is monitored in the message queue by using the second type of resident process started in advance, and update the Es database based on the acquired Topic message.
In an embodiment of the present invention, the writing module 420 is further adapted to add a hook function to the data access object Dao of the Mysql database, and when the Mysql database generates a data update event, write a data update record corresponding to the data update event into the hook function. The data update record is written to the local message list using a hook function.
In one embodiment of the invention, the message queue comprises a message queue implemented based on a storage system Redis or a distributed publish-subscribe message system kafka.
The embodiment of the invention also provides another device for updating the data of the Mysql database to the Es database, and fig. 5 shows a schematic structural diagram of the device for updating the data of the Mysql database to the Es database according to another embodiment of the invention. Referring to fig. 5, the apparatus 400 for updating data of mysql database to Es database may further include a start module 450, a listening module 460, a detection module 470, and a compensation module 480, in addition to the above modules.
The starting module 450 is adapted to start the second type of resident process for each Es database if there are a plurality of Es databases focusing on the corresponding Topic message.
The monitoring module 460 is coupled to the starting module 450 and the updating module 440, respectively, and is adapted to monitor the Topic message focused by any Es database in the message queue by the second type resident process corresponding to the Es database.
The detection module 470, coupled to the creation module 410, is adapted to detect whether there is a data update in the Mysql database at preset time intervals.
And a compensation module 480 coupled to the detection module 470 and adapted to compensate the updated data in the preset time interval to the Es database if the detection module 470 detects that there is an update in the Mysql database at the preset time interval.
In one embodiment of the present invention, the data update event includes at least one of adding data, deleting data, and modifying data.
In an embodiment of the invention, the update module 440 is further adapted to analyze the type of the acquired Topic message. Triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
In an embodiment of the present invention, the type of the Topic message is related to a data update event, and if the data update event is adding data, the type of the Topic message is an adding type. If the data update event is deleting data, the type of the Topic message is a delete type. If the data update event is modification data, the type of the Topic message is a modification type.
In an embodiment of the present invention, the update module 440 is further adapted to trigger a data addition event if the type of the Topic message is an addition type, and add data corresponding to the Topic message in the Es database. If the type of the Topic message is a deletion type, triggering a data deletion event, and deleting the data corresponding to the Topic message in the Es database. If the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
In an embodiment of the present invention, the detection module 470 is further adapted to detect whether the page on which the server attribute list is located receives an editing operation, where the server list includes attribute information related to a server, and the attribute information related to the server is stored in the Mysql database. The updating module 440 is further adapted to, if the detecting module 470 detects that the page on which the server attribute list is located receives an editing operation, invoke a real-time updating interface preset on the Es database to update the data corresponding to the editing operation in the Es database.
In one embodiment of the invention, the server-related attribute information includes at least one of a server IP address, a server configuration parameter, a server brand, a server model, and the like.
According to yet another aspect of the present invention there is also provided a computer storage medium having stored thereon computer program code which, when run on a computing device, causes the computing device to perform the method of updating data of a Mysql database of any of the embodiments above to an Es database.
According to yet another aspect of the present invention, there is also provided a computing device including: a processor; a memory storing computer program code; the computer program code, when executed by the processor, causes the computing device to perform the method of updating data of the Mysql database to the Es database of any of the embodiments above.
According to any one of the above preferred embodiments or a combination of the preferred embodiments, the following advantageous effects can be achieved according to the embodiments of the present invention:
in the embodiment of the invention, firstly, a local message list is created based on a Mysql database, and when the Mysql database generates a data update event, a data update record corresponding to the data update event is written into the local message list. And then, the data update record in the local message list is read in real time by utilizing a first type of resident process started in advance, and the read data update record is issued to the Topic corresponding to the message queue. If a pre-started second type resident process is adopted to monitor the Topic message focused by the Es database in the message queue, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message. Therefore, the embodiment of the invention does not directly write the data update record in the Mysql database into the message queue, but writes the data update record in the Mysql database into the newly created message list local to the Mysql database, so that after the complete Mysql database update data is recorded in the local message list, the complete update data is written into the Es database, and the data finally updated to the Es database is high available complete data. The implementation process of updating the data in the Mysql database to the Es database is simple and efficient, and the real-time property and accuracy of the data updated to the Es database are effectively ensured. Further, when a user queries the data list, the user does not need to directly search a plurality of data tables from the Mysql database in a combined way, but can directly search the required data in the Es database, so that the data query efficiency is improved, and the data query time is reduced.
It will be clear to those skilled in the art that the specific working procedures of the above-described systems, devices and units may refer to the corresponding procedures in the foregoing method embodiments, and are not repeated herein for brevity.
In addition, each functional unit in the embodiments of the present invention may be physically independent, two or more functional units may be integrated together, or all functional units may be integrated in one processing unit. The integrated functional units may be implemented in hardware or in software or firmware.
Those of ordinary skill in the art will appreciate that: the integrated functional units, if implemented in software and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in essence or in whole or in part in the form of a software product stored in a storage medium, comprising instructions for causing a computing device (e.g., a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present invention when the instructions are executed. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a read-only memory (ROM), a random-access memory (RAM), a magnetic disk, or an optical disk, etc.
Alternatively, all or part of the steps of implementing the foregoing method embodiments may be implemented by hardware (such as a personal computer, a server, or a computing device such as a network device) associated with program instructions, where the program instructions may be stored on a computer-readable storage medium, and where the program instructions, when executed by a processor of the computing device, perform all or part of the steps of the method according to the embodiments of the present invention.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all technical features thereof can be replaced by others within the spirit and principle of the present invention; such modifications and substitutions do not depart from the scope of the invention.
Claims (24)
1. A method of updating data of a Mysql database to an Es database, comprising:
creating a local message list based on the Mysql database;
when the Mysql database generates a data update event, writing a data update record corresponding to the data update event into the local message list;
reading data update records in the local message list in real time by using a first type of resident process started in advance, and issuing the read data update records to Topic corresponding to the message queue;
if a pre-started second type resident process is adopted to monitor the Topic message focused by the Es database in the message queue, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message.
2. The method of claim 1, wherein when the Mysql database generates a data update event, writing a data update record corresponding to the data update event into the local message list comprises:
adding a hook function into a data access object Dao of the Mysql database;
when a Mysql database generates a data update event, writing a data update record corresponding to the data update event into the hook function;
and writing the data update record into the local message list by using the hook function.
3. The method according to claim 1 or 2, wherein the message queue comprises a message queue implemented based on a storage system dis or a distributed publish-subscribe message system kafka.
4. The method according to claim 1 or 2, further comprising:
if a plurality of Es databases pay attention to the corresponding Topic messages, respectively starting a second type resident process for each Es database;
and monitoring the Topic messages focused by any Es database in the message queue by a second resident process corresponding to any Es database.
5. The method of claim 1 or 2, wherein the data update event comprises at least one of: adding data, deleting data, modifying data.
6. The method of claim 5, wherein updating the Es database based on the acquired Topic message comprises:
analyzing the type of the acquired Topic message;
triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
7. The method of claim 6, wherein the type of the Topic message is associated with the data update event,
if the data update event is adding data, the type of the Topic message is an adding type;
if the data update event is deleting data, the type of the Topic message is a deleting type;
and if the data update event is modification data, the type of the Topic message is modification type.
8. The method of claim 7, wherein triggering a corresponding data processing event in accordance with the type of the Topic message, processing data corresponding to the Topic message in the Es database based on the data processing event, comprises:
if the type of the Topic message is an adding type, triggering a data adding event, and adding data corresponding to the Topic message in the Es database;
if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting the data corresponding to the Topic message in the Es database;
and if the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
9. The method according to claim 1 or 2, further comprising:
detecting whether data update exists in the Mysql database according to a preset time interval;
if yes, the updated data in the preset time interval is compensated to the Es database.
10. The method according to claim 1 or 2, further comprising:
detecting whether an editing operation is received on a page where a server attribute list is located, wherein the server attribute list comprises attribute information related to a server, and the attribute information related to the server is stored in the Mysql database;
if yes, calling a real-time updating interface preset on the Es database to update the data corresponding to the editing operation in the Es database.
11. The method of claim 10, wherein the attribute information associated with the server comprises at least one of:
server IP address, server configuration parameters, server brand, server model.
12. An apparatus for updating data of a Mysql database to an Es database, comprising:
the creation module is suitable for creating a local message list based on the Mysql database;
the writing module is suitable for writing a data update record corresponding to a data update event into the local message list when the Mysql database generates the data update event;
the reading module is suitable for reading the data update record in the local message list in real time by utilizing a first type of resident process started in advance, and issuing the read data update record to the Topic corresponding to the message queue;
and the updating module is suitable for acquiring the monitored Topic message if the Topic message focused by the Es database is monitored in the message queue by adopting the second type resident process started in advance, and updating the Es database based on the acquired Topic message.
13. The apparatus of claim 12, wherein the writing module is further adapted to:
adding a hook function into a data access object Dao of the Mysql database;
when a Mysql database generates a data update event, writing a data update record corresponding to the data update event into the hook function;
and writing the data update record into the local message list by using the hook function.
14. The apparatus of claim 12 or 13, wherein the message queue comprises a message queue implemented based on a storage system dis or a distributed publish-subscribe message system kafka.
15. The apparatus of claim 12 or 13, further comprising:
the starting module is suitable for starting the second type resident process for each Es database if a plurality of Es databases pay attention to the corresponding Topic messages;
and the monitoring module is suitable for monitoring the Topic messages focused by any Es database in the message queue by a second type resident process corresponding to the Es database.
16. The apparatus of claim 12 or 13, wherein the data update event comprises at least one of: adding data, deleting data, modifying data.
17. The apparatus of claim 16, wherein the update module is further adapted to:
analyzing the type of the acquired Topic message;
triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
18. The apparatus of claim 17, wherein the type of the Topic message is associated with the data update event,
if the data update event is adding data, the type of the Topic message is an adding type;
if the data update event is deleting data, the type of the Topic message is a deleting type;
and if the data update event is modification data, the type of the Topic message is modification type.
19. The apparatus of claim 18, wherein the update module is further adapted to:
if the type of the Topic message is an adding type, triggering a data adding event, and adding data corresponding to the Topic message in the Es database;
if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting the data corresponding to the Topic message in the Es database;
and if the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
20. The apparatus of claim 12 or 13, further comprising:
the detection module is suitable for detecting whether data update exists in the Mysql database according to a preset time interval;
and the compensation module is suitable for compensating the updated data in the preset time interval into the Es database if the detection module detects that the data update exists in the Mysql database according to the preset time interval.
21. The apparatus of claim 12 or 13, further comprising:
the detection module is suitable for detecting whether the page where the server attribute list is located receives editing operation or not, wherein the server attribute list comprises attribute information related to the server, and the attribute information related to the server is stored in the Mysql database;
and the updating module is further adapted to, if the detecting module detects that the page where the server attribute list is located receives the editing operation, call a real-time updating interface preset on the Es database to update the data corresponding to the editing operation in the Es database.
22. The apparatus of claim 21, wherein the attribute information associated with the server comprises at least one of:
server IP address, server configuration parameters, server brand, server model.
23. A computer storage medium storing computer program code which, when run on a computing device, causes the computing device to perform a method of updating data of a Mysql database of any of claims 1-11 to an Es database.
24. A computing device, comprising: a processor; a memory storing computer program code; when the computer program code is run by the processor, causes the computing device to perform the method of updating data of a Mysql database of any of claims 1-11 to an Es database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811028161.6A CN110928885B (en) | 2018-09-04 | 2018-09-04 | Method and device for updating data of Mysql database to Es database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811028161.6A CN110928885B (en) | 2018-09-04 | 2018-09-04 | Method and device for updating data of Mysql database to Es database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110928885A CN110928885A (en) | 2020-03-27 |
| CN110928885B true CN110928885B (en) | 2023-05-16 |
Family
ID=69855739
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811028161.6A Active CN110928885B (en) | 2018-09-04 | 2018-09-04 | Method and device for updating data of Mysql database to Es database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110928885B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111954038B (en) * | 2020-08-21 | 2021-12-21 | 湖南快乐阳光互动娱乐传媒有限公司 | Configurable search method and system based on ES efficient automatic synchronization |
| CN111966719B (en) * | 2020-10-21 | 2021-07-13 | 四川新网银行股份有限公司 | Method for refreshing local data cache of distributed consumer credit system in real time |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016101752A1 (en) * | 2014-12-22 | 2016-06-30 | 北京奇虎科技有限公司 | Method and device for data synchronization |
| CN106960007A (en) * | 2017-02-28 | 2017-07-18 | 北京京东尚科信息技术有限公司 | A kind of data-updating method, device and electronic equipment |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN107885804A (en) * | 2017-11-02 | 2018-04-06 | 平安科技（深圳）有限公司 | Database synchronization method, application server and computer-readable recording medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8510270B2 (en) * | 2010-07-27 | 2013-08-13 | Oracle International Corporation | MYSQL database heterogeneous log based replication |
-
2018
- 2018-09-04 CN CN201811028161.6A patent/CN110928885B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016101752A1 (en) * | 2014-12-22 | 2016-06-30 | 北京奇虎科技有限公司 | Method and device for data synchronization |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN106960007A (en) * | 2017-02-28 | 2017-07-18 | 北京京东尚科信息技术有限公司 | A kind of data-updating method, device and electronic equipment |
| CN107885804A (en) * | 2017-11-02 | 2018-04-06 | 平安科技（深圳）有限公司 | Database synchronization method, application server and computer-readable recording medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110928885A (en) | 2020-03-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9122694B1 (en) | Systems and methods for highly scalable system log analysis, deduplication and management | |
| US9836346B2 (en) | Error troubleshooting using a correlated knowledge base | |
| CN110162512B (en) | Log retrieval method, device and storage medium | |
| US11188560B2 (en) | Synchronizing object in local object storage node | |
| CN107040576B (en) | Information pushing method and device and communication system | |
| US9727394B2 (en) | Establishing causality order of computer trace records | |
| CN111597382A (en) | Network security auditing method and system | |
| US10855750B2 (en) | Centralized management of webservice resources in an enterprise | |
| CN110928885B (en) | Method and device for updating data of Mysql database to Es database | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| WO2016095716A1 (en) | Fault information processing method and related device | |
| CN111723245A (en) | Method for establishing incidence relation of different types of storage objects in data storage system | |
| CN113238815A (en) | Interface access control method, device, equipment and storage medium | |
| CN110928945B (en) | Data processing method and device for database and data processing system | |
| CN110515803B (en) | Processing method and device for log message and electronic equipment | |
| CN115454773A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN108228613B (en) | Data reading method and device | |
| CN114547501A (en) | Page back-off method and device based on vue management and computer equipment | |
| CN112988776A (en) | Method, device and equipment for updating text parsing rule and readable storage medium | |
| CN112948020A (en) | Configuration file processing method and device | |
| CN109739883B (en) | Method and device for improving data query performance and electronic equipment | |
| CN112632428A (en) | Page partial refreshing method, device, equipment and storage medium | |
| CN113779117A (en) | Data monitoring method and device, storage medium and electronic equipment | |
| JP2009282563A (en) | Data storage system, program, method, and monitoring device | |
| US10311032B2 (en) | Recording medium, log management method, and log management apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230424 Address after: 300450 No. 9-3-401, No. 39, Gaoxin 6th Road, Binhai Science Park, high tech Zone, Binhai New Area, Tianjin Applicant after: 3600 Technology Group Co.,Ltd. Address before: 100088 room 112, block D, 28 new street, new street, Xicheng District, Beijing (Desheng Park) Applicant before: BEIJING QIHOO TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |