CN110928885A - Method and device for updating data of Mysql database to Es database - Google Patents
Method and device for updating data of Mysql database to Es database Download PDFInfo
- Publication number
- CN110928885A CN110928885A CN201811028161.6A CN201811028161A CN110928885A CN 110928885 A CN110928885 A CN 110928885A CN 201811028161 A CN201811028161 A CN 201811028161A CN 110928885 A CN110928885 A CN 110928885A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- message
- updating
- type
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a device for updating data of a Mysql database to an Es database, wherein the method comprises the following steps: and creating a local message list based on the Mysql database, and writing a data updating record corresponding to the data updating event into the local message list when the Mysql database generates the data updating event. And then, reading the data updating record in the local message list in real time by using a first type of resident process which is started in advance, and issuing the read data updating record to the corresponding Topic of the message queue. And if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message. Therefore, in the embodiment of the invention, the data updating record in the Mysql database is firstly written into the local message list of the Mysql database, and then the complete updating data is written into the Es database, so that the data updated to the Es database is the high-availability complete data.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a device for updating data of a Mysql database to an Es database.
Background
Generally, a list of items and pages needs more information, and if the information is directly read from the Mysql database, many data tables need to be searched, so that the Search speed of the information is slow, and therefore, the data in the database can be considered to be indexed into an Es (Elastic Search server) database.
However, in the prior art, for example, for update (update) action of Mysql database, updated data is usually written directly into the message queue, and the resident process immediately consumes the message once monitoring the message written into the message queue, so that it is easy to happen that the updated data is not completely written into the message queue, and the resident process updates incomplete or unreliable or unavailable data into the Es database, thereby affecting the accuracy of data update. If the complete updated data is written into the Es database subsequently, compensation is needed by means of a compensation mechanism, and therefore instantaneity of data updating is poor.
Disclosure of Invention
In view of the above problems, the present invention is proposed to provide a method and apparatus for updating data of Mysql database to Es database that overcomes or at least partially solves the above problems.
According to an aspect of the invention, a method for updating data of a Mysql database to an Es database is provided, which comprises the following steps:
creating a local message list based on the Mysql database;
when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into the local message list;
reading data updating records in the local message list in real time by utilizing a first type of resident processes started in advance, and issuing the read data updating records to the corresponding Topic of the message queue;
and if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message.
Optionally, when the Mysql database generates a data update event, writing a data update record corresponding to the data update event into the local message list, including:
adding a hook function in a data access object Dao of the Mysql database;
when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into the hook function;
and writing the data updating record into the local message list by using the hook function.
Optionally, the message queue includes a message queue implemented based on a storage system Redis or a distributed publish-subscribe message system kafka.
Optionally, the method further comprises:
if a plurality of Es databases concern the corresponding Topic message, respectively starting a second type of resident process for each Es database;
and monitoring a Topic message concerned by any Es database in the message queue by a second resident process corresponding to the Es database.
Optionally, the data update event comprises at least one of: add data, delete data, modify data.
Optionally, updating the Es database based on the obtained Topic message includes:
analyzing the type of the acquired Topic message;
triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
Optionally, the type of the Topic message is related to the data update event,
if the data updating event is adding data, the type of the Topic message is an adding type;
if the data updating event is deletion data, the type of the Topic message is a deletion type;
and if the data updating event is modification data, the type of the Topic message is a modification type.
Optionally, triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event, including:
if the type of the Topic message is an adding type, triggering a data adding event, and adding data corresponding to the Topic message in the Es database;
if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting data corresponding to the Topic message in the Es database;
if the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
Optionally, the method further comprises:
detecting whether data updating exists in the Mysql database or not according to a preset time interval;
and if so, compensating the updated data in the preset time interval to the Es database.
Optionally, the method further comprises:
detecting whether a page where a server attribute list is located receives an editing operation, wherein the server list comprises attribute information related to the server, and the attribute information related to the server is stored in the Mysql database;
and if so, calling a real-time updating interface preset on the Es database to update the data corresponding to the editing operation in the Es database.
Optionally, the attribute information related to the server includes at least one of:
server IP address, server configuration parameters, server brand, server model.
According to another aspect of the present invention, there is also provided an apparatus for updating data of a Mysql database to an Es database, including:
the creating module is suitable for creating a local message list based on the Mysql database;
the writing module is suitable for writing a data updating record corresponding to a data updating event into the local message list when the Mysql database generates the data updating event;
the reading module is suitable for reading the data updating record in the local message list in real time by utilizing a first type of resident process which is started in advance, and issuing the read data updating record to the corresponding Topic of the message queue;
and the updating module is suitable for acquiring the monitored Topic message if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, and updating the Es database based on the acquired Topic message.
Optionally, the writing module is further adapted to:
adding a hook function in a data access object Dao of the Mysql database;
when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into the hook function;
and writing the data updating record into the local message list by using the hook function.
Optionally, the message queue includes a message queue implemented based on a storage system Redis or a distributed publish-subscribe message system kafka.
Optionally, the apparatus further comprises:
the starting module is suitable for respectively starting a second type of resident process for each Es database if the Es databases concern the corresponding Topic message;
and the monitoring module is suitable for monitoring the Topic message concerned by any Es database in the message queue by the second resident process corresponding to the Es database.
Optionally, the data update event comprises at least one of: add data, delete data, modify data.
Optionally, the update module is further adapted to:
analyzing the type of the acquired Topic message;
triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
Optionally, the type of the Topic message is related to the data update event,
if the data updating event is adding data, the type of the Topic message is an adding type;
if the data updating event is deletion data, the type of the Topic message is a deletion type;
and if the data updating event is modification data, the type of the Topic message is a modification type.
Optionally, the update module is further adapted to:
if the type of the Topic message is an adding type, triggering a data adding event, and adding data corresponding to the Topic message in the Es database;
if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting data corresponding to the Topic message in the Es database;
if the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
Optionally, the apparatus further comprises:
the detection module is suitable for detecting whether data updating exists in the Mysql database according to a preset time interval;
and the compensation module is suitable for compensating the updated data in the preset time interval to the Es database if the detection module detects that the data in the Mysql database is updated according to the preset time interval.
Optionally, the detecting module is further adapted to detect whether a page where a server attribute list is located receives an editing operation, where the server list includes attribute information related to the server, and the attribute information related to the server is stored in the Mysql database;
and the updating module is further adapted to call a real-time updating interface preset on the Es database to update data corresponding to the editing operation in the Es database if the detecting module detects that the page where the server attribute list is located receives the editing operation.
Optionally, the attribute information related to the server includes at least one of:
server IP address, server configuration parameters, server brand, server model.
According to yet another aspect of the present invention, there is also provided a computer storage medium storing computer program code which, when run on a computing device, causes the computing device to perform the method of updating data of a Mysql database to an Es database as described in any of the embodiments above.
According to yet another aspect of the present invention, there is also provided a computing device comprising: a processor; a memory storing computer program code; the computer program code, when executed by the processor, causes the computing device to perform a method of data update of Mysql database to Es database as described in any of the embodiments above.
In the embodiment of the invention, firstly, a local message list is created based on the Mysql database, and when a data updating event is generated in the Mysql database, a data updating record corresponding to the data updating event is written into the local message list. And then, reading the data updating record in the local message list in real time by using a first type of resident process which is started in advance, and issuing the read data updating record to the corresponding Topic of the message queue. And if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message. Therefore, in the embodiment of the invention, the data updating record in the Mysql database is not directly written into the message queue, but is written into the newly created message list which is located locally in the Mysql database, so that the complete updating data can be written into the Es database after the complete Mysql database updating data is recorded in the local message list, and the data finally updated into the Es database is highly available complete data. The implementation process of updating the data in the Mysql database to the Es database is simple and efficient, and the real-time performance and the accuracy of the data updated to the Es database are effectively guaranteed. Furthermore, when a user queries the data list, the user does not need to directly jointly query a plurality of data tables from the Mysql database, but can directly search the required data in the Es database, so that the data query efficiency is improved, and the data query time is shortened.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments thereof, taken in conjunction with the accompanying drawings.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 shows a flow diagram of a method for updating data of a Mysql database to an Es database, according to one embodiment of the invention;
FIG. 2 shows a schematic diagram of the process of publishing data of a Mysql database into a message queue according to one embodiment of the invention;
FIG. 3 is a diagram illustrating a process for updating a message in a message queue to an Es database according to an embodiment of the invention;
FIG. 4 is a schematic diagram of an apparatus for updating data of a Mysql database to an Es database according to an embodiment of the invention; and
fig. 5 is a schematic structural diagram illustrating an apparatus for updating data of the Mysql database to the Es database according to another embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
In order to solve the above technical problem, an embodiment of the present invention provides a method for updating data of a Mysql database to an Es database. Fig. 1 is a flowchart illustrating a method for updating data of a Mysql database to an Es database according to an embodiment of the present invention. Referring to fig. 1, the method includes at least steps S102 to S108.
Step S102, a local message list is created based on the Mysql database.
And step S104, when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into a local message list.
Step S106, reading the data updating record in the local message list in real time by utilizing the first type of resident process started in advance, and issuing the read data updating record to the corresponding Topic of the message queue.
In this step, since the Mysql database includes a plurality of data tables, and each data table corresponds to one Topic, when data in the data tables in the Mysql database is updated, the first type resident process issues the read updated data to the Topic corresponding to the message queue. For example, the Mysql database includes a device detail table, and then the name of the topic corresponding to the device detail table may be cmdb.
And S108, if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message.
In the embodiment of the invention, the data updating record in the Mysql database is not directly written into the message queue, but is written into the newly created message list which is positioned at the local of the Mysql database, so that the complete updating data can be written into the Es database after the complete Mysql database updating data is recorded in the local message list, and the data finally updated into the Es database is highly available complete data. The implementation process of updating the data in the Mysql database to the Es database is simple and efficient, and the real-time performance and the accuracy of the data updated to the Es database are effectively guaranteed. Through actual operation, the scheme of the invention can update the updated data generated by the Mysql database to the Es database within 30ms, thereby greatly improving the updating speed of the data in the Es database. Furthermore, when a user queries the data list, the user does not need to directly jointly query a plurality of data tables from the Mysql database, but can directly search the required data in the Es database, so that the data query efficiency is improved, and the data query time is shortened.
Referring to step S104, in an embodiment of the present invention, when the data update record corresponding to the data update event is written into the local message list, the data update record may also be written into the local message list through a hook function. Specifically, first, a hook function is added to a Data Access Object Dao (Data Access Object) in the Mysql database. Then, when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into the hook function, and further writing the data updating record into the local message list publish _ list by using the hook function.
Referring to step S106 above, in an embodiment of the present invention, the message queue may include a message queue implemented based on a key-value storage system Redis or a distributed publish-subscribe message system kafka. For example, if the message queue is implemented based on the storage system Redis, it adopts a pubsub (Publish/Subscribe mode) function of the storage system Redis. Of course, the message queue may also be implemented based on other systems, and the embodiment of the present invention does not specifically limit the type of the message queue.
Referring to step S108 above, in an embodiment of the present invention, if there are multiple Es databases that concern the corresponding Topic message, a second type of resident process may be respectively started for each Es database, and then the second type of resident process corresponding to any one Es database may monitor the Topic message that concerns the Es database in the message queue. When any one of the second-type resident processes monitors the Topic message concerned by the corresponding Es database in the message queue, the corresponding Es database can be updated based on the acquired Topic message.
In an embodiment of the invention, the data update event may comprise at least one of adding data, deleting data, modifying data. In step S108, when the Es database is updated based on the obtained Topic message, the obtained Topic message may be analyzed first, that is, the type of the obtained Topic message is analyzed, then a corresponding data processing event is triggered according to the analyzed type of the Topic message, and then the data corresponding to the Topic message in the Es database is processed based on the data processing event, so as to update the Es database.
In this embodiment, the type of the Topic message is related to the data update event, for example, if the data update event is add data, the type of the Topic message is an add type. If the data update event is delete data, the type of the Topic message is a delete type. If the data update event is modification data, the type of the Topic message is modification type.
In an optional embodiment of the present invention, a corresponding data processing event is triggered according to the type of the Topic message, and when data corresponding to the Topic message in the Es database is processed based on the data processing event, if the type of the Topic message is an addition type, a data addition event is triggered, and data corresponding to the Topic message is added in the Es database. And if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting the data corresponding to the Topic message in the Es database. If the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
In order to more clearly embody the solution of the present invention, a detailed description will now be given of a process of updating data of the Mysql database to the Es database according to the present invention.
First, the process of publishing data of the Mysql database to the message queue is introduced.
Referring to fig. 2, in the embodiment, a hook function is added in advance in a data access object Dao of the Mysql database, and when a data update event is generated in the Mysql database (the data update event shown in fig. 2 is an add data add), a data update record corresponding to the data update event is written in the hook function, and then the data update record is written in a local message list publish _ list by using the hook function. As described above, the Mysql database includes a plurality of data tables, each of which corresponds to a Topic, and thus, in the embodiment shown in FIG. 2, it is assumed that the data list in the Mysql database to which data is added corresponds to Topic-01. Then, a first type of pre-started resident process is adopted to read the data updating record from the local message list publish _ list in real time, and the first type of resident process issues the data updating record publish to a message queue realized based on the key-value storage system Redis.
In this embodiment, the Model shown in fig. 2 refers to a service logic layer, through which corresponding service processing logic can be set, for example, the set service processing logic may include adding data add, modifying data update, and deleting data delete. The Mysql database generates data update events according to corresponding business processing logic preset in the business logic layer.
Then, a procedure of how to update the Es database with the data update record in the message queue is described.
Referring to fig. 3, the data in the Topic corresponding to the message queue includes the data update record in the Mysql database. The previous description has introduced that the message queue is monitored for Topic messages of interest to the Es database by a second type of resident process that is started in advance. In this embodiment, the second type resident process is started by the Console, which may be a terminal server. The second type of resident process monitors the Topic message concerned by the Es database in a message queue realized based on the Redis of the storage system through a data access object EDao. For example, the Es database may focus on at least one Topic message in Topic-01, Topic-02, and so on. And when the second resident process monitors the Topic message concerned by the Es database, the corresponding Topic message is acquired, the EDao-event is triggered to process the Topic message in the event, and the Es database is updated.
In an embodiment of the present invention, in order to ensure that the updated data in the Mysql database can be accurately updated to the Es database, an embodiment of the present invention may further provide a compensation mechanism based on the foregoing method, and the compensation mechanism may adjust time to increment the updated data according to the needs of the user. Specifically, whether data updating exists in the Mysql database is detected according to a preset time interval, and if yes, the updated data in the preset time interval is compensated to the Es database. In this embodiment, the preset time interval may be any preset time interval such as 5 minutes, 6 minutes, 7 minutes, and the like, which is not specifically limited in this embodiment of the present invention.
For example, if the preset time interval is 5 minutes, a new resident process may be started first, so as to detect whether updated data exists in the Mysql database every 5 minutes by using the new resident process, and when it is detected that updated data exists in the Mysql database at a certain time, the updated data in 5 minutes before the current time is acquired from the Mysql database, and the acquired updated data is compensated to the Es database. The compensation mechanism can be used as a guaranteed operation to avoid the situation that data in the Mysql database is updated and cannot be timely and accurately updated to the Es database due to other reasons (such as that updated data is not successfully written into a local message list, or a data update record is not successfully issued to a Topic corresponding to a message queue, and the like).
In another embodiment of the present invention, it may further be detected whether the page where the server attribute list is located receives an editing operation (for example, an editing operation of a user on the page), where the server list includes attribute information related to the server, and the attribute information related to the server is stored in the Mysql database. If yes, calling a real-time updating interface preset on the Es database to update data corresponding to the editing operation in the Es database. The checking process of the page is carried out in real time, namely for the editing operation received by the page, the data information generated by the editing operation is updated to the Es database in real time, and the process is approximate to 0 delay. Of course, the data information generated during the editing operation on the page is also stored in the Mysql database.
In this embodiment, the server-related attribute information may include at least one of a server IP address, a server configuration parameter, a server brand, a server model, information on a person in charge related to the server, a manufacturer of the server, and the like.
Based on the same inventive concept, the embodiment of the present invention further provides a device for updating data of the Mysql database to the Es database, and fig. 4 shows a schematic structural diagram of the device for updating data of the Mysql database to the Es database according to an embodiment of the present invention. Referring to fig. 4, the apparatus 400 for updating data of Mysql database to Es database includes at least a creating module 410, a writing module 420, a reading module 430, and an updating module 440.
Now, the functions of the components or devices of the apparatus 400 for updating data of the Mysql database to the Es database and the connection relationship between the components are described:
a creation module 410 adapted to create a local message list based on the Mysql database;
a writing module 420, coupled to the creating module 410, adapted to write a data update record corresponding to the data update event into the local message list when the Mysql database generates the data update event;
a reading module 430, coupled to the writing module 420, adapted to read the data update record in the local message list in real time by using a first type of pre-started resident process, and issue the read data update record to the corresponding Topic of the message queue;
and an updating module 440, coupled to the reading module 430, adapted to acquire the monitored Topic message if the Topic message concerned by the Es database is monitored in the message queue by using a pre-started second type resident process, and update the Es database based on the acquired Topic message.
In an embodiment of the present invention, the writing module 420 is further adapted to add a hook function to the data access object Dao of the Mysql database, and write a data update record corresponding to the data update event into the hook function when the Mysql database generates the data update event. And writing the data updating record into the local message list by using a hook function.
In an embodiment of the invention, the message queue comprises a message queue implemented based on a storage system Redis or a distributed publish-subscribe message system kafka.
The embodiment of the invention also provides another device for updating the data of the Mysql database to the Es database, and fig. 5 is a schematic structural diagram of the device for updating the data of the Mysql database to the Es database according to another embodiment of the invention. Referring to fig. 5, the apparatus 400 for updating data of the Mysql database to the Es database may further include a starting module 450, a monitoring module 460, a detecting module 470, and a compensating module 480, in addition to the above modules.
The starting module 450 is adapted to start a second type of resident process for each Es database, if there are multiple Es databases that focus on the corresponding Topic message.
And the monitoring module 460, coupled to the starting module 450 and the updating module 440, is adapted to monitor the Topic message concerned by any Es database in the message queue by the second type resident process corresponding to the Es database.
A detecting module 470, coupled to the creating module 410, is adapted to detect whether there is a data update in the Mysql database according to a preset time interval.
And the compensation module 480 is coupled to the detection module 470, and is adapted to compensate the data updated in the preset time interval to the Es database if the detection module 470 detects that data update exists in the Mysql database according to the preset time interval.
In an embodiment of the invention, the data update event comprises at least one of adding data, deleting data, modifying data.
In an embodiment of the present invention, the updating module 440 is further adapted to analyze the type of the obtained Topic message. And triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
In an embodiment of the present invention, the type of the Topic message is related to the data update event, and if the data update event is the add data, the type of the Topic message is the add type. If the data update event is delete data, the type of the Topic message is a delete type. If the data update event is modification data, the type of the Topic message is modification type.
In an embodiment of the present invention, the updating module 440 is further adapted to, if the type of the Topic message is an addition type, trigger a data addition event, and add data corresponding to the Topic message in the Es database. And if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting the data corresponding to the Topic message in the Es database. If the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
In an embodiment of the present invention, the detecting module 470 is further adapted to detect whether the page where the server attribute list is located receives an editing operation, where the server attribute list includes attribute information related to the server, and the attribute information related to the server is stored in the Mysql database. The updating module 440 is further adapted to, if the detecting module 470 detects that the page where the server attribute list is located receives the editing operation, call a real-time updating interface preset on the Es database to update data corresponding to the editing operation in the Es database.
In an embodiment of the invention, the attribute information related to the server includes at least one of a server IP address, a server configuration parameter, a server brand, a server model, and the like.
According to yet another aspect of the present invention, there is also provided a computer storage medium storing computer program code which, when run on a computing device, causes the computing device to perform the method of updating data of the Mysql database to the Es database of any of the above embodiments.
According to yet another aspect of the present invention, there is also provided a computing device comprising: a processor; a memory storing computer program code; the computer program code, when executed by the processor, causes the computing device to perform the method of data update of the Mysql database to the Es database of any of the embodiments above.
According to any one or a combination of the above preferred embodiments, the following advantages can be achieved by the embodiments of the present invention:
in the embodiment of the invention, firstly, a local message list is created based on the Mysql database, and when a data updating event is generated in the Mysql database, a data updating record corresponding to the data updating event is written into the local message list. And then, reading the data updating record in the local message list in real time by using a first type of resident process which is started in advance, and issuing the read data updating record to the corresponding Topic of the message queue. And if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message. Therefore, in the embodiment of the invention, the data updating record in the Mysql database is not directly written into the message queue, but is written into the newly created message list which is located locally in the Mysql database, so that the complete updating data can be written into the Es database after the complete Mysql database updating data is recorded in the local message list, and the data finally updated into the Es database is highly available complete data. The implementation process of updating the data in the Mysql database to the Es database is simple and efficient, and the real-time performance and the accuracy of the data updated to the Es database are effectively guaranteed. Furthermore, when a user queries the data list, the user does not need to directly jointly query a plurality of data tables from the Mysql database, but can directly search the required data in the Es database, so that the data query efficiency is improved, and the data query time is shortened.
It is clear to those skilled in the art that the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and for the sake of brevity, further description is omitted here.
In addition, the functional units in the embodiments of the present invention may be physically independent of each other, two or more functional units may be integrated together, or all the functional units may be integrated in one processing unit. The integrated functional units may be implemented in the form of hardware, or in the form of software or firmware.
Those of ordinary skill in the art will understand that: the integrated functional units, if implemented in software and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computing device (e.g., a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention when the instructions are executed. And the aforementioned storage medium includes: u disk, removable hard disk, Read Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disk, and other various media capable of storing program code.
Alternatively, all or part of the steps of implementing the foregoing method embodiments may be implemented by hardware (such as a computing device, e.g., a personal computer, a server, or a network device) associated with program instructions, which may be stored in a computer-readable storage medium, and when the program instructions are executed by a processor of the computing device, the computing device executes all or part of the steps of the method according to the embodiments of the present invention.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments can be modified or some or all of the technical features can be equivalently replaced within the spirit and principle of the present invention; such modifications or substitutions do not depart from the scope of the present invention.
In an embodiment of the present invention, a1, a method for updating data of a Mysql database to an Es database is provided, including:
creating a local message list based on the Mysql database;
when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into the local message list;
reading data updating records in the local message list in real time by utilizing a first type of resident processes started in advance, and issuing the read data updating records to the corresponding Topic of the message queue;
and if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message.
A2, the method according to A1, wherein when the Mysql database generates a data update event, writing a data update record corresponding to the data update event into the local message list includes:
adding a hook function in a data access object Dao of the Mysql database;
when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into the hook function;
and writing the data updating record into the local message list by using the hook function.
A3, the method according to A1 or A2, wherein the message queue comprises a message queue implemented based on a storage system Redis or a distributed publish-subscribe messaging system kafka.
A4, the method according to A1 or A2, further comprising:
if a plurality of Es databases concern the corresponding Topic message, respectively starting a second type of resident process for each Es database;
and monitoring a Topic message concerned by any Es database in the message queue by a second resident process corresponding to the Es database.
A5, the method of A1 or A2, wherein the data update event includes at least one of: add data, delete data, modify data.
A6, the method according to A5, wherein updating the Es database based on the acquired Topic message includes:
analyzing the type of the acquired Topic message;
triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
A7, the method of A6, wherein the type of the Topic message relates to the data update event,
if the data updating event is adding data, the type of the Topic message is an adding type;
if the data updating event is deletion data, the type of the Topic message is a deletion type;
and if the data updating event is modification data, the type of the Topic message is a modification type.
A8, the method according to A7, wherein the triggering of the corresponding data processing event according to the type of the Topic message and the processing of the data corresponding to the Topic message in the Es database based on the data processing event include:
if the type of the Topic message is an adding type, triggering a data adding event, and adding data corresponding to the Topic message in the Es database;
if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting data corresponding to the Topic message in the Es database;
if the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
A9, the method according to A1 or A2, further comprising:
detecting whether data updating exists in the Mysql database or not according to a preset time interval;
and if so, compensating the updated data in the preset time interval to the Es database.
A10, the method according to A1 or A2, further comprising:
detecting whether a page where a server attribute list is located receives an editing operation, wherein the server list comprises attribute information related to the server, and the attribute information related to the server is stored in the Mysql database;
and if so, calling a real-time updating interface preset on the Es database to update the data corresponding to the editing operation in the Es database.
A11, the method according to A10, wherein the attribute information related to the server includes at least one of:
server IP address, server configuration parameters, server brand, server model.
B12, a device for updating the data of the Mysql database to the Es database, comprising:
the creating module is suitable for creating a local message list based on the Mysql database;
the writing module is suitable for writing a data updating record corresponding to a data updating event into the local message list when the Mysql database generates the data updating event;
the reading module is suitable for reading the data updating record in the local message list in real time by utilizing a first type of resident process which is started in advance, and issuing the read data updating record to the corresponding Topic of the message queue;
and the updating module is suitable for acquiring the monitored Topic message if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, and updating the Es database based on the acquired Topic message.
B13, the apparatus of B12, wherein the write module is further adapted to:
adding a hook function in a data access object Dao of the Mysql database;
when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into the hook function;
and writing the data updating record into the local message list by using the hook function.
B14, the apparatus according to B12 or B13, wherein the message queue comprises a message queue implemented based on a storage system Redis or a distributed publish-subscribe messaging system kafka.
B15, the device according to B12 or B13, wherein further comprising:
the starting module is suitable for respectively starting a second type of resident process for each Es database if the Es databases concern the corresponding Topic message;
and the monitoring module is suitable for monitoring the Topic message concerned by any Es database in the message queue by the second resident process corresponding to the Es database.
B16, the apparatus of B12 or B13, wherein the data update event comprises at least one of: add data, delete data, modify data.
B17, the apparatus of B16, wherein the update module is further adapted to:
analyzing the type of the acquired Topic message;
triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
B18, the apparatus of B17, wherein the type of the Topic message is related to the data update event,
if the data updating event is adding data, the type of the Topic message is an adding type;
if the data updating event is deletion data, the type of the Topic message is a deletion type;
and if the data updating event is modification data, the type of the Topic message is a modification type.
B19, the apparatus of B18, wherein the update module is further adapted to:
if the type of the Topic message is an adding type, triggering a data adding event, and adding data corresponding to the Topic message in the Es database;
if the type of the Topic message is a deletion type, triggering a data deletion event, and deleting data corresponding to the Topic message in the Es database;
if the type of the Topic message is a modification type, triggering a data modification event, and correspondingly modifying the data corresponding to the Topic message in the Es database.
B20, the device according to B12 or B13, wherein further comprising:
the detection module is suitable for detecting whether data updating exists in the Mysql database according to a preset time interval;
and the compensation module is suitable for compensating the updated data in the preset time interval to the Es database if the detection module detects that the data in the Mysql database is updated according to the preset time interval.
B21, the device according to B12 or B13, wherein,
the detection module is further adapted to detect whether a page where a server attribute list is located receives an editing operation, wherein the server list includes attribute information related to the server, and the attribute information related to the server is stored in the Mysql database;
and the updating module is further adapted to call a real-time updating interface preset on the Es database to update data corresponding to the editing operation in the Es database if the detecting module detects that the page where the server attribute list is located receives the editing operation.
B22, the apparatus according to B21, wherein the attribute information related to the server includes at least one of:
server IP address, server configuration parameters, server brand, server model.
C23, a computer storage medium storing computer program code which, when run on a computing device, causes the computing device to perform the method of Mysql database data update to Es database of any of a1-a 11.
D24, a computing device, comprising: a processor; a memory storing computer program code; the computer program code, when executed by the processor, causes the computing device to perform a method of any of A1-A11 for updating data of the Mysql database to the Es database.
Claims (10)
1. A method for updating data of a Mysql database to an Es database comprises the following steps:
creating a local message list based on the Mysql database;
when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into the local message list;
reading data updating records in the local message list in real time by utilizing a first type of resident processes started in advance, and issuing the read data updating records to the corresponding Topic of the message queue;
and if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, acquiring the monitored Topic message, and updating the Es database based on the acquired Topic message.
2. The method of claim 1, wherein when the Mysql database generates a data update event, writing a data update record corresponding to the data update event into the local message list comprises:
adding a hook function in a data access object Dao of the Mysql database;
when the Mysql database generates a data updating event, writing a data updating record corresponding to the data updating event into the hook function;
and writing the data updating record into the local message list by using the hook function.
3. The method of claim 1 or 2, wherein the message queue comprises a message queue implemented based on a storage system Redis or a distributed publish-subscribe messaging system kafka.
4. The method of claim 1 or 2, further comprising:
if a plurality of Es databases concern the corresponding Topic message, respectively starting a second type of resident process for each Es database;
and monitoring a Topic message concerned by any Es database in the message queue by a second resident process corresponding to the Es database.
5. The method of claim 1 or 2, wherein the data update event comprises at least one of: add data, delete data, modify data.
6. The method of claim 5, wherein updating the Es database based on the obtained Topic message comprises:
analyzing the type of the acquired Topic message;
triggering a corresponding data processing event according to the type of the Topic message, and processing data corresponding to the Topic message in the Es database based on the data processing event so as to update the Es database.
7. The method of claim 6, wherein the type of the Topic message is related to the data update event,
if the data updating event is adding data, the type of the Topic message is an adding type;
if the data updating event is deletion data, the type of the Topic message is a deletion type;
and if the data updating event is modification data, the type of the Topic message is a modification type.
8. An apparatus for updating data of Mysql database to Es database, comprising:
the creating module is suitable for creating a local message list based on the Mysql database;
the writing module is suitable for writing a data updating record corresponding to a data updating event into the local message list when the Mysql database generates the data updating event;
the reading module is suitable for reading the data updating record in the local message list in real time by utilizing a first type of resident process which is started in advance, and issuing the read data updating record to the corresponding Topic of the message queue;
and the updating module is suitable for acquiring the monitored Topic message if the Topic message concerned by the Es database is monitored in the message queue by adopting a second type of resident process which is started in advance, and updating the Es database based on the acquired Topic message.
9. A computer storage medium storing computer program code which, when run on a computing device, causes the computing device to perform a method of Mysql database data update to Es database as claimed in any of claims 1-7.
10. A computing device, comprising: a processor; a memory storing computer program code; the computer program code, when executed by the processor, causes the computing device to perform a method of updating data of the Mysql database of any of claims 1-7 to the Es database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811028161.6A CN110928885B (en) | 2018-09-04 | 2018-09-04 | Method and device for updating data of Mysql database to Es database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811028161.6A CN110928885B (en) | 2018-09-04 | 2018-09-04 | Method and device for updating data of Mysql database to Es database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110928885A true CN110928885A (en) | 2020-03-27 |
| CN110928885B CN110928885B (en) | 2023-05-16 |
Family
ID=69855739
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811028161.6A Active CN110928885B (en) | 2018-09-04 | 2018-09-04 | Method and device for updating data of Mysql database to Es database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110928885B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111954038A (en) * | 2020-08-21 | 2020-11-17 | 湖南快乐阳光互动娱乐传媒有限公司 | Configurable search method and system based on ES efficient automatic synchronization |
| CN111966719A (en) * | 2020-10-21 | 2020-11-20 | 四川新网银行股份有限公司 | Method for refreshing local data cache of distributed consumer credit system in real time |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120030172A1 (en) * | 2010-07-27 | 2012-02-02 | Oracle International Corporation | Mysql database heterogeneous log based replication |
| WO2016101752A1 (en) * | 2014-12-22 | 2016-06-30 | 北京奇虎科技有限公司 | Method and device for data synchronization |
| CN106960007A (en) * | 2017-02-28 | 2017-07-18 | 北京京东尚科信息技术有限公司 | A kind of data-updating method, device and electronic equipment |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN107885804A (en) * | 2017-11-02 | 2018-04-06 | 平安科技（深圳）有限公司 | Database synchronization method, application server and computer-readable recording medium |
-
2018
- 2018-09-04 CN CN201811028161.6A patent/CN110928885B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120030172A1 (en) * | 2010-07-27 | 2012-02-02 | Oracle International Corporation | Mysql database heterogeneous log based replication |
| WO2016101752A1 (en) * | 2014-12-22 | 2016-06-30 | 北京奇虎科技有限公司 | Method and device for data synchronization |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN106960007A (en) * | 2017-02-28 | 2017-07-18 | 北京京东尚科信息技术有限公司 | A kind of data-updating method, device and electronic equipment |
| CN107885804A (en) * | 2017-11-02 | 2018-04-06 | 平安科技（深圳）有限公司 | Database synchronization method, application server and computer-readable recording medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111954038A (en) * | 2020-08-21 | 2020-11-17 | 湖南快乐阳光互动娱乐传媒有限公司 | Configurable search method and system based on ES efficient automatic synchronization |
| CN111954038B (en) * | 2020-08-21 | 2021-12-21 | 湖南快乐阳光互动娱乐传媒有限公司 | Configurable search method and system based on ES efficient automatic synchronization |
| CN111966719A (en) * | 2020-10-21 | 2020-11-20 | 四川新网银行股份有限公司 | Method for refreshing local data cache of distributed consumer credit system in real time |
| CN111966719B (en) * | 2020-10-21 | 2021-07-13 | 四川新网银行股份有限公司 | Method for refreshing local data cache of distributed consumer credit system in real time |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110928885B (en) | 2023-05-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7200259B2 (en) | Data retention handling for data object stores | |
| US9836346B2 (en) | Error troubleshooting using a correlated knowledge base | |
| US9965480B2 (en) | Smart archiving of real-time performance monitoring data | |
| US10204016B1 (en) | Incrementally backing up file system hard links based on change logs | |
| US11093340B2 (en) | Summary file change log for faster forever incremental backup | |
| US9031909B2 (en) | Provisioning and/or synchronizing using common metadata | |
| CN110737682A (en) | cache operation method, device, storage medium and electronic equipment | |
| CN108809704B (en) | Data deduplication statistical method and device based on dynamic time window | |
| CN107688626B (en) | Slow query log processing method and device and electronic equipment | |
| US20200233837A1 (en) | Intelligent method to index storage system files accounting for snapshots | |
| US20230359587A1 (en) | Analysis of streaming data using deltas and snapshots | |
| CN110928885A (en) | Method and device for updating data of Mysql database to Es database | |
| EP3832477A1 (en) | Efficient storage method for time series data | |
| US11243979B1 (en) | Asynchronous propagation of database events | |
| US9767180B2 (en) | Floating time dimension design | |
| CN111435327A (en) | Log record processing method, device and system | |
| CN115454773A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN104317820A (en) | Statistical method and device of report | |
| CN114547501A (en) | Page back-off method and device based on vue management and computer equipment | |
| CN110515803B (en) | Processing method and device for log message and electronic equipment | |
| JP6988178B2 (en) | Information processing device, log management program and log management method | |
| CN112463741A (en) | Cleaning method for aggregated large files and related equipment | |
| CN112632428A (en) | Page partial refreshing method, device, equipment and storage medium | |
| US9069751B1 (en) | Systems and methods for managing document pedigrees | |
| US10311032B2 (en) | Recording medium, log management method, and log management apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230424 Address after: 300450 No. 9-3-401, No. 39, Gaoxin 6th Road, Binhai Science Park, high tech Zone, Binhai New Area, Tianjin Applicant after: 3600 Technology Group Co.,Ltd. Address before: 100088 room 112, block D, 28 new street, new street, Xicheng District, Beijing (Desheng Park) Applicant before: BEIJING QIHOO TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |