CN117149763B - Index switching synchronization method and device and storage medium - Google Patents
Index switching synchronization method and device and storage medium Download PDFInfo
- Publication number
- CN117149763B CN117149763B CN202310998608.7A CN202310998608A CN117149763B CN 117149763 B CN117149763 B CN 117149763B CN 202310998608 A CN202310998608 A CN 202310998608A CN 117149763 B CN117149763 B CN 117149763B
- Authority
- CN
- China
- Prior art keywords
- index
- data information
- index data
- message queue
- switching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 134
- 230000001360 synchronised effect Effects 0.000 claims abstract description 39
- 230000010354 integration Effects 0.000 claims abstract description 8
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012544 monitoring process Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24578—Query processing with adaptation to user needs using ranking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The application relates to an index switching synchronization method, which is characterized in that after a new index is rebuilt and in the process of switching indexes, second index data information updated in the index to be rebuilt is continuously synchronized to a message queue, identifiers of the second index data information updated in the index switching process are integrated, after the index switching is completed, the identifiers of third index data information updated in the index after target rebuilding are judged to be in a first set after integration, and the second index data information and the third index data information are written into the index after target rebuilding according to the synchronization process of a write lock for acquiring the third index data information or the second index data information. According to the method, consumption of pause information is not needed when index switching is carried out, synchronization of all index data updated in the index switching process to the target reconstruction index is achieved through control of the time stamp and the write lock of the index data information after index switching is completed, and data consistency after index switching is completed is guaranteed.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to an index switching synchronization method, an index switching synchronization device, and a storage medium.
Background
In the internet system, due to reliability of providing services to the outside and the need to quickly retrieve data from massive data, a non-relational database is generally used to store the data. In the non-relational database, an elastic search (search analysis engine) is often used as a search index for data lines, and in order to ensure consistency between index data and data in a service system, ES indexes are usually regularly reconstructed to keep consistency between index data and service data. However, after the index is rebuilt, the consumption of the pause message is still needed when the new index and the old index are switched, so that the consistency of the index data and the service data after the index is switched can be ensured, and the synchronization of the update data in the index switching process after the index is rebuilt can not be realized.
Disclosure of Invention
Based on the above, the invention aims to provide an index switching synchronization method, which realizes the data consistency after index switching under the condition that consumption in the index switching process is not needed to be paused, and solves the problem that the consumption of pause information is still needed in the new and old index switching process after index reconstruction to ensure the data consistency. The specific scheme is as follows:
In a first aspect, an index switching synchronization method is characterized in that: comprising the following steps:
when the starting of an index rebuilding process is monitored, rebuilding an index to be rebuilt to obtain an initial rebuilt index, and synchronizing first index data information updated in the index to be rebuilt to a first message queue in the process of obtaining the initial rebuilt index;
writing the first index data information and the service data information in the first message queue into the initial reconstructed index to obtain a target reconstructed index; the service data information is data in a commodity service system;
traversing the first message queue, switching an index alias to the target rebuilt index if the first message queue is empty, and continuously synchronizing second index data information updated in the index to be rebuilt to the first message queue in the process of switching the index alias to the target rebuilt index;
acquiring the identification of the second index data information, and integrating the identification of the second index data information to obtain first set information; the first set of information comprises all identifiers of the second index data information;
After the index alias is switched to the target reconstructed index, if the identifier of the third index data information updated in the target reconstructed index is judged to be in the first set, acquiring the write lock of the third index data information or the second index data information, and writing the second index data information and the third index data information into the target reconstructed index according to the synchronization process of the write lock of the third index data information or the second index data information.
In a second aspect, an index switching synchronization device is characterized in that: the device comprises:
the first synchronization module is used for reconstructing the index to be reconstructed to obtain an initial reconstructed index when the starting of the index reconstruction process is monitored, and synchronizing the first index data information updated in the index to be reconstructed to the first message queue in the process of obtaining the initial reconstructed index;
the writing module is used for writing the first index data information and the service data information in the first message queue into the initial reconstructed index to obtain a target reconstructed index; the service data information is data in a commodity service system;
The second synchronization module is used for traversing the first message queue, switching an index alias to the target rebuilt index if the first message queue is empty, and continuously synchronizing second index data information updated in the index to be rebuilt to the first message queue in the process of switching the index alias to the target rebuilt index;
the integration module is used for acquiring the identification of the second index data information and integrating the identification of the second index data information to obtain first set information; the first set of information comprises all identifiers of the second index data information;
and the third synchronization module is used for acquiring the write lock of the third index data information or the second index data information if the identification of the third index data information updated in the target reconstructed index is judged to be in the first set after the index alias is switched to the target reconstructed index, and writing the second index data information and the third index data information into the target reconstructed index according to the synchronization process of the write lock of the third index data information or the second index data information.
Therefore, according to the index switching synchronization method, a new message queue is set, when the starting of an index rebuilding process is monitored, updated first index data information in an index to be rebuilt is synchronized into the new message queue, after the new index is rebuilt, the new message queue is traversed to consume the first index data information in the new message queue into the new index, and when all updated data information in the message queue is read, index aliases are switched to the target rebuilt index. Simultaneously, continuing to synchronize the updated second index data information in the index to be rebuilt to the new message queue in the process of switching the alias to the target rebuilt index, namely, not suspending consumption in the message queue, and integrating the identifier of the updated second index data information in the index switching process to obtain first set information; after the index alias is switched to the target reconstructed index, if the identifier of the third index data information updated in the target reconstructed index is judged to be in the first set, acquiring the write lock of the third index data information or the second index data information, and writing the second index data information and the third index data information into the target reconstructed index according to the synchronous process of the write lock of the third index data information or the second index data information. Therefore, when the index is switched after the index is rebuilt, the consumption of the message is not needed to be paused, the index data in the queue is continuously updated, and after the index is finished, all the index data updated in the index switching process are synchronized into the target rebuilt index by controlling the time stamp and the write lock of the index data information, so that the index data information is continuously updated in real time in the index switching process, and the data consistency after the index switching is finished is ensured.
For a clearer understanding of the present invention, specific embodiments of the invention will be set forth in the following description taken in conjunction with the accompanying drawings.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or in the prior art will be briefly described below, it being obvious that the drawings described below are only examples of the present invention, and that other drawings can be obtained according to the drawings provided without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of an index switching synchronization method disclosed in the present application;
FIG. 2 is a flowchart of a specific index switching synchronization method disclosed in the present application;
FIG. 3 is a flowchart of a specific index switching synchronization method disclosed in the present application;
FIG. 4 is a flowchart of a specific index switching synchronization method disclosed in the present application;
FIG. 5 is a flowchart of a specific index switching synchronization method disclosed in the present application;
FIG. 6 is a flowchart of a specific index switching synchronization method disclosed in the present application;
FIG. 7 is a flowchart of a specific index switching synchronization method disclosed in the present application;
Fig. 8 is a schematic structural diagram of a specific index switching synchronization device disclosed in the present application;
fig. 9 is a schematic structural diagram of a computer device disclosed in the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.
It should be understood that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments, and that any other embodiments obtained by one of ordinary skill in the art without making any inventive effort based on the embodiments in the present application are within the scope of the embodiments of the present application.
The terminology used in the embodiments of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the application. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein means that the cake contains any or all possible combinations of one or more of the associated listed items.
When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. In the description of the present application, it should be understood that the terms "first," "second," "third," and the like are used merely to distinguish between similar objects and are not necessarily used to describe a particular order or sequence, nor should they be construed to indicate or imply relative importance. The specific meaning of the terms in this application will be understood by those of ordinary skill in the art as the case may be.
In the ES index, the real-time modification of the index and the structure type of the update index are not supported, so when the structure of the index needs to be modified and updated, a new index needs to be rebuilt and then the service data is reintroduced, and the new index is started to realize the data update because the structure of the index cannot be directly changed on the old index. However, in the process of switching between the new index and the old index after the new index is rebuilt, if continuous index update data is imported from the outside, the partial data will be lost in the index switching process, and synchronization of the update data in the index switching process after the index is rebuilt cannot be ensured. The application provides an index switching synchronization method, which continuously updates index data in the new and old index switching process, stores the identification of the updated index data in a set, and synchronizes the latest index update data to a new index through the write lock of a synchronization thread to the index update data, so that the data consistency after index update can be ensured without suspending the consumption of information in the new and old index switching process.
In one aspect, referring to fig. 1, in order to address the technical problem mentioned in the background art, an embodiment of the present application provides an index switching synchronization method, which is characterized in that: comprising the following steps:
step S1: when the starting of an index rebuilding process is monitored, rebuilding an index to be rebuilt to obtain an initial rebuilt index, and synchronizing first index data information updated in the index to be rebuilt to a first message queue in the process of obtaining the initial rebuilt index;
in this embodiment, in order to ensure consistency between index data of the commodity index system and service data in the commodity service system, the commodity index system generally reconstructs the index in full periodically, and monitors the index reconstruction process in real time, when the commodity index system monitors that the index reconstruction process is started, the original index to be reconstructed is reconstructed to obtain an initial reconstructed index, and it can be understood that the index to be reconstructed is the original old index.
Further, the commodity index system is provided with a second message queue, when index updating data exist, the index updating data are written into the second message queue in real time to be stored, the commodity index system monitors the second message queue in real time, and when the commodity index system monitors that the second message queue is updated, the commodity index system reads the index updating data in the second message queue into the commodity index system to update the index data.
Thus in a specific embodiment, referring to fig. 2, before said step S1, the method further comprises:
step S0: and acquiring index updating data information in real time, and writing the index updating data information into the second message queue. The index in the commodity index system is stored in an alias mode, the index update data information is updated in real time when the update of the second message queue is monitored, the original old index can read the index update data information from the second message queue in real time, and the index update data information in the second message queue is consumed to the old index to realize the real-time update of the old index.
Further, it may be appreciated that, when the index rebuilding process is monitored to start, the commodity index system continues to monitor the data update of the second message queue in real time, and at the same time, the commodity index system may fix or create a first message queue when the index rebuilding process is started, where the commodity index system writes the updated first index data information in the index to be rebuilt after the index rebuilding process is started into the second message queue, and writes the updated first index data information into the first message queue together for synchronous storage.
Thus in a specific embodiment, said step S1 specifically comprises:
step S11: when the starting of the index rebuilding process is monitored, the first index data information updated in the index to be rebuilt in the process of obtaining the initial rebuilt index is written into the second message queue, and simultaneously, the first index data information in the second message queue is synchronously written into the first message queue. The embodiment can use the first message queue to save the first index data information in the index rebuilding process, so that the first message queue which saves all index update data information of the index to be rebuilt in the index rebuilding process can be obtained.
Step S2: writing the first index data information and the service data information in the first message queue into the initial reconstructed index to obtain a target reconstructed index; the service data information is data in a commodity service system;
in this embodiment, referring to fig. 3, the step S2 specifically includes:
step S21: after the initial rebuilt index is obtained, traversing all commodity information in the commodity service system, and writing all service data information in the commodity service system into the initial rebuilt index in batches;
Step S22: writing the first index data information synchronously stored in the first message queue into the initial reconstructed index to obtain a target reconstructed index until the first message queue is empty.
In this embodiment, after the initial rebuilt index is obtained, all service data information in the commodity service system is written into the initial rebuilt index in batches, meanwhile, a consumption process in a first message queue is started, first index data information in the first message queue is consumed from the beginning, all first index data information is read into the initial rebuilt index to obtain a target rebuilt index, so that the target rebuilt index includes all service data information of the commodity service system and all first index data information written into the first message queue in the process of obtaining the target rebuilt index, namely, the initial rebuilt index sequentially reads the first index data information in the first message queue, the first index data information is written into the initial rebuilt index, and when the first index data information in the first message queue is read, namely, the first message queue is empty, an index switching process is started.
Step S3: traversing the first message queue, switching an index alias to the target rebuilt index if the first message queue is empty, and continuously synchronizing second index data information updated in the index to be rebuilt to the first message queue in the process of switching the index alias to the target rebuilt index;
as described above, it can be understood that, when the first index data information updated after the start of the index rebuilding process and stored in the first message queue is written into the initial rebuilt index, the initial rebuilt index traverses the first message queue, and after the first index data information in the first message queue is sequentially read, it is determined whether the first index data information in the first message queue is read completely, if the first message queue is empty due to the completion of the reading, the index switching process is started to switch the index alias to the target rebuilt index.
Further, in this embodiment, after the target rebuilt index is created, a new index and an old index need to be switched, and in the process of switching the index alias from the index to be rebuilt to the target rebuilt index, the commodity index system may continue to monitor the message update of the second message queue, and continue to synchronously write the updated second index data information into the first message queue. It can be understood that, the consumption process of the second message queue and the first message queue is not suspended, the updated second index data information of the consumption process is updated to the second message queue in real time, and the second message queue continues to synchronously write the updated second index data information to the first message queue in real time for storage.
Step S4: acquiring the identification of the second data information, and integrating the identification of the second index data information to obtain first set information; the first set of information comprises all identifiers of the second index data information;
in this embodiment, referring to fig. 4, the step S4 specifically includes:
step S41: integrating the identification of the second index data information according to the timestamp of the second index data information;
step S42: and storing the integrated second index data information in the first set information. The second index data information comprises identifiers of the second index data information, and the first set information comprises identifiers of all the second index data information.
Further, the identification of the second index data information may be primary key information, and in the commodity index system, the primary key information of each index data information is unique, so that different index data information may be identified by the primary key information. And each index data information comprises a time stamp, and updated time information of the index data information can be obtained according to the time stamp of the index data information. Specifically, if there is update of the repeated message in the index switching process, the second index data information with the latest timestamp is integrated into the first set information according to the timestamp of the second index data information, so that the latest updated second index data can be synchronized to the target reconstructed index, and consistency in the data synchronization process is ensured.
Step S5: after the index alias is switched to the target reconstructed index, if the identification of the updated third index data information in the target reconstructed index is judged to be in the first set information, acquiring the write lock of the third index data information or the second index data information, and writing the second index data information and the third index data information into the target reconstructed index according to the synchronization process of the write lock of the third index data information or the second index data information.
In this embodiment, in the process of switching the index alias from the index to be rebuilt to the target rebuilt index, updating of the index data is not suspended, and in order to ensure data consistency of the index to be rebuilt and the target rebuilt index after the rebuild, it is necessary to continuously record the index data updated in the switching process, so that the commodity index system continuously monitors the message update of the second message queue, and simultaneously continuously writes the second index data information updated in the switching process into the first message queue for storage. However, after the switching is completed, the alias of the index is switched to the latest target rebuilt index, so that the latest information update in the second message queue is synchronized to the switched target rebuilt index in real time, and the second index data information updated in the first message queue is saved as not being the latest data, so that in the process of synchronizing the second index data information updated in the first message queue to the target rebuilt index after the completion of the index switching, cooperative control is needed to be performed with the process of consuming the third index data in the second message queue to the target rebuilt index in real time, thereby realizing that the latest index update data is synchronized to the target rebuilt index, and further realizing that the data consistency of the index to be rebuilt and the reconstructed target rebuilt index is ensured on the premise that the information update in the message queue is not suspended in the index switching process.
Further, the process of writing the updated second index data information in the second message queue into the first message queue for synchronization is denoted as a first synchronization thread, that is, the process of writing the second index data information obtained by continuing to update in the second message queue into the first message queue for storage in the index switching process is denoted by the first synchronization thread. And after the switching process is finished, writing the third index data information updated in real time in the second message queue into the target rebuilt index, wherein the process of writing the third index data information updated in real time in the second message queue into the target rebuilt index is marked as a second synchronous thread, namely the second synchronous thread represents the process of writing the third index data information obtained by updating in real time in the second message queue into the target rebuilt index for synchronization after the index switching is finished.
Specifically, after the index alias is switched to the target rebuilt index, the second index data information updated to the first message queue in real time in the index switching process needs to be synchronized to the target rebuilt index to ensure data consistency, that is, in the process of synchronizing the second index data information stored in the first message queue to the target rebuilt index one by one after the index switching is completed, the first synchronization thread needs to be cooperatively controlled with the second synchronization thread to determine the latest index updating data synchronized to the target rebuilt index.
Further, referring to fig. 5, the step S5 specifically includes:
step S51: acquiring the identification of the third index data information, judging whether the identification of the third index data information updated in the target reconstructed index is in the first set information, and if so, determining a synchronous thread for acquiring the write lock of the corresponding third index data information or the second index data information; the synchronous threads comprise the first synchronous thread and the second synchronous thread;
step S52: and if the second synchronous thread acquires the write lock of the third index data information, writing the third index data information into the target reconstructed index after locking the identification of the third index data information.
Step S53: if the first synchronous thread acquires the write lock of the second index data information, comparing the time stamp of the second index data information with the time stamp of the third index data information, and if the time stamp of the second index data information is later than the time stamp of the third index data information, writing the second index data information into the target reconstructed index;
Step S54: and circularly executing the steps until all second index data information in the first message queue is written into the target reconstructed index.
In this embodiment, in the process of synchronizing the second index data information stored in the first message queue to the target rebuilt index one by one after the completion of the index switching, if it is determined that the identifier of the third index data information updated in the target rebuilt index is in the first set, this indicates that the second index data information synchronized to the first message queue after the completion of the index is updated in new real time, so that it is necessary to acquire a write lock for the second index data information or the third index data information to control the cooperation of the first synchronization thread and the second synchronization thread, thereby ensuring that the primary key of the same index update data is not updated concurrently.
Further, if the identifier of the third index data information is in the first set, it indicates that there is real-time update of index data after the completion of the indexing, and if the second synchronization thread first acquires the write lock of the third index data information, the main key of the third index data information is locked, and then the latest updated third index data information is written into the target reconstructed index. If the first synchronization thread acquires the write lock of the second index data, the second index data information in the first synchronization thread and the timestamp of the third index data information need to be compared, if the timestamp of the second index data information is later than the timestamp of the third index data information, the fact that the part of index data stored in the first message queue in the index switching process is not updated after the index switching is completed is indicated, the second index data information is written into the target reconstructed index for synchronization, and the consistency of the index to be reconstructed and the data of the target reconstructed index after the index is ensured.
In another embodiment, with further reference to fig. 5, the step S5 further includes:
step S55: discarding the second index data information if the time stamp of the second index data information is earlier than the time stamp of the third index data information.
In this embodiment, if the timestamp of the second index data information is earlier than the timestamp of the third index data information, it is indicated that the portion of index data stored in the first message queue during the index switching process is updated after the index switching is completed, and the portion of index data information is already updated in real time to the target reconstructed index for synchronization in the third index data information, so that the portion of index data information is directly discarded.
In another embodiment, with further reference to fig. 5, the step S5 further includes:
step S56: and if the identification of the updated third index data information in the target reconstructed index is judged not to be in the first set, writing the third index data information into the target reconstructed index in real time.
The foregoing aspects of the present application are described below in conjunction with a specific embodiment.
Referring to fig. 6, an index switching synchronization method provided in an embodiment of the present application includes:
Step 0: setting a change message queue, and updating to the old index in real time when the update in the change message queue is monitored;
step 1: when the index reestablishing task is monitored, writing the change information updated in real time in the change information queue into the fixed or created temporary storage queue;
step 2: creating a new index, and writing the business data into the new index in batches until all business data are written into the new index;
step 4: starting a synchronization process, and consuming change messages in a temporary storage queue from the beginning to a new index until all the change messages in the temporary storage queue are consumed, wherein the temporary storage queue is empty;
step 5: when confirming that the temporary storage queue is empty, switching the index alias to a new index;
step 6: and (3) continuously executing the steps 0 and 1 when the switching is started, updating the updated information in the change information queue to the old index in real time, and simultaneously writing the updated change information in the change information queue in real time into the temporary storage queue.
Further, referring to fig. 7, the to-be-synchronized change record is the message record written into the temporary storage queue by the change message queue in the index switching process, and it can be understood that the to-be-synchronized change record is the second index data information related in the foregoing scheme; the T thread can be understood as a first synchronous thread involved in the scheme, and the R thread can be understood as a second synchronous thread involved in the scheme; one record in the R thread can be understood to be the third index data information related in the scheme; after the index switching is completed, the switching synchronization method further comprises the following steps:
1: integrating the to-be-synchronized change records by the T thread according to the main key and the time stamp of the to-be-synchronized change records to generate a tidied change record set;
2: recording a primary key set to obtain a K set, monitoring the message update in the change message queue by the R thread in real time, judging whether the change message record is in the K set one by one according to the message record, and if not, updating normally; if the change message records are in the K set, the T thread is also ready to synchronize to a new index one by one according to the change records to be synchronized, so that the R thread and the T thread acquire the write locks of the change message records respectively, and if the R thread acquires the write locks of the change message records, the change message records are updated to the new index after being locked;
3: and traversing the generated set of change records to be synchronized by the T thread, preparing to synchronize to new indexes one by one according to the change records to be synchronized, locking the change message records to be synchronized if the T thread acquires the write locks of the corresponding change records to be synchronized, judging whether the change message records to be synchronized are newer than index data updated in real time, if so, updating, otherwise, discarding the change message records to be synchronized.
Therefore, according to the index switching synchronization method provided by the application, by setting a new message queue, when the starting of an index rebuilding process is monitored, the updated first index data information in the index to be rebuilt is synchronized into the new message queue, after the new index is rebuilt, the new message queue is traversed to consume the first index data information in the new message queue into the new index, and when all the updated data information in the message queue is read, the index alias is switched to the target rebuilt index. Simultaneously, continuing to synchronize the updated second index data information in the index to be rebuilt to the new message queue in the process of switching the alias to the target rebuilt index, namely, not suspending consumption in the message queue, and integrating the identifier of the updated second index data information in the index switching process to obtain first set information; after the index alias is switched to the target reconstructed index, if the identifier of the third index data information updated in the target reconstructed index is judged to be in the first set, acquiring the write lock of the third index data information or the second index data information, and writing the second index data information and the third index data information into the target reconstructed index according to the synchronous process of the write lock of the third index data information or the second index data information. Therefore, according to the index switching synchronization method provided by the application, when index switching is performed after index reconstruction, consumption of a pause message is not needed, index data in a queue is continuously updated, and after the index is completed, synchronization of all index data updated in the index switching process to a target reconstruction index is realized by controlling the time stamp and the write lock of the index data information, so that the index data information is continuously updated in real time in the index switching process, and data consistency after the index switching is completed is ensured.
On the other hand, referring to fig. 8, the present application provides an index switching synchronization device, including:
the first synchronization module 81 is configured to reconstruct the index to be reconstructed to obtain an initial reconstructed index when the start of the index reconstruction process is monitored, and synchronize first index data information updated in the index to be reconstructed to the first message queue in the process of obtaining the initial reconstructed index;
a writing module 82, configured to write the updated index data information and service data information in the first message queue into the initial reconstructed index to obtain a target reconstructed index; the service data information is data in a commodity service system;
a second synchronization module 83, configured to traverse the first message queue, switch an index alias to the target reconstructed index if the first message queue is empty, and continuously synchronize second index data information updated in the index to be reconstructed to the first message queue in a process of switching the index alias to the target reconstructed index;
the integration module 84 is configured to obtain an identifier of the second index data information, and integrate the identifier information of the second index data to obtain first set information; the first set of information comprises all identifiers of the second index data information;
And a third synchronization module 85, configured to, after completing switching the index alias to the target post-reconstruction index, obtain a write lock of the third index data information or the second index data information if it is determined that the identifier of the third index data information updated in the target post-reconstruction index is in the first set, and write the second index data information and the third index data information to the target post-reconstruction index according to a synchronization process of the write lock of the third index data information or the second index data information.
In a specific embodiment, the first synchronization module 81 is specifically configured to: when the starting of the index rebuilding process is monitored, first index data information updated in the index to be rebuilt in the process of obtaining the initial rebuilt index is written into a second message queue, and simultaneously, first index data information in the second message queue is synchronously written into a first message queue.
In a specific embodiment, the writing module 82 includes:
the first writing unit is used for traversing all commodity information in the commodity service system after the initial rebuilt index is obtained, and writing all service data information in the commodity service system into the initial rebuilt index in batches;
And the second writing unit is used for writing the first index data information which is synchronously stored in the first message queue and is updated after the index rebuilding process is started into the initial rebuilt index to obtain a target rebuilt index until the first message queue is empty.
In a specific embodiment, the integration module 84 includes:
the integration unit is used for integrating the identification of the second index data information according to the timestamp of the second index data information;
and the storage unit is used for storing the second index data information obtained through integration in the first set information. The second index data information comprises identifiers of the second index data information, and the first set information comprises identifiers of all the second index data information.
In a specific embodiment, the third synchronization module 85 includes:
the determining unit is used for acquiring the identification of the third index data information, and determining the synchronous thread of the third index data information or the second index data information if the identification of the updated third index data information in the index after target reconstruction is judged to be in the first set; the synchronization thread comprises a first synchronization thread and a second synchronization thread, the first synchronization thread is used for synchronizing the second index data information to the target reconstructed index, and the second synchronization thread is used for synchronizing the third index data information to the target reconstructed index;
And the third writing unit is used for writing the third index data information into the target reconstructed index after locking the identification of the third index data information if the second synchronous thread acquires the write lock of the third index data information.
A fourth writing unit, configured to compare the timestamp of the second index data information with the timestamp of the third index data information if the first synchronization thread obtains the write lock of the second index data information, and write the second index data information into the target reconstructed index if the timestamp of the second index data information is later than the timestamp of the third index data information;
and the circulation unit is used for circularly executing the steps until all the second index data information in the first message queue is written into the target reconstructed index.
In a specific embodiment, the third synchronization module 85 further includes:
and the discarding unit is used for discarding the second index data information if the time stamp of the second index data information is earlier than the time stamp of the third index data information.
In a specific embodiment, the third synchronization module 85 further includes:
And the fifth writing unit is used for writing the third index data information into the target reconstructed index in real time if judging that the identifier of the updated third index data information in the target reconstructed index is not in the first set.
Further, the application also discloses a computer readable storage medium for storing a computer program, wherein the computer program realizes the index switching synchronization method provided by the previous description when being executed by a processor.
Further, the application also discloses a computer device 9 comprising a memory 91 and a processor 92, the memory 91 being used for storing a computer program; the processor 92 is configured to execute the computer program to implement the index switching synchronization method provided above. For specific steps of the method, reference may be made to the corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. For the device disclosed in the examples, since it corresponds to the method disclosed in the examples, the description is relatively simple, and the relevant points are referred to in the description of the method section.
The above examples illustrate only a few embodiments of the invention, which are described in detail and are not to be construed as limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention.
Claims (10)
1. An index switching synchronization method is characterized in that: comprising the following steps: when the starting of an index rebuilding process is monitored, rebuilding an index to be rebuilt to obtain an initial rebuilt index, writing first index data information updated in the index to be rebuilt in the process of obtaining the initial rebuilt index into a second message queue, and synchronously writing first index data information in the second message queue into a first message queue;
writing the first index data information and the service data information in the first message queue into the initial reconstructed index to obtain a target reconstructed index; the service data information is data in a commodity service system;
traversing the first message queue, switching an index alias to the target rebuilt index if the first message queue is empty, and continuously synchronizing second index data information updated in the index to be rebuilt to the first message queue in the process of switching the index alias to the target rebuilt index; the second index data information is information synchronized to the first message queue in the process of switching an index alias to the target reconstructed index;
Acquiring the identification of the second index data information, and integrating the identification of the second index data information to obtain first set information; the first set of information comprises all identifiers of the second index data information;
after switching an index alias to the target reconstructed index, if the identifier of third index data information updated in the target reconstructed index is judged to be in the first set, acquiring write locks of the third index data information or the second index data information, and writing the second index data information and the third index data information into the target reconstructed index according to a synchronization process of acquiring the write locks of the third index data information or the second index data information; the third index data information is information synchronized into the second message queue in the process of switching index aliases to the target reconstructed index.
2. The index switching synchronization method as claimed in claim 1, wherein: before monitoring the start of the index rebuilding process, the method further comprises: and acquiring index updating data information in real time, and writing the index updating data information into a second message queue.
3. The index switching synchronization method as claimed in claim 1, wherein: the writing the first index data information and the service data information in the first message queue into the initial reconstructed index to obtain a target reconstructed index includes:
after the initial rebuilt index is obtained, traversing all commodity information in a commodity service system, and writing all service data information in the commodity service system into the initial rebuilt index in batches;
writing the first index data information synchronously stored in the first message queue into the initial reconstructed index to obtain a target reconstructed index until the first message queue is empty.
4. The index switching synchronization method as claimed in claim 1, wherein: the obtaining the identification of the second index data information, and integrating the identification of the second index data information to obtain the first set information includes:
integrating the identification of the second index data information according to the timestamp of the second index data information;
storing the second index data information obtained by integration in the first set information; wherein the second index data information stored in the first set of information includes an identification of the second index data information.
5. The index switching synchronization method as claimed in claim 4, wherein: after the switching of the index alias to the target reconstructed index is completed, if it is determined that the identifier of the updated third index data information in the target reconstructed index is in the first set, acquiring the write lock of the third index data information or the second index data information, and writing the second index data information and the third index data information to the target reconstructed index according to the synchronization process of the write lock of the third index data information or the second index data information, where the writing of the third index data information and the third index data information to the target reconstructed index includes:
acquiring the identification of the third index data information, and if the identification of the third index data information updated in the index after target reconstruction is judged to be in the first set information, determining a synchronous thread for acquiring the write lock of the corresponding third index data information or the second index data information; the synchronization thread comprises a first synchronization thread and a second synchronization thread, the first synchronization thread is used for synchronizing the second index data information to the target reconstructed index, and the second synchronization thread is used for synchronizing the third index data information to the target reconstructed index;
If the second synchronous thread acquires the write lock of the third index data information, writing the third index data information into the target reconstructed index after locking the identification of the third index data information;
if the first synchronous thread acquires the write lock of the second index data information, comparing the time stamp of the second index data information with the time stamp of the third index data information, and if the time stamp of the second index data information is later than the time stamp of the third index data information, writing the second index data information into the target reconstructed index;
and circularly executing the steps until all second index data information in the first message queue is written into the target reconstructed index.
6. The index switching synchronization method as claimed in claim 5, wherein: the method further comprises the steps of:
discarding the second index data information if the time stamp of the second index data information is earlier than the time stamp of the third index data information.
7. The index switching synchronization method according to any one of claims 5 or 6, characterized by: the method further comprises the steps of:
And if the identification of the updated third index data information in the target reconstructed index is judged not to be in the first set, writing the third index data information into the target reconstructed index in real time.
8. An index switching synchronization device is characterized in that: the device comprises:
the first synchronization module is used for reconstructing the index to be reconstructed to obtain an initial reconstructed index when the starting of the index reconstruction process is monitored, writing first index data information updated in the index to be reconstructed in the process of obtaining the initial reconstructed index into a second message queue, and synchronously writing first index data information in the second message queue into a first message queue;
the writing module is used for writing the first index data information and the service data information in the first message queue into the initial reconstructed index to obtain a target reconstructed index; the service data information is data in a commodity service system;
the second synchronization module is used for traversing the first message queue, switching an index alias to the target rebuilt index if the first message queue is empty, and continuously synchronizing second index data information updated in the index to be rebuilt to the first message queue in the process of switching the index alias to the target rebuilt index; the second index data information is information synchronized to the first message queue in the process of switching an index alias to the target reconstructed index;
The integration module is used for acquiring the identification of the second index data information and integrating the identification of the second index data information to obtain first set information; the first set of information comprises all identifiers of the second index data information;
the third synchronization module is used for acquiring the write lock of the third index data information or the second index data information if the identification of the updated third index data information in the target reconstructed index is judged to be in the first set after the switching of the index alias to the target reconstructed index is completed, and writing the second index data information and the third index data information into the target reconstructed index according to the synchronization process of the write lock of the third index data information or the second index data information; the third index data information is information synchronized into the second message queue in the process of switching index aliases to the target reconstructed index.
9. A computer storage medium, on which a computer program is arranged, which computer program, when being executed by a processor, implements the index switching synchronization method according to any one of claims 1-7.
10. A computer device comprising a memory and a processor, the memory for storing a computer program; the processor is configured to execute the computer program to implement the index switching method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310998608.7A CN117149763B (en) | 2023-08-08 | 2023-08-08 | Index switching synchronization method and device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310998608.7A CN117149763B (en) | 2023-08-08 | 2023-08-08 | Index switching synchronization method and device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117149763A CN117149763A (en) | 2023-12-01 |
| CN117149763B true CN117149763B (en) | 2024-04-02 |
Family
ID=88883315
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310998608.7A Active CN117149763B (en) | 2023-08-08 | 2023-08-08 | Index switching synchronization method and device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117149763B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110609865A (en) * | 2018-05-29 | 2019-12-24 | 优信拍（北京）信息科技有限公司 | Information synchronization method, device and system |
| CN112835980A (en) * | 2021-02-05 | 2021-05-25 | 北京字跳网络技术有限公司 | Index reconstruction method, device, equipment, computer readable storage medium and product |
| CN113672627A (en) * | 2021-09-08 | 2021-11-19 | 湖南惠农科技有限公司 | Elasticissearch search engine index construction method and device |
| CN114020745A (en) * | 2021-11-03 | 2022-02-08 | 山东库睿科技有限公司 | Index construction method and device, electronic equipment and storage medium |
| CN114579596A (en) * | 2022-05-06 | 2022-06-03 | 达而观数据(成都)有限公司 | Method and system for updating index data of search engine in real time |
| CN115098525A (en) * | 2022-07-25 | 2022-09-23 | 湖南快乐阳光互动娱乐传媒有限公司 | Index updating method and device, electronic equipment and storage medium |
| CN115658697A (en) * | 2022-11-14 | 2023-01-31 | 政采云有限公司 | Index reconstruction method, device, equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9996568B2 (en) * | 2013-12-31 | 2018-06-12 | International Business Machines Corporation | Index maintenance based on a comparison of rebuild vs. update |
| US11531645B2 (en) * | 2017-12-15 | 2022-12-20 | Slack Technologies, Llc | Method, apparatus and computer program product for improving data indexing in a group-based communication platform |
-
2023
- 2023-08-08 CN CN202310998608.7A patent/CN117149763B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110609865A (en) * | 2018-05-29 | 2019-12-24 | 优信拍（北京）信息科技有限公司 | Information synchronization method, device and system |
| CN112835980A (en) * | 2021-02-05 | 2021-05-25 | 北京字跳网络技术有限公司 | Index reconstruction method, device, equipment, computer readable storage medium and product |
| CN113672627A (en) * | 2021-09-08 | 2021-11-19 | 湖南惠农科技有限公司 | Elasticissearch search engine index construction method and device |
| CN114020745A (en) * | 2021-11-03 | 2022-02-08 | 山东库睿科技有限公司 | Index construction method and device, electronic equipment and storage medium |
| CN114579596A (en) * | 2022-05-06 | 2022-06-03 | 达而观数据(成都)有限公司 | Method and system for updating index data of search engine in real time |
| CN115098525A (en) * | 2022-07-25 | 2022-09-23 | 湖南快乐阳光互动娱乐传媒有限公司 | Index updating method and device, electronic equipment and storage medium |
| CN115658697A (en) * | 2022-11-14 | 2023-01-31 | 政采云有限公司 | Index reconstruction method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117149763A (en) | 2023-12-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107038162B (en) | Real-time data query method and system based on database log | |
| CN106878376B (en) | Configuration management method and system | |
| US10747745B2 (en) | Transaction execution commitment without updating of data row transaction status | |
| EP2590087A1 (en) | Database log parallelization | |
| US20070276884A1 (en) | Method and apparatus for managing backup data and journal | |
| CN109992628B (en) | Data synchronization method, device, server and computer readable storage medium | |
| CN109766194B (en) | Method and system for realizing low-coupling plan task component based on message | |
| CN109933632B (en) | Data migration method, device and equipment for database | |
| CN112612799B (en) | Data synchronization method and terminal | |
| US20030055805A1 (en) | Data processing | |
| CN111581219B (en) | Method and device for acquiring monitoring data | |
| CN110457157A (en) | Distributed transaction abnormality eliminating method, device, computer equipment and storage medium | |
| CN117149763B (en) | Index switching synchronization method and device and storage medium | |
| CN113900788A (en) | Distributed work scheduling method and distributed workflow engine system | |
| CN111737331B (en) | Transaction consistency processing method and system for database and object storage | |
| CN109725964A (en) | Skin change method, device, the electronic equipment of mixed type frame | |
| CN115981921A (en) | Backup method | |
| CN113672167B (en) | Data consistency verification method, device and equipment for distributed storage system | |
| CN111737348B (en) | Aging synchronization method and device based on database table | |
| CN110297673B (en) | Method and storage medium for optimizing loading of memory data | |
| CN111444281B (en) | Database parameter synchronization method and system | |
| CN113986939A (en) | Data change method and device for database | |
| CN109144788B (en) | Method, device and system for reconstructing OSD | |
| CN113420088B (en) | Method, device and terminal for synchronizing cache database and database | |
| CN114327283B (en) | Data migration method in distributed storage system and distributed storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |