CN112507187B - Index changing method and device - Google Patents
Index changing method and device Download PDFInfo
- Publication number
- CN112507187B CN112507187B CN202011257710.4A CN202011257710A CN112507187B CN 112507187 B CN112507187 B CN 112507187B CN 202011257710 A CN202011257710 A CN 202011257710A CN 112507187 B CN112507187 B CN 112507187B
- Authority
- CN
- China
- Prior art keywords
- index
- consumer group
- data
- alias
- reindex
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Abstract
The embodiment of the invention provides an index changing method and device, wherein the method comprises the following steps: according to the index change requirement, a second index corresponding to an original first index is created, a first consumer group which is in a first subscription relation with a preset topic is created for the first index, a second consumer group which is used for submitting consumption offset acquired from the first consumer group is created for the second index, the second consumer group consumes data according to the consumption offset, meanwhile, Reindex processing is adopted to copy the data of the first index to the second index, a first alias is switched to the second index, the second index replaces the first index, and data index service is provided through the second consumer group. The embodiment of the invention avoids double writing of the incremental data in the index changing process, reduces the pressure of the ES cluster, and also avoids the conflict between the incremental data and the migration data.
Description
Technical Field
The invention relates to the technical field of internet, in particular to an index changing method and device.
Background
A search server elastic search (hereinafter referred to as 'ES') based on Lucene is an enterprise-level search engine, is widely applied to the field of big data, can achieve real-time search, and is stable, reliable, rapid, and convenient to install and use.
However, the ES does not support operations of field deletion, field type change, participle type change, and whether the change can be retrieved on the original index mapping structure after the ES creates the index. Therefore, in the prior art, the ES provides a reindex function to support migration of original index data into a changed index, but when the ES index is changed, incremental data write traffic needs to be written into the original index and a new index after the change, so that ES cluster pressure is caused, and the incremental data write of the original index and the original index migration data conflict.
Therefore, how to provide a method that can reduce the ES cluster pressure during index change and avoid the conflict between incremental data and migration data is a problem to be solved.
Disclosure of Invention
To solve the problems in the prior art, embodiments of the present invention provide an index changing method and apparatus.
The embodiment of the invention provides an index changing method, which comprises the following steps:
creating a second index corresponding to the original first index according to the index change requirement; wherein the first index has a first alias;
creating a first consumer group for the first index, wherein the first consumer group and a preset topic establish a first subscription relationship in advance, and the first consumer group is used for consuming data in the preset topic;
creating a second consumer group for the second index, wherein a second subscription relationship is pre-established between the second consumer group and the preset topic, and the second consumer group is used for submitting the consumption offset acquired from the first consumer group and consuming the data in the preset topic according to the consumption offset; wherein the second consumer group shares the same consumption progress variable with the first consumer group;
copying the data of the first index to the second index by using Reindex;
and switching the first alias to the second index, and establishing a corresponding relation between the second index and the first alias so as to enable the second index to replace the first index and provide data index service through the second consumer group.
According to an embodiment of the present invention, after switching the first alias to the second index, the index changing method further includes:
judging whether the second index replaces the first index to provide data index service, if so, recovering the first index; if not, switching the first alias to the first index, establishing a corresponding relation between the first index and the first alias, so that the first index replaces the second index, and providing the data index service through the first consumer group.
According to an embodiment of the index changing method, before switching the first alias to the second index, the method further includes:
polling the task ID of the Reindex at fixed time, calling a task API in an ES (application program interface) and acquiring the task state of the Reindex;
and switching the first alias to the second index according to the task state of the Reindex.
According to an embodiment of the present invention, the method for changing an index, in which the switching the first alias to the second index according to the task status of the Reindex includes:
if the task state of the Reindex is finished, switching the first alias to the second index;
if the task state of the Reindex is in progress, continuously polling the task ID of the Reindex;
and if the task state of the Reindex indicates that an error is caused, performing alarm prompt.
According to the index changing method of the embodiment of the invention, the second consumer group obtains the consumption offset of the first consumer group by calling a partitionfor function, and submits the consumption offset by a committed mode.
An embodiment of the present invention further provides an index changing apparatus, including:
the first creating unit is used for creating a second index corresponding to the original first index according to the index change requirement; wherein the first index has a first alias;
a second creating unit, configured to create a first consumer group for the first index, where the first consumer group establishes a first subscription relationship with a preset topic in advance, and the first consumer group is used to consume data in the preset topic;
a third creating unit, configured to create a second consumer group for the second index, where the second consumer group establishes a second subscription relationship with the preset topic in advance, and the second consumer group is configured to submit a consumption offset obtained from the first consumer group, and consume the data in the preset topic according to the consumption offset;
a data migration unit, configured to copy the data of the first index to the second index by using a Reindex;
and the index changing unit is used for switching the first alias to the second index, establishing the corresponding relation between the second index and the first alias, enabling the second index to replace the first index, and providing data index service through the second consumer group.
According to the index changing apparatus of an embodiment of the present invention, the index changing unit is further configured to determine whether the second index provides a data index service instead of the first index after switching the first alias to the second index, and if so, recover the first index; if not, switching the first alias to the first index, establishing a corresponding relation between the first index and the first alias, so that the first index replaces the second index, and providing the data index service through the first consumer group.
According to the index changing apparatus of an embodiment of the present invention, before switching the first alias to the second index, the index changing unit is further configured to:
polling the task ID of the Reindex at fixed time, calling a task API in an ES (application program interface) and acquiring the task state of the Reindex;
and switching the first alias to the second index according to the task state of the Reindex.
According to an embodiment of the present invention, the index changing unit is configured to switch the first alias to the second index according to a task state of the Reindex, and includes:
if the task state of the Reindex is finished, switching the first alias to the second index;
if the task state of the Reindex is in progress, continuously polling the task ID of the Reindex;
and if the task state of the Reindex is that an error is prompted, performing alarm prompting.
According to the index changing apparatus of an embodiment of the present invention, the second consumer group obtains the consumption offset of the first consumer group by calling a partitionfor function, and submits the consumption offset by committed.
An embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements any of the steps of the index changing method when executing the program.
Embodiments of the present invention further provide a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of any of the index changing methods described above.
According to the index changing method and device provided by the embodiment of the invention, the second index corresponding to the original first index is created according to the index changing requirement, the first consumer group which is in the first subscription relation with the preset topic is created for the first index, and the second consumer group which is used for submitting the consumption offset acquired from the first consumer group is created for the second index, so that the second consumer group consumes data according to the consumption offset, double writing of incremental data in the index changing process is avoided, the ES cluster pressure is reduced, and the conflict between the incremental data and the migration data caused by the double writing of the data is also avoided; and meanwhile, the data of the first index is copied to a second index by adopting Reindex processing, and the first alias is switched to the second index so as to enable the second index to replace the first index and provide data index service through a second consumer group, thereby completing index change.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of an index changing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of consuming indicia provided by an embodiment of the present invention;
FIG. 3 is a flow chart of index switching according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an index changing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
The index changing method according to the embodiment of the present invention is described below with reference to fig. 1 to 3.
Fig. 1 is a schematic flow chart of an index changing method according to an embodiment of the present invention, and as shown in fig. 1, an index changing method according to an embodiment of the present invention includes:
In particular, ES is a document oriented search engine, and the index defines the logical storage and field types of the documents. However, when the original index needs to be subjected to field deletion, field type change and word segmentation type change, the original index needs to be changed.
Fig. 2 is a flowchart of a consumption marking process according to an embodiment of the present invention, and as shown in fig. 2, according to a requirement of index change, the embodiment of the present invention creates a second index indeb corresponding to an original first index indeca, and at the same time, a first alias is pointed to the first index indea, which indicates that data indexing service is provided by the first index indeca at this time.
And 120, creating a first consumer group for the first index, wherein a first subscription relationship is pre-established between the first consumer group and the preset topic, and the first consumer group is used for consuming data in the preset topic.
In this step, it should be noted that kafka is a distributed publish-subscribe messaging system. The application scene mainly comprises: online applications (messages) and offline applications (data files, logs). That is, kafka is a distributed, partitionable, redundant backup message middleware service that is primarily used to process active streaming data. the topic can be called as a topic, the topic is a category or a subscription name of a message under kafka, kafka is published through topic when publishing the message, that is, topic is a basic unit of kafka data writing or reading operation, it should be noted that each message belongs to and only belongs to one topic, a message Producer must specify to which topic the message is published when publishing the data, and similarly, a Consumer Group must specify information of which topic is subscribed (consumed) when subscribing to the message (also called consuming the message). The Consumer Group Consumer Group is the scalable and fault tolerant Consumer mechanism provided by kafka. There may be multiple consumers in the same consumer group, all consumers in the group coordinating together to consume the subscription topic, and for the same topic, each consumer group may take the same full data.
In this embodiment, a first Consumer Group is created for the first index, and the Consumer Group, i.e. Consumer Group, is the scalable and fault-tolerant Consumer mechanism provided by kafka. Within the same consumer group are multiple consumers and consumer entities that share a common ID. As shown in fig. 2, a first consumer group created for the first index indecea and having a first subscription relationship with a preset topic is established, such that group a consumes topic data and indexes the data to the first index indecea. Meanwhile, when the data in the preset topic is consumed by the first consumer group, the consumption offset at the time is recorded, if a consumer in the group A crashes or a new consumer joins the group A, rebalancing is triggered, and after the rebalancing is completed, each consumer in the group A may be allocated to a new partition instead of the one processed before. In order to continue the previous work, the consumer needs to continue consuming data from the designated place according to the consumption offset.
And step 130, creating a second consumer group for the second index, wherein a second subscription relationship is pre-established between the second consumer group and the preset topic, and the second consumer group is used for submitting the consumption offset acquired from the first consumer group and consuming the data in the preset topic according to the consumption offset.
In this step, it should be noted that the concept of consumer group is: when a plurality of application programs need to acquire messages from kafka, each application program corresponds to a consumer group, so that each application program can acquire all messages of one or more topics; in each consumer group, consumers are added to the consumer group to scale reading and processing capabilities, each consumer in the consumer group only processes a portion of the messages of each topic, and each consumer corresponds to a thread.
Specifically, as shown in fig. 2, a second consumer group groupB is created for the second index indeb, the groupB also establishes a subscription relationship with a preset topic, the groupB submits the consumption offset of the groupA after acquiring the consumption offset of the groupA, at this time, the groupB can consume the topic data according to the consumption offset, when the first index indea data is migrated to the second index indeb, the groupB can consume the data according to the consumption offset of the groupA, and it is not necessary that incremental data in the first index indea needs to be written into the first index indea and the second index indeb in a double writing manner during data migration in the conventional method, which results in a double writing flow in the entire process of the ES cluster. Meanwhile, in the conventional method, when the incremental data is written into the second index indeceb, the migration data after the index is changed is overwritten, and the overwritten data is not the required index change data, so that data collision is caused.
It should be noted that, for multiple consumer groups that have a subscription relationship with the same topic, all the same data can be obtained. Therefore, for the embodiment, the subscription relationship is established between the second consumer group and the first consumer group and the preset topic, and the second consumer group submits the consumption offset acquired from the first consumer group, so that the second consumer group can consume the data identical to the first consumer group according to the consumption offset.
And 140, copying the data of the first index to the second index by adopting the Reindex.
Specifically, the first index data is migrated to the second index by using a Reindex API provided by the ES, wherein the Reindex can specify a field to be migrated and operation metadata, so as to realize an index field change function, and a parameter wait _ for _ completion is selectable during migration to obtain a Reindex task ID and store the ID in a database. Therefore, in the step, an index is newly created, and then the data of the first index is copied to the second index by using a mode of leading the original index data in the Reindex.
The method and the device can realize the operations of field deletion, field type change, participle type change and whether the index change can be searched or not in the index mapping based on the Reindex function of the ES, thereby migrating the original index data to a new index.
Therefore, in the embodiment, based on the kafka message middleware, an ES data index channel is constructed, and version control of ES data indexes is realized by using the kafka sequential consumption characteristic. As can be seen, the present embodiment achieves the following effects: A. the method can support operations of field deletion, field type change, participle type change and whether the change can be searched or not for index mapping based on the Reindex function of the ES; B. through the consumption marking technology, double writing of data indexes in the index field changing process is avoided, and the pressure of an ES cluster is reduced; meanwhile, the problem of conflict between incremental data and migration data is avoided. In addition, in other embodiments, the index changing method may further achieve the following effects: c, when the index change operation needs to be rolled back, the index change operation can be rolled back quickly, and incremental data in the operation process cannot be lost; D. through the task state query function of the ES, the full-flow automation of index mapping change is realized, meanwhile, the manual intervention processing can be performed on the alarm of the task failure, and the whole flow is automatic, safe and efficient. The description of effects C and D is explained in detail in the following embodiments, which are not described in much detail in the present embodiment.
Fig. 3 is an index switching flowchart provided by an embodiment of the present invention, and as shown in fig. 3, after the data of the first index indecea is copied to the second index indecb, the first alias is switched to the second index indecb, and at this time, the second index indecb is responsible for providing data index services, and the second consumer group groupB consumes data. If the data indexing service provided by the second index indeceB is normal, the first index indeceA is recovered, the first consumer group A stops consuming data, and full-process automation operation is performed to realize index change.
According to the embodiment of the invention, the consumption offset acquired from the first consumer group is submitted by the second consumer group through the kafka consumption marking technology, so that the second consumer group can avoid data double writing in the index field changing process according to the consumption offset and the data same as the first consumer group, the ES cluster pressure is reduced, the conflict between incremental data and migration data is avoided, and the cost is low.
According to the index changing method provided by the embodiment of the invention, the second index corresponding to the original first index is created according to the index changing requirement, the first consumer group which establishes the first subscription relation with the preset topic is created for the first index, the second consumer group which is used for submitting the consumption offset obtained from the first consumer group is created for the second index, the second consumer group consumes data according to the consumption offset, then the data of the first index is copied to the second index by adopting Reindex processing, the first alias is switched to the second index, the second index replaces the first index, and the data index service is provided through the second consumer group, so that the index changing is completed. Therefore, the embodiment of the invention constructs an ES data index channel based on kafka message middleware, and realizes the version control of the ES data index by utilizing kafka sequential consumption characteristics; based on a grouping consumption principle of kafka and a basic function of submitting consumption offset, when an index is changed, a consumption position corresponding to topic with an original index name as a consumption group is obtained, then a new index name is used as the consumption group to mark the consumption position, then the data in the original index is transferred to the new index by utilizing a Reindex function of an ES, the new consumption group starts to consume after the transfer is completed, and the change process of smoothing and no perception of the whole index field is completed.
Based on the content of the foregoing embodiment, as an optional embodiment, after the first alias is switched to the second index, it is determined whether the second index provides data index service instead of the first index, and if yes, the first index is recovered; if not, the first alias is switched to the first index, and the corresponding relation between the first index and the first alias is established, so that the first index replaces the second index, and data index service is provided through the first consumer group.
Specifically, after the first alias is switched to the second index, if it is determined that the second index provides data index service instead of the first index, the second index can provide data index service normally, and the index field is changed completely, so that the first index is recovered. If the second index is judged not to replace the first index to provide the data index service, it is indicated that a problem occurs in the process of replacing the first index with the second index to provide the data index service, for example, the second index cannot normally provide the data index service or the second consumer group cannot normally consume data, the first alias is switched to the first index, so that the corresponding relation between the first index and the first alias is established, the first consumer group normally consumes the data again, the process rollback is completed, and the first consumer group consumes the data according to the consumption offset, so that any incremental data cannot be lost.
According to the index changing method provided by the embodiment of the invention, when a problem occurs in the process of providing the data index service by replacing the first index with the second index, the first alias is switched to the first index, so that the second index is replaced with the first index, the data index service is provided, the process rollback is completed, any incremental data cannot be lost, and the index change is completed safely and efficiently.
Based on the content of the above embodiment, as an optional embodiment, the task ID of the Reindex is polled at regular time, and a task API in the ES is called to obtain the task state of the Reindex;
and switching the first alias to the second index according to the task state of the Reindex.
Specifically, an Application Programming Interface (API) provided by the ES is called, and the task polls the Reindex task ID stored in the database at regular time to obtain the task state of the Reindex. And judging whether to switch the first alias to the second index according to the task state of the Reindex.
Based on the content of the foregoing embodiment, as an optional embodiment, switching the first alias to the second index according to the task state of the Reindex includes:
if the task state of the Reindex is finished, switching the first alias to the second index;
if the task state of the Reindex is in progress, continuously polling the task ID of the Reindex;
and if the task state of the Reindex is that the prompt is wrong, performing alarm prompt.
Specifically, a task API provided by the ES is called, and the regular task polls the Reindex task ID stored in the database to acquire the Reindex task state. If the task state is still in normal operation, continuously polling; if the task state return is completed, switching the first alias to the second index; if the task state returns an error, a short message, a WeChat or a telephone alarm is carried out, and manual intervention processing is carried out. At this time, the index changing process is terminated until the manual processing finishes the error information, and then the process is restarted. The embodiment of the invention realizes the full flow automation, smoothness, low cost and no service perception of the index mapping change of the ES of the service level through the task state query function of the ES, and can quickly rollback without losing incremental data in the operation process when the index change operation needs rollback.
According to the index changing method provided by the embodiment of the invention, whether the first alias is switched to the second index is judged according to the task state of the Reindex, so that the second index is responsible for providing data index service after the first index data is transferred to the second index, and the index changing is accurately finished.
Based on the content of the above embodiment, as an optional embodiment, the second consumer group obtains the consumption offset of the first consumer group by calling a partitionfor function, and submits the consumption offset by committed.
Specifically, after the second Consumer group is created for the second index, the second Consumer group obtains the consumption offset of the first Consumer group by calling partitionfor in the kafka Consumer, and submits the consumption offset in a committed manner, and the second Consumer group can obtain the consumption progress of the first Consumer group, so that the second Consumer group can continue to consume at the specified position, double writing of incremental data to the first index and the second index in the conventional method is avoided, the ES cluster pressure is reduced, and data collision is avoided.
According to the index changing method provided by the embodiment of the invention, after the second consumer group is created for the second index, the second consumer group acquires the consumption offset of the first consumer group and submits the consumption offset, so that the second consumer group can consume according to the consumption offset, double writing and data collision of incremental data are avoided, and the ES cluster pressure is reduced.
In the following, the index changing apparatus provided in the embodiment of the present invention is described, and the index changing apparatus described below and the index changing method described above may be referred to correspondingly.
Fig. 4 is a schematic structural diagram of an index changing apparatus according to an embodiment of the present invention, and as shown in fig. 4, the index changing apparatus according to the embodiment of the present invention includes:
a first creating unit 410, configured to create a second index corresponding to the original first index according to an index change requirement; wherein the first index has a first alias.
Specifically, the first creating unit 410 creates a second index corresponding to the original first index according to the requirement of index change, and the first alias points to the first index, which indicates that the first index provides data index service at this time.
The second creating unit 420 is configured to create a first consumer group for the first index, where the first consumer group establishes a first subscription relationship with the preset topic in advance, and the first consumer group is used for consuming data in the preset topic.
Specifically, the second creating unit 420 creates a first consumer group for the first index and establishes a first subscription relationship with the preset topic, so that the first consumer group consumes the topic data and indexes the data to the first index. Meanwhile, when the first consumer group consumes the data in the preset topic, the consumption offset at the time is recorded, if the consumers in the first consumer group crash or a new consumer joins the first consumer group, rebalancing is triggered, and after the rebalancing is completed, each consumer in the first consumer group may be assigned to a new partition instead of the previously processed one. In order to be able to continue with the previous work, the consumer needs to continue consuming data from the specified place according to the consumption offset.
And a third creating unit 430, configured to create a second consumer group for the second index, where the second consumer group establishes a second subscription relationship with the preset topic in advance, and the second consumer group is configured to submit the consumption offset obtained from the first consumer group, and consume the data in the preset topic according to the consumption offset.
Specifically, the third creating unit 430 creates a second consumer group for the second index, the second consumer group establishes a second subscription relationship with the preset topic, and meanwhile, the second consumer group submits the consumption offset of the first consumer group after acquiring the consumption offset of the first consumer group, and when the first index data is migrated to the second index, the second consumer group may consume the data according to the consumption offset of the first consumer group.
And a data migration unit 440, configured to copy the data of the first index to the second index by using the Reindex.
Specifically, the data migration unit 440 migrates the first index data to the second index using a Reindex API provided by the ES, where Reindex can specify fields to be migrated and operation metadata, implement an index field change function, select a parameter wait _ for _ completion ═ false at the time of migration, obtain a Reindex task ID, and store the ID in the database.
The index changing unit 450 is configured to switch the first alias to the second index, and establish a corresponding relationship between the second index and the first alias, so that the second index replaces the first index, and data indexing service is provided through the second consumer group.
Specifically, after the data of the first index is copied to the second index, the index changing unit 450 switches the first alias to the second index, and at this time, the second index is responsible for providing the data index service, and the second consumer group consumes the data. If the data index service provided by the second index is normal, the first index is recovered, the first consumer group stops consuming the data, and the full-process automation operation realizes index change.
According to the index changing device provided by the embodiment of the invention, the second index corresponding to the original first index is created according to the index changing requirement, meanwhile, the first consumer group which is preset to establish a first subscription relation with the preset topic is created for the first index, and the second consumer group which is used for submitting the consumption offset acquired from the first consumer group is created for the second index, so that the second consumer group consumes data according to the consumption offset, double writing of incremental data in the index changing process is avoided, the ES cluster pressure is reduced, and the conflict between the incremental data and the migration data caused by the double writing of the data is also avoided; and meanwhile, the data of the first index is copied to a second index by adopting Reindex processing, and the first alias is switched to the second index so as to enable the second index to replace the first index and provide data index service through a second consumer group, thereby completing index change.
Based on the content of the foregoing embodiment, as an optional embodiment, the index changing unit 450 is further configured to, after switching the first alias to the second index, determine whether the second index provides a data index service instead of the first index, and if so, recycle the first index; if not, the first alias is switched to the first index, and the corresponding relation between the first index and the first alias is established, so that the first index replaces the second index, and data index service is provided through the first consumer group.
Specifically, if the second index replaces the first index to provide the data index service, indicating that the second index can normally provide the data index service, the first index is recovered; if a problem occurs in the process of providing the data index service by replacing the first index with the second index, for example, the second index cannot normally provide the data index service or the second consumer group cannot normally consume data, the second changing unit switches the first alias to the first index, so that the corresponding relation between the first index and the first alias is established, the first consumer group normally consumes the data again, the process rollback is completed, and the first consumer group consumes the data according to the consumption offset, so that any incremental data cannot be lost.
According to the index changing device provided by the embodiment of the invention, when a problem occurs in the process of providing the data index service by replacing the first index with the second index, the first alias is switched to the first index, so that the second index is replaced with the first index, the data index service is provided, the process rollback is completed, any incremental data cannot be lost, and the index change is completed safely and efficiently.
Based on the content of the foregoing embodiment, as an optional embodiment, before switching the first alias to the second index, the index changing unit 450 is further configured to:
polling the task ID of the Reindex at regular time, calling a task API in the ES, and acquiring the task state of the Reindex;
and switching the first alias to the second index according to the task state of the Reindex.
Based on the content of the foregoing embodiment, as an optional embodiment, the index changing unit 450 is configured to switch the first alias to the second index according to the task status of the Reindex, and includes:
if the task state of the Reindex is finished, switching the first alias to the second index;
if the task state of the Reindex is in progress, continuously polling the task ID of the Reindex;
and if the task state of the Reindex is that the prompt is wrong, performing alarm prompt.
Specifically, the second changing unit calls a task API provided by the ES, and the timed task polls the Reindex task ID stored in the database to acquire a Reindex task state. If the task state is still in normal operation, continuously polling; if the task state return is completed, switching the first alias to the second index; if the task state returns errors, short messages, WeChat or telephone alarms are carried out, and manual intervention processing is carried out. At this time, the index changing process is terminated until the manual processing finishes the error information, and then the process is restarted.
The index changing device provided by the embodiment of the invention judges whether to switch the first alias to the second index according to the task state of the Reindex, so that the second index is responsible for providing data index service after the first index data is transferred to the second index, and the index changing is accurately finished.
Based on the content of the above embodiment, as an optional embodiment, after the third creating unit 430 creates the second consumer group for the second index, the second consumer group obtains the consumption offset of the first consumer group by calling the partitionfor function, and submits the consumption offset by committed.
Specifically, after creating the second Consumer group for the second index, the third creating unit 430 may obtain the consumption offset of the first Consumer group by calling partitionfor in the kafka Consumer, and submit the consumption offset in a committed manner, and the second Consumer group may obtain the consumption progress of the first Consumer group, so that the second Consumer group may continue to consume at the specified position, thereby avoiding double-writing the incremental data to the first index and the second index as in the conventional method, reducing the ES cluster pressure, and avoiding data collision.
According to the index changing device provided by the embodiment of the invention, after the second consumer group is created for the second index, the second consumer group obtains the offset of the first consumer group and submits the consumption offset, so that the second consumer group can consume according to the consumption offset, double writing of incremental data and data collision are avoided, and the ES cluster pressure is reduced.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 5, the electronic device may include: a processor (processor)510, a communication Interface (Communications Interface)520, a memory (memory)530 and a communication bus 540, wherein the processor 510, the communication Interface 520 and the memory 530 communicate with each other via the communication bus 540. Processor 510 may call logic instructions in memory 530 to perform the index change method in the above-described embodiments, for example, to: according to the index change requirement, a second index corresponding to the original first index is created; wherein the first index has a first alias; creating a first consumer group for the first index, wherein a first subscription relation is pre-established between the first consumer group and a preset topic, and the first consumer group is used for consuming data in the preset topic; creating a second consumer group for the second index, wherein a second subscription relationship is pre-established between the second consumer group and the preset topic, and the second consumer group is used for submitting the consumption offset acquired from the first consumer group and consuming the data in the preset topic according to the consumption offset; copying the data of the first index to a second index by adopting Reindex processing; and switching the first alias to the second index, and establishing a corresponding relation between the second index and the first alias so as to enable the second index to replace the first index and provide data index service through a second consumer group.
Furthermore, the logic instructions in the memory 530 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, an embodiment of the present invention further provides a computer program product, where the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is capable of executing the index changing method in the foregoing embodiment, for example, performing the following operations: creating a second index corresponding to the original first index according to the index change requirement; wherein the first index has a first alias; creating a first consumer group for the first index, wherein a first subscription relation is pre-established between the first consumer group and a preset topic, and the first consumer group is used for consuming data in the preset topic; creating a second consumer group for the second index, wherein a second subscription relationship is pre-established between the second consumer group and the preset topic, and the second consumer group is used for submitting the consumption offset acquired from the first consumer group and consuming the data in the preset topic according to the consumption offset; copying the data of the first index to a second index by adopting Reindex processing; and switching the first alias to the second index, and establishing a corresponding relation between the second index and the first alias so as to enable the second index to replace the first index and provide data index service through a second consumer group.
In still another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to perform the index changing method in the foregoing embodiment when executed by a processor, for example, to perform the following operations: creating a second index corresponding to the original first index according to the index change requirement; wherein the first index has a first alias; creating a first consumer group for the first index, wherein a first subscription relation is pre-established between the first consumer group and a preset topic, and the first consumer group is used for consuming data in the preset topic; creating a second consumer group for the second index, wherein a second subscription relationship is pre-established between the second consumer group and the preset topic, and the second consumer group is used for submitting the consumption offset acquired from the first consumer group and consuming the data in the preset topic according to the consumption offset; copying the data of the first index to a second index by adopting Reindex processing; and switching the first alias to the second index, and establishing a corresponding relation between the second index and the first alias so as to enable the second index to replace the first index and provide data index service through a second consumer group.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on the understanding, the above technical solutions substantially or otherwise contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the various embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, and not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. An index changing method, comprising:
creating a second index corresponding to the original first index according to the index change requirement; wherein the first index has a first alias;
creating a first consumer group for the first index, wherein the first consumer group and a preset topic establish a first subscription relationship in advance, and the first consumer group is used for consuming data in the preset topic;
creating a second consumer group for the second index, wherein a second subscription relationship is pre-established between the second consumer group and the preset topic, and the second consumer group is used for submitting consumption offset acquired from the first consumer group and consuming data in the preset topic according to the consumption offset;
copying the data of the first index to the second index by using Reindex;
switching the first alias to the second index, establishing a corresponding relation between the second index and the first alias, so that the second index replaces the first index, and providing data index service through the second consumer group;
prior to switching the first alias to the second index, the method further comprises:
polling the task ID of the Reindex at fixed time, calling a task API in an ES (application program interface) and acquiring the task state of the Reindex;
switching the first alias to the second index according to the task state of the Reindex;
the switching the first alias to the second index according to the task state of the Reindex includes:
if the task state of the Reindex is finished, switching the first alias to the second index;
if the task state of the Reindex is in progress, continuously polling the task ID of the Reindex;
and if the task state of the Reindex is that an error is prompted, performing alarm prompting.
2. The index changing method according to claim 1, further comprising, after switching the first alias to the second index:
judging whether the second index replaces the first index to provide the data index service or not, and if so, recovering the first index; if not, switching the first alias to the first index, establishing a corresponding relation between the first index and the first alias, so that the first index replaces the second index, and providing the data index service through the first consumer group.
3. The index change method of claim 1, wherein the second consumer group obtains the consumption offset of the first consumer group by calling a partitionfor function, and submits the consumption offset by committted.
4. An index changing apparatus, comprising:
the first creating unit is used for creating a second index corresponding to the original first index according to the index change requirement; wherein the first index has a first alias;
a second creating unit, configured to create a first consumer group for the first index, where the first consumer group establishes a first subscription relationship with a preset topic in advance, and the first consumer group is used to consume data in the preset topic;
a third creating unit, configured to create a second consumer group for the second index, where the second consumer group establishes a second subscription relationship with the preset topic in advance, and the second consumer group is configured to submit the consumption offset obtained from the first consumer group, and consume the data in the preset topic according to the consumption offset;
a data migration unit, configured to copy the data of the first index to the second index by using a Reindex;
an index changing unit, configured to switch the first alias to the second index, establish a corresponding relationship between the second index and the first alias, so that the second index replaces the first index, and provide a data index service through the second consumer group;
the index changing unit is further configured to, before switching the first alias to the second index:
polling the task ID of the Reindex at fixed time, calling a task API in an ES (application program interface) and acquiring the task state of the Reindex;
switching the first alias to the second index according to the task state of the Reindex;
the index changing unit is configured to switch the first alias to the second index according to a task state of the Reindex, and includes:
if the task state of the Reindex is finished, switching the first alias to the second index;
if the task state of the Reindex is in progress, continuously polling the task ID of the Reindex;
and if the task state of the Reindex is that an error is prompted, performing alarm prompting.
5. The index changing apparatus according to claim 4, wherein the index changing unit is further configured to determine whether the second index provides the data index service in place of the first index after switching the first alias to the second index, and if so, to recover the first index; if not, switching the first alias to the first index, and establishing a corresponding relation between the first index and the first alias so as to enable the first index to replace the second index, and providing the data index service through the first consumer group.
6. The index changing apparatus as claimed in claim 4, wherein the second consumer group obtains the consumption offset of the first consumer group by calling partitionfor function, and submits the consumption offset by commimited manner.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the index changing method as claimed in any one of claims 1 to 3 when executing the program.
8. A non-transitory computer readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the index changing method according to any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011257710.4A CN112507187B (en) | 2020-11-11 | 2020-11-11 | Index changing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011257710.4A CN112507187B (en) | 2020-11-11 | 2020-11-11 | Index changing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112507187A CN112507187A (en) | 2021-03-16 |
| CN112507187B true CN112507187B (en) | 2022-09-27 |
Family
ID=74957123
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011257710.4A Active CN112507187B (en) | 2020-11-11 | 2020-11-11 | Index changing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112507187B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113220662B (en) * | 2021-04-30 | 2024-02-09 | 广州市百果园信息技术有限公司 | Index migration method, device, electronic equipment and storage medium |
| CN113268466A (en) * | 2021-06-07 | 2021-08-17 | 上海数禾信息科技有限公司 | Method and system for smoothly migrating message cluster |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103631937A (en) * | 2013-12-06 | 2014-03-12 | 北京趣拿信息技术有限公司 | Method, device and system for establishing column storage indexes |
| CN108431780A (en) * | 2016-01-19 | 2018-08-21 | 微软技术许可有限责任公司 | Use the versioned record management for restarting the epoch |
| US10347370B1 (en) * | 2015-08-17 | 2019-07-09 | Aetion Inc. | Deriving a patient level longitudinal database for rapid cycle analytics |
| CN110609865A (en) * | 2018-05-29 | 2019-12-24 | 优信拍（北京）信息科技有限公司 | Information synchronization method, device and system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9767108B2 (en) * | 2012-07-13 | 2017-09-19 | Hitachi Solutions, Ltd. | Retrieval device, method for controlling retrieval device, and recording medium |
| US10140149B1 (en) * | 2015-05-19 | 2018-11-27 | Pure Storage, Inc. | Transactional commits with hardware assists in remote memory |
| CN106055622A (en) * | 2016-05-26 | 2016-10-26 | 浪潮软件集团有限公司 | Data searching method and system |
| CN109144994B (en) * | 2017-06-19 | 2022-04-29 | 华为技术有限公司 | Index updating method, system and related device |
| CN111352936A (en) * | 2020-02-06 | 2020-06-30 | 福建天泉教育科技有限公司 | Method and storage medium for ES index reconstruction |
-
2020
- 2020-11-11 CN CN202011257710.4A patent/CN112507187B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103631937A (en) * | 2013-12-06 | 2014-03-12 | 北京趣拿信息技术有限公司 | Method, device and system for establishing column storage indexes |
| US10347370B1 (en) * | 2015-08-17 | 2019-07-09 | Aetion Inc. | Deriving a patient level longitudinal database for rapid cycle analytics |
| CN108431780A (en) * | 2016-01-19 | 2018-08-21 | 微软技术许可有限责任公司 | Use the versioned record management for restarting the epoch |
| CN110609865A (en) * | 2018-05-29 | 2019-12-24 | 优信拍（北京）信息科技有限公司 | Information synchronization method, device and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112507187A (en) | 2021-03-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109493076B (en) | Kafka message unique consumption method, system, server and storage medium | |
| CN102594849B (en) | Data backup and recovery method and device, virtual machine snapshot deleting and rollback method and device | |
| US8688642B2 (en) | Systems and methods for managing application availability | |
| CN112507187B (en) | Index changing method and device | |
| US20150213100A1 (en) | Data synchronization method and system | |
| CN110096336A (en) | Data monitoring method, device, equipment and medium | |
| CN110968586A (en) | Distributed transaction processing method and device | |
| WO2015135370A1 (en) | Data update method and system | |
| US11500738B2 (en) | Tagging application resources for snapshot capability-aware discovery | |
| US9195528B1 (en) | Systems and methods for managing failover clusters | |
| CN112783868A (en) | Distributed database table structure gray scale upgrading method, device and system | |
| CN111352936A (en) | Method and storage medium for ES index reconstruction | |
| CN113779149A (en) | Message processing method and device, electronic equipment and readable storage medium | |
| US20210334165A1 (en) | Snapshot capability-aware discovery of tagged application resources | |
| EP4250119A1 (en) | Data placement and recovery in the event of partition failures | |
| US11093290B1 (en) | Backup server resource-aware discovery of client application resources | |
| CN111049913B (en) | Data file transmission method and device, storage medium and electronic equipment | |
| US11386060B1 (en) | Techniques for verifiably processing data in distributed computing systems | |
| CN114328007B (en) | Container backup and restoration method, device and medium thereof | |
| CN109344001B (en) | Concurrent backup method and system for virtual machine files | |
| CN116594734A (en) | Container migration method and device, storage medium and electronic equipment | |
| CN111209138A (en) | Operation and maintenance method and device of data storage system | |
| CN112084171A (en) | Operation log writing method, device, equipment and medium based on Cassandra database | |
| CN111147554A (en) | Data storage method and device and computer system | |
| CN111241068B (en) | Information processing method, device and equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |