CN112632083A - Method, device and equipment for updating index data - Google Patents
Method, device and equipment for updating index data Download PDFInfo
- Publication number
- CN112632083A CN112632083A CN202011640212.8A CN202011640212A CN112632083A CN 112632083 A CN112632083 A CN 112632083A CN 202011640212 A CN202011640212 A CN 202011640212A CN 112632083 A CN112632083 A CN 112632083A
- Authority
- CN
- China
- Prior art keywords
- written
- index
- index data
- data
- library
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000012795 verification Methods 0.000 claims description 64
- 230000001960 triggered effect Effects 0.000 abstract description 3
- 238000013523 data management Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a method, a device and equipment for updating index data. Sending the index base information to be written into the index base to the cluster node so that the cluster node can determine the index base to be written according to the index base information; and transmitting the index data to be written to the cluster node. And controlling the cluster node to write the index data to be written into the index database to be written. And finally, if the updating of the index data to be written is finished, switching the index database to be written into to an index database providing service. Therefore, the complexity of updating the index data required by different updates is reduced on the basis of not influencing the query performance of the search engine. Based on the obtained index data to be written sent by the user terminal, the index data can be automatically triggered and updated, and the updating efficiency of the index data is improved.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a method, an apparatus, and a device for updating index data.
Background
Based on the rapid development of internet technology, a user generates a large amount of data when using the internet, and the generated data needs to be stored and analyzed in order to provide better service to the user.
At present, a search engine is used to manage massive data and perform data search and data analysis in real time. The search engine has an index repository therein for managing index data. The server side stores a large amount of index data into an index database so as to search and analyze the index data through a search engine. When data maintenance is performed on index data, the types of updating operations to be performed on the index data are many, and the process of performing index data maintenance is complex, so that the updating efficiency of the index data is low.
Disclosure of Invention
In view of this, embodiments of the present application provide a method, an apparatus, and a device for implementing index data update, which can improve the update efficiency of index data and the flexibility of updating index data on the basis of not affecting the query performance of a search engine.
In order to solve the above problem, the technical solution provided by the embodiment of the present application is as follows:
in a first aspect, the present application provides a method for implementing index data update, where the method includes:
receiving index data to be written sent by a user terminal; the index data to be written is full index data;
selecting a standby index library as an index library to be written in, wherein the standby index library is an index library which does not provide service;
sending the index base information to be written into the index base to cluster nodes so that the cluster nodes can determine the index base to be written according to the index base information;
transmitting the index data to be written to the cluster node;
controlling the cluster nodes to write the index data to be written into the index library to be written;
and if the updating of the index data to be written is finished, switching the index database to be written into to an index database providing service.
In a possible implementation manner, after receiving index data to be written sent by a user terminal, the method further includes:
judging whether a target user corresponding to the user terminal has an updating authority or not and whether the index data to be written has data validity or not; wherein, the updating authority is determined by inquiring the user authority corresponding to the target user; the data validity is that only index data to be written in the same batch sent by the user terminal is received in a target time period;
the transmitting the index data to be written to the cluster node includes:
and if the target user corresponding to the user terminal has the updating authority and the index data to be written has the data validity, transmitting the index data to be written to the cluster node.
In a possible implementation manner, the to-be-written index data has a corresponding user identifier, and after the sending of the index base information of the to-be-written index base to the cluster node, the method further includes:
determining a target user according to the user identification, and starting the data writing authority of the target user; the data writing permission of the target user is closed by default;
the controlling the cluster nodes to write the index data to be written into the index library to be written includes:
judging whether the data writing authority of the target user is opened or not;
and if the data writing authority of the target user is opened, controlling the cluster nodes to write the index data to be written with the user identification into the index library to be written.
In a possible implementation manner, the switching the index library to be written to the index library providing the service if the update of the current index data to be written is completed includes:
taking the time when the cluster node starts to write the index data to be written into the index library to be written as the starting time;
after a preset verification time period from the starting time, verifying whether the index data to be written is completely written into the index database to be written;
if all the index data to be written are written into the index library to be written, switching the index library to be written into an index library providing services;
if the index data to be written is not completely written into the index library to be written, after a preset verification time period, verifying whether the index data to be written is completely written into the index library to be written or not and subsequent steps are executed again until the index data to be written is completely written into the index library to be written or the verification times reach preset verification times.
In one possible implementation, the method further includes:
and adjusting the preset verification time period according to the verification times, wherein the preset verification time period corresponding to the writing of the index data to be written in the current verification is larger than the preset verification time period corresponding to the writing of the index data to be written in the last verification.
In a possible implementation manner, after the selecting one standby index library as the index library to be written into, the method further includes:
judging whether the index database to be written has historical index data or not;
and if so, emptying the historical index data to be written into the index library.
In one possible implementation, the method further includes:
if the updating of the index data to be written fails, using the index library which provides the service currently to provide the service;
and feeding back the update failure information to the user terminal.
In a second aspect, the present application provides an apparatus for implementing index data update, the apparatus comprising:
the receiving unit is used for receiving index data to be written sent by a user terminal; the index data to be written is full index data;
the device comprises a selecting unit, a writing unit and a reading unit, wherein the selecting unit is used for selecting a standby index library as an index library to be written in, and the standby index library is an index library which does not provide service;
the first sending unit is used for sending the index library information to be written into the index library to cluster nodes so that the cluster nodes can determine the index library to be written according to the index library information;
the second sending unit is used for transmitting the index data to be written to the cluster node;
the writing unit is used for controlling the cluster nodes to write the index data to be written into the index library to be written;
and the switching unit is used for switching the index database to be written into to the index database providing service if the updating of the index data to be written is completed.
In one possible implementation, the apparatus further includes:
the first judging unit is used for judging whether a target user corresponding to the user terminal has an updating authority or not and whether the index data to be written has data validity or not; wherein, the updating authority is determined by inquiring the user authority corresponding to the target user; the data validity is that only index data to be written in the same batch sent by the user terminal is received in a target time period;
and the second sending unit is specifically configured to transmit the index data to be written to the cluster node if the target user corresponding to the user terminal has the update permission and the index data to be written has the data validity.
In one possible implementation, the apparatus further includes:
the permission opening unit is used for determining a target user according to the user identifier and opening the data writing permission of the target user; the data writing permission of the target user is closed by default;
the writing unit is specifically configured to determine whether a data writing authority of the target user is turned on;
and if the data writing authority of the target user is opened, controlling the cluster nodes to write the index data to be written with the user identification into the index library to be written.
In a possible implementation manner, the switching unit is specifically configured to use a time when the cluster node starts to write the index data to be written into the index library to be written as an initial time;
after a preset verification time period from the starting time, verifying whether the index data to be written is completely written into the index database to be written;
if all the index data to be written are written into the index library to be written, switching the index library to be written into an index library providing services;
if the index data to be written is not completely written into the index library to be written, after a preset verification time period, verifying whether the index data to be written is completely written into the index library to be written or not and subsequent steps are executed again until the index data to be written is completely written into the index library to be written or the verification times reach preset verification times.
In one possible implementation, the apparatus further includes:
and the adjusting unit is used for adjusting the preset verification time period according to the verification times, wherein the preset verification time period corresponding to the writing of the index data to be written in the current verification is larger than the preset verification time period corresponding to the writing of the index data to be written in the last verification.
In one possible implementation, the apparatus further includes:
the second judging unit is used for judging whether the index library to be written has historical index data or not;
and if so, clearing the historical index data to be written into the index library.
In one possible implementation, the apparatus further includes:
the execution unit is used for providing service by using the index library which provides service at present if the updating of the index data to be written fails;
and feeding back the update failure information to the user terminal.
In a third aspect, the present application provides an apparatus for implementing index data update, including: a processor, a memory, a system bus;
the processor and the memory are connected through the system bus;
the memory is used for storing one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform the method of the above embodiment.
In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions that, when executed on a terminal device, cause the terminal device to perform the method of the foregoing embodiment.
Therefore, the embodiment of the application has the following beneficial effects:
the method, the device and the equipment for realizing the index data updating provided by the embodiment of the application firstly receive index data to be written, which is sent by a user terminal; the index data to be written is full index data; selecting a standby index library as an index library to be written in, wherein the standby index library is an index library which does not provide service; sending the index base information to be written into the index base to the cluster node so that the cluster node can determine the index base to be written according to the index base information; transmitting the index data to be written to the cluster node; controlling the cluster node to write the index data to be written into the index database to be written; and finally, if the updating of the index data to be written is finished, switching the index database to be written into to an index database providing service. By selecting the standby index database to write in the index data to be written in, asynchronous index data updating can be realized, and the query performance of a search engine is not influenced. The full amount of index data is used as the index data to be written, so that the complexity of updating the index data required by different updates is reduced, and the index data can be updated conveniently. In addition, based on the obtained to-be-written index data sent by the user terminal, the index data can be automatically triggered and updated, the index data can be flexibly updated, manual updating operation is reduced, and the updating efficiency of the index data is improved.
Drawings
Fig. 1 is a flowchart of a method for updating index data according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of an apparatus for implementing index data update according to an embodiment of the present disclosure.
Detailed Description
In order to facilitate understanding and explaining the technical solutions provided by the embodiments of the present application, the following description will first describe the background art of the present application.
After studying on a conventional method for updating index data of a search engine, the inventors found that, during the use of the search engine, a user needs to update, for example, add, delete, modify, etc., the index data in the search engine to ensure the correctness of the index data in the search engine. The updating operation performed on the index data by the user is various, and corresponding manual updating operation needs to be performed according to different updating requirements during actual updating, so that the updating process is complex, and the index data is not convenient to update. In addition, when index data is updated, it is necessary to ensure that a search engine can provide services normally. However, at present, the index data update occupies more resources, and the query performance of the search engine is easily affected. If the index data fails to be updated, the search engine may be caused to stop providing services.
Based on this, the embodiment of the present application provides a method for implementing index data update, which first receives index data to be written, which is sent by a user terminal; the index data to be written is full index data; selecting a standby index library as an index library to be written in, wherein the standby index library is an index library which does not provide service; sending the index base information to be written into the index base to the cluster node so that the cluster node can determine the index base to be written according to the index base information; transmitting the index data to be written to the cluster nodes; controlling the cluster node to write the index data to be written into the index database to be written; and finally, if the updating of the index data to be written is finished, switching the index database to be written into to an index database providing service. Therefore, asynchronous index data updating can be realized by updating the index data in batch, and the query performance of a search engine is not influenced. And the full amount of index data is used as the index data to be written, so that the complexity of updating the index data required by different operations is reduced, and the index data can be updated conveniently. In addition, based on the obtained index data to be written sent by the user terminal, the index data can be automatically updated, manual updating operation is reduced, and the updating efficiency of the index data is improved.
In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, a method for implementing index data update provided by the embodiments of the present application is described below with reference to the accompanying drawings.
First, it should be noted that the embodiments of the present application do not limit the execution subject of the method for implementing index data update. In one possible implementation, it may be an index data management system.
Referring to fig. 1, the figure is a flowchart of a method for implementing index data update according to an embodiment of the present application, where the method includes steps S101 to S106.
S101: receiving index data to be written sent by a user terminal; the index data to be written is full index data.
The user terminal is a terminal that provides index data to a search engine. The user terminals may be user devices, such as smart phones, personal computers, mid-to-large computers, and the like, that interact with each other through wired and/or wireless connections.
The user terminal may send the index data to be written to the index data management system. The index data to be written is the full amount of index data stored and processed by the search engine. The full index data comprises original index data which does not need to be updated in the updating and index data which needs to be updated and is obtained after updating.
By taking the full amount of index data as the index data to be written, the writing operation can be performed only on the index data to be written. Therefore, the full-scale synchronous updating of the index data is realized, different updating operations on the index data with various updating requirements are avoided, the operation complexity of updating the index data is reduced, and the updating efficiency of the index data is improved.
In a possible implementation manner, the user terminal needs to preprocess the index data to be written, encapsulate the index data to be written according to the interface specification, generate a corresponding data file, and send the encapsulated data file to the index data management system, so that the subsequent index data management system can write the index data to be written into the index database based on the data file. Specifically, the data files obtained after encapsulation can be classified into 2 types. One type is a data file comprising index data to be written which needs to be written into an index library; the other type is a data file comprising summary information of a data file formed by the current index data to be written.
S102: and selecting a standby index library as an index library to be written in, wherein the standby index library is an index library which does not provide services.
It will be appreciated that the search engine needs to read the index data from the index repository when providing services. In order to ensure that the search engine can provide services normally, one spare index library is selected from the unserviceable spare index libraries to be written into the index library.
The standby index base can be pre-established or newly established. The number of the spare index banks is not limited in the embodiment of the present application, and in a possible implementation manner, the number of the spare index banks may be 2. One spare index library is prepared for writing index data to be written, and the other spare index library is used for storing historical index data.
S103: and sending the index base information to be written into the index base to the cluster node so that the cluster node can determine the index base to be written according to the index base information.
The index library to be written has corresponding index library information, and the index library information is used for distinguishing different index libraries. The index repository information may specifically be an index repository identification.
The cluster node is a node for executing the writing of the index data to be written into the index library to be written. And sending the index base information to be written into the index base to the cluster node. The cluster node can determine the index database to which the index data to be written is written according to the index database information.
S104: and transmitting the index data to be written to the cluster node.
The embodiment of the application does not limit the specific way of transmitting the index data to be written to the cluster node. In a possible implementation manner, the index data to be written may be divided into a plurality of groups of index data to be written, and each group of index data to be written is transmitted to a corresponding cluster node, so that the cluster node performs an operation of writing the group of index data to be written into the index library.
In a specific implementation manner, each cluster node may obtain the index data to be written from the task list, so as to transmit the index data to be written to the cluster node.
It should be noted that the embodiment of the present application does not limit the execution order among S102 to S103 and S104. In a possible implementation manner, the index database to be written may be determined first, and then the index data to be written may be transmitted to the cluster node; or the index data to be written can be transmitted to the cluster node, and then the index database to be written can be determined; and simultaneously determining an index database to be written, and transmitting the index data to be written to the cluster node.
S105: and controlling the cluster node to write the index data to be written into the index database to be written.
After the cluster node acquires the index base information and the index data to be written in the index base, the cluster node is controlled to write the index data to be written in the index base. Through the cluster nodes, concurrent batch writing operation of the index data to be written can be realized, and the writing efficiency of the index data to be written is improved.
S106: and if the updating of the index data to be written is finished, switching the index database to be written into to an index database providing services.
After the index data to be written is written into the index library to be written, the writing condition of the index data to be written can be checked. And if all the index data to be written are written into the index database to be written, finishing the updating of the index data to be written.
After the updating of the current index data to be written is ensured to be completed, the index database to be written is switched to the index database providing services, and the services of the search engine are provided by utilizing the updated index data.
The embodiment of the present application provides a specific implementation manner for switching the index library to be written to the index library providing services if the update of the current index data to be written is completed, and please refer to the following.
In one possible implementation, historical index data in the last served index repository may be retained for data recovery using the historical index data when an abnormal condition is encountered.
Based on the related contents of S101 to S106, it can be seen that by acquiring the entire amount of index data from the user terminal as the index data to be written, the complexity of updating the index data required for different updates can be reduced, and the operation of updating the index data is simplified. By selecting the standby index database to write in the index data to be written, asynchronous index data updating can be realized, the query performance of a search engine is not influenced, the resource consumption during updating is reduced, and the index data updating is convenient to realize. In addition, based on the obtained index data to be written sent by the user terminal, the index data can be automatically triggered and updated, manual updating operation is reduced, and the updating efficiency of the index data is improved.
In a possible implementation manner, after receiving index data to be written sent by a user terminal, the user terminal and the written index data need to be checked.
After receiving the index data to be written sent by the user terminal, the method further comprises the following steps:
judging whether a target user corresponding to the user terminal has an updating authority or not and whether index data to be written has data validity or not; the updating authority is determined by inquiring the user authority corresponding to the target user; the data validity is that only the index data to be written which is sent by the user terminal and belongs to the same batch is received in the target time period.
It can be understood that, in order to ensure the data security of the index database, the user corresponding to the user terminal has the updating right. After obtaining the index data to be written sent by the user terminal, it is necessary to verify whether the user terminal can send the index data to be written for updating the index data. Specifically, the user terminal has a corresponding target user, and whether the target user has the update right can be determined by querying the user right of the target user. If the user authority of the target user has the updating authority, the updating operation of writing the index database into the index data to be written sent by the user terminal can be carried out.
In addition, data validity verification needs to be performed on the index data to be written. The data validity refers to whether the index data to be written is valid index data. The data validity can be verified by whether only one batch of index data to be written sent by the user terminal is received in the target time period. The target time period may be determined according to a frequency of the user terminal performing normal index data update, and the target time period may be a preset time period smaller than a time interval between adjacent updated index data. For example, if the time interval between adjacent update index data is 24 hours, the target time period may be 10 minutes.
Specifically, the index data to be written that is sent by the user terminal each time and needs to be updated may be used as the index data to be written in the same batch. The batches of the index data to be written can be determined by the batch identifiers, the index data to be written in the same batch have the same batch identifier, and the batch identifiers of the index data to be written in different batches are different.
If only one batch of index data to be written is received in the target time period, it may be determined that the index data to be written has data validity. If the index data to be written of a plurality of batches is received in the target time period, the user terminal is indicated to send the index data to be written for a plurality of times in a short time, and the received index data to be written may be invalid.
If the index data to be written has data validity, subsequent operations of determining the index database to be written and transmitting the index data to be written to the cluster node for writing can be performed. If the index data to be written does not have data validity, feedback information of update failure can be sent to the user terminal so as to determine the valid index data to be written.
Transmitting the index data to be written to the cluster node, specifically comprising:
and if the target user corresponding to the user terminal has the updating authority and the index data to be written has the data validity, transmitting the index data to be written to the cluster node.
After determining that the target user corresponding to the user terminal has the updating right and the index data to be written has the data validity, it may be determined that the verification of the user terminal and the index data to be written passes. And transmitting the index data to be written to the cluster node so that the cluster node writes the index data to be written into the index database to be written.
Based on the above, by verifying the update permission of the target terminal corresponding to the user terminal and the data validity of the index data to be written after receiving the index data to be written sent by the user terminal, the data validity of the index data to be written and the update safety of the index data can be ensured, and the repeated invalid index data to be written are prevented from occupying the storage resource and the writing resource of the index library.
And before the index library to be written is not determined, the default index library which carries out read-write operation is the index library which provides service. If the operation of writing the index data to be written in by the cluster node is directly performed after the index data to be written sent by the user terminal is received, part of the index data to be written in may be written in the index library which provides the service currently.
Based on this, in a possible implementation manner, an embodiment of the present application provides a method for implementing index data update, where index data to be written has a corresponding user identifier, and after sending index library information to be written in an index library to a cluster node, the method further includes the following steps:
determining a target user according to the user identification, and starting the data writing authority of the target user; the data write permission of the target user is closed by default.
In order to realize the write control of the index data to be written, data write permission can be correspondingly set for a user who updates the index data. The data writing authority is an authority for controlling whether to write the index data to be written provided by the user into the index library.
The index data to be written has a user identifier, the user identifier is used for determining a user who sends the index data to be written, and the user identifier can be determined according to a user terminal.
And determining a target user by using the user identification, wherein the target user has data writing authority, and the data writing authority is in a closed state by default. After determining the index library to be written and sending the index library information of the index library to be written to the cluster node, the cluster node may determine the index library to be written to which the index data to be written is written, so as to implement correct writing of the index data to be written. And correspondingly starting the data writing authority of the target user so as to realize that the subsequent cluster nodes start to write the index data to be written.
Correspondingly, the controlling the cluster node to write the index data to be written into the index library to be written includes:
judging whether the data writing authority of the target user is opened or not;
and if the data writing authority of the target user is opened, controlling the cluster node to write the index data to be written with the user identification into the index database to be written.
And acquiring a user identifier of the index data to be written, and judging whether the data writing authority of the target user corresponding to the user identifier is opened or not. And if the data writing authority of the target user is opened, controlling the cluster node to write the index data to be written with the user identification corresponding to the target user into the index database to be written.
In a possible implementation manner, after the update of the current index data to be written is completed, the data writing permission of the target user corresponding to the user identifier of the index data to be written is closed, so as to implement the writing control of the next index data to be written.
In the embodiment of the application, by setting the data writing authority and performing opening control on the data writing authority, the index data to be written can be written after the index database to be written is determined, so that the index data to be written is prevented from being written into the index database providing services. Therefore, the index data to be written can be written into the correct index library, and the updating process of the index data is ensured to be correct.
The embodiment of the present application provides a specific implementation manner for switching an index library to be written to an index library providing services if the update of the current index data to be written is completed, and specifically includes the following four steps:
a1: and taking the time when the cluster nodes start to write the index data to be written into the index library as the starting time.
It can be understood that a certain writing time is required for the cluster nodes to completely write the index data to be written into the index library to be written. After it is determined that the cluster node can write all the index data to be written into the index library to be written, it is determined whether the update of the index data to be written is completed.
And taking the time when the cluster nodes start to write the index data to be written into the index library as the starting time. The completion time of the index data to be written can be determined by using the starting time.
In a possible implementation manner, the time when the cluster node switches to the to-be-written index library may be determined as the time when the cluster node starts to write the to-be-written index data into the to-be-written index library, that is, the starting time.
A2: and judging whether the index data to be written is completely written into the index database to be written after a preset verification time period from the starting time.
The preset verification time period is a preset time period required for writing all index data to be written into the index library to be written. The preset check time period may be determined according to a historical length of time required for writing the index data to be written. The preset check time period can be adjusted according to the data volume of the index data to be written and the number of the cluster nodes.
Starting from the starting time, after a preset verification time period elapses, it can be determined that under a normal condition, the cluster node can write all index data to be written into the index library to be written. And checking whether the index data to be written is completely written into the index library to be written or not, thereby determining whether the writing of the index data to be written is successful or not. Specifically, if the to-be-written index data has the user identifier and the batch identifier, whether the to-be-written index data written into the to-be-written index library is complete or not may be verified by using the user identifier and the batch identifier.
A3: and if all the index data to be written are written into the index library to be written, switching the index library to be written into an index library providing services.
And if all the index data to be written is written into the index database to be written, updating the index data to be written this time successfully, switching the index database to be written into the index database providing service, and using the written index data to be written as new index data.
A4: and if the index data to be written is not completely written into the index database to be written, after a preset verification time period, verifying whether the index data to be written is completely written into the index database to be written or not and subsequent steps are carried out again until the index data to be written is completely written into the index database to be written or the verification times reach the preset verification times.
And if the index data to be written is not completely written into the index database to be written, the writing of the index data to be written is unsuccessful. And after the preset verification time period, verifying whether the index data to be written is completely written into the index database to be written. And if the index data to be written is completely written into the index database to be written, stopping checking whether the index data to be written is successfully written. And if the index data to be written is not completely written into the index database to be written, continuously verifying whether the index data to be written is successfully written. And if the checking times reach the preset checking times or the checking is successful, stopping continuing the checking.
It should be noted that, in order to prevent the index database to be written from having repeated index data to be written, the last written index data to be written may be overwritten every time the index data to be written is written into the index database to be written again. And clearing the index data to be written in the index library to be written when the fact that the index data to be written is not completely written in the index library to be written is determined.
Based on the above content, in the embodiment of the present application, after the index data to be written starts to be written, the written index data to be written in the index library is verified after the preset verification time period, so that whether the current index data to be written is successfully written in the index library to be written can be verified. And the writing operation is carried out again when the writing is not successful, so that the fault tolerance of the index data updating is improved. And the time for performing the verification at this time can be correspondingly adjusted through presetting the verification time, so that on one hand, the index data to be written can be ensured to be completely written into the index database under the normal condition, on the other hand, the overlong time for waiting the verification can be avoided, and the efficiency for updating the index data is improved.
In one possible implementation, the index data to be written may take longer to write to the index library to be written.
Further, the method further comprises:
and adjusting a preset verification time period according to the verification times, wherein the preset verification time period corresponding to the writing of the index data to be written in the current verification is larger than the preset verification time period corresponding to the writing of the index data to be written in the last verification.
In order to prevent the cluster nodes from being unable to write all index data to be written into the index database within the preset verification time period, the preset verification time period may be adjusted according to the verification times of the index data to be written into, so that the preset verification time period of this time is greater than the last preset verification time period. Therefore, all index data to be written can be normally written into the index library to be written in the cluster nodes, and then the integrity of the index data to be written is verified.
In a possible implementation manner, after selecting one spare index library as the index library to be written into, the method further includes:
judging whether historical index data exist in an index database to be written;
and if so, clearing the historical index data to be written into the index database.
The index database to be written may have historical index data, and the historical index data to be written in the index database needs to be emptied before the index data to be written is written. Therefore, the data to be written into the index database can be ensured to be written into the index database, other index data do not exist, and normal service of the search engine is ensured.
In one possible implementation, the current index data to be written may fail to be updated. Correspondingly, an embodiment of the present application provides a method for implementing index data update, where in addition to the above steps, the method further includes:
if the updating of the index data to be written fails, using the index library which provides the service currently to provide the service;
and feeding back the update failure information to the user terminal.
And when the updating of the index to be written fails, the index library to be written cannot be used as the index library for providing service. The service continues to be provided using the index repository currently providing the service.
And feeding back the update failure information to the user terminal so that the user can check the reason of the update failure.
In the embodiment of the application, when the updating fails, the index library which provides the service currently is kept to continue providing the service, and the service providing of the search engine is not influenced when the updating fails. And feeding back update failure information to the user terminal so as to check the reason of update failure.
Based on the method for updating index data provided by the above method embodiment, an embodiment of the present application further provides a device for updating index data, and the device for updating index data will be described below with reference to the accompanying drawings.
Referring to fig. 2, this figure is a schematic structural diagram of an apparatus for implementing index data update according to an embodiment of the present application. As shown in fig. 2, the apparatus for implementing index data update includes:
a receiving unit 201, configured to receive index data to be written, sent by a user terminal; the index data to be written is full index data;
a selecting unit 202, configured to select a standby index library as an index library to be written in, where the standby index library is an index library that does not provide a service;
a first sending unit 203, configured to send the index library information of the index library to be written to a cluster node, so that the cluster node determines the index library to be written according to the index library information;
a second sending unit 204, configured to transmit the index data to be written to the cluster node;
a writing unit 205, configured to control the cluster node to write the index data to be written into the index library to be written;
the switching unit 206 is configured to switch the index library to be written to an index library providing a service if the update of the index data to be written is completed this time.
In one possible implementation, the apparatus further includes:
the first judging unit is used for judging whether a target user corresponding to the user terminal has an updating authority or not and whether the index data to be written has data validity or not; wherein, the updating authority is determined by inquiring the user authority corresponding to the target user; the data validity is that only index data to be written in the same batch sent by the user terminal is received in a target time period;
the second sending unit 204 is specifically configured to transmit the index data to be written to the cluster node if the target user corresponding to the user terminal has the update permission and the index data to be written has the data validity.
In one possible implementation, the apparatus further includes:
the permission opening unit is used for determining a target user according to the user identifier and opening the data writing permission of the target user; the data writing permission of the target user is closed by default;
the writing unit 205 is specifically configured to determine whether the data writing authority of the target user is turned on;
and if the data writing authority of the target user is opened, controlling the cluster nodes to write the index data to be written with the user identification into the index library to be written.
In a possible implementation manner, the switching unit 206 is specifically configured to use a time when the cluster node starts to write the index data to be written into the index library to be written as a starting time;
after a preset verification time period from the starting time, verifying whether the index data to be written is completely written into the index database to be written;
if all the index data to be written are written into the index library to be written, switching the index library to be written into an index library providing services;
if the index data to be written is not completely written into the index library to be written, after a preset verification time period, verifying whether the index data to be written is completely written into the index library to be written or not and subsequent steps are executed again until the index data to be written is completely written into the index library to be written or the verification times reach preset verification times.
In one possible implementation, the apparatus further includes:
and the adjusting unit is used for adjusting the preset verification time period according to the verification times, wherein the preset verification time period corresponding to the writing of the index data to be written in the current verification is larger than the preset verification time period corresponding to the writing of the index data to be written in the last verification.
In one possible implementation, the apparatus further includes:
the second judging unit is used for judging whether the index library to be written has historical index data or not;
and if so, clearing the historical index data to be written into the index library.
In one possible implementation, the apparatus further includes:
the execution unit is used for providing service by using the index library which provides service at present if the updating of the index data to be written fails;
and feeding back the update failure information to the user terminal.
Based on the method for updating index data provided by the above method embodiment, an embodiment of the present application further provides an apparatus for updating index data, including: a processor, a memory, a system bus;
the processor and the memory are connected through the system bus;
the memory is used for storing one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform the method of the above embodiment.
Based on the method for updating index data provided by the foregoing method embodiment, an embodiment of the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are run on a terminal device, the terminal device is enabled to execute the method described in the foregoing embodiment.
It should be noted that, in the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the system or the device disclosed by the embodiment, the description is simple because the system or the device corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
It should be understood that in the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" for describing an association relationship of associated objects, indicating that there may be three relationships, e.g., "a and/or B" may indicate: only A, only B and both A and B are present, wherein A and B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of single item(s) or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method for implementing index data update, the method comprising:
receiving index data to be written sent by a user terminal; the index data to be written is full index data;
selecting a standby index library as an index library to be written in, wherein the standby index library is an index library which does not provide service;
sending the index base information to be written into the index base to cluster nodes so that the cluster nodes can determine the index base to be written according to the index base information;
transmitting the index data to be written to the cluster node;
controlling the cluster nodes to write the index data to be written into the index library to be written;
and if the updating of the index data to be written is finished, switching the index database to be written into to an index database providing service.
2. The method according to claim 1, wherein after receiving index data to be written sent by a user terminal, the method further comprises:
judging whether a target user corresponding to the user terminal has an updating authority or not and whether the index data to be written has data validity or not; wherein, the updating authority is determined by inquiring the user authority corresponding to the target user; the data validity is that only index data to be written in the same batch sent by the user terminal is received in a target time period;
the transmitting the index data to be written to the cluster node includes:
and if the target user corresponding to the user terminal has the updating authority and the index data to be written has the data validity, transmitting the index data to be written to the cluster node.
3. The method according to claim 1, wherein the index data to be written has a corresponding user identifier, and after the sending of the index base information of the index base to be written to the cluster node, the method further comprises:
determining a target user according to the user identification, and starting the data writing authority of the target user; the data writing permission of the target user is closed by default;
the controlling the cluster nodes to write the index data to be written into the index library to be written includes:
judging whether the data writing authority of the target user is opened or not;
and if the data writing authority of the target user is opened, controlling the cluster nodes to write the index data to be written with the user identification into the index library to be written.
4. The method according to claim 1, wherein the switching the to-be-written index library to an index library providing services if the updating of the current to-be-written index data is completed comprises:
taking the time when the cluster node starts to write the index data to be written into the index library to be written as the starting time;
after a preset verification time period from the starting time, verifying whether the index data to be written is completely written into the index database to be written;
if all the index data to be written are written into the index library to be written, switching the index library to be written into an index library providing services;
if the index data to be written is not completely written into the index library to be written, after a preset verification time period, verifying whether the index data to be written is completely written into the index library to be written or not and subsequent steps are executed again until the index data to be written is completely written into the index library to be written or the verification times reach preset verification times.
5. The method of claim 4, further comprising:
and adjusting the preset verification time period according to the verification times, wherein the preset verification time period corresponding to the writing of the index data to be written in the current verification is larger than the preset verification time period corresponding to the writing of the index data to be written in the last verification.
6. The method of claim 1, wherein after the selecting one of the spare index banks as the index bank to be written to, the method further comprises:
judging whether the index database to be written has historical index data or not;
and if so, emptying the historical index data to be written into the index library.
7. The method of claim 1, further comprising:
if the updating of the index data to be written fails, using the index library which provides the service currently to provide the service;
and feeding back the update failure information to the user terminal.
8. An apparatus for implementing index data update, the apparatus comprising:
the receiving unit is used for receiving index data to be written sent by a user terminal; the index data to be written is full index data;
the device comprises a selecting unit, a writing unit and a reading unit, wherein the selecting unit is used for selecting a standby index library as an index library to be written in, and the standby index library is an index library which does not provide service;
the first sending unit is used for sending the index library information to be written into the index library to cluster nodes so that the cluster nodes can determine the index library to be written according to the index library information;
the second sending unit is used for transmitting the index data to be written to the cluster node;
the writing unit is used for controlling the cluster nodes to write the index data to be written into the index library to be written;
and the switching unit is used for switching the index database to be written into to the index database providing service if the updating of the index data to be written is completed.
9. An apparatus for implementing index data update, comprising: a processor, a memory, a system bus;
the processor and the memory are connected through the system bus;
the memory is to store one or more programs, the one or more programs comprising instructions, which when executed by the processor, cause the processor to perform the method of any of claims 1-7.
10. A computer-readable storage medium having stored therein instructions that, when executed on a terminal device, cause the terminal device to perform the method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011640212.8A CN112632083A (en) | 2020-12-31 | 2020-12-31 | Method, device and equipment for updating index data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011640212.8A CN112632083A (en) | 2020-12-31 | 2020-12-31 | Method, device and equipment for updating index data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112632083A true CN112632083A (en) | 2021-04-09 |
Family
ID=75290183
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011640212.8A Pending CN112632083A (en) | 2020-12-31 | 2020-12-31 | Method, device and equipment for updating index data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112632083A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114595410A (en) * | 2022-03-24 | 2022-06-07 | 中国农业银行股份有限公司 | Webpage parsing method and system and electronic equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102779185A (en) * | 2012-06-29 | 2012-11-14 | 浙江大学 | High-availability distribution type full-text index method |
| CN105069151A (en) * | 2015-08-24 | 2015-11-18 | 用友网络科技股份有限公司 | HBase secondary index construction apparatus and method |
| CN105208105A (en) * | 2015-08-26 | 2015-12-30 | 北京奇虎科技有限公司 | Index updating method and apparatus |
| CN107315825A (en) * | 2017-07-05 | 2017-11-03 | 北京奇艺世纪科技有限公司 | A kind of index upgrade system, method and device |
-
2020
- 2020-12-31 CN CN202011640212.8A patent/CN112632083A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102779185A (en) * | 2012-06-29 | 2012-11-14 | 浙江大学 | High-availability distribution type full-text index method |
| CN105069151A (en) * | 2015-08-24 | 2015-11-18 | 用友网络科技股份有限公司 | HBase secondary index construction apparatus and method |
| CN105208105A (en) * | 2015-08-26 | 2015-12-30 | 北京奇虎科技有限公司 | Index updating method and apparatus |
| CN107315825A (en) * | 2017-07-05 | 2017-11-03 | 北京奇艺世纪科技有限公司 | A kind of index upgrade system, method and device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114595410A (en) * | 2022-03-24 | 2022-06-07 | 中国农业银行股份有限公司 | Webpage parsing method and system and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109474578B (en) | Message checking method, device, computer equipment and storage medium | |
| CN112685391B (en) | Service data migration method and device, computer equipment and storage medium | |
| US11544245B2 (en) | Transaction processing method, apparatus, and device and computer storage medium | |
| CN111290696A (en) | Flow control method and device for application program assembly | |
| CN110688168A (en) | Method, device and equipment for improving starting speed of application program and storage medium | |
| CN110209529A (en) | The guard method of radio frequency parameter and electronic equipment | |
| CN114362983A (en) | Firewall policy management method and device, computer equipment and storage medium | |
| CN112306383B (en) | Method for executing operation, computing node, management node and computing equipment | |
| CN112202633A (en) | Block chain network testing method and device, electronic equipment and readable storage medium | |
| CN112632083A (en) | Method, device and equipment for updating index data | |
| CN113157411B (en) | Celery-based reliable configurable task system and device | |
| CN112104671B (en) | Interface authorization monitoring method and device, computer equipment and storage medium | |
| CN112667270A (en) | Voice processing resource updating method, computer equipment and storage device | |
| CN111343212A (en) | Message processing method, device, equipment and storage medium | |
| CN114895879B (en) | Management system design scheme determining method, device, equipment and storage medium | |
| CN115442137A (en) | Service access processing method and equipment based on identity authentication | |
| CN115239450A (en) | Financial data processing method and device, computer equipment and storage medium | |
| CN114490135A (en) | Task processing method and device, electronic equipment and storage medium | |
| CN115017027A (en) | Interface automation continuous integration test method, device, equipment and storage medium | |
| CN111367694A (en) | Event processing method, server and computer storage medium | |
| CN111711537B (en) | Method, device and equipment for updating standby main node list | |
| CN115774679B (en) | AB experiment regression testing method, device and storage medium | |
| CN114780267B (en) | Interface calling method, device, equipment and storage medium | |
| CN115189932B (en) | Rights management method, device, electronic equipment and readable storage medium | |
| CN109683926B (en) | Network component updating method, device, equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |