CN114461678A - Data retrieval method and device, electronic equipment and readable storage medium - Google Patents
Data retrieval method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN114461678A CN114461678A CN202111600536.3A CN202111600536A CN114461678A CN 114461678 A CN114461678 A CN 114461678A CN 202111600536 A CN202111600536 A CN 202111600536A CN 114461678 A CN114461678 A CN 114461678A
- Authority
- CN
- China
- Prior art keywords
- data
- updated
- local cache
- request information
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data retrieval, and discloses a data retrieval method, a data retrieval device, electronic equipment and a readable storage medium. Wherein, the method comprises the following steps: acquiring retrieval request information of target data; judging whether matching data corresponding to the retrieval request information exists in a local cache corresponding to the current node; when the matching data corresponding to the retrieval request information exists in the local cache, judging whether the matching data needs to be updated or not; and when the matched data needs to be updated, acquiring the updated data from the database corresponding to the local cache, and updating the updated data to the local cache to take the updated data as target data. By implementing the invention, the target data acquired from each node is ensured to be consistent, so that the data consistency among the nodes is ensured, meanwhile, the target data acquired from the local cache reduces the access pressure of the database, and the data retrieval efficiency, the system performance of the distributed system and the data processing capacity are improved.
Description
Technical Field
The invention relates to the technical field of data retrieval, in particular to a data retrieval method, a data retrieval device, electronic equipment and a readable storage medium.
Background
The distributed system has a plurality of nodes, the nodes are deployed in a plurality of physical hosts, and the distributed system needs to pre-store data into a cache through a cache mechanism for hot data. And when the high concurrency condition occurs, the data searching request preferentially retrieves the data in the cache, and if the cache data does not match with the corresponding data, the database is accessed for data searching. When the network of the physical machine is delayed, fails or the node is restarted, the problem of inconsistent cache data of the node and other normal nodes occurs.
For an open-source Infrastructure as a Service (IaaS) platform (such as OpenStack, cloudstock, etc.), in terms of processing of cache data, the method mainly realizes Message communication and data consistency between nodes through a Message Queue (MQ), a MySQL database or a MongoDB database, and does not adopt a third-party data cache component such as redis, although the method can ensure that data acquired by each node is consistent, the method has a large influence on system performance, response speed is obviously reduced, and data retrieval efficiency is reduced; for most of the conventional distributed systems, a mode of combining the MQ, the database and the redis is adopted, although the consistency of the cache data of each node can be ensured through the MQ and the efficiency of data retrieval is ensured through the redis cluster, the distributed system is added with components such as the MQ and the redis, the probability of failure of the distributed system is greatly increased, and the data retrieval efficiency is also influenced.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data retrieval method, an apparatus, an electronic device, and a readable storage medium, so as to solve the problem of low data retrieval efficiency in the existing distributed system.
According to a first aspect, an embodiment of the present invention provides a data retrieval method, including: acquiring retrieval request information of target data; judging whether matching data corresponding to the retrieval request information exists in a local cache corresponding to the current node; when the matching data corresponding to the retrieval request information exists in the local cache, judging whether the matching data needs to be updated or not; and when the matched data needs to be updated, acquiring updated data from a database corresponding to the local cache, and updating the updated data to the local cache, wherein the updated data is used as the target data.
The data retrieval method provided by the embodiment of the invention determines whether the local cache corresponding to the retrieval request information exists in the matching data corresponding to the retrieval request information by acquiring the retrieval request information of the target data and inquiring based on the local cache corresponding to the retrieval request information, when the matching data corresponding to the retrieval request information exists in the local cache, further judges whether the matching data needs to be updated, if the matching data needs to be updated, acquires the updating data from the database corresponding to the local cache, stores the updating data in the local cache, and takes the updating data as the target data corresponding to the retrieval request information, so that the target data corresponding to the retrieval request information can be acquired from the local cache and is the latest cache data, thereby ensuring that the target data acquired from each node are consistent, the data consistency among all the nodes is guaranteed, meanwhile, the target data are obtained from the local cache, the access pressure of the database is reduced, and the data retrieval efficiency, the system performance of the distributed system and the data processing capacity are improved.
With reference to the first aspect, in a first implementation manner of the first aspect, the determining whether the matching data needs to be updated includes: acquiring a current version value and a target version value corresponding to the matching data; and determining whether the matching data needs to be updated or not based on the size relation between the current version value and the target version value.
With reference to the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the determining whether the matching data needs to be updated according to a size relationship between the current version value and the target version value includes: judging whether the current version value is equal to the target version value; and when the current version value is smaller than the target version value, judging that the matching data needs to be updated.
According to the data retrieval method provided by the embodiment of the invention, the current version value and the target version value corresponding to the matching data are obtained, when the current version value is smaller than the target version value, the matching data is not the latest data, and the matching data can be determined to need to be updated at the moment, so that the data in the local cache is the latest data, and the accuracy of data acquisition is ensured.
With reference to the first aspect, in a third implementation of the first aspect, the method further includes: when the matching data corresponding to the retrieval request information does not exist in the local cache, retrieving the matching data again after a preset time interval; and when cache information corresponding to the retrieval request information is not retrieved yet, sending the retrieval request information to a database corresponding to a local cache.
According to the data retrieval method provided by the embodiment of the invention, when the matching data corresponding to the retrieval request information does not exist in the local cache, the matching data is retrieved again after a preset time interval so as to determine whether the matching data corresponding to the retrieval request information is updated to the local cache, and if the cache information corresponding to the retrieval request information is not retrieved yet, the retrieval request information is sent to the database corresponding to the local cache so as to obtain the target data corresponding to the retrieval request information from the database, so that the corresponding target data is obtained according to the retrieval request information, and the data retrieval failure is avoided.
With reference to the third embodiment of the first aspect, in a fourth embodiment of the first aspect, the method further comprises: when the database does not return the target data corresponding to the retrieval request information, uploading the keywords corresponding to the target data to the database; detecting whether value information corresponding to the keywords returned by the database is acquired or not; and deleting the keywords in the database when the value information corresponding to the keywords returned by the database is acquired.
According to the data retrieval method provided by the embodiment of the invention, when the database does not return the target data corresponding to the retrieval request information, the keyword corresponding to the target data is uploaded to the database, so that other nodes corresponding to the database can monitor the keyword, and the corresponding data is updated to the cache. When detecting that the current node acquires the value information corresponding to the keyword returned by the database, deleting the keyword in the database to prevent other nodes from continuously monitoring the keyword and simultaneously prevent other nodes from continuously updating the value information corresponding to the keyword.
With reference to the first aspect, in a fifth implementation of the first aspect, the method further comprises: and when the matching data does not need to be updated, taking the matching data as target data corresponding to the retrieval request information.
According to the data retrieval method provided by the embodiment of the invention, when the matching data does not need to be updated, the matching data is represented as the latest data, and at the moment, the matching data can be directly used as the target data corresponding to the retrieval request information. The method reduces the access pressure of the database by retrieving the required data from the local cache without accessing the database.
With reference to the first aspect, in a sixth implementation of the first aspect, the method further includes: detecting whether the data in the database is updated or not; and when the data in the database are detected to be updated, storing the updated data into the local cache corresponding to each node, wherein the local cache stores the data based on Hash mapping.
According to the data retrieval method provided by the embodiment of the invention, whether the data are updated or not is judged by comparing the data version of the target data in the local cache with the data version of the target data in the database, when the data in the database are detected to be updated, the updated data are stored in the local cache corresponding to each node, and the local cache stores the data based on Hash mapping and realizes an optimistic locking mechanism based on data version comparison, so that the data corresponding to each node are ensured to be consistent, and the concurrency performance of the system is improved.
According to a second aspect, an embodiment of the present invention provides a data retrieval apparatus, including: the acquisition module is used for acquiring retrieval request information of the target data; the first judgment module is used for judging whether the matching data corresponding to the retrieval request information exists in a local cache or not; the second judging module is used for judging whether the matched data needs to be updated or not when the matched data corresponding to the retrieval request information exists in the local cache; and the updating module is used for acquiring the updated data from the database corresponding to the local cache when the matched data needs to be updated, updating the updated data to the local cache, and taking the updated data as the target data.
According to a third aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, and the processor executing the computer instructions to perform the data retrieval method according to the first aspect or any embodiment of the first aspect.
According to a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where computer instructions are stored, and the computer instructions are configured to cause a computer to execute the data retrieval method according to the first aspect or any implementation manner of the first aspect.
It should be noted that, for corresponding beneficial effects of the data retrieval device, the electronic device and the computer-readable storage medium provided in the embodiment of the present invention, please refer to the description of corresponding contents in the data retrieval method, which is not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow diagram of a data retrieval method according to an embodiment of the present invention;
FIG. 2 is another flow diagram of a data retrieval method according to an embodiment of the present invention;
FIG. 3 is another flow diagram of a data retrieval method according to an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of a distributed system according to an embodiment of the invention;
FIG. 5 is a data structure diagram of a local cache according to an embodiment of the invention;
FIG. 6 is a detailed diagram of a data retrieval method according to an embodiment of the invention;
FIG. 7 is a block diagram of a data retrieval apparatus according to an embodiment of the present invention;
fig. 8 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
According to the technical scheme, when the node receives the retrieval request information, the corresponding data is obtained from the local cache, if the data is not the latest data, the latest data corresponding to the retrieval request information is obtained from the database, the latest data corresponding to the retrieval request information can be obtained from the local cache, the access pressure of the database is reduced, the system performance and the data processing capacity of the distributed system are improved, and therefore the retrieval efficiency of the data is improved.
The overall design scheme of the distributed system according to the technical scheme of the invention is shown in fig. 4, and the distributed system comprises a plurality of nodes and corresponding databases, wherein the plurality of nodes are API nodes, Engine nodes, schedule nodes and the like in a cloud platform, and the database is an ETCD database. Each node includes a main program and a cache module. The cache module comprises a cache monitoring module, a cache updating module and a cache searching module. Each node of the distributed system stores data of the ETCD database through a cache module, wherein the cache monitoring module is used for monitoring changed key-value data in the ETCD database; the cache updating module is responsible for updating the new data acquired by the cache monitoring module; the cache searching module is responsible for searching the data in the cache.
The data structure of the local cache in the technical scheme of the invention is shown in fig. 5, the HashMap is adopted to store data, and the time complexity of data retrieval is ensured to be O (1) through the Hash value, so that the data retrieval efficiency is improved. The data structure totally adopts 3 HashMap storage data, including: VersionMap for data version, DataMap for buffered data, and TypeMap for data type. The VersionMap is used for storing a storage directory, a data version and a Value address of data of the data, the storage directory of the data is used for locating a storage path of the data in the ETCD, the data version is used for realizing optimistic locking, and whether the local cache data needs to be updated or not is determined by comparing the data version of the local cache data with the data version of the ETCD database; the Value address of the data can quickly acquire the data Value by moving the pointer, so that the memory space occupied by the VersionMap is reduced, and the data retrieval efficiency is improved; the DataMap is used for storing a storage directory for storing data, a data version and a Value of the data, wherein the Value of the data is used for representing a real data storage position, and when cache data is searched, the corresponding Value is retrieved according to a Key of the data; the TypeMap is used for storing a storage path of the data in the ETCD database and a type corresponding to the data, and judging whether the current data exists in a local cache or not by combining the VersionMap.
In accordance with an embodiment of the present invention, there is provided an embodiment of a data retrieval method, it should be noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
In this embodiment, a data retrieval method is provided, which can be used in electronic devices, such as servers, computers, and the like, corresponding to each node of a distributed system, and fig. 1 is a flowchart of the data retrieval method according to an embodiment of the present invention, as shown in fig. 1, the flowchart includes the following steps:
s11, search request information of the target data is acquired.
The target data is data required to be retrieved by the electronic device, the retrieval request information is retrieval information corresponding to the data retrieval request, a user can send the data retrieval request to the electronic device adopting the distributed system through a communication interface of the client, and correspondingly, the electronic device corresponding to the current node can receive the data retrieval request to determine the retrieval request information to be characterized.
And S12, judging whether the local cache corresponding to the current node has the matching data corresponding to the retrieval request information.
The matching data is the data corresponding to the retrieval request information, the current node is the node for processing the retrieval request information, and the current node corresponds to the corresponding local cache. When the electronic equipment distributes the retrieval request information to the current node, the current node is controlled to inquire the corresponding local cache of the current node so as to determine whether the matching data corresponding to the retrieval request information exists in the corresponding local cache. When there is matching data corresponding to the retrieval request information in the local cache, step S13 is executed, otherwise, other operations are executed, where the other operations may continue to wait for the local cache to pull data corresponding to the retrieval request information from the database, or send data request information to the database, and obtain data corresponding to the retrieval request information from the database.
And S13, judging whether the matching data needs to be updated.
When the matching data corresponding to the retrieval request information exists in the local cache, continuously judging whether the matching data is the data of the latest version, specifically, the data in the local cache is stored in a data cache code packet form, and the electronic device corresponding to the current node can detect whether the data in the local cache is the data of the latest version through Check and Get interfaces provided by the data cache code packet. I.e. detecting whether the matching data needs to be updated, when the matching data needs to be updated, executing step S14, otherwise, directly returning the matching data to the user client. Specifically, the electronic device corresponding to the current node may obtain data corresponding to the matching data from a database corresponding to the local cache, determine whether the matching data in the local cache is consistent with the data in the database, and if not, indicate that the matching data needs to be updated.
And S14, acquiring the updated data from the database corresponding to the local cache, and storing the updated data in the local cache to take the updated data as the target data.
The update data is the latest version of data corresponding to the match data, and when the match data needs to be updated, the electronic device may obtain the update data from the database corresponding to the local cache, store the update data in the local cache corresponding to the current node, and return the update data serving as the target data to the user client.
The data retrieval method provided in this embodiment determines whether matching data corresponding to the retrieval request information exists in a local cache corresponding to a current node by obtaining retrieval request information of target data and querying based on the local cache corresponding to the current node of the retrieval request information, further determines whether the matching data needs to be updated when the matching data corresponding to the retrieval request information exists in the local cache, and obtains updated data from a database corresponding to the local cache if the matching data needs to be updated, stores the updated data in the local cache, and uses the updated data as the target data corresponding to the retrieval request information, so that the target data corresponding to the retrieval request information can be obtained from the local cache, and the target data is latest cache data, thereby ensuring that the target data obtained from each node are consistent, the data consistency among all the nodes is guaranteed, meanwhile, the target data are obtained from the local cache, the access pressure of the database is reduced, and the data retrieval efficiency, the system performance of the distributed system and the data processing capacity are improved.
In this embodiment, a data retrieval method is provided, which can be used in electronic devices employing a distributed system, such as a server, a computer, etc., and fig. 2 is a flowchart of the data retrieval method according to an embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
s21, search request information of the target data is acquired. For a detailed description, refer to the corresponding related description of the above embodiments, which is not repeated herein.
And S22, judging whether the local cache corresponding to the current node has the matching data corresponding to the retrieval request information. When there is matching data corresponding to the retrieval request information in the local cache, step S23 is executed, otherwise, step S25 is executed.
And S23, when the matching data corresponding to the retrieval request information exists in the local cache, judging whether the matching data needs to be updated.
Specifically, the step S23 may include:
and S231, acquiring a current version value and a target version value corresponding to the matching data.
The current version value is a data version corresponding to the matching data in the local cache, the target version value is a data version corresponding to the matching data in the ETCD database, specifically, a storage path of the matching data in the ETCD database can be located through a storage directory of the data in the VersionMap, and the ETCD database is inquired according to the storage path so as to determine a data version of the matching data in the ETCD database.
S232, whether the matched data needs to be updated or not is determined based on the magnitude relation between the current version value and the target version value.
And comparing the current version value with the target version value to determine whether the current version value is the same as the target version value, if the current version value is different from the target version value, indicating that the matching data in the local cache is inconsistent with the matching data in the ETCD database, and determining that the matching data in the local cache needs to be updated.
Specifically, the step S232 may include:
(1) it is determined whether the current version value is equal to the target version value.
And (3) the data in the ETCD database is the latest data, the current node can compare the current version value corresponding to the matching data with the target version value to determine whether the current version value is equal to the target version value, and if the current version value is smaller than the target version value, the step (2) is executed, otherwise, the matching data in the local cache corresponding to the current node is the latest version data, namely the data does not need to be updated.
(2) And if the current version value is smaller than the target version value, judging that the matched data needs to be updated.
If the current version value is smaller than the target version value, the matching data in the local cache corresponding to the current node is older data and not the latest version data, and the matching data in the local caches corresponding to other nodes may be the latest version data, so that the matching data in the local cache corresponding to the current node may not be consistent with the matching data in the local caches corresponding to other nodes, and data consistency among the nodes is affected.
And S24, when the matched data needs to be updated, acquiring the updated data from the database corresponding to the local cache, and storing the updated data in the local cache to take the updated data as the target data. For a detailed description, refer to the corresponding related description of the above embodiments, which is not repeated herein.
And S25, when the matching data corresponding to the retrieval request information does not exist in the local cache, retrieving the matching data again after a preset time interval.
The preset time is a time for the data retrieval process to sleep to wait for the current node to update the local cache, and the preset time may be 1s, 1.5s, or 2s, or may be other times, which is not specifically limited herein, and may be determined by a person skilled in the art according to actual needs.
And when the matching data corresponding to the retrieval request information does not exist in the local cache, the data retrieval process of the current node sleeps for a preset time, and the data retrieval process is executed again after the preset time so as to acquire the matching data corresponding to the retrieval request information from the local cache.
And S26, when the cache information corresponding to the retrieval request information is not retrieved yet, sending the retrieval request information to the database corresponding to the local cache.
If the cache information corresponding to the retrieval request information is not queried in the local cache when the data retrieval process is executed after the preset time, the electronic device corresponding to the current node can send the retrieval request information to the ETCD database at the moment, and target data corresponding to the retrieval request information is queried from the ETCD database.
And S27, when the database does not return the target data corresponding to the retrieval request information, uploading the keywords corresponding to the target data to the database.
If the ETCD database does not return the target data corresponding to the retrieval request information due to factors such as network interruption and the like, the data retrieval is not hit, and at the moment, keywords corresponding to the target data can be uploaded to the ETCD database, so that each node corresponding to the ETCD database can monitor the target data, and the data consistency among the nodes is ensured.
And S28, detecting whether value information corresponding to the key returned by the database is acquired.
The current node continuously acquires a keyword (namely, a Key of local cache missing data) corresponding to target data through a watch monitoring mechanism, and sends a data searching request to the ETCD database so as to detect whether value information corresponding to the keyword returned by the ETCD database is received in real time, wherein the value information is the target data corresponding to the retrieval request information. When the value information corresponding to the key word returned by the database is acquired, step S29 is executed, otherwise, the monitoring is continued.
And S29, deleting the keywords in the database.
When the value information corresponding to the keyword returned by the database is acquired, the target data of the retrieval request information is stored in the local cache of the current node, at this time, the current node can inform the ETCD database, and the keyword in the ETCD database is deleted, so that other nodes are prevented from continuously monitoring the keyword, and the local cache of the current node is prevented from continuously updating the data corresponding to the keyword.
In the data retrieval method provided by this embodiment, by obtaining the current version value and the target version value corresponding to the matching data, when the current version value is smaller than the target version value, it indicates that the matching data is not the latest data, and at this time, it may be determined that the matching data needs to be updated, so that it is ensured that the data in the local cache is the latest data, and the accuracy of data acquisition is ensured. When the matching data corresponding to the retrieval request information does not exist in the local cache, the matching data is retrieved again after a preset time interval to determine whether the matching data corresponding to the retrieval request information is updated to the local cache, and if the cache information corresponding to the retrieval request information is not retrieved yet, the retrieval request information is sent to the database corresponding to the local cache to obtain the target data corresponding to the retrieval request information from the database, so that the corresponding target data is obtained according to the retrieval request information, and the data retrieval failure is avoided. And when the database does not return the target data corresponding to the retrieval request information, uploading the key words corresponding to the target data to the database so that other nodes corresponding to the database can monitor the key words and update the corresponding data to the cache. When detecting that the current node acquires the value information corresponding to the keyword returned by the database, deleting the keyword in the database to prevent other nodes from continuously monitoring the keyword and simultaneously prevent other nodes from continuously updating the value information corresponding to the keyword.
In this embodiment, a data retrieval method is provided, which can be used in electronic devices employing a distributed system, such as a server, a computer, etc., and fig. 3 is a flowchart of the data retrieval method according to an embodiment of the present invention, as shown in fig. 3, the flowchart includes the following steps:
s31, search request information of the target data is acquired. For a detailed description, refer to the corresponding related description of the above embodiments, which is not repeated herein.
And S32, judging whether the local cache corresponding to the current node has the matching data corresponding to the retrieval request information. For a detailed description, refer to the corresponding related description of the above embodiments, which is not repeated herein.
And S33, when the matching data corresponding to the retrieval request information exists in the local cache, judging whether the matching data needs to be updated. When the matching data needs to be updated, step S34 is performed, otherwise step S35 is performed.
And S34, when the matched data needs to be updated, acquiring the updated data from the database corresponding to the local cache, and storing the updated data in the local cache to take the updated data as the target data. For a detailed description, refer to the corresponding related description of the above embodiments, which is not repeated herein.
And S35, when the matching data does not need to be updated, using the matching data as the target data corresponding to the retrieval request information.
When the matching data does not need to be updated, the matching data in the local cache corresponding to the current node is the data of the latest version, and at this time, the electronic device corresponding to the current node may use the matching data as the target data corresponding to the retrieval request information and return the target data to the user client.
Optionally, before step S31, the method may further include:
s301, whether the data in the database is updated or not is detected.
In the process of initializing the distributed system, each corresponding node can acquire data in the ETCD database through a watch monitoring mechanism, store the data in VersionMap and DataMap cached locally, and store data directory field information needing caching in TypeMap. Each node can monitor the data Key in the ETCD database in real time through a watch monitoring mechanism to determine whether the data is updated. And when detecting that the data in the database is updated, executing the step S302, otherwise, continuing to monitor.
S302, storing the updated data into local caches corresponding to the nodes, wherein the local caches store the data based on Hash mapping.
When the data in the ETCD database is detected to be updated, each node corresponding to the ETCD database executes data updating operation, and the updated data in the ETCD database is stored in the local cache corresponding to each node, so that the stored data in the local cache is in the latest state. The local cache stores data based on hash mapping, and realizes an optimistic lock mechanism based on data version comparison, as shown in fig. 5, the time complexity of data retrieval is ensured to be O (1) through a hash value, so as to improve the data retrieval efficiency.
In the data retrieval method provided by this embodiment, when the matching data does not need to be updated, it indicates that the matching data is the latest data, and at this time, the matching data may be directly used as the target data corresponding to the retrieval request information. According to the method, the required data are retrieved from the local cache, the database does not need to be accessed, and the access pressure of the database is reduced. When the data in the database are detected to be updated, the updated data are stored in the local caches corresponding to the nodes, and the local caches store the data based on Hash mapping and realize an optimistic locking mechanism based on data version comparison, so that the data corresponding to the nodes are ensured to be consistent, and the concurrency performance of the system is improved.
In this embodiment, the data retrieval method is described with a specific example, as shown in fig. 6, when a system is initialized, each node acquires data in an ETCD database through a watch monitoring mechanism, stores the data in a VersionMap or DataMap of a local cache, and stores data directory field information to be cached in a TypeMap. When the node monitors that the data Key in the ETCD database is changed through the watch monitoring mechanism, the data in the ETCD database is updated, the node can execute data updating operation, and the updated data in the ETCD database is stored in a local cache, so that the cache data in the local cache is in a latest state.
A user may send search request information of the data instance2 through a client, and a proxy node may allocate the search request information to a certain node, for example, node 1 in fig. 6, when receiving the search request information, node 1 searches data cached locally according to the search request information.
When the node 1 determines that the target data corresponding to the instance2 (i.e. the cache data corresponding to the instance2 exists in the local cache) does not exist in the VersionMap, the data retrieval process of the node 1 sleeps for 1 second to wait for the watch snoop mechanism to update the local cache of the node. After 1 second, if the target data corresponding to the instance2 is not retrieved in the local cache, the data lookup request is sent to the ETCD database again to obtain the data corresponding to the instance 2. If the data search request cannot acquire the data corresponding to the instance2 due to factors such as network interruption, it indicates that the data is not hit this time.
When the data is not hit, the node 1 reports the instance2 data to the MissingCache directory of the ETCD, and stores the Key of the missing data into the directory, so that each node can monitor the data. Meanwhile, the node 1 continuously acquires the Key of the missing data through a watch monitoring mechanism, and sends a data searching request to the ETCD database to retrieve the data. Only after the node 1 successfully acquires the data, the instance2 data in the MissingCache directory is deleted, so as to prevent other nodes from continuously monitoring and updating the data.
When other nodes monitor that the MisingCache directory has cache data instance2 to be updated, if the local cache corresponding to the node does not have instance2 data, the instance2 data can be directly obtained from the ETCD database and stored in the local cache. If the local cache corresponding to the node has instance2 data, comparing whether the version A of the instance2 data in the ETCD database is consistent with the version B of the instance2 data in the local cache. If the version A > the version B, the instance2 data version of the local cache is older and needs to be updated to the latest version, otherwise, the update operation of the local cache data is not performed.
The embodiment can realize the consistency of local cache data among multiple nodes, and effectively reduce the problem of data retrieval delay; a non-centralized cache storage method is adopted to replace the traditional redis component, so that the probability of inconsistency of cache data and database data is reduced, and the problem of data consistency of double writing of redis and the database is avoided; the watch mechanism of the ETCD database is used for replacing message transmission of MQ components, the transfer times of data in the network are reduced, the system node can obtain the function similar to MQ message subscription only by adding watch monitoring, and therefore the system maintenance difficulty is reduced while the data consistency is ensured.
In this embodiment, a data retrieval apparatus is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, which have already been described and are not described again. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
The present embodiment provides a data retrieval apparatus, as shown in fig. 7, including:
the obtaining module 41 is configured to obtain the retrieval request information of the target data. For a detailed description, reference is made to the corresponding related description of the above method embodiments, which is not repeated herein.
The first determining module 42 is configured to determine whether matching data corresponding to the retrieval request information exists in a local cache corresponding to the current node. For a detailed description, reference is made to the corresponding related description of the above method embodiments, which is not repeated herein.
The second judging module 43 is configured to judge whether the matching data needs to be updated when the matching data corresponding to the retrieval request information exists in the local cache. For a detailed description, reference is made to the corresponding related description of the above method embodiments, which is not repeated herein.
And the updating module 44 is configured to, when the matching data needs to be updated, obtain the update data from the database corresponding to the local cache, and store the update data in the local cache, where the update data is used as the target data. For a detailed description, reference is made to the corresponding related description of the above method embodiments, which is not repeated herein.
The data retrieval device provided in this embodiment determines whether matching data corresponding to the retrieval request information exists in a local cache corresponding to a current node by obtaining retrieval request information of target data and querying based on the local cache corresponding to the current node of the retrieval request information, further determines whether the matching data needs to be updated when the matching data corresponding to the retrieval request information exists in the local cache, and obtains updated data from a database corresponding to the local cache if the matching data needs to be updated, stores the updated data in the local cache, and uses the updated data as the target data corresponding to the retrieval request information, so that the target data corresponding to the retrieval request information can be obtained from the local cache, and the target data is latest cache data, thereby ensuring that the target data obtained from each node are consistent, the data consistency among all the nodes is guaranteed, meanwhile, the target data are obtained from the local cache, the access pressure of the database is reduced, and the data retrieval efficiency, the system performance of the distributed system and the data processing capacity are improved.
The data retrieval means in this embodiment is presented in the form of functional units, where a unit refers to an ASIC circuit, a processor and memory executing one or more software or fixed programs, and/or other devices that may provide the above-described functionality.
Further functional descriptions of the modules are the same as those of the corresponding embodiments, and are not repeated herein.
An embodiment of the present invention further provides an electronic device, which has the data retrieval apparatus shown in fig. 7.
Referring to fig. 8, fig. 8 is a schematic structural diagram of an electronic device according to an alternative embodiment of the present invention, and as shown in fig. 8, the electronic device may include: at least one processor 501, such as a CPU (Central Processing Unit), at least one communication interface 503, memory 504, and at least one communication bus 502. Wherein a communication bus 502 is used to enable connective communication between these components. The communication interface 503 may include a Display (Display) and a Keyboard (Keyboard), and the optional communication interface 503 may also include a standard wired interface and a standard wireless interface. The Memory 504 may be a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The memory 504 may optionally be at least one storage device located remotely from the processor 501. Wherein the processor 501 may be in connection with the apparatus described in fig. 7, an application program is stored in the memory 504, and the processor 501 calls the program code stored in the memory 504 for performing any of the above-mentioned method steps.
The communication bus 502 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 502 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 8, but this is not intended to represent only one bus or type of bus.
The memory 504 may include a volatile memory (RAM), such as a random-access memory (RAM); the memory may also include a non-volatile memory (english: non-volatile memory), such as a flash memory (english: flash memory), a hard disk (english: hard disk drive, abbreviated: HDD) or a solid-state drive (english: SSD); the memory 504 may also comprise a combination of the above types of memory.
The processor 501 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of CPU and NP.
The processor 501 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.
Optionally, the memory 504 is also used to store program instructions. Processor 501 may invoke program instructions to implement the data retrieval method as shown in the embodiments of fig. 1-3 of the present application.
The embodiment of the invention also provides a non-transitory computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions can execute the processing method of the data retrieval method in any method embodiment. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. A method of data retrieval, comprising:
acquiring retrieval request information of target data;
judging whether matching data corresponding to the retrieval request information exists in a local cache corresponding to the current node;
when the matching data corresponding to the retrieval request information exists in the local cache, judging whether the matching data needs to be updated or not;
and when the matched data needs to be updated, acquiring updated data from a database corresponding to the local cache, and storing the updated data to the local cache, wherein the updated data is used as the target data.
2. The method of claim 1, wherein the determining whether the matching data needs to be updated comprises:
acquiring a current version value and a target version value corresponding to the matching data;
and determining whether the matching data needs to be updated or not based on the size relation between the current version value and the target version value.
3. The method of claim 2, wherein determining whether the matching data needs to be updated based on a magnitude relationship between the current version value and the target version value comprises:
judging whether the current version value is equal to the target version value;
and when the current version value is smaller than the target version value, judging that the matching data needs to be updated.
4. The method of claim 1, further comprising:
when the matching data corresponding to the retrieval request information does not exist in the local cache, retrieving the matching data again after a preset time interval;
and when cache information corresponding to the retrieval request information is not retrieved yet, sending the retrieval request information to a database corresponding to a local cache.
5. The method of claim 4, further comprising:
when the database does not return the target data corresponding to the retrieval request information, uploading the keywords corresponding to the target data to the database;
detecting whether value information corresponding to the keywords returned by the database is acquired or not;
and deleting the keywords in the database when the value information corresponding to the keywords returned by the database is acquired.
6. The method of claim 1, further comprising:
and when the matching data does not need to be updated, taking the matching data as target data corresponding to the retrieval request information.
7. The method of claim 1, further comprising:
detecting whether the data in the database is updated or not;
and when the data in the database are detected to be updated, storing the updated data into the local cache corresponding to each node, wherein the local cache stores the data based on Hash mapping.
8. A data retrieval device, comprising:
the acquisition module is used for acquiring retrieval request information of the target data;
the first judgment module is used for judging whether the matching data corresponding to the retrieval request information exists in a local cache or not;
the second judging module is used for judging whether the matched data needs to be updated or not when the matched data corresponding to the retrieval request information exists in the local cache;
and the updating module is used for acquiring the updated data from the database corresponding to the local cache when the matched data needs to be updated, updating the updated data to the local cache, and taking the updated data as the target data.
9. An electronic device, comprising:
a memory and a processor, the memory and the processor being communicatively coupled to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the data retrieval method of any of claims 1-7.
10. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the data retrieval method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111600536.3A CN114461678A (en) | 2021-12-24 | 2021-12-24 | Data retrieval method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111600536.3A CN114461678A (en) | 2021-12-24 | 2021-12-24 | Data retrieval method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114461678A true CN114461678A (en) | 2022-05-10 |
Family
ID=81407708
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111600536.3A Pending CN114461678A (en) | 2021-12-24 | 2021-12-24 | Data retrieval method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114461678A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858593A (en) * | 2023-02-22 | 2023-03-28 | 昆明昆船逻根机场系统有限公司 | Luggage data retrieval method and system based on memory cache |
-
2021
- 2021-12-24 CN CN202111600536.3A patent/CN114461678A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115858593A (en) * | 2023-02-22 | 2023-03-28 | 昆明昆船逻根机场系统有限公司 | Luggage data retrieval method and system based on memory cache |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10795817B2 (en) | Cache coherence for file system interfaces | |
| US8548945B2 (en) | Database caching utilizing asynchronous log-based replication | |
| US10579615B2 (en) | Method for data retrieval from a distributed data storage system | |
| US9442858B2 (en) | Solid state drives as a persistent cache for database systems | |
| US9529731B1 (en) | Contention-free approximate LRU for multi-threaded access | |
| CN107818111B (en) | Method for caching file data, server and terminal | |
| US10528590B2 (en) | Optimizing a query with extrema function using in-memory data summaries on the storage server | |
| US20200134043A1 (en) | Duplicate Request Checking for File System Interfaces | |
| US11799961B2 (en) | File directory traversal method, apparatus, device, and medium | |
| CN113032335A (en) | File access method, device, equipment and storage medium | |
| US20140164549A1 (en) | Managing direct attached cache and remote shared cache | |
| CN111399760B (en) | NAS cluster metadata processing method and device, NAS gateway and medium | |
| CN114461678A (en) | Data retrieval method and device, electronic equipment and readable storage medium | |
| CN110311975B (en) | Data request processing method and device | |
| US8549274B2 (en) | Distributive cache accessing device and method for accelerating to boot remote diskless computers | |
| US10936500B1 (en) | Conditional cache persistence in database systems | |
| CN114238518A (en) | Data processing method, device, equipment and storage medium | |
| CN113703917A (en) | Multi-cluster resource data processing system and method and non-transient storage medium | |
| US10402373B1 (en) | Filesystem redirection | |
| US10067678B1 (en) | Probabilistic eviction of partial aggregation results from constrained results storage | |
| CN113722362B (en) | Remote-based cache data query method, device and system | |
| CN114610691B (en) | Storage object acquisition method, storage object acquisition device, equipment and medium | |
| US11762854B2 (en) | Extended caching and query-time validation | |
| CN116257521B (en) | KV storage method based on FPGA | |
| CN107172447B (en) | Data loading method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |