CN117914944A - Distributed three-level caching method and device based on Internet of things - Google Patents

Distributed three-level caching method and device based on Internet of things Download PDF

Info

Publication number
CN117914944A
CN117914944A CN202410316945.8A CN202410316945A CN117914944A CN 117914944 A CN117914944 A CN 117914944A CN 202410316945 A CN202410316945 A CN 202410316945A CN 117914944 A CN117914944 A CN 117914944A
Authority
CN
China
Prior art keywords
metadata
internet
things
memory cache
distributed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202410316945.8A
Other languages
Chinese (zh)
Inventor
孙壮
庞国涛
谢超君
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Dark Matter Beijing Intelligent Technology Co ltd
DMAI Guangzhou Co Ltd
Original Assignee
Dark Matter Beijing Intelligent Technology Co ltd
DMAI Guangzhou Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Dark Matter Beijing Intelligent Technology Co ltd, DMAI Guangzhou Co Ltd filed Critical Dark Matter Beijing Intelligent Technology Co ltd
Priority to CN202410316945.8A priority Critical patent/CN117914944A/en
Publication of CN117914944A publication Critical patent/CN117914944A/en
Pending legal-status Critical Current

Links

Classifications

    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application provides a distributed three-level caching method and device based on the Internet of things, wherein the method comprises the following steps: storing the equipment metadata of the client equipment of the Internet of things into a preset basic database according to preset service requirements, responding to the equipment metadata request sent by the client equipment of the Internet of things to the Internet of things system, inquiring and acquiring the equipment metadata of the client equipment of the Internet of things in a preset multi-level cache system based on the equipment metadata request, updating the data of the target equipment metadata in the basic database, and updating the target equipment metadata cached in the local memory cache library and the distributed memory cache library.

Description

Distributed three-level caching method and device based on Internet of things
Technical Field
The application relates to the technical field of the Internet of things, in particular to a distributed three-level caching method and device based on the Internet of things.
Background
In an internet of things (IoT, internet of Things) scenario, an internet of things device needs to query and acquire its device metadata (i.e., basic information of the device) to report, for example, the device metadata is frequently read by a database to report the device data, but when the device metadata is read in a large scale, huge pressure is caused on the database.
At present, when an application program is accessed by the Internet of things device, the application program firstly acquires cache data from the distributed cache according to the key, and if the cache data corresponding to the key does not exist in the distributed cache, the application program queries the data from the MySQL database and returns the data to the Internet of things device, and meanwhile, the data is updated to the distributed cache.
However, as the number of physical network devices increases, the resources of the distributed cache are limited, so that the distributed cache pressure is high, the performance of the cache system is low, the data reporting speed of the internet of things device is influenced, the database burden caused by device metadata query is high, and in addition, when the data cached at one end is updated, the problem of inconsistent data cached at multiple ends is often caused.
Disclosure of Invention
In view of the above, the application aims to provide a distributed three-level caching method and device based on the internet of things, by adopting a three-level caching strategy, device metadata is obtained by inquiring in a multi-level caching system based on a device metadata request sent by an internet of things client device to the internet of things system, so that the distributed caching pressure is reduced, the performance of the caching system is improved, the speed of reporting data by the internet of things device is improved, and the problem of database burden caused by device metadata inquiry under a huge amount of internet of things devices is effectively solved. Meanwhile, when the metadata of the target equipment is updated, the local memory cache library and the distributed memory cache library are synchronously updated, so that the data consistency of a plurality of caches in the multi-level cache system is improved.
In a first aspect, an embodiment of the present application provides a distributed three-level caching method based on the internet of things, where the method includes:
Storing the equipment metadata of the client equipment of the Internet of things into a preset basic database according to preset service requirements;
Responding to the equipment metadata request sent by the Internet of things client equipment to an Internet of things system, and inquiring and acquiring the equipment metadata of the Internet of things client equipment in a preset multi-level cache system based on the equipment metadata request; the multi-level cache system comprises a local memory cache library, a distributed memory cache library and the basic database;
And updating the data of the target device metadata in the basic database, and updating the target device metadata cached in the local memory cache library and the distributed memory cache library.
In a possible implementation manner, the querying and acquiring the device metadata of the client device of the internet of things in a multi-level cache system based on the device metadata request includes:
inquiring and acquiring the equipment metadata of the client equipment of the Internet of things in the local memory cache library based on the equipment metadata request;
In response to the device metadata being unable to be obtained in the local memory cache, querying and obtaining device metadata of the client device of the internet of things from a distributed memory cache, and writing the device metadata into the local memory cache;
And in response to the fact that the device metadata cannot be acquired in the distributed memory cache, querying and acquiring the device metadata of the client device of the Internet of things from the basic database, and writing the device metadata into the local memory cache and the distributed memory cache.
In a possible implementation manner, the querying and acquiring the device metadata of the client device of the internet of things in a multi-level cache system based on the device metadata request includes:
inquiring and acquiring the equipment metadata of the client equipment of the Internet of things in the local memory cache library based on the equipment metadata request;
In response to the device metadata being unable to be obtained in the local memory cache, querying and obtaining device metadata of the client device of the internet of things from a distributed memory cache, and writing the device metadata into the local memory cache;
And in response to the fact that the device metadata cannot be acquired in the distributed memory cache, querying and acquiring the device metadata of the client device of the Internet of things from the basic database, and writing the device metadata into the local memory cache and the distributed memory cache.
In a possible implementation manner, the deleting the target device metadata corresponding to the key in the local memory cache bank and the distributed memory cache bank includes:
transmitting corresponding data invalidation operation to the distributed memory cache library to delete target equipment metadata corresponding to the secret key in the distributed memory cache library;
Triggering the data invalidation operation of the local memory cache library to delete the target device metadata in the local memory cache library.
In one possible implementation manner, the distributed three-level caching method based on the internet of things further comprises:
Acquiring a message subscription mode;
Target device metadata in the distributed local repository is invalidated based on the message subscription pattern.
In a possible implementation manner, the invalidating the target device metadata in the preset distributed local cache library based on the message subscription mode includes:
transmitting a key deletion message corresponding to the key to a target channel based on the message subscription mode;
And responding to the subscription of the distributed local cache library to the target channel, receiving the key deletion message based on the target channel and invalidating corresponding target device metadata in the distributed local cache library.
In a possible implementation manner, the storing, according to a preset service requirement, device metadata of the client device of the internet of things into a preset base database includes:
Creating a device metadata table of the client device of the internet of things; the device metadata table at least comprises an attribute name, an attribute type and a device type which are reported by the client device of the Internet of things;
And defining the equipment data of the client equipment of the Internet of things as equipment metadata based on the equipment metadata table and the service requirement, and storing the equipment metadata into the basic database.
In a second aspect, an embodiment of the present application further provides a distributed three-level caching apparatus based on the internet of things, where the apparatus includes:
The storage module is used for storing the equipment metadata of the client equipment of the Internet of things into a preset basic database according to preset service requirements;
The query module is used for responding to the equipment metadata request sent by the client equipment of the Internet of things to the system of the Internet of things, and querying and acquiring the equipment metadata of the client equipment of the Internet of things in a preset multi-level cache system based on the equipment metadata request; the multi-level cache system comprises a local memory cache library, a distributed memory cache library and the basic database;
and the updating module is used for carrying out data updating on the target device metadata in the basic database and updating the target device metadata cached in the local memory cache library and the distributed memory cache library.
In a possible implementation manner, the query module is specifically configured to:
inquiring and acquiring the equipment metadata of the client equipment of the Internet of things in the local memory cache library based on the equipment metadata request;
In response to the device metadata being unable to be obtained in the local memory cache, querying and obtaining device metadata of the client device of the internet of things from a distributed memory cache, and writing the device metadata into the local memory cache;
And in response to the fact that the device metadata cannot be acquired in the distributed memory cache, querying and acquiring the device metadata of the client device of the Internet of things from the basic database, and writing the device metadata into the local memory cache and the distributed memory cache.
In a possible implementation manner, the updating module is specifically configured to:
inquiring and acquiring the equipment metadata of the client equipment of the Internet of things in the local memory cache library based on the equipment metadata request;
In response to the device metadata being unable to be obtained in the local memory cache, querying and obtaining device metadata of the client device of the internet of things from a distributed memory cache, and writing the device metadata into the local memory cache;
And in response to the fact that the device metadata cannot be acquired in the distributed memory cache, querying and acquiring the device metadata of the client device of the Internet of things from the basic database, and writing the device metadata into the local memory cache and the distributed memory cache.
In a possible implementation manner, the updating module is specifically configured to:
transmitting corresponding data invalidation operation to the distributed memory cache library to delete target equipment metadata corresponding to the secret key in the distributed memory cache library;
Triggering the data invalidation operation of the local memory cache library to delete the target device metadata in the local memory cache library.
In one possible implementation manner, the distributed three-level caching device based on the internet of things further includes:
the acquisition module is used for acquiring a message subscription mode;
and the invalidation module is used for invalidating the target device metadata in the distributed local cache library based on the message subscription mode.
In one possible implementation, the disabling module is specifically configured to:
transmitting a key deletion message corresponding to the key to a target channel based on the message subscription mode;
And responding to the subscription of the distributed local cache library to the target channel, receiving the key deletion message based on the target channel and invalidating corresponding target device metadata in the distributed local cache library.
In a possible implementation manner, the storage module is specifically configured to:
Creating a device metadata table of the client device of the internet of things; the device metadata table at least comprises an attribute name, an attribute type and a device type which are reported by the client device of the Internet of things;
And defining the equipment data of the client equipment of the Internet of things as equipment metadata based on the equipment metadata table and the service requirement, and storing the equipment metadata into the basic database.
In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor and the storage medium communicating over the bus when the electronic device is running, the processor executing the machine-readable instructions to perform the steps of the internet of things-based distributed three-level caching method of any of the first aspects.
In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, where a computer program is stored, where the computer program when executed by a processor performs the steps of the distributed three-level caching method according to any one of the first aspects, where the three-level caching method is based on the internet of things.
According to the distributed three-level caching method and device based on the Internet of things, device metadata of client devices of the Internet of things are stored in a preset basic database according to preset service requirements, device metadata requests are sent to an Internet of things system in response to the client devices of the Internet of things, device metadata of the client devices of the Internet of things are inquired and obtained in the preset multi-level caching system based on the device metadata requests, data updating is conducted on target device metadata in the basic database, and target device metadata cached in a local memory cache bank and a distributed memory cache bank are updated. According to the method and the device, the three-level caching strategy is adopted, the device metadata is obtained by inquiring the device metadata in the multi-level caching system based on the device metadata request sent by the client device of the Internet of things to the Internet of things system, so that the distributed caching pressure is reduced, the performance of the caching system is improved, the data reporting speed of the Internet of things device is improved, and the database burden problem caused by the device metadata inquiry under the condition of a large number of devices of the Internet of things is effectively solved. Meanwhile, when the metadata of the target equipment is updated, the local memory cache library and the distributed memory cache library are synchronously updated, so that the data consistency of a plurality of caches in the multi-level cache system is improved.
In order to make the above objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a distributed three-level caching method based on the internet of things, which is provided by an embodiment of the application;
FIG. 2 is a schematic diagram of a multi-level cache system;
FIG. 3 is a flowchart of querying and acquiring device metadata of an Internet of things client device in a multi-level cache system based on a device metadata request according to an embodiment of the present application;
FIG. 4 is a flow chart providing for data updating of target device metadata in a base database and updating of target device metadata cached in a local memory cache and a distributed memory cache in accordance with an embodiment of the present application;
Fig. 5 is a flowchart of a distributed three-level caching method based on the internet of things according to another embodiment of the present application;
Fig. 6 is a flowchart of storing device metadata of an internet of things client device into a preset base database according to preset service requirements according to an embodiment of the present application;
Fig. 7 is a schematic structural diagram of a distributed three-level caching device based on the internet of things according to an embodiment of the present application;
Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application, and it should be understood that the drawings in the present application are for the purpose of illustration and description only and are not intended to limit the scope of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. A flowchart, as used in this disclosure, illustrates operations implemented according to some embodiments of the present application. It should be understood that the operations of the flow diagrams may be implemented out of order and that steps without logical context may be performed in reverse order or concurrently. Moreover, one or more other operations may be added to or removed from the flow diagrams by those skilled in the art under the direction of the present disclosure.
In addition, the described embodiments are only some, but not all, embodiments of the application. The components of the embodiments of the present application generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the application, as presented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present application.
It should be noted that the term "comprising" will be used in embodiments of the application to indicate the presence of the features stated hereafter, but not to exclude the addition of other features.
In view of the fact that in the scene of the internet of things, the internet of things equipment needs to inquire and acquire own equipment metadata (namely basic information of the equipment) for reporting, for example, the equipment metadata are frequently read by a database to report the equipment data, but huge pressure is caused to the database when the equipment metadata are read in a large scale. At present, when an application program is accessed by the Internet of things device, the application program firstly acquires cache data from the distributed cache according to the key, and if the cache data corresponding to the key does not exist in the distributed cache, the application program queries the data from the MySQL database and returns the data to the Internet of things device, and meanwhile, the data is updated to the distributed cache.
However, as the number of physical network devices increases, the resources of the distributed cache are limited, so that the distributed cache pressure is high, the performance of the cache system is low, the data reporting speed of the internet of things device is influenced, the database burden caused by device metadata query is high, and in addition, when the data cached at one end is updated, the problem of inconsistent data cached at multiple ends is often caused.
Aiming at the problem, the application provides a distributed three-level caching method and device based on the Internet of things, which are used for inquiring and obtaining device metadata in a multi-level caching system based on a device metadata request sent by client devices of the Internet of things to the Internet of things system by adopting a three-level caching strategy, so that the distributed caching pressure is reduced, the performance of the caching system is improved, the speed of reporting data by the Internet of things device is improved, and the problem of database burden caused by device metadata inquiry under huge amount of devices of the Internet of things is effectively solved. Meanwhile, when the metadata of the target equipment is updated, the local memory cache library and the distributed memory cache library are synchronously updated, so that the data consistency of a plurality of caches in the multi-level cache system is improved.
Fig. 1 is a flowchart of a distributed three-level caching method based on the internet of things, which is provided according to an embodiment of the present application, as shown in fig. 1, where the distributed three-level caching method based on the internet of things according to the embodiment of the present application specifically may include:
s101, storing the device metadata of the client device of the Internet of things into a preset basic database according to preset service requirements.
S102, responding to the device metadata request sent by the client device of the Internet of things to the system of the Internet of things, and inquiring and acquiring the device metadata of the client device of the Internet of things in a preset multi-level cache system based on the device metadata request.
S103, data updating is carried out on the target device metadata in the basic database, and the target device metadata cached in the local memory cache library and the distributed memory cache library are updated.
According to the distributed three-level caching method based on the Internet of things, the three-level caching strategy is adopted, and the device metadata is obtained by inquiring the device metadata in the multi-level caching system based on the device metadata request sent by the client device of the Internet of things to the Internet of things system, so that the distributed caching pressure is reduced, the performance of the caching system is improved, the data reporting speed of the Internet of things device is improved, and the database burden problem caused by the device metadata inquiry under the huge amount of devices of the Internet of things is effectively solved. Meanwhile, when the metadata of the target equipment is updated, the local memory cache library and the distributed memory cache library are synchronously updated, so that the data consistency of a plurality of caches in the multi-level cache system is improved.
The following fig. 1 and 2 illustrate the above exemplary steps of an embodiment of the present application with reference to specific examples:
s101, storing the equipment metadata of the client equipment of the Internet of things into a preset basic database according to preset service requirements.
In the embodiment of the present application, the service requirement is a preset requirement for the content of the required device metadata, for example, a service requirement requires that the device metadata include a device type, for example, as shown in fig. 2, the IOT device represents the internet of things client device, that is, the basic information of the internet of things client device, and the device metadata of the internet of things client device is stored in a preset basic database according to the preset service requirement for subsequent processing.
It should be noted that, in the present application, mySQL database is taken as a basic database as an example, but the basic database is not limited, and may be set according to practical situations, for example, as shown in fig. 2, mySQL clusters represent clusters formed by the basic database of the present application. As can be appreciated by those skilled in the art, the MySQL database is a relational database, is widely applied to large, medium and small Internet systems, has data persistence, and can ensure data consistency by supporting transactions. It will be appreciated that the query performance of MySQL databases appears to be a fly-through when faced with an excessive number of queries, since the bottleneck in the query performance of MySQL depends on the disk performance limitations.
S102, responding to the device metadata request sent by the client device of the Internet of things to the system of the Internet of things, and inquiring and acquiring the device metadata of the client device of the Internet of things in a preset multi-level cache system based on the device metadata request.
In the embodiment of the application, the internet of things system, namely the network system where the internet of things equipment is located, the equipment metadata request is the equipment metadata of the request internet of things client equipment, the multi-level cache system comprises a local memory cache library, a distributed memory cache library and a basic database, and when the internet of things client equipment sends the equipment metadata request to the internet of things system, the equipment metadata of the request corresponding to the equipment metadata request is queried in the multi-level cache system according to the equipment metadata request.
It should be noted that CAFFEINE CACHE is taken as a local memory cache bank and Redis (Remote Dictionary Server, remote dictionary service) cache is taken as a distributed memory cache bank as an example for description, but the application is not limited thereto and can be set according to practical situations. For example, as shown in FIG. 2, a micro-service cluster represents a cluster of local memory caches of the present application, and a Redis cluster represents a cluster of distributed memory caches of the present application. As can be appreciated by those skilled in the art, the Redis is an open-source memory cache database, which has many advantages, and the Redis data is stored in the memory, so that the read-write speed is very fast, the request can be generally responded at the level of sub-millisecond, and the Redis also naturally supports the distributed type, so that a distributed cache cluster can be constructed, high availability and load balance are provided, and the data reliability and the performance expansion are ensured; CAFFEINE CACHE is a memory cache in Java, because of the use of efficient data structures and data storage techniques, it provides data access at extremely high speed, provides rich cache policies, including LRU (LEAST RECENTLY Used, least recently Used elimination algorithm), LFU (Least Frequently Used, least frequently Used elimination algorithm), timing failure, etc., and can set the maximum cache size to avoid excessive occupation of local memory, when the cache occupation is excessive, CAFFEINE CACHE will automatically reject the least frequently Used data. Compared with MySQL stored by using a disk, CAFFEINE CACHE uses a memory to store data, so that the read-write speed is improved by thousands of times, and compared with Redis which is also read-write by using the memory, CAFFEINE CACHE has the advantage of storing data locally, and can save network bandwidth resources.
S103, data updating is carried out on the target device metadata in the basic database, and the target device metadata cached in the local memory cache library and the distributed memory cache library are updated.
In the embodiment of the application, the target device metadata is selected to be subjected to data updating, the target device metadata in the basic database is subjected to data updating, and the target device metadata cached in the local memory cache library and the distributed memory cache library are updated so as to improve the data consistency of the multi-level cache system.
According to the distributed three-level caching method based on the Internet of things, device metadata of client devices of the Internet of things are stored in a preset basic database according to preset service requirements, device metadata requests are sent to an Internet of things system in response to the client devices of the Internet of things, device metadata of the client devices of the Internet of things are inquired and obtained in a preset multi-level caching system based on the device metadata requests, data updating is conducted on target device metadata in the basic database, and target device metadata cached in a local memory cache bank and a distributed memory cache bank are updated. According to the distributed three-level caching method based on the Internet of things, the three-level caching strategy is adopted, and the device metadata is obtained by inquiring the device metadata in the multi-level caching system based on the device metadata request sent by the client device of the Internet of things to the Internet of things system, so that the distributed caching pressure is reduced, the performance of the caching system is improved, the data reporting speed of the Internet of things device is improved, and the database burden problem caused by the device metadata inquiry under the huge amount of devices of the Internet of things is effectively solved. Meanwhile, when the metadata of the target equipment is updated, the local memory cache library and the distributed memory cache library are synchronously updated, so that the data consistency of a plurality of caches in the multi-level cache system is improved.
Further, as shown in fig. 3, in step S102 in the foregoing embodiment, "query and obtain device metadata of an internet of things client device in a multi-level cache system based on a device metadata request", specifically may include the following steps:
s301, inquiring and acquiring device metadata of the client device of the Internet of things in a local memory cache library based on the device metadata request.
In the embodiment of the application, when the client device of the internet of things requests the device metadata request, the corresponding device metadata is firstly queried and obtained in the local memory cache bank CAFFEINE CACHE based on the device metadata request. Obviously, when the device metadata exists in the local memory cache bank, the device metadata can be acquired from the local memory cache bank for processing. It should be noted that, in the internet of things system, the speed of acquiring the device metadata in the local memory cache is very fast, so that the data can be directly acquired from the memory without creating a network connection and acquiring the data through a network IO.
It can be seen that the device metadata is obtained from the local memory cache, which shares the pressure of the distributed memory cache, and the processing speed is very fast, and no additional network request is needed, thereby saving network bandwidth resources.
S302, in response to the fact that the device metadata cannot be acquired in the local memory cache, the device metadata of the client device of the Internet of things are queried and acquired from the distributed memory cache, and the device metadata are written into the local memory cache.
In the embodiment of the present application, when the device metadata cannot be obtained in the local memory cache CAFFEINE CACHE (i.e., the local memory cache does not have the device metadata of the client device of the internet of things), the device metadata of the client device of the internet of things is queried from the distributed memory cache REDIS CACHE, if the device metadata of the client device of the internet of things exists in REDIS CACHE, the obtained device metadata is obtained for processing, and the obtained device metadata is written into the local memory cache CAFFEINE CACHE for supplementing, so that the device metadata can be conveniently obtained in the local memory cache when the next device metadata request is read. It should be noted that, under the condition that the acquisition of the device metadata in the local memory cache bank fails, the data is automatically acquired from the second-level cache, that is, the distributed memory cache bank REDIS CACHE, where the storage of the distributed memory cache bank REDIS CACHE is also based on the memory, so that the speed of acquiring the device metadata in REDIS CACHE is very fast, and only a part of network overhead is needed to be borne, so that the device metadata can be acquired.
It should be further noted that, if the device metadata cannot be obtained from the local memory cache CAFFEINE CACHE, generally, when the client device of the internet of things requests the device metadata for the first time, the data in the cache is not refreshed yet or the memory of the local JVM heap is insufficient or exceeds the buffer size and the number limit set by CAFFEINE CACHE, which results in clearing a part of the cache data, and the obtained device metadata from the cleared local memory cache fails.
S303, in response to the fact that the device metadata cannot be acquired in the distributed memory cache, the device metadata of the client device of the Internet of things are queried and acquired from the basic database, and the device metadata are written into the local memory cache and the distributed memory cache.
In the embodiment of the present application, if the device metadata cannot be obtained in the distributed memory cache, that is, the device metadata of the client device of the internet of things does not exist in REDIS CACHE, the device metadata is queried from the basic database, that is, the MySQL database, if the device metadata of the client device of the internet of things exists in MySQL, the device metadata is obtained for processing, and the obtained device metadata is written into the local memory cache CAFFEINE CACHE and the distributed memory cache REDIS CACHE for supplementing, so that the device metadata can be obtained in the local memory cache CAFFEINE CACHE and the distributed memory cache REDIS CACHE when the next device metadata requests for reading.
It should be noted that REDIS CACHE is a distributed cache, which has the characteristic that all the internet of things systems share one piece of data, and the data utilization rate is higher, but the data is mainly stored in the memory, so that the data in the data are lost or outdated, and thus the acquisition of the device metadata in the distributed memory cache library fails, and the device metadata of the client device of the internet of things is acquired from the MySQL database.
It can be understood that the MySQL database of the base database is the device metadata of the client device of the internet of things, because MySQL is a persistent database, the system ensures that the device metadata is always written into the MySQL database through transactions after designing the device metadata, so that the memory is utilized to query the data to the greatest extent, a large number of network requests are avoided to connect the databases, and server resources are saved.
Further, as shown in fig. 4, step S103 "in the above embodiment, the step of updating the target device metadata in the base database and updating the target device metadata cached in the local memory cache library and the distributed memory cache library" may specifically include the following steps:
S401, determining a key corresponding to the metadata of the target device in response to the basic database acquiring and executing the data updating statement aiming at the metadata of the target device.
In the embodiment of the application, the data update statement is a statement for performing data update on the metadata of the target device, the Key is a dead Key (Key) corresponding to the metadata of the target device, when the basic database executes the data update statement for the metadata of the target device, a data update event (data change or data deletion) is generated, and the Key corresponding to the metadata of the target device is determined for subsequent processing. It will be appreciated that when a user needs to update target device metadata in MySQL of the underlying database, a data update statement of MySQL needs to be executed first.
It should be noted that, in fact, device metadata of the base database may be added, but operations that need to be synchronized to the local memory cache repository CAFFEINE CACHE and the distributed memory cache repository REDIS CACHE mainly include change and delete operations, and the data update of the present application refers to data change and delete, because the operations for adding data are not described excessively.
S402, deleting target device metadata corresponding to the secret keys in the local memory cache library and the distributed memory cache library.
In the embodiment of the present application, based on the key corresponding to the metadata of the target device determined in step S401, the data of the corresponding key in the local memory cache CAFFEINE CACHE and the data of the corresponding key in the distributed memory cache are deleted.
The method does not limit the specific mode of deleting the target device metadata corresponding to the keys of the local memory cache library and the distributed memory cache library too much, and can be set according to actual conditions.
As a possible implementation manner, sending a corresponding data invalidation operation to the distributed memory cache library to delete the target device metadata corresponding to the key in the distributed memory cache library; triggering a data invalidation operation of the local memory cache to delete the target device metadata in the local memory cache. The data invalidation operation is an operation of invalidating (deleting) old data before the data update. For example, when the user needs to update the metadata of the target device in the MySQL of the base database, after first executing the data update statement of MySQL, the invalidation operation of the local jvm cache is sent, and then the invalidation operation of the dis cache is triggered to delete the corresponding data in REDIS CACHE and CAFFEINE CACHE successively.
It should be noted that, the data corresponding to the Key in REDIS CACHE must be deleted first, then the data corresponding to the Key in CAFFEINE CACHE must be deleted, and the sequence cannot be changed, otherwise, the result of inconsistent data may occur, if the data corresponding to the Key in CAFFEINE CACHE is deleted first and then REDIS CACHE is deleted, if in the middle of these two invalidation operations, one internet of things client device performs a device metadata reading operation, then the data to be deleted will be rewritten REDIS CACHE into the local cache CAFFEINE CACHE, and then the data in the local cache will be the old data before being changed.
Further, as shown in fig. 5, the distributed three-level caching method based on the internet of things in the embodiment of the application may specifically further include the following steps:
S501, acquiring a message subscription mode.
In the embodiment of the present application, REDIS CHANNEL is described as a message subscription mode, but this is not limited thereto, and may be set according to practical situations. It will be appreciated by those skilled in the art that REDIS CHANNEL is a messaging mechanism of a publish/subscribe model provided by Redis. It allows publishers to publish messages to specific channels, while subscribers receive messages in real-time by subscribing to these channels.
S502, invalidating target device metadata in a distributed local cache library based on a message subscription mode.
In the embodiment of the present application, the target device metadata in the distributed local repository is invalidated based on the message subscription mode determined in step S501.
It should be noted that, the present application does not limit the specific manner of disabling the metadata of the target device in the distributed local repository based on the message subscription mode too much, and can be set according to the actual situation.
As a possible implementation manner, a key deletion message corresponding to the key is sent to the target channel based on the message subscription mode; and in response to the distributed local cache library subscribing to the target channel, receiving a key deletion message based on the target channel and invalidating corresponding target device metadata in the distributed local cache library. The target channel is a specific channel for transmitting and receiving Key deletion messages in REDIS CHANNEL. For example, in the message subscription mechanism of REDIS CHANNEL, a Key delete message of the target device metadata is sent to a specific channel, and the distributed local cache library REDIS CACHE subscribes to the specific channel to listen for messages therein, when REDIS CACHE receives the Key delete message from the channel, the corresponding target device metadata in REDIS CACHE is immediately invalidated, and then a subsequent query about the device metadata is started from CAFFEINE CACHE to form a closed loop. It should be noted that, new device metadata will be written into each level of cache again in the process of reading each level of cache.
Further, as shown in fig. 6, step S101 "store device metadata of the client device of the internet of things into a preset base database according to a preset service requirement" in the above embodiment may specifically include the following steps:
s601, creating a device metadata table of the client device of the Internet of things.
And S602, defining the device data of the client device of the Internet of things as the device metadata based on the device metadata table and the service requirement, and storing the device metadata into a basic database.
In the embodiment of the application, the device metadata table at least comprises an attribute name, an attribute type and a device type reported by the client device of the internet of things, and the device metadata is stored in a basic database, namely, the device metadata is persisted in the basic database (the step is completed by the system of the internet of things), specifically, the device metadata table of the corresponding client device of the internet of things is newly built, and the table mainly comprises: and defining the device data of the client device of the Internet of things as device metadata according to the device metadata table and the service requirement, and writing the device metadata into a MySQL database.
Further, reporting data reported by the client device of the Internet of things is obtained, and the reporting data and the queried device metadata are stored in the Internet of things system after being correlated.
Obviously, as described in the above embodiment, the CAFFEINE CACHE local cache module may reduce the request pressure of the distributed cache; REDSI CACHE the distributed caching module can reduce the caching pressure of Mysql and improve the efficiency of data query.
Thus, the persistence, cache inquiry and real-time updating of the device metadata are realized, and the consistency and reliability of the data are ensured.
Fig. 7 is a schematic structural diagram of a distributed three-level caching device based on the internet of things, according to an embodiment of the present application, as shown in fig. 7, a distributed three-level caching device 700 based on the internet of things according to an embodiment of the present application may specifically include:
The storage module 701 is configured to store device metadata of the client device of the internet of things into a preset base database according to a preset service requirement.
The query module 702 is configured to respond to a request for device metadata sent by an internet of things client device to an internet of things system, and query and acquire device metadata of the internet of things client device in a preset multi-level cache system based on the request for device metadata; the multi-level cache system comprises a local memory cache bank, a distributed memory cache bank and a basic database.
And the updating module 703 is configured to perform data updating on the target device metadata in the base database, and update the target device metadata cached in the local memory cache library and the distributed memory cache library.
In one possible implementation, the query module is specifically configured to:
Inquiring and acquiring device metadata of the client device of the Internet of things in a local memory cache based on the device metadata request;
In response to the failure to obtain the device metadata in the local memory cache, querying and obtaining the device metadata of the client device of the Internet of things from the distributed memory cache, and writing the device metadata into the local memory cache;
and in response to the failure to acquire the device metadata in the distributed memory cache, querying and acquiring the device metadata of the client device of the Internet of things from the basic database, and writing the device metadata into the local memory cache and the distributed memory cache.
In one possible implementation, the updating module is specifically configured to:
Inquiring and acquiring device metadata of the client device of the Internet of things in a local memory cache based on the device metadata request;
In response to the failure to obtain the device metadata in the local memory cache, querying and obtaining the device metadata of the client device of the Internet of things from the distributed memory cache, and writing the device metadata into the local memory cache;
and in response to the failure to acquire the device metadata in the distributed memory cache, querying and acquiring the device metadata of the client device of the Internet of things from the basic database, and writing the device metadata into the local memory cache and the distributed memory cache.
In one possible implementation, the updating module is specifically configured to:
Transmitting corresponding data invalidation operation to the distributed memory cache library to delete the target device metadata corresponding to the key in the distributed memory cache library;
triggering a data invalidation operation of the local memory cache to delete the target device metadata in the local memory cache.
In one possible implementation manner, the distributed three-level caching device based on the internet of things further includes:
the acquisition module is used for acquiring a message subscription mode;
and the invalidation module is used for invalidating the target device metadata in the distributed local cache library based on the message subscription mode.
In one possible implementation, the disabling module is specifically configured to:
transmitting a key deleting message corresponding to the key to the target channel based on the message subscription mode;
And in response to the distributed local cache library subscribing to the target channel, receiving a key deletion message based on the target channel and invalidating corresponding target device metadata in the distributed local cache library.
In one possible implementation, the storage module is specifically configured to:
creating a device metadata table of the client device of the Internet of things; the device metadata table at least comprises an attribute name, an attribute type and a device type which are reported by client devices of the Internet of things;
device data of the client device of the Internet of things is defined as device metadata based on the device metadata table and the service requirements, and the device metadata is stored in the basic database.
According to the distributed three-level caching device based on the Internet of things, device metadata of client devices of the Internet of things are stored in the preset basic database according to preset service requirements, device metadata requests are sent to an Internet of things system in response to the client devices of the Internet of things, device metadata of the client devices of the Internet of things are inquired and obtained in the preset multi-level caching system based on the device metadata requests, data updating is conducted on target device metadata in the basic database, and target device metadata cached in a local memory cache bank and a distributed memory cache bank are updated. According to the distributed three-level caching device based on the Internet of things, the three-level caching strategy is adopted, and the device metadata is obtained by inquiring the device metadata in the multi-level caching system based on the device metadata request sent by the client device of the Internet of things to the Internet of things system, so that the distributed caching pressure is reduced, the performance of the caching system is improved, the data reporting speed of the Internet of things device is improved, and the database burden problem caused by the device metadata inquiry under the huge amount of devices of the Internet of things is effectively solved. Meanwhile, when the metadata of the target equipment is updated, the local memory cache library and the distributed memory cache library are synchronously updated, so that the data consistency of a plurality of caches in the multi-level cache system is improved.
As shown in fig. 8, an electronic device 800 provided in an embodiment of the present application includes: a processor 801, a memory 802, and a bus, the memory 802 storing machine readable instructions executable by the processor 801, the processor 801 and the memory 802 communicating via the bus when the electronic device is running, the processor 801 executing the machine readable instructions to perform the steps of the distributed three-level caching method based on the internet of things as described above.
Specifically, the above-mentioned memory 802 and the processor 801 can be general-purpose memories and processors, which are not limited herein, and when the processor 801 runs a computer program stored in the memory 802, the above-mentioned distributed three-level caching method based on the internet of things can be executed.
Corresponding to the above-mentioned distributed three-level caching method based on the internet of things, the embodiment of the application also provides a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the computer program executes the steps of the above-mentioned distributed three-level caching method based on the internet of things when being run by a processor.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described system and apparatus may refer to corresponding procedures in the method embodiments, and are not repeated in the present disclosure. In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, and the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, and for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, indirect coupling or communication connection of devices or modules, electrical, mechanical, or other form.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the deployment method described in the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk, etc.
The foregoing is merely illustrative of the present application, and the present application is not limited thereto, and any person skilled in the art will readily appreciate variations or alternatives within the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.

Claims (10)

1. The distributed three-level caching method based on the Internet of things is characterized by comprising the following steps of:
Storing the equipment metadata of the client equipment of the Internet of things into a preset basic database according to preset service requirements;
Responding to the equipment metadata request sent by the Internet of things client equipment to an Internet of things system, and inquiring and acquiring the equipment metadata of the Internet of things client equipment in a preset multi-level cache system based on the equipment metadata request; the multi-level cache system comprises a local memory cache library, a distributed memory cache library and the basic database;
And updating the data of the target device metadata in the basic database, and updating the target device metadata cached in the local memory cache library and the distributed memory cache library.
2. The method of claim 1, wherein the querying and obtaining device metadata of the internet of things client device in a multi-level cache system based on the device metadata request comprises:
inquiring and acquiring the equipment metadata of the client equipment of the Internet of things in the local memory cache library based on the equipment metadata request;
In response to the device metadata being unable to be obtained in the local memory cache, querying and obtaining device metadata of the client device of the internet of things from a distributed memory cache, and writing the device metadata into the local memory cache;
And in response to the fact that the device metadata cannot be acquired in the distributed memory cache, querying and acquiring the device metadata of the client device of the Internet of things from the basic database, and writing the device metadata into the local memory cache and the distributed memory cache.
3. The method of claim 1, wherein the data updating the target device metadata in the base database and updating the target device metadata cached in the local memory cache and the distributed memory cache comprises:
determining a key corresponding to the target device metadata in response to the base database acquiring and executing a data update statement for the target device metadata;
and deleting target device metadata corresponding to the secret key in the local memory cache and the distributed memory cache.
4. The method of claim 3, wherein the deleting the target device metadata corresponding to the key in the local memory cache and the distributed memory cache comprises:
transmitting corresponding data invalidation operation to the distributed memory cache library to delete target equipment metadata corresponding to the secret key in the distributed memory cache library;
Triggering the data invalidation operation of the local memory cache library to delete the target device metadata in the local memory cache library.
5. The method according to claim 4, wherein the method further comprises:
Acquiring a message subscription mode;
Target device metadata in the distributed local repository is invalidated based on the message subscription pattern.
6. The method of claim 5, wherein invalidating target device metadata in a preset distributed local repository based on the message subscription mode comprises:
transmitting a key deletion message corresponding to the key to a target channel based on the message subscription mode;
And responding to the subscription of the distributed local cache library to the target channel, receiving the key deletion message based on the target channel and invalidating corresponding target device metadata in the distributed local cache library.
7. The method according to claim 1, wherein storing the device metadata of the client device of the internet of things into the preset base database according to the preset service requirement comprises:
Creating a device metadata table of the client device of the internet of things; the device metadata table at least comprises an attribute name, an attribute type and a device type which are reported by the client device of the Internet of things;
And defining the equipment data of the client equipment of the Internet of things as equipment metadata based on the equipment metadata table and the service requirement, and storing the equipment metadata into the basic database.
8. Three-level caching device of distributing type based on thing networking, its characterized in that, the device includes:
The storage module is used for storing the equipment metadata of the client equipment of the Internet of things into a preset basic database according to preset service requirements;
The query module is used for responding to the equipment metadata request sent by the client equipment of the Internet of things to the system of the Internet of things, and querying and acquiring the equipment metadata of the client equipment of the Internet of things in a preset multi-level cache system based on the equipment metadata request; the multi-level cache system comprises a local memory cache library, a distributed memory cache library and the basic database;
and the updating module is used for carrying out data updating on the target device metadata in the basic database and updating the target device metadata cached in the local memory cache library and the distributed memory cache library.
9. An electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication over the bus when the electronic device is running, the machine-readable instructions when executed by the processor performing the steps of the internet of things-based distributed three-level cache method of any one of claims 1 to 7.
10. A computer readable storage medium, characterized in that it has stored thereon a computer program which, when executed by a processor, performs the steps of the internet of things based distributed three level caching method according to any one of claims 1 to 7.
CN202410316945.8A 2024-03-20 2024-03-20 Distributed three-level caching method and device based on Internet of things Pending CN117914944A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410316945.8A CN117914944A (en) 2024-03-20 2024-03-20 Distributed three-level caching method and device based on Internet of things

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410316945.8A CN117914944A (en) 2024-03-20 2024-03-20 Distributed three-level caching method and device based on Internet of things

Publications (1)

Publication Number Publication Date
CN117914944A true CN117914944A (en) 2024-04-19

Family

ID=90684052

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410316945.8A Pending CN117914944A (en) 2024-03-20 2024-03-20 Distributed three-level caching method and device based on Internet of things

Country Status (1)

Country Link
CN (1) CN117914944A (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112749198A (en) * 2021-01-21 2021-05-04 中信银行股份有限公司 Multi-level data caching method and device based on version number
CN113420052A (en) * 2021-07-08 2021-09-21 上海浦东发展银行股份有限公司 Multi-level distributed cache system and method
CN114691703A (en) * 2022-03-22 2022-07-01 深圳追一科技有限公司 Data updating method and device, electronic equipment and storage medium
US20220276998A1 (en) * 2020-06-10 2022-09-01 Tencent Technology (Shenzhen) Company Limited Database transaction processing method and apparatus, server, and storage medium
CN116226189A (en) * 2022-12-28 2023-06-06 多点生活(中国)网络科技有限公司 Cache data query method, device, electronic equipment and computer readable medium
CN116701413A (en) * 2023-08-08 2023-09-05 北京久其金建科技有限公司 Main data processing method and device

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220276998A1 (en) * 2020-06-10 2022-09-01 Tencent Technology (Shenzhen) Company Limited Database transaction processing method and apparatus, server, and storage medium
CN112749198A (en) * 2021-01-21 2021-05-04 中信银行股份有限公司 Multi-level data caching method and device based on version number
CN113420052A (en) * 2021-07-08 2021-09-21 上海浦东发展银行股份有限公司 Multi-level distributed cache system and method
CN114691703A (en) * 2022-03-22 2022-07-01 深圳追一科技有限公司 Data updating method and device, electronic equipment and storage medium
CN116226189A (en) * 2022-12-28 2023-06-06 多点生活(中国)网络科技有限公司 Cache data query method, device, electronic equipment and computer readable medium
CN116701413A (en) * 2023-08-08 2023-09-05 北京久其金建科技有限公司 Main data processing method and device

Similar Documents

Publication Publication Date Title
US7058783B2 (en) Method and mechanism for on-line data compression and in-place updates
US8380931B2 (en) Memory cache data center
US10089317B2 (en) System and method for supporting elastic data metadata compression in a distributed data grid
US10102147B1 (en) Phased based distributed LRU for shared cache systems
CN112039979A (en) Distributed data cache management method, device, equipment and storage medium
CN112035528A (en) Data query method and device
CN107888687B (en) Proxy client storage acceleration method and system based on distributed storage system
CN105426321A (en) RDMA friendly caching method using remote position information
US9928174B1 (en) Consistent caching
CN107992270B (en) Method and device for globally sharing cache of multi-control storage system
CN114817195A (en) Method, system, storage medium and equipment for managing distributed storage cache
CN112148736A (en) Method, device and storage medium for caching data
CN114490744B (en) Data caching method, storage medium and electronic device
CN115858409A (en) Data prefetching method, computing node and storage system
CN106164874B (en) Method and device for accessing data visitor directory in multi-core system
US8028011B1 (en) Global UNIX file system cylinder group cache
CN117914944A (en) Distributed three-level caching method and device based on Internet of things
CN115934583A (en) Hierarchical caching method, device and system
CN115203250A (en) Method and system for realizing high-performance distributed cache
CN113672652A (en) Data access method, device, equipment and storage medium
CN115408431A (en) Data access method and device, electronic equipment and storage medium
CN115080459A (en) Cache management method and device and computer readable storage medium
CN112650694A (en) Data reading method and device, cache proxy server and storage medium
CN113849119A (en) Storage method, storage device, and computer-readable storage medium
CN112395453A (en) Self-adaptive distributed remote sensing image caching and retrieval method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination