CN117785501A

CN117785501A - Data caching method and device, storage medium and electronic equipment

Info

Publication number: CN117785501A
Application number: CN202311515157.3A
Authority: CN
Inventors: 张建刚; 谢鹏; 郭坤
Original assignee: Suzhou Metabrain Intelligent Technology Co Ltd
Current assignee: Suzhou Metabrain Intelligent Technology Co Ltd
Priority date: 2023-11-14
Filing date: 2023-11-14
Publication date: 2024-03-29

Abstract

The application provides a data caching method, a device, a storage medium and electronic equipment, and relates to the technical field of computers, wherein the method comprises the following steps: under the condition that the storage management system receives a target access request, calculating the relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched according to a target dependency relationship; then, at least one object identifier to be cached, the relation of which is greater than a preset relation threshold, is screened from the plurality of object identifiers to be matched; and finally, caching the object corresponding to each object identifier to be cached in the target object and the at least one object identifier to be cached into a memory space through a cache data updating mechanism. The data caching method, the data caching device, the storage medium and the electronic equipment are used for improving the cache hit rate of the application program so as to improve the response efficiency of the application program.

Description

Data caching method and device, storage medium and electronic equipment

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a data caching method, a data caching device, a storage medium, and an electronic device.

Background

Caching is a common performance optimization technique that relies on storing copies of data from certain operations previously performed in memory or other high-speed storage for use by the next operation. The application of the caching technology can greatly improve the running efficiency of the application program.

In the related art, in order to quickly respond to an access request of a client, a storage management system generally uses a cache technology to store frequently accessed data in a memory so as to reduce access to a disk or a network. However, in the cache policy in the related art, some situations of low cache hit rate still exist.

Based on the above, a cache method capable of effectively improving the cache hit rate of the storage management system is urgently needed, so as to shorten the response time of the storage management system and improve the response efficiency.

Disclosure of Invention

The application aims to provide a data caching method, a data caching device, a storage medium and electronic equipment, which are used for improving the cache hit rate of an application program so as to improve the response efficiency of the application program.

The application provides a data caching method, which comprises the following steps:

under the condition that the storage management system receives a target access request, calculating the relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched according to a target dependency relationship; the target access request is used for requesting a target object corresponding to the target object identifier; the plurality of objects to be matched are identified as follows: object identifiers with dependency relations with the target object identifiers; screening at least one object identifier to be cached, wherein the relation degree of the object identifier to be cached is greater than a preset relation degree threshold value, from the plurality of object identifiers to be matched; caching the target object and the object corresponding to each object identifier to be cached in the at least one object identifier to be cached into a memory space through a cache data updating mechanism; wherein the target dependency relationship includes: the dependency relationship between each object identifier generated based on the log data and the dependency relationship between each object identifier are strong and weak; the cache data updating mechanism is used for caching the object into the memory according to the monitoring result of the system event message.

Optionally, before calculating the relationship degree between the target object identifier and each object identifier to be matched in the plurality of object identifiers to be matched according to the target dependency relationship, the method further includes: acquiring a plurality of operation logs, and executing preprocessing operation on each operation log to obtain a plurality of target information; the target information includes: object identification, operation time; an operation log corresponds to a target message; dividing the plurality of target information according to time periods, and combining object identifiers contained in at least one piece of target information in the same time period into one data item based on the object identifiers to obtain a plurality of data items; generating an object dependency relation set to be adjusted, which contains different object identification combinations, based on the plurality of data items; each object identification combination comprises two object identifications; the set of object dependencies includes: the plurality of object dependencies, and any one of the object dependencies includes: the frequency and relation degree of two object identifications; deleting object dependency relationships with the frequency smaller than a preset frequency threshold from the object dependency relationship set to be adjusted to obtain a target dependency relationship set, and generating the target dependency relationship based on the target dependency relationship set; wherein the frequency is used to represent: meanwhile, the ratio of the data items of the first object identifier and the second object identifier indicated by the object dependency relationship to the total data items is contained; the relationship is used to represent: and the ratio of the data items of the first object identifier and the second object identifier indicated by the object dependency relationship to the data items only containing the first object identifier is contained.

Optionally, the calculating the relationship between the target object identifier and each object identifier to be matched in the plurality of object identifiers to be matched according to the target dependency relationship includes: and determining the relation degree between each object to be matched in the plurality of object identifications to be matched and the target object identification according to the object dependency relationship between the target object identification and the plurality of object identifications to be matched in the target dependency relationship.

Optionally, the caching, by using a cache data update mechanism, the object corresponding to each of the target object and the at least one object identifier to be cached in the memory space includes: calculating the total amount of memory space occupied by an object to be cached, and judging whether the target memory usage rate after the object to be cached is cached in a memory is greater than or equal to a preset memory usage rate threshold value based on the total amount of memory space occupied by the object to be cached; the object to be cached comprises: the object cache corresponding to the target object and each object identifier to be cached; if the target memory utilization rate is greater than or equal to the preset memory utilization rate threshold, caching the object to be cached into a memory in a first data caching mode; otherwise, caching the object to be cached into a memory in a second data caching mode; the first data caching mode is as follows: directly caching the object to be cached into a memory; the second data caching mode is as follows: and caching the object to be cached in the memory in an object replacement mode.

Optionally, the caching, by using a cache data update mechanism, the object corresponding to each of the target object and the at least one object identifier to be cached in the memory space includes: screening out an object to be deleted with the lowest relation degree with the target object from objects cached in a memory according to the target dependency relation; and deleting the object to be deleted from the data cached in the memory under the condition that the object to be deleted only comprises one object, and caching the object to be cached in the memory.

Optionally, the caching, by using a cache data update mechanism, the object corresponding to each of the target object and the at least one object identifier to be cached in the memory space includes: acquiring the access times of each object in the object to be deleted under the condition that the object to be deleted contains a plurality of objects; deleting the object with the lowest object access frequency from the data cached in the memory, and caching the object to be cached in the memory.

Optionally, the caching, by using a cache data update mechanism, the object corresponding to each of the target object and the at least one object identifier to be cached in the memory space includes: under the condition that the access times of each object in the objects to be deleted are the same, acquiring the cache time of each object in the objects to be deleted; deleting the object with the earliest caching time from the data cached in the memory, and caching the object to be cached in the memory.

Optionally, the caching the object to be cached in the memory includes: monitoring system event information, and storing the object to be cached in a memory by a data structure of a key value pair under the condition of monitoring target system event information, wherein the data structure is used as cache data of the storage management system in the memory; wherein the system event message includes at least one of: event type, object identification, object; the event types included in the system event message include: creating an event, deleting the event, modifying the event and emptying the data cache event; the event types included in the target system event are as follows: an event is created.

The application also provides a data caching device, which comprises:

the computing module is used for computing the relation degree between the target object identifier and each object identifier to be matched in the plurality of object identifiers to be matched according to the target dependency relationship under the condition that the storage management system receives the target access request; the target access request is used for requesting a target object corresponding to the target object identifier; the plurality of objects to be matched are identified as follows: object identifiers with dependency relations with the target object identifiers; the object screening module is used for screening at least one object identifier to be cached, the relation degree of which is greater than a preset relation degree threshold value, from the plurality of object identifiers to be matched; the cache updating module is used for caching the object corresponding to each object to be cached in the at least one object to be cached identifier into a memory space through a cache data updating mechanism; wherein the target dependency relationship includes: the dependency relationship between each object identifier generated based on the log data and the dependency relationship between each object identifier are strong and weak; the cache data updating mechanism is used for caching the object into the memory according to the monitoring result of the system event message.

Optionally, the apparatus further comprises: the system comprises an acquisition module, a classification module, a generation module and a relation screening module; the acquisition module is used for acquiring a plurality of operation logs, and executing preprocessing operation on each operation log to obtain a plurality of target information; the target information includes: object identification, operation time; an operation log corresponds to a target message; the classification module is used for dividing the plurality of target information according to time periods, and combining object identifiers contained in at least one piece of target information in the same time period into one data item based on the object identifiers to obtain a plurality of data items; the generating module is used for generating an object dependency relation set to be adjusted, which contains different object identification combinations, based on the plurality of data items; each object identification combination comprises two object identifications; the set of object dependencies includes: the plurality of object dependencies, and any one of the object dependencies includes: the frequency and relation degree of two object identifications; the relation screening module is used for deleting the object dependency relation with the frequency smaller than a preset frequency threshold value from the object dependency relation set to be adjusted to obtain a target dependency relation set, and generating the target dependency relation based on the target dependency relation set; wherein the frequency is used to represent: meanwhile, the ratio of the data items of the first object identifier and the second object identifier indicated by the object dependency relationship to the total data items is contained; the relationship is used to represent: and the ratio of the data items of the first object identifier and the second object identifier indicated by the object dependency relationship to the data items only containing the first object identifier is contained.

Optionally, the calculating module is specifically configured to determine a degree of relationship between each object to be matched in the plurality of object to be matched identifiers and the target object identifier according to an object dependency relationship between the target object identifier and the plurality of object identifiers to be matched in the target dependency relationship.

Optionally, the calculating module is further configured to calculate a total amount of memory space occupied by an object to be cached, and determine whether a target memory usage rate after the object to be cached is cached in the memory is greater than or equal to a preset memory usage rate threshold based on the total amount of memory space occupied by the object to be cached; the object to be cached comprises: the object cache corresponding to the target object and each object identifier to be cached; the cache updating module is specifically configured to cache the object to be cached into the memory by adopting a first data caching manner if the target memory usage is greater than or equal to the preset memory usage threshold; otherwise, caching the object to be cached into a memory in a second data caching mode; the first data caching mode is as follows: directly caching the object to be cached into a memory; the second data caching mode is as follows: and caching the object to be cached in the memory in an object replacement mode.

Optionally, the object screening module is further configured to screen, according to the target dependency relationship, an object to be deleted with the lowest relationship degree with the target object from objects cached in the memory; the cache updating module is specifically configured to delete the object to be deleted from the data cached in the memory and cache the object to be cached in the memory when the object to be deleted only includes one object.

Optionally, the obtaining module is further configured to obtain, when the object to be deleted includes a plurality of objects, the number of access times of each object in the object to be deleted; the cache updating module is specifically configured to delete an object with the lowest access frequency of the object from the data cached in the memory, and cache the object to be cached in the memory.

Optionally, the obtaining module is further configured to obtain a cache time of each object in the to-be-deleted objects when the access times of each object in the to-be-deleted objects are the same; the cache updating module is specifically configured to delete an object with earliest cache time from data cached in the memory, and cache the object to be cached in the memory.

Optionally, the cache update module is specifically configured to monitor a system event message, and store the object to be cached in the memory as cache data of the storage management system in the memory in a data structure of a key value pair under the condition that a target system event message is monitored; wherein the system event message includes at least one of: event type, object identification, object; the event types included in the system event message include: creating an event, deleting the event, modifying the event and emptying the data cache event; the event types included in the target system event are as follows: an event is created.

The present application also provides a computer program product comprising computer programs/instructions which when executed by a processor implement the steps of a data caching method as described in any one of the above.

The application also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the data caching method as described in any one of the above when executing the program.

The present application also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a data caching method as described in any of the above.

According to the data caching method, the device, the storage medium and the electronic equipment, under the condition that the storage management system receives a target access request, the relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched is calculated according to the target dependency relationship; then, at least one object identifier to be cached, the relation of which is greater than a preset relation threshold, is screened from the plurality of object identifiers to be matched; and finally, caching the object corresponding to each object identifier to be cached in the target object and the at least one object identifier to be cached into a memory space through a cache data updating mechanism. Therefore, the cache hit rate of the storage management system can be improved, and the response efficiency of the storage management system to the client access request can be improved.

Drawings

In order to more clearly illustrate the technical solutions of the present application or the prior art, the following description will briefly introduce the drawings used in the embodiments or the description of the prior art, and it is obvious that, in the following description, the drawings are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a storage management system provided herein;

FIG. 2 is a flow chart of a data caching method provided in the present application;

FIG. 3 is a flow chart of a memory usage calculation method provided in the present application;

FIG. 4 is a schematic flow chart of a method for updating cache data provided in the present application;

FIG. 5 is a second flowchart of a method for updating cache data according to the present application;

FIG. 6 is a schematic diagram of a data buffering device provided in the present application;

fig. 7 is a schematic structural diagram of an electronic device provided in the present application.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application, and it is apparent that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type and not limited to the number of objects, e.g., the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.

The following description is made with respect to terms of art referred to in the embodiments of the present application:

caching is a common performance optimization technique that relies on storing copies of data from certain operations previously performed in memory or other high-speed storage for use by the next operation. In applications, caching techniques are typically used to cache frequently accessed but computationally difficult data to avoid recalculation at each access. In network applications, caching techniques are typically used to cache responses to network requests to avoid repeated requests for the same data, thereby increasing response speed and reducing network traffic.

The role of the caching technique is mainly represented in the following aspects:

the access speed is improved. The cache can reduce disk or network IO operations, and the response time is reduced and the processing speed of the program is improved by providing quick access to the data in the memory. The burden of the server is lightened. Caching may reduce the load on the server because caching may avoid repeated computations or querying the same data. Thus, the resources of the server can be more fully utilized, and the request to the server is reduced. And the system expandability is improved. The cache can store data in a place which is easy to access, thereby reducing delay and resource requirements generated when the data is acquired from an external system, and enabling the system to be more easily expanded. And the user experience is improved. The cache may allow the most recently used data to be in a fast available state, thereby improving user experience. This is particularly important in Web applications because network delays and response times can lead to user churn.

System message mechanism: refers to a mechanism provided by an operating system or program runtime environment for the manner in which information and notification events are passed between different components. Through a system message mechanism, different components can communicate with each other to realize information transmission and event triggering. In operating systems, system messaging mechanisms are commonly used in the context of inter-process and inter-thread communications (Thread Communication). Common system message mechanisms include:

signal (Signal): the signal is a software interrupt that notifies the process that a particular event has occurred. The operating system may send different types of signals to the process, such as SIGINT representing an interrupt signal (e.g., keyboard input ctrl+c), sigtherm representing a termination signal, etc. The process may choose to ignore, capture or default process signals. Message Queue (Message Queue): message queues are a communication mechanism for passing messages between processes. One process may send a message to a message queue and another process may read the message from the queue. Message queues are commonly applied in the context of interprocess decoupling and asynchronous communications. Shared Memory (Shared Memory): shared memory is a mechanism that allows different processes to perform read and write operations in the same memory area. Multiple processes can map the same memory area to respective address spaces to realize data sharing between the memory areas. Socket (Socket): sockets are a mechanism for inter-process communication in a network. Through sockets, processes can communicate over the network, sending and receiving messages on different hosts. Event-driven (Event-driven): the event driven mechanism is a communication mode based on event and callback functions. When an event occurs, the operating system triggers a corresponding callback function to inform relevant components to process. The system message mechanism plays an important role in an operating system and a runtime environment, so that different components can cooperate with each other to carry out information transmission and event triggering, thereby realizing more flexible and efficient programming.

Aiming at the technical problem that the cache hit rate of a storage management system in a related technology is low in certain scenes, as shown in fig. 1, the embodiment of the application provides a data caching method, firstly, the storage management system sets the recommendation basis of a default cache object, deep mining analysis is carried out from a plurality of client usage log data to obtain a general user operation recommendation rule, and the rule is set as a default object association recommendation criterion. Secondly, when the client requests access, the storage management system identifies an object identifier corresponding to the request, firstly, a query result is returned to the client, a new thread is initiated, an object access counter is set, the object is cached, and further, the object strongly related to the current object is identified by combining the object association recommendation rule generated in the first step, and the query and the caching are performed. Thirdly, the storage management system is provided with a memory monitoring module, and the memory space occupied by the object to be cached is calculated to be used for managing and updating the cache mainly when the object is cached, so that the cache size occupied by the storage object is ensured, the normal operation of the system service cannot be influenced, and the management of the cache is dynamically realized. And finally, the storage management system is provided with a cache data updating module, and the hit rate of a user frequently operating the object and the instantaneity of the data are improved through an updating mechanism of the cache data.

According to the data caching method, under the condition that the operation of the system service is not affected, as many objects with user relations are cached as possible, the cache hit rate is improved based on machine learning, and meanwhile a cache updating mechanism is set, so that the real-time effectiveness of cached data is ensured. When a user uses the system, the request response is completed quickly, and the usability of the system is improved.

The data caching method provided by the embodiment of the application is described in detail below through specific embodiments and application scenarios thereof with reference to the accompanying drawings.

As shown in fig. 2, a data caching method provided in an embodiment of the present application may include the following steps 201 to 203:

step 201, under the condition that the storage management system receives a target access request, calculating a relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched according to a target dependency relation.

The target access request is used for requesting a target object corresponding to the target object identifier; the plurality of objects to be matched are identified as follows: and the object identification has a dependency relationship with the target object identification. The target dependency relationship includes: the dependency relationship between the object identifications generated based on the log data, and the dependency relationship between the object identifications are strong and weak.

Illustratively, the object identification in the embodiment of the present application may include: pools, volumes, volume groups, hosts, host groups, remote replication, local replication, file systems, directories, shares, etc. An object identifier represents a class of objects.

The target access request may be a data access request sent by the client for accessing a target object corresponding to the target object identifier. The storage management system can determine other object identifiers with stronger dependency relationship with the target object identifier according to the target dependency relationship, and store the target object corresponding to the target object identifier and the object corresponding to the other object identifier into the memory together as cache data of the storage management system in the memory so as to improve the cache hit rate of the storage management system.

It can be understood that when the client requests the target object corresponding to the target object identifier, other object identifiers having a strong dependency relationship with the target object identifier will be accessed in the subsequent access request, so that the objects corresponding to the object identifiers are cached in the memory in advance, which can greatly improve the cache hit rate of the storage management system, and further improve the response efficiency of the storage management system to the client access request.

Illustratively, before the step 201, the data caching method provided in the embodiment of the present application may further include the following steps 101 to 104:

step 101, obtaining a plurality of operation logs, and executing preprocessing operation on each operation log to obtain a plurality of target information.

Wherein the target information includes: object identification, operation time; an operation log corresponds to a target information.

For example, before the storage management system is first used, a large amount of log data generated by other collected running storage management systems can be used as basic sample data, and meanwhile, according to a set dependency relationship threshold value between object identifications, a dependency relationship rule between objects is generated and is used as a default dependency rule. After the storage management system operates for a certain period, the update of the object dependency relationship can be completed through an operation log generated by the storage management system. By collecting the client oplog data of the storage management system as new sample data, new object relationship rules (i.e., the target dependencies described above) are generated. And through the dependency relationship between the updated object identifiers, the operation habit of the client user is combined more accurately, and the cache object management is performed.

For example, after the operation logs are obtained, the object identification and the operation time may be extracted from log data of each operation log. Each operation log can extract one piece of target information, and all operation logs can obtain a plurality of pieces of target information.

For example, for log data of 10:01mk pool-name pool1-size 1024, an operation object pool and an operation time of 10:01 can be extracted from the log data, and the object information can be noted as { "pool", "10:01" }, as one object belief.

Step 102, dividing the plurality of target information according to the time period, and combining the object identifications contained in at least one piece of target information in the same time period into one data item based on the object identifications to obtain a plurality of data items.

Illustratively, after all the operation logs are processed, the obtained plurality of target information is divided according to time, and object identifiers contained in all the target information in the same time period are combined into one data item based on the object identifiers.

For example, the data items are divided by minutes, and the target information of all operation logs in the same minute is combined to obtain one data item. Taking the target information as { "pool",10:01}, { "vdisk",10:01}, { "vidsk group",10:01}, for example, object identifiers contained in all the target information with the operation time of 10:01 may be combined to obtain a data item, where the data item is recorded as:

{ pool, vdisk, vdsik group }. All target data can get multiple data items, which are marked as: { pool, vdisk, host }, { pool, filesystem, domain }, { vdisk, host, hostmap }, … }.

Step 103, generating a to-be-adjusted object dependency relation set containing different object identification combinations based on the plurality of data items.

Wherein each object identification combination comprises two object identifications; the set of object dependencies includes: the plurality of object dependencies, and any one of the object dependencies includes: the frequency and relationship of two object identifications.

Illustratively, any of the above object dependencies is an expression shaped as X- > Y, where X (i.e., the above first object identification) and Y (i.e., the above second object identification) are referred to as the leading and trailing of the dependency, respectively, where the dependency rule XY, there is a degree of frequency and a degree of relationship. Wherein frequency = number of data items containing both X and Y/total number of all data items; relationship = number of data items containing both X and Y/total number of data items containing only X.

And 104, deleting the object dependency relationship with the frequency smaller than a preset frequency threshold from the object dependency relationship set to be adjusted to obtain a target dependency relationship set, and generating the target dependency relationship based on the dependency relationship set.

Wherein the frequency is used to represent: meanwhile, the ratio of the data items of the first object identifier and the second object identifier indicated by the object dependency relationship to the total data items is contained; the relationship is used to represent: and the ratio of the data items of the first object identifier and the second object identifier indicated by the object dependency relationship to the data items only containing the first object identifier is contained.

For example, in order to eliminate the influence of sporadic operations, the object dependency relationship with smaller frequency needs to be deleted, that is, the object dependency relationship with the frequency smaller than the preset frequency threshold in the image dependency relationship set to be adjusted is deleted, so as to obtain a target dependency relationship set, and further obtain a target dependency relationship.

Illustratively, the target dependency relationship is used to indicate the strength of the dependency relationship between any two object identifiers.

Step 202, screening at least one object identifier to be cached, the relationship of which is greater than a preset relationship threshold, from the plurality of object identifiers to be matched.

For example, after determining a plurality of object identifiers to be matched, which have a dependency relationship with the target object identifier, according to the target dependency relationship, it is further required to determine the dependency relationship strength between each object identifier to be matched and the target object identifier.

For example, whether the dependency relationship between the two object identifiers meets the requirement of simultaneous caching can be directly determined according to the relationship degree between the target object identifier and any object identifier to be matched.

Specifically, the step 202 may further include the following step 202a:

step 202a, determining a degree of relationship between each object to be matched in the plurality of object identifications to be matched and the target object identification according to the object dependency relationship between the target object identification and the plurality of object identifications to be matched in the target dependency relationship.

It should be noted that, in the embodiment of the present application, the target dependency relationship does not delete the dependency relationship with the relationship degree smaller than the preset relationship degree threshold, which is to determine the object to be replaced when the memory usage rate is high in the subsequent step.

Step 203, caching the object corresponding to each object identifier to be cached in the memory space by the target object and each object identifier to be cached in the at least one object identifier to be cached through a cache data updating mechanism.

The cache data updating mechanism is used for caching the object into the memory according to the monitoring result of the system event message.

Illustratively, in the embodiment of the present application, a cache data update mechanism is used to update the cached object, instead of flushing the cached object first and then caching new data into the memory as used in the related art.

For example, after determining the object to be cached that needs to be cached, the object to be cached may be stored in the memory through the cache data update mechanism in the embodiment of the present application.

Specifically, the step 203 may further include the following steps 203a and 203b:

step 203a, calculating the total amount of memory space occupied by the object to be cached, and determining whether the target memory usage rate after caching the object to be cached in the memory is greater than or equal to a preset memory usage rate threshold based on the total amount of memory space occupied by the object to be cached.

Wherein, the object to be cached comprises: and the target object and the object cache corresponding to each object identifier to be cached.

For example, before storing the object to be cached in the memory, it is first required to calculate the size of the memory space occupied by the object to be cached, and determine whether the memory usage threshold is exceeded after storing the object to be cached in the memory. Memory usage = memory space size in use/total memory space size.

It should be noted that the memory usage refers to the ratio between the amount of memory currently being used by the system and the total memory capacity. It can be used to measure the utilization of the system memory resources and whether the system has memory pressure. A high memory usage may indicate that the system is using a large amount of memory resources, and there may be memory pressure, resulting in reduced system performance, and thus the memory size occupied by the memory module needs to be reduced.

Illustratively, the current query object and the associated object are fetched from the ready cache queue, and the memory size to be occupied by each object is sequentially calculated, wherein the calculation mode is mainly based on: the size of memory that an object's instance variables occupy depends on the type and number of variables. The memory size occupied by different data types is also different. Taking java development as an example, the memory size occupied by some common data types is as follows: boost: 1 byte, byte:1 byte, char:2 bytes, short:2 bytes, int:4 bytes, float:4 bytes, long:8 bytes, double:8 bytes, reference type (e.g., object reference, array reference, etc.): 4 bytes (32 bit JVM) or 8 bytes (64 bit JVM).

As shown in fig. 3, according to the above calculation criteria, it is first required to determine the object to be cached contained in the object to be cached, calculate the memory size occupied by the calculator according to the attribute of each object to be cached (i.e. the data type of each data contained in the object to be cached), and calculate the memory size to be occupied by the object according to the number of instance objects, so as to further obtain the total memory size occupied by the object to be cached. And then, calculating the memory utilization rate after the objects to be cached are all stored in the memory, and judging whether the memory utilization rate threshold value is exceeded. In this embodiment, the memory usage threshold may be 80%.

Step 203b, if the target memory usage rate is greater than or equal to the preset memory usage rate threshold, caching the object to be cached in the memory by adopting a first data caching mode; otherwise, the object to be cached is cached in the memory in a second data caching mode.

The first data caching mode is as follows: directly caching the object to be cached into a memory; the second data caching mode is as follows: and caching the object to be cached in the memory in an object replacement mode.

For example, as shown in fig. 4, when it is determined that the memory usage rate after all the objects to be cached are stored in the memory does not exceed the memory usage rate threshold, then the objects to be cached are cached in the memory; otherwise, the object to be cached is cached in the memory in an object replacement mode.

Specifically, when the object to be cached needs to be cached in the memory by means of object replacement, the above step 203b may further include the following steps 203b1 and 203b2:

step 203b1, screening the object to be deleted with the lowest relation degree with the target object from the objects cached in the memory according to the target dependency relationship.

Step 203b2, deleting the object to be deleted from the data cached in the memory, and caching the object to be cached in the memory, where the object to be deleted only includes one object.

For example, as shown in fig. 4, when it is determined that the object to be cached needs to be cached in the memory by means of object replacement, the association degree (i.e., the relationship degree) between all objects cached in the memory module and the current access object (i.e., the target object) may be obtained, and it may be determined whether there is 1 object with the smallest confidence degree (i.e., the relationship degree is the smallest) in the cached data in the memory of the storage management system. If yes, directly removing the cache object with the minimum confidence, recalculating the memory utilization rate, and repeating the steps until the memory utilization rate is smaller than the memory utilization rate threshold after the object to be cached is stored in the memory.

It should be noted that, the association degree between the objects is determined based on the relationship degree between the object identifications of the objects.

For example, if the storage management system has a plurality of objects to be deleted with the minimum relation degree with the target object in the cache data of the memory, it is necessary to further determine the access times of each object to be deleted, and remove the object to be deleted with the minimum access times from the memory.

Specifically, after the step 203b1, the step 203b may further include the following steps 203b3 and 203b4:

step 203b3, obtaining the access times of each object in the object to be deleted when the object to be deleted contains a plurality of objects.

Step 203b4, deleting the object with the lowest object access frequency from the data cached in the memory, and caching the object to be cached in the memory.

For example, as shown in fig. 4, when there are a plurality of objects with the smallest confidence, the data of each object access counter may be acquired, the access times of each object are determined, and one cache object with the smallest access times is removed from the memory. Meanwhile, similarly to the above steps, after the cache object is removed from the memory, the memory usage needs to be recalculated and the above steps are re-executed until the memory usage is less than the memory usage threshold after the object to be cached is stored in the memory.

Specifically, after the step 203b3, the step 203b may further include the following steps 203b5 and 203b6:

step 203b5, obtaining the cache time of each object in the objects to be deleted when the access times of each object in the objects to be deleted are the same.

Step 203b6, deleting the object with the earliest caching time from the data cached in the memory, and caching the object to be cached in the memory.

For example, as shown in fig. 4, if there are multiple objects to be deleted with the same access times, the object with the earliest storage buffering time needs to be removed from the memory. And then, re-executing the steps until the memory utilization rate is smaller than the memory utilization rate threshold after the object to be cached is stored in the memory.

For example, after determining that the object to be cached is stored in the memory, the object to be cached may be stored in the memory after the memory usage is less than the memory usage threshold.

Illustratively, in embodiments of the present application, the cache update operation that needs to be performed may be determined by listening for a system event message.

Specifically, the step 203 may further include the following step 203c1:

and 203c1, monitoring a system event message, and storing the object to be cached in a memory in a data structure of a key value pair as cache data in the memory of the storage management system under the condition that a target system event message is monitored.

Wherein the system event message includes at least one of: event type, object identification, object; the event types included in the system event message include: creating an event, deleting the event, modifying the event and emptying the data cache event; the event types included in the target system event are as follows: an event is created.

The object data is stored in the cache module in the form of key-value pair { key: value }, wherein the key is the object identifier, such as the pool object data, and the data structure is stored in the cache

{pool:[pool1,pool2,pool3...]}。

By way of example, the event types described above may include: the CREATE event, the creation event described above, represents a newly created object instance; DELETE event, the DELETE event described above, represents DELETE object instance data; the MODIFY event, i.e., the modification event described above, represents modification object instance data; INVALIDALL event, the above flush data cache event, indicates that all cache data is invalid, meaning that all object data and access counters in the cache module are flushed.

The event operation monitoring is mainly used for monitoring all event change messages in the system, and according to user operation, event message packaging, namely event change types, object identification and entity class data of change events, is completed, and release of the event change messages is completed. Encapsulated event messages such as { actionType: CREATE, objectId: "pool", bean: { "id":1, "name": "pool 4"..degree}.

For example, when the object to be cached needs to be stored in the memory, a system event message including the creation event, that is, the target system event message, may be encapsulated, and then the object to be cached is stored in the memory through the system event message.

For example, if the object to be cached needs to be stored in the memory by replacing the object, a system event message including a deletion event needs to be encapsulated before the target system event message to delete the object to be replaced in the memory.

Specifically, in the case where the object to be cached needs to be cached in the memory by means of object replacement, before the step 203c1, the step 203 may further include the following step 203c2:

and 203c2, monitoring the system event message, and removing the object to be deleted from the memory under the condition that the deleting system event message is monitored.

By means of a cache data updating mechanism, accuracy of data in the cache module is guaranteed, event change types are identified through monitoring of system events, local updating of the cache data is further achieved, and waste of system resources due to whole query of object data is avoided.

For example, as shown in fig. 5, a detailed flow chart of a cache data updating mechanism provided in an embodiment of the present application is shown, a server performs a user operation on a client as soon as possible, and when an access request sent by the client is received, obtains an object identifier of the access request operation, and determines an event operation type. And then, event message encapsulation processing is carried out, and the caching module receives event messages through event subscription. After receiving the event message, firstly judging whether the event message is a INVALIDALL event message, if so, directly emptying all caches. Otherwise, further matching the object identifier carried in the event message with the object identifier in the cache module, and if the matching fails, directly discarding the event message; if the matching is successful, further acquiring the event type and executing the operation corresponding to the event type. For example, a CREATE event requires that the data set indicated by the object identification be inserted into a cache.

According to the data caching method, user log data are collected and analyzed to serve as basic data, log data preprocessing is conducted, object identification and operation time are extracted, and the object identification operated within a period of time is further extracted to serve as a transaction item, so that a transaction item set is generated. Outputting the relation degree of each object through the occurrence frequency and the dependency relationship among the objects, and dynamically recommending the objects with high relation degree to pre-cache the data, so as to provide a management basis for the cached objects in the cache management; secondly, designing a system memory utilization rate monitoring module, and judging the system memory utilization rate in real time by using the buffer memory occupied by the actual service scene of the system operation and the memory size occupied by the object to be buffered, and dynamically buffering the management strategy; thirdly, designing an updating and replacing algorithm of the cache object, taking a user as a basic starting point, using three important index factors such as an object relation degree, an object access counter, a time point for storing the cache and the like as references, updating and replacing object data of a frequent operation relation of the user in real time, and improving the cache hit rate; fourthly, designing a data updating algorithm, monitoring user operation behaviors in the system, identifying and packaging operation objects and operation types, realizing monitoring of user operation by using a server-side publishing-subscribing design mode, updating cached data in real time, and simultaneously adopting a local updating mode to reduce occupation of system resources due to updating of the cache.

According to the data caching method, under the condition that the storage management system receives a target access request, the relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched is calculated according to a target dependency relationship; then, at least one object identifier to be cached, the relation of which is greater than a preset relation threshold, is screened from the plurality of object identifiers to be matched; and finally, caching the object corresponding to each object identifier to be cached in the target object and the at least one object identifier to be cached into a memory space through a cache data updating mechanism. Therefore, the cache hit rate of the storage management system can be improved, and the response efficiency of the storage management system to the client access request can be improved.

It should be noted that, in the data caching method provided in the embodiment of the present application, the execution body may be a data caching device, or a control module in the data caching device for executing the data caching method. In this embodiment, a data caching device executes a data caching method as an example, which describes the data caching device provided in this embodiment of the present application.

In the embodiment of the application, the method is shown in the drawings. The data caching method is exemplified by combining one drawing in the embodiment of the application. In specific implementation, the data caching method shown in the foregoing method drawings may also be implemented in combination with any other drawing that may be combined and illustrated in the foregoing embodiments, which is not repeated herein.

The data caching apparatus provided in the present application is described below, and the data caching method described below and the data caching method described above may be referred to correspondingly to each other.

Fig. 6 is a schematic structural diagram of a data caching apparatus provided in an embodiment of the present application, as shown in fig. 6, specifically including:

the calculating module 601 is configured to calculate, according to a target dependency relationship, a relationship degree between a target object identifier and each of a plurality of object identifiers to be matched when the storage management system receives a target access request; the target access request is used for requesting a target object corresponding to the target object identifier; the plurality of objects to be matched are identified as follows: object identifiers with dependency relations with the target object identifiers; the object screening module 602 is configured to screen at least one object identifier to be cached, where the relationship degree of the object identifier to be matched is greater than a preset relationship degree threshold, from the plurality of object identifiers to be matched; the cache updating module 604 is configured to cache, through a cache data updating mechanism, the target object and an object corresponding to each object identifier to be cached in the at least one object identifier to be cached into a memory space; wherein the target dependency relationship includes: the dependency relationship between each object identifier generated based on the log data and the dependency relationship between each object identifier are strong and weak; the cache data updating mechanism is used for caching the object into the memory according to the monitoring result of the system event message.

Optionally, the calculating module 601 is specifically configured to determine a degree of relationship between each object to be matched in the plurality of object to be matched identifiers and the target object identifier according to an object dependency relationship between the target object identifier and the plurality of object to be matched in the target dependency relationship.

Optionally, the calculating module 601 is further configured to calculate a total amount of memory space occupied by an object to be cached, and determine whether a target memory usage rate after the object to be cached is cached in the memory is greater than or equal to a preset memory usage rate threshold based on the total amount of memory space occupied by the object to be cached; the object to be cached comprises: the object cache corresponding to the target object and each object identifier to be cached; the cache update module 604 is specifically configured to cache the object to be cached in the memory by adopting a first data cache manner if the target memory usage is greater than or equal to the preset memory usage threshold; otherwise, caching the object to be cached into a memory in a second data caching mode; the first data caching mode is as follows: directly caching the object to be cached into a memory; the second data caching mode is as follows: and caching the object to be cached in the memory in an object replacement mode.

Optionally, the object screening module 602 is further configured to screen, according to the target dependency relationship, an object to be deleted with the lowest relationship degree with the target object from objects cached in the memory; the cache update module 604 is specifically configured to delete the object to be deleted from the data cached in the memory and cache the object to be cached in the memory when the object to be deleted only includes one object.

Optionally, the obtaining module is further configured to obtain, when the object to be deleted includes a plurality of objects, the number of access times of each object in the object to be deleted; the cache update module 604 is specifically configured to delete an object with the lowest access frequency of the object from the data cached in the memory, and cache the object to be cached in the memory.

Optionally, the obtaining module is further configured to obtain a cache time of each object in the to-be-deleted objects when the access times of each object in the to-be-deleted objects are the same; the cache update module 604 is specifically configured to delete an object with the earliest cache time from the data cached in the memory, and cache the object to be cached in the memory.

Optionally, the cache update module 604 is specifically configured to monitor a system event message, and store the object to be cached in the memory as cache data of the storage management system in the memory in a data structure of a key value pair when the target system event message is monitored; wherein the system event message includes at least one of: event type, object identification, object; the event types included in the system event message include: creating an event, deleting the event, modifying the event and emptying the data cache event; the event types included in the target system event are as follows: an event is created.

According to the data caching device, under the condition that the storage management system receives a target access request, the relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched is calculated according to a target dependency relationship; then, at least one object identifier to be cached, the relation of which is greater than a preset relation threshold, is screened from the plurality of object identifiers to be matched; and finally, caching the object corresponding to each object identifier to be cached in the target object and the at least one object identifier to be cached into a memory space through a cache data updating mechanism. Therefore, the cache hit rate of the storage management system can be improved, and the response efficiency of the storage management system to the client access request can be improved.

Fig. 7 illustrates a physical schematic diagram of an electronic device, as shown in fig. 7, which may include: processor 710, communication interface (Communications Interface) 720, memory 730, and communication bus 740, wherein processor 710, communication interface 720, memory 730 communicate with each other via communication bus 740. Processor 710 may invoke logic instructions in memory 730 to perform a data caching method comprising: under the condition that the storage management system receives a target access request, calculating the relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched according to a target dependency relationship; the target access request is used for requesting a target object corresponding to the target object identifier; the plurality of objects to be matched are identified as follows: object identifiers with dependency relations with the target object identifiers; screening at least one object identifier to be cached, wherein the relation degree of the object identifier to be cached is greater than a preset relation degree threshold value, from the plurality of object identifiers to be matched; caching the target object and the object corresponding to each object identifier to be cached in the at least one object identifier to be cached into a memory space through a cache data updating mechanism; wherein the target dependency relationship includes: the dependency relationship between each object identifier generated based on the log data and the dependency relationship between each object identifier are strong and weak; the cache data updating mechanism is used for caching the object into the memory according to the monitoring result of the system event message.

Further, the logic instructions in the memory 730 described above may be implemented in the form of software functional units and may be stored in a computer readable storage medium when sold or used as a stand alone product. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

In another aspect, the present application also provides a computer program product comprising a computer program stored on a computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform a data caching method provided by the above methods, the method comprising: under the condition that the storage management system receives a target access request, calculating the relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched according to a target dependency relationship; the target access request is used for requesting a target object corresponding to the target object identifier; the plurality of objects to be matched are identified as follows: object identifiers with dependency relations with the target object identifiers; screening at least one object identifier to be cached, wherein the relation degree of the object identifier to be cached is greater than a preset relation degree threshold value, from the plurality of object identifiers to be matched; caching the target object and the object corresponding to each object identifier to be cached in the at least one object identifier to be cached into a memory space through a cache data updating mechanism; wherein the target dependency relationship includes: the dependency relationship between each object identifier generated based on the log data and the dependency relationship between each object identifier are strong and weak; the cache data updating mechanism is used for caching the object into the memory according to the monitoring result of the system event message.

In yet another aspect, the present application further provides a computer readable storage medium having stored thereon a computer program which when executed by a processor is implemented to perform the above provided data caching methods, the method comprising: under the condition that the storage management system receives a target access request, calculating the relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched according to a target dependency relationship; the target access request is used for requesting a target object corresponding to the target object identifier; the plurality of objects to be matched are identified as follows: object identifiers with dependency relations with the target object identifiers; screening at least one object identifier to be cached, wherein the relation degree of the object identifier to be cached is greater than a preset relation degree threshold value, from the plurality of object identifiers to be matched; caching the target object and the object corresponding to each object identifier to be cached in the at least one object identifier to be cached into a memory space through a cache data updating mechanism; wherein the target dependency relationship includes: the dependency relationship between each object identifier generated based on the log data and the dependency relationship between each object identifier are strong and weak; the cache data updating mechanism is used for caching the object into the memory according to the monitoring result of the system event message.

The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and are not limiting thereof; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.

Claims

1. A data caching method, applied to a storage management system, the method comprising:

under the condition that the storage management system receives a target access request, calculating the relation degree between a target object identifier and each object identifier to be matched in a plurality of object identifiers to be matched according to a target dependency relationship; the target access request is used for requesting a target object corresponding to the target object identifier; the plurality of objects to be matched are identified as follows: object identifiers with dependency relations with the target object identifiers;

screening at least one object identifier to be cached, wherein the relation degree of the object identifier to be cached is greater than a preset relation degree threshold value, from the plurality of object identifiers to be matched;

Caching the target object and the object corresponding to each object identifier to be cached in the at least one object identifier to be cached into a memory space through a cache data updating mechanism;

wherein the target dependency relationship includes: the dependency relationship between each object identifier generated based on the log data and the dependency relationship between each object identifier are strong and weak; the cache data updating mechanism is used for caching the object into the memory according to the monitoring result of the system event message.

2. The method of claim 1, wherein before calculating the degree of relationship between the target object identifier and each of the plurality of object identifiers to be matched according to the target dependency relationship, the method further comprises:

acquiring a plurality of operation logs, and executing preprocessing operation on each operation log to obtain a plurality of target information; the target information includes: object identification, operation time; an operation log corresponds to a target message;

dividing the plurality of target information according to time periods, and combining object identifiers contained in at least one piece of target information in the same time period into one data item based on the object identifiers to obtain a plurality of data items;

Generating an object dependency relation set to be adjusted, which contains different object identification combinations, based on the plurality of data items; each object identification combination comprises two object identifications; the set of object dependencies includes: the plurality of object dependencies, and any one of the object dependencies includes: the frequency and relation degree of two object identifications;

deleting object dependency relationships with the frequency smaller than a preset frequency threshold from the object dependency relationship set to be adjusted to obtain a target dependency relationship set, and generating the target dependency relationship based on the target dependency relationship set;

3. The method according to claim 1 or 2, wherein calculating the degree of relationship between the target object identifier and each of the plurality of object identifiers to be matched according to the target dependency relationship comprises:

And determining the relation degree between each object to be matched in the plurality of object identifications to be matched and the target object identification according to the object dependency relationship between the target object identification and the plurality of object identifications to be matched in the target dependency relationship.

4. The method of claim 1, wherein caching, by the cache data update mechanism, the object corresponding to each of the target object and the at least one object identifier to be cached in the memory space includes:

calculating the total amount of memory space occupied by an object to be cached, and judging whether the target memory usage rate after the object to be cached is cached in a memory is greater than or equal to a preset memory usage rate threshold value based on the total amount of memory space occupied by the object to be cached; the object to be cached comprises: the object cache corresponding to the target object and each object identifier to be cached;

if the target memory utilization rate is greater than or equal to the preset memory utilization rate threshold, caching the object to be cached into a memory in a first data caching mode; otherwise, caching the object to be cached into a memory in a second data caching mode;

5. The method of claim 4, wherein caching, by the cache data update mechanism, the object corresponding to each of the target object and the at least one object identifier to be cached in the memory space includes:

screening out an object to be deleted with the lowest relation degree with the target object from objects cached in a memory according to the target dependency relation;

and deleting the object to be deleted from the data cached in the memory under the condition that the object to be deleted only comprises one object, and caching the object to be cached in the memory.

6. The method of claim 5, wherein caching, by the cache data update mechanism, the object corresponding to each of the target object and the at least one object identifier to be cached in the memory space includes:

acquiring the access times of each object in the object to be deleted under the condition that the object to be deleted contains a plurality of objects;

Deleting the object with the lowest object access frequency from the data cached in the memory, and caching the object to be cached in the memory.

7. The method of claim 6, wherein caching, by the cache data update mechanism, the object corresponding to each of the target object and the at least one object identifier to be cached in the memory space includes:

under the condition that the access times of each object in the objects to be deleted are the same, acquiring the cache time of each object in the objects to be deleted;

deleting the object with the earliest caching time from the data cached in the memory, and caching the object to be cached in the memory.

8. The method according to any one of claims 5 to 7, wherein the caching the object to be cached in the memory includes:

monitoring system event information, and storing the object to be cached in a memory by a data structure of a key value pair under the condition of monitoring target system event information, wherein the data structure is used as cache data of the storage management system in the memory;

9. A data caching apparatus for use with a storage management system, the apparatus comprising:

the computing module is used for computing the relation degree between the target object identifier and each object identifier to be matched in the plurality of object identifiers to be matched according to the target dependency relationship under the condition that the storage management system receives the target access request; the target access request is used for requesting a target object corresponding to the target object identifier; the plurality of objects to be matched are identified as follows: object identifiers with dependency relations with the target object identifiers;

the object screening module is used for screening at least one object identifier to be cached, the relation degree of which is greater than a preset relation degree threshold value, from the plurality of object identifiers to be matched;

the cache updating module is used for caching the object corresponding to each object to be cached in the at least one object to be cached identifier into a memory space through a cache data updating mechanism;

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the data caching method according to any one of claims 1 to 8 when the program is executed.

11. A computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, implements the steps of the data caching method according to any one of claims 1 to 8.