CN115757352A - State data processing system, method, device, computer equipment and storage medium - Google Patents
State data processing system, method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN115757352A CN115757352A CN202211493539.6A CN202211493539A CN115757352A CN 115757352 A CN115757352 A CN 115757352A CN 202211493539 A CN202211493539 A CN 202211493539A CN 115757352 A CN115757352 A CN 115757352A
- Authority
- CN
- China
- Prior art keywords
- state data
- data
- storage system
- module
- state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present application provides a state data processing system, the system comprising: the state data caching module is used for receiving and storing state data to be written; the state data management module is used for monitoring the use condition of the memory, acquiring the read-write condition of the state data, and determining a migrated state data set according to the use condition of the memory, the read-write condition of the state data and a preset data migration strategy; and the state data migration management module is used for migrating the state data to be migrated from the state data cache module to the state data back-end storage system and migrating the state data to be migrated from the state data back-end storage system to the state data cache module. According to the technical scheme, the consumption of I/O resources can be reduced, and the read-write performance of state data is improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a system, a method, an apparatus, a computer device, and a storage medium for processing state data.
Background
Flink is an open source streaming framework developed by the Apache software foundation that provides a high throughput, low latency streaming data engine and support for event-time processing and state management. In order to achieve fault tolerance and store intermediate results in the calculation process, the Flink stores State (State data) into a Flink State backend (for example, rocksDB), so that self-recovery can be performed conveniently when an exception occurs, wherein the RocksDB is a high-performance embedded database for key-value data.
However, the inventor finds that the conventional method of storing the state data to the backend of the flag state consumes more I/O resources when the error tolerance is realized by reading the state data.
Disclosure of Invention
The present application aims to provide a state data processing system, a method, an apparatus, a computer device and a storage medium, so as to solve the problem that the current storage method consumes more I/O resources when state data needs to be read.
One aspect of the embodiments of the present application provides a state data processing system deployed in a memory, the system including:
the state data caching module is used for receiving and storing state data to be written;
the state data management module is used for monitoring the use condition of the memory, acquiring the read-write condition of state data, and determining a migrated state data set according to the use condition of the memory, the read-write condition of the state data and a preset data migration strategy, wherein the state data set comprises the state data to be migrated from the state data cache module to a state data back-end storage system and the state data to be migrated from the state data back-end storage system to the state data cache module;
and the state data migration management module is used for migrating the state data to be migrated from the state data cache module to the state data back-end storage system and migrating the state data to be migrated from the state data back-end storage system to the state data cache module.
Optionally, the status data management module includes:
the memory monitoring unit is used for monitoring the use condition of the memory;
the data reading and writing index acquisition unit is used for acquiring the reading and writing conditions of the state data, wherein the reading and writing conditions comprise the writing conditions of the state data stored in the state data caching module and the reading and writing conditions of the state data in the state data rear-end storage system; and
and the migration policy unit is used for configuring the data migration policy by a user and determining a migrated state data set according to the use condition of the memory, the read-write condition of the state data and a preset data migration policy.
Optionally, the state data migration management module includes:
the scheduling unit is used for triggering a data migration task; and
and the data migration unit is used for migrating the state data to be migrated from the state data cache module to the state data back-end storage system and migrating the state data to be migrated from the state data back-end storage system to the state data cache module when the data migration task is received.
Optionally, the state data migration management module further includes:
the memory management unit is used for executing memory application operation before migrating the state data in the state data back-end storage system to the state data cache module; and executing memory recovery operation after migrating the state data in the state data caching module to a state data back-end storage system.
Optionally, the state data caching module is further configured to:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system, and triggering the state data back-end storage system to execute data snapshot operation according to the Checkpoint operation after the data writing operation is completed.
Optionally, the different-state data caching module is further configured to:
and when a Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system in an asynchronous mode, so that the state data back-end storage system executes data snapshot operation according to the Checkpoint operation in an asynchronous mode.
Optionally, the status data is keyed status data.
Another aspect of the embodiments of the present application provides a status data processing method, which is applied to the status data processing system described above, and the method includes:
receiving a state data query request;
searching whether query data corresponding to the state data query request exists in a state data cache module;
if the query data exists in the state data cache module, acquiring the query data from the state data cache module;
and if the query data does not exist in the state data caching module, acquiring the query data from a state data rear-end storage system.
Optionally, after the step of obtaining the query data from the state data back-end storage system if the query data does not exist in the state data caching module, the method further includes:
and writing the query data into the state data caching module.
Optionally, the method further comprises:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system, and triggering the state data back-end storage system to execute data snapshot operation according to the Checkpoint operation after the data writing operation is completed.
Optionally, the method further comprises:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system in an asynchronous mode, so that the state data back-end storage system executes data snapshot operation according to the Checkpoint operation in an asynchronous mode.
Yet another aspect of an embodiment of the present application provides a status data processing apparatus, including:
the receiving module is used for receiving a state data query request;
the searching module is used for searching whether the state data caching module has the query data corresponding to the state data query request;
an obtaining module, configured to obtain the query data from the state data caching module if the query data exists in the state data caching module; and if the query data does not exist in the state data caching module, acquiring the query data from a state data rear-end storage system.
A further aspect of embodiments of the present application provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the state data processing method described above when executing the computer program.
Yet another aspect of embodiments of the present application provides a computer-readable storage medium having stored thereon a computer program executable by at least one processor to cause the at least one processor to perform the steps of the state data processing method as described above.
The state data processing system, the state data processing method, the state data processing device, the computer equipment and the computer readable storage medium provided by the embodiment of the application have the following technical advantages:
the state data generated by the streaming data calculation engine in the running process can be written into the memory first, and the state data is stored by a state data cache module in a state data processing system deployed in the memory. And finally, the state data migration management module executes the data migration operation to migrate part of the state data stored in the memory to the state data rear-end storage system, and meanwhile migrates part of the state data stored in the state data rear-end storage system back to the memory. Therefore, when the state data needs to be acquired, the state data can be directly acquired from the memory, and the state data back-end storage system is queried only when the state data to be acquired does not exist in the memory, so that the state data is acquired from the state data back-end storage system. Because the consumption of the I/O resources for directly acquiring the state data from the memory is much lower than that of the I/O resources for directly acquiring the state data from the state data rear-end storage system, the state data processing system can obviously reduce the consumption of the I/O resources and improve the read-write performance of the state data when acquiring the state, and the speed of acquiring the state data is higher.
Drawings
FIG. 1 schematically illustrates an environment schematic according to an embodiment of the present application;
FIG. 2 is a diagram schematically illustrating an architecture of a state data processing system according to a first embodiment of the present application;
fig. 3 schematically shows a flow chart of a state data processing method according to a second embodiment of the present application;
FIG. 4 schematically illustrates a flow diagram of an embodiment of a Checkpoint operation;
FIG. 5 schematically illustrates a flow diagram of another embodiment of a Checkpoint operation;
fig. 6 schematically shows a block diagram of a state data processing apparatus according to a third embodiment of the present application; and
fig. 7 schematically shows a hardware architecture diagram of a computer device suitable for implementing the state data processing method according to a fourth embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and the embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the descriptions relating to "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present application.
In the description of the present application, it should be understood that the numerical references before the steps do not identify the order of performing the steps, but merely serve to facilitate the description of the present application and to distinguish each step, and therefore should not be construed as limiting the present application.
The following are some explanations of terms that the present application refers to:
flink is an open source stream processing framework developed by Apache software foundation, and is at the core a distributed stream data stream engine written in Java and Scala, executing arbitrary stream data programs in a data parallel and pipeline manner, and a pipeline runtime system can execute batch processing and stream processing programs.
RocksDB, a high-performance embedded database for key-value data, is a branch of google LevelDB, and optimized, can utilize many CPU cores, and effectively utilize fast storage such as Solid State Drives (SSD) to process input/output (I/O) bound workloads, as a log-structure-based merge tree (LSM tree) data structure.
Fig. 1 schematically shows an environment schematic according to an embodiment of the application, as shown:
the streaming data calculation engine 100 is connected to the memory 200 and the state data back-end storage system 300. During the operation of the streaming data calculation engine 100, state data may be generated, the state data may be written into a memory by the streaming data calculation engine 100, and the state data may be stored by a state data processing system deployed in the memory, and then, the state data processing system may migrate a part of the state data stored in the memory to the state data backend storage system 300 according to a preset data migration policy, and at the same time, may migrate a part of the state data stored by the state data backend storage system 300 back to the memory. Thus, when the state data needs to be acquired, the state data can be directly acquired from the memory first, and when the state data to be acquired does not exist in the memory, the state data backend storage system 300 is queried, so as to acquire the state data from the state data backend storage system 300.
The streaming data computation engine 100 may include, but is not limited to, a computation engine that is running to generate state data, such as Flink, where state data refers to data that is intermediate results generated during a computation. The state data back-end storage system 300 is a storage system for persisting state data, such as an LSM-Tree (LSM Tree) database, which may be RocksDB or Hbase, specifically.
In the related art, when status data generated during the Flink operation is directly stored in RocksDB, the status data needs to be read from a disk when the status data needs to be read, and therefore, a large amount of I/O resources need to be consumed when the status data is read.
According to the state data processing scheme, the Flink can be preferentially obtained from the memory when the state data are read, and the consumed I/O resources are less when the state data are obtained from the memory.
In the following, several embodiments will be provided in the above exemplary application environment to illustrate the state data processing scheme.
Example one
Fig. 2 schematically shows an architecture diagram of a state data processing system 20 according to a first embodiment of the present application. The state data processing system is deployed in a memory, and the state data processing system 20 may include a state data caching module 21, a state data management module 22, and a state data migration management module 23.
And the state data caching module 21 is configured to receive and store state data to be written.
After the streaming data calculation engine generates the state data, the state data is written into the state data caching module 21.
In this embodiment, the status data is divided according to a basic type, and may include Keyed status data (Keyed status Date) and Operator status data (Operator status Date).
Among them, keyed State is a State type related to Key, and it can only be used by methods based on operations above KeyedStream. We can logically understand that this state is the correspondence of a parallelism-operator-instance and a Key, < parallel-operator-instance, key >.
Operator State, which is a State type that is not Key-related. Accordingly, we understand the concept logically, which is equivalent to a parallelism instance, corresponding to a piece of state data. Since the concept of Key is not referred to here, there is a process of redistribution of state data when the parallelism (expansion/contraction) changes.
In this embodiment, the status data is preferably keying status data.
The state data caching module 21 stores the state in the form of a state table (state table), and the copyonwritesetmap is used in the state table to actually store the data. Wherein, the copyonWriteStateMap is a hash table formed by an array + linked list. The element types in copyonwritesetmapare all: stateMapEntry. The first layer of the hash table is an array of StateMapEntry types, i.e.: statemApEntry [ ]. In the StatemAppEntry class, there is a StatemAppEntry next pointer to form a linked list.
In this embodiment, the state data caching module 21 is created in the memory as a coordinator of resources, and is used to access the underlying state data backend storage system in an agent manner.
The state data management module 22 is configured to monitor a use condition of the memory, acquire a read-write condition of state data, and determine a migrated state data set according to the use condition of the memory, the read-write condition of the state data, and a preset data migration policy, where the state data set includes state data to be migrated from the state data cache module to a state data back-end storage system and state data to be migrated from the state data back-end storage system to the state data cache module.
Specifically, the use condition of the memory includes a memory use rate. In one embodiment, when the memory includes heap memory and out-of-heap memory, the memory usage includes usage rates of the heap memory and the out-of-heap memory.
The read-write condition of the state data may include, but is not limited to, how many pieces of state data are written in a preset time, how many pieces of state data are read, the sizes of the written and read state data obtained by sampling, which pieces of state data are frequently read, which pieces of state data are rarely read, and the like.
The data migration policy is a policy for performing a migration operation on data, and the data migration policy may include, but is not limited to, an LRU (Least Recently Used) policy, an LFU (Least frequently Used) policy, a FIFO (First Input First Output) policy, and the like.
In an optional embodiment, the status data management module 22 includes a memory monitoring unit, a data reading and writing index collecting unit, and a migration policy unit.
And the memory monitoring unit is used for monitoring the service condition of the memory.
In an embodiment, the memory monitoring unit may monitor a system memory status indicator, for example, related data such as a GC indicator of the JVM, besides the memory usage.
And the data reading and writing index acquisition unit is used for acquiring the reading and writing conditions of the state data, wherein the reading and writing conditions comprise the writing conditions of the state data stored in the state data caching module and the reading and writing conditions of the state data in the state data rear-end storage system.
In an embodiment, the data reading and writing index collecting unit may collect indexes such as a hit rate of the memory data, a state data size sampling estimation value, and the like, in addition to the reading and writing condition of the state data. When the data reading and writing index acquisition unit acquires the size sampling estimated value of the state data, the Memory Estimator can be called to realize the data acquisition.
And the migration strategy unit is used for configuring the data migration strategy by a user and determining a migrated state data set according to the use condition of the memory, the read-write condition of the state data and a preset data migration strategy.
Specifically, a user may configure a data migration policy through the migration policy unit, where when configuring the data migration policy, the user may select one data migration policy from a plurality of and set data migration policies as the data migration policy configured by the user, or the user may customize one data migration policy as the data migration policy configured by the user according to an actual situation.
In this embodiment, after the data reading and writing index collecting unit obtains the use condition of the memory from the memory monitoring unit and obtains the reading and writing condition of the state data from the data reading and writing index collecting unit, a state data set to be migrated is determined according to the data and a data migration policy configured by a user, where the state data set includes all state data to be migrated.
The state data migration management module 23 is configured to migrate the state data to be migrated from the state data caching module 21 to the state data backend storage system, and migrate the state data to be migrated from the state data backend storage system to the state data caching module 21.
In an alternative embodiment, the status data migration management module 23 may include a scheduling unit and a data migration unit.
And the scheduling unit is used for triggering the data migration task.
In this embodiment, the scheduling unit may trigger the data migration task at regular time, or may trigger the data migration task as needed, where a time interval for triggering the data migration task in a timing manner may be set and modified according to an actual situation, and is not limited in this embodiment. In some scenarios, the data migration task may be triggered immediately after the state data is written, so as to prevent a memory overflow condition when a large amount of state data is written.
In a specific scenario, when the scheduling unit triggers the data migration task, the state data set that needs to perform the data migration operation is first taken from the migration policy unit, and then the data migration task is triggered, so that the data migration unit can implement data migration according to the data migration task.
And a data migration unit, configured to, when receiving the data migration task, migrate the state data to be migrated from the state data caching module 21 to the state data backend storage system in the state data backend storage system to the state data backend storage system, and migrate the state data to be migrated from the state data backend storage system to the state data caching module 21.
In this embodiment, when performing data migration, the data migration unit serializes the state data that needs to be migrated to the state data backend storage system, and then writes the serialized state data into the state data backend storage system in batch. Meanwhile, the state data to be migrated to the state data cache module 21 is read out in batch from the state data back-end storage system, then deserialized, and finally the deserialized state data is written into the state data cache module 21.
In this embodiment, the state data that needs to be migrated to the state data backend storage system is generally "cold data," which is state data that is rarely used. The state data cache module 21 to be migrated is generally "temperature data", which is the state data to be used recently.
In an exemplary embodiment, in order to increase the utilization rate of the memory, the state data migration management module 23 further includes: and a memory management unit.
The memory management unit is used for executing memory application operation before migrating the state data in the state data back-end storage system to the state data cache module; and executing memory recovery operation after migrating the state data in the state data caching module to a state data back-end storage system.
In this embodiment, the memory management unit is used as a component responsible for memory application and recovery, and executes a memory application operation before state data in the state data back-end storage system needs to be migrated to the state data caching module, so as to obtain a memory space to store the migrated state data. After the state data is migrated from the state data caching module to the state data back-end storage system, the memory recovery operation is executed, so that the memory space can be timely recovered for storing other state data.
In a specific scenario, only heap memory may be used as a storage space of the state data caching module to store state data.
In an exemplary embodiment, the state data caching module 21 is further configured to:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system, and triggering the state data back-end storage system to execute data snapshot operation according to the Checkpoint operation after the data writing operation is completed.
Among them, the Checkpoint mechanism is a cornerstone to guarantee the reliability of the streaming data computation engine. When a task fails in the running process, the task can be restored to a certain state before the failure according to the information of Checkpoint, and then the running of the task is restored from the state. In the Flink, a Checkpoint mechanism adopts a chandy-snapshot algorithm, and the exact Once semantics in the Flink program are ensured through the Checkpoint mechanism.
In an exemplary embodiment, in order to improve snapshot efficiency, the state data caching module 21 is further configured to:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system in an asynchronous mode, so that the state data back-end storage system executes data snapshot operation according to the Checkpoint operation in an asynchronous mode.
In this embodiment, the state data stored in the state data caching module is written into the state data backend storage system in an asynchronous manner, so that the snapshot operation in the state data caching module 21 and the snapshot operation in the state data backend storage system can be triggered at the same time.
In the state data processing system of this embodiment, when the streaming data calculation engine generates state data in the running process, the state data is written into the memory first, and the state data is stored by the state data cache module in the state data processing system deployed in the memory. And finally, the state data migration management module executes the data migration operation to migrate part of the state data stored in the memory to the state data rear-end storage system, and simultaneously migrates part of the state data stored in the state data rear-end storage system back to the memory. Therefore, when the state data needs to be acquired, the state data can be directly acquired from the memory, and the state data back-end storage system is queried only when the state data to be acquired does not exist in the memory, so that the state data is acquired from the state data back-end storage system. Because the consumption of the I/O resources for directly acquiring the state data from the memory is much lower than that of the I/O resources for directly acquiring the state data from the state data rear-end storage system, the state data processing system can obviously reduce the consumption of the I/O resources and improve the read-write performance of the state data when acquiring the state, and the speed of acquiring the state data is higher.
Example two
The present embodiment provides a status data processing method, which is applied to the status data processing system in the above embodiment, and reference may be made to the above for details and effects.
Fig. 3 schematically shows a flow chart of a state data processing method according to the second embodiment of the present application.
As shown in fig. 3, the status data processing method may include steps S30 to S33, in which:
step S30, receiving a state data query request;
step S31, searching whether the state data cache module has the query data corresponding to the state data query request;
step S32, if the query data exists in the state data cache module, acquiring the query data from the state data cache module;
step S33, if the query data does not exist in the state data caching module, obtaining the query data from a state data back-end storage system.
In this embodiment, when the state data needs to be obtained, it is first queried whether the queried data exists in the state data caching module, and if so, it is directly obtained from the state data caching module, and if not, it is then continued to obtain the queried data from the state data backend storage system.
In an exemplary embodiment, if the query data does not exist in the state data caching module, after the step of obtaining the query data from the state data back-end storage system, the method further includes: and writing the query data into the state data caching module.
In this embodiment, after obtaining query data from the state data caching module, the query data may be written into the state data caching module for facilitating subsequent query, so that when the data needs to be obtained again subsequently, the query data may be directly obtained from the state data caching module.
In an exemplary embodiment, the method further comprises:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system, and triggering the state data back-end storage system to execute data snapshot operation according to the Checkpoint operation after the data writing operation is completed.
For ease of understanding, how the stateful data processing system implements Checkpoint operations is described below in conjunction with FIG. 4.
When receiving a Checkpoint operation, the State data processing system first performs snapshot processing on State data (which may be 1-N in a corresponding Map) stored in the State data caching module to save snapshot data. In the process of executing the snapshot, since the data in the state data caching module is stored by using a data structure of the copyonwritesettable, in the snapshot process, the copyonwritesetmappshot in the copyonwritesettable generates a copyonwritesetmappshot, and a linked list of data entries requiring snapshot is stored in the copyonwritesetmappshot. And then, writing the snapshot data in the copyonwritesetmappshost into the state data back-end storage system in batch, and continuously triggering the state data back-end storage system to execute data snapshot operation according to the Checkpoint operation after finishing the data writing operation.
In an exemplary embodiment, the method further comprises:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system in an asynchronous mode, so that the state data back-end storage system executes data snapshot operation according to the Checkpoint operation in an asynchronous mode.
For ease of understanding, how the stateful data processing system implements Checkpoint operations is described below in conjunction with FIG. 5.
When a State data processing system receives a Checkpoint operation, snapshot processing is triggered to be performed on State data (which may be 1-N in correspondence to Copy on Write State Map i, i in the figure) stored in a State data cache module to save snapshot data, and snapshot processing is triggered to be performed on State data (which corresponds to backup States Date in the figure) stored in a State data back-end storage system to save snapshot data in an asynchronous manner. In the process of executing the snapshot, since the data in the state data caching module is stored by using a data structure of the copyonwritedatasheet, in the snapshot process, a copyonwritesetemapsnaphot is generated by a copyonwritesetmap in the copyonwritedatasheet, and a linked list of data entries requiring a snapshot is stored in the copyonwritesetemapsnaphot. Thereafter, the snapshot data in copyonwritestatemapsnaphot is written in batch in an asynchronous manner into a data table (corresponding to Cache States data snapshot in the figure) in the state data back-end storage system. And after all the data are written into the state data back-end storage system, continuously carrying out Snapshot processing on the part of data in the state data back-end storage system.
EXAMPLE III
Fig. 6 is a block diagram of a state data processing apparatus according to a third embodiment of the present application, which may be divided into one or more program modules, the one or more program modules being stored in a storage medium and executed by one or more processors to implement the third embodiment of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments that can perform specific functions, and the following description will specifically describe the functions of the program modules in the embodiments.
As shown in fig. 6, the status data processing apparatus 50 may include a receiving module 51, a searching module 52 and an obtaining module 53, wherein:
a receiving module 51, configured to receive a status data query request;
the searching module 52 is configured to search whether query data corresponding to the status data query request exists in the status data caching module;
an obtaining module 53, configured to obtain the query data from the status data caching module if the query data exists in the status data caching module; and if the query data does not exist in the state data caching module, acquiring the query data from a state data rear-end storage system.
In an exemplary embodiment, the apparatus further includes a write module configured to:
and writing the query data into the state data caching module.
In an exemplary embodiment, the apparatus further comprises a snapshot module configured to:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system, and triggering the state data back-end storage system to execute data snapshot operation according to the Checkpoint operation after the data writing operation is completed.
In an exemplary embodiment, the snapshot module is further configured to:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system in an asynchronous mode, so that the state data back-end storage system executes data snapshot operation according to the Checkpoint operation in an asynchronous mode.
Example four
Fig. 7 schematically shows a hardware architecture diagram of a computer device suitable for implementing the state data processing method according to a fourth embodiment of the present application. The computer device 10000 is a device capable of automatically performing numerical calculation and/or information processing according to a command set in advance or stored. For example, the server may be a rack server, a blade server, a tower server, or a rack server (including an independent server or a server cluster composed of a plurality of servers). As shown in fig. 7, computer device 10000 includes at least, but is not limited to: the memory 10010, the processor 10020, and the network interface 10030 may be communicatively linked to each other through a system bus. Wherein:
the memory 10010 includes at least one type of computer-readable storage medium including a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the storage 10010 may be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device 10000. Of course, the memory 10010 may also include both internal and external memory modules of the computer device 10000. In this embodiment, the memory 10010 is generally used for storing an operating system installed in the computer device 10000 and various application software, such as program codes of the status data processing method. In addition, the memory 10010 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 10020, in some embodiments, can be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip. The processor 10020 is generally configured to control overall operations of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is configured to execute program codes stored in the memory 10010 or process data.
Network interface 10030 may comprise a wireless network interface or a wired network interface, and network interface 10030 is generally configured to establish a communication connection between computer device 10000 and another computer device. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, establish a data transmission channel and a communication connection between the computer device 10000 and the external terminal, and the like. The network may be an Intranet (Internet), the Internet (Internet), a Global System of Mobile communication (GSM), wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, bluetooth (Bluetooth), wi-Fi, or other wireless or wired network.
It should be noted that fig. 7 only illustrates a computer device having components 10010-10030, but it is to be understood that not all illustrated components need be implemented and that more or fewer components may be implemented instead.
In this embodiment, the status data processing method stored in the memory 10010 can be further divided into one or more program modules, and executed by one or more processors (in this embodiment, the processor 10020) to complete the present application.
EXAMPLE five
The present embodiment also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the state data processing method in the embodiments.
In this embodiment, the computer-readable storage medium includes a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the computer readable storage medium may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium may be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device. Of course, the computer-readable storage medium may also include both internal and external storage devices of the computer device. In the present embodiment, the computer-readable storage medium is generally used for storing an operating system and various types of application software installed in the computer device, for example, the program code of the state data processing method in the embodiment, and the like. Further, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the present application described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different from that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are included in the scope of the present application.
Claims (14)
1. A state data processing system deployed in a memory, the system comprising:
the state data caching module is used for receiving and storing state data to be written;
the state data management module is used for monitoring the use condition of the memory, acquiring the read-write condition of state data, and determining a migrated state data set according to the use condition of the memory, the read-write condition of the state data and a preset data migration strategy, wherein the state data set comprises the state data to be migrated from the state data cache module to a state data back-end storage system and the state data to be migrated from the state data back-end storage system to the state data cache module;
and the state data migration management module is used for migrating the state data to be migrated from the state data cache module to the state data back-end storage system and migrating the state data to be migrated from the state data back-end storage system to the state data cache module.
2. The system of claim 1, wherein the state data management module comprises:
the memory monitoring unit is used for monitoring the use condition of the memory;
the data reading and writing index acquisition unit is used for acquiring the reading and writing conditions of the state data, wherein the reading and writing conditions comprise the writing conditions of the state data stored in the state data cache module and the reading and writing conditions of the state data in the state data back-end storage system; and
and the migration policy unit is used for configuring the data migration policy by a user and determining a migrated state data set according to the use condition of the memory, the read-write condition of the state data and a preset data migration policy.
3. The system of claim 1, wherein the state data migration management module comprises:
the scheduling unit is used for triggering a data migration task; and
and the data migration unit is used for migrating the state data to be migrated from the state data cache module to the state data back-end storage system and migrating the state data to be migrated from the state data back-end storage system to the state data cache module when the data migration task is received.
4. The system of claim 3, wherein the state data migration management module further comprises:
the memory management unit is used for executing memory application operation before migrating the state data in the state data back-end storage system to the state data cache module; and executing memory recovery operation after migrating the state data in the state data caching module to a state data back-end storage system.
5. The system of claim 1, wherein the state data caching module is further configured to:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system, and triggering the state data back-end storage system to execute data snapshot operation according to the Checkpoint operation after the data writing operation is completed.
6. The system of claim 1, wherein the meta-state data caching module is further configured to:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system in an asynchronous mode, so that the state data back-end storage system executes data snapshot operation according to the Checkpoint operation in an asynchronous mode.
7. The system of claim 1, wherein the status data is keyed status data.
8. A status data processing method applied to the status data processing system according to any one of claims 1 to 7, the method comprising:
receiving a state data query request;
searching whether query data corresponding to the state data query request exists in a state data cache module;
if the query data exists in the state data cache module, acquiring the query data from the state data cache module;
and if the query data does not exist in the state data caching module, acquiring the query data from a state data rear-end storage system.
9. The method according to claim 8, wherein after the step of obtaining the query data from the state data backend storage system if the query data does not exist in the state data caching module, the method further comprises:
and writing the query data into the state data caching module.
10. The method of claim 8, further comprising:
and when the Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system, and triggering the state data back-end storage system to execute data snapshot operation according to the Checkpoint operation after the data writing operation is completed.
11. The method of claim 8, further comprising:
and when a Checkpoint operation is received, writing the state data stored in the state data caching module into the state data back-end storage system in an asynchronous mode, so that the state data back-end storage system executes data snapshot operation according to the Checkpoint operation in an asynchronous mode.
12. A state data processing apparatus, characterized in that the apparatus comprises:
the receiving module is used for receiving a state data query request;
the searching module is used for searching whether the state data caching module has the query data corresponding to the state data query request;
an obtaining module, configured to obtain the query data from the state data caching module if the query data exists in the state data caching module; and if the query data does not exist in the state data caching module, acquiring the query data from a state data rear-end storage system.
13. A computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor is adapted to carry out the steps of the state data processing method according to any one of claims 8 to 11 when executing the computer program.
14. A computer-readable storage medium, on which a computer program is stored, which computer program is executable by at least one processor to cause the at least one processor to carry out the steps of the state data processing method according to any one of claims 8 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211493539.6A CN115757352A (en) | 2022-11-25 | 2022-11-25 | State data processing system, method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211493539.6A CN115757352A (en) | 2022-11-25 | 2022-11-25 | State data processing system, method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115757352A true CN115757352A (en) | 2023-03-07 |
Family
ID=85338382
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211493539.6A Pending CN115757352A (en) | 2022-11-25 | 2022-11-25 | State data processing system, method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115757352A (en) |
-
2022
- 2022-11-25 CN CN202211493539.6A patent/CN115757352A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10657101B2 (en) | Techniques for implementing hybrid flash/HDD-based virtual disk files | |
| US9182927B2 (en) | Techniques for implementing hybrid flash/HDD-based virtual disk files | |
| Park et al. | {iJournaling}:{Fine-Grained} Journaling for Improving the Latency of Fsync System Call | |
| US9946569B1 (en) | Virtual machine bring-up with on-demand processing of storage requests | |
| US9280300B2 (en) | Techniques for dynamically relocating virtual disk file blocks between flash storage and HDD-based storage | |
| US9672245B2 (en) | Memory storage apparatus, method of supporting transaction function for database, and memory system | |
| US9274798B2 (en) | Multi-threaded logging | |
| US9292359B2 (en) | System and method for memory management | |
| US10049036B2 (en) | Reliable distributed messaging using non-volatile system memory | |
| US9767015B1 (en) | Enhanced operating system integrity using non-volatile system memory | |
| US9760480B1 (en) | Enhanced logging using non-volatile system memory | |
| Laga et al. | Lynx: A learning linux prefetching mechanism for ssd performance model | |
| US20130111103A1 (en) | High-speed synchronous writes to persistent storage | |
| EP3262511B1 (en) | Automatic recovery of application cache warmth | |
| CN107003897B (en) | Monitoring utilization of transaction processing resources | |
| CN103034577A (en) | Method and device for positioning slow shutdown | |
| CN115793957A (en) | Method and device for writing data and computer storage medium | |
| US10089220B1 (en) | Saving state information resulting from non-idempotent operations in non-volatile system memory | |
| CN111694806A (en) | Transaction log caching method, device, equipment and storage medium | |
| CN115757352A (en) | State data processing system, method, device, computer equipment and storage medium | |
| WO2017078707A1 (en) | Method and apparatus for recovering in-memory data processing system | |
| Ren et al. | {Memory-Centric} Data Storage for Mobile Systems | |
| Yang et al. | Optimizing synchronization mechanism for block-based file systems using persistent memory | |
| Kalita et al. | DurableFS: a file system for NVRAM | |
| Song et al. | CADedup: High-performance consistency-aware deduplication based on persistent memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |