CN116107509A - Data processing method and device and electronic equipment - Google Patents
Data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN116107509A CN116107509A CN202310082303.1A CN202310082303A CN116107509A CN 116107509 A CN116107509 A CN 116107509A CN 202310082303 A CN202310082303 A CN 202310082303A CN 116107509 A CN116107509 A CN 116107509A
- Authority
- CN
- China
- Prior art keywords
- data
- storage area
- cached
- solid state
- state disk
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0622—Securing storage systems in relation to access
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the application discloses a data processing method, a device and equipment, wherein the data processing method comprises the following steps: when the residual memory space of the system is smaller than a first threshold value, storing data to be cached into a first storage area of the solid state disk; the physical address of each storage partition in the first storage area and the virtual address in the system meet the mapping relation; periodically obtaining the access times corresponding to the virtual addresses of the data to be cached; and migrating the data to be cached corresponding to the target virtual address with the access times smaller than a second threshold out of the first storage area.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, an apparatus, and an electronic device.
Background
The operating system of the computer has a SWAP mechanism, that is, a SWAP space exists on the solid state disk, and when the operating system decides that the physical memory space is to be allocated to the active process and the available physical memory is insufficient, the SWAP space is used. In general, the operating system will write partial data with low priority into the solid state disk first, and when the data is needed again, the operating system will read the needed data from the solid state disk. Most of the data written into the solid state disk are often hot data, and the frequency of repeated reading later is high.
However, under the current operating system environment and architecture of the computer, when the host writes data into the solid state disk, the solid state disk cannot distinguish the cold and hot degrees of the data, so that the written data are all subjected to the same cache processing, and the data stored into the solid state disk lack of systematic management, so that the subsequent processing efficiency of the hot data is affected.
Disclosure of Invention
The embodiment of the application expects to provide a data processing method, a data processing device and electronic equipment.
The technical scheme of the application is realized as follows:
an embodiment of a first aspect of the present application provides a data processing method, including:
when the residual memory space of the system is smaller than a first threshold value, storing data to be cached into a first storage area of the solid state disk; the physical address of each storage partition in the first storage area and the virtual address in the system meet the mapping relation;
periodically obtaining the access times corresponding to the virtual addresses of the data to be cached;
and migrating the data to be cached corresponding to the target virtual address with the access times smaller than a second threshold out of the first storage area.
Optionally, the physical address of each storage partition in the first storage area and the virtual address in the system satisfy a paging mapping mechanism.
Optionally, before storing the data to be cached in the first storage area of the solid state disk, the method further includes:
acquiring a first storage space required by storing the data to be cached;
if the residual space of the solid state disk is larger than the first storage space, a first storage area is created in the solid state disk.
Optionally, before storing the data to be cached in the first storage area of the solid state disk, the method further includes:
acquiring a first storage space required by storing the data to be cached;
and if the residual space of the solid state disk is smaller than the first storage space, releasing at least part of data in the used space of the solid state disk.
Optionally, the first storage area is a high performance area.
Optionally, after storing the data to be cached in the first storage area of the solid state disk, the method further includes:
and when the virtual address of the data to be cached is accessed, writing the data to be cached into a storage space of the system.
Optionally, the migrating the data to be cached corresponding to the target virtual address with the access frequency smaller than the second threshold out of the first storage area includes:
migrating the data to be cached corresponding to the target virtual address from the first storage area to a second storage area; the second storage area is an area except the first storage area in the solid state disk.
Optionally, after migrating the data to be cached corresponding to the virtual address with the access frequency smaller than the second threshold from the first storage area to the second storage area, the method further includes:
and clearing the physical space corresponding to the target virtual address in the solid state disk.
Embodiments of a second aspect of the present application provide a data processing apparatus comprising
The storage module is used for storing the data to be cached into the first storage area of the solid state disk when the residual memory space of the system is smaller than a first threshold value; each storage partition address in the first storage area and a virtual address in the system meet the mapping relation;
the acquisition module is used for periodically acquiring the access times corresponding to the virtual address of the data to be cached;
and the migration module is used for migrating the data to be cached corresponding to the virtual address with the access frequency smaller than a second threshold value out of the first storage area.
An embodiment of the third aspect of the present application provides an electronic device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, performs the steps of the method of the first aspect.
The embodiment of the application provides a data processing method, a device and equipment, wherein the data processing method comprises the following steps: when the residual memory space of the system is smaller than a first threshold value, storing data to be cached into a first storage area of the solid state disk; the physical address of each storage partition in the first storage area and the virtual address in the system meet the mapping relation; periodically obtaining the access times corresponding to the virtual addresses of the data to be cached; and migrating the data to be cached corresponding to the target virtual address with the access times smaller than a second threshold out of the first storage area. By adopting the technical scheme, the independent first storage area is created in the solid state disk and used for storing the data to be cached of the system, and cold and hot data are distinguished by detecting the access times corresponding to the virtual address of the data to be cached, so that the cold data in the first storage area are migrated, the data in the first storage area can be ensured to be hot data, the subsequent processing efficiency of the hot data is improved, and the time required for reading the hot data is reduced.
Drawings
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present application;
FIG. 2 is a schematic flow chart of data access according to an embodiment of the present application;
FIG. 3 is a flow chart of a data management mechanism according to an embodiment of the present disclosure;
FIG. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
Furthermore, the drawings are only schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.
The flow diagrams depicted in the figures are exemplary only and not necessarily all steps are included. For example, some steps may be decomposed, and some steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term "and/or" includes any and all combinations of the associated listed items.
In some embodiments, referring to fig. 1, fig. 1 is a flow chart of a data processing method according to an embodiment of the present application, where the data processing method includes:
step S110, when the residual memory space of the system is smaller than a first threshold value, storing data to be cached into a first storage area of the solid state disk; the physical address of each storage partition in the first storage area and the virtual address in the system meet the mapping relation.
When an operating system decides to allocate physical memory space for an active process, it often prefers to choose the physical memory of the system itself. However, when the occupancy rate of the physical memory is too high, the computer is blocked, the system application cannot normally run, and when the occupancy rate is severe, the CPU utilization rate may be increased suddenly, and the system crashes.
In this embodiment, a first threshold may be preset, and when the remaining memory space (physical memory) of the system is smaller than the first threshold, the physical hard disk is preferably selected to store the data to be cached. Here, the first threshold may be set in a proper proportion according to the size matching of the physical memory of the system. For example, when the remaining memory space of the system is less than 30% of the total memory of the system, the data to be cached is stored in the first storage area of the solid state disk. It should be noted that 30% is only an exemplary illustration, and the second threshold may be other values.
In one example, the data to be cached may include data such as program startup items and system startup items. The program starting item and the system starting item related data are stored into the appointed virtual memory, so that the corresponding starting item data can be read preferentially after the system is started, and the aim of accelerating the starting is fulfilled.
In some embodiments, the physical address of each storage partition in the first storage area and the virtual address within the system satisfy a page mapping mechanism.
In a virtual memory, addresses generated by a program are virtual addresses, which constitute a virtual address space, and these virtual addresses are mapped to physical addresses by an MMU (memory management unit). Paging divides memory into fixed length units, each unit being a page. For a virtual address space, the paging mechanism partitions the address space into fixed-size units, each called a page. For the physical address space, physical memory is abstracted into fixed-size units, each of which is called a page frame (frame).
In this embodiment, the system employs a page mapping mechanism, and the virtual address space is divided in units of pages, and is divided into a plurality of equal-sized pages. The physical address space is also divided into page frames, or page frames, per block in units of pages. Each virtual page may optionally correspond to a physical page frame or may correspond to a page file on a disk. The problem of fragmentation outside the memory caused by segmentation can be avoided by managing the memory through paging.
In some embodiments, before storing the data to be cached in the first storage area of the solid state disk, the method further includes:
acquiring a first storage space required for storing data to be cached;
if the remaining space of the solid state disk is larger than the first storage space, a first storage area is created in the solid state disk.
Before the data to be cached is stored in the first storage area of the solid state disk, the size of the storage space required for storing the data to be cached can be estimated, and whether the first storage area is enough to be created is determined by combining the residual space of the solid state disk. And creating a first storage area in the solid state disk only when the remaining space of the solid state disk is larger than the first storage space required for storing the data to be cached.
In some embodiments, before storing the data to be cached in the first storage area of the solid state disk, the method further includes:
acquiring a first storage space required for storing data to be cached;
and if the residual space of the solid state disk is smaller than the first storage space, releasing at least part of data in the used space of the solid state disk.
When the remaining space of the solid state disk is smaller than the first storage space required for storing the data to be cached, at least part of the data in the used space of the solid state disk can be released until the first storage area is enough to be created.
In an optional embodiment, the number of accesses corresponding to the data in the used space of the solid state disk in the preset time period may also be obtained, and if the number of accesses is smaller than the data of the third threshold, the part of the data is classified into the sub-thermal data. And if the residual space of the solid state disk is smaller than the first storage space, releasing the secondary thermal data, and reserving the thermal data with the access times not smaller than a third threshold value. Here, the sub-thermal data is thermal data having a relatively small number of accesses. The third threshold is greater than a threshold for dividing the cold data and the hot data.
In some embodiments, the first storage area is a high performance area.
Different from a common storage area, the high-performance storage area has larger data throughput, can store more data to be cached in unit time, is favorable for improving the processing efficiency of hot data, has smaller time delay, can reduce the time required by data storage and reading, and improves the response speed of a system.
Step S120, periodically obtaining the access times corresponding to the virtual addresses of the data to be cached.
The virtual address of the data to be cached corresponds to the physical address of the data to be cached in the solid state disk, and the access times corresponding to the virtual address represent the times of the data to be cached in the solid state disk. Here, the number of accesses corresponding to the virtual address of the data to be cached may be obtained at the same time interval. According to actual requirements, the interval duration can be flexibly adjusted.
Step S130, the data to be cached corresponding to the target virtual address with the access times smaller than the second threshold value is migrated out of the first storage area.
In this embodiment, the cold and hot data are divided by the second threshold, that is, if the number of accesses is smaller than the second threshold, the corresponding data to be cached are divided into cold data; and if the access times are greater than a second threshold value, dividing the corresponding data to be cached into hot data. The access times corresponding to the virtual addresses of the data to be cached are periodically obtained, and the data to be cached corresponding to the target virtual addresses with the access times smaller than the second threshold value are migrated out of the first storage area, so that the data in the first storage area can be ensured to be hot data, the subsequent processing efficiency of the hot data is improved, and the time required for reading the hot data is reduced. It should be noted that, the second threshold value is smaller than the third threshold value mentioned in the above embodiment.
In some embodiments, after storing the data to be cached in the first storage area of the solid state disk, the method further includes:
when the virtual address of the data to be cached is accessed, the data to be cached is written into the storage space of the system.
When the system CPU accesses data, firstly, whether the data to be accessed exist in a storage space of the system or not is confirmed, if not, the solid state disk is accessed through the virtual address, and if the data to be accessed exists in the solid state disk, the data to be accessed is read, and the read data to be cached is written into the storage space of the system.
In some embodiments, referring to fig. 2, fig. 2 is a schematic flow chart of data access provided in the embodiments of the present application; when the CPU accesses the data, firstly, whether the data exists in the physical memory is required to be judged, if yes, the data is directly mapped to the physical memory for access, if not, whether the data exists in the virtual memory is required to be judged again, and if the data is determined to exist in the virtual memory, the data can be read from the first storage area and written into the physical memory, and the data is accessed by being mapped to the physical memory.
In some embodiments, migrating the data to be cached corresponding to the target virtual address with the access frequency smaller than the second threshold out of the first storage area includes:
migrating data to be cached corresponding to the target virtual address from the first storage area to the second storage area; the second storage area is an area except the first storage area in the solid state disk.
As described in the above embodiment, the cold and hot data are divided by the second threshold, and if the number of accesses is smaller than the second threshold, the corresponding data to be cached is divided into the cold data. When the remaining space of the solid state disk is sufficient, a second storage area can be established in the solid state disk except for the first storage area and used for storing cold data. After the original hot data in the first storage area is converted into cold data, the data can be migrated from the first storage area to the second storage area, so that the data in the first storage area is ensured to be hot data.
In some embodiments, referring to fig. 3, fig. 3 is a flow chart of a data management mechanism provided in an embodiment of the present application; the access times corresponding to the virtual addresses of the data to be cached are obtained periodically, whether the access times are smaller than a second threshold value or not is judged, and if the access times are not smaller than the second threshold value, the corresponding data to be cached are continuously stored in the first storage area; otherwise, if the access times are smaller than the second threshold value, the corresponding data to be cached is migrated from the first storage area to the second storage area.
In some embodiments, after migrating the data to be cached corresponding to the virtual address having the access number less than the second threshold from the first storage area to the second storage area, the method further includes:
and (3) clearing the physical space corresponding to the target virtual address in the solid state disk.
The physical space of the solid state disk is limited, so that after the data to be cached corresponding to the virtual address with the access frequency smaller than the second threshold value is transferred from the first storage area to the second storage area, the physical space corresponding to the target virtual address in the solid state disk needs to be emptied, so that the utilization rate of the first storage area is improved.
According to the embodiment of the application, the independent first storage area is created in the solid state disk and used for storing the data to be cached of the system, and cold and hot data are distinguished by detecting the access times corresponding to the virtual address of the data to be cached, so that the cold data in the first storage area are migrated, the data in the first storage area can be ensured to be hot data, the subsequent processing efficiency of the hot data is improved, and the time required for reading the hot data is shortened.
In some embodiments, referring to fig. 4, fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application; an embodiment of the present application provides a data processing apparatus 400, including:
the storage module 410 is configured to store data to be cached in a first storage area of the solid state disk when a remaining memory space of the system is smaller than a first threshold; the addresses of all storage partitions in the first storage area and the virtual addresses in the system meet the mapping relation.
The obtaining module 420 is configured to periodically obtain the number of accesses corresponding to the virtual address of the data to be cached;
and a migration module 430, configured to migrate the data to be cached corresponding to the virtual address with the access number less than the second threshold out of the first storage area.
When an operating system decides to allocate physical memory space for an active process, it often prefers to choose the physical memory of the system itself. However, when the occupancy rate of the physical memory is too high, the computer is blocked, the system application cannot normally run, and when the occupancy rate is severe, the CPU utilization rate may be increased suddenly, and the system crashes.
In this embodiment, a first threshold may be preset, and when the remaining memory space (physical memory) of the system is smaller than the first threshold, the physical hard disk is preferably selected to store the data to be cached. Here, the first threshold may be set in a proper proportion according to the size matching of the physical memory of the system. For example, when the remaining memory space of the system is less than 30% of the total memory of the system, the data to be cached is stored in the first storage area of the solid state disk. It should be noted that 30% is only an exemplary illustration, and the second threshold may be other values.
In one example, the data to be cached may include data such as program startup items and system startup items. By storing the program starting item and the system starting item into the appointed virtual memory, corresponding starting item data can be read preferentially after the system is started, so that the aim of accelerating starting is fulfilled.
In some embodiments, the physical address of each storage partition in the first storage area and the virtual address within the system satisfy a page mapping mechanism.
In a virtual memory, addresses generated by a program are virtual addresses, which constitute a virtual address space, and these virtual addresses are mapped to physical addresses by an MMU (memory management unit). Paging divides memory into fixed length units, each unit being a page. For a virtual address space, the paging mechanism partitions the address space into fixed-size units, each called a page. For the physical address space, physical memory is abstracted into fixed-size units, each of which is called a page frame (frame).
In this embodiment, the system employs a page mapping mechanism, and the virtual address space is divided in units of pages, and is divided into a plurality of equal-sized pages. The physical address space is also divided into page frames, or page frames, per block in units of pages. Each virtual page may optionally correspond to a physical page frame or may correspond to a page file on a disk. The problem of fragmentation outside the memory caused by segmentation can be avoided by managing the memory through paging.
In some embodiments, the data processing apparatus 400 further comprises a creation module, specifically configured to:
acquiring a first storage space required for storing data to be cached;
if the residual space of the solid state disk is larger than the first storage space, a first storage area is created in the solid state disk.
Before the data to be cached is stored in the first storage area of the solid state disk, the size of the storage space required for storing the data to be cached can be estimated, and whether the first storage area is enough to be created is determined by combining the residual space of the solid state disk. And creating a first storage area in the solid state disk only when the remaining space of the solid state disk is larger than the first storage space required for storing the data to be cached.
In some embodiments, the data processing apparatus 400 further comprises a release module, specifically configured to:
acquiring a first storage space required for storing data to be cached;
and if the residual space of the solid state disk is smaller than the first storage space, releasing at least part of data in the used space of the solid state disk.
When the remaining space of the solid state disk is smaller than the first storage space required for storing the data to be cached, at least part of the data in the used space of the solid state disk can be released until the first storage area is enough to be created.
In an optional embodiment, the number of accesses corresponding to the data in the used space of the solid state disk in the preset time period may also be obtained, and if the number of accesses is smaller than the data of the third threshold, the part of the data is classified into the sub-thermal data. And if the residual space of the solid state disk is smaller than the first storage space, releasing the secondary thermal data, and reserving the thermal data with the access times not smaller than a third threshold value. Here, the sub-thermal data is thermal data having a relatively small number of accesses. The third threshold is greater than a threshold for dividing the cold data and the hot data.
In some embodiments, the first storage area is a high performance area.
Different from a common storage area, the high-performance storage area has larger data throughput, can store more data to be cached in unit time, is favorable for improving the processing efficiency of hot data, has smaller time delay, can reduce the time required by data storage and reading, and improves the response speed of a system.
In this embodiment, the cold and hot data are divided by the second threshold, that is, if the number of accesses is smaller than the second threshold, the corresponding data to be cached are divided into cold data; and if the access times are greater than a second threshold value, dividing the corresponding data to be cached into hot data. The access times corresponding to the virtual addresses of the data to be cached are periodically obtained, and the data to be cached corresponding to the target virtual addresses with the access times smaller than the second threshold value are migrated out of the first storage area, so that the data in the first storage area can be ensured to be hot data, the subsequent processing efficiency of the hot data is improved, and the time required for reading the hot data is reduced. It should be noted that, the second threshold value is smaller than the third threshold value mentioned in the above embodiment.
In some embodiments, the data processing apparatus 400 further comprises a writing module, specifically configured to:
when the virtual address of the data to be cached is accessed, the data to be cached is written into the storage space of the system.
When the system CPU accesses data, firstly, whether the data to be accessed exist in a storage space of the system or not is confirmed, if not, the solid state disk is accessed through the virtual address, and if the data to be accessed exists in the solid state disk, the data to be accessed is read, and the read data to be cached is written into the storage space of the system.
In some embodiments, migrating the data to be cached corresponding to the target virtual address with the access frequency smaller than the second threshold out of the first storage area includes:
migrating data to be cached corresponding to the target virtual address from the first storage area to the second storage area; the second storage area is an area except the first storage area in the solid state disk.
As described in the above embodiment, the cold and hot data are divided by the second threshold, and if the number of accesses is smaller than the second threshold, the corresponding data to be cached is divided into the cold data. When the remaining space of the solid state disk is sufficient, a second storage area can be established in the solid state disk except for the first storage area and used for storing cold data. After the original hot data in the first storage area is converted into cold data, the data can be migrated from the first storage area to the second storage area, so that the data in the first storage area is ensured to be hot data.
In some embodiments, the data processing apparatus 400 further comprises a flushing module, specifically configured to:
and (3) clearing the physical space corresponding to the target virtual address in the solid state disk.
The physical space of the solid state disk is limited, so that after the data to be cached corresponding to the virtual address with the access frequency smaller than the second threshold value is transferred from the first storage area to the second storage area, the physical space corresponding to the target virtual address in the solid state disk needs to be emptied, so that the utilization rate of the first storage area is improved.
According to the embodiment of the application, the independent first storage area is created in the solid state disk and used for storing the data to be cached of the system, and cold and hot data are distinguished by detecting the access times corresponding to the virtual address of the data to be cached, so that the cold data in the first storage area are migrated, the data in the first storage area can be ensured to be hot data, the subsequent processing efficiency of the hot data is improved, and the time required for reading the hot data is shortened.
It should be noted here that: the description of the data processing apparatus embodiments above is similar to that of the data processing method embodiments above, with similar advantageous effects as those of the data processing method embodiments. For technical details not disclosed in the embodiments of the data processing apparatus of the present application, please refer to the description of the embodiments of the data processing method of the present application, which is not described herein in detail.
In some embodiments, referring to fig. 5, fig. 5 is a schematic structural diagram of an electronic device provided in the embodiments of the present application, where the electronic device includes a memory and a processor, and the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the data processing method.
The electronic device may be a terminal, a server or similar computing means. The electronic device may vary considerably in configuration or performance and may include one or more central processing units (Central Processing Units, CPU) including, but not limited to, a microprocessor MCU or programmable logic device FPGA, memory for storing data, one or more storage media (e.g., one or more mass storage devices) for storing applications or data. The memory and storage medium may be transitory or persistent. The program stored on the storage medium may include one or more modules, each of which may include a series of instruction operations in the electronic device. Still further, the central processor may be configured to communicate with a storage medium and execute a series of instruction operations in the storage medium on an electronic device. The electronic device may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input/output interfaces, and/or one or more operating systems, such as Windows Server, mac OS X, unix, linux, freeBSD, etc. The input-output interface may be used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the electronic device.
In one example, the input-output interface includes a network adapter (Network Interface Controller, NIC) that can connect to other network devices through the base station to communicate with the internet. In an exemplary embodiment, the input/output interface may be a Radio Frequency (RF) module for communicating with the internet wirelessly.
It will be appreciated by those of ordinary skill in the art that the configuration shown in fig. 5 is merely illustrative and is not intended to limit the configuration of the electronic device described above. For example, the electronic device may also include more or fewer components than shown in FIG. 5, or have a different configuration than shown in FIG. 5.
In some embodiments, embodiments of the present application provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data processing method described above.
It should be noted here that: the above description of the electronic device embodiments and the storage medium embodiments is similar to the description of the data processing method embodiments described above, with similar advantageous effects as the data processing method embodiments. For technical details not disclosed in the electronic device embodiments and the storage medium embodiments of the present application, please refer to the description of the data processing method embodiments of the present application, which are not described herein in detail.
In the several embodiments provided in the present application, it should be understood that the disclosed data processing method, apparatus, device, and storage medium may be implemented in other manners. The above-described method, apparatus, electronic device, and storage medium embodiments are merely illustrative.
The embodiments described in the embodiments of the present application are merely examples of the embodiments of the present application, but are not limited thereto, and the data processing method, the apparatus, the electronic device, and the storage medium are all within the scope of the present application.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic thereof, and should not constitute any limitation on the implementation process of the embodiments of the present application. The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about or replace the present application within the technical scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A data processing method, comprising:
when the residual memory space of the system is smaller than a first threshold value, storing data to be cached into a first storage area of the solid state disk; the physical address of each storage partition in the first storage area and the virtual address in the system meet the mapping relation;
periodically obtaining the access times corresponding to the virtual addresses of the data to be cached;
and migrating the data to be cached corresponding to the target virtual address with the access times smaller than a second threshold out of the first storage area.
2. The data processing method of claim 1, wherein the physical address of each storage partition in the first storage area and the virtual address within the system satisfy a page mapping mechanism.
3. The data processing method according to claim 1, further comprising, before storing the data to be cached in the first storage area of the solid state disk:
acquiring a first storage space required by storing the data to be cached;
if the residual space of the solid state disk is larger than the first storage space, a first storage area is created in the solid state disk.
4. The data processing method according to claim 1, further comprising, before storing the data to be cached in the first storage area of the solid state disk:
acquiring a first storage space required by storing the data to be cached;
and if the residual space of the solid state disk is smaller than the first storage space, releasing at least part of data in the used space of the solid state disk.
5. The data processing method according to claim 1, wherein the first memory area is a high-performance area.
6. The data processing method according to claim 1, after storing the data to be cached in the first storage area of the solid state disk, the method further comprising:
and when the virtual address of the data to be cached is accessed, writing the data to be cached into a storage space of the system.
7. The data processing method according to claim 1, wherein the migrating the data to be cached corresponding to the target virtual address with the access number smaller than the second threshold out of the first storage area includes:
migrating the data to be cached corresponding to the target virtual address from the first storage area to a second storage area; the second storage area is an area except the first storage area in the solid state disk.
8. The data processing method according to claim 7, after migrating data to be cached corresponding to the virtual address for which the access number is smaller than a second threshold from the first storage area to a second storage area, the method further comprising:
and clearing the physical space corresponding to the target virtual address in the solid state disk.
9. A data processing apparatus comprising:
the storage module is used for storing the data to be cached into the first storage area of the solid state disk when the residual memory space of the system is smaller than a first threshold value; each storage partition address in the first storage area and a virtual address in the system meet the mapping relation;
the acquisition module is used for periodically acquiring the access times corresponding to the virtual address of the data to be cached;
and the migration module is used for migrating the data to be cached corresponding to the virtual address with the access frequency smaller than a second threshold value out of the first storage area.
10. An electronic device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, performs the steps of the method of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310082303.1A CN116107509A (en) | 2023-01-31 | 2023-01-31 | Data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310082303.1A CN116107509A (en) | 2023-01-31 | 2023-01-31 | Data processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116107509A true CN116107509A (en) | 2023-05-12 |
Family
ID=86265143
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310082303.1A Pending CN116107509A (en) | 2023-01-31 | 2023-01-31 | Data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116107509A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116560585A (en) * | 2023-07-05 | 2023-08-08 | 支付宝(杭州)信息技术有限公司 | Data hierarchical storage method and system |
-
2023
- 2023-01-31 CN CN202310082303.1A patent/CN116107509A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116560585A (en) * | 2023-07-05 | 2023-08-08 | 支付宝(杭州)信息技术有限公司 | Data hierarchical storage method and system |
| CN116560585B (en) * | 2023-07-05 | 2024-04-09 | 支付宝(杭州)信息技术有限公司 | Data hierarchical storage method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10705965B2 (en) | Metadata loading in storage systems | |
| KR101786871B1 (en) | Apparatus for processing remote page fault and method thereof | |
| US8775755B2 (en) | Peer-to-peer transcendent memory | |
| US20150234669A1 (en) | Memory resource sharing among multiple compute nodes | |
| US20050097384A1 (en) | Data processing system with fabric for sharing an I/O device between logical partitions | |
| US11314689B2 (en) | Method, apparatus, and computer program product for indexing a file | |
| US20230214956A1 (en) | Resiliency Schemes for Distributed Storage Systems | |
| US6968382B2 (en) | Activating a volume group without a quorum of disks in the volume group being active | |
| JPH06222999A (en) | Swap-block assembling method and memory control mechanism | |
| US10268592B2 (en) | System, method and computer-readable medium for dynamically mapping a non-volatile memory store | |
| CN110865989A (en) | Business processing method for large-scale computing cluster | |
| CN110968529A (en) | Method and device for realizing non-cache solid state disk, computer equipment and storage medium | |
| US10922147B2 (en) | Storage system destaging based on synchronization object with watermark | |
| CN116107509A (en) | Data processing method and device and electronic equipment | |
| US11086558B2 (en) | Storage system with storage volume undelete functionality | |
| US10061725B2 (en) | Scanning memory for de-duplication using RDMA | |
| CN106537321B (en) | Method, device and storage system for accessing file | |
| CN115168317B (en) | LSM tree storage engine construction method and system | |
| US20230021883A1 (en) | Migrating virtual machines in cluster memory systems | |
| US20040073907A1 (en) | Method and system of determining attributes of a functional unit in a multiple processor computer system | |
| CN116483740B (en) | Memory data migration method and device, storage medium and electronic device | |
| US11150827B2 (en) | Storage system and duplicate data management method | |
| CN113448517B (en) | Solid state disk big data writing processing method, device, equipment and medium | |
| CN114489848B (en) | Task unloading method based on computable storage architecture and computable storage system | |
| WO2024082702A1 (en) | Data processing method and apparatus, and chip and computer-readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |