CN116303119B - Method, system and storage medium for identifying cold and hot data - Google Patents
Method, system and storage medium for identifying cold and hot data Download PDFInfo
- Publication number
- CN116303119B CN116303119B CN202310588422.4A CN202310588422A CN116303119B CN 116303119 B CN116303119 B CN 116303119B CN 202310588422 A CN202310588422 A CN 202310588422A CN 116303119 B CN116303119 B CN 116303119B
- Authority
- CN
- China
- Prior art keywords
- data
- cold
- hot
- frequency table
- block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000013507 mapping Methods 0.000 claims description 8
- 230000005540 biological transmission Effects 0.000 description 8
- 238000012545 processing Methods 0.000 description 4
- 230000006641 stabilisation Effects 0.000 description 4
- 238000011105 stabilization Methods 0.000 description 4
- 238000012546 transfer Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000007704 transition Effects 0.000 description 3
- 238000009825 accumulation Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
- G06F12/0238—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory
- G06F12/0246—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory in block erasable memory, e.g. flash memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
- G06F12/0253—Garbage collection, i.e. reclamation of unreferenced memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/0292—User address space allocation, e.g. contiguous or non contiguous base addressing using tables or multilevel address translation means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0632—Configuration or reconfiguration of storage systems by initialisation or re-initialisation of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Read Only Memory (AREA)
Abstract
The application discloses a cold and hot data identification method, a system and a storage medium, which relate to the technical field of storage and comprise the following steps: acquiring a logic block of data to be written and a first data frequency table; updating the write frequency of the logic block in the first data frequency table; judging whether the current moment is in an initialization time period or a stable time period of the current reset period; when the data is in the initialization time period, distinguishing whether the data to be written is hot data or cold data according to the second data frequency table; when the data is in the stable time period, distinguishing whether the data to be written is hot data or cold data according to the first data frequency table; storing the thermal data into a thermal block; and initializing a second data frequency table of the current reset period when the current reset period is ended, taking the first data frequency table of the current reset period as a second data frequency table of the next reset period, and taking the initialized second data frequency table as the first data frequency table of the next reset period. The application can effectively distinguish cold data from hot data.
Description
Technical Field
The present application relates to the field of storage technologies, and in particular, to a method, a system, and a storage medium for identifying cold and hot data.
Background
Direct memory access (Direct Memory Access, DMA) transfers are capable of copying data from one address space to another, providing high speed data transfer between a peripheral and memory or between memory and memory. The DMA transmission mode does not need direct control transmission of a CPU, and does not reserve the site and recover the site process like an interrupt processing mode, but opens up a direct data transmission channel for the RAM and the IO equipment through hardware to realize data transmission, so that the efficiency of the CPU is greatly improved.
In the related art, the flash memory device does not distinguish whether the written data is cold data or hot data, so that the cold data and the hot data are stored in a mixed manner, and therefore, when the hot data is recovered, part of the cold data is recovered, so that the amount of data recovered by garbage increases, thereby affecting the performance and the use of the flash memory device. How to effectively distinguish cold data from hot data is a problem that is currently in need of discussion and resolution.
Disclosure of Invention
The present application aims to solve at least one of the technical problems existing in the prior art. Therefore, the application provides a cold and hot data identification method, a cold and hot data identification system and a storage medium, which can effectively distinguish cold data from hot data and improve the performance of a flash memory device.
In order to solve the technical problems, the application provides the following technical scheme:
an embodiment of a first aspect of the present application provides a method for identifying hot and cold data, applied to a flash memory device, including:
acquiring a logic block of data to be written and a first data frequency table;
updating the write frequency of the logic block in the first data frequency table;
judging whether the current moment is in an initialization time period or a stable time period of the current reset period;
when the data is in the initialization time period, distinguishing whether the data to be written is hot data or cold data according to the second data frequency table;
when the data is in the stable time period, distinguishing whether the data to be written is hot data or cold data through the first data frequency table, wherein the second data frequency table is obtained in the stable time period of the last reset period;
storing the hot data into a hot block and the cold data into a cold block, wherein the hot block and the cold block are physical spaces divided in advance;
and initializing the second data frequency table in the current reset period when the current reset period is ended, taking the first data frequency table in the current reset period as the second data frequency table in the next reset period, and taking the initialized second data frequency table as the first data frequency table in the next reset period.
The method for identifying cold and hot data according to the embodiment of the first aspect of the application has at least the following beneficial effects: the cold and hot data identification method can acquire the logic block of the data to be written and the first data frequency table, update the writing frequency of the logic block in the first frequency table, judge the frequency table used for distinguishing the cold and hot data according to the current moment after updating the writing frequency, effectively distinguish the hot data from the cold data according to the first data frequency table or the second data frequency table, further realize the storage of the hot data into the pre-divided hot area block, store the cold data into the pre-divided cold area block, provide judgment basis for the split of the cold data and the hot data, and improve the performance of the flash memory device. Furthermore, the application carries out initialization and then accumulation again on the update of the first data frequency table according to each reset period, and carries out transition with the second data frequency table in the initialization time period of each reset period, thereby more effectively identifying the cold data and the hot data.
According to some embodiments of the first aspect of the present application, the logic for obtaining data to be written includes:
obtaining a plurality of logic blocks with equal proportion sizes according to the total physical space corresponding to the hot block and the cold block;
and obtaining the address of the data to be written, and matching the address of the data to be written with the logic blocks with the equal proportional sizes to obtain the logic block corresponding to the address of the data to be written.
According to some embodiments of the first aspect of the present application, the obtaining a plurality of logic blocks with equal proportional sizes according to the total physical space corresponding to the hot block and the cold block includes:
acquiring the maximum data length allowed to be transmitted by the master control for transmitting the data to be written;
accumulating the physical space of the hot zone block and the physical space of the cold zone block to obtain a total physical space;
and performing equal proportion logic division on the total physical space according to the maximum data length to obtain a plurality of logic blocks with the maximum data length.
According to some embodiments of the first aspect of the application, the method further comprises:
when the reset period at the current moment is the first reset period and the current moment is in the initialization time period, the data to be written received in the initialization time period are taken as cold data or hot data;
correspondingly, when the data to be written is hot data or cold data in the initialization period, the distinguishing by the second data frequency table includes:
and when the current moment is in the initialization time period and the current reset period is a reset period except the first reset period, distinguishing whether the data to be written is hot data or cold data according to the second data frequency table.
According to some embodiments of the first aspect of the present application, the updating the write frequency of the logic block in the first data frequency table includes:
matching the logic block with the first data frequency table to obtain the historical writing frequency of the logic block recorded by the first data frequency table;
and adding one to the historical write frequency to obtain the write frequency of the logic block at the current moment, and updating the write frequency into the first data frequency table.
According to some embodiments of the first aspect of the present application, the distinguishing, by the second data frequency table, whether the data to be written is hot data or cold data includes:
when the writing frequency corresponding to the data to be written is larger than a preset frequency threshold value, the data is hot data;
when the writing frequency corresponding to the data to be written is smaller than a preset frequency threshold value, the data is cold data;
wherein the frequency threshold is determined according to the reset period and the performance of the flash memory device.
According to some embodiments of the first aspect of the application, the storing the thermal data into a thermal block comprises:
acquiring the hot data length of the hot data and the logical address of the hot data;
determining a first storage space in the thermal block according to the thermal data length;
copying the thermal data to the first storage space;
establishing a mapping relation between the first storage space and the logical address;
the storing the cold data into a cold block includes:
acquiring the cold data length of the cold data and the logic address of the cold data;
determining a second storage space in the cold block according to the cold data length;
copying the cold data to the second storage space;
and establishing a mapping relation between the second storage space and the logic address.
An embodiment of a second aspect of the present application provides a system for identifying cold and hot data, including:
the acquisition module is used for acquiring a logic block of data to be written and a first data frequency table;
the updating module is used for updating the writing frequency of the logic block in the first data frequency table;
the judging module is used for judging whether the current moment is in the initialization time period or the stable time period of the current reset period; when the data is in the initialization time period, distinguishing whether the data to be written is hot data or cold data according to the second data frequency table; when the data is in the stable time period, distinguishing whether the data to be written is hot data or cold data through the first data frequency table, wherein the second data frequency table is obtained in the stable time period of the last reset period;
the storage module is used for storing the hot data into a hot block and storing the cold data into a cold block, wherein the hot block and the cold block are physical spaces divided in advance;
and the initialization module is used for initializing the second data frequency table in the current reset period when the current reset period is ended, taking the first data frequency table in the current reset period as the second data frequency table in the next reset period, and taking the initialized second data frequency table as the first data frequency table in the next reset period.
An embodiment of a third aspect of the present application provides a controller, comprising:
at least one memory;
at least one processor;
at least one program;
the program is stored in the memory, and the processor executes at least one of the programs to implement:
the method for identifying cold and hot data according to the first aspect of the present application.
A fourth aspect of the present application provides a computer-readable storage medium storing a computer-executable signal for performing:
the method for identifying cold and hot data according to the first aspect of the present application.
Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.
Drawings
Additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a main flow chart of a method for identifying cold and hot data according to some embodiments of the present application;
FIG. 2 is a flow chart of a method for obtaining logical blocks of data to be written according to some embodiments of the present application;
FIG. 3 is a flow chart of a method for obtaining a plurality of logic blocks with equal proportional sizes according to the total physical space corresponding to a hot block and a cold block according to some embodiments of the present application;
FIG. 4 is a flowchart of a method for updating the write frequency of a logic block in a first data frequency table according to some embodiments of the present application;
FIG. 5 is a flow chart of a method for storing thermal data into a thermal block according to some embodiments of the present application;
FIG. 6 is a flowchart of a method for storing cold data in a cold block according to some embodiments of the present application;
FIG. 7 is a block diagram of a method for identifying hot and cold data according to some embodiments of the present application;
fig. 8 is a block diagram of a controller according to some embodiments of the application.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
It should be noted that although a logical order is illustrated in the flowchart, in some cases, the steps illustrated or described may be performed in an order different from that in the flowchart. The terms and the like in the description and in the claims, and in the above-described drawings, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
In the description of the present application, the description of the first and second is only for the purpose of distinguishing technical features, and should not be construed as indicating or implying relative importance or implying the number of technical features indicated or the precedence of the technical features indicated.
In the description of the present application, unless explicitly defined otherwise, terms such as arrangement, installation, connection, etc. should be construed broadly and the specific meaning of the terms in the present application can be reasonably determined by a person skilled in the art in combination with the specific contents of the technical scheme.
Direct memory access (Direct Memory Access, DMA) transfers are capable of copying data from one address space to another, providing high speed data transfer between a peripheral and memory or between memory and memory. The DMA transmission mode does not need direct control transmission of a CPU, and does not reserve the site and recover the site process like an interrupt processing mode, but opens up a direct data transmission channel for the RAM and the IO equipment through hardware to realize data transmission, so that the efficiency of the CPU is greatly improved. In the related art, the flash memory device does not distinguish whether the written data is cold data or hot data, so that the cold data and the hot data are stored in a mixed manner, and therefore, when the hot data is recovered, part of the cold data is recovered, so that the amount of data recovered by garbage increases, thereby affecting the performance and the use of the flash memory device. How to effectively distinguish cold data from hot data is a problem that is currently in need of discussion and resolution.
Based on the method, the cold data and the hot data can be effectively distinguished, and the performance of the flash memory device is improved.
Referring to fig. 1, in a first aspect, an embodiment of the present application provides a method for identifying cold and hot data, including but not limited to step S110, step S120, step S130, step S140, step S150, step S160, and step S170.
Step S110, a logic block of data to be written and a first data frequency table are obtained;
step S120, updating the write frequency of the logic block in the first data frequency table;
step S130, judging whether the current moment is in the initialization time period or the stable time period of the current reset period;
step S140, when the data is in the initialization time period, distinguishing whether the data to be written is hot data or cold data through the second data frequency table;
step S150, distinguishing whether the data to be written is hot data or cold data through a first data frequency table when the data is in a stable time period, wherein a second data frequency table is obtained in the stable time period of the last reset period;
step S160, storing hot data into a hot block and cold data into a cold block, wherein the hot block and the cold block are pre-divided physical spaces;
step S170, initializing the second data frequency table of the current reset period and taking the first data frequency table of the current reset period as the second data frequency table of the next reset period when the current reset period is finished, and taking the initialized second data frequency table as the first data frequency table of the next reset period.
It should be noted that, the cold and hot data identification method of the present application can obtain the logic block of the data to be written and the first data frequency table, update the writing frequency of the logic block in the first frequency table, judge the used frequency table for distinguishing the cold and hot data according to the current time after updating the writing frequency, effectively distinguish the hot data and the cold data according to the first data frequency table or the second data frequency table, further realize the storage of the hot data into the pre-divided hot area block, and the storage of the cold data into the pre-divided cold area block, thereby providing a judging basis for the split of the cold data and the hot data and improving the performance of the flash memory device. Furthermore, the application carries out initialization and then accumulation again on the update of the first data frequency table according to each reset period, and carries out transition with the second data frequency table in the initialization time period of each reset period, thereby more effectively identifying the cold data and the hot data.
The first data frequency table is a data table used for recording the frequency of the current data to be written in real time in each reset period; the second data frequency table is a first data frequency table of the last reset period that can be used to distinguish between cold data and hot data. The first data frequency table and the second data frequency table are with respect to a reset period.
It should be noted that the physical space of the flash memory device is divided into a plurality of logic blocks, and the data to be written transmitted by the host is stored according to the logic blocks. The first data frequency table and the second data frequency table are each for recording the number of writes of the divided plurality of logical blocks.
According to one embodiment of the present application, when a host transmits data to a flash memory device, the present application first obtains a logic block to be written with data and a first data frequency table already established in the flash memory device in real time, and updates the write frequency of the logic block to be written with data. In order to ensure the real-time property of shunting the cold data and the hot data, before judging the cold data and the hot data, firstly, judging whether the current moment is in an initialization time period or a stable time period of a reset period, when the data to be written is written, distinguishing whether the data to be written is the hot data or the cold data through a second data frequency table, and when the data to be written is written, distinguishing whether the data to be written is the hot data or the cold data through a first data frequency table.
Hot data refers to frequently accessed online class data with high storage performance requirements, and cold data refers to infrequently accessed offline class data, such as backup and archive data, with relatively low storage performance requirements.
In addition, in order to facilitate the split flow of cold data and hot data, the application divides the physical space for storage in the flash memory device in advance to obtain a cold block and a hot block, wherein the cold block is used for storing the cold data, and the hot block is used for storing the hot data. After distinguishing hot data from cold data by the first data frequency table or the second data frequency table, the present application achieves splitting by storing hot data into hot blocks and storing cold data into cold blocks. The application considers the limitation of the space of the flash memory device, and by establishing two data frequency tables and distinguishing cold data and hot data by the logic blocks, the application effectively avoids the mixed storage condition of the cold data and the hot data, reduces the probability of misjudgment of the cold data to occupy a large amount of storage space, and improves the performance and the use of the flash memory device.
Meanwhile, because the usage scenes of the data transmitted by the host are different in each reset period, the second data frequency table established in the last reset period is possibly not suitable for the data transmitted by the host in the current reset period, and the limitation of the existing time is considered.
It should be noted that, the initialization time period is for updating the first data frequency table again in each reset stage, so as to ensure that cold and hot data can still be distinguished in the time period.
For example, the reset period of the present application is 12 hours, the initialization period is 3 hours, the stabilization period is 9 hours, i.e., every 12 hours, the first data frequency table is updated again and the transition is made using the second data frequency table during the initialization period. Such as: the time of the last read host for writing the data is 5:00 a.m., because the reset period is 12 hours, the second data frequency table obtained in the last reset period is automatically initialized when the reset period is 17:00 a.m., the data transmitted from the host to the flash memory device after 17:00 a corresponding writing frequency is read, and the data frequency table after initialization, namely the first data frequency table, is built according to the data read after 17:00 a and the corresponding writing frequency of the data. And the 3 hours from 17:00 to 20:00 are initialization time periods, the first data frequency table is still in an updating perfection stage within the 3 hours, so that the data frequency table for distinguishing cold and hot data in the 3 hours from 17:00 to 20:00 is still used for shunting in the application, until 20:00, the initialization time periods are finished, which means that the current reset period has completed updating the first data frequency table, and the attribute judgment of the cold data and the hot data of the logic block in the first data frequency table is unchanged from 20:00 to the corresponding reset period, so that the shunting of the cold data and the hot data is realized by using the first data frequency table.
According to another embodiment of the present application, the logic block of the data A, B, C to be written is obtained, the writing frequency of all the logic blocks in the first data frequency table is updated according to the logic block of the data A, B, C to be written, the writing frequency of the data a is 30/reset period, the writing frequency of the data B is 40/reset period, the writing frequency of the data C is 60/reset period, after the data is read, the present application determines whether the current time is in the initialization time period or the stable time period of the reset period, and the corresponding data frequency table is selected according to the current time to split. If the data with the access frequency below 50/period is cold data and the data with the access frequency above 50/period is hot data, the data A and the data B are cold data and the data C is hot data according to the corresponding data frequency table. Specifically, the usage scenario of the data written by the possible host in the next reset period is different from the usage scenario of the data a, the data B and the data C in the current reset period, and in order to ensure the accuracy of the data frequency table for data splitting in the present application, the data frequency table is initialized in the interval reset period, and the hot data and the cold data are distinguished according to the updated data frequency table.
Referring to fig. 2, in a first aspect, an embodiment of the present application provides a method for identifying cold and hot data, including but not limited to step S210 and step S220.
Step S210, obtaining a plurality of logic blocks with equal proportion size according to the total physical space corresponding to the hot block and the cold block;
step S220, the address of the data to be written is obtained, and the address of the data to be written is matched with a plurality of logic blocks with equal proportion sizes, so that the logic block corresponding to the address of the data to be written is obtained.
It should be noted that, in order to achieve the accurate update and reading of the write-in frequency of the data transmitted by the host, before the host transmits the data to the flash memory device, the application further performs equal proportion logic division on the thermal block and the cold block according to the total physical space corresponding to the thermal block and the cold block, so as to obtain a plurality of logic blocks with equal proportion sizes. After the flash memory device receives data transmitted by the host, the address of the data to be written is acquired, the address of the data to be written is matched with a plurality of logic blocks with equal proportion sizes, the logic blocks of the address of the data to be written are obtained, and meanwhile the writing frequency of the data is updated. Specifically, the method and the device have the advantages that the logic blocks are obtained by carrying out equal-proportion logic division on the physical space, and the data transmitted by the host computer is written into the logic blocks, so that the accuracy of the writing frequency of the data transmitted by the read host computer is ensured, and the accuracy of the cold and hot data identification method is improved.
Referring to fig. 3, in a first aspect, an embodiment of the present application provides a method for identifying cold and hot data, including but not limited to step S310, step S320, and step S330.
Step S310, obtaining the maximum data length allowed to be transmitted by the master control for transmitting the data to be written;
step S320, accumulating the physical space of the hot block and the physical space of the cold block to obtain a total physical space;
step S330, performing equal proportion logic division on the total physical space according to the maximum data length to obtain a plurality of logic blocks with the maximum data length.
According to one embodiment of the application, the first data frequency table and the second data frequency table record the times of writing the data address of each logic block in the flash memory device, in order to avoid the situation that the logic blocks cannot be written due to overlong data addresses transmitted by a host, and also in order to ensure that the logic blocks of the application can realize writing of the data addresses, the application firstly obtains the maximum data length allowed to be transmitted by a main control transmitting data to be written, and simultaneously accumulates the physical space of a hot block and the physical space of a cold block to obtain the total physical space, after that, the total physical space is logically divided in equal proportion according to the maximum data length to obtain a plurality of logic blocks, and the size of each logic block is the maximum data length, thereby effectively reducing the risk that the flash memory device cannot smoothly write data due to overlong data addresses transmitted by the host.
It will be appreciated that the method of the present application further comprises: when the reset period at the current moment is the first reset period and the current moment is in the initialization time period, the data to be written received in the initialization time period are taken as cold data or hot data; correspondingly, when the data is in the initialization period, distinguishing whether the data to be written is hot data or cold data through the second data frequency table comprises the following steps: when the current moment is in the initialization time period and the current reset period is the reset period except the first reset period, distinguishing whether the data to be written is hot data or cold data through the second data frequency chart.
According to one embodiment of the present application, the reset period of the present application is 12 hours, the initialization period is 3 hours, and the stabilization period is 9 hours, i.e., the present application initializes the data frequency table at intervals of 12 hours, and perfects the data frequency table using 3 hours after the initialization. The first reset period starts at 00:00, the initialization period is 00:00-03:00, the stabilization period is 3:00-12:00, the second reset period starts at 12:00, the initialization period is 12:00-15:00, and the stabilization period is 15:00-24:00. When the reset period at the current moment is the first reset period and the time of the logic block to be written with data is in an initialization time period, namely the current moment is between 00:00 and 03:00, the data to be written received in the 00:00 and 03:00 are all used as cold data or hot data, the second data frequency table is updated in the period, and cold and hot data distribution is realized by using the data frequency table of the last reset period. And at the time of 3:00-12:00, cold and hot data shunting is realized through the second data frequency table. Correspondingly, when the current moment is 12:00-15:00, the logic block of the data to be written is obtained again, the first data frequency table is updated according to the logic block of the data to be written and the writing frequency, and during the period, cold data and hot data are still shunted by using the second data frequency table, and when the moment is 15:00-24:00, cold and hot data are shunted by using the first data frequency table.
Specifically, in order to avoid the situation that cold data and hot data cannot be accurately distinguished due to the fact that the first data frequency table is used for distinguishing the first data frequency table when the first data frequency table is not perfect, the second data frequency table obtained in the last reset period is still used for distinguishing the cold data and the hot data in the initialization period until the initialization period is finished, at this time, the first data frequency table is already perfect, and at this time, the first data frequency table can be used for distinguishing the cold data and the hot data. More specifically, before the settling period of each reset period, that is, before the first data frequency table of the current reset period is available, the first data frequency table available in the previous reset period is used for splitting the cold data and the hot data.
Referring to fig. 4, in a first aspect, an embodiment of the present application provides a method for identifying cold and hot data, including but not limited to step S410 and step S420.
Step S410, matching the logic block with the first data frequency table to obtain the historical writing frequency of the logic block recorded by the first data frequency table;
step S420, the historical write frequency is added with one to obtain the write frequency of the logic block at the current moment, and the write frequency is updated to the first data frequency table.
According to one embodiment of the application, after obtaining the logic block of the data A to be written, the flash memory device first matches the logic block of the data A to be written with the first data frequency table to obtain the historical writing frequency of the data A recorded by the first data frequency table, adds one to the historical writing frequency of the data A to obtain the writing frequency at the current moment, and updates the writing frequency to the first data frequency table. Such as: the historical writing frequency of the data A recorded by the first data frequency table is 30/period, at the moment, the historical writing frequency of the data A needs to be added by one to obtain the writing frequency 31/period at the current moment, and the data is updated into the first data frequency table.
It is understood that distinguishing whether the data to be written is hot data or cold data by the second data frequency table includes: when the writing frequency corresponding to the data to be written is larger than a preset frequency threshold, the data is hot data; when the writing frequency corresponding to the data to be written is smaller than a preset frequency threshold, the data is cold data; wherein the frequency threshold is determined based on the reset period and the performance of the flash memory device.
According to one embodiment of the present application, in order to better distinguish and shunt the data recorded in the second data frequency table, the present application sets a frequency threshold according to the reset period and the performance of the flash memory device, when the write frequency corresponding to the data in the second data frequency table is greater than the preset frequency threshold, the data is hot data, and when the write frequency corresponding to the data in the second data frequency table is less than the preset frequency threshold, the data is cold data. Specifically, the writing frequency corresponding to the data A transmitted by the host is 30/reset period, the writing frequency corresponding to the data B transmitted by the host is 40/reset period, and the writing frequency corresponding to the data C transmitted by the host is 60/reset period.
Further, when the frequency threshold is 50/period, it indicates that the data with the access frequency below 50/period is cold data and the data with the access frequency above 50/period is hot data, according to the established second data frequency table, it can be known that the data a and the data B are cold data and the data C is hot data.
Referring to fig. 5, in a first aspect, an embodiment of the present application provides a method for identifying cold and hot data, including but not limited to step S510, step S520, step S530, and step S540.
Step S510, obtaining the hot data length of the hot data and the logical address of the hot data;
step S520, determining a first storage space in the hot zone block according to the thermal data length;
step S530, copying the hot data to the first storage space;
in step S540, a mapping relationship between the first storage space and the logical address is established.
It should be noted that, after the cold data and the hot data are distinguished by the first data frequency table or the second data frequency table to obtain the hot data, first, the hot data length of the hot data and the logical address of the hot data need to be obtained, and the first storage space for storing the hot data length is determined in the divided hot block according to the hot data length, so that the hot data is copied into the first storage space, and a mapping relationship between the first storage space and the logical address is established, so that the addressing of the data to be written later is facilitated.
Referring to fig. 6, in a first aspect, an embodiment of the present application provides a method for identifying cold and hot data, including but not limited to step S610, step S620, step S630, and step S640.
Step S610, obtaining the cold data length of the cold data and the logic address of the cold data;
step S620, determining a second storage space in the cold block according to the cold data length;
step S630, copying the cold data to the second storage space;
in step S640, a mapping relationship between the second storage space and the logical address is established.
It should be noted that, after the cold data is obtained by distinguishing the cold data from the cold data through the first data frequency table or the second data frequency table, the cold data length of the cold data and the logical address of the cold data need to be obtained first, and the second storage space for storing the cold data length is determined in the already-divided cold block according to the cold data length, so as to copy the cold data into the second storage space, when the mapping relationship between the second storage space and the logical address is established, the addressing of the data to be written later is facilitated.
In a second aspect, referring to fig. 7, an embodiment of the present application provides a cold and hot data identification system 300, including: the acquisition module 310, the acquisition module 310 is configured to acquire a logic block of data to be written and a first data frequency table; the updating module 320, the updating module 320 is configured to update the write frequency of the logic block in the first data frequency table; the judging module 330, the judging module 330 is configured to judge whether the current time is in an initialization period or a stable period of the current reset period; when the data is in the initialization time period, distinguishing whether the data to be written is hot data or cold data according to the second data frequency table; when the data is in the stable time period, distinguishing whether the data to be written is hot data or cold data through a first data frequency table, wherein a second data frequency table is obtained in the stable time period of the last reset period; the storage module 340, the storage module 340 is configured to store hot data into a hot block and cold data into a cold block, wherein the hot block and the cold block are physical spaces divided in advance; the initialization module 350, the initialization module 350 is configured to initialize the second data frequency table of the current reset period and take the first data frequency table of the current reset period as the second data frequency table of the next reset period, and take the initialized second data frequency table as the first data frequency table of the next reset period when the current reset period is over.
In a third aspect, referring to fig. 8, an embodiment of the present application provides a controller, including:
at least one memory 200;
at least one processor 100;
at least one program;
the programs are stored in the memory 200, and the processor 100 executes at least one program to implement:
the method for identifying cold and hot data according to the embodiment of the first aspect of the application.
The processor 100 and the memory 200 may be connected by a bus or other means.
Memory 200, as a non-transitory readable storage medium, may be used to store non-transitory software instructions as well as non-transitory directives. In addition, memory 200 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. It will be appreciated that the memory 200 may alternatively comprise memory 200 located remotely from the processor 100, such remote memory 200 being connectable to the processor 100 through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The processor 100 implements a method for identifying hot and cold data according to the above-described first embodiment by executing non-transitory software instructions, commands, and signals stored in the memory 200, thereby implementing various functional applications and data processing.
Non-transitory software instructions and instructions required to implement a controller of the above embodiments are stored in the memory 200, and when executed by the processor 100, perform a method of identifying hot and cold data of the first aspect of the embodiment of the present application, for example, performing the method steps S110 to S170 in fig. 1, the method steps S210 to S220 in fig. 2, the method steps S310 to S320 in fig. 3, the method steps S410 to S420 in fig. 4, the method steps S510 to S540 in fig. 5, and the method steps S610 to S640 in fig. 6 described above.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing a computer-executable signal for performing:
a method for identifying hot and cold data according to an embodiment of the first aspect is disclosed.
For example, the above-described method steps S110 to S170 in fig. 1, method steps S210 to S220 in fig. 2, method steps S310 to S320 in fig. 3, method steps S410 to S420 in fig. 4, method steps S510 to S540 in fig. 5, and method steps S610 to S640 in fig. 6 are performed.
The apparatus embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
From the description of the embodiments above, those skilled in the art will appreciate that all or some of the steps, systems, and methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable signals, data structures, instruction modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable signals, data structures, instruction modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and may include any information delivery media.
The embodiments of the present application have been described in detail with reference to the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge of one of ordinary skill in the art without departing from the spirit of the present application.
Claims (10)
1. A method for identifying hot and cold data, applied to a flash memory device, the method comprising:
acquiring a logic block of data to be written and a first data frequency table;
updating the write frequency of the logic block in the first data frequency table;
judging whether the current moment is in an initialization time period or a stable time period of the current reset period;
when the data is in the initialization time period, distinguishing whether the data to be written is hot data or cold data according to the second data frequency table;
when the data is in the stable time period, distinguishing whether the data to be written is hot data or cold data through the first data frequency table, wherein the second data frequency table is obtained in the stable time period of the last reset period;
storing the hot data into a hot block and the cold data into a cold block, wherein the hot block and the cold block are physical spaces divided in advance;
and initializing the second data frequency table in the current reset period when the current reset period is ended, taking the first data frequency table in the current reset period as the second data frequency table in the next reset period, and taking the initialized second data frequency table as the first data frequency table in the next reset period.
2. The method for identifying hot and cold data according to claim 1, wherein the logic block for acquiring the data to be written comprises:
obtaining a plurality of logic blocks with equal proportion sizes according to the total physical space corresponding to the hot block and the cold block;
and obtaining the address of the data to be written, and matching the address of the data to be written with the logic blocks with the equal proportional sizes to obtain the logic block corresponding to the address of the data to be written.
3. The method for identifying cold and hot data according to claim 2, wherein the obtaining a plurality of logic blocks with equal proportional sizes according to the total physical space corresponding to the hot block and the cold block includes:
acquiring the maximum data length allowed to be transmitted by the master control for transmitting the data to be written;
accumulating the physical space of the hot zone block and the physical space of the cold zone block to obtain a total physical space;
and performing equal proportion logic division on the total physical space according to the maximum data length to obtain a plurality of logic blocks with the maximum data length.
4. The method for identifying cold and hot data according to claim 1, further comprising:
when the reset period at the current moment is the first reset period and the current moment is in the initialization time period, the data to be written received in the initialization time period are taken as cold data or hot data;
correspondingly, when the data to be written is hot data or cold data in the initialization period, the distinguishing by the second data frequency table includes:
and when the current moment is in the initialization time period and the current reset period is a reset period except the first reset period, distinguishing whether the data to be written is hot data or cold data according to the second data frequency table.
5. The method of claim 2, wherein updating the write frequency of the logical block in the first data frequency table comprises:
matching the logic block with the first data frequency table to obtain the historical writing frequency of the logic block recorded by the first data frequency table;
and adding one to the historical write frequency to obtain the write frequency of the logic block at the current moment, and updating the write frequency into the first data frequency table.
6. The method for identifying cold and hot data according to claim 1, wherein the distinguishing, by the second data frequency table, whether the data to be written is hot data or cold data, comprises:
when the writing frequency corresponding to the data to be written is larger than a preset frequency threshold value, the data is hot data;
when the writing frequency corresponding to the data to be written is smaller than a preset frequency threshold value, the data is cold data;
wherein the frequency threshold is determined according to the reset period and the performance of the flash memory device.
7. The method of claim 2, wherein storing the thermal data into a thermal block comprises:
acquiring the hot data length of the hot data and the logical address of the hot data;
determining a first storage space in the thermal block according to the thermal data length;
copying the thermal data to the first storage space;
establishing a mapping relation between the first storage space and the logical address;
the storing the cold data into a cold block includes:
acquiring the cold data length of the cold data and the logic address of the cold data;
determining a second storage space in the cold block according to the cold data length;
copying the cold data to the second storage space;
and establishing a mapping relation between the second storage space and the logic address.
8. A system for identifying hot and cold data, comprising:
the acquisition module is used for acquiring a logic block of data to be written and a first data frequency table;
the updating module is used for updating the writing frequency of the logic block in the first data frequency table;
the judging module is used for judging whether the current moment is in the initialization time period or the stable time period of the current reset period; when the data is in the initialization time period, distinguishing whether the data to be written is hot data or cold data according to the second data frequency table; when the data is in the stable time period, distinguishing whether the data to be written is hot data or cold data through the first data frequency table, wherein the second data frequency table is obtained in the stable time period of the last reset period;
the storage module is used for storing the hot data into a hot block and storing the cold data into a cold block, wherein the hot block and the cold block are physical spaces divided in advance;
and the initialization module is used for initializing the second data frequency table in the current reset period when the current reset period is ended, taking the first data frequency table in the current reset period as the second data frequency table in the next reset period, and taking the initialized second data frequency table as the first data frequency table in the next reset period.
9. A controller, comprising:
at least one memory;
at least one processor;
at least one program;
the program is stored in the memory, and the processor executes at least one of the programs to realize the identification method of cold and hot data according to any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer-executable signal for performing the method of identifying cold and hot data according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310588422.4A CN116303119B (en) | 2023-05-19 | 2023-05-19 | Method, system and storage medium for identifying cold and hot data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310588422.4A CN116303119B (en) | 2023-05-19 | 2023-05-19 | Method, system and storage medium for identifying cold and hot data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116303119A CN116303119A (en) | 2023-06-23 |
| CN116303119B true CN116303119B (en) | 2023-08-11 |
Family
ID=86820756
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310588422.4A Active CN116303119B (en) | 2023-05-19 | 2023-05-19 | Method, system and storage medium for identifying cold and hot data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116303119B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116820351B (en) * | 2023-07-21 | 2024-04-09 | 北京得瑞领新科技有限公司 | Cold and hot data calibration method and device, storage medium and electronic equipment |
| CN116627355B (en) * | 2023-07-21 | 2023-10-13 | 北京得瑞领新科技有限公司 | Cold and hot data calibration method and device and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104951403A (en) * | 2015-07-06 | 2015-09-30 | 中国科学技术大学 | Low-overhead and error-free cold and hot data recognition method |
| CN106201906A (en) * | 2016-07-11 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | A kind of cold and hot data separation method for flash memory and system |
| CN109558075A (en) * | 2018-10-29 | 2019-04-02 | 珠海妙存科技有限公司 | A kind of method and device using the cold and hot attribute storing data of data |
| CN109725841A (en) * | 2017-10-30 | 2019-05-07 | 华为技术有限公司 | Method for writing data, device, storage system, storage medium and controller |
| CN114063888A (en) * | 2020-07-31 | 2022-02-18 | 中移(苏州)软件技术有限公司 | Data storage system, data processing method, terminal and storage medium |
| CN114461143A (en) * | 2022-01-11 | 2022-05-10 | 上海英方软件股份有限公司 | Cold and hot data distinguishing method and device for mass data |
| KR20220138269A (en) * | 2021-04-05 | 2022-10-12 | 숙명여자대학교산학협력단 | Data storage device dynamically determinning whether data is hot or cold and operation method of the same |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150356125A1 (en) * | 2014-06-06 | 2015-12-10 | Plexistor Ltd. | Method for data placement based on a file level operation |
| KR20160008365A (en) * | 2014-07-14 | 2016-01-22 | 삼성전자주식회사 | storage medium, memory system and method for managing storage space in memory system |
| US10915475B2 (en) * | 2017-10-12 | 2021-02-09 | Western Digital Technologies, Inc. | Methods and apparatus for variable size logical page management based on hot and cold data |
-
2023
- 2023-05-19 CN CN202310588422.4A patent/CN116303119B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104951403A (en) * | 2015-07-06 | 2015-09-30 | 中国科学技术大学 | Low-overhead and error-free cold and hot data recognition method |
| CN106201906A (en) * | 2016-07-11 | 2016-12-07 | 浪潮(北京)电子信息产业有限公司 | A kind of cold and hot data separation method for flash memory and system |
| CN109725841A (en) * | 2017-10-30 | 2019-05-07 | 华为技术有限公司 | Method for writing data, device, storage system, storage medium and controller |
| CN109558075A (en) * | 2018-10-29 | 2019-04-02 | 珠海妙存科技有限公司 | A kind of method and device using the cold and hot attribute storing data of data |
| CN114063888A (en) * | 2020-07-31 | 2022-02-18 | 中移(苏州)软件技术有限公司 | Data storage system, data processing method, terminal and storage medium |
| KR20220138269A (en) * | 2021-04-05 | 2022-10-12 | 숙명여자대학교산학협력단 | Data storage device dynamically determinning whether data is hot or cold and operation method of the same |
| CN114461143A (en) * | 2022-01-11 | 2022-05-10 | 上海英方软件股份有限公司 | Cold and hot data distinguishing method and device for mass data |
Non-Patent Citations (1)
| Title |
|---|
| 适应冷热数据存储的多编码架构的设计与实证;魏学才 等;计算机应用与软件;第34卷(第2期);第35-41页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116303119A (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116303119B (en) | Method, system and storage medium for identifying cold and hot data | |
| CN108647176B (en) | Data transmission method for Internet of vehicles equipment and Internet of vehicles equipment | |
| CN111666044B (en) | Method for accessing shingled magnetic recording SMR hard disk and server | |
| CN108989432B (en) | User-mode file sending method, user-mode file receiving method and user-mode file receiving and sending device | |
| EP2122975B1 (en) | Wireless media system with embedded media server | |
| CN109669622B (en) | File management method, file management device, electronic equipment and storage medium | |
| CN101520792A (en) | Method and system for automatically mounting and recognizing system file | |
| CN106537353B (en) | A kind of data copy method, device and system | |
| CN107704325B (en) | Method and device for transmitting messages between processes | |
| CN111651127B (en) | Monitoring data storage method and device based on shingled magnetic recording disk | |
| CN111107123A (en) | Method and device for continuous transmission in broken network | |
| WO2022022215A1 (en) | Method for controlling charging of vehicle, and device, program, medium, and vehicle | |
| CN109120384B (en) | HDMI channel switching method and device and smart television | |
| CN103856516A (en) | Data storage and reading method and data storage and reading device | |
| CN106201778B (en) | Information processing method and storage equipment | |
| CN108616598B (en) | Data synchronization method and device and distributed storage system | |
| CN102710481B (en) | Token turnover control method, device and system | |
| CN105681222A (en) | Method and apparatus for data receiving and caching, and communication system | |
| US20160037119A1 (en) | Video recording apparatus, and video recording method when temporary network failure occurs | |
| CN112328549A (en) | Small file storage method, electronic device and storage medium | |
| CN117009087A (en) | Image output method, device, equipment and storage medium | |
| CN112148220B (en) | Method, device, computer storage medium and terminal for realizing data processing | |
| US20210303419A1 (en) | Data processing method and memory controller utilizing the same | |
| CN111949558A (en) | Garbage data recovery method and device and storage equipment | |
| EP2306462A1 (en) | Semiconductor memory device and its early-warning system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |