CN113721854A - Data storage method and device - Google Patents
Data storage method and device Download PDFInfo
- Publication number
- CN113721854A CN113721854A CN202111015623.2A CN202111015623A CN113721854A CN 113721854 A CN113721854 A CN 113721854A CN 202111015623 A CN202111015623 A CN 202111015623A CN 113721854 A CN113721854 A CN 113721854A
- Authority
- CN
- China
- Prior art keywords
- data
- resource pool
- cold
- storage
- access
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 238000013500 data storage Methods 0.000 title claims abstract description 28
- 238000013507 mapping Methods 0.000 claims abstract description 56
- 239000011159 matrix material Substances 0.000 claims abstract description 49
- 238000012544 monitoring process Methods 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 2
- BASFCYQUMIYNBI-UHFFFAOYSA-N platinum Chemical compound [Pt] BASFCYQUMIYNBI-UHFFFAOYSA-N 0.000 description 24
- BQCADISMDOOEFD-UHFFFAOYSA-N Silver Chemical compound [Ag] BQCADISMDOOEFD-UHFFFAOYSA-N 0.000 description 12
- 229910052697 platinum Inorganic materials 0.000 description 12
- 229910052709 silver Inorganic materials 0.000 description 12
- 239000004332 silver Substances 0.000 description 12
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 10
- 229910052737 gold Inorganic materials 0.000 description 10
- 239000010931 gold Substances 0.000 description 10
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 4
- 229910052802 copper Inorganic materials 0.000 description 4
- 239000010949 copper Substances 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0622—Securing storage systems in relation to access
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Abstract
The invention provides a data storage method and a data storage device, wherein the method comprises the following steps: acquiring data information of each datum in a storage system, wherein the storage system is provided with a resource pool with a plurality of storage levels; generating a statistical matrix corresponding to each resource pool based on each data information; judging whether cold data exists in the storage system or not based on the data information corresponding to each data; when the cold data exists, determining a first resource pool for currently storing the cold data and a storage grade of the first resource pool; setting a mobile storage position of the cold data based on the data information of the cold data and the storage grade of the first resource pool; determining a second resource pool corresponding to the mobile storage position of the cold data, and constructing a mapping file corresponding to the cold data based on the data information of the cold data and a statistical matrix corresponding to the second resource pool; and saving the mapping file, and movably storing the cold data to a second resource pool. By applying the method provided by the invention, the storage position of the data can be reasonably adjusted, and a large amount of cold data is prevented from being stored in a high-level resource pool.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data storage method and apparatus.
Background
In the internet era, the core business data and other transaction data volume of banks are in geometric figure and mass growth, and the requirements on data storage efficiency and security are higher and higher. In order to ensure the security of the stored data, many enterprises divide the attributes of the storage resource pool so as to store the important data in a grading manner. The data is stored in order to ensure that the application can access the data again, but the security and performance requirements of important data on the storage resource pool are also high, if a large amount of cold data exists in the storage resource pool, the data operation and maintenance cost of an enterprise can be increased, and meanwhile, the existing resources can be greatly wasted.
Disclosure of Invention
In view of this, the present invention provides a data storage method, by which a storage location of data can be reasonably adjusted, and a large amount of cold data is prevented from being stored in a high-level resource pool.
The invention also provides a data storage device for ensuring the realization and the application of the method in practice.
A method of data storage, comprising:
acquiring data information of each datum in a storage system, wherein the storage system is provided with a plurality of resource pools with storage levels, and at least one datum is stored in each resource pool;
generating a statistical matrix corresponding to each resource pool based on the data information;
judging whether cold data exists in the storage system or not based on data information corresponding to each piece of data;
when cold data exists in the storage system, determining that a resource pool for storing the cold data currently is a first resource pool and the storage grade of the first resource pool;
setting a mobile storage position of the cold data based on the data information of the cold data and the storage level of the first resource pool;
determining a resource pool corresponding to the mobile storage position of the cold data as a second resource pool, and constructing a mapping file corresponding to the cold data based on data information of the cold data and a statistical matrix corresponding to the second resource pool, wherein the mapping file is used for positioning the mobile storage position of the cold data;
and saving the mapping file, and movably storing the cold data to the second resource pool.
Optionally, in the method, generating a statistical matrix corresponding to each resource pool based on each piece of data information includes:
acquiring an access parameter of each data stored in each data information in a preset monitoring period;
determining the data type and data attribute of each data stored in each resource pool;
counting a first access frequency corresponding to each data type, a second access frequency corresponding to each data attribute and an access frequency in each resource pool based on an access parameter corresponding to each data in each resource pool;
and constructing a statistical matrix corresponding to each resource pool based on the first access times corresponding to each data type, the second access times corresponding to each data attribute and the access frequency in each resource pool.
Optionally, the method for determining whether cold data exists in the storage system based on the data information corresponding to each piece of data includes:
calculating the access frequency of each data based on the access parameter corresponding to each data;
judging whether the access frequency of each piece of data is lower than a frequency threshold corresponding to a resource pool to which the data belongs;
if the access frequency of any data is lower than the access frequency corresponding to the resource pool to which the data belongs, determining that the data is cold data;
and if the access frequency of any data is not lower than the access frequency corresponding to the resource pool to which the data belongs, determining that the data is not cold data.
The method, optionally, may set the storage level of each resource pool, including:
determining a current support service of a storage system and a target data type related to the service;
determining a proportion of each of the resource pools in the storage system that contains data of the target data type;
setting a weight value corresponding to each resource pool according to the proportion of the data of the target data type contained in each resource pool in the storage system;
and setting the storage grade of each resource pool in the storage system to which the resource pool belongs according to the weight value of each resource pool in the storage system to which the resource pool belongs.
The above method, optionally, further includes:
when a data access request corresponding to the cold data sent by a server is received, acquiring a mapping file corresponding to the cold data;
determining a target resource pool currently storing the cold data based on the mapping file;
and executing the access operation of the cold data corresponding to the data access request in the target resource pool, and returning an operation result corresponding to the access operation to the server.
A data storage device comprising:
the data information acquisition unit is used for acquiring data information of each data in a storage system, the storage system is provided with a plurality of resource pools with storage levels, and at least one data is stored in each resource pool;
the generating unit is used for generating a statistical matrix corresponding to each resource pool based on the data information;
the judging unit is used for judging whether cold data exists in the storage system or not based on the data information corresponding to each piece of data;
the determining unit is used for determining that a resource pool for storing cold data currently is a first resource pool and the storage grade of the first resource pool when the cold data exists in the storage system;
the setting unit is used for setting a mobile storage position of the cold data based on the data information of the cold data and the storage level of the first resource pool;
the construction unit is used for determining that the resource pool corresponding to the mobile storage position of the cold data is a second resource pool, and constructing a mapping file corresponding to the cold data based on the data information of the cold data and the statistical matrix corresponding to the second resource pool, wherein the mapping file is used for positioning the mobile storage position of the cold data;
and the storage unit is used for storing the mapping file and movably storing the cold data to the second resource pool.
The above apparatus, optionally, the generating unit includes:
the acquisition subunit is used for acquiring an access parameter of each piece of data stored in each piece of data information in a preset monitoring period;
the first determining subunit is used for determining the data type and the data attribute of each piece of data stored in each resource pool;
the counting subunit is configured to count, based on an access parameter corresponding to each piece of data in each resource pool, a first access frequency corresponding to each data type, a second access frequency corresponding to each data attribute, and an access frequency in each resource pool;
and the constructing subunit is used for constructing a statistical matrix corresponding to each resource pool based on the first access times corresponding to each data type, the second access times corresponding to each data attribute and the access frequency in each resource pool.
The above apparatus, optionally, the determining unit includes:
the calculating subunit is used for calculating the access frequency of each piece of data based on the access parameter corresponding to each piece of data;
the judging subunit is used for judging whether the access frequency of each piece of data is lower than a frequency threshold corresponding to the resource pool to which the data belongs;
the second determining subunit is used for determining that the data is cold data if the access frequency of any data is lower than the access frequency corresponding to the resource pool to which the data belongs;
and the third determining subunit is used for determining that the data is not cold data if the access frequency of any data is not lower than the access frequency corresponding to the resource pool to which the data belongs.
The above apparatus, optionally, further comprises:
the system comprises a grade division unit, a storage unit and a service processing unit, wherein the grade division unit is used for determining the current service supporting service of a storage system and the target data type related to the service; determining a proportion of each of the resource pools in the storage system that contains data of the target data type; setting a weight value corresponding to each resource pool according to the proportion of the data of the target data type contained in each resource pool in the storage system; and setting the storage grade of each resource pool in the storage system to which the resource pool belongs according to the weight value of each resource pool in the storage system to which the resource pool belongs.
The above apparatus, optionally, further comprises:
the access unit is used for acquiring a mapping file corresponding to the cold data when receiving a data access request corresponding to the cold data sent by a server; determining a target resource pool currently storing the cold data based on the mapping file; and executing the access operation of the cold data corresponding to the data access request in the target resource pool, and returning an operation result corresponding to the access operation to the server.
A storage medium, the storage medium comprising stored instructions, wherein when the instructions are executed, the apparatus on which the storage medium is located is controlled to execute the above data storage method.
An electronic device comprising a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by the one or more processors to perform the data storage method described above.
Compared with the prior art, the invention has the following advantages:
the invention provides a data storage method, which comprises the following steps: acquiring data information of each datum in a storage system, wherein the storage system is provided with a plurality of resource pools with storage levels, and at least one datum is stored in each resource pool; generating a statistical matrix corresponding to each resource pool based on the data information; judging whether cold data exists in the storage system or not based on data information corresponding to each piece of data; when cold data exists in the storage system, determining that a resource pool for storing the cold data currently is a first resource pool and the storage grade of the first resource pool; setting a mobile storage position of the cold data based on the data information of the cold data and the storage level of the first resource pool; determining a resource pool corresponding to the mobile storage position of the cold data as a second resource pool, and constructing a mapping file corresponding to the cold data based on data information of the cold data and a statistical matrix corresponding to the second resource pool, wherein the mapping file is used for positioning the storage position of the cold data; and saving the mapping file, and movably storing the cold data to the second resource pool. By applying the method provided by the invention, the storage position of the data can be reasonably adjusted, and a large amount of cold data is prevented from being stored in a high-level resource pool.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a method of storing data according to an embodiment of the present invention;
FIG. 2 is a flowchart of another method of a data storage method according to an embodiment of the present invention;
FIG. 3 is a flowchart of another method of a data storage method according to an embodiment of the present invention;
FIG. 4 is a block diagram of a data storage device according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In this application, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions, and the terms "comprises", "comprising", or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The invention is operational with numerous general purpose or special purpose computing device environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multi-processor apparatus, distributed computing environments that include any of the above devices or equipment, and the like.
An embodiment of the present invention provides a data storage method, where the method may be applied to multiple system platforms, an execution subject of the method may be a computer terminal or a processor of various mobile devices, and a flowchart of the method is shown in fig. 1, and specifically includes:
s101: data information of each data in the storage system is acquired.
The storage system is provided with a plurality of resource pools with storage levels, and at least one datum is stored in each resource pool.
In the invention, the Storage system is an SAN (Storage area Network) system or an NAS (Network Attached Storage Network) system.
For the SAN system, a production resource pool, a multi-active resource pool and a disaster recovery resource pool are arranged, different types of storage resource pools are divided according to application levels, the storage resource pools can be divided into three levels, and at least one data is stored in each level of resource pool.
For the NAS system, a production resource pool, a multi-active resource pool, a disaster recovery resource pool and an AP storage resource pool are arranged in the NAS system, different types of storage resource pools are divided according to application levels, the storage resource pools can be divided into four levels, and at least one piece of data is stored in each level of resource pool.
Specifically, the data information of the data includes a data type, an access time and an access frequency of each access, and the like.
S102: and generating a statistical matrix corresponding to each resource pool based on the data information.
It should be noted that the statistical matrix of each resource pool is used to indicate the situation that each data in the resource pool is accessed. And the access habit of the accessed data in a preset time period is counted by applying a preset counting matrix module, the data attribute, the data access frequency and the like of the accessed data are extracted, and a counting matrix of the resource pool is established by utilizing an analysis mode of big data.
The statistical matrix corresponding to the resource pool further includes resource pool information of the resource pool, such as a serial number of the resource pool, a name of the resource pool, and a location in the storage system.
S103: and judging whether cold data exists in the storage system or not based on the data information corresponding to each piece of data.
In the invention, the processor comprises a cold data judging module, and whether cold data exists in each resource pool of the storage system is judged through the cold data judging module so as to manage the data subsequently.
Specifically, when there is cold data in the storage system, S104 is performed, and otherwise S108 is performed.
S104: and determining the current resource pool for storing cold data as a first resource pool and the storage grade of the first resource pool.
In the present invention, cold data refers to data in a resource pool that has not been accessed for a long period of time or has a low access rate.
S105: and setting a mobile storage position of the cold data based on the data information of the cold data and the preset storage level of the first resource pool.
In the invention, if the storage level of the resource pool for storing the cold data is not the resource pool with the minimum storage level, the access frequency of the cold data is determined according to the data information of the cold data, and the mobile storage position of the cold data is determined according to the access frequency.
The mobile storage location refers to a resource pool location for replacing a resource pool for storing cold data.
S106: and determining a resource pool corresponding to the mobile storage position of the cold data as a second resource pool, and constructing a mapping file corresponding to the cold data based on the data information of the cold data and a statistical matrix corresponding to the second resource pool.
Wherein the mapping file is used for locating a mobile storage location of the cold data.
In the present invention, before determining whether cold data is present. And firstly, constructing a statistical matrix corresponding to each resource pool, and reflecting the accessed condition of each data in the resource pool through the statistical matrix. In addition, the statistical matrix mainly provides reference for establishing the mapping file. The reference providing specifically refers to extracting the access frequency of a user to certain data within a period of time by establishing a data access model, and providing auxiliary information for a statistical matrix by using the model, so that the information with the highest user access frequency can be conveniently extracted from a plurality of data information; and establishing information required by the mapping file according to the information extracted from the statistical matrix so as to determine the position of the transferred file, namely, when a user accesses the information, the data can be accessed through the mapping file.
It should be noted that the second resource pool may be the first resource pool in S104, and when the second resource pool is the first resource pool, it is not necessary to transfer and store the cold data.
S107: and saving the mapping file, and movably storing the cold data to the second resource pool.
In the present invention, the map file is saved so that the data can be accessed through the map file after the cold data is transferred to the storage location.
S108: when no cold data exists in the storage system, no operation is performed on the respective data in the storage system.
In the data storage method provided by the embodiment of the invention, the data information of each data in the storage system is acquired, and the statistical matrix corresponding to each resource pool is generated based on the data information of each data. The storage system is provided with a plurality of resource pools with storage levels, each resource pool is used for storing data, and each resource pool stores at least one piece of data. And judging whether cold data exists in the storage system or not based on the data information corresponding to each piece of data, if so, determining the resource pool which currently stores the cold data as a first resource pool, determining the storage grade of the first resource pool, and setting the mobile storage position of the cold data based on the data information of the cold data and the storage grade of the first resource pool. And determining a second resource pool corresponding to the mobile storage position, constructing a mapping file corresponding to the cold data based on the data information of the cold data and the statistical matrix corresponding to the second resource pool, storing the mapping file and moving the cold data to the second resource pool.
Based on the method provided by the above embodiment, when the storage system is an SAN system, the process of storing cold data in the SAN system has the following specific implementation:
the production resource pool, the multi-active resource pool and the disaster recovery resource pool in the SAN system are divided into different types of resource pools according to application levels, and the different types of resource pools can be divided into three storage levels, namely an SAN platinum-level resource pool, an SAN gold-level resource pool and an SAN silver-level resource pool. The storage levels of the three resource pools are as follows from large to small in sequence: SAN platinum-level resource pool, SAN gold-level resource pool and SAN silver-level resource pool. Respectively creating statistical matrixes corresponding to the SAN platinum-level resource pool, the SAN gold-level resource pool and the SAN silver-level resource pool, and sequentially detecting whether cold data exists in the SAN platinum-level resource pool, the SAN gold-level resource pool and the SAN silver-level resource pool. And if the SAN platinum-level resource pool has cold data and the mobile storage position of the cold data is the position of the SAN gold-level resource pool, the SAN platinum-level resource pool is a first resource pool, and the SAN gold-level resource pool is a second resource pool. And generating a mapping file corresponding to the cold data according to the data information of the cold data and the statistical matrix of the SAN golden-level resource pool, storing the mapping file and transferring and storing the cold data to the SAN golden-level resource pool.
Based on the method provided by the above embodiment, when the storage system is an NAS system, the process of storing cold data in the NAS system has the following specific implementation manners:
the method comprises the steps of dividing a production resource pool, a multi-active resource pool, a disaster recovery resource pool and an AP storage resource pool in the NAS storage system into three types according to the application level, wherein the three types are respectively an NAS platinum-level resource pool, an NAS gold-level resource pool, an NAS silver-level resource pool and an NAS copper-level resource pool. The storage levels of the four resource pools are as follows from large to small in sequence: the NAS platinum-level resource pool, the NAS gold-level resource pool, the NAS silver-level resource pool and the NAS copper-level resource pool. Respectively creating statistical matrixes corresponding to the NAS platinum-level resource pool, the NAS gold-level resource pool, the NAS silver-level resource pool and the NAS copper-level resource pool, and sequentially detecting whether cold data exists in the NAS platinum-level resource pool, the NAS gold-level resource pool, the NAS silver-level resource pool and the NAS copper-level resource pool or not. And if the NAS platinum-level resource pool has cold data and the mobile storage position of the cold data is the position of the NAS silver-level resource pool, the NAS platinum-level resource pool is a first resource pool, and the NAS silver-level resource pool is a second resource pool. And generating a mapping file corresponding to the cold data according to the data information of the cold data and the statistical matrix of the NAS silver-level resource pool, storing the mapping file and transferring and storing the cold data to the NAS silver-level resource pool.
By applying the method provided by the embodiment of the invention, the storage position of the data can be reasonably adjusted, and a large amount of cold data is prevented from being stored in a high-level resource pool.
As shown in fig. 2, based on the method provided in the foregoing embodiment, the generating a statistical matrix corresponding to each resource pool based on each piece of data information includes:
s201: and acquiring the access parameter of each data stored in each data information in a preset monitoring period.
In the present invention, the access parameter includes information such as access time, access times, and access device.
It should be noted that, in the present invention, for each data in the storage system, the data in the storage system is scanned in real time according to the set monitoring period, and a data storage method corresponding to each data in the storage system is executed.
S202: and determining the data type and the data attribute of each data stored in each resource pool.
In the invention, each resource pool contains data of a plurality of data types, the attributes of each data type are different, and the application mode of the data can be determined according to the data type and the data attributes of the data.
S203: and counting a first access frequency corresponding to each data type, a second access frequency corresponding to each data attribute and an access frequency in each resource pool based on the access parameter corresponding to each data in each resource pool.
In the invention, the access condition of the application to the storage resource pool is counted by counting the first access times of each data type, the second access times corresponding to each data attribute and the access frequency.
S204: and constructing a statistical matrix corresponding to each resource pool based on the first access times corresponding to each data type, the second access times corresponding to each data attribute and the access frequency in each resource pool.
In the invention, a statistical matrix corresponding to the resource pool is established according to the access condition of each data in the resource pool. The statistical matrix comprises the resource pool number, the position address, the access times of each data type, the access times of each data attribute and the access frequency of each data attribute of the corresponding resource pool.
Further, the process of generating the mapping file corresponding to the cold data based on the data information of the cold data and the statistical matrix corresponding to the second resource pool includes:
counting the access frequency of the cold data in the current monitoring period based on the data information of the cold data;
determining a resource pool code, a first access frequency of each data type, a second access frequency corresponding to each data attribute and an arrangement format of the access frequency in a statistical matrix corresponding to the second resource pool;
and generating a mapping file corresponding to the cold data according to the arrangement format, the data information of the cold data, the access frequency, and the resource pool number and position of the second resource pool.
In the method provided by the embodiment of the invention, in the process of generating the statistical matrix corresponding to each resource pool, the access parameter of each data stored in each data information in a preset monitoring period is obtained, and the data type and the data attribute of all data stored in the resource pool are determined. And counting the access times of each data type and the access times of each data attribute according to the access parameters, and calculating the access frequency of each data attribute. That is, the number of times of accessing each data type in the data of each data type stored in the resource pool and the number of times and frequency of accessing each data attribute in the data of each data attribute stored in the resource pool are counted. And constructing a statistical matrix corresponding to each resource pool based on the first access times of each data type corresponding to each resource pool, the second access times corresponding to each data attribute and the access frequency.
By applying the method provided by the embodiment of the invention, the access condition of the data in each resource pool can be determined by establishing the statistical matrix, and a corresponding template is provided for the cold data creation mapping file.
As shown in fig. 3, in the method according to the embodiment of the present invention, the determining whether cold data exists in the storage system based on the data information corresponding to each piece of data includes:
s301: and calculating the access frequency of each data based on the access parameter corresponding to each data.
In the invention, for each resource pool of the storage level, the higher the access frequency of the data in the resource pool is, the more times of representing the data needs. When data with high access frequency is continuously stored in the same resource pool, the processor can know the storage condition of the data according to the historical access condition without establishing a mapping file of the data. On the contrary, when data with low access frequency is continuously stored in the same resource pool, the storage space of the resource pool is occupied by the data with low access frequency for a long time, and when other data needs to be stored in the resource pool, the space is easily insufficient.
S302: and judging whether the access frequency of each piece of data is lower than a frequency threshold corresponding to the resource pool to which the data belongs.
If the access frequency of any data is lower than the access frequency corresponding to the resource pool to which the data belongs, executing S303; if the access frequency of any data is not lower than the access frequency corresponding to the resource pool to which the data belongs, S304 is executed.
S303: determining the data to be cold data.
In the present invention, cold data is data that has not been accessed for a long period of time.
S304: determining that the data is not cold data.
In the method provided by the embodiment of the invention, a time period of one year, one month or one week and the like can be used as a monitoring period for monitoring the storage system. And monitoring all data in the storage system by using preset monitoring equipment, and checking the access frequency of the data according to the data information of the data. If the access frequency of the data is low, for example, the access frequency is lower than 1%, the data is characterized as cold data, and conversely, the data is not characterized as cold data.
In addition, the change of the storage location of the cold data is usually performed by transferring and storing the cold data from a resource pool having a higher storage level to a resource pool having a lower storage level.
By applying the method provided by the embodiment of the invention, whether the data is cold data or not is determined through the access frequency of the data, so that the cold data in the resource pool is prevented from occupying the storage space for a long time.
In the method provided by the embodiment of the present invention, the process of setting the storage level of each resource pool includes:
determining a current support service of a storage system and a target data type related to the service;
determining a proportion of each of the resource pools in the storage system that contains data of the target data type;
setting a weight value corresponding to each resource pool according to the proportion of the data of the target data type contained in each resource pool in the storage system;
and setting the storage grade of each resource pool in the storage system to which the resource pool belongs according to the weight value of each resource pool in the storage system to which the resource pool belongs.
In the data storage method provided by the embodiment of the invention, the grade of each resource pool in the storage system can be changed according to the set requirement. The storage system can set a corresponding supported service business according to the service requirement, for example, if the current storage system supports the account checking business, the data type of the data related to the account checking business is account checking data; and if the service business supported by the storage system is the withdrawal business after one year, the data type of the data related to the withdrawal business is withdrawal transaction data. For a storage system, all data contained in the storage system is not data corresponding to a target data type related to a service supported by the storage system, and therefore, in the step of dividing the storage levels of the resource pools, the storage levels of the resource pools are divided according to the data types of the data stored in each resource pool.
Specifically, the proportion of data containing the target data type in each resource pool is determined, and the weight value corresponding to each resource pool is determined according to the proportion of the data of the target data type. The larger the weight value of the resource pool is, the higher the corresponding storage level is.
It should be noted that, the more data of the target data type is contained in the resource pool, the greater the access probability of the storage system to each data in the resource pool is in the application process, and when newly generated data is stored, the higher the storage level is, the storage level is stored preferentially in the resource pool. When certain data needs to be accessed, whether each resource pool stores the data to be accessed or not can be inquired according to the storage grade, if the data type of the data to be accessed is the target data type, the probability of the data to be accessed in the resource pool with the higher storage grade is higher, and the relevant response of the data to be accessed is faster.
The method provided by the embodiment of the invention further comprises the following steps:
when a data access request corresponding to the cold data sent by a server is received, acquiring a mapping file corresponding to the cold data;
determining a target resource pool currently storing the cold data based on the mapping file;
and executing the access operation of the cold data corresponding to the data access request in the target resource pool, and returning an operation result corresponding to the access operation to the server.
It can be understood that when a data access request is received, data information such as a data number of data to be accessed included in the data access request is obtained, and whether the data to be accessed is cold data can be determined through the data information such as the data number. If the data is cold data, a mapping file corresponding to the cold data can be queried according to the data number, a target resource pool storing the cold data at present is queried through the mapping file, access operations of the cold data, such as operations of copying, transmitting, packaging and the like, are executed through the target resource pool, and an operation result is returned.
By applying the method provided by the embodiment of the invention, the resource pool storing the cold data at present can be inquired through the mapping file, so that the condition that the cold data cannot be inquired after the position of the cold data is adjusted is avoided.
The specific implementation procedures and derivatives thereof of the above embodiments are within the scope of the present invention.
Corresponding to the method described in fig. 1, an embodiment of the present invention further provides a data storage apparatus, which is used for implementing the method in fig. 1 specifically, the data storage apparatus provided in the embodiment of the present invention may be applied to a computer terminal or various mobile devices, and a schematic structural diagram of the data storage apparatus is shown in fig. 4, and specifically includes:
an obtaining unit 401, configured to obtain data information of each data in a storage system, where the storage system is provided with resource pools of multiple storage levels, and each resource pool stores at least one data;
a generating unit 402, configured to generate a statistical matrix corresponding to each resource pool based on each piece of data information;
a determining unit 403, configured to determine whether cold data exists in the storage system based on data information corresponding to each piece of data;
a determining unit 404, configured to determine, when cold data exists in the storage system, that a resource pool currently storing the cold data is a first resource pool and a storage level of the first resource pool;
a setting unit 405, configured to set a mobile storage location of the cold data based on the data information of the cold data and the storage level of the first resource pool;
a constructing unit 406, configured to determine that a resource pool corresponding to the mobile storage location of the cold data is a second resource pool, and construct a mapping file corresponding to the cold data based on data information of the cold data and a statistical matrix corresponding to the second resource pool, where the mapping file is used to locate the mobile storage location of the cold data;
the storage unit 407 is configured to save the mapping file, and store the cold data to the second resource pool in a mobile manner.
In the data storage device provided by the embodiment of the invention, the data information of each data in the storage system is acquired, and the statistical matrix corresponding to each resource pool is generated based on the data information of each data. The storage system is provided with a plurality of resource pools with storage levels, each resource pool is used for storing data, and each resource pool stores at least one piece of data. And judging whether cold data exists in the storage system or not based on the data information corresponding to each piece of data, if so, determining the resource pool which currently stores the cold data as a first resource pool, determining the storage grade of the first resource pool, and setting the mobile storage position of the cold data based on the data information of the cold data and the storage grade of the first resource pool. And determining a second resource pool corresponding to the mobile storage position, constructing a mapping file corresponding to the cold data based on the data information of the cold data and the statistical matrix corresponding to the second resource pool, storing the mapping file and moving the cold data to the second resource pool.
By applying the device provided by the embodiment of the invention, the storage position of the data can be reasonably adjusted, and a large amount of cold data is prevented from being stored in a high-level resource pool.
In the apparatus provided in the embodiment of the present invention, the generating unit includes:
the acquisition subunit is used for acquiring an access parameter of each piece of data stored in each piece of data information in a preset monitoring period;
the first determining subunit is used for determining the data type and the data attribute of each piece of data stored in each resource pool;
the counting subunit is configured to count, based on an access parameter corresponding to each piece of data in each resource pool, a first access frequency corresponding to each data type, a second access frequency corresponding to each data attribute, and an access frequency in each resource pool;
and the constructing subunit is used for constructing a statistical matrix corresponding to each resource pool based on the first access times corresponding to each data type, the second access times corresponding to each data attribute and the access frequency in each resource pool.
In the apparatus provided in the embodiment of the present invention, the determining unit includes:
the calculating subunit is used for calculating the access frequency of each piece of data based on the access parameter corresponding to each piece of data;
the judging subunit is used for judging whether the access frequency of each piece of data is lower than a frequency threshold corresponding to the resource pool to which the data belongs;
the second determining subunit is used for determining that the data is cold data if the access frequency of any data is lower than the access frequency corresponding to the resource pool to which the data belongs;
and the third determining subunit is used for determining that the data is not cold data if the access frequency of any data is not lower than the access frequency corresponding to the resource pool to which the data belongs.
The device provided by the embodiment of the invention further comprises:
the system comprises a grade division unit, a storage unit and a service processing unit, wherein the grade division unit is used for determining the current service supporting service of a storage system and the target data type related to the service; determining a proportion of each of the resource pools in the storage system that contains data of the target data type; setting a weight value corresponding to each resource pool according to the proportion of the data of the target data type contained in each resource pool in the storage system; and setting the storage grade of each resource pool in the storage system to which the resource pool belongs according to the weight value of each resource pool in the storage system to which the resource pool belongs.
The device provided by the embodiment of the invention further comprises:
the access unit is used for acquiring a mapping file corresponding to the cold data when receiving a data access request corresponding to the cold data sent by a server; determining a target resource pool currently storing the cold data based on the mapping file; and executing the access operation of the cold data corresponding to the data access request in the target resource pool, and returning an operation result corresponding to the access operation to the server.
The specific working processes of each unit and sub-unit in the data storage device disclosed in the above embodiment of the present invention can be referred to the corresponding contents in the data storage method disclosed in the above embodiment of the present invention, and are not described herein again.
The embodiment of the invention also provides a storage medium, which comprises a stored instruction, wherein when the instruction runs, the device where the storage medium is located is controlled to execute the data storage method.
An electronic device is provided in an embodiment of the present invention, and the structural diagram of the electronic device is shown in fig. 5, which specifically includes a memory 501 and one or more instructions 502, where the one or more instructions 502 are stored in the memory 501, and are configured to be executed by one or more processors 503 to perform the following operations according to the one or more instructions 502:
acquiring data information of each datum in a storage system, wherein the storage system is provided with a plurality of resource pools with storage levels, and at least one datum is stored in each resource pool;
generating a statistical matrix corresponding to each resource pool based on the data information;
judging whether cold data exists in the storage system or not based on data information corresponding to each piece of data;
when cold data exists in the storage system, determining that a resource pool for storing the cold data currently is a first resource pool and the storage grade of the first resource pool;
setting a mobile storage position of the cold data based on the data information of the cold data and the storage level of the first resource pool;
determining a resource pool corresponding to the mobile storage position of the cold data as a second resource pool, and constructing a mapping file corresponding to the cold data based on data information of the cold data and a statistical matrix corresponding to the second resource pool, wherein the mapping file is used for positioning the mobile storage position of the cold data;
and saving the mapping file, and movably storing the cold data to the second resource pool.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the system or system embodiments are substantially similar to the method embodiments and therefore are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described system and system embodiments are only illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both.
To clearly illustrate this interchangeability of hardware and software, various illustrative components and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A method of storing data, comprising:
acquiring data information of each datum in a storage system, wherein the storage system is provided with a plurality of resource pools with storage levels, and at least one datum is stored in each resource pool;
generating a statistical matrix corresponding to each resource pool based on the data information;
judging whether cold data exists in the storage system or not based on data information corresponding to each piece of data;
when cold data exists in the storage system, determining that a resource pool for storing the cold data currently is a first resource pool and the storage grade of the first resource pool;
setting a mobile storage position of the cold data based on the data information of the cold data and the storage level of the first resource pool;
determining a resource pool corresponding to the mobile storage position of the cold data as a second resource pool, and constructing a mapping file corresponding to the cold data based on data information of the cold data and a statistical matrix corresponding to the second resource pool, wherein the mapping file is used for positioning the mobile storage position of the cold data;
and saving the mapping file, and movably storing the cold data to the second resource pool.
2. The method of claim 1, wherein the generating a statistical matrix corresponding to each resource pool based on the respective data information comprises:
acquiring an access parameter of each data stored in each data information in a preset monitoring period;
determining the data type and data attribute of each data stored in each resource pool;
counting a first access frequency corresponding to each data type, a second access frequency corresponding to each data attribute and an access frequency in each resource pool based on an access parameter corresponding to each data in each resource pool;
and constructing a statistical matrix corresponding to each resource pool based on the first access times corresponding to each data type, the second access times corresponding to each data attribute and the access frequency in each resource pool.
3. The method of claim 2, wherein the determining whether cold data exists in the storage system based on the data information corresponding to each piece of data comprises:
calculating the access frequency of each data based on the access parameter corresponding to each data;
judging whether the access frequency of each piece of data is lower than a frequency threshold corresponding to a resource pool to which the data belongs;
if the access frequency of any data is lower than the access frequency corresponding to the resource pool to which the data belongs, determining that the data is cold data;
and if the access frequency of any data is not lower than the access frequency corresponding to the resource pool to which the data belongs, determining that the data is not cold data.
4. The method of claim 1, wherein the process of setting the storage rank of each resource pool comprises:
determining a current support service of a storage system and a target data type related to the service;
determining a proportion of each of the resource pools in the storage system that contains data of the target data type;
setting a weight value corresponding to each resource pool according to the proportion of the data of the target data type contained in each resource pool in the storage system;
and setting the storage grade of each resource pool in the storage system to which the resource pool belongs according to the weight value of each resource pool in the storage system to which the resource pool belongs.
5. The method of claim 1, further comprising:
when a data access request corresponding to the cold data sent by a server is received, acquiring a mapping file corresponding to the cold data;
determining a target resource pool currently storing the cold data based on the mapping file;
and executing the access operation of the cold data corresponding to the data access request in the target resource pool, and returning an operation result corresponding to the access operation to the server.
6. A data storage device, comprising:
the data information acquisition unit is used for acquiring data information of each data in a storage system, the storage system is provided with a plurality of resource pools with storage levels, and at least one data is stored in each resource pool;
the generating unit is used for generating a statistical matrix corresponding to each resource pool based on the data information;
the judging unit is used for judging whether cold data exists in the storage system or not based on the data information corresponding to each piece of data;
the determining unit is used for determining that a resource pool for storing cold data currently is a first resource pool and the storage grade of the first resource pool when the cold data exists in the storage system;
the setting unit is used for setting a mobile storage position of the cold data based on the data information of the cold data and the storage level of the first resource pool;
the construction unit is used for determining that the resource pool corresponding to the mobile storage position of the cold data is a second resource pool, and constructing a mapping file corresponding to the cold data based on the data information of the cold data and the statistical matrix corresponding to the second resource pool, wherein the mapping file is used for positioning the mobile storage position of the cold data;
and the storage unit is used for storing the mapping file and movably storing the cold data to the second resource pool.
7. The apparatus of claim 6, wherein the generating unit comprises:
the acquisition subunit is used for acquiring an access parameter of each piece of data stored in each piece of data information in a preset monitoring period;
the first determining subunit is used for determining the data type and the data attribute of each piece of data stored in each resource pool;
the counting subunit is configured to count, based on an access parameter corresponding to each piece of data in each resource pool, a first access frequency corresponding to each data type, a second access frequency corresponding to each data attribute, and an access frequency in each resource pool;
and the constructing subunit is used for constructing a statistical matrix corresponding to each resource pool based on the first access times corresponding to each data type, the second access times corresponding to each data attribute and the access frequency in each resource pool.
8. The apparatus according to claim 7, wherein the determining unit comprises:
the calculating subunit is used for calculating the access frequency of each piece of data based on the access parameter corresponding to each piece of data;
the judging subunit is used for judging whether the access frequency of each piece of data is lower than a frequency threshold corresponding to the resource pool to which the data belongs;
the second determining subunit is used for determining that the data is cold data if the access frequency of any data is lower than the access frequency corresponding to the resource pool to which the data belongs;
and the third determining subunit is used for determining that the data is not cold data if the access frequency of any data is not lower than the access frequency corresponding to the resource pool to which the data belongs.
9. The apparatus of claim 6, further comprising:
the system comprises a grade division unit, a storage unit and a service processing unit, wherein the grade division unit is used for determining the current service supporting service of a storage system and the target data type related to the service; determining a proportion of each of the resource pools in the storage system that contains data of the target data type; setting a weight value corresponding to each resource pool according to the proportion of the data of the target data type contained in each resource pool in the storage system; and setting the storage grade of each resource pool in the storage system to which the resource pool belongs according to the weight value of each resource pool in the storage system to which the resource pool belongs.
10. The apparatus of claim 6, further comprising:
the access unit is used for acquiring a mapping file corresponding to the cold data when receiving a data access request corresponding to the cold data sent by a server; determining a target resource pool currently storing the cold data based on the mapping file; and executing the access operation of the cold data corresponding to the data access request in the target resource pool, and returning an operation result corresponding to the access operation to the server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111015623.2A CN113721854A (en) | 2021-08-31 | 2021-08-31 | Data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111015623.2A CN113721854A (en) | 2021-08-31 | 2021-08-31 | Data storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113721854A true CN113721854A (en) | 2021-11-30 |
Family
ID=78680042
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111015623.2A Pending CN113721854A (en) | 2021-08-31 | 2021-08-31 | Data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113721854A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101263494A (en) * | 2005-09-30 | 2008-09-10 | 新途径网络公司 | Accumulating access frequency and file attributes for supporting policy based storage management |
| CN103605615A (en) * | 2013-11-21 | 2014-02-26 | 浪潮电子信息产业股份有限公司 | Block-level-data-based directional allocation method for hierarchical storage |
| US20140189236A1 (en) * | 2012-12-28 | 2014-07-03 | Huawei Technologies Co., Ltd. | Data storage method and storage device |
| CN107870916A (en) * | 2016-09-23 | 2018-04-03 | 伊姆西Ip控股有限责任公司 | Memory management method and equipment |
| CN109918448A (en) * | 2019-03-06 | 2019-06-21 | 电子科技大学 | A kind of cloud storage data classification method based on user behavior |
| CN112825023A (en) * | 2019-11-20 | 2021-05-21 | 上海商汤智能科技有限公司 | Cluster resource management method and device, electronic equipment and storage medium |
-
2021
- 2021-08-31 CN CN202111015623.2A patent/CN113721854A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101263494A (en) * | 2005-09-30 | 2008-09-10 | 新途径网络公司 | Accumulating access frequency and file attributes for supporting policy based storage management |
| US20140189236A1 (en) * | 2012-12-28 | 2014-07-03 | Huawei Technologies Co., Ltd. | Data storage method and storage device |
| CN103605615A (en) * | 2013-11-21 | 2014-02-26 | 浪潮电子信息产业股份有限公司 | Block-level-data-based directional allocation method for hierarchical storage |
| CN107870916A (en) * | 2016-09-23 | 2018-04-03 | 伊姆西Ip控股有限责任公司 | Memory management method and equipment |
| CN109918448A (en) * | 2019-03-06 | 2019-06-21 | 电子科技大学 | A kind of cloud storage data classification method based on user behavior |
| CN112825023A (en) * | 2019-11-20 | 2021-05-21 | 上海商汤智能科技有限公司 | Cluster resource management method and device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 屠要峰;杨洪章;韩银俊;陈正华;: "基于分布式块存储系统的服务质量研究与优化", 电信科学, no. 04, pages 124 - 134 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3258397A1 (en) | Text address processing method and apparatus | |
| CN101370025A (en) | Storing method, scheduling method and management system for geographic information data | |
| CN103905503A (en) | Data storage method, data scheduling method, device and system | |
| WO2016145993A1 (en) | Method and system for user device identification | |
| CN103544153A (en) | Data updating method and system based on database | |
| CN107181821A (en) | A kind of information push method and device based on SSE specifications | |
| CN110381151A (en) | A kind of warping apparatus detection method and device | |
| CN110046174B (en) | population migration analysis method and system based on big data | |
| CN113204580A (en) | Overwater target position information query method, system, device, equipment and medium | |
| CN114095567A (en) | Data access request processing method and device, computer equipment and medium | |
| CN108009205B (en) | Search result caching method based on position, search method, client and system | |
| CN109271453A (en) | A kind of determination method and apparatus of database volume | |
| CN102843369B (en) | The Network Access Method at UI interface and system | |
| US20140180574A1 (en) | Electronic device and method for updating rendezvous location of communication devices | |
| CN113721854A (en) | Data storage method and device | |
| CN115238837A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN109241110A (en) | Order management method and system, electronic equipment, storage medium | |
| CN111611337B (en) | Terminal data processing system | |
| CN111311014B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN114896462A (en) | Rescue material recommendation method and device, electronic equipment and readable storage medium | |
| CN103024091A (en) | Method and device for acquiring real physical address of network client | |
| CN107766216A (en) | It is a kind of to be used to obtain the method and apparatus using execution information | |
| CN111147556A (en) | Load balancing method, device, equipment and storage medium | |
| CN112052302B (en) | Method, system, device and storage medium for dynamically generating position word stock | |
| CN105678155A (en) | Verification method and device for information storage |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |