CN111078144A - Method, system, terminal and storage medium for improving automatic layering efficiency - Google Patents

Method, system, terminal and storage medium for improving automatic layering efficiency Download PDF

Info

Publication number
CN111078144A
CN111078144A CN201911208391.5A CN201911208391A CN111078144A CN 111078144 A CN111078144 A CN 111078144A CN 201911208391 A CN201911208391 A CN 201911208391A CN 111078144 A CN111078144 A CN 111078144A
Authority
CN
China
Prior art keywords
data
pseudo
partition
hot spot
threshold
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911208391.5A
Other languages
Chinese (zh)
Inventor
吴丙涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Suzhou Inspur Intelligent Technology Co Ltd
Original Assignee
Suzhou Inspur Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Suzhou Inspur Intelligent Technology Co Ltd filed Critical Suzhou Inspur Intelligent Technology Co Ltd
Priority to CN201911208391.5A priority Critical patent/CN111078144A/en
Publication of CN111078144A publication Critical patent/CN111078144A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604Improving or facilitating administration, e.g. storage management
    • G06F3/0607Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0629Configuration or reconfiguration of storage systems
    • G06F3/0631Configuration or reconfiguration of storage systems by allocating resources to storage systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0628Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638Organizing or formatting or addressing of data
    • G06F3/0644Management of space entities, e.g. partitions, extents, pools

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a method, a system, a terminal and a storage medium for improving automatic layering efficiency, which comprises the following steps: dividing pseudo hot spot data from the stored data according to the data access degree; storing the metadata of the pseudo hot spot data to a high-speed layer; and storing the data part of the pseudo hot spot data to a low speed layer. The invention reduces the access times of the low-speed storage space and can effectively improve the system performance.

Description

Method, system, terminal and storage medium for improving automatic layering efficiency
Technical Field
The invention relates to the technical field of storage, in particular to a method, a system, a terminal and a storage medium for improving automatic layering efficiency.
Background
The data in the information age is explosively increased, the data volume is in a geometric growth trend due to the development of technologies such as big data and cloud computing. To back end storage access efficiency more and more high, the user is in order to obtain high access efficiency, and purchase the storage medium (like NVME, SSD etc.) that the price is expensive, directly leads to user equipment cost continuously to increase, and the storage manufacturer promotes data access efficiency through automatic layering function with cold and hot data separation, when data access degree constantly changes at cold and hot boundary value, will appear the problem that data constantly make a round trip to shift in the storage layer of difference, influences overall efficiency.
Disclosure of Invention
In view of the above-mentioned deficiencies of the prior art, the present invention provides a method, a system, a terminal and a storage medium for improving automatic layering efficiency, so as to solve the above-mentioned technical problems.
In a first aspect, the present invention provides a method for improving automatic layering efficiency, including:
dividing pseudo hot spot data from the stored data according to the data access degree;
storing the metadata of the pseudo hot spot data to a high-speed layer;
and storing the data part of the pseudo hot spot data to a low speed layer.
Further, the method further comprises:
dividing a high-speed layer into a first partition and a second partition;
storing hotspot data frequently accessed by a user in the first partition;
and storing metadata of the pseudo hotspot data to a second partition, wherein the metadata comprises attribute information of the pseudo hotspot data.
Further, the dividing of the pseudo hotspot data from the storage data according to the data access degree comprises
Setting a hot data threshold and a cold data threshold;
and dividing the data with the access degree between the hot data threshold and the cold data threshold into pseudo hot spot data.
In a second aspect, the present invention provides a system for improving automatic layering efficiency, comprising:
the data dividing unit is configured for dividing the pseudo hot spot data from the storage data according to the data access degree;
the high-speed storage unit is configured for storing the metadata of the pseudo hot spot data to a high-speed layer;
and the low-speed storage unit is configured to store the data part of the pseudo hot spot data to a low-speed layer.
Further, the system further comprises:
a high speed partition module configured to divide a high speed layer into a first partition and a second partition;
the first storage module is configured to store hotspot data frequently accessed by a user in the first partition;
and the second storage module is configured to store metadata of the pseudo hotspot data to a second partition, wherein the metadata comprises attribute information of the pseudo hotspot data.
Further, the data dividing unit includes:
a threshold setting module configured to set a hot data threshold and a cold data threshold;
and the data dividing module is configured for dividing the data with the access degree between the hot data threshold and the cold data threshold into pseudo hot spot data.
In a third aspect, a terminal is provided, including:
a processor, a memory, wherein,
the memory is used for storing a computer program which,
the processor is used for calling and running the computer program from the memory so as to make the terminal execute the method of the terminal.
In a fourth aspect, a computer storage medium is provided having stored therein instructions that, when executed on a computer, cause the computer to perform the method of the above aspects.
The beneficial effect of the invention is that,
according to the method, the system, the terminal and the storage medium for improving the automatic layering efficiency, the first high-speed layer is divided into two partitions, and the partition A is used for storing hot spot data frequently accessed by a user; the partition B is used for storing the metadata of the pseudo hot spot data, and the influence on the space of the whole first high-speed layer is low due to the fact that the metadata of the data occupies small space. And the data part of the pseudo hot spot data is stored in the next-layer low-speed storage space, when the user accesses the attribute (such as file name, size, date and other information) of the pseudo hot spot data, the pseudo hot spot data can be directly returned from the pseudo data partition of the first layer, the low-speed storage space is not required to be accessed, the data access efficiency is improved, and the low-speed storage space can be accessed only when the user actually reads the pseudo hot spot data. The invention reduces the access times of the low-speed storage space and can effectively improve the system performance.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention.
FIG. 2 is a schematic block diagram of a system of one embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The following explains key terms appearing in the present invention.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention. The execution subject of fig. 1 may be a system for improving automatic layering efficiency.
As shown in fig. 1, the method 100 includes:
step 110, dividing pseudo hot spot data from storage data according to the data access degree;
step 120, storing the metadata of the pseudo hot spot data to a high-speed layer;
and step 130, storing the data part of the pseudo hot spot data to a low speed layer.
Optionally, as an embodiment of the present invention, the method further includes:
dividing a high-speed layer into a first partition and a second partition;
storing hotspot data frequently accessed by a user in the first partition;
and storing metadata of the pseudo hotspot data to a second partition, wherein the metadata comprises attribute information of the pseudo hotspot data.
Optionally, as an embodiment of the present invention, the dividing the pseudo hotspot data from the storage data according to the data access degree includes
Setting a hot data threshold and a cold data threshold;
and dividing the data with the access degree between the hot data threshold and the cold data threshold into pseudo hot spot data.
In order to facilitate understanding of the present invention, the method for improving automatic layering efficiency provided by the present invention is further described below with reference to the principle of the method for improving automatic layering efficiency of the present invention and the process of processing data in the embodiment.
Specifically, the method for improving automatic layering efficiency includes:
step 1, initializing a rear-end automatic layering space, and dividing a first high-speed layer into a hot spot data area A and a pseudo hot spot data area B.
And 2, setting two thresholds x and y, wherein the data higher than the threshold x is hot spot data, the data lower than the threshold y is cold data, and the data between the x and the y is pseudo hot spot data.
And 3, taking the period T as a default cycle, acquiring the access amount of the data block in one cycle, and dividing hot spot data, pseudo hot spot data and cold data according to the threshold values x and y.
And 4, storing the hot data in the partition A, storing the pseudo hot data metadata in the partition B and storing the pseudo hot data part and the cold data in the low-speed storage layer through a back-end dump process.
And 5, judging the front-end access type, returning access hot data from the partition A, directly returning access cold data from the low-speed storage layer, and directly returning access to the low-speed storage space only when a user actually reads the pseudo hot data if accessing the pseudo hot data metadata part (such as file name, size, date and the like) from the partition B.
As shown in fig. 2, the system 200 includes:
the data dividing unit 210 is configured to divide the pseudo hotspot data from the stored data according to the data access degree;
a high-speed storage unit 220 configured to store metadata of the pseudo hotspot data to a high-speed layer;
and the low-speed storage unit 230 is configured to store the data part of the pseudo hot spot data to a low-speed layer.
Optionally, as an embodiment of the present invention, the system further includes:
a high speed partition module configured to divide a high speed layer into a first partition and a second partition;
the first storage module is configured to store hotspot data frequently accessed by a user in the first partition;
and the second storage module is configured to store metadata of the pseudo hotspot data to a second partition, wherein the metadata comprises attribute information of the pseudo hotspot data.
Optionally, as an embodiment of the present invention, the data dividing unit includes:
a threshold setting module configured to set a hot data threshold and a cold data threshold;
and the data dividing module is configured for dividing the data with the access degree between the hot data threshold and the cold data threshold into pseudo hot spot data.
Fig. 3 is a schematic structural diagram of a terminal system 300 according to an embodiment of the present invention, where the terminal system 300 may be used to execute the method for improving automatic layering efficiency according to the embodiment of the present invention.
The terminal system 300 may include: a processor 310, a memory 320, and a communication unit 330. The components communicate via one or more buses, and those skilled in the art will appreciate that the architecture of the servers shown in the figures is not intended to be limiting, and may be a bus architecture, a star architecture, a combination of more or less components than those shown, or a different arrangement of components.
The memory 320 may be used for storing instructions executed by the processor 310, and the memory 320 may be implemented by any type of volatile or non-volatile storage terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The executable instructions in memory 320, when executed by processor 310, enable terminal 300 to perform some or all of the steps in the method embodiments described below.
The processor 310 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by operating or executing software programs and/or modules stored in the memory 320 and calling data stored in the memory. The processor may be composed of an Integrated Circuit (IC), for example, a single packaged IC, or a plurality of packaged ICs connected with the same or different functions. For example, the processor 310 may include only a Central Processing Unit (CPU). In the embodiment of the present invention, the CPU may be a single operation core, or may include multiple operation cores.
A communication unit 330, configured to establish a communication channel so that the storage terminal can communicate with other terminals. And receiving user data sent by other terminals or sending the user data to other terminals.
The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, and the program may include some or all of the steps in the embodiments provided by the present invention when executed. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a Random Access Memory (RAM).
Therefore, the first high-speed layer is divided into two partitions, and the partition A is used for storing hot spot data frequently accessed by a user; the partition B is used for storing the metadata of the pseudo hot spot data, and the influence on the space of the whole first high-speed layer is low due to the fact that the metadata of the data occupies small space. The data part of the pseudo hotspot data is stored in the next-layer low-speed storage space, when the user accesses the attribute (such as file name, size, date and other information) of the pseudo hotspot data, the pseudo hotspot data can be directly returned from the pseudo data partition of the first layer, the low-speed storage space does not need to be accessed, the data access efficiency is improved, and the user can access the low-speed storage space only when actually reading the pseudo hotspot data.
Those skilled in the art will readily appreciate that the techniques of the embodiments of the present invention may be implemented as software plus a required general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied in the form of a software product, where the computer software product is stored in a storage medium, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like, and the storage medium can store program codes, and includes instructions for enabling a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, and the like) to perform all or part of the steps of the method in the embodiments of the present invention.
The same and similar parts in the various embodiments in this specification may be referred to each other. Especially, for the terminal embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant points can be referred to the description in the method embodiment.
In the embodiments provided in the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, the division of the units is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, systems or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (8)

1. The patent provides a method for improving automatic layering efficiency, which is characterized by comprising the following steps:
dividing pseudo hot spot data from the stored data according to the data access degree;
storing the metadata of the pseudo hot spot data to a high-speed layer;
and storing the data part of the pseudo hot spot data to a low speed layer.
2. The method of claim 1, further comprising:
dividing a high-speed layer into a first partition and a second partition;
storing hotspot data frequently accessed by a user in the first partition;
and storing metadata of the pseudo hotspot data to a second partition, wherein the metadata comprises attribute information of the pseudo hotspot data.
3. The method of claim 1, wherein partitioning the pseudo-hotspot data from the stored data according to the degree of data access comprises
Setting a hot data threshold and a cold data threshold;
and dividing the data with the access degree between the hot data threshold and the cold data threshold into pseudo hot spot data.
4. A system for improving the efficiency of automatic stratification, comprising:
the data dividing unit is configured for dividing the pseudo hot spot data from the storage data according to the data access degree;
the high-speed storage unit is configured for storing the metadata of the pseudo hot spot data to a high-speed layer;
and the low-speed storage unit is configured to store the data part of the pseudo hot spot data to a low-speed layer.
5. The system of claim 4, further comprising:
a high speed partition module configured to divide a high speed layer into a first partition and a second partition;
the first storage module is configured to store hotspot data frequently accessed by a user in the first partition;
and the second storage module is configured to store metadata of the pseudo hotspot data to a second partition, wherein the metadata comprises attribute information of the pseudo hotspot data.
6. The system of claim 4, wherein the data partitioning unit comprises:
a threshold setting module configured to set a hot data threshold and a cold data threshold;
and the data dividing module is configured for dividing the data with the access degree between the hot data threshold and the cold data threshold into pseudo hot spot data.
7. A terminal, comprising:
a processor;
a memory for storing instructions for execution by the processor;
wherein the processor is configured to perform the method of any one of claims 1-3.
8. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-3.
CN201911208391.5A 2019-11-30 2019-11-30 Method, system, terminal and storage medium for improving automatic layering efficiency Pending CN111078144A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911208391.5A CN111078144A (en) 2019-11-30 2019-11-30 Method, system, terminal and storage medium for improving automatic layering efficiency

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911208391.5A CN111078144A (en) 2019-11-30 2019-11-30 Method, system, terminal and storage medium for improving automatic layering efficiency

Publications (1)

Publication Number Publication Date
CN111078144A true CN111078144A (en) 2020-04-28

Family

ID=70312261

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911208391.5A Pending CN111078144A (en) 2019-11-30 2019-11-30 Method, system, terminal and storage medium for improving automatic layering efficiency

Country Status (1)

Country Link
CN (1) CN111078144A (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101369451A (en) * 2007-08-14 2009-02-18 三星电子株式会社 Solid state memory (ssm), computer system including an ssm, and method of operating an ssm
CN101281542B (en) * 2008-05-09 2011-04-20 成都市华为赛门铁克科技有限公司 Method and device for storing file
CN104866438A (en) * 2014-02-20 2015-08-26 联想(北京)有限公司 Storage device, storage device control method and electronic equipment
CN105069002A (en) * 2014-03-21 2015-11-18 Aupera技术有限公司 Flash memory file system and method using different types of storage media
CN107888687A (en) * 2017-11-15 2018-04-06 长沙证通云计算有限公司 A kind of agent client storage accelerated method and system based on distributed memory system
US20180260154A1 (en) * 2017-03-09 2018-09-13 Netapp Inc. Selectively storing data into allocations areas using streams

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101369451A (en) * 2007-08-14 2009-02-18 三星电子株式会社 Solid state memory (ssm), computer system including an ssm, and method of operating an ssm
CN101281542B (en) * 2008-05-09 2011-04-20 成都市华为赛门铁克科技有限公司 Method and device for storing file
CN104866438A (en) * 2014-02-20 2015-08-26 联想(北京)有限公司 Storage device, storage device control method and electronic equipment
CN105069002A (en) * 2014-03-21 2015-11-18 Aupera技术有限公司 Flash memory file system and method using different types of storage media
US20180260154A1 (en) * 2017-03-09 2018-09-13 Netapp Inc. Selectively storing data into allocations areas using streams
CN107888687A (en) * 2017-11-15 2018-04-06 长沙证通云计算有限公司 A kind of agent client storage accelerated method and system based on distributed memory system

Similar Documents

Publication Publication Date Title
CN109445690B (en) RAID card performance optimization method, device, terminal and storage medium
CN101763433B (en) Data storage system and method
CN112667405B (en) Information processing method, device, equipment and storage medium
CN111984557A (en) Data processing method, device and system
CN112214384A (en) Hard disk serial number management method, system, terminal and storage medium
CN113687949B (en) Server deployment method, device, deployment equipment and storage medium
CN112906332B (en) Comprehensive implementation method and device for FPGA (field programmable Gate array) design
CN111966682B (en) White list protection matching method, system, terminal and storage medium
CN112003730A (en) Method, system, terminal and storage medium for rapid cluster deployment
CN111142787A (en) Storage-layered data exchange testing method, system, terminal and storage medium
CN111858590B (en) Storage system metadata organization method, system, terminal and storage medium
CN111078144A (en) Method, system, terminal and storage medium for improving automatic layering efficiency
CN111124890A (en) Distributed database performance test method, system, terminal and storage medium
CN110989936A (en) Storage space allocation method and device and storage medium
CN111858665B (en) Method, system, terminal and storage medium for improving soft copy reading performance
CN111026682B (en) Data access method and device of board card chip and computer readable storage medium
CN110703988B (en) Storage pool creating method, system, terminal and storage medium for distributed storage
CN111966286A (en) Multi-data pool hierarchical migration method and system
CN111008195A (en) Database free space management method, system, terminal and storage medium
CN111949617A (en) Aggregate file object header management method, system, terminal and storage medium
CN111090633A (en) Small file aggregation method, device and equipment of distributed file system
CN111273867B (en) Block-based data relocation method, system, terminal and storage medium
CN111949414B (en) Method, system, terminal and storage medium for optimizing performance of storage controller
CN110780992A (en) Cloud computing platform optimized deployment method, system, terminal and storage medium
CN112988250B (en) Multi-mainboard USB service configuration method, system, terminal and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20200428