CN114371815A - Data storage method, device and system - Google Patents

Data storage method, device and system Download PDF

Info

Publication number
CN114371815A
CN114371815A CN202111522331.8A CN202111522331A CN114371815A CN 114371815 A CN114371815 A CN 114371815A CN 202111522331 A CN202111522331 A CN 202111522331A CN 114371815 A CN114371815 A CN 114371815A
Authority
CN
China
Prior art keywords
storage
storage space
stored
data
zone
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111522331.8A
Other languages
Chinese (zh)
Other versions
CN114371815B (en
Inventor
姜慧敏
吕庭昌
李文俊
江文龙
周明伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhejiang Dahua Technology Co Ltd
Original Assignee
Zhejiang Dahua Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhejiang Dahua Technology Co Ltd filed Critical Zhejiang Dahua Technology Co Ltd
Priority to CN202111522331.8A priority Critical patent/CN114371815B/en
Priority claimed from CN202111522331.8A external-priority patent/CN114371815B/en
Publication of CN114371815A publication Critical patent/CN114371815A/en
Application granted granted Critical
Publication of CN114371815B publication Critical patent/CN114371815B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G06F3/0644—Management of space entities, e.g. partitions, extents, pools
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0673—Single storage device
    • G06F3/0674—Disk device
    • G06F3/0676—Magnetic disk device

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Security & Cryptography (AREA)
  • Signal Processing For Digital Recording And Reproducing (AREA)

Abstract

The application provides a data storage method, a device and a system, which are used for solving the technical problem that the data storage cannot be continuously carried out and further the data loss is caused in the existing data storage method, wherein the method comprises the following steps: determining at least one storage Zone of the data stream to be stored on the SMR disk according to the storage information of the data stream to be stored; writing the data streams to be stored into at least one storage Zone in sequence, and determining whether the residual storage space of the at least one storage Zone can store the residual data of the data streams to be stored according to the storage space information sent by the SMR disk at regular time; if not, adjusting the size of the reserved storage space of the SMR disk to be capable of triggering an emergency coverage strategy; and writing the residual data of the data stream to be stored into the residual storage space of at least one storage Zone and the storage space of the SMR disk released after the emergency coverage strategy is triggered.

Description

Data storage method, device and system
Technical Field
The present application relates to the field of data storage technologies, and in particular, to a data storage method, apparatus, and system.
Background
With the development of society, more and more data need to be stored in daily life, and in order to ensure that data cannot be lost due to the replacement of devices such as mobile phones and computers, people usually perform data storage and backup at a cloud end. Among various disks in the cloud, a Shingled Magnetic Recording (SMR) disk is used as a high-capacity disk adopting a novel Magnetic storage technology, has the characteristics of high track density, high unit area storage density and the like, and is very suitable for large-scale data storage. The SMR disk is divided into a plurality of areas (zones), random writing can be carried out between the zones, but sequential writing is required inside the zones, so that the SMR disk has the waste of storage space caused by the fact that upper layer business cannot control the size of data written into the zones under normal conditions, and the waste of storage space caused by the fact that the data written into the zones are frequently interrupted under abnormal conditions (disk damage, network frequent fluctuation and the like), and further the actually usable storage space of the SMR disk is reduced.
At present, when a user perceives that the actually usable storage space of an SMR disk is reduced to a value that data cannot be stored continuously, the user generally adjusts the storage space of the SMR disk by triggering a lifecycle policy and an emergency coverage policy, so that data storage can be performed continuously, where the lifecycle policy is to delete data whose storage time reaches a lifecycle, and the emergency coverage policy is to delete data whose storage time is longest when a ratio between a used storage space and a total storage space is not less than a threshold, so that a ratio between the used storage space and the total storage space is less than the threshold. However, for a user, the storage space actually usable by the SMR disk does not include a wasted storage space, and the user may not perceive the reduction of the storage space actually usable by the SMR disk due to the waste of the storage space of the SMR disk, and may not trigger the life cycle policy and the emergency coverage policy, so that data storage cannot be continued, and data loss may be caused.
Therefore, the existing data storage method has the problems that data storage cannot be continuously carried out, and data loss is caused.
Disclosure of Invention
The embodiment of the application provides a data storage method, a data storage device and a data storage system, which are used for solving the technical problem that data cannot be stored continuously and data is lost in the existing data storage method.
In a first aspect, to solve the foregoing technical problem, an embodiment of the present application provides a data storage method, including:
determining at least one storage area Zone of the data stream to be stored on a Shingled Magnetic Recording (SMR) disk according to storage information of the data stream to be stored, wherein the storage Zone is a Zone which does not store data or a Zone which stores data, and the storage information of the stored data is the same as the storage information of the data stream to be stored;
writing the data stream to be stored into the at least one storage Zone in sequence, and determining whether the remaining storage space of the at least one storage Zone can store the remaining data of the data stream to be stored according to the storage space information sent by the SMR disk at regular time;
if the SMR disk cannot be used, adjusting the size of the reserved storage space of the SMR disk to be capable of triggering an emergency coverage strategy;
and writing the residual data of the data stream to be stored into the residual storage space of the at least one storage Zone and the storage space of the SMR disk released after the emergency coverage strategy is triggered in sequence.
In this embodiment of the application, at least one storage Zone of a data stream to be stored on an SMR disk may be determined according to storage information of the data stream to be stored, where the storage Zone is a Zone that does not store data or a Zone that stores data, the storage information of the stored data is the same as the storage information of the data stream to be stored, the data stream to be stored is sequentially written into the at least one storage Zone, and whether the remaining storage space of the at least one storage Zone is capable of storing remaining data of the data stream to be stored is determined according to storage space information sent by the SMR disk at regular time, if not, the size of the reserved storage space of the SMR disk is adjusted to be capable of triggering an emergency coverage policy, and the remaining data of the data stream to be stored is sequentially written into the remaining storage space of the at least one storage Zone and a storage space of the SMR disk that is released after the emergency coverage policy is triggered. When data writing is normal, data with the same storage information is written into the same Zone, the utilization rate of the storage space of the SMR disk is improved, waste of the storage space of the SMR disk is further reduced, when the data writing is determined to be abnormal according to the storage space information sent by the SMR disk at regular time, the size of the reserved storage space of the SMR disk is adjusted to be capable of triggering an emergency coverage strategy, a certain storage space of the SMR disk is released, data writing is guaranteed to be normal, data storage can be carried out continuously, and data loss is avoided.
In an optional embodiment, the storage information of the data stream to be stored includes any one or more of a life cycle, a redundancy mode, or a data type of the data stream to be stored.
In an optional implementation manner, determining, according to storage space information sent by the SMR disk at regular time, whether the remaining storage space of the at least one storage Zone is capable of storing the data stream to be stored includes:
receiving storage space information sent by the SMR disk at regular time, wherein the storage space information is used for indicating the used storage space of the at least one storage Zone;
determining whether a first ratio is not smaller than a first preset threshold and whether a second ratio is not smaller than a second preset threshold according to the storage space information, wherein the first ratio is a ratio between a used storage space of the data stream to be stored and a total storage space of the at least one storage Zone, the second ratio is a ratio between a first difference and the total storage space of the at least one storage Zone, and the first difference is a difference between the used storage space of the at least one storage Zone and the used storage space of the data stream to be stored;
if the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold, determining that the remaining storage space of the at least one storage Zone cannot store the data stream to be stored.
In the embodiment of the present application, storage space information sent by the SMR disk timing may be received, where the storage space information is used to indicate used storage space of at least one storage Zone, determining whether the first ratio is not less than a first preset threshold and whether the second ratio is not less than a second preset threshold according to the storage space information, wherein the first ratio is a ratio between an used storage space of the data stream to be stored and a total storage space of the at least one storage Zone, the second ratio is a ratio between a first difference and the total storage space of the at least one storage Zone, the first difference is a difference between the used storage space of the at least one storage Zone and the used storage space of the data stream to be stored, and if the first ratio is not less than a first preset threshold and the second ratio is not less than a second preset threshold, it is determined that the remaining storage space of the at least one storage Zone is not capable of storing the data stream to be stored. The method comprises the steps of determining whether a first ratio between a used storage space of a data stream to be stored and a total storage space of at least one storage Zone is not smaller than a first preset threshold value or not and whether a second ratio between a waste storage space of the at least one storage Zone and the total storage space of the at least one storage Zone is not smaller than a second preset threshold value or not through storage space information sent by an SMR disk at regular time, determining that data writing is abnormal when the first ratio is not smaller than the first preset threshold value and the second ratio is not smaller than the second preset threshold value, and avoiding that a user cannot perceive reduction of an actually usable storage space of the SMR disk due to waste of the storage space of the SMR disk, so that data storage can be continuously carried out, and data loss is avoided.
In an optional implementation manner, after writing the remaining data of the data stream to be stored into the remaining storage space of the at least one storage Zone and the storage space of the SMR disk released after the emergency coverage policy is triggered in sequence, the method further includes:
and restoring the size of the reserved storage space, and migrating the data stored in the restored reserved storage space to a non-reserved storage space.
In a second aspect, embodiments of the present application further provide a data storage device, including:
the determining module is configured to determine, according to storage information of a data stream to be stored, at least one storage area Zone of the data stream to be stored on the shingled magnetic recording SMR disk, where the storage Zone is a Zone where data is not stored or a Zone where data is stored, and the storage information of the stored data is the same as the storage information of the data stream to be stored;
a first processing module, configured to write the data stream to be stored into the at least one storage Zone in sequence, and determine, according to storage space information sent by the SMR disk at regular time, whether a remaining storage space of the at least one storage Zone is capable of storing remaining data of the data stream to be stored;
the adjusting module is used for adjusting the size of the reserved storage space of the SMR disk to be capable of triggering an emergency coverage strategy if the SMR disk cannot be used for storing the emergency coverage strategy;
and a second processing module, configured to write the remaining data of the data stream to be stored into the remaining storage space of the at least one storage Zone and the storage space of the SMR disk released after the emergency coverage policy is triggered in sequence.
In an optional embodiment, the storage information of the data stream to be stored includes any one or more of a life cycle, a redundancy mode, or a data type of the data stream to be stored.
In an optional implementation manner, the first processing module is specifically configured to:
receiving storage space information sent by the SMR disk at regular time, wherein the storage space information is used for indicating the used storage space of the at least one storage Zone;
determining whether a first ratio is not smaller than a first preset threshold and whether a second ratio is not smaller than a second preset threshold according to the storage space information, wherein the first ratio is a ratio between a used storage space of the data stream to be stored and a total storage space of the at least one storage Zone, the second ratio is a ratio between a first difference and the total storage space of the at least one storage Zone, and the first difference is a difference between the used storage space of the at least one storage Zone and the used storage space of the data stream to be stored;
if the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold, determining that the remaining storage space of the at least one storage Zone cannot store the data stream to be stored.
In an optional embodiment, the apparatus further comprises a third processing module configured to:
and restoring the size of the reserved storage space, and migrating the data stored in the restored reserved storage space to a non-reserved storage space.
In a third aspect, an embodiment of the present application further provides a data storage system, including:
a memory for storing program instructions;
and the processor is used for calling the program instructions stored in the memory and executing the steps included in any one of the implementation modes of the first aspect according to the obtained program instructions.
In a fourth aspect, embodiments of the present application provide a storage medium storing computer-executable instructions for causing a computer to perform the steps included in any one of the embodiments of the first aspect.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application.
Fig. 1 is a schematic structural diagram of a data storage system according to an embodiment of the present application;
fig. 2a is a schematic flowchart of a data storage method according to an embodiment of the present application;
FIG. 2b is a schematic diagram of an SMR disk according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data storage device according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of another data storage system according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application. In the present application, the embodiments and features of the embodiments may be arbitrarily combined with each other without conflict. Also, while a logical order is shown in the flow diagrams, in some cases, the steps shown or described can be performed in an order different than here.
The terms "first" and "second" in the description and claims of the present application and the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the term "comprises" and any variations thereof, which are intended to cover non-exclusive protection. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
In the embodiments of the present application, "at least one" may mean at least two, for example, two, three, or more, and the embodiments of the present application are not limited.
In addition, the term "and/or" herein is only one kind of association relationship describing an associated object, and means that there may be three kinds of relationships, for example, a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" in this document generally indicates that the preceding and following related objects are in an "or" relationship unless otherwise specified.
At present, when a user perceives that the actually usable storage space of an SMR disk is reduced to a value that data cannot be stored continuously, the user generally adjusts the storage space of the SMR disk by triggering a lifecycle policy and an emergency coverage policy, so that data storage can be performed continuously, where the lifecycle policy is to delete data whose storage time reaches a lifecycle, and the emergency coverage policy is to delete data whose storage time is longest when a ratio between a used storage space and a total storage space is not less than a threshold, so that a ratio between the used storage space and the total storage space is less than the threshold. However, for a user, the storage space actually usable by the SMR disk does not include a wasted storage space, and the user may not perceive the reduction of the storage space actually usable by the SMR disk due to the waste of the storage space of the SMR disk, and may not trigger the life cycle policy and the emergency coverage policy, so that data storage cannot be continued, and data loss may be caused. Therefore, the existing data storage method has the problems that data storage cannot be continuously carried out, and data loss is caused.
In view of this, embodiments of the present application provide a data storage method, which may determine at least one storage Zone of a data stream to be stored on an SMR disk according to storage information of the data stream to be stored, wherein, the storage Zone is a Zone which does not store data or a Zone which stores data, the storage information of the stored data is the same as the storage information of the data stream to be stored, the data stream to be stored is written into at least one storage Zone in sequence, and determining whether the residual storage space of at least one storage Zone can store the residual data of the data stream to be stored or not according to the storage space information sent by the SMR disk at regular time, if not, adjusting the size of the reserved storage space of the SMR disk to be capable of triggering an emergency coverage strategy, and writing the residual data of the data stream to be stored into the residual storage space of at least one storage Zone in sequence and the storage space of the SMR disk released after the emergency coverage strategy is triggered. When data writing is normal, data with the same storage information is written into the same Zone, the utilization rate of the storage space of the SMR disk is improved, waste of the storage space of the SMR disk is further reduced, when the data writing is determined to be abnormal according to the storage space information sent by the SMR disk at regular time, the size of the reserved storage space of the SMR disk is adjusted to be capable of triggering an emergency coverage strategy, a certain storage space of the SMR disk is released, data writing is guaranteed to be normal, data storage can be carried out continuously, and data loss is avoided.
In order to better understand the technical solutions, the technical solutions of the present application are described in detail below through the drawings and the specific embodiments of the specification, and it should be understood that the specific features of the embodiments and examples of the present application are detailed descriptions of the technical solutions of the present application, and are not limitations of the technical solutions of the present application, and the technical features of the embodiments and examples of the present application may be combined with each other without conflict.
As shown in fig. 1, a schematic structural diagram of a data storage system to which the method provided in the embodiment of the present application is applicable is shown, but the method provided in the embodiment of the present application can be applied to a variety of data storage systems, and it should be understood that the data storage system shown in fig. 1 is a simple illustration of a data storage system to which the method provided in the embodiment of the present application is applicable, and is not a limitation of a data storage system to which the method provided in the embodiment of the present application is applicable.
The Data storage system shown in fig. 1 includes a management server 101 and an SMR disk 102, where a user or an upper service manages Data stored in the SMR disk 102 through the management server 101, the management server 101 includes an Index Manager (Index Manager), a Sector Manager (Sector Manager), and a Data Manager (Data Manager), where the Index Manager is responsible for managing sorting and storage of Index records, the Sector Manager is responsible for allocating storage space for Data input by the user or the upper service, and the Data Manager is responsible for converting the storage space allocated by the Sector Manager into a read/write instruction and sending the read/write instruction to the SMR disk.
Fig. 2a is a schematic flow chart of a data storage method provided in an embodiment of the present application, where the method may be executed by the data storage system shown in fig. 1. The specific flow of the method is described below.
Step 201: and determining at least one storage Zone of the data stream to be stored on the SMR disk according to the storage information of the data stream to be stored.
In this embodiment of the application, after receiving a to-be-stored data stream input by a user or an upper layer service, a management server may determine, according to storage information of the to-be-stored data stream, at least one storage Zone of the to-be-stored data stream on an SMR disk, where the storage information of the to-be-stored data stream includes any one or more of a life cycle, a redundancy mode, and a data type of the to-be-stored data stream, the storage Zone is a Zone in which data is not stored or a Zone in which data is stored, and the storage information of the stored data is the same as the storage information of the to-be-stored data stream.
For example, as shown in fig. 2b, a schematic diagram of an SMR disk provided in this embodiment of the present application includes a Conventional Magnetic Recording (CMR) area and an SMR area, where the CMR area occupies 1% of a storage space of the SMR disk, and includes a Master Boot Record (MBR) area, a super block (for storing a disk id), a file system xfs (a high-performance log file system for managing metadata), a Cyclic Redundancy Check (CRC) area (for storing Check codes of a Zone in the SMR area), the SMR area occupies 99% of the storage space of the SMR disk, and includes a plurality of zones (e.g., Zone _1 to Zone _ N) for storing written data, a part of the Zone at the tail of the sector of the SMR disk is reserved as the storage space of the SMR disk (typically, 1.5% to 5% of the storage space of the SMR disk), such as Zone _ N-2 to Zone _ N). If the Zone _15 and the Zone _16 of the SMR disk are zones which do not store data, the Zone _20 is a Zone which has stored data, and the storage information of the data stored in the Zone _20 is the same as that of the data stream to be stored, it is determined that the storage zones of the data stream to be stored on the SMR disk are Zone _15, Zone _16 and Zone _20, so that when the data stream to be stored is written into the storage zones in sequence, the remaining storage space of the Zone _20 which has stored data can be written first, and if there is remaining data, the remaining data is written into the Zone _15 or the Zone _16, so that when the data is written normally, the data with the same storage information is written into the same Zone, thereby improving the storage space utilization rate of the SMR disk, and further reducing the waste of the storage space of the SMR disk.
Step 202: and writing the data stream to be stored into at least one storage Zone in sequence, and determining whether the residual storage space of the at least one storage Zone can store the residual data of the data stream to be stored according to the storage space information sent by the SMR disk at regular time.
In this embodiment of the application, after determining at least one storage Zone of the data stream to be stored on the SMR disk, the management server may write the data stream to be stored in the at least one storage Zone in sequence, and determine whether the remaining storage space of the at least one storage Zone is capable of storing the remaining data of the data stream to be stored according to the storage space information sent by the SMR disk at regular time.
Specifically, if abnormal conditions such as disk damage and frequent network fluctuation occur when the data stream to be stored is written into the at least one storage Zone in sequence, the data written into the at least one storage Zone is frequently interrupted, and the storage space of the at least one storage Zone is wasted. For the management server, the remaining storage space of the at least one storage Zone is the total storage space of the at least one storage Zone, i.e. the used storage space of the to-be-stored data stream, but for the disk SMR disk, the remaining storage space of the at least one storage Zone is the total storage space of the at least one storage Zone, i.e. the used storage space of the at least one storage Zone, wherein the used storage space of the at least one storage Zone includes the used storage space of the to-be-stored data stream and the wasted storage space of the at least one storage Zone. It can be seen that the management server itself does not sense the waste of the storage space of the at least one storage Zone, which results in a reduction of the storage space actually used by the at least one storage Zone, but can receive the storage space information periodically sent by the SMR disk, where the storage space information is used to indicate the used storage space of the at least one storage Zone.
When the management server determines that the remaining storage space of the at least one storage Zone can store the remaining data of the data stream to be stored according to the storage space information sent by the SMR disk at regular time, it may first determine whether a first ratio is not smaller than a first preset threshold, where the first ratio is a ratio between an already used storage space of the data stream to be stored and a total storage space of the at least one storage Zone. If the first ratio is smaller than a first preset threshold, determining that the ratio between the used storage space of the current data stream to be stored and the total storage space of the at least one storage Zone is not close to the threshold for triggering the emergency coverage strategy, and waiting for triggering the life cycle strategy and the emergency coverage strategy without determining whether the writing is abnormal in advance. And if the first ratio is not smaller than the first preset threshold, determining whether a second ratio is not smaller than a second preset threshold, wherein the second ratio is a ratio between a first difference and the total storage space of the at least one storage Zone, and the first difference is a difference between the used storage space of the at least one storage Zone and the used storage space of the data stream to be stored. If the first ratio is not smaller than the first preset threshold value but the second ratio is smaller than the second preset threshold value, it is determined that the ratio between the used storage space of the current data stream to be stored and the total storage space of the at least one storage Zone is close to the threshold value for triggering the emergency coverage policy, but no exception occurs in writing, and the life cycle policy and the emergency coverage policy are waited to be triggered. If the first ratio is not smaller than the first preset threshold and the second ratio is not smaller than the second preset threshold, it is determined that the ratio between the used storage space of the current data stream to be stored and the total storage space of the at least one storage Zone is close to the threshold for triggering the emergency coverage policy, and an exception is written, so that it is determined that the remaining storage space of the at least one storage Zone cannot store the remaining data of the data stream to be stored.
Illustratively, the management server receives storage space information sent by the SMR disk at regular time, wherein the storage space information is used for indicating that the used storage space of at least one storage Zone is 100 TB. If the total storage space of the at least one storage Zone is 100TB, the used storage space of the data stream to be stored is 94TB, the first preset threshold is 94%, and the second preset threshold is 6%, it may be determined that the first ratio is 94%, and is not less than the first preset threshold, and the second ratio is 6%, and is not less than the second preset threshold, and it may be further determined that the remaining storage space of the at least one storage Zone is not capable of storing the remaining data of the data stream to be stored.
Step 203: and if not, adjusting the size of the reserved storage space of the SMR disk to trigger the emergency coverage strategy.
In this embodiment of the application, if it is determined that the remaining storage space of the at least one storage Zone is not capable of storing the remaining data of the data stream to be stored, that is, it is determined that a ratio between an already used storage space of the data stream to be stored and a total storage space of the at least one storage Zone is close to a threshold value for triggering the emergency coverage policy, but the emergency coverage policy cannot be triggered when the write-in is abnormal, the management server adjusts the size of the reserved storage space of the SMR disk to be capable of triggering the emergency coverage policy.
Illustratively, when the emergency coverage policy is that the ratio of the used storage space of the data stream to be stored to the total storage space of the at least one storage Zone is not less than 95%, deleting the data with the longest storage time, so that the ratio of the used storage space of the data stream to be stored to the total storage space of the at least one storage Zone is less than 95%, when writing the data stream to be stored of 100TB, abnormal situations such as disk damage, frequent network fluctuation occur, and the like, resulting in a wasted storage space reaching 6TB, at this time, for the SMR disk, the used storage space of the at least one storage Zone is 100TB, the at least one storage Zone cannot continue to write the data stream to be stored, for the management server, the used storage space of the data stream to be stored is 94TB, the threshold of the emergency coverage policy cannot be reached, and the reserved storage space of the SMR disk is reduced by 1%, the ratio of the used storage space of the data stream to be stored to the total storage space of the at least one storage Zone is 95%, and an emergency coverage strategy is triggered, so that the storage space of a certain SMR disk is released to guarantee normal data writing, data storage can be continuously carried out, and data loss is avoided.
Step 204: and writing the residual data of the data stream to be stored into the residual storage space of at least one storage Zone and the storage space of the SMR disk released after the emergency coverage strategy is triggered.
In this embodiment of the application, after the management server adjusts the size of the reserved storage space of the SMR disk to be capable of triggering the emergency coverage policy, the remaining data of the data stream to be stored may be written into the remaining storage space of the at least one storage Zone in sequence and the storage space of the SMR disk released after the emergency coverage policy is triggered, and after the remaining data of the data stream to be stored is all written into the SMR disk, the size of the reserved storage space of the SMR disk is recovered, and the data stored in the recovered reserved storage space is migrated to the non-reserved storage space.
According to the scheme, the management server can determine at least one storage Zone of the data stream to be stored on the SMR disk according to the storage information of the data stream to be stored, where the storage Zone is a Zone that does not store data or a Zone that stores data, the storage information of the stored data is the same as the storage information of the data stream to be stored, write the data stream to be stored into the at least one storage Zone in sequence, determine whether the remaining storage space of the at least one storage Zone can store the remaining data of the data stream to be stored according to the storage space information sent by the SMR disk at regular time, if not, adjust the size of the reserved storage space of the SMR disk to be capable of triggering the emergency coverage policy, and write the remaining data of the data stream to be stored into the remaining storage space of the at least one storage Zone in sequence and the storage space of the SMR disk released after the emergency coverage policy is triggered. When data are written normally, the data with the same storage information are written into the same Zone, the utilization rate of the storage space of the SMR disk is improved, waste of the storage space of the SMR disk is further reduced, when the data are determined to be written abnormally according to the storage space information sent by the SMR disk at regular time, the size of the reserved storage space of the SMR disk is adjusted to be capable of triggering an emergency coverage strategy, the storage space of the SMR disk is released, the data are guaranteed to be written normally, data storage can be carried out continuously, and data loss is avoided.
Based on the same inventive concept, the embodiment of the present application further provides a data storage device, which can be applied to the data storage system shown in fig. 1. The data storage device can realize the corresponding functions of the data storage method. The data storage device may be a hardware structure, a software module, or a hardware structure plus a software module. The data storage device may be implemented by a system-on-chip, which may be formed by a chip, or may include a chip and other discrete devices. Referring to fig. 3, a schematic structural diagram of a data storage device according to an embodiment of the present disclosure is shown, where the data storage device includes a determining module 301, a first processing module 302, an adjusting module 303, and a second processing module 304.
A determining module 301, configured to determine, according to storage information of a data stream to be stored, at least one storage area Zone of the data stream to be stored on a shingled magnetic recording SMR disk, where the storage Zone is a Zone in which data is not stored or a Zone in which data is stored, and the storage information of the stored data is the same as the storage information of the data stream to be stored;
a first processing module 302, configured to write the data stream to be stored into the at least one storage Zone in sequence, and determine whether the remaining storage space of the at least one storage Zone can store the remaining data of the data stream to be stored according to the storage space information sent by the SMR disk at regular time;
an adjusting module 303, configured to adjust the size of the reserved storage space of the SMR disk to be able to trigger an emergency coverage policy if the size of the reserved storage space of the SMR disk is not able to be triggered;
a second processing module 304, configured to write the remaining data of the data stream to be stored into the remaining storage space of the at least one storage Zone and the storage space of the SMR disk released after the emergency coverage policy is triggered in sequence.
In an optional embodiment, the storage information of the data stream to be stored includes any one or more of a life cycle, a redundancy mode, or a data type of the data stream to be stored.
In an optional implementation manner, the first processing module 302 is specifically configured to:
receiving storage space information sent by the SMR disk at regular time, wherein the storage space information is used for indicating the used storage space of the at least one storage Zone;
determining whether a first ratio is not smaller than a first preset threshold and whether a second ratio is not smaller than a second preset threshold according to the storage space information, wherein the first ratio is a ratio between a used storage space of the data stream to be stored and a total storage space of the at least one storage Zone, the second ratio is a ratio between a first difference and the total storage space of the at least one storage Zone, and the first difference is a difference between the used storage space of the at least one storage Zone and the used storage space of the data stream to be stored;
if the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold, determining that the remaining storage space of the at least one storage Zone cannot store the data stream to be stored.
In an optional embodiment, the apparatus further comprises a third processing module configured to:
and restoring the size of the reserved storage space, and migrating the data stored in the restored reserved storage space to a non-reserved storage space.
Based on the same inventive concept, an embodiment of the present application further provides a data storage system, please refer to fig. 4, which is a schematic structural diagram of a data storage system provided in the embodiment of the present application, the data storage system includes at least one processor 402 and a memory 401 connected to the at least one processor, a specific connection medium between the processor 402 and the memory 401 is not limited in the embodiment of the present application, fig. 4 is an example in which the processor 402 and the memory 401 are connected by a bus 400, the bus 400 is shown by a thick line in fig. 4, and a connection manner between other components is only schematically illustrated and not limited. The bus 400 may be divided into an address bus, a data bus, a control bus, etc., and is shown with only one thick line in fig. 4 for ease of illustration, but does not represent only one bus or type of bus.
In the embodiment of the present application, the memory 401 stores instructions executable by the at least one processor 402, and the at least one processor 402 may perform the steps included in the foregoing data storage method by calling the instructions stored in the memory 401. The processor 402 is a control center of the data storage system, and may be connected to various parts of the entire data storage system through various interfaces and lines, and implement various functions of the data storage system by executing instructions stored in the memory 401. Optionally, the processor 402 may include one or more processing units, and the processor 402 may integrate an application processor and a modem processor, wherein the application processor mainly handles operating systems, user interfaces, application programs, and the like, and the modem processor mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 402. In some embodiments, processor 402 and memory 401 may be implemented on the same chip, or in some embodiments, they may be implemented separately on separate chips.
Memory 401, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The Memory 401 may include at least one type of storage medium, and may include, for example, a flash Memory, a hard disk, a multimedia card, a card-type Memory, a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Programmable Read Only Memory (PROM), a Read Only Memory (ROM), a charge Erasable Programmable Read Only Memory (EEPROM), a magnetic Memory, a magnetic disk, an optical disk, and so on. The memory 401 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 401 in the embodiments of the present application may also be a circuit or any other device capable of implementing a storage function for storing program instructions and/or data.
In the embodiments of the present application, the processor 402 may be a general-purpose processor, such as a Central Processing Unit (CPU), a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or discrete hardware components, and may implement or perform the methods, steps, and logic blocks disclosed in the embodiments of the present application. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the data storage method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.
By programming the processor 402, the code corresponding to the data storage method described in the foregoing embodiment may be solidified in the chip, so that the chip can execute the steps of the data storage method when running, and how to program the processor 402 is a technique known by those skilled in the art and will not be described herein again.
Based on the same inventive concept, the present application also provides a storage medium storing computer instructions, which when executed on a computer, cause the computer to perform the steps of the data storage method as described above.
In some possible embodiments, the various aspects of the data storage method provided herein may also be implemented in the form of a program product comprising program code for causing a data storage system to perform the steps of the data storage method according to various exemplary embodiments of the present application described above in this specification when the program product is run on the data storage system.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims (10)

1. A method of storing data, comprising:
determining at least one storage area Zone of the data stream to be stored on a Shingled Magnetic Recording (SMR) disk according to storage information of the data stream to be stored, wherein the storage Zone is a Zone which does not store data or a Zone which stores data, and the storage information of the stored data is the same as the storage information of the data stream to be stored;
writing the data stream to be stored into the at least one storage Zone in sequence, and determining whether the remaining storage space of the at least one storage Zone can store the remaining data of the data stream to be stored according to the storage space information sent by the SMR disk at regular time;
if the SMR disk cannot be used, adjusting the size of the reserved storage space of the SMR disk to be capable of triggering an emergency coverage strategy;
and writing the residual data of the data stream to be stored into the residual storage space of the at least one storage Zone and the storage space of the SMR disk released after the emergency coverage strategy is triggered in sequence.
2. The method of claim 1, wherein the storage information of the data stream to be stored comprises any one or more of a lifecycle, a redundancy pattern, or a data type of the data stream to be stored.
3. The method as claimed in claim 1 or 2, wherein determining whether the remaining storage space of the at least one storage Zone can store the data stream to be stored according to the storage space information sent by the SMR disk timing comprises:
receiving storage space information sent by the SMR disk at regular time, wherein the storage space information is used for indicating the used storage space of the at least one storage Zone;
determining whether a first ratio is not smaller than a first preset threshold and whether a second ratio is not smaller than a second preset threshold according to the storage space information, wherein the first ratio is a ratio between a used storage space of the data stream to be stored and a total storage space of the at least one storage Zone, the second ratio is a ratio between a first difference and the total storage space of the at least one storage Zone, and the first difference is a difference between the used storage space of the at least one storage Zone and the used storage space of the data stream to be stored;
if the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold, determining that the remaining storage space of the at least one storage Zone cannot store the data stream to be stored.
4. The method of claim 3, wherein after writing the remaining data of the data stream to be stored in the remaining storage space of the at least one storage Zone and the storage space of the SMR disk released after the triggering of the emergency coverage policy in order, further comprising:
and restoring the size of the reserved storage space, and migrating the data stored in the restored reserved storage space to a non-reserved storage space.
5. A data storage device, comprising:
the determining module is configured to determine, according to storage information of a data stream to be stored, at least one storage area Zone of the data stream to be stored on the shingled magnetic recording SMR disk, where the storage Zone is a Zone where data is not stored or a Zone where data is stored, and the storage information of the stored data is the same as the storage information of the data stream to be stored;
a first processing module, configured to write the data stream to be stored into the at least one storage Zone in sequence, and determine, according to storage space information sent by the SMR disk at regular time, whether a remaining storage space of the at least one storage Zone is capable of storing remaining data of the data stream to be stored;
the adjusting module is used for adjusting the size of the reserved storage space of the SMR disk to be capable of triggering an emergency coverage strategy if the SMR disk cannot be used for storing the emergency coverage strategy;
and a second processing module, configured to write the remaining data of the data stream to be stored into the remaining storage space of the at least one storage Zone and the storage space of the SMR disk released after the emergency coverage policy is triggered in sequence.
6. The apparatus of claim 5, wherein the storage information of the data stream to be stored comprises any one or more of a lifecycle, a redundancy pattern, or a data type of the data stream to be stored.
7. The apparatus of claim 5 or 6, wherein the first processing module is specifically configured to:
receiving storage space information sent by the SMR disk at regular time, wherein the storage space information is used for indicating the used storage space of the at least one storage Zone;
determining whether a first ratio is not smaller than a first preset threshold and whether a second ratio is not smaller than a second preset threshold according to the storage space information, wherein the first ratio is a ratio between a used storage space of the data stream to be stored and a total storage space of the at least one storage Zone, the second ratio is a ratio between a first difference and the total storage space of the at least one storage Zone, and the first difference is a difference between the used storage space of the at least one storage Zone and the used storage space of the data stream to be stored;
if the first ratio is not less than the first preset threshold and the second ratio is not less than the second preset threshold, determining that the remaining storage space of the at least one storage Zone cannot store the data stream to be stored.
8. The apparatus of claim 7, further comprising a third processing module to:
and restoring the size of the reserved storage space, and migrating the data stored in the restored reserved storage space to a non-reserved storage space.
9. A data storage system, comprising:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory and for executing the steps comprised by the method of any one of claims 1 to 4 in accordance with the obtained program instructions.
10. A storage medium storing computer-executable instructions for causing a computer to perform the steps comprising the method of any one of claims 1-4.
CN202111522331.8A 2021-12-13 Data storage method, device and system Active CN114371815B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111522331.8A CN114371815B (en) 2021-12-13 Data storage method, device and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111522331.8A CN114371815B (en) 2021-12-13 Data storage method, device and system

Publications (2)

Publication Number Publication Date
CN114371815A true CN114371815A (en) 2022-04-19
CN114371815B CN114371815B (en) 2024-05-28

Family

ID=

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109597568A (en) * 2018-09-18 2019-04-09 天津字节跳动科技有限公司 A kind of date storage method, device, terminal device and storage medium
CN111666046A (en) * 2020-05-20 2020-09-15 西安奥卡云数据科技有限公司 Data storage method, device and equipment
EP3809708A1 (en) * 2018-06-13 2021-04-21 Hangzhou Hikvision System Technology Co., Ltd. Video data storage method and device in cloud storage system
CN113495889A (en) * 2021-07-07 2021-10-12 浙江大华技术股份有限公司 Distributed object storage method and device, electronic equipment and storage medium
CN113672171A (en) * 2021-07-26 2021-11-19 浙江大华技术股份有限公司 Distributed object storage method, device and system and metadata server
CN113741814A (en) * 2021-08-24 2021-12-03 深圳Tcl新技术有限公司 Data processing method, data processing device, computer readable storage medium and computer equipment

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3809708A1 (en) * 2018-06-13 2021-04-21 Hangzhou Hikvision System Technology Co., Ltd. Video data storage method and device in cloud storage system
CN109597568A (en) * 2018-09-18 2019-04-09 天津字节跳动科技有限公司 A kind of date storage method, device, terminal device and storage medium
CN111666046A (en) * 2020-05-20 2020-09-15 西安奥卡云数据科技有限公司 Data storage method, device and equipment
CN113495889A (en) * 2021-07-07 2021-10-12 浙江大华技术股份有限公司 Distributed object storage method and device, electronic equipment and storage medium
CN113672171A (en) * 2021-07-26 2021-11-19 浙江大华技术股份有限公司 Distributed object storage method, device and system and metadata server
CN113741814A (en) * 2021-08-24 2021-12-03 深圳Tcl新技术有限公司 Data processing method, data processing device, computer readable storage medium and computer equipment

Similar Documents

Publication Publication Date Title
US10649953B2 (en) Blockchain-based data migration method and apparatus
CN107220148B (en) Reconstruction method and device for redundant array of independent disks
CN102934094B (en) Hierarchical allocation for file system storage device
EP3777026A1 (en) Multi-level storage method and apparatus for blockchain data
US20120117304A1 (en) Managing memory with limited write cycles in heterogeneous memory systems
CN104111897A (en) Data processing method, data processing device and computer system
KR20160022226A (en) Heterogeneous unified memory section and method for manaing extended unified memory space thereof
CN113176858B (en) Data processing method, storage system and storage device
KR20200113047A (en) Operation method of open-channel storage device
CN105574141B (en) Method and device for carrying out data migration on database
CN109558335B (en) Nor Flash memory based file storage format of embedded system
CN107665098B (en) Information processing method, storage device, and computer storage medium
CN113495889A (en) Distributed object storage method and device, electronic equipment and storage medium
CN108958660B (en) Distributed storage system and data processing method and device thereof
CN109375868B (en) Data storage method, scheduling device, system, equipment and storage medium
CN112463058B (en) Fragmented data sorting method and device and storage node
CN114371815B (en) Data storage method, device and system
CN115469803B (en) Queue message storage scheduling method and application
CN114371815A (en) Data storage method, device and system
US11507458B2 (en) Storage management method, device, and computer program product
US20070061530A1 (en) Method for storage of digital data in a mainframe data center and associated device
CN103064926A (en) Data processing method and device
CN115543859A (en) Wear leveling optimization method, device, equipment and medium for multi-partition SSD
CN110262758B (en) Data storage management method, system and related equipment
CN113467722A (en) Data migration method and device of distributed storage system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant