CN111221468A - Storage block data deleting method and device, electronic equipment and cloud storage system - Google Patents
Storage block data deleting method and device, electronic equipment and cloud storage system Download PDFInfo
- Publication number
- CN111221468A CN111221468A CN201811419430.1A CN201811419430A CN111221468A CN 111221468 A CN111221468 A CN 111221468A CN 201811419430 A CN201811419430 A CN 201811419430A CN 111221468 A CN111221468 A CN 111221468A
- Authority
- CN
- China
- Prior art keywords
- storage
- time
- block
- storage block
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0623—Securing storage systems in relation to content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0652—Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a method and a device for deleting storage block data, electronic equipment and a cloud storage system, wherein the method comprises the following steps: acquiring target time and the last block writing time of each storage block; determining a target storage block with the last writing time of the block being less than the target time, and deleting data in the target storage block; acquiring the last writing time of the video segment in each storage block; judging the size of the last writing time and the target time of each video segment; and deleting the video segment with the last writing time less than the target time. In the method for deleting the data of the storage block, when the storage space of the resource pool is cleared, the video segment with the last writing time smaller than the target time is deleted, and the video segment is used as the minimum detection unit, so that the release of the storage space in the resource pool can be increased, and the effective utilization rate of the storage resource is improved. Meanwhile, the time span of data in the same storage block can be reduced, and the time updating of the resource pool is facilitated.
Description
Technical Field
The application relates to the technical field of data storage, in particular to a method and a device for deleting data of storage blocks, electronic equipment and a cloud storage system.
Background
In the existing cloud storage system, storage resources are managed by a virtualized resource pool, and the overall architecture of the existing cloud storage system is shown in fig. 1. The cloud storage system comprises a management node and storage nodes, wherein the storage nodes are physical media for storing data, block devices on the storage nodes are divided into storage blocks according to fixed sizes, and the management node logically manages the storage blocks on the storage nodes through a virtual storage resource pool.
The resource pool is divided into a covering pool and a non-covering pool according to the user requirement. The covering pool can carry out space reclamation according to the strategy. Specifically, in the overlay pool, the data in the storage block before the specified target time at the last write time of the storage block is cleared by taking the storage block as a unit to release the storage space.
However, by adopting the method, the release of the storage space in the resource pool is small, and the effective utilization rate of the storage resources is low.
Disclosure of Invention
An object of the embodiments of the present application is to provide a method and an apparatus for deleting storage block data, an electronic device, and a cloud storage system, so as to increase the effective utilization rate of storage resources. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present application provides a method for deleting storage block data, where the method includes:
acquiring target time and the last block writing time of each storage block;
determining a target storage block with the last writing time of the block being less than the target time, and deleting data in the target storage block;
acquiring the last writing time of the video segment in each storage block;
judging the size of the last writing time and the target time of each video segment;
and deleting the video segment with the last writing time smaller than the target time.
Optionally, the obtaining the target time and the last block writing time of each storage block includes:
and when the preset cleaning time is reached, acquiring the target time and the block last writing time of each storage block.
Optionally, the obtaining the last writing time of the video segment in each storage block includes:
and acquiring the last writing time of the video segments in other storage blocks except the target storage block.
Optionally, after deleting the video segment whose last writing time is less than the target time, the method further includes:
detecting the storage space utilization rate of each storage block to be sorted, wherein the storage blocks to be sorted are the storage blocks of deleted video segments in each storage block;
judging the storage space utilization rate of each storage block to be sorted and the size of a preset storage threshold value:
and performing defragmentation on the storage blocks to be defragmented, wherein the utilization rate of the storage space is less than a preset storage threshold value.
In a second aspect, an embodiment of the present application provides a storage block data deletion apparatus, where the apparatus includes:
the first time acquisition module is used for acquiring target time and the last block writing time of each storage block;
the first data deleting module is used for determining a target storage block of which the last writing time of the block is less than the target time and deleting data in the target storage block;
a second time obtaining module, configured to obtain a last writing time of the video segment of the video segments in each storage block;
the time judgment module is used for judging the size of the last writing time and the target time of each video segment;
and the second data deleting module is used for deleting the video segment of which the last writing time is less than the target time.
Optionally, the first time obtaining module is specifically configured to:
and when the preset cleaning time is reached, acquiring the target time and the block last writing time of each storage block.
Optionally, the second time obtaining module is specifically configured to:
and acquiring the last writing time of the video segments in other storage blocks except the target storage block.
Optionally, the device for deleting the storage block data according to the embodiment of the present application further includes:
the usage rate detection module is used for detecting the usage rate of storage space of each storage block to be sorted, wherein the storage blocks to be sorted are the storage blocks of deleted video segments in each storage block;
and the threshold value judging module is used for judging the storage space utilization rate of each storage block to be sorted and the size of a preset storage threshold value:
and the defragmentation module is used for defragmenting the storage blocks to be defragmented, wherein the utilization rate of the storage space is less than a preset storage threshold value.
In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to implement the method for deleting block data according to any one of the first aspect described above when executing the program stored in the memory.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program implements the method for deleting storage block data according to any one of the first aspect.
In a fifth aspect, an embodiment of the present application provides a cloud storage system, where the cloud storage system includes: the system comprises a management node and a storage node, wherein the storage node comprises a storage block;
the storage block is used for storing data;
the management node is configured to implement, for the storage block, the storage block data deletion method according to any one of the first aspect.
According to the method and the device for deleting the data of the storage blocks, the electronic equipment and the cloud storage system, the target time and the last block writing time of each storage block are obtained; determining a target storage block with the last writing time of the block being less than the target time, and deleting data in the target storage block; acquiring the last writing time of the video segment in each storage block; judging the size of the last writing time and the target time of each video segment; and deleting the video segment with the last writing time less than the target time. When the storage space of the resource pool is cleared, the video segment with the last writing time smaller than the target time is deleted, and the video segment is used as the minimum detection unit, so that the release of the storage space in the resource pool can be increased, and the effective utilization rate of the storage resources is improved. Meanwhile, the time span of data in the same storage block can be reduced, and the time updating of the resource pool is facilitated. Of course, not all advantages described above need to be achieved at the same time in the practice of any one product or method of the present application.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic diagram of an application scenario of a storage block data deletion method according to an embodiment of the present application;
FIG. 2 is a diagram illustrating a writing time of a video segment in a storage block according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating a method for deleting storage block data according to an embodiment of the present application;
fig. 4 is another schematic flow chart of a storage block data deletion method according to an embodiment of the present application;
FIG. 5 is a diagram illustrating an apparatus for deleting storage block data according to an embodiment of the present application;
FIG. 6 is a schematic diagram of an electronic device according to an embodiment of the present application;
fig. 7 is a schematic diagram of a cloud storage system according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms in the examples of the present application are explained first:
video cloud storage: in the security industry, video data have the requirements of continuous storage according to a time sequence and cyclic coverage according to a certain period or capacity, and the traditional file storage mode has certain difference on the video streaming storage characteristics. The video cloud storage data storage mode enables video data storage to better accord with the characteristics of streaming data storage by simplifying a storage flow and optimizing a data block structure, so that the read-write efficiency and later-stage application expansion are optimized, and the video cloud storage data storage mode can better serve security application.
CVM (Cloud Virtual Machine, Cloud server): management node of cloud storage.
CVS (Cloud Video Storage ): storage nodes of cloud storage.
Resource pool: a virtually partitioned logical storage resource.
A storage block: the minimum unit of space allocation management on a storage node.
A hollow block: there are more than two memory blocks with a large difference in time between segments.
In the existing cloud storage system, storage resources are managed by a virtualized resource pool, and the overall architecture of the existing cloud storage system is shown in fig. 1. The cloud storage system comprises a management node and storage nodes, wherein the storage nodes are physical media for storing data, block devices on the storage nodes are divided into storage blocks according to fixed sizes, and the management node logically manages the storage blocks on the storage nodes through a virtual storage resource pool. The resource pool records the number of all storage blocks used in the pool, the minimum time is the starting time of the resource pool, the maximum time is the ending time of the resource pool, and the coverage time related to the strategy.
The resource pool is divided into a covering pool and a non-covering pool according to the user requirement. The data which the user desires to permanently store is stored in the non-coverage pool, the data in the pool cannot be deleted, and the coverage time of the non-coverage pool is always 0. And the covering pool is a pool for space recovery and resource reuse according to the strategy, and the covering time is the target time in the strategy executed at the last time. The coverage pool is divided into a period pool and a capacity pool according to a strategy. The period pool corresponds to a storage period, the time for storing the data stored in the storage block is checked at regular time, and if the storage period is met, the storage block is triggered to be cleaned and the data is deleted; and the capacity covering pool corresponds to a total storage space and a capacity threshold, if the used storage block reaches the capacity threshold, a target time for cleaning the data is determined according to the data time stored in the resource pool, and the storage block with the data storage time before the target time is subjected to data cleaning and data deletion.
In a conventional method for deleting data of a storage block, when data is cleared, data is deleted in units of data blocks. The following description will be made by taking fig. 2 as an example.
When the periodic resource pool and the capacity resource pool are used for space cleaning, the final cleaning instruction is converted into data before a certain target time T. When the current cleaning target time is T, and T4< T < T5, the storage node will find out the execution block deletion in the storage blocks before T according to the end time of the storage block, i.e. the last writing time of the storage block. The end time T2< T of memory block 1, the data in memory block 1 will be deleted, freeing up space. The end time of the storage block 2 and the end time of the storage block 3 are respectively T6 and T8 which are both larger than T, and the storage blocks cannot be inquired out and deleted during cleaning. The space of memory block 1 is released and the start time of the resource pool is updated to T3.
The following steps are known through the above process: there are two small segments of video data in storage block 2: when the time difference between the two segments is relatively large, the two segments are called a hole block, and the videos in the time segments of [ T3 and T4] in the block are overdue and need to be deleted. However, since the search block is queried according to the storage block when the block is deleted, the block deletion is executed only when the end time of the storage block expires. Therefore, the expired data is always stored in the storage node, the starting time of the resource pool can only be updated to T3, and the estimation of the coverage time of the capacity coverage is also influenced by the fact that the starting time cannot be updated normally.
Such a video segmentation scenario in the storage block 2 is also common, for example, in the process of allocating one storage block to write data, some exceptions (network, power off, etc.) occur, after half of the data is written, the video is written on another storage node, and after a period of time, the block of the node is reallocated, and the video is continuously written, so that two blocks with a large time span are obtained. In extreme cases, when a user stops writing or recording or closes a certain storage node and continues to use the storage node for a long time, the block time span is very large, and may exceed one or two months.
Due to the existence of such a block, the starting time of the resource pool and even the coverage time cannot be updated, which may have a significant influence on the coverage policy of the management node for the resource pool, so that the calculation coverage time point in the coverage policy is invalid due to the missing basis, the space of the resource pool cannot be released, and the loss of data will be caused by the absence of space write data by the user.
In view of this, an embodiment of the present application provides a method for deleting storage block data, and referring to fig. 3, the method includes:
s301, acquiring target time and the last block writing time of each storage block.
The method for deleting the stored block data in the embodiment of the application can be realized by a storage system, and the storage system is any system capable of realizing the method for deleting the stored block data in the embodiment of the application. For example:
the storage system may be an apparatus comprising: a processor, a memory, a communication interface, and a bus; the processor, the memory and the communication interface are connected through a bus and complete mutual communication; the memory stores executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to execute the storage block data deletion method according to the embodiment of the present application.
The storage system may also be an application program configured to execute the storage block data deletion method according to the embodiment of the present application at runtime.
The storage system may also be a storage medium for storing executable code for performing the storage block data deletion method according to the embodiment of the present application.
And when the preset covering condition is met, the storage system acquires the target moment. The preset coverage condition may be set according to an actual situation, for example, when the preset coverage period is reached, it is determined that the preset coverage condition is satisfied; or when the occupancy rate of the storage space reaches a preset occupancy rate threshold value, judging that a preset coverage condition is met, and the like. The target time is a specified data retention point, that is, a video segment before the target time needs to be deleted, and a target video segment after the target time needs to be retained, and the target time can be set according to actual requirements. For example, in the security field, the time required to store the video segment is generally one month, and when the clearing time is set to be zero per day, the target time may be zero one month before the current clearing time. The block last write time of the memory block, i.e. the end time of the memory block, taking fig. 2 as an example, the block last write times of the memory block 1, the memory block 2 and the memory block 3 are sequentially: t2, T6 and T8.
Optionally, the obtaining the target time and the last block writing time of each storage block includes: and when the preset cleaning time is reached, acquiring the target time and the block last writing time of each storage block. For example, when the resource pool is a periodic pool, the storage period corresponding to the resource pool is acquired, and the storage space of each storage block in the resource pool is released at regular time. And (4) with the current check time as the reference time, subtracting the duration of the storage period on the basis of the reference time to obtain the target time.
Optionally, when the resource pool is a capacity coverage pool, if the total capacity of the used storage blocks reaches a capacity threshold, a target time for cleaning data is determined according to the data time stored in the resource pool.
S302, determining a target storage block with the block last writing time smaller than the target time, and deleting the data in the target storage block.
The storage system judges the sizes of the block last writing time and the target time of each storage block, and takes the storage block of which the block last writing time is less than the target time as the target storage block. The storage system deletes the data in each target storage block. When there are many video segments in the resource pool, the processing directly using the video segments as the basic unit will excessively consume the performance of the system and affect the processing efficiency, so the data deletion can be performed by using the storage block as the basic unit first, and then the data deletion is performed by using the video segments as the basic unit, thereby increasing the processing efficiency.
S303, acquiring the last writing time of the video segment of the video segments in each storage block.
A storage block may contain a plurality of video segments, and the storage system may traverse each storage block to obtain the last write time of the video segments of each video segment in each storage block. For example, the storage block 2 shown in fig. 2 includes two recorded segments, and the recorded segments of the two recorded segments are written at the time T4 and T6 in this order.
Optionally, the obtaining of the last writing time of the video segment in each storage block includes:
and acquiring the last writing time of the video segments in other storage blocks except the target storage block.
The data in the target storage block has been deleted and therefore does not include a recorded segment. Therefore, the last write time of the recorded segment can be obtained only in other storage blocks except the target storage block without detecting the target storage block. In the embodiment of the application, the storage system detects the last writing time of the video segments of all the video segments in other storage blocks except the target storage block, so that the time for detecting the target storage block is saved, and the processing efficiency can be further increased.
S304, judging the size of the last writing time of each video segment and the target time.
The storage system judges the size of the last writing time and the target time of the video segment of each video segment in each storage block.
S305, deleting the video segment with the last writing time of the video segment less than the target time.
Aiming at the video segment with the last writing time of the deleted video segment not less than the target time, the storage system does not process the video segment; and the storage system deletes the video segment aiming at the video segment with the last writing time of the deleted video segment being less than the target time. Optionally, after deleting the video segment whose last writing time of the video segment is less than the target time, the method further includes: and updating the starting time of the resource pool in which each storage block is positioned to be the target time.
In the embodiment of the application, when the storage space of the resource pool is cleared, the video segment with the last writing time smaller than the target time is deleted, and the video segment is taken as the minimum judgment unit, so that the release of the storage space in the resource pool can be increased, and the effective utilization rate of the storage resource is improved. The time span of data in the same storage block is reduced, and the time updating of the resource pool is facilitated.
Optionally, referring to fig. 4, after deleting the recorded segment whose last writing time is less than the target time, the method further includes:
s306, detecting the utilization rate of the storage space of each storage block to be sorted, wherein the storage block to be sorted is a storage block of the deleted video segment in each storage block.
And taking the storage block with the video segment deleted as a storage block to be sorted. And the storage system detects the storage space utilization rate of each storage block to be sorted.
And S307, judging the storage space utilization rate of each storage block to be sorted and the size of a preset storage threshold.
The preset storage threshold is set according to actual requirements, for example, the preset storage threshold is set to be 5%, 10%, 20%, or the like.
And S308, performing defragmentation on the storage blocks to be defragmented, wherein the utilization rate of the storage space is less than a preset storage threshold value.
If the utilization rate of the storage space of the storage blocks to be sorted is smaller than a preset storage threshold value, a defragmentation task is generated, the background asynchronously performs data migration on the storage blocks to be sorted, the utilization rate of the storage space of which is smaller than the preset storage threshold value, the data are migrated to other storage blocks, the data in the original storage blocks are deleted, and space resources are released.
In the embodiment of the application, the storage blocks to be sorted, of which the utilization rate of the storage space is smaller than the preset storage threshold value, are subjected to defragmentation, so that data management can be facilitated, more storage spaces can be released, and the utilization rate of storage resources is improved.
In the following, taking fig. 2 as an example, a storage block data deletion method according to an embodiment of the present application is described as an example:
when the CVS executes the covering strategy, the CVS firstly searches according to the storage blocks, inquires the blocks of which the end time of the storage blocks is before the covering target time T to execute deletion, the end time T2< T of the storage block 1, and deletes the data in the storage blocks which meet the conditions when the blocks are deleted.
Then, video segment searching is executed, video segments before the target covering moment are searched, and video segment deleting is executed, at this time, because block searching and deleting are executed, the searched video segments are not too many (segment searching and deleting are directly executed, the searched segments are possibly too many, and the performance is influenced), the video segments [ T3, T4] in the storage block 2 are searched and deleted, the block starting time of the storage block 2 is also updated to T5, the starting time of the corresponding resource pool is also successfully updated to T5, and the subsequent updating and operation maintenance of the resource pool can also be normally carried out.
If the execution segment is deleted successfully, the proportion of the effective video segment in the current storage block to the total capacity of the storage block is checked, if the effective video segment is smaller than a preset storage threshold value, a defragmentation task is generated, a background asynchronously carries out data migration on the defragmentation block and migrates the defragmentation block to a new block, and the original defragmentation block executes block deletion to release space resources.
In the embodiment of the application, when the storage space of the resource pool is cleared, the video segment with the last writing time smaller than the target time is deleted, and the video segment is taken as a basic unit, so that the release of the storage space in the resource pool can be increased, and the effective utilization rate of the storage resource is improved. The time span of data in the same storage block is reduced, and the time updating of the resource pool is facilitated.
An embodiment of the present application provides a storage block data deleting apparatus, referring to fig. 5, the apparatus includes:
a first time obtaining module 501, configured to obtain a target time and a last block writing time of each storage block;
a first data deleting module 502, configured to determine a target storage block whose block last writing time is less than the target time, and delete data in the target storage block;
a second time obtaining module 503, configured to obtain a last writing time of the video segment of the video segments in each storage block;
a time judgment module 504, configured to judge the last writing time of each video segment and the target time;
and a second data deleting module 505, configured to delete the video segment with the last writing time being less than the target time.
Optionally, in the device for deleting block data stored in the embodiment of the present application, the first time obtaining module is specifically configured to:
and when the preset cleaning time is reached, acquiring the target time and the block last writing time of each storage block.
Optionally, the second time obtaining module 503 is specifically configured to:
and acquiring the last writing time of the video segments in other storage blocks except the target storage block.
Optionally, the device for deleting the storage block data according to the embodiment of the present application further includes:
the usage rate detection module is used for detecting the usage rate of storage spaces of the storage blocks to be sorted, wherein the storage blocks to be sorted are the storage blocks of deleted video segments in the storage blocks;
and the threshold value judging module is used for judging the storage space utilization rate of each storage block to be sorted and the size of a preset storage threshold value:
and the defragmentation module is used for defragmenting the storage blocks to be defragmented, wherein the utilization rate of the storage space is less than a preset storage threshold value.
In the embodiment of the application, when the storage space of the resource pool is cleared, the video segment with the last writing time smaller than the target time is deleted, and the video segment is taken as a basic unit, so that the release of the storage space in the resource pool can be increased, and the effective utilization rate of the storage resource is improved. The time span of data in the same storage block is reduced, and the time updating of the resource pool is facilitated.
An electronic device according to an embodiment of the present application is provided, and referring to fig. 6, the electronic device includes a processor 601 and a memory 602;
the memory 602 is used for storing computer programs;
the processor 601 is configured to implement the following steps when executing the program stored in the memory 602:
acquiring target time and the last block writing time of each storage block;
determining a target storage block with the last writing time smaller than the target time, and deleting data in the target storage block;
acquiring the last writing time of the video segment in each storage block;
judging the size of the last writing time and the target time of each video segment;
and deleting the video segment with the last writing time smaller than the target time.
In the embodiment of the application, when the storage space of the resource pool is cleared, the video segment with the last writing time smaller than the target time is deleted, and the video segment is taken as a basic unit, so that the release of the storage space in the resource pool can be increased, and the effective utilization rate of the storage resource is improved. The time span of data in the same storage block is reduced, and the time updating of the resource pool is facilitated.
Optionally, the processor 601 is configured to implement any of the above methods for deleting block data when executing the program stored in the memory 602.
Optionally, the electronic device according to the embodiment of the present application further includes a communication interface and a communication bus, where the processor 601, the communication interface, and the memory 602 complete mutual communication through the communication bus.
Optionally, the electronic device according to the embodiment of the present application may be a CVM.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.
An embodiment of the present application provides a cloud storage system, and referring to fig. 7, the cloud storage system includes: a management node 701 and a storage node 702, where the storage node 702 includes a storage block;
the storage block is used for storing data;
the management node 701 is configured to implement any storage block data deletion method for the storage block.
Alternatively, the management node 701 may be a CVM and the storage node 702 may be a CVS. Management node 701 may also logically divide storage node 702 into a plurality of resource pools, as shown in FIG. 1.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the following steps:
acquiring target time and the last block writing time of each storage block;
determining a target storage block with the last writing time smaller than the target time, and deleting data in the target storage block;
acquiring the last writing time of the video segment in each storage block;
judging the size of the last writing time and the target time of each video segment;
and deleting the video segment with the last writing time smaller than the target time.
In the embodiment of the application, when the storage space of the resource pool is cleared, the video segment with the last writing time smaller than the target time is deleted, and the video segment is taken as a basic unit, so that the release of the storage space in the resource pool can be increased, and the effective utilization rate of the storage resource is improved. The time span of data in the same storage block is reduced, and the time updating of the resource pool is facilitated.
Optionally, when being executed by a processor, the computer program can also implement any of the above methods for deleting storage block data.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the embodiments of the apparatus, the electronic device, the cloud storage system and the storage medium, since they are substantially similar to the method embodiments, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiments.
The above description is only for the preferred embodiment of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims (11)
1. A method for deleting storage block data, the method comprising:
acquiring target time and the last block writing time of each storage block;
determining a target storage block with the last writing time of the block being less than the target time, and deleting data in the target storage block;
acquiring the last writing time of the video segment in each storage block;
judging the size of the last writing time and the target time of each video segment;
and deleting the video segment with the last writing time smaller than the target time.
2. The method of claim 1, wherein the obtaining the target time and the last block writing time of each storage block comprises:
and when the preset cleaning time is reached, acquiring the target time and the block last writing time of each storage block.
3. The method of claim 1, wherein said obtaining a last write time of a recorded segment of recorded segments in each of said storage blocks comprises:
and acquiring the last writing time of the video segments in other storage blocks except the target storage block.
4. The method of claim 1, wherein after deleting the recorded segment having a recorded segment last written time less than the target time, the method further comprises:
detecting the storage space utilization rate of each storage block to be sorted, wherein the storage blocks to be sorted are the storage blocks of deleted video segments in each storage block;
judging the storage space utilization rate of each storage block to be sorted and the size of a preset storage threshold value:
and performing defragmentation on the storage blocks to be defragmented, wherein the utilization rate of the storage space is less than a preset storage threshold value.
5. An apparatus for deleting storage block data, the apparatus comprising:
the first time acquisition module is used for acquiring target time and the last block writing time of each storage block;
the first data deleting module is used for determining a target storage block of which the last writing time of the block is less than the target time and deleting data in the target storage block;
a second time obtaining module, configured to obtain a last writing time of the video segment of the video segments in each storage block;
the time judgment module is used for judging the size of the last writing time and the target time of each video segment;
and the second data deleting module is used for deleting the video segment of which the last writing time is less than the target time.
6. The apparatus of claim 5, wherein the first time acquisition module is specifically configured to:
and when the preset cleaning time is reached, acquiring the target time and the block last writing time of each storage block.
7. The apparatus of claim 5, wherein the second time obtaining module is specifically configured to:
and acquiring the last writing time of the video segments in other storage blocks except the target storage block.
8. The apparatus of claim 5, further comprising:
the usage rate detection module is used for detecting the usage rate of storage space of each storage block to be sorted, wherein the storage blocks to be sorted are the storage blocks of deleted video segments in each storage block;
and the threshold value judging module is used for judging the storage space utilization rate of each storage block to be sorted and the size of a preset storage threshold value:
and the defragmentation module is used for defragmenting the storage blocks to be defragmented, wherein the utilization rate of the storage space is less than a preset storage threshold value.
9. An electronic device comprising a processor and a memory;
the memory is used for storing a computer program;
the processor, when executing the program stored in the memory, implementing the method steps of any of claims 1-4.
10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 4.
11. A cloud storage system, the cloud storage system comprising: the system comprises a management node and a storage node, wherein the storage node comprises a storage block;
the storage block is used for storing data;
the management node is configured to implement the storage block data deletion method according to any one of claims 1 to 4 for the storage block.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811419430.1A CN111221468B (en) | 2018-11-26 | 2018-11-26 | Storage block data deleting method and device, electronic equipment and cloud storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811419430.1A CN111221468B (en) | 2018-11-26 | 2018-11-26 | Storage block data deleting method and device, electronic equipment and cloud storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111221468A true CN111221468A (en) | 2020-06-02 |
| CN111221468B CN111221468B (en) | 2023-10-13 |
Family
ID=70827089
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811419430.1A Active CN111221468B (en) | 2018-11-26 | 2018-11-26 | Storage block data deleting method and device, electronic equipment and cloud storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111221468B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112463058A (en) * | 2020-11-27 | 2021-03-09 | 杭州海康威视系统技术有限公司 | Fragmented data sorting method and device and storage node |
| CN116521094A (en) * | 2023-07-03 | 2023-08-01 | 之江实验室 | Metadata storage method and device, computer equipment and storage medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080195799A1 (en) * | 2007-02-13 | 2008-08-14 | Samsung Electronics Co., Ltd. | Systems, methods and computer program products for operating a data processing system in which a file delete command is sent to an external storage device for invalidating data thereon |
| CN101819603A (en) * | 2010-05-07 | 2010-09-01 | 中兴通讯股份有限公司 | Embedded system based method and device for searching information record based on |
| US20150088838A1 (en) * | 2013-09-25 | 2015-03-26 | International Business Machines Corporation | Data storage device deferred secure delete |
| US20160246799A1 (en) * | 2015-02-20 | 2016-08-25 | International Business Machines Corporation | Policy-based, multi-scheme data reduction for computer memory |
| US20160350296A1 (en) * | 2013-12-19 | 2016-12-01 | Shell Internet (Beijing) Security Technology Co., Ltd. | Method, device, and mobile terminal for cleaning up temporary files in a mobile terminal |
| CN106326439A (en) * | 2016-08-26 | 2017-01-11 | 东方网力科技股份有限公司 | Real-time video storage and retrieval method and device |
| CN106469120A (en) * | 2016-08-30 | 2017-03-01 | 华为技术有限公司 | Scrap cleaning method, device and equipment |
| CN107092437A (en) * | 2016-02-17 | 2017-08-25 | 杭州海康威视数字技术股份有限公司 | Data write-in, read method and device, cloud storage system |
| CN107678684A (en) * | 2017-08-22 | 2018-02-09 | 深圳市硅格半导体股份有限公司 | Invalid data sweep-out method, device and the memory of memory |
| US20180081894A1 (en) * | 2016-09-22 | 2018-03-22 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Method and apparatus for clearing data in cloud storage system |
| CN107967117A (en) * | 2016-10-20 | 2018-04-27 | 杭州海康威视数字技术股份有限公司 | A kind of data storage, reading, method for cleaning, device and cloud storage system |
| CN108153594A (en) * | 2017-12-25 | 2018-06-12 | 联想(北京)有限公司 | The resource fragmentation method for sorting and electronic equipment of a kind of artificial intelligence cloud platform |
-
2018
- 2018-11-26 CN CN201811419430.1A patent/CN111221468B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080195799A1 (en) * | 2007-02-13 | 2008-08-14 | Samsung Electronics Co., Ltd. | Systems, methods and computer program products for operating a data processing system in which a file delete command is sent to an external storage device for invalidating data thereon |
| CN101819603A (en) * | 2010-05-07 | 2010-09-01 | 中兴通讯股份有限公司 | Embedded system based method and device for searching information record based on |
| US20150088838A1 (en) * | 2013-09-25 | 2015-03-26 | International Business Machines Corporation | Data storage device deferred secure delete |
| US20160350296A1 (en) * | 2013-12-19 | 2016-12-01 | Shell Internet (Beijing) Security Technology Co., Ltd. | Method, device, and mobile terminal for cleaning up temporary files in a mobile terminal |
| US20160246799A1 (en) * | 2015-02-20 | 2016-08-25 | International Business Machines Corporation | Policy-based, multi-scheme data reduction for computer memory |
| CN107092437A (en) * | 2016-02-17 | 2017-08-25 | 杭州海康威视数字技术股份有限公司 | Data write-in, read method and device, cloud storage system |
| CN106326439A (en) * | 2016-08-26 | 2017-01-11 | 东方网力科技股份有限公司 | Real-time video storage and retrieval method and device |
| CN106469120A (en) * | 2016-08-30 | 2017-03-01 | 华为技术有限公司 | Scrap cleaning method, device and equipment |
| US20180081894A1 (en) * | 2016-09-22 | 2018-03-22 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Method and apparatus for clearing data in cloud storage system |
| CN107967117A (en) * | 2016-10-20 | 2018-04-27 | 杭州海康威视数字技术股份有限公司 | A kind of data storage, reading, method for cleaning, device and cloud storage system |
| CN107678684A (en) * | 2017-08-22 | 2018-02-09 | 深圳市硅格半导体股份有限公司 | Invalid data sweep-out method, device and the memory of memory |
| CN108153594A (en) * | 2017-12-25 | 2018-06-12 | 联想(北京)有限公司 | The resource fragmentation method for sorting and electronic equipment of a kind of artificial intelligence cloud platform |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112463058A (en) * | 2020-11-27 | 2021-03-09 | 杭州海康威视系统技术有限公司 | Fragmented data sorting method and device and storage node |
| CN116521094A (en) * | 2023-07-03 | 2023-08-01 | 之江实验室 | Metadata storage method and device, computer equipment and storage medium |
| CN116521094B (en) * | 2023-07-03 | 2023-11-14 | 之江实验室 | Metadata storage method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111221468B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111399762B (en) | Data storage method, device and storage system | |
| JP6014925B2 (en) | Memory recovery method and apparatus | |
| CN107729558B (en) | Method, system and device for defragmenting file system and computer storage medium | |
| CN107818269B (en) | File storage method and system based on Flash | |
| CN109428913B (en) | Storage capacity expansion method and device | |
| EP3252609A1 (en) | Cache data determination method and device | |
| CN107122130B (en) | Data deduplication method and device | |
| CN106294206B (en) | Cache data processing method and device | |
| CN111061752B (en) | Data processing method and device and electronic equipment | |
| CN108153594B (en) | Resource fragment sorting method of artificial intelligence cloud platform and electronic equipment | |
| CN112463058B (en) | Fragmented data sorting method and device and storage node | |
| CN112631953A (en) | TRIM method and device for solid state disk data, electronic equipment and storage medium | |
| CN105376269B (en) | Virtual machine storage system and its implementation and device | |
| CN111221468A (en) | Storage block data deleting method and device, electronic equipment and cloud storage system | |
| CN106598508A (en) | Solid-state hard disc and write-in arbitrating method and system thereof | |
| CN112306408A (en) | Storage block processing method, device, equipment and storage medium | |
| CN114840449A (en) | Data storage method, device, equipment and storage medium based on MCU chip flash | |
| CN109189739B (en) | Cache space recovery method and device | |
| TW202242664A (en) | Method of performing garbage collection with partial clean operation and related controller and storage system | |
| CN110618946A (en) | Stack memory allocation method, device, equipment and storage medium | |
| CN114595066A (en) | Reserved memory processing method and device, electronic equipment and medium | |
| EP3296878A1 (en) | Electronic device and page merging method therefor | |
| CN106959888B (en) | Task processing method and device in cloud storage system | |
| CN114237489A (en) | Method and device for writing logical resources into SMR disk, electronic equipment and storage medium | |
| CN111143288A (en) | Data storage method, system and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |