CN112181277B - Data storage space management method and device, storage medium and electronic equipment - Google Patents
Data storage space management method and device, storage medium and electronic equipment Download PDFInfo
- Publication number
- CN112181277B CN112181277B CN201910596063.0A CN201910596063A CN112181277B CN 112181277 B CN112181277 B CN 112181277B CN 201910596063 A CN201910596063 A CN 201910596063A CN 112181277 B CN112181277 B CN 112181277B
- Authority
- CN
- China
- Prior art keywords
- trigger condition
- data
- data storage
- capacity
- deleted
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 135
- 238000007726 management method Methods 0.000 title claims abstract description 53
- 238000000034 method Methods 0.000 claims abstract description 47
- 230000004913 activation Effects 0.000 claims description 9
- 230000007246 mechanism Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 239000003550 marker Substances 0.000 claims description 7
- 230000003213 activating effect Effects 0.000 claims description 5
- 230000002035 prolonged effect Effects 0.000 abstract description 5
- 239000002609 medium Substances 0.000 description 13
- 238000012544 monitoring process Methods 0.000 description 13
- 238000010586 diagram Methods 0.000 description 12
- 238000012217 deletion Methods 0.000 description 11
- 230000037430 deletion Effects 0.000 description 11
- 230000008569 process Effects 0.000 description 9
- 241000306271 Colombian datura virus Species 0.000 description 7
- 238000012545 processing Methods 0.000 description 5
- 238000012163 sequencing technique Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000001960 triggered effect Effects 0.000 description 4
- 230000014759 maintenance of location Effects 0.000 description 3
- UHLRPXXFPYMCAE-UHFFFAOYSA-N 4-isopropylcalix[4]arene Chemical compound C1C(C=2O)=CC(C(C)C)=CC=2CC(C=2O)=CC(C(C)C)=CC=2CC(C=2O)=CC(C(C)C)=CC=2CC2=CC(C(C)C)=CC1=C2O UHLRPXXFPYMCAE-UHFFFAOYSA-N 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000005236 sound signal Effects 0.000 description 2
- 102100022256 Clustered mitochondria protein homolog Human genes 0.000 description 1
- 101000902167 Homo sapiens Clustered mitochondria protein homolog Proteins 0.000 description 1
- 101000942697 Homo sapiens Clusterin Proteins 0.000 description 1
- 101001010731 Homo sapiens Intraflagellar transport protein 81 homolog Proteins 0.000 description 1
- 102100030001 Intraflagellar transport protein 81 homolog Human genes 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000009533 lab test Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 239000012120 mounting media Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data storage space management method and device, a storage medium and electronic equipment. The method comprises the following steps: if the trigger condition of the update capacity of the target data storage interval is detected, determining whether the trigger condition simultaneously comprises a first trigger condition and a second trigger condition; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule; and if the data to be deleted simultaneously comprises the first trigger condition and the second trigger condition, determining data to be deleted from all data in the target data storage interval based on a preset rule, and deleting the data to be deleted. By executing the technical scheme, the storage data can be effectively managed according to the requirement, and the storage capacity of the storage equipment for the newly-coming data is not influenced while the storage time of the data is prolonged.
Description
Technical Field
The embodiment of the application relates to the technical field of data processing, in particular to a data storage space management method and device, a storage medium and an electronic device.
Background
With the rapid development of science and technology, the data storage capacity is increasingly enhanced, and the storage device is used for storing a large amount of data for a long time so as to achieve the purpose of calling the data in the using process.
For the retention of video monitoring data, the data plays a crucial role. From the current trend, the retention time of video surveillance data and the number of cameras accessed will continue to increase, and storage will also face the need for access to a greater variety of metadata and greater capacity storage. However, the storage capacity cannot be expanded without limit, so that the orderly update and deletion of a large amount of data in a certain amount of storage space is particularly important, which requires that the storage must be more rigorous and scientific for the management of metadata. Therefore, how to utilize the characteristics of the video monitoring data and combine the storage management technology to stably and orderly update and clean the video monitoring data and improve the resistance of the data storage space management to known and unknown risks is a problem to be solved currently.
Disclosure of Invention
The embodiment of the application provides a data storage space management method and device, a storage medium and an electronic device, which can realize effective management of stored data according to requirements, and do not influence the storage capacity of the storage device on new data while increasing the storage time of the data.
In a first aspect, an embodiment of the present application provides a data storage space management method, where the method includes:
if the trigger condition of the update capacity of the target data storage interval is detected, determining whether the trigger condition comprises a first trigger condition and a second trigger condition at the same time; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule;
and if the data to be deleted simultaneously comprises the first trigger condition and the second trigger condition, determining data to be deleted from all data in the target data storage interval based on a preset rule, and deleting the data to be deleted.
Further, the obtaining manner of the first trigger condition includes:
determining whether a first preset condition is met or not according to the size relation between the residual capacity space of the target data storage interval and the capacity requirement of the data to be stored;
and if so, generating a first trigger condition.
Further, the obtaining manner of the second trigger condition includes:
determining a capacity timer of a target data storage interval according to the residual capacity space of the target data storage interval and the data storage rate of the target data storage interval;
and when the timing of the capacity timer meets a second preset condition, generating a second trigger condition.
Further, after detecting the trigger condition of the update capacity of the target data storage interval, the method further comprises:
if the triggering condition is determined to comprise a first triggering condition, and the target data storage interval reaches the first triggering condition for the first time, updating the capacity of the data storage space based on the storage time of each data in the target data storage interval, and activating a time triggering mechanism to work so as to generate a second triggering condition based on a preset time rule after activation.
Further, before detecting a trigger condition for updating the capacity of the target data storage interval, the method further includes:
storing data to be stored in a target data storage interval; the data to be stored comprises a mark field, and the mark field is used for storing a mark numerical value;
correspondingly, determining data to be deleted from each data in the target data storage interval based on a preset rule, including:
and determining the data to be deleted based on the marking value of the marking field of each data.
Further, before determining the data to be deleted based on the tag value of the tag field of each data, the method further includes:
generating a mark numerical value recording table;
correspondingly, the step of determining the data to be deleted based on the tag value of the tag field of each piece of data comprises the following steps:
and determining data to be deleted based on the marker values in the record table, and updating the record table after deleting the data.
Further, after generating the tag value record table, the method further includes: determining whether the first trigger condition is valid according to the number of the marked numerical values in the record table; if the number exceeds the threshold of quota number, determining that the first trigger condition is valid, and if the number does not exceed the threshold of quota number, determining that the first trigger condition is invalid.
In a second aspect, an embodiment of the present application provides an apparatus for managing a data storage space, where the apparatus includes:
the trigger condition judging module is used for determining whether the trigger conditions comprise a first trigger condition and a second trigger condition at the same time if the trigger conditions of the update capacity of the target data storage interval are detected; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule;
and the data deleting module is used for determining data to be deleted from all data in the target data storage interval based on a preset rule and deleting the data to be deleted if the triggering condition judging module judges that the data to be deleted simultaneously comprises the first triggering condition and the second triggering condition.
Further, the obtaining manner of the first trigger condition includes:
determining whether a first preset condition is met or not according to the size relation between the residual capacity space of the target data storage interval and the capacity requirement of the data to be stored;
and if so, generating a first trigger condition.
Further, the obtaining manner of the second trigger condition includes:
determining a capacity timer of a target data storage interval according to the residual capacity space of the target data storage interval and the data storage rate of the target data storage interval;
and when the timing of the capacity timer meets a second preset condition, generating a second trigger condition.
Further, the apparatus further comprises:
and the second trigger condition activation module is used for updating the capacity of the data storage space based on the storage time of each data in the target data storage interval and activating a time trigger mechanism to work so as to generate a second trigger condition based on a preset time rule after activation if the trigger condition is determined to comprise the first trigger condition and the target data storage interval reaches the first trigger condition for the first time.
Further, the apparatus further comprises:
the data marking module is used for storing the data to be stored in the target data storage interval; the data to be stored comprises a mark field, and the mark field is used for storing a mark numerical value;
correspondingly, the data deleting module is specifically configured to:
and determining the data to be deleted based on the mark value of the mark field of each data.
Further, the apparatus further comprises:
the record table generating module is used for generating a mark numerical value record table;
correspondingly, the data deleting module is specifically configured to:
and determining data to be deleted based on the marker values in the record table, and updating the record table after deleting the data.
Further, after generating the tag value record table, the method further includes:
determining whether the first trigger condition is valid according to the number of the marked numerical values in the record table; if the number exceeds a quota number threshold, determining that the first trigger condition is valid, and if the number does not exceed the quota number threshold, determining that the first trigger condition is invalid.
In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements a data storage space management method according to an embodiment of the present application.
In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable by the processor, where the processor executes the computer program to implement the data storage space management method according to the embodiment of the present application.
According to the technical scheme provided by the embodiment of the application, if the trigger condition of the update capacity of the target data storage interval is detected, whether the trigger condition comprises a first trigger condition and a second trigger condition is determined; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule; and if the data to be deleted simultaneously comprises the first trigger condition and the second trigger condition, determining data to be deleted from all data in the target data storage interval based on a preset rule, and deleting the data to be deleted. By adopting the technical scheme provided by the application, the storage data can be effectively managed according to the requirement, and the storage capacity of the storage equipment for the newly-coming data is not influenced while the storage time of the data is prolonged.
Drawings
Fig. 1 is a schematic diagram of data storage space management provided in an embodiment of the present application. (ii) a
FIG. 2 is a flowchart of a data storage space management method according to an embodiment of the present application;
FIG. 3 is a schematic diagram of data management provided in the second embodiment of the present application;
FIG. 4 is a schematic diagram of a table of generated records provided in the second embodiment of the present application;
FIG. 5 is a diagram illustrating a data storage space management method according to a second embodiment of the present application;
fig. 6 is a schematic structural diagram of a data storage space management apparatus according to a third embodiment of the present application;
fig. 7 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some of the structures related to the present application are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in greater detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Fig. 1 is a schematic diagram of data storage space management provided in an embodiment of the present application. As shown in fig. 1, the recorded data of IPC is stored in each CDV in CLU1, and each CDV is only responsible for managing the recorded data on the server, taking CDV1 as an example: the CDV metadata management takes a POL pool as a unit, the POL pool is checked every 20min, when the total residual capacity of the POL pool is detected to be incapable of meeting the storage of any USR, a capacity shortage request is reported to DN, a deletion mechanism is triggered, and the residual capacity of each USR under the POL pool is checked one by one. When detecting that the capacity of the USR is insufficient, the DN deletes the oldest file according to the sequence of the video data creation Time stored in the current USR (for example, the metadata of the Time1 is deleted in the figure, but the premise is that all data under the USR are obtained for sequencing, and then the data of the oldest Time is deleted), and the capacity is updated.
Wherein the nouns are indicated below:
CDM: and the metadata management server provides metadata cluster management service.
CDV: and the storage node manages the server.
And (3) CLU: the storage cluster management unit comprises a plurality of CDVs and is responsible for balanced storage of data.
VM: and the monitoring business management server is responsible for management services of various monitoring businesses.
DN: and (6) a data node. Managing resources and taking charge of the read-write control of data.
POL: the strategy pool provides a virtualized resource pool for data storage, and is mainly used for virtualizing resources provided by the bottom disk array and scheduling and managing the resources by upper-layer services.
IPC: IP Camera IP. Is the trend of IP video monitoring and high-definition video monitoring. The IPC integrates the camera and the encoder, digitalizes and compresses and encodes the analog video and audio signals to form IP data packets, and transmits the IP data packets to a designated destination address by using a network.
And (3) USR: the virtual write address of the IPC is bound on the POL pool, and a memory and a path are simulated on the POL to be responsible for storing IPC write data.
It can be seen that the current solution has several problems:
1. along with the access of diversified data storage, the storage capacity of the USR depends on capacity acquisition and calculation of upper-layer software, and when the calculation is wrong or the acquired capacity is unreliable, the DN can delete the video data under the condition that the actual capacity of the USR is sufficient, so that the problem that the actual retention period of the corresponding camera is shortened is caused;
2. when the DN clears the USR, the DN can acquire and sequence all data under the USR in advance, if a large amount of data exists, the DN occupies more memory, the sequencing processing consumes longer time, and the influence on the performance of the server is larger;
DN based on the creation time of the video data is sorted (the server time is used as a timestamp to record the data creation time during storage), when the server jumps in time, the DN can preferentially delete the video with the time jump, the sorting mode is unreasonable, and the continuity of the video data is influenced.
In view of the above technical problems and the related technical problems, the present application provides the following technical solutions.
Example one
Fig. 2 is a flowchart of a data storage space management method according to an embodiment of the present application, where the present embodiment is applicable to monitoring and managing a data storage space, and the method may be executed by a data storage space management apparatus according to an embodiment of the present application, where the apparatus may be implemented by software and/or hardware, and may be integrated in an electronic device such as an intelligent terminal.
As shown in fig. 2, the data storage space management method includes:
s210, if the trigger condition of the update capacity of the target data storage interval is detected, determining whether the trigger condition simultaneously comprises a first trigger condition and a second trigger condition; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule.
The target data storage interval may be a hardware device for performing data storage or a specific part of a storage interval of the hardware device, such as a storage interval of a hard disk, a removable disk, a network disk, or a server. Because the target data storage interval is limited, in the process of storing data similar to video monitoring and the like, the data can be continuously transmitted for storage, and storage pressure can be brought to a fixed storage interval. In order to normally store data, after the stored data occupies a certain proportion of the storage interval, it is necessary to perform a deletion process on the data that was stored in the past. Since the remaining capacity of the storage section is updated after deletion, the trigger condition for updating the capacity may be that the storage section meets the condition that the previous stored data needs to be deleted.
Taking the storage of the monitor camera data as an example, if the data is stored with a video recorded every ten minutes as one data, six data are generated every hour. In the process of continuous recording, when the storage interval receives one data per ten minutes for storage, the capacity needs to be updated according to the data size of the video recorded per ten minutes. For example, the data may be updated when the storage amount reaches 90%, or after the video data is received, it may be determined that the currently remaining capacity of the storage interval is not enough to store the video data, and the data needs to be updated.
In the present embodiment, the first trigger condition is a trigger condition generated based on the capacity space, and both of the above two examples are trigger conditions generated based on the capacity space. In addition, the trigger condition may further include a second trigger condition. The second trigger condition is a trigger condition generated based on a preset time rule, and may be, for example, updating the capacity of the storage interval once every preset time.
And S220, if the first trigger condition and the second trigger condition are included, determining data to be deleted from all data in the target data storage interval based on a preset rule, and deleting the data to be deleted.
Specifically, the method may include determining data to be deleted from each data in the target data storage interval, and performing a deletion operation on the data to be deleted.
Based on the preset rule, the time length of the data in the storage interval or the sequence of the timestamps during data storage can be used as the basis. For example, when the first trigger condition and the second trigger condition are satisfied at the same time, the oldest stored data may be deleted, thereby expanding the remaining capacity to store the received data. The method has the advantages that when the capacity needs to be updated or not is determined based on the space, the situation that the normal storage of newly transmitted data cannot be guaranteed due to the fact that the current capacity is read wrongly or the data cannot be read or the data stored in the past is deleted when the data cannot be deleted is avoided, the storage time of the data is shortened, and the playback of the data content is influenced.
According to the technical scheme provided by the embodiment of the application, if the trigger condition of the update capacity of the target data storage interval is detected, whether the trigger condition comprises a first trigger condition and a second trigger condition is determined; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule; and if the first trigger condition and the second trigger condition are included at the same time, determining data to be deleted from all data in the target data storage interval based on a preset rule, and deleting the data to be deleted. By adopting the technical scheme provided by the application, the storage data can be effectively managed according to the requirement, and the storage capacity of the storage equipment for the newly-coming data is not influenced while the storage time of the data is prolonged.
On the basis of the above technical solution, optionally, the obtaining manner of the first trigger condition includes: determining whether a first preset condition is met or not according to the size relation between the residual capacity space of the target data storage interval and the capacity requirement of the data to be stored; and if so, generating a first trigger condition. Determining the size relation between the residual capacity space of the target data storage interval and the capacity requirement of the data to be stored, if the residual capacity space is smaller than the capacity requirement of the data to be stored, generating a first trigger condition, and if the residual capacity space is larger than the capacity requirement of the data to be stored, not generating the first trigger condition. The technical scheme has the advantages that the target data storage interval can be fully utilized, and the time length of data storage is prolonged.
On the basis of the above technical solutions, optionally, the obtaining manner of the second trigger condition includes: determining a capacity timer of a target data storage interval according to the residual capacity space of the target data storage interval and the data storage rate of the target data storage interval; and when the timing of the capacity timer meets a second preset condition, generating a second trigger condition. The data storage rate of the target data storage interval may be obtained statistically according to historical storage data, for example, for the target data storage interval, video information recorded by five cameras may be stored, and each camera stores video data every thirty minutes, so that the data storage rate of the target storage interval may be determined according to the size of video data stored every thirty minutes. Further, the second trigger condition may be generated in the case where it is determined how long it takes to make the remaining capacity space insufficient to store new data based on the size of the remaining capacity space. In this embodiment, it may be determined in the form of a capacity timer that when a certain value of the timer is reached, a second trigger condition is generated. The technical scheme has the advantages that the problem that capacity calculation data is inaccurate due to the fact that whether capacity updating operation is triggered or not is determined according to the space only can be avoided, and therefore the management effect of the data storage space is improved.
On the basis of the foregoing technical solutions, optionally, after detecting a trigger condition of updating the capacity of the target data storage interval, the method further includes: if the triggering condition is determined to comprise a first triggering condition, and the target data storage interval reaches the first triggering condition for the first time, updating the capacity of the data storage space based on the storage time of each data in the target data storage interval, and activating a time triggering mechanism to work so as to generate a second triggering condition based on a preset time rule after activation. In this scheme, there may be a situation where it is not necessary to control the storage efficiency when the data content of the target storage section is not full. Therefore, whether the first trigger condition is the first time or not can be determined, if yes, the updating operation on the capacity can be directly executed, and the time trigger mechanism is activated to work, so that the second trigger condition is generated when the preset time rule is met after activation. The preset time rule may be that after activation, if the timing reaches 1 hour, the preset time rule is determined to be satisfied. If the first trigger condition is not the first time, it indicates that the first trigger condition has been generated, and the data capacity of the target storage interval can be updated only if the first trigger condition and the second trigger condition are received at the same time. The technical scheme has the advantages that effective implementation of space management of data storage is guaranteed, and the executed capacity updating operation can be guaranteed to be accurate.
On the basis of the foregoing technical solutions, optionally, before detecting a trigger condition of updating the capacity of the target data storage interval, the method further includes: storing data to be stored in a target data storage interval; the data to be stored comprises a mark field, and the mark field is used for storing a mark numerical value; correspondingly, determining data to be deleted from each data in the target data storage interval based on a preset rule, including: and determining the data to be deleted based on the marking value of the marking field of each data. The data may be generated with a tag field, or the data may be stored with a tag field for tagging a value, for example, if the first stored data is, the first stored data may be tagged as 0000, the next data is 0001, and so on. Then when a delete operation is performed on the data, it may be determined from the tag value that data is stored earliest and then determined as the data to be deleted. In the technical scheme, optionally, the date of the system can also be recorded, so that when data needs to be deleted, a batch of data stored earliest can be determined according to the date, and then the data to be deleted can be determined according to the marker value. Thus, the data to be deleted can be determined more quickly. Meanwhile, the method can avoid the situation that the storage time jumps or the system time is called back due to the system time or equipment reasons, so that the data stored later is preferentially deleted to influence the playback of the data content.
On the basis of the above technical solution, optionally, before determining the data to be deleted based on the tag value of the tag field of each data, the method further includes: generating a mark numerical value recording table; correspondingly, determining the data to be deleted based on the tag value of the tag field of each data includes: and determining data to be deleted based on the marker values in the record table, and updating the record table after deleting the data. The record table can be stored separately, and when the data to be deleted needs to be determined, the record table is read and the mark value of the data to be deleted is determined, so that the data to be deleted is determined. Wherein the content recorded in the recording table may include a flag value. After the data is deleted, the number in the data table may be updated, for example, after the data with the number 0000 is deleted, the number 0000 may be deleted from the record table, and the number value of the oldest stored data after deletion is 0001. In this embodiment, the record table may be a summary table, or may be divided according to dates, that is, one record table is generated every day, or may be flexibly set in other manners. The method has the advantages that the data to be deleted can be determined only according to the mark values in the record table, and the method is simple, convenient and quick.
On the basis of the above technical solutions, optionally, after generating the mark numerical value record table, the method further includes: determining whether the first trigger condition is valid according to the number of the marked numerical values in the record table; if the number exceeds a quota number threshold, determining that the first trigger condition is valid, and if the number does not exceed the quota number threshold, determining that the first trigger condition is invalid. For example, if the quota number threshold is 100, after receiving the first trigger condition, it may be determined whether the first trigger condition is valid according to the number of flag values recorded in the record table, where the first trigger condition is valid if the number exceeds 100, and is invalid if the number does not exceed 100. The method has the advantages that the amount of the data stored in the target storage interval can be reflected from the side surface based on the number of the mark values in the record table, and the situation that the data is not deleted due to the error in reading the storage capacity is avoided, the first trigger condition is received, the data deletion operation is executed, and the storage time of the data is influenced.
Example two
In order to enable a person skilled in the art to more accurately understand the technical solutions provided in the present application, the present application also provides a preferred embodiment.
Fig. 3 is a schematic diagram of data management provided in the second embodiment of the present application. As shown in fig. 3, the nouns are illustrated as follows:
CDM: and the metadata management server provides metadata cluster management service.
CDV: and the storage node manages the server.
And (3) CLU: the storage cluster management unit comprises a plurality of CDVs and is responsible for balanced storage of data.
And (4) VM: and the monitoring business management server is responsible for management services of various monitoring businesses.
DN: and (6) a data node. Managing resources and taking charge of read-write control of data.
POL: and (4) strategy pool. A virtualization pool of different policies is provided. The main role is to virtualize the underlying LV resources
IPC: IP Camera IP. Is the trend of IP video monitoring and high-definition video monitoring. The IPC integrates the camera and the encoder, digitalizes and compresses and encodes the analog video and audio signals to form IP data packets, and transmits the IP data packets to a designated destination address by using a network.
And (3) USR: the virtual write address of the IPC is bound on the POL pool, and a memory and a path are simulated on the POL to be responsible for storing IPC write data.
The invention improves and optimizes the following three contents based on the original metadata basic management mode:
(1) And modifying the file format of metadata storage, adding a numbering field on the original file format, and numbering according to the sequence of file creation. Meanwhile, a recording table is added, and the start-stop number of each day file is recorded by taking the day as a unit;
(2) Modifying a deleting mechanism, deleting the oldest file, not sequencing any more, but directly accessing a recording table generated during storage, searching the oldest file according to the initial code, saving sequencing time and eliminating the dependency of sequencing on the creation time of the video data;
(3) The deletion trigger mechanism in the normal storage process of the metadata is modified, a capacity demand timer is added on the basis of the original judgment basis of capacity insufficiency only, the reasonability of the deletion demand is ensured again on the deletion frequency, and the unreasonable deletion request triggered by capacity calculation errors is prevented.
Fig. 4 is a schematic diagram of generating a record table according to the second embodiment of the present application. As shown in fig. 4, when the oldest file of the USR is deleted by DN, the first record (the record inserted first) of the USR in the record table is directly searched, and according to the USR, the date (Day), and the StartNum information, the corresponding recorded data is directly found and deleted, and then the StartNum is updated. If the video data of the day is deleted, the record of the day is deleted in the data table.
Fig. 5 is a schematic diagram of a data storage space management method according to a second embodiment of the present application. As shown in fig. 5, when the USR is insufficient in capacity, it is first determined whether the capacity insufficiency alarm is true or valid, and if it is determined that the capacity insufficiency alarm is valid, the capacity insufficiency alarm is sent to the DN, otherwise, the capacity insufficiency alarm is not sent. The judgment basis is that the number of files written in the current USR is calculated according to a record table generated by storage, whether the number of the files reaches a quota capacity threshold value is judged according to the number of the files, if the number of the files exceeds the quota capacity threshold value, the capacity shortage alarm is judged to be valid, and if not, the capacity shortage alarm is judged to be invalid and is not sent to the DN.
When the timer is started, the capacity demand timer is in a closed state, after the DN receives the first effective insufficient capacity alarm of the USR, the DN judges that the current timer is in the closed state, the capacity processing operation is directly carried out, the oldest file is deleted, the USR capacity is updated, meanwhile, the timer is activated, and the initial time is set for the timer.
After the timer is activated, when the USR reports that the DN capacity is insufficient again, the DN can judge whether the timer gives an alarm or not, if the timer gives an alarm at the moment, the capacity processing operation is carried out, the oldest file is deleted, the USR capacity is updated, and the set value of the timer is initialized; if the timer is not alarmed at this time, the non-processing is ignored.
And calculating a reasonable numerical value as a set value of the timer according to the time span of the latest file storage at present and by combining an optimization formula of laboratory test data.
The technical scheme provided by the application starts from the two aspects of 'effectiveness of insufficient capacity request' and 'capacity demand frequency', and manages the life cycle of the video metadata on the basis of simultaneously meeting two conditions of time and space. When the USR sends out a capacity shortage request, the data are stably and orderly deleted according to the effectiveness of the capacity request and the correctness of the required rate of the capacity, and the storage period of the metadata is doubly guaranteed. The method reduces the dependence of metadata management on a time line, always manages the metadata in a first-in first-out principle, and visually ensures reasonable alternation of the metadata.
By adopting the technical scheme, the metadata management starts from two aspects of time and space, on one hand, whether the capacity shortage alarm is effective or not can be judged, and the error alarm is sent out when the capacity acquisition is unreliable; on the other hand, the data deleting frequency is controlled from the capacity demand frequency, so that the capacity update is kept at a certain rate, and the data can be effectively protected from being triggered and deleted by other abnormal conditions. And deleting according to the sequence of file writing without depending on the time line of the server, and when the server jumps in time, the management of the metadata is not influenced.
EXAMPLE III
Fig. 6 is a schematic structural diagram of a data storage space management apparatus according to a third embodiment of the present application. As shown in fig. 6, the data storage space management apparatus includes:
a trigger condition determining module 610, configured to determine, if a trigger condition for updating the capacity of the target data storage interval is detected, whether the trigger condition includes a first trigger condition and a second trigger condition at the same time; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule;
and a data deleting module 620, configured to determine, based on a preset rule, data to be deleted from each data in the target data storage interval if the triggering condition determining module determines that the data to be deleted includes the first triggering condition and the second triggering condition at the same time, and delete the data to be deleted.
According to the technical scheme provided by the embodiment of the application, if the trigger condition of the update capacity of the target data storage interval is detected, whether the trigger condition comprises a first trigger condition and a second trigger condition is determined; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule; and if the first trigger condition and the second trigger condition are included at the same time, determining data to be deleted from all data in the target data storage interval based on a preset rule, and deleting the data to be deleted. By adopting the technical scheme provided by the application, the storage data can be effectively managed according to the requirement, and the storage capacity of the storage equipment for the newly-coming data is not influenced while the storage time of the data is prolonged.
On the basis of the above technical solutions, optionally, the obtaining manner of the first trigger condition includes:
determining whether a first preset condition is met or not according to the size relation between the residual capacity space of the target data storage interval and the capacity requirement of the data to be stored;
and if so, generating a first trigger condition.
On the basis of the above technical solutions, optionally, the obtaining manner of the second trigger condition includes:
determining a capacity timer of a target data storage interval according to the residual capacity space of the target data storage interval and the data storage rate of the target data storage interval;
and when the timing of the capacity timer meets a second preset condition, generating a second trigger condition.
On the basis of the above technical solutions, optionally, the apparatus further includes:
and the second trigger condition activation module is used for updating the capacity of the data storage space based on the storage time of each data in the target data storage interval and activating a time trigger mechanism to work so as to generate a second trigger condition based on a preset time rule after activation if the trigger condition is determined to comprise the first trigger condition and the target data storage interval reaches the first trigger condition for the first time.
On the basis of the above technical solutions, optionally, the apparatus further includes:
the data marking module is used for storing the data to be stored in the target data storage interval; the data to be stored comprises a mark field, and the mark field is used for storing a mark numerical value;
correspondingly, the data deleting module is specifically configured to:
and determining the data to be deleted based on the marking value of the marking field of each data.
On the basis of the above technical solutions, optionally, the apparatus further includes:
the record table generating module is used for generating a mark numerical value record table;
correspondingly, the data deleting module is specifically configured to:
and determining data to be deleted based on the marker values in the record table, and updating the record table after deleting the data.
The product can execute the method provided by any embodiment of the application, and has corresponding functional modules and beneficial effects of the execution method.
Example four
Embodiments of the present application also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, are configured to perform a method of data storage space management, the method comprising:
if the trigger condition of the update capacity of the target data storage interval is detected, determining whether the trigger condition simultaneously comprises a first trigger condition and a second trigger condition; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule;
and if the first trigger condition and the second trigger condition are included at the same time, determining data to be deleted from all data in the target data storage interval based on a preset rule, and deleting the data to be deleted.
Storage medium-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system connected to the computer system through a network (such as the internet). The second computer system may provide the program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations, such as in different computer systems that are connected via a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.
Of course, the storage medium provided in this embodiment of the present application includes computer-executable instructions, where the computer-executable instructions are not limited to the data storage space management operation described above, and may also perform related operations in the data storage space management method provided in any embodiment of the present application.
EXAMPLE five
The embodiment of the application provides electronic equipment, and the data storage space management device provided by the embodiment of the application can be integrated into the electronic equipment. Fig. 7 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present application. As shown in fig. 7, the present embodiment provides an electronic device 700, which includes: one or more processors 720; the storage device 710 is configured to store one or more programs, and when the one or more programs are executed by the one or more processors 720, the one or more processors 720 implement the data storage space management method provided in the embodiment of the present application, the method includes:
if the trigger condition of the update capacity of the target data storage interval is detected, determining whether the trigger condition comprises a first trigger condition and a second trigger condition at the same time; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule;
and if the data to be deleted simultaneously comprises the first trigger condition and the second trigger condition, determining data to be deleted from all data in the target data storage interval based on a preset rule, and deleting the data to be deleted.
Of course, it can be understood by those skilled in the art that the processor 720 also implements the technical solution of the data storage space management method provided in any embodiment of the present application.
The electronic device 700 shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 7, the electronic device 700 includes a processor 720, a storage 710, an input 730, and an output 740; the number of the processors 720 in the electronic device may be one or more, and one processor 720 is taken as an example in fig. 7; the processor 720, the storage device 710, the input device 730, and the output device 740 in the electronic apparatus may be connected by a bus or other means, and are exemplified by a bus 750 in fig. 7.
The storage device 710, which is a computer-readable storage medium, can be used to store software programs, computer-executable programs, and module units, such as program instructions corresponding to the data storage space management method in the embodiments of the present application.
The storage device 710 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the storage 710 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, storage 710 may further include memory located remotely from processor 720, which may be connected via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 730 may be used to receive input numbers, character information, or voice information, and to generate key signal inputs related to user settings and function control of the electronic apparatus. The output device 740 may include a display screen, speakers, etc.
The electronic equipment provided by the embodiment of the application can effectively manage the stored data according to the requirement, and does not influence the storage capacity of the storage equipment to the newly-coming data while increasing the storage time of the data.
The data storage space management device, the storage medium and the electronic device provided in the above embodiments may execute the data storage space management method provided in any embodiment of the present application, and have corresponding functional modules and beneficial effects for executing the method. Technical details that are not described in detail in the above embodiments may be referred to a data storage space management method provided in any embodiments of the present application.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present application and the technical principles employed. It will be understood by those skilled in the art that the present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the application. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims (9)
1. A method for managing data storage space, comprising:
if the trigger condition of the update capacity of the target data storage interval is detected, determining whether the trigger condition simultaneously comprises a first trigger condition and a second trigger condition; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule;
if the first trigger condition and the second trigger condition are included at the same time, determining data to be deleted from all data in the target data storage interval based on a preset rule, and deleting the data to be deleted;
wherein, the obtaining mode of the second trigger condition comprises:
determining a capacity timer of a target data storage interval according to the residual capacity space of the target data storage interval and the data storage rate of the target data storage interval;
and when the timing of the capacity timer meets a second preset condition, generating a second trigger condition.
2. The method of claim 1, wherein the first trigger condition is derived by:
determining whether a first preset condition is met or not according to the size relation between the residual capacity space of the target data storage interval and the capacity requirement of the data to be stored;
and if so, generating a first trigger condition.
3. The method of any of claims 1-2, wherein after detecting a trigger condition for an update capacity of a target data storage interval, the method further comprises:
and if the trigger condition is determined to comprise a first trigger condition and the target data storage interval reaches the first trigger condition for the first time, updating the capacity of the data storage space based on the storage time of each data in the target data storage interval, and activating a time trigger mechanism to work so as to generate a second trigger condition based on a preset time rule after activation.
4. The method of any of claims 1-2, wherein prior to detecting a trigger condition for an update capacity of a target data storage interval, the method further comprises:
storing data to be stored in a target data storage interval; the data to be stored comprises a mark field, and the mark field is used for storing a mark numerical value;
correspondingly, determining data to be deleted from each data in the target data storage interval based on a preset rule, including:
and determining the data to be deleted based on the mark value of the mark field of each data.
5. The method of claim 4, wherein prior to determining data to delete based on the tag value of the tag field of each data, the method further comprises:
generating a mark numerical value recording table;
correspondingly, determining the data to be deleted based on the tag value of the tag field of each data includes:
and determining data to be deleted based on the marker values in the record table, and updating the record table after deleting the data.
6. The method of claim 5, wherein after generating the marker value record table, the method further comprises:
determining whether the first trigger condition is valid according to the number of the marked numerical values in the record table; if the number exceeds the threshold of quota number, determining that the first trigger condition is valid, and if the number does not exceed the threshold of quota number, determining that the first trigger condition is invalid.
7. A data storage space management apparatus, comprising:
the trigger condition judging module is used for determining whether the trigger conditions comprise a first trigger condition and a second trigger condition at the same time if the trigger conditions of the updating capacity of the target data storage interval are detected; the first trigger condition comprises a trigger condition generated based on a capacity space, and the second trigger condition comprises a trigger condition generated based on a preset time rule;
the data deleting module is used for determining data to be deleted from all data in the target data storage interval based on a preset rule and deleting the data to be deleted if the triggering condition judging module judges that the data to be deleted simultaneously comprises a first triggering condition and a second triggering condition;
wherein, the obtaining mode of the second trigger condition comprises:
determining a capacity timer of a target data storage interval according to the residual capacity space of the target data storage interval and the data storage rate of the target data storage interval;
and when the timing of the capacity timer meets a second preset condition, generating a second trigger condition.
8. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a data storage space management method according to any one of claims 1 to 6.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data storage space management method according to any of claims 1-6 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910596063.0A CN112181277B (en) | 2019-07-03 | 2019-07-03 | Data storage space management method and device, storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910596063.0A CN112181277B (en) | 2019-07-03 | 2019-07-03 | Data storage space management method and device, storage medium and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112181277A CN112181277A (en) | 2021-01-05 |
| CN112181277B true CN112181277B (en) | 2022-10-04 |
Family
ID=73915638
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910596063.0A Active CN112181277B (en) | 2019-07-03 | 2019-07-03 | Data storage space management method and device, storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112181277B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112835525A (en) * | 2021-02-07 | 2021-05-25 | 东方网力科技股份有限公司 | Method, device and terminal for improving utilization rate of storage space |
| CN114356856A (en) * | 2021-12-21 | 2022-04-15 | 苏州凌云视界智能设备有限责任公司 | Data deleting method, device, equipment and storage medium |
| CN114416677A (en) * | 2021-12-30 | 2022-04-29 | 中国电信股份有限公司 | A method, device, device and storage medium for updating cold storage data |
| CN114528052A (en) * | 2022-02-24 | 2022-05-24 | 浙江大华技术股份有限公司 | Data loading method and device, storage medium and electronic device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102799395A (en) * | 2012-06-29 | 2012-11-28 | 华为技术有限公司 | Dynamic balancing processing method and device for disk spaces and disk system |
| CN106445414A (en) * | 2016-09-20 | 2017-02-22 | 广州视源电子科技股份有限公司 | Method and system for expanding available storage space of terminal equipment |
| CN107301020A (en) * | 2017-06-22 | 2017-10-27 | 苏州交运电子科技有限公司 | Data managing method and managing device |
| CN109284066A (en) * | 2017-07-19 | 2019-01-29 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device, equipment and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10915498B2 (en) * | 2017-03-30 | 2021-02-09 | International Business Machines Corporation | Dynamically managing a high speed storage tier of a data storage system |
-
2019
- 2019-07-03 CN CN201910596063.0A patent/CN112181277B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102799395A (en) * | 2012-06-29 | 2012-11-28 | 华为技术有限公司 | Dynamic balancing processing method and device for disk spaces and disk system |
| CN106445414A (en) * | 2016-09-20 | 2017-02-22 | 广州视源电子科技股份有限公司 | Method and system for expanding available storage space of terminal equipment |
| CN107301020A (en) * | 2017-06-22 | 2017-10-27 | 苏州交运电子科技有限公司 | Data managing method and managing device |
| CN109284066A (en) * | 2017-07-19 | 2019-01-29 | 阿里巴巴集团控股有限公司 | A kind of data processing method, device, equipment and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112181277A (en) | 2021-01-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112181277B (en) | Data storage space management method and device, storage medium and electronic equipment | |
| CN109960686B (en) | Log processing method and device for database | |
| US7111136B2 (en) | Method and apparatus for backup and recovery system using storage based journaling | |
| US8103840B2 (en) | Snapshot mechanism and method thereof | |
| US20200019320A1 (en) | Capacity forecasting based on capacity policies and transactions | |
| US20070239806A1 (en) | Methods and apparatus for a fine grained file data storage system | |
| CN113111129A (en) | Data synchronization method, device, equipment and storage medium | |
| CN103197988A (en) | Data backup and recovery method, device and database system | |
| CN106682162A (en) | Log management method and device | |
| CN111367755A (en) | User log writing method and system of mobile terminal | |
| CN101174225A (en) | A device and method for realizing message communication | |
| CN113515317A (en) | Method and device for data recovery | |
| CN111399753B (en) | Method and device for writing pictures | |
| US8938593B2 (en) | Tracking ownership of memory in a data processing system through use of a memory monitor | |
| CN112783711A (en) | Method and storage medium for analyzing program memory on NodeJS | |
| CN109521969B (en) | Solid state disk data recovery method and device and computer readable storage medium | |
| US7657664B2 (en) | Method and system for tracking device driver requests | |
| CN118964489A (en) | Offline synchronization method of incremental data in data development | |
| CN111078418B (en) | Operation synchronization method, device, electronic equipment and computer readable storage medium | |
| CN112000971A (en) | File permission recording method, system and related device | |
| CN112685329A (en) | Method for processing data and related device | |
| CN117056313A (en) | Memory database transaction management method and device, electronic equipment and medium | |
| CN111831206B (en) | Storage space switching method and back-end monitoring system | |
| CN107632792A (en) | The method and apparatus that virtual disk is managed in cloud data system | |
| CN106959888B (en) | Task processing method and device in cloud storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |