CN111090617B - File management method and device in monitoring system and computer readable storage medium - Google Patents
File management method and device in monitoring system and computer readable storage medium Download PDFInfo
- Publication number
- CN111090617B CN111090617B CN201811235338.XA CN201811235338A CN111090617B CN 111090617 B CN111090617 B CN 111090617B CN 201811235338 A CN201811235338 A CN 201811235338A CN 111090617 B CN111090617 B CN 111090617B
- Authority
- CN
- China
- Prior art keywords
- file
- files
- media
- current
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0625—Power saving in storage systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a file management method, a device and a computer readable storage medium in a monitoring system, wherein the method comprises the following steps: storing the monitoring file based on three layers of directories including a channel, a date and a file size; when determining that a monitoring file with a preset size needs to be added, transferring the monitoring file from a subdirectory with the corresponding size of an idle directory to a corresponding subdirectory on the current date; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory; two preset media slices are recycled in the live broadcast process, and the media slices are stored in a file merging file; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period.
Description
Technical Field
The present invention relates to the field of mobile communications technologies, and in particular, to a method and an apparatus for managing files in a monitoring system, and a computer-readable storage medium.
Background
In an industrial monitoring system, the storage period of monitoring audio and video is generally between seven days and two years, most audio and video monitoring files are stored in a media platform according to time periods at present, and disk fragments are caused when the audio and video monitoring files are circularly deleted.
In order to reduce the occurrence of disk fragmentation caused by deletion, the related method is to combine a small file with a large file, for example, a channel name is used as a primary directory, a date is used as a secondary directory to manage the file of each channel, so as to realize dynamic creation of the file and realize dynamic expansion or reduction of the channel, but after the file is dynamically created, the writing speed of the disk can be influenced because the operation of '0' of pre-writing is not carried out; in addition, since each file is fixed in size, when each fixed-size file may not be fully written, particularly when the data to be written in the last file of each day is small, it causes a waste of storage. Moreover, creating a file when there are insufficient free files reduces the first write rate of the file.
Disclosure of Invention
In view of the above, embodiments of the present invention are intended to provide a method and an apparatus for managing files in a monitoring system, and a computer-readable storage medium.
In order to achieve the above purpose, the technical solution of the embodiment of the present invention is realized as follows:
the embodiment of the invention provides a file management method in a monitoring system, which comprises the following steps:
storing the monitoring file based on three layers of directories including a channel, a date and a file size;
when determining that a monitoring file with a preset size needs to be added, transferring the monitoring file from a subdirectory with the corresponding size of an idle directory to a corresponding subdirectory on the current date; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory;
two preset media slices are recycled in the live broadcasting process, and the media slices are stored in a merged file; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period.
Optionally, the method further includes:
creating files with corresponding number and size, wherein the total capacity of the files is larger than the sum of daily storage capacity of the channels, according to the estimated number of the channels;
the file is pre-written \0' and moved to the corresponding subdirectory.
Optionally, the method further includes:
checking the number of idle files under each subdirectory regularly;
and when the number of the idle files is determined to be smaller than the threshold value, creating corresponding files and preprocessing the files.
Wherein, the method also comprises the following steps:
after the video in the preset period is packaged into MP4, judging whether the residual storage capacity of the current file can be written into the MP4 media data;
if not, closing the current file, determining the sum of the MP4 capacity and the residual capacity of the current day, transferring the file with the corresponding size from the idle file directory to the corresponding subdirectory of the current channel, and writing the file; if so, the current file is written sequentially.
Wherein after the sequentially writing the current file, the method further comprises:
determining whether the writing and the starting time of the MP4 are in the same day;
if the current file is in the same day, determining the residual capacity of the current file, determining whether the residual capacity can be written into the media data of the next preset period according to the media capacity of the previous preset period, and if so, returning to the writing operation; if the time length of the media file which can be written is not the same as the time length of the media file which can be written, and the operation of closing the current file or returning to the writing operation is executed based on the time length;
if not, repeating the on-demand process when the first storage period is determined not to be expired; and if the second storage period is up, renaming the date subdirectories under the channel directories, and moving all the original files in the subdirectories under the date to the corresponding subdirectories under the free file directory.
Wherein, the method also comprises the following steps in the live broadcast process:
inquiring timeliness of media slice merging files and index files under each channel directory at intervals of preset time;
and when the last modification time of the media slice merging file or the index file is determined to be larger than a preset threshold value, recovering the related files under the corresponding channel directories.
Wherein, the method also comprises the following steps in the live broadcast process:
inquiring the number of media slice merging files and index files in a live idle file directory at intervals of preset time;
and when the number of the media slice merging files or the index files is less than a preset threshold value, creating corresponding files.
An embodiment of the present invention further provides a file management apparatus in a monitoring system, where the apparatus includes:
the storage module is used for storing the monitoring file based on three layers of directories including a channel, a date and a file size;
the processing module is used for transferring the monitoring file from the subdirectory with the corresponding size of the idle directory to the corresponding subdirectory on the current date when the monitoring file with the preset size needs to be added is determined; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory;
the live broadcast processing module is used for circularly using two preset media slices in the live broadcast process to merge the file storage media slices; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period.
An embodiment of the present invention further provides a file management apparatus in a monitoring system, where the apparatus includes: a processor and a memory for storing a computer program capable of running on the processor,
wherein the processor is configured to perform the steps of the above method when running the computer program.
Embodiments of the present invention also provide a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-mentioned method.
The file management method, the file management device and the computer-readable storage medium in the monitoring system provided by the embodiment of the invention store the monitoring files based on three layers of directories including a channel, a date and a file size; when determining that a monitoring file with a preset size needs to be added, transferring the monitoring file from a subdirectory with the corresponding size of an idle directory to a corresponding subdirectory on the current date; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory; two preset media slices are recycled in the live broadcasting process, and the media slices are stored in a merged file; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period. The embodiment of the invention adopts the three-layer directory to manage and store the files, and when the storage period is up, the files do not need to be deleted, only the date subdirectories under the corresponding channel directories need to be renamed, and the original files in each subdirectory under the date are moved to the corresponding subdirectories under the idle file directory, thereby not only realizing the rapidness, but also reducing the fragments caused by the file deletion. In addition, in the live broadcasting process, the two preset media slices are recycled to merge the file storage media slices, so that the number of small files is reduced, the maintenance cost is reduced, and disk fragments cannot be introduced due to file deletion.
In addition, according to the estimated channel quantity, files with the corresponding number and size of the total capacity larger than the sum of the daily storage capacity of the channels are created; the number of idle files under each subdirectory is checked regularly at the later stage, and when the number of the idle files is determined to be smaller than a threshold value, corresponding files are created; the method not only ensures the preprocessing speed, but also can not cause performance overhead caused by file creation of a writing thread due to insufficient use of files, and the size of the stored file is not fixed, and can be dynamically adjusted according to the needs, so that the storage loss caused by file switching can be reduced.
Drawings
Fig. 1 is a schematic flow chart of a file management method in a monitoring system according to an embodiment of the present invention;
FIG. 2 is a first schematic structural diagram of a file management apparatus in a monitoring system according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a file management device in the monitoring system according to the embodiment of the present invention.
Detailed Description
The invention is described below with reference to the figures and examples.
An embodiment of the present invention provides a file management method in a monitoring system, and as shown in fig. 1, the method includes:
step 101: storing the monitoring file based on three layers of directories including a channel, a date and a file size;
step 102: when determining that a monitoring file with a preset size needs to be added, transferring the monitoring file from a subdirectory with the corresponding size of an idle directory to a corresponding subdirectory on the current date; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory;
step 103: two preset media slices are recycled in the live broadcasting process, and the media slices are stored in a merged file; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period.
The sequence of the above steps is not limited.
The embodiment of the invention adopts three layers of directories to manage and store the files, when the storage period is up, the files do not need to be deleted, only the date subdirectory under the corresponding channel directory needs to be renamed, and the original files in each subdirectory under the date are moved to the corresponding subdirectory under the idle file directory, thereby not only realizing the rapidness, but also reducing the fragments caused by the file deletion. In addition, in the live broadcasting process, the two preset media slices are recycled to merge the file storage media slices, so that the number of small files is reduced, the maintenance cost is reduced, and disk fragments cannot be introduced due to file deletion.
In one embodiment, the method further comprises:
creating files with corresponding number and size of total capacity larger than the sum of daily storage capacity of the channels according to the estimated number of the channels;
the file is pre-written \0' and moved to the corresponding subdirectory.
In one embodiment, the method further comprises:
regularly checking the number of idle files under each subdirectory;
and when the number of the idle files is determined to be smaller than the threshold value, creating corresponding files and preprocessing the files.
In one embodiment, the method further comprises the following steps:
after the video in the preset period is packaged into MP4, judging whether the residual storage capacity of the current file can be written into the MP4 media data;
if not, closing the current file, determining the sum of the MP4 capacity and the residual capacity of the current day, transferring the file with the corresponding size from the idle file directory to the corresponding subdirectory of the current channel, and writing the file; if so, the current file is written sequentially.
In one embodiment, after the sequentially writing the current file, the method further comprises:
determining whether the writing is in the same day as the starting time of the MP 4;
if the current file is in the same day, determining the residual capacity of the current file, determining whether the residual capacity can write the media data of the next preset period according to the media capacity of the previous preset period, and if so, returning to the writing operation; if the media file cannot be written, determining the duration of the media file which can be written, and executing the operation of closing the current file or returning the writing operation based on the duration;
if not, repeating the on-demand process when the first storage period is determined not to be expired; and if the second storage period is up, renaming the sub-directories of the dates under the channel directories, and completely moving the original files in the sub-directories under the dates to the corresponding sub-directories under the free file directories.
In one embodiment, the method further comprises, during the live broadcast:
inquiring timeliness of media slice merging files and index files under each channel directory at intervals of preset time;
and when the last modification time of the media slice merging file or the index file is determined to be larger than a preset threshold value, recovering the related files under the corresponding channel directories.
In one embodiment, the method further comprises, during the live broadcast:
inquiring the number of media slice merging files and index files in a live idle file directory at intervals of preset time;
and when the number of the media slice merging files or the index files is less than a preset threshold value, establishing corresponding files.
An embodiment of the present invention further provides a file management apparatus in a monitoring system, and as shown in fig. 2, the apparatus includes:
the storage module 201 is used for storing the monitoring file based on three layers of directories including a channel, a date and a file size;
the processing module 202 is configured to, when it is determined that a monitoring file with a preset size needs to be added, transfer the monitoring file from a subdirectory with a corresponding size of an idle directory to a corresponding subdirectory on a current date; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory;
the live broadcast processing module 203 is used for cyclically using two preset media slices in the live broadcast process to merge the file storage media slices; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period.
In one embodiment, as shown in fig. 3, the apparatus further comprises:
a creating module 204, configured to create, according to the estimated number of channels, files with corresponding numbers and sizes, where a total capacity is greater than a sum of daily storage capacities of the channels;
carry on '0' operation of prewrite to file, and move the file to the corresponding subdirectory.
In one embodiment, the creating module 204 is further configured to periodically check the number of free files in each sub-directory;
and when the number of the idle files is determined to be smaller than the threshold value, creating corresponding files and preprocessing the files.
In one embodiment, as shown in fig. 3, the apparatus further comprises:
the on-demand processing module 205 is configured to, after the video in the preset period is encapsulated into the MP4, determine whether the remaining storage capacity of the current file can be written into the current MP4 media data;
if the current file cannot be written, closing the current file, determining the sum of the MP4 capacity and the residual capacity of the current day, transferring the file with the corresponding size from the idle file directory to the corresponding subdirectory of the current channel, and writing the file; if so, the current file is written sequentially.
In an embodiment, the on-demand processing module 205 is further configured to determine whether the writing is on the same day as the start time of the current MP 4;
if the determination is that the file is in the same day, determining the residual capacity of the current file, determining whether the residual capacity can be written into the media data of the next preset period according to the media capacity of the previous preset period, and if so, returning to the writing operation; if the time length of the media file which can be written is not the same as the time length of the media file which can be written, and the operation of closing the current file or returning to the writing operation is executed based on the time length;
determining that the data are not in the same day, and repeating the on-demand process when the first storage period is determined not to be expired; and if the second storage period is up, renaming the sub-directories of the dates under the channel directories, and completely moving the original files in the sub-directories under the dates to the corresponding sub-directories under the free file directories.
In one embodiment, the live broadcast processing module 203 is further configured to query timeliness of a media slice merge file and an index file in each channel directory at intervals of preset time;
and when the last modification time of the media slice merging file or the index file is determined to be larger than a preset threshold value, recovering the related files under the corresponding channel directories.
In one embodiment, the live broadcast processing module 203 is further configured to query, at intervals of preset time, the number of media slice merge files and index files in a live broadcast idle file directory;
and when the number of the media slice merging files or the index files is less than a preset threshold value, establishing corresponding files.
An embodiment of the present invention further provides a file management apparatus in a monitoring system, where the apparatus includes: a processor and a memory for storing a computer program capable of running on the processor,
wherein the processor is configured to, when running the computer program, perform:
storing the monitoring file based on three layers of directories, namely a channel, a date and a file size;
when determining that a monitoring file with a preset size needs to be added, transferring the monitoring file from a subdirectory with the corresponding size of an idle directory to a corresponding subdirectory on the current date; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory;
two preset media slices are recycled in the live broadcast process, and the media slices are stored in a file merging file; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period.
The processor is further configured to, when executing the computer program, perform:
creating files with corresponding number and size, wherein the total capacity of the files is larger than the sum of daily storage capacity of the channels, according to the estimated number of the channels;
carry on '0' operation of prewrite to file, and move the file to the corresponding subdirectory.
The processor is further configured to, when executing the computer program, perform:
checking the number of idle files under each subdirectory regularly;
and when the number of the idle files is smaller than the threshold value, creating corresponding files and preprocessing the files.
The processor is further configured to, when executing the computer program, perform:
after the video in the preset period is packaged into MP4, judging whether the residual storage capacity of the current file can be written into the MP4 media data;
if not, closing the current file, determining the sum of the MP4 capacity and the residual capacity of the current day, transferring the file with the corresponding size from the idle file directory to the corresponding subdirectory of the current channel, and writing the file; if so, the current file is written sequentially.
After the sequentially writing the current file, the processor is further configured to, when running the computer program, perform:
determining whether the writing is in the same day as the starting time of the MP 4;
if the current file is in the same day, determining the residual capacity of the current file, determining whether the residual capacity can be written into the media data of the next preset period according to the media capacity of the previous preset period, and if so, returning to the writing operation; if the time length of the media file which can be written is not the same as the time length of the media file which can be written, and the operation of closing the current file or returning to the writing operation is executed based on the time length;
if not, repeating the on-demand process when the first storage period is determined not to be expired; and if the second storage period is up, renaming the sub-directories of the dates under the channel directories, and completely moving the original files in the sub-directories under the dates to the corresponding sub-directories under the free file directories.
In the live broadcasting process, the processor is further configured to execute, when running the computer program:
inquiring timeliness of media slice merging files and index files under each channel directory at intervals of preset time;
and when the last modification time of the media slice merging file or the index file is determined to be larger than a preset threshold value, recovering the related files under the corresponding channel directories.
In the live broadcasting process, the processor is further configured to execute, when running the computer program:
inquiring the number of media slice merging files and index files in a live idle file directory at intervals of preset time;
and when the number of the media slice merging files or the index files is less than a preset threshold value, establishing corresponding files.
It should be noted that: in the device provided in the above embodiment, when performing file management, only the division of each program module is illustrated, and in practical applications, the above processing allocation may be completed by different program modules according to needs, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the apparatus provided in the above embodiments and the corresponding method embodiments belong to the same concept, and the specific implementation process thereof is described in the method embodiments, which is not described herein again.
In an exemplary embodiment, the embodiment of the present invention also provides a computer-readable storage medium, which may be a Memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash Memory, magnetic surface Memory, optical disc, or CD-ROM; or a variety of devices, such as mobile phones, computers, tablet devices, personal digital assistants, etc., that include one or any combination of the above memories.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs:
storing the monitoring file based on three layers of directories including a channel, a date and a file size;
when determining that a monitoring file with a preset size needs to be added, transferring the monitoring file from a subdirectory with the corresponding size of an idle directory to a corresponding subdirectory on the current date; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory;
two preset media slices are recycled in the live broadcasting process, and the media slices are stored in a merged file; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period.
The computer program, when executed by the processor, further performs:
creating files with corresponding number and size, wherein the total capacity of the files is larger than the sum of daily storage capacity of the channels, according to the estimated number of the channels;
the file is pre-written \0' and moved to the corresponding subdirectory.
The computer program, when executed by the processor, further performs:
checking the number of idle files under each subdirectory regularly;
and when the number of the idle files is determined to be smaller than the threshold value, creating corresponding files and preprocessing the files.
The computer program, when executed by the processor, further performs:
after the video in the preset period is packaged into MP4, judging whether the residual storage capacity of the current file can be written into the MP4 media data;
if not, closing the current file, determining the sum of the MP4 capacity and the residual capacity of the current day, transferring the file with the corresponding size from the idle file directory to the corresponding subdirectory of the current channel, and writing the file; if so, the current file is written sequentially.
After the sequential writing to the current file, the computer program, when executed by the processor, further performs:
determining whether the writing is in the same day as the starting time of the MP 4;
if the current file is in the same day, determining the residual capacity of the current file, determining whether the residual capacity can write the media data of the next preset period according to the media capacity of the previous preset period, and if so, returning to the writing operation; if the media file cannot be written, determining the duration of the media file which can be written, and executing the operation of closing the current file or returning the writing operation based on the duration;
if not, repeating the on-demand process when the first storage period is determined not to be expired; and if the second storage period is up, renaming the sub-directories of the dates under the channel directories, and moving all original files in the sub-directories under the dates to the corresponding sub-directories under the free file directories.
In the live broadcast process, when the computer program is executed by the processor, the computer program further executes:
inquiring timeliness of media slice merging files and index files under each channel directory at intervals of preset time;
and when the last modification time of the media slice merging file or the index file is determined to be larger than a preset threshold value, recovering the related files under the corresponding channel directories.
In the live broadcasting process, when the computer program is executed by the processor, the method further executes:
inquiring the number of media slice merging files and index files in a live idle file directory at intervals of preset time;
and when the number of the media slice merging files or the index files is less than a preset threshold value, creating corresponding files.
The present invention is described below with reference to scene embodiments.
This embodiment provides a live, on-demand system based on trade control field, includes: the system comprises seven main modules of preprocessing, camera media access, on-demand storage, live broadcasting caching, instant review and on-demand; wherein, the first and the second end of the pipe are connected with each other,
the preprocessing module is responsible for formatting the memory disc, creating a small number of files with different sizes according to the estimated number of channels, and carrying out '0' prewriting operation on the files so as to improve the writing speed;
the camera access module is responsible for analyzing an H.264 bare stream from a media stream of GBT 28181;
the on-demand storage module renames a date subdirectory of the last day of the previous period at 0 point 0 min 0 sec every day, moves a lower video file to an idle file directory, acquires a file with a proper size from the idle file directory when a writing requirement exists, and sequentially writes the contents of the MP4 files packaged by H.264 in the TM time period into the files;
the live broadcast module is responsible for segmenting the H.264 bare stream to generate a ts slicing file;
the live broadcast storage module is responsible for pre-distributing and initializing two media slice merging files, one is used for storing ts slices in the current TM time, and the other is used for storing ts slices in the last TM time;
the instant review module is responsible for reading the media data from the HFA and the HFB and returning the media data to the client;
and the on-demand module is responsible for reading the MP4 files in the specified time period from the storage files and sending the files to the client.
The preprocessing module of the embodiment of the invention comprises the following steps:
step one, creating an idle file directory, and creating sub-directories with names of 1G, 800M, 600M, 400M, 200M and 100M under the idle file directory. The directory name may be: the ratio of/mnt/storage/idle/1G;
secondly, a plurality of files with the total capacity being larger than the sum of the daily storage capacity of the channels are pre-created according to the estimated number of the channels, as the code rate of the monitoring video is basically between 2M and 8M, for example, the storage capacity of each day is about 21G, and the storage capacity of each day is 168G calculated by using the 8M code rate, the total capacity of the pre-created storage files is Volum = $ Camsum × 168G δ, wherein δ is an empirical parameter, the number of bytes additionally introduced by MP4 packaging is mainly considered, and δ is tentatively 1.01;
step three, determining the size and the number of files to be created according to the total daily storage capacity, creating the files in subdirectories with corresponding sizes under the free file directory, and performing pre-writing '\0' operation, wherein the file name naming rule $ size _ $ seq is as follows, for example: /mnt/storage/idle/1G/1G_1. The file sizes are 1G, 800M, 600M, 400M, 200M and 100M, wherein the number of files with different sizes is as follows:
FSum 1G ＝$Volum/1G；
FSum 800M ＝FSum 600M ＝FSum 400M ＝FSum 200M ＝FSum 100M ＝$CamSum*2。
and step four, when the number of files with various sizes in the free file directory is periodically detected to be less than $ Camsum x 2, corresponding files need to be created, the number of the files with various sizes is ensured to be equal to $ Camsum x 2, and the operation of '0' pre-writing is carried out, so that the real-time property of the written files is ensured.
The implementation steps of the on-demand storage module of the embodiment of the invention are as follows:
step one, after equipment starts to push flow, setting a sequence number $ sequence of MP4 to be 0;
and step two, when a day in the first storage period begins, creating a channel name directory for each channel, and creating a date subdirectory under the channel name directory. Subdirectories with names 1G, 800M, 600M, 400M, 200M, 100M are created under the date subdirectories.
Example of directory format is 65010000001327662615/20180405/800M/.
And step three, estimating the remaining storage capacity of the current day according to the equipment code rate and the remaining time of the current day, and if the remaining storage capacity of the current day is greater than 1G, transferring a 1G file from the free file directory to a 1G subdirectory under the current date directory. Otherwise, transferring the file with the corresponding size from the free file directory to the subdirectory with the corresponding size of the current storage directory. The method for calculating the residual storage capacity of the day according to the equipment code rate comprises the following steps: vol left ＝$Bitrate*T left * γ, where γ is the storage overhead introduced by the MP4 encapsulation, tentatively 1.01.
Step four, the on-demand storage module stores T M After the video in the time is packaged into the MP4, whether the residual storage capacity of the current file can be written into the MP4 media data is judged. If not, closing the current file, predicting the sum of the MP4 capacity and the residual capacity of the current day, transferring the file with the corresponding size from the idle file directory to the corresponding subdirectory of the current channel, and writing the file; if so, the current file is written sequentially. After the file is successfully written, an MP4 file name is generated, the file name naming mode is $ machien ID _ $ Date _ $ sequence.MP4, and the MP4 file name, the MP4 file length, the MP4 start-stop time, the storage file name and the MP4 section starting position are written into a database. $ sequence is increased by 1. Turning to the step five, namely, turning to the step five,
step five, checking the current time after the MP4 writing is finished each time, if the current time is not the same as the starting time of the MP4, jumping to the step ten, otherwise, jumping to the step six;
step six, calculating the residual capacity of the current file, and estimating whether the residual capacity can be written into the media data of the next TM time period according to the media capacity of the previous TM period, wherein the estimation method of the media capacity of the next TM time period is Vol Tm ’＝Vol Tm * Eta, wherein Vol Tm Medium capacity, vol, of last Tm period Tm ' is the medium capacity of the next Tm period, and η is an empirical value, and is generally set to 1.01 in consideration of the fluctuation of the code rate.
Step seven, if yes, the step four is carried out;
step eight, if not, estimating the time length of the media file which can be written, such as T M ', then the next MP4 package is only for T M ' and the data is packaged and written into the current file, and the estimation method comprises the following steps: t is M ’＝(Vol left *T M )/Vol Tm * Eta if T M ' greater than 10s, adding T M Temporarily set to T M ' go to step four;
step nine, if T M ' less than 10s, close the current file. Repeating the third step to the ninth step;
and step ten, if the first storage period is not expired, repeating two to nine, if the second storage period is reached, renaming sub-directories of $ Date- $ Duration time under each channel directory, and moving all files in the sub-directories under the sub-directories into corresponding sub-directories of the free file directory. And repeating the third step to the ninth step.
The mechanism for realizing live broadcast caching in the embodiment of the invention is as follows:
step one, when the system is started, a plurality of media slice merging files (merge files) and index files (m 3u8 files) are created in an idle file directory and are filled with '0' characters. Because the maximum code rate of video monitoring is 8192kbps, the size of merge files is set to Vol for convenient realization in consideration of the overhead of audio and TS headers merge ＝8192kps*T M * Eta/8, wherein eta is 1.2 ≈ 1280 × T M kB. In particular, when T M =300s, the merge file size is 384M. The size of the space pre-allocated for each ts slice is then: TS (transport stream) vol ＝1280*T ts KB, M3u8 index file is preset to 1M.
And step two, when a client requests the live broadcast of a certain channel, checking whether the current live broadcast exists, and if so, directly jumping to a live broadcast module for processing. If the live broadcast does not exist, checking whether the live broadcast exists in the live broadcast catalog or notAnd if the subdirectory named the channel does not exist, creating a channel directory, moving a merge file from the free file directory to the position below the channel directory, and renaming the file at the current time by using a naming rule: $ sequence _ $ unique _ live. Merge, move two m3u8 index files to the current directory, one for storing the ts slice index and duration of the current merge file, the naming convention, $ sequence _ $ unique _ replay.m3u8, one for storing the live ts slice index and duration, the naming convention, $ sequence _ $ unique _ live.m3u8. If there are subdirectories, the live m3u8 index file, the merge file with the earlier m3u8 time, and the review m3u8 file are renamed to the current time. Serial number ($ TS) of TS slice seq ) Initialize to 0, $ MEDIA-SEQUENCE is initialized to 0.
Step three, every T ts Time generates ts slice information, and the position of the ts slice information in the merge file is calculated firstly, and the calculation formula is as follows: TS (transport stream) loc ＝TS seq *TS vol . And then sequentially writing the TS slice information into the TS of the merge file loc Location, and generates TS slice filename, filename naming rule $ machien id $ merge filename $ TS loc Length, write this file name and duration to the live m3u8 index file, and update the MEDIA-SEQUENCE entry of the live m3u8 index file. Write back this file name and duration to the m3u8 index file. $ TS seq Increase by 1, $ MEDIA-SEQUENCE by 1.
Step four, T M3u8 When the time is up, adding Enlist to the currently processed m3u8 file to indicate that the index file is ended, and closing the merge file and the m3u8 file. If another merge file and review m3u8 file are available under the current directory, then the two files are renamed at the current time. Subsequent ts slices will be written to the new merge file and slice information will be written to the new m3u8 file. If there is only one merge file and one review m3u8 file under the current directory, then one merge file and one review m3u8 file are moved from the free file directory and the two files are renamed at the current time. And setting the serial number of the Ts slice to be 0, and repeating the second step, the third step until the live broadcast is finished.
And fifthly, inquiring timeliness of merge files and m3u8 files under the channel directories at certain time intervals, and if the last modification time of the files is larger than a threshold value, recycling the related files under the channel directories.
And step six, inquiring the number of merge files and m3u8 files in the idle file directory at intervals, and if the number of merge files and m3u8 files in the idle file directory is less than a preset threshold value, creating corresponding files.
In the embodiment of the present invention, for HLS live broadcast, the implementation process is as follows:
when a client requests direct broadcasting, the http server sends the index file information of the live m3u8 to the client, and when the client requests a ts slice file in the live m3u8 file, the http server reads media data from a merge file according to the name of the ts file and sends the media data to the client, and the judgment process is as follows:
the method comprises the steps of obtaining the name of a merge file according to the name of the $ merge file, determining the position of the merge file according to the $ TSloc, reading the number of bytes with the length of $ and sending the number of bytes to a server side.
In the embodiment of the invention, for the instant review request of the user, the implementation process is as follows:
instant review refers to the user wanting to see the first few seconds or minutes of video during the live broadcast. When the user requests the distance T from the current time M And when the client performs playback within the time, the client sends a playback request to the current live broadcast server, and the live broadcast server reads corresponding ts slice file names and durations according to the time period selected by the user and the starting time of two playback m3u8 index files in the channel directory, splices playback m3u8 file information in the corresponding time period and sends the information to the client.
The client requests ts files in the m3u8 index file, the http server reads media data from the merge file according to the ts file name and sends the media data to the client, and the judgment process is as follows:
obtaining the name of merge file according to $ merge file name, according to $ TS loc And determining the position of the terminal in the merge file, reading the byte number of the size of $ length, and sending the byte number to the server side.
In the embodiment of the present invention, for the on-demand request of the user, the implementation process is as follows:
and inquiring the database to obtain the MP4 files in the time period requested by the user, and sending the MP4 file names and the start-stop time meeting the conditions to the client.
The client accesses the requested MP4 file, and the http server acquires the MP4 file information according to the MP4 storage file name, the starting position and the file length and sends the MP4 file information to the client.
The embodiment of the invention adopts three layers of directories to manage and store the files, when the storage period is up, the files do not need to be deleted, only the date subdirectory under the corresponding channel directory needs to be renamed, and the original files in each subdirectory under the date are moved to the corresponding subdirectory under the idle file directory, thereby not only realizing the rapidness, but also reducing the fragments caused by the file deletion. In addition, in the live broadcasting process, the two preset media slices are recycled to merge the file storage media slices, so that the number of small files is reduced, the maintenance cost is reduced, and disk fragments cannot be introduced due to file deletion.
According to the estimated channel quantity, files with the corresponding number and size of the total capacity larger than the sum of the daily storage capacity of the channels are created; in the later period, the number of idle files in each subdirectory is regularly checked, and when the number of the idle files is determined to be smaller than a threshold value, corresponding files are created; the method not only ensures the preprocessing speed, but also can not cause performance overhead caused by the fact that files are not used enough to create the files by the write thread, and the size of the stored files is not fixed and can be dynamically adjusted according to the needs, so that the storage loss caused by file switching can be reduced.
In addition, the media storage in the embodiment of the invention adopts MP4 encapsulation, thereby reducing the storage overhead brought by TS header, determining the review mode according to the difference value between the request time and the current time, and ensuring that the video information of the latest 3s can be seen.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention.
Claims (10)
1. A method of file management in a monitoring system, the method comprising:
storing the monitoring file based on three layers of directories including a channel, a date and a file size;
when determining that a monitoring file with a preset size needs to be added, transferring the monitoring file from a subdirectory with the corresponding size of an idle directory to a corresponding subdirectory on the current date; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory;
two preset media slices are recycled in the live broadcast process, and the media slices are stored in a file merging file; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period.
2. The method of claim 1, further comprising:
creating files with corresponding number and size of total capacity larger than the sum of daily storage capacity of the channels according to the estimated number of the channels;
the file is pre-written \0' and moved to the corresponding subdirectory.
3. The method of claim 2, further comprising:
regularly checking the number of idle files under each subdirectory;
and when the number of the idle files is smaller than the threshold value, creating corresponding files and preprocessing the files.
4. The method of claim 1, wherein the on-demand process further comprises:
after the video in the preset period is packaged into MP4, judging whether the residual storage capacity of the current file can be written into the MP4 media data;
if not, closing the current file, determining the sum of the MP4 capacity and the residual capacity of the current day, transferring the file with the corresponding size from the idle file directory to the corresponding subdirectory of the current channel, and writing the file; if so, the current file is written sequentially.
5. The method of claim 4, wherein after the sequentially writing the current file, the method further comprises:
determining whether the writing and the starting time of the MP4 are in the same day;
if the current file is in the same day, determining the residual capacity of the current file, determining whether the residual capacity can be written into the media data of the next preset period according to the media capacity of the previous preset period, and if so, returning to the writing operation; if the time length of the media file which can be written is not the same as the time length of the media file which can be written, and the operation of closing the current file or returning to the writing operation is executed based on the time length;
if the current storage period is not the same day, when the first storage period is determined not to be expired, the on-demand process is repeated; and if the second storage period is up, renaming the date subdirectories under the channel directories, and moving all the original files in the subdirectories under the date to the corresponding subdirectories under the free file directory.
6. The method of claim 1, wherein the method further comprises, during the live broadcast:
inquiring timeliness of media slice merging files and index files under each channel directory at intervals of preset time;
and when the last modification time of the media slice merging file or the index file is determined to be larger than a preset threshold value, recovering the related files under the corresponding channel directories.
7. The method of claim 1, wherein the method further comprises, during the live broadcast:
inquiring the number of media slice merging files and index files in a live idle file directory at intervals of preset time;
and when the number of the media slice merging files or the index files is less than a preset threshold value, establishing corresponding files.
8. A file management apparatus in a monitoring system, the apparatus comprising:
the storage module is used for storing the monitoring file based on three layers of directories including a channel, a date and a file size;
the processing module is used for transferring the monitoring files from the subdirectories with the corresponding size of the idle directory to the corresponding subdirectories under the current date when determining that the monitoring files with the preset size need to be added; when the storage period is determined to be reached, renaming the date subdirectories under the corresponding channel directories, and moving original files in each subdirectory under the date to the corresponding subdirectory under the free file directory;
the live broadcast processing module is used for circularly using two preset media slices in the live broadcast process to merge the file storage media slices; one file is used for storing the media slices in the current preset period, and the other file is used for storing the media slices in the last preset period.
9. A file management apparatus in a monitoring system, the apparatus comprising: a processor and a memory for storing a computer program capable of running on the processor,
wherein the processor is adapted to perform the steps of the method of any one of claims 1-7 when running the computer program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811235338.XA CN111090617B (en) | 2018-10-23 | 2018-10-23 | File management method and device in monitoring system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811235338.XA CN111090617B (en) | 2018-10-23 | 2018-10-23 | File management method and device in monitoring system and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111090617A CN111090617A (en) | 2020-05-01 |
| CN111090617B true CN111090617B (en) | 2023-03-31 |
Family
ID=70391299
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811235338.XA Active CN111090617B (en) | 2018-10-23 | 2018-10-23 | File management method and device in monitoring system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111090617B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117472297B (en) * | 2023-12-28 | 2024-04-16 | 深圳万物安全科技有限公司 | Method, device and storage medium for prolonging storage time of video file |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7761497B1 (en) * | 2001-07-13 | 2010-07-20 | Vignette Software, LLC | Storage medium having a manageable file directory structure |
| CN101539884B (en) * | 2009-04-30 | 2012-07-11 | 杭州华三通信技术有限公司 | Method and device for data storage |
| CN105589662B (en) * | 2015-12-21 | 2018-10-02 | 上海华测导航技术股份有限公司 | A kind of GNSS receiver data storage and management method |
| CN107341227B (en) * | 2017-07-01 | 2020-12-25 | 中国科学院计算机网络信息中心 | File processing method, server and computer readable storage medium |
-
2018
- 2018-10-23 CN CN201811235338.XA patent/CN111090617B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111090617A (en) | 2020-05-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102240557B1 (en) | Method, device and system for storing data | |
| US10803016B2 (en) | Predictive models of file access patterns by application and file type | |
| US7134041B2 (en) | Systems and methods for data backup over a network | |
| CN110213672A (en) | Video generation, playback method, system, device, storage medium and equipment | |
| CN102725755B (en) | Method and system of file access | |
| CN105303456A (en) | Method for processing monitoring data of electric power transmission equipment | |
| CN111198856B (en) | File management method, device, computer equipment and storage medium | |
| CN101510838B (en) | Method for implementing perdurable data queue | |
| KR20140024262A (en) | Apparatus and method for transmitting live media content | |
| US8060697B1 (en) | Dynamically allocated secondary browser cache | |
| CN102761614A (en) | Method and system for realizing breakpoint resume of network data transmission | |
| CN102902724A (en) | Mass raster tile map release method | |
| CN111090617B (en) | File management method and device in monitoring system and computer readable storage medium | |
| US11409720B2 (en) | Metadata reduction in a distributed storage system | |
| CN102523301A (en) | Method for caching data on client in cloud storage | |
| CN102104581A (en) | Network karaoke on-demand system and method thereof | |
| CN109062500B (en) | Metadata management server, data storage system and data storage method | |
| US20180203636A1 (en) | Likelihood of access based object storage in a cloud environment | |
| CN110737389A (en) | Method and device for storing data | |
| CN101848366B (en) | Filing and program broadcasting method and equipment of time shift slicing files in IPTV (Internet Protocol Television) system | |
| CN105677579A (en) | Data access method and system in cache system | |
| JP2023531751A (en) | Vehicle data storage method and system | |
| CN111158606A (en) | Storage method, storage device, computer equipment and storage medium | |
| US10915270B2 (en) | Random file I/O and chunked data upload | |
| EP3293647A1 (en) | Methods of storing essence data in media file systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |