CN108124112B - Performance improving method for multi-path storage stream media data - Google Patents
Performance improving method for multi-path storage stream media data Download PDFInfo
- Publication number
- CN108124112B CN108124112B CN201711457166.6A CN201711457166A CN108124112B CN 108124112 B CN108124112 B CN 108124112B CN 201711457166 A CN201711457166 A CN 201711457166A CN 108124112 B CN108124112 B CN 108124112B
- Authority
- CN
- China
- Prior art keywords
- file
- streaming media
- data
- path
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/78—Television signal recording using magnetic recording
- H04N5/781—Television signal recording using magnetic recording on disks or drums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/91—Television signal processing therefor
Abstract
The invention provides a performance improvement method of multi-path storage streaming media data, which is characterized in that when a storage device is initialized, original fragments of the storage device are removed, and the size and the number of each video file are calculated according to the storage capacity and the set video recording time and code rate; then, establishing a corresponding number of streaming media files for each path, and allocating storage space according to the size of each video file; directly selecting an oldest streaming media file to perform renaming for circular coverage when the storage space is full; when data is written, the data is firstly copied into a cache pool managed by an application, a background thread is established to be responsible for directly writing back to a storage device by directly bypassing the cache of an operating system in a direct-io mode, and the performance is improved by filling 0 and modifying the size of a file.
Description
Technical Field
The invention relates to the field of multi-path streaming media storage, in particular to a method for improving the performance of multi-path stored streaming media data.
Background
The existing multi-path streaming media storage is stored in a storage medium through a file interface provided by an operating system, each file usually stores only fixed-time (adjustable) streaming media data, and after the space is full, the oldest file is deleted to release the storage space, which has the following disadvantages:
1. due to the fact that multiple paths of concurrent writing are conducted, storage space needs to be allocated for each writing, and the storage space is concurrently allocated, so that file fragments are brought, and the performance of the file fragments is reduced;
2. after the storage space is full, the old file needs to be deleted to release the space, which causes that deletion and writing can occur concurrently, file fragmentation can be caused, and the performance is reduced;
3. when disk spaces are allocated and released concurrently, mutual exclusion is needed to be performed in order to ensure data consistency, and uncontrollable performance is caused by uncontrollable mutual exclusion (which way of mutual exclusion is executed first cannot be determined);
4. when the performance is reduced, the cache occupation of the file system is increased, and the cost is increased under an embedded platform;
5. the streaming media data can be cached by a file of an operating system, and the operating system can write back the streaming media data to the memory only when specific conditions are met, and the write-back mechanism cannot necessarily meet the requirement of multi-path concurrent streaming media;
6. when an operating system writes back a cache, two types of data are provided, namely file system metadata (mainly indexes or bitmaps generated when a storage space is allocated, such as fat in a fat file system and bitmap in an ntfs file system) and streaming media data, the two types of data are usually far away from each other, under the condition of cross-writing of the two types of data, most storage devices cannot achieve the best performance (the hard disk needs seek overhead; and flash type stores mapping and garbage recycling overhead).
As disclosed in 2014-04-02, chinese invention with publication number CN103702057A discloses a block storage algorithm adapted to multiple concurrent write-in streaming media data, which bypasses the file system, so that the final streaming media file can only be accessed through the specially developed software, and is inconvenient to view, play and distribute, and poor in compatibility; in addition, the method must divide a fixed-size area on the storage device, and the area can only be dedicated to video recording and cannot be used for other purposes.
2014-11-19, chinese invention publication No. CN104159058A discloses an efficient method for simultaneous recording of multiple media streams, which also results in poor compatibility due to bypassing of the file system.
As disclosed in 2010-06-16, chinese invention with publication number CN101742263A discloses a method for storing video surveillance record data, the data format of the file is dedicated and can be accessed by dedicated software, which is not compatible with third party application; in addition, the video recording time of each file is fixed and cannot be switched; finally this does not consider the case of multi-way concurrency and cache-controlled write back.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a method for improving performance of multi-path stored streaming media data, which mainly improves performance by avoiding generation of file fragments and controlling the time for writing back data, and has high compatibility.
The invention is realized by the following steps: a method for improving performance of multi-path storage stream media data is characterized in that: the method comprises an initial formatting process of the storage device, a renaming covering process after the storage space is full and a processing process of writing data:
the storage device initial formatting process is as follows: removing original fragments of the storage equipment, and calculating the size fsize and the number fnum of each video file according to the storage capacity and the set video recording time and code rate; then, creating a number fnum of streaming media files obtained by calculation for each path, and allocating a storage space for each file according to the size fsize of each video file obtained by calculation after the file is created;
the renaming covering process after the storage space is full is as follows: when the storage space is full, the old streaming media file is not deleted, but the oldest streaming media file is directly selected for renaming, and new streaming media data are written into the file after renaming to achieve circular coverage;
the processing flow of writing data is as follows: when data is written, the data is firstly copied into a cache pool managed by an application, a background thread is created to be responsible for directly bypassing the cache of an operating system and directly writing the data back to a storage device at a proper time in a direct-io mode, and because the data of the direct-io needs to be aligned, the data at the tail of a file can be supplemented by 0, and finally the size of the file can be modified (namely, the metadata of the file system is directly modified, so that the performance cost is low), but the supplemented 0 is not included.
Further, the formula for calculating the size fsize and the number fnum of each video file is as follows:
fsize＝b*t；
fnum＝c/fsize。
wherein b is the estimated average bit rate of the path streaming media, t is the recording time, c is the storage space allocated to the path, and fnum is the number of the path streaming media files.
Further, the control method of the appropriate timing is:
writing back is firstly carried out according to a preset time period t _ interval, if the writing back of the data takes time
If t _ write > t _ interval, the time period t _ interval is modified to t _ write/p, where p is a constant;
if the equipment is abnormally powered off and the data in the cache pool is lost, the control is carried out by setting a maximum allowed data loss time t _ loss at the upper layer, and if t _ loss is greater than 0, t _ interval is reset to min (t _ loss, t _ interval).
Furthermore, the invention also comprises a video recording time flow modification step;
the video recording time modifying flow comprises the following steps: and if the video recording time needs to be modified, selecting the oldest streaming media file for merging or splitting, and only modifying the metadata of the file system without moving the streaming media data during merging and splitting.
Furthermore, the invention also comprises an adjusting process of the storage space of each path of streaming media;
the adjustment process of the storage space of each path of streaming media comprises the following steps: if the storage space of each path of streaming media is to be adjusted, the path with reduced space is selected to be the oldest required streaming media file, a new file is merged, and then the new file is split into the streaming media files of the path with increased space.
The invention has the following advantages:
1. space is allocated when a file is created, so that only streaming media data need to be written without writing metadata (index or bitmap of storage space) of a file system during video recording, file fragments are avoided, and data ping-pong is prevented;
2. when the space is full, the name is changed, and file fragments and multipath mutual exclusion are avoided;
3. the video recording time is adjusted, and the old data does not need to be formatted or deleted when the space is stored;
4. only the metadata is modified when the files are merged and split, so that the performance is improved;
5. the opportunity of writing back data is controlled by the application layer, and the performance is controllable.
Drawings
The invention will be further described with reference to the following examples with reference to the accompanying drawings.
Fig. 1 is a general flowchart of a process of multi-path storing streaming media data according to the present invention.
FIG. 2 is a flow chart illustrating the process of initializing the memory according to the present invention.
FIG. 3 is a flow chart illustrating the execution of the concurrent store to cache pool process of the present invention.
FIG. 4 is a flowchart illustrating the write back process according to the present invention.
Detailed Description
Referring to fig. 1, the process of multi-path storing streaming media data of the present invention includes the steps of initializing a memory, concurrently storing, writing back data, and the like. To avoid performance degradation, file fragmentation is avoided, and concurrent allocation and freeing of space is avoided; to avoid the uncontrollable write-back opportunity, the file cache of the operating system needs to be bypassed. Therefore, the method for improving the performance of the multi-path storage streaming media data of the invention is to perform some relevant processing in these steps.
Referring to fig. 2 to 4, the method for improving performance of multi-path storage streaming media data of the present invention includes an initial formatting process of a storage device, a renaming covering process after a storage space is full, and a processing process of writing data:
the storage device initial formatting process is as follows: as shown in fig. 2, in the process of initializing the memory, the original fragments of the storage device are removed, and the size fsize and the number fnum of each video file are calculated according to the storage capacity and the set video recording time and code rate; then, creating a number fnum of streaming media files obtained by calculation for each path, and allocating a storage space for each file after the creation according to the size fsize of each video file obtained by calculation, so that the storage spaces of all the streaming media files are continuous and have no fragments; the formula for calculating the size fsize and the number fnum of each video file is as follows:
fsize＝b*t；
fnum＝c/fsize。
wherein b is the estimated average bit rate of the path streaming media, t is the recording time, c is the storage space allocated to the path, and fnum is the number of the path streaming media files.
The renaming covering process after the storage space is full is as follows: when the storage space is full, the old streaming media file is not deleted, but the oldest streaming media file is directly selected for renaming, and new streaming media data are written into the file after renaming to achieve circular coverage; the process of deleting old files to release space is omitted, so that the concurrent deletion and writing cannot occur, and the generation of file fragments cannot be caused.
The processing flow of writing data is as follows: as shown in fig. 3 and 4, when writing data, copying the data into a cache pool managed by an application, and creating a separate background thread to be responsible for directly writing the data back to the storage device at a proper time by directly bypassing the cache of the operating system in a direct-io manner, because the data of the direct-io needs to be aligned, the data at the tail of the file will be supplemented with 0, and finally the file size will be modified (i.e., the metadata of the file system is directly modified, the performance overhead is small), but the supplemented 0 is not included. Wherein the control method of the proper time is as follows:
(1) firstly, writing back according to a preset time period t _ interval (for example, t _ interval is 1s, the adjustment is performed according to the size of the memory, and the memory occupied by the cache pool is smaller), if the data write-back time t _ write > t _ interval, the time period t _ interval is modified to t _ write/p, wherein p is
Is a constant, such as 2, 3, etc.;
(2) if the equipment is abnormally powered off and the data in the cache pool is lost, the control is carried out by setting a maximum allowed data loss time t _ loss at the upper layer, and if t _ loss is greater than 0, t _ interval is reset to min (t _ loss, t _ interval).
The storage device can reach the best performance by directly bypassing the cache of the operating system and directly writing back the cache to the storage device through a single background thread in a proper time machine so as to meet the requirement of multi-path concurrent streaming media, wherein the file size is modified to be better compatible with third-party players, and since some third-party players encounter the additional 0 data when playing certain video formats, an exception may occur, so that the exception can be eliminated by directly modifying the size in the metadata of the file system.
When the small files are copied, the consumed time is short, the occupied space for making the evidence is small, and the advantage of the large files is that the speed is high when the videos are searched, so that most of the automobile data recorders have the function of controlling the video recording time of each video file. Therefore, the invention also comprises a video recording time flow modification step;
the video recording time modifying flow comprises the following steps: and if the video recording time needs to be modified, selecting the oldest streaming media file for merging or splitting, and only modifying the metadata of the file system without moving the streaming media data during merging and splitting.
If there are multiple cameras, the initial camera a is 720p, and the initial camera a is changed to a 1080p camera, or even the camera is not changed, but the resolution or different image quality modes are switched, so that the code rate is changed, the original 10g can record for 2 hours, and now only can record for 1 hour, at this time, the client may want to adjust the storage space of each path of streaming media. Therefore, the method of the invention also comprises the adjustment flow of the storage space of each path of streaming media;
the adjustment process of the storage space of each path of streaming media comprises the following steps: if the storage space of each path of streaming media is to be adjusted, the path with reduced space is selected to be the oldest required streaming media file, a new file is merged, and then the new file is split into the streaming media files of the path with increased space.
The invention has the following advantages:
1. space is allocated when a file is created, file fragments are avoided, and data ping-pong is prevented;
2. when the space is full, the name is changed, and file fragments and multipath mutual exclusion are avoided;
3. the video recording time is adjusted, and the old data does not need to be formatted or deleted when the space is stored;
4. only the metadata is modified when the files are merged and split, so that the performance is improved;
5. the opportunity of writing back data is controlled by the application layer, and the performance is controllable.
Although specific embodiments of the invention have been described above, it will be understood by those skilled in the art that the specific embodiments described are illustrative only and are not limiting upon the scope of the invention, and that equivalent modifications and variations can be made by those skilled in the art without departing from the spirit of the invention, which is to be limited only by the appended claims.
Claims (5)
1. A method for improving performance of multi-path storage stream media data is characterized in that: the method comprises an initial formatting process of the storage device, a renaming covering process after the storage space is full and a processing process of writing data:
the storage device initial formatting process is as follows: removing original fragments of the storage equipment, and calculating the size fsize and the number fnum of each video file according to the storage capacity and the set video recording time and code rate; then, creating a number fnum of streaming media files obtained by calculation for each path, and allocating a storage space for each file according to the size fsize of each video file obtained by calculation after the file is created;
the renaming covering process after the storage space is full is as follows: when the storage space is full, the old streaming media file is not deleted, but the oldest streaming media file is directly selected for renaming, and new streaming media data are written into the file after renaming to achieve circular coverage;
the processing flow of writing data is as follows: when data is written, the data is firstly copied to a cache pool managed by an application, a background thread is created to be responsible for directly bypassing the cache of an operating system and directly writing the data back to a storage device in a direct-io mode, the data at the tail of a file can be filled with 0, and finally the size of the file can be modified, but the size of the file does not include the filled 0.
2. The method of claim 1, wherein the performance of the multi-way storage streaming media data is improved by: the formula for calculating the size fsize of each video file and the number fnum of the path flow media files is as follows:
fsize＝b*t；
fnum＝c/fsize；
wherein b is the estimated average bit rate of the streaming media of the path, t is the recording time of each file, and c is the storage space allocated to the path.
3. The method of claim 1, wherein the performance of the multi-way storage streaming media data is improved by: in the processing flow of the write data, the background thread is responsible for directly bypassing the cache of the operating system and directly writing back to the storage device at a proper time in a direct-io mode, wherein the control method at the proper time comprises the following steps:
writing back according to a preset time period t _ interval, and if the data writing back takes t _ write > t _ interval, modifying the time period t _ interval to t _ write/p, wherein p is a constant;
if the equipment is abnormally powered off, the data in the cache pool can be lost, so the control is carried out by setting a maximum allowed data loss time t _ loss at the upper layer, and if t _ loss >0, t _ interval is reset to min (t _ loss, t _ interval).
4. A method for improving the performance of multi-way storage streaming media data according to any one of claims 1 to 3, wherein: the method also comprises a video recording time flow modification;
the video recording time modifying flow comprises the following steps: and if the video recording time needs to be modified, selecting the oldest streaming media file for merging or splitting, and only modifying the metadata of the file system without moving the streaming media data during merging and splitting.
5. A method for improving the performance of multi-way storage streaming media data according to any one of claims 1 to 3, wherein: the method also comprises the adjustment flow of the storage space of each path of streaming media;
the adjustment process of the storage space of each path of streaming media comprises the following steps: if the storage space of each path of streaming media is to be adjusted, the path with reduced space is selected to be the oldest required streaming media file, a new file is merged, and then the new file is split into the streaming media files of the path with increased space.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711457166.6A CN108124112B (en) | 2017-12-28 | 2017-12-28 | Performance improving method for multi-path storage stream media data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711457166.6A CN108124112B (en) | 2017-12-28 | 2017-12-28 | Performance improving method for multi-path storage stream media data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108124112A CN108124112A (en) | 2018-06-05 |
| CN108124112B true CN108124112B (en) | 2020-04-03 |
Family
ID=62232116
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711457166.6A Active CN108124112B (en) | 2017-12-28 | 2017-12-28 | Performance improving method for multi-path storage stream media data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108124112B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110045918B (en) * | 2018-12-03 | 2021-09-03 | 蚂蚁金服（杭州）网络技术有限公司 | Efficient data unit reusing method and system |
| CN110825697B (en) * | 2019-11-07 | 2022-12-20 | 珠海研果科技有限公司 | Method and apparatus for formatting a storage device |
| CN111506271B (en) * | 2020-06-29 | 2020-10-09 | 南京鹏云网络科技有限公司 | Data segment unit transmission differentiation management method and distributed block storage system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7987384B2 (en) * | 2008-02-12 | 2011-07-26 | International Business Machines Corporation | Method, system, and computer program product for handling errors in a cache without processor core recovery |
| CN102103740B (en) * | 2010-12-20 | 2013-01-02 | 福州瑞芯微电子有限公司 | Method and circuit for processing written address and width unaligned image |
| US8850125B2 (en) * | 2011-10-25 | 2014-09-30 | Cavium, Inc. | System and method to provide non-coherent access to a coherent memory system |
| CN105867836A (en) * | 2015-01-23 | 2016-08-17 | 中兴通讯股份有限公司 | Storage management method and apparatus as well as stream media system |
| CN104966528A (en) * | 2015-06-25 | 2015-10-07 | 广东工业大学 | Multi-path audio/video stream storage method for preventing disk fragments from formation |
-
2017
- 2017-12-28 CN CN201711457166.6A patent/CN108124112B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN108124112A (en) | 2018-06-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7447836B2 (en) | Disk drive storage defragmentation system | |
| CN108124112B (en) | Performance improving method for multi-path storage stream media data | |
| US8578431B2 (en) | Adaptive timeshift service | |
| US7660837B2 (en) | Method for automatically managing disk fragmentation | |
| US7096327B2 (en) | System for and method of accessing blocks on a storage medium | |
| US10725925B2 (en) | Metadata-specific cache policy for device reliability | |
| US7613892B2 (en) | Recording device, recording method, recording medium, and program | |
| US8750679B2 (en) | Information processing apparatus, recording method, and computer program | |
| US9036982B2 (en) | Recording apparatus | |
| US9336293B2 (en) | Data management apparatus and method for surveillance system | |
| US8416658B2 (en) | Recording apparatus and recording method for data and file system information | |
| US8625968B2 (en) | Recording apparatus and control method thereof | |
| US7924456B1 (en) | Data distribution and buffering | |
| KR20060120674A (en) | Information recording medium data processing apparatus and data recording method | |
| US6898610B2 (en) | Apparatus and method for space allocation of image and audio information | |
| US8730348B2 (en) | Recording apparatus and control method thereof | |
| KR102094786B1 (en) | File system and method of storing files based on the file system | |
| US8655139B2 (en) | Video recording and reproducing system and reading method of video data | |
| EP1362290A1 (en) | Device and method for managing the access to a storage medium | |
| JP4847672B2 (en) | Method and apparatus for recording multimedia digital data | |
| CN110214352B (en) | Recording method adapted to wear level of storage medium | |
| US6496311B1 (en) | Data recording/reproducing unit, data recording method, data reproducing method, data recording/reproducing method, hard disk controller, and AV-data recording/reproducing method | |
| EP2256648A1 (en) | Method for storing data files, method for reading data content, and data store | |
| KR100379573B1 (en) | A file system for recording and displaying a digital broadcasting data | |
| JP2005032382A (en) | Disk recording device and its file erasing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 350000 building, No. 89, software Avenue, Gulou District, Fujian, Fuzhou 18, China Patentee after: Ruixin Microelectronics Co., Ltd Address before: 350000 building, No. 89, software Avenue, Gulou District, Fujian, Fuzhou 18, China Patentee before: Fuzhou Rockchips Electronics Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder |