CN107870973B

CN107870973B - File storage system capable of accelerating simultaneous playback of multiple paths of monitoring

Info

Publication number: CN107870973B
Application number: CN201710855634.9A
Authority: CN
Inventors: 周水平; 黎华健
Original assignee: Guangdong Dongwait Information Technology Co ltd
Current assignee: Guangdong Dongwait Information Technology Co ltd
Priority date: 2017-09-20
Filing date: 2017-09-20
Publication date: 2021-11-26
Anticipated expiration: 2037-09-20
Also published as: CN107870973A

Abstract

The invention discloses a file storage system for accelerating simultaneous playback of multipath monitoring, which carries out formatting operation on a disk when the file storage system is used for the first time, formats the disk into 4 partitions of a system attribute area, a date index area, a file index area and a data block area, resets the file format of the partitions, retrieves the first-level index area through time period parameters by directly reading and writing the file storage system when recording the video, obtains index records of the time period and the date, retrieves the second-level index area through the recorded offset address field value, obtains the initial offset addresses of all video data blocks of the time period corresponding to the date, and takes out the data and returns the data to a client; the first-level index area takes date as a retrieval standard, the second-level index area takes block timestamp and block type as the retrieval standard, and the purpose of rapid data retrieval is achieved according to the setting of the second-level index content. The invention can effectively solve the problem of slow speed of hard disk storage, read-write, search and positioning during the playback operation of the multi-channel monitoring equipment.

Description

File storage system capable of accelerating simultaneous playback of multiple paths of monitoring

Technical Field

The invention relates to the field of audio and video monitoring and file storage systems, in particular to a file storage system for accelerating simultaneous playback of multiple paths of monitoring.

Background

The file storage systems such as FAT32, NTFS and the like of the traditional Windows platform can meet the requirements when operating for storage, reading and writing, searching, inquiring, exchanging and the like with small data volume, along with increasingly complex social security and protection conditions, video monitoring engineering becomes an important guarantee of social security, video monitoring equipment and monitoring data grow in a geometric grade, and when operating for storage, reading and writing, searching and the like of large or even massive data, for example, when operating for simultaneous playback of multiple paths of monitoring equipment, the hard disk data storage and reading and writing speed of the traditional file storage system is low, the efficiency is low or even the traditional file storage system cannot operate; meanwhile, the more files stored in the traditional file storage system, the greater and greater management burden is imposed on the sector area, and the more files are stored, the slower the storage speed is, so that the requirements of rapidly viewing video resources and playing back videos in some major cases cannot be met.

At present, in order to facilitate monitoring data retrieval and data recovery, many file storage systems store data in units of blocks, and establish one-level or multi-level indexes to achieve the purpose of fast storage.

Disclosure of Invention

In order to solve the problems in the prior art, the invention provides a file storage system for accelerating the simultaneous playback of multi-channel monitoring, which adopts the scheme of storing and retrieving video data in a file storage system format in a bare disc, establishes self-defined primary and secondary index contents and formats,

it is desirable to speed up the server's processing of requests for simultaneous playback by multiple devices.

The purpose of the invention is realized by at least one of the following technical solutions.

The invention provides a user-defined storage file storage system for accelerating simultaneous playback of multiple paths of monitoring equipment videos, wherein the minimum hard disk space of the file storage system is 1GB, and the maximum space is not limited.

The invention provides a file storage system for accelerating simultaneous playback of multipath monitoring, which is a file storage system special for storing monitoring videos in the security industry, wherein a disk is formatted when the file storage system is used for the first time, and system partitions comprise a system attribute area, a date index area (namely a primary index area), a file index area (namely a secondary index area) and a data block area; when the video is played back, the file storage system in the disk array is directly read and written without passing through a windows file storage system, and the specific video data is quickly positioned by indexing the content in the primary index area and the secondary index area, so that the aim of quick playback is fulfilled.

Furthermore, the service logic of the file storage system is to cover from the beginning after the disk is full, and the video in a certain period of time is not deleted from the middle of the disk, and disk fragments are not generated.

Furthermore, the system attribute AREA is marked with a ". around _ AREA" string, occupies a storage space of 512 bytes, and includes system creation time (8 bytes), a version number (2 bytes), a system total space size (8 bytes), a system remaining space size (8 bytes), an attribute AREA size (4 bytes), a date index AREA size (4 bytes), a file index AREA size (4 bytes), a data block AREA size (8 bytes), a single data block size (4 bytes), a date index AREA free AREA start block address (4 bytes), a file index AREA free AREA start block address (4 bytes), and a data block AREA free AREA start block address (8 bytes).

Furthermore, the DATE INDEX AREA (primary INDEX AREA) and the FILE INDEX AREA (secondary INDEX AREA) are respectively divided into a plurality of equal partitions with the same size, the start of the equal partitions is marked by strings of 'DATE _ INDEX _ AREA' and 'FILE _ INDEX _ AREA', and the total size of the two INDEX AREAs is determined by a partition algorithm; the format of the primary index area is a date stamp (4 bytes), the date stamp corresponds to the first address (4 bytes) of the first file in the date in the file index area (secondary index area), and the first address of the first file in a certain date in the secondary index area is used for being quickly positioned.

Furthermore, the FILE INDEX AREA, i.e. the secondary INDEX AREA, is marked with a string ". FILE _ INDEX _ AREA", and stores an AREA of FILE INDEX information for quickly locating the video data of a certain monitoring device in a certain day and a certain period of time, the content of the secondary INDEX AREA is core search content for quick search, and the format and size of each INDEX in the secondary INDEX AREA include a start date timestamp (8 bytes), an end date timestamp (8 bytes), a video type (1 byte), a stream format (13 bytes), a device ID (20 bytes), and a FILE header address (8 bytes) of a corresponding data block AREA for quickly locating the header address of the FILE in the data block AREA.

Furthermore, the DATA BLOCK AREA is marked with a string of 'DATA _ BLOCK _ AREA', the video DATA stream file is stored in units of BLOCKs, the first DATA BLOCK of the file contains file header information in front, the file header information is used for recovering the file storage system due to accidental damage of the index AREA, and the size of the file storage system is determined by a partitioning algorithm; the data block format comprises a file header (occupying 50 bytes, the first data block of each file has the part of data), a data block (the fixed size is stored in a system attribute area), and a head address (8 bytes) of the next data block, the data block format is used for storing a binary video stream data block, and the data blocks are connected through a block tail address; except for the file header, the data block area occupies the same storage space.

Further, when the space of the disk is firstly used, the space is formatted into four partitions, after the formatting is completed, the space is locked into a self-defined file storage system format, the four partitions are arranged in sequence, and when the remaining space of the file storage system is too small, the replacement is covered from the beginning.

Furthermore, when the file storage system writes data, a buffer space with a fixed size is taken out from the memory pool by the current thread for the monitoring equipment currently processed and is used for continuously storing the frame data of the equipment until the data block area is full, the data block area is written into the file storage system once again, the data blocks of the file storage system are connected and next to each other by a block tail structure, no disk fragments are generated, the utilization rate of the disk space is saved, and after the data is written, the buffer space is returned to the memory pool for use in the next processing.

Further, when it is detected that the remaining storage space of the file storage system is written into the last data block, overwriting storage is performed from the earliest date by default in units of blocks, and when part of data is important and cannot be overwritten, the corresponding data block is marked with a lock, so that the overwrite strategy automatically identifies the lock mark and skips over the lock area for erasing.

Furthermore, when video playback is carried out, the efficiency evaluation of retrieving the video data by the client according to time periods is carried out, the video data stored in the file storage system can be retrieved and downloaded by the client according to any time period, when the client requests the video playback, the client directly reads and writes the file storage system in the disk array without passing through a windows file storage system, and the specific video data are quickly positioned through indexes at all levels, so that the aim of quickly playing back is fulfilled; the parameters to be transmitted to the server of the management file storage system comprise equipment ID, video recording time period and video recording type, the server retrieves the primary index area through the time period parameters to obtain index records of all dates of the time period, then retrieves the secondary index area through the offset address field value in the retrieved records to obtain the initial offset addresses of all video recording data blocks in the data area of the time period corresponding to the date, and takes out the data to return to the client.

Compared with the prior art, the invention has the following advantages: compared with the traditional windows file storage system, the system has the following characteristics:

1. adopting a multi-level index mode of dates and files: this file storage system adopts the mode of two-stage index to fix a position the video file, date index district (one-level index district) and file index district (second grade index district), its form and size of second grade index district include the initial date time stamp, end date time stamp, the video type, the stream format, equipment ID and the file head initial address that corresponds the data block district, a first address for fix a position this file in data block district fast, the file type is single, thereby index district structure is very simple, the space that occupies is very little, can accelerate the retrieval speed of file.

2. Covering mode no magnetic disc fragment: because the file storage system does not delete the service logic of a certain period of time video from the middle of the disk, but covers from the beginning after the disk is full, disk fragments are not generated like other file storage systems, the utilization rate of the disk is improved, and the reading speed of the disk is accelerated.

Drawings

FIG. 1 is a UML application architecture diagram of one embodiment of a file storage system.

FIG. 2 illustrates a file storage system reading a file during video playback.

FIG. 3 is a flow chart of an embodiment of a file storage system storage application.

Fig. 4 is a flowchart of an embodiment of a file storage system playback, download application.

Detailed Description

The invention will now be described in detail with reference to a few embodiments as illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art, that the present invention may be practiced without some or all of these specific details. In other instances, well known process steps and/or structures have not been described in detail in order not to unnecessarily obscure the present invention. Additionally, while the invention will be described in conjunction with the specific embodiments, it will be understood that they are not intended to limit the invention to the described embodiments. On the contrary, the description is intended to cover alternatives, modifications, and equivalents, which may be included within the spirit and scope of the invention as defined by the appended claims.

FIG. 1 is a UML application architecture diagram of one embodiment of a file storage system. As shown in fig. 1, the UML application framework diagram of this embodiment includes three parts, which are a server 101, a client 102, and a file storage system 103. The client 101 includes a device that sends a request, establishes a connection with the server, has a service to process and feed back data, and during the period, the server 101 accesses the file storage system 103 in a mutually exclusive operation manner. The server 101 function module and flow are as follows: firstly, a server side is started 104, a main thread loads the content of an index area of a file storage system to a global cache 105, and at the moment, the server side needs to perform an index area timing updating thread 106, a video storage service processing thread 107, a video playback service processing thread 113 and a request receiving processing thread 108, wherein a timing updating method is adopted when the index area timing updating thread 106 is used for updating data of a terminal system attribute area, a primary index area and a secondary index area of the file storage system 103. The client 102 comprises a 1-path device, a 2-path device and the like, when the device initiates establishment of connection and request queue, the server judges whether to request to receive a processing thread 108, the requests are divided into two types, the first type is a storage push request 109 and a playback push request 111, after the storage push request is initiated, the server performs a video storage request queue 110, and simultaneously, a route storage service processing thread 107 is started and traverses the video storage request queue 110; and storing the data obtained by traversal into a database memory pool 114, wherein the memory pool comprises more than 1-way device data block, and when the data block is full of 10 frames, the data area of the file storage system 103 is accessed in a mutually exclusive operation mode. When the request receiving processing thread 108 processes the playback push request 111, the server performs a video playback request queue 112, and simultaneously, the video playback service processing thread 113 traverses the request queue and updates a thread pool 115 obtained by traversal, where the thread pool includes more than 1 device thread, and the more than one device thread accesses a data area of the file storage system in a mutually exclusive operation manner and fetches data, and the thread returns the data to the client 102. In the mutual exclusion operation, considering that frequent cross operation of the non-data area and the data area can cause frequent jump of a magnetic disk head, and adverse effect is brought to the service life of hardware, a buffer timing refresh mechanism is adopted for refreshing the non-data area, and the mutual exclusion operation is performed with the operation on the data area, namely, the index area is not updated once the data area is operated once, but an index area update thread is specially opened, and the index area is erased and written and updated regularly.

FIG. 2 illustrates a file storage system reading a file during video playback. Video playback, the efficiency evaluation of client retrieving video data according to time period, the video data stored in the file storage system can be retrieved and downloaded by client according to any time period, when client requests video playback, as shown in fig. 2, the application 201 under the linux of the operating system does not pass through the windows file storage system, the file storage system 202 in the disk array is directly read and written, the disk 203 includes but is not limited to 1, 2, 3, 4, the specific video data is quickly positioned through indexes of all levels, thereby achieving the purpose of quick playback, the parameters needing to be transmitted to the server include device ID, video time period, video type and the like, the server retrieves the first-level index area through the time period parameters, obtains index records of all days of the time period, then retrieves the second-level index area through the offset address field value in the retrieved records, obtains the initial offset address of all video data blocks in the data area corresponding to the time period on the date, and takes out the data to return to the client And (4) an end.

FIG. 3 is a flow chart of an embodiment of a file storage system storage application. As shown in FIG. 3, the file storage system makes a storage request 301, the server determines whether the storage is the first storage 302, if so, the file storage system 303 is formatted by performing a formatting partition operation on the disk space, and then the index areas and the data block areas are updated regularly 304, otherwise, the file storage system 303 is not formatted, after the partition is completed, the space is locked in the file storage system format of the present invention, and the space is divided into 4 areas, namely, a system attribute area, a primary index area, a secondary index area and a data area, until the file storage system is full of storage, and when the file storage system is full 305, the file storage system is covered by default from the earliest date by taking the block as a unit 306, namely, the index areas and the data block areas are updated again.

Fig. 4 is a flowchart of an embodiment of a file storage system playback, download application. As shown in fig. 4, the playback 401 operation starts, the video data stored in the file storage system can be retrieved, played back, and downloaded by the client according to any time period, when the client requests the video playback, the parameter that needs to be transmitted to the server is input: the file storage system searches a primary index area through a time period parameter to obtain index records of all dates in the time period, fixes offset addresses 403 corresponding to the date index area according to the start date, then searches a secondary index area through the offset address field values in the searched records to find the start offset addresses 404 of all files in the file index area, finds corresponding file header offset addresses in the data block area according to the offset addresses of the files, and can traverse all video data blocks 405 of the files according to a block tail pointer.

Claims

1. A file storage system for accelerating simultaneous playback of multiple paths of monitoring is characterized in that the system is a file storage system special for storing monitoring videos in the security industry, formatting operation is carried out on a magnetic disk when the file storage system is used for the first time, and system partitions comprise a system attribute partition, a date index partition, namely a primary index partition, a file index partition, namely a secondary index partition, and a data block partition; when the video is played back, the file storage system in the disk array is directly read and written without a windows file storage system, and the specific video data is quickly positioned by indexing the content in the primary index area and the secondary index area, so that the aim of quick playback is fulfilled; the service logic of the file storage system is to cover from the beginning after the disk is full, and the video in a certain period of time is not deleted from the middle of the disk, and disk fragments are not generated; the system attribute zone is marked with a 'PROPERTY _ AREA' string, occupies a storage space of 512 bytes, and comprises system creation time, a version number, a system total space size, a system residual space size, an attribute zone size, a date index zone size, a file index zone size, a data block zone size, a single data block size, a date index zone free zone starting block address, a file index zone free zone starting block address and a data block zone free zone starting block address; the DATE INDEX AREA and the FILE INDEX AREA are divided into a plurality of equal divisions with the same size, the start of the equal divisions is marked by a string of 'data _ INDEX _ AREA' and 'FILE _ INDEX _ AREA', and the total size of the two INDEX AREAs is determined by a division algorithm; the format of the first-level index area is a date stamp, the date stamp corresponds to the first address of the first file in the date in the file index area and is used for quickly positioning the first address of the first file in a certain date in the second-level index area; the FILE INDEX AREA, namely the secondary INDEX AREA, is marked with a string of "" FILE _ INDEX _ AREA "", an AREA for storing FILE INDEX information is used for quickly positioning video data of a certain path of monitoring equipment in a certain day in a certain time period, the content of the secondary INDEX AREA is core retrieval content of quick retrieval, and the format and the size of each INDEX in the secondary INDEX AREA comprise a start date timestamp, an end date timestamp, a video type, a stream format, an equipment ID and a FILE head address of a corresponding data block AREA, and are used for quickly positioning the head address of the FILE in the data block AREA; the DATA BLOCK AREA is marked with a string of 'DATA _ BLOCK _ AREA', a video DATA stream file is stored by taking a BLOCK as a unit, the first DATA BLOCK of the file contains file header information which is used for recovering a file storage system due to accidental damage of the index AREA, and the size of the file storage system is determined by a partitioning algorithm; the data block format comprises a file header, a data block and a head address of the next data block, the partition is used for storing a binary video stream data block, and the data blocks are connected through a block tail address; except for the file header, each data block occupies the same storage space in the data block area;

the file storage system adopts a UML application architecture and comprises three parts, namely a server, a client and the file storage system; the client-side internal equipment sends out a request, establishes connection with the server-side, processes and feeds back data by the service, and accesses the file storage system by the server-side in a mutual exclusion operation mode; the server function module and the flow are as follows: the method comprises the steps that firstly, a server is started, a main thread loads the content of an index area of a file storage system to a global cache, and at the moment, the server needs to perform an index area timing updating thread, a video storage service processing thread, a video playback service processing thread and a request receiving processing thread; the client comprises a 1-path device and a 2-path device, when the device initiates establishment of connection and request queues, the server judges whether a request is to receive a processing thread, the requests are divided into two types, the first type is a storage push request and a playback push request, after the storage push request is initiated, the server carries out video storage request queues, and simultaneously, a video storage service processing thread is started and traverses the storage request queues; storing the data obtained by traversal into a database memory pool, wherein the memory pool comprises more than 1 path of equipment data blocks, and when the data blocks are full of 10 frames, accessing a data area of the file storage system in a mutual exclusion operation mode; when a request receives a processing thread to process a playback push request, a server side carries out a video playback request queue, meanwhile, a video playback service processing thread traverses the request queue and updates a thread pool obtained by traversal, wherein the thread pool comprises more than 1 device thread, the more than 1 device thread accesses a data area of a file storage system in a mutual exclusion operation mode and fetches data, and the 1 device thread transmits the data back to a client side; in the mutual exclusion operation, considering that frequent cross operation of a non-data area and a data area can cause frequent jump of a magnetic head of a magnetic disk and bring adverse effects to the service life of hardware, a buffer timing refresh mechanism is adopted for refreshing the non-data area, and the mutual exclusion operation is performed with the operation on the data area, namely, the data area is not updated once every time the data area is operated, but an index area update thread is specially opened, and the index area is erased and written and updated regularly;

video playback, efficiency evaluation of client side to search video data according to time period, searching and downloading video data stored in file storage system according to any time period, when client side requests video playback, an application program under the linux of the operating system directly reads and writes a file storage system in the disk array without passing through a windows file storage system, quickly locates specific video data through indexes at all levels, thereby achieving the purpose of quick playback, the parameters needing to be transmitted to the server comprise equipment ID, video time period and video type, the server searches the primary index area through the time period parameters to obtain index records of all the days in the time period, then, retrieving a secondary index area through the offset address field value in the retrieved record to obtain the initial offset addresses of all the image data blocks in the data area in the time period corresponding to all the dates, and taking out the data and returning the data to the client; the file storage system provides a storage request, the server judges whether the storage is the first storage, if so, the server performs formatting partition operation on the disk space, and then updates each index area and each data block area at regular time, otherwise, the file storage system is not formatted, after the partition is completed, the disk space is locked into the format of the file storage system, and is divided into 4 areas, namely a system attribute area, a primary index area, a secondary index area and a data area, until the file storage system is fully stored, when the file storage system is fully stored, the file storage system is defaulted to perform overlay storage from the earliest date by taking the block as a unit, namely, each index area and each data block area are updated again; the playback operation starts, the video data stored in the file storage system can be retrieved, played back and downloaded by the client according to any time period, when the client requests the video playback, the parameters need to be transmitted to the server, and the parameters are input: the file storage system searches a primary index area through a time period parameter to obtain index records of all dates of the time period, fixes offset addresses corresponding to the date index area according to the start date, then searches a secondary index area through the offset address field value in the searched records to find the start offset addresses of all files in the file index area, finds corresponding file head offset addresses in the data block area according to the offset addresses of the files, and can traverse all video data blocks of the files according to a block tail pointer.

2. The file storage system for accelerating multiple monitored simultaneous playback as claimed in claim 1 wherein disk space is formatted into four partitions for first use, and after formatting is complete, the space is locked into a custom file storage system format, and the four partitions are arranged in sequence and replaced from scratch when the remaining space in the file storage system is too small.

3. The file storage system for accelerating simultaneous playback of multiple paths of monitors as claimed in claim 1, wherein when writing data, the file storage system fetches a buffer space of fixed size from the memory pool for the currently processed monitor device by the current thread for continuously storing the frame data of the device until the data block area is full, and writes the data into the file storage system once again, the data blocks of the file storage system are connected and next to each other by a block tail structure, so that no disk fragments are generated, the utilization rate of the disk space is saved, and after the data is written, the buffer space is returned to the memory pool for use in the next processing.

4. The file storage system for accelerating multi-channel monitoring simultaneous playback as claimed in claim 1, wherein when it is detected that the remaining storage space of the file storage system is written to the last data block, the overwriting storage is performed by default from the earliest date in units of blocks, and when part of the data is important and cannot be overwritten, the corresponding data block is marked with a lock, so that the overwriting policy automatically identifies the lock mark and skips over the lock area for erasing.