WO2020041928A1

WO2020041928A1 - Data storage method and system and terminal device

Info

Publication number: WO2020041928A1
Application number: PCT/CN2018/102490
Authority: WO
Inventors: 王斌; 刘文涛; 归建章; 苏晓天
Original assignee: 深圳市锐明技术股份有限公司
Priority date: 2018-08-27
Filing date: 2018-08-27
Publication date: 2020-03-05
Also published as: CN110149803B; CN110149803A

Abstract

The present application is applicable to the technical field of computers, and discloses a data storage method and system and a terminal device. The method comprises: reading a data block index table, searching for an idle data block according to the data block index table, and updating a data block index node corresponding to the idle data block; allocating a buffer for an in-block index table in idle data block; cyclically reading data in a data buffer of each channel; if a time length of data in a data buffer of a target channel is greater than or equal to a preset time length, reading data of the preset time length in the data buffer of the target channel, writing the data of the preset time length to the idle data block, and updating the data block index node corresponding to the idle data block and the in-block index table in the buffer; and if the idle data block is full, writing the in-block index table in the buffer to an end of the idle data block, and continuing performing the step of searching for an idle data block according to the data block index table. The present application reduces the number of writing positions, and improves a writing speed of a disc.

Description

Data storage method, system and terminal equipment

Technical field

The present application belongs to the field of computer technology, and particularly relates to a data storage method, system, and terminal device.

Background technique

With the high-definition monitoring data and the increase in the number of monitoring channels, the amount of data that needs to be stored for monitoring is also increasing. Hard disk as a cost-effective storage medium is widely used in various industries, but the disk density is limited by the laws of physics. At present, the perpendicular magnetic recording technology used by hard disks is about to reach its limit. Therefore, shingled magnetic recording (Shingled Magnetic Recording, SMR) disk came into being. For SMR disks, a cache with a relatively high write speed is usually set. Data is written to the cache first. When the SMR disk is idle, data is written from the cache to the shingled track, which can improve the writing efficiency. However, due to the limited cache space, when there are too many write locations, the cache space is insufficient, resulting in a significant reduction in write speed.

At present, the data of one channel is usually used as a file. Since all channels will generate data at the same time, multiple files will be written to the SMR disk at the same time. This will increase the write position and cause the write speed to decrease.

technical problem

In view of this, embodiments of the present application provide a data storage method, system, and terminal device to solve the problem that the writing speed decreases when multi-channel data is written to an SMR disk in the prior art.

Technical solutions

A first aspect of the embodiments of the present application provides a data storage method, including:

Read the data block index table, find the free data block according to the data block index table, and update the data block index node corresponding to the idle data block. The data block index table includes multiple data block index nodes;

Allocate a cache for the in-block index table in the free data block;

Read the data in the data buffer area of each channel cyclically;

If the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, then read the data of the preset time length in the data buffer area of the target channel and write the data of the preset time length to the idle data Block, and update the data block index node corresponding to the idle data block and the in-block index table in the cache, and the target channel is any one of all channels;

If the free data block is full, the in-block index table in the cache is written to the tail of the free data block, and the step of finding the free data block according to the data block index table is continued.

A second aspect of the embodiments of the present application provides a data storage system, including:

The idle data block search module is used to read the data block index table, find the free data block according to the data block index table, and update the data block index node corresponding to the idle data block. The data block index table includes multiple data block index nodes;

A cache allocation module, configured to allocate a cache for an in-block index table in an idle data block;

Cyclic reading module, which is used to read the data in the data buffer area of each channel cyclically;

A data writing module is configured to read data of a preset time length in the data buffer area of the target channel if the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, and set the preset time The length of data is written into the free data block, and the data block index node corresponding to the free data block and the in-block index table in the cache are updated, and the target channel is any one of all channels;

The index table writing module is configured to write the index table in the block in the cache to the end of the free data block if the free data block is full, and continue to perform the steps of finding the free data block according to the data block index table.

A third aspect of the embodiments of the present application provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the data storage method described above is implemented. step.

A fourth aspect of the embodiments of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by one or more processors, the steps of the data storage method described above are implemented.

Beneficial effect

The embodiment of the present application first reads a data block index table, searches for a free data block according to the data block index table, and updates a data block index node corresponding to the idle data block, allocates a buffer for the in-block index table in the free data block, and then reads the data circularly Take the data in the data buffer area of each channel. If the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, then read the data of the preset time length in the data buffer area of the target channel. , Write the data of the preset time length into the free data block, and update the data block index node corresponding to the free data block and the in-block index table in the cache, the target channel is any one of all channels, and finally if the free data block is written When it is full, the in-block index table in the cache is written to the end of the free data block, and the steps of finding the free data block according to the data block index table are continued. In the embodiment of the present application, the data in the data buffer area of each channel is read cyclically, and it is determined whether the time length of the data in the data buffer area of each channel is greater than or equal to the preset time length. If so, the preset time is set. The length of data is written into the free data block, which can combine multi-channel data into one channel of data for write operations, reducing the write position and improving the disk write speed; it also uses the data block index table and the in-block index table in the data block Build a two-level index to speed up retrieval and reading.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flowchart of a data storage method provided by an embodiment of the present application;

2 is a schematic diagram of a data block index table provided by an embodiment of the present application;

3 is a schematic diagram of an intra-block index table in a data block provided by an embodiment of the present application;

4 is a schematic flowchart of a data storage method provided by another embodiment of the present application;

5 is a schematic flowchart of an implementation method of a data storage method according to another embodiment of the present application;

6 is a schematic block diagram of a data storage system according to an embodiment of the present application;

FIG. 7 is a schematic block diagram of a terminal device according to an embodiment of the present application.

Embodiments of the invention

In order to explain the technical solution described in this application, specific examples are used for description below.

FIG. 1 is a schematic diagram of an implementation process of a data storage method provided by an embodiment of the present application. For ease of description, only parts related to the embodiment of the present application are shown. The execution subject in this embodiment of the present application may be a terminal device. As shown in Figure 1, the method may include the following steps:

Step S101: Read a data block index table, find a free data block according to the data block index table, and update a data block index node corresponding to the idle data block. The data block index table includes a plurality of data block index nodes.

In the embodiment of the present application, the storage space of the SMR disk is divided into a plurality of data blocks according to a preset size, and each data block may be numbered, and the numbering may start from 0 and increase sequentially. The preset size can be 256 megabytes (MByte, MB). For example, a 4T hard disk can be divided into 256MB to obtain 16,384 data blocks. The 16384 data blocks are numbered in sequence, with numbers ranging from 0 to 16383.

All the divided data blocks are stored through a data block index table. As shown in FIG. 2, the data block index table includes multiple data block index nodes, and each data block index node corresponds to one data block. The data block index node includes the file ownership stamp, the file internal number, the file start time, the file end time, channel information, data block type, reserved fields, and check digits, which are defined as follows:

typedefstruct _block_info_

{

unsigned long owner_stamp;

unsigned long inner_num;

unsigned long start_time;

unsigned long end_time;

unsigned long chn_info;

unsigned char block_type;

unsigned char reserve [7];

unsigned long check_num;

} block_info_t;

Among them, owner_stamp is a file ownership stamp, which can also be referred to as a file identifier. Different files have different file ownership stamps. You can use the file ownership stamp to find out which data blocks are stored in the same file. The files in the embodiments of the present application are files with time attributes, such as video files, audio files, and so on. Because the monitoring channels are multiple channels, all channels will generate data at the same time. A file is made up of data from all channels that are continuous in time.

inner_num is the internal number of the file, which is used to indicate the order in which the files are stored. The internal number of the same file can be a continuous natural number starting from 1. A file is often stored in multiple data blocks. Assuming a file is stored in data blocks 1, 3, and 5, the internal number of the file corresponding to data block 1 is 3, the internal number of the file corresponding to data block 2 is 2, and the data block The internal number of the file corresponding to 5 is 1. Then the order of storing the file is first stored in data block 5, after data block 5 is full, then it is stored in data block 3. After data block 3 is full, Stored in data block 1.

start_time is the file start time, which indicates the start time of the file in the data block.

end_time is the end time of the file, indicating the end time of the file in the data block. That is, the data stored in the data block is data in the file from the file start time to the file end time.

chn_info is channel information, which indicates which channels of data are stored in the data block. Bits are used to indicate whether data for the corresponding channel is stored. For example, if the 0th bit is 1, it means that the data of channel 0 is stored. If the 0th bit is 0, it means that the data of channel 0 is not stored. The other channels are similar and will not be described again.

block_type is a data block type and is used to distinguish the type of the data block. You can use 0 to indicate free data blocks, that is, data blocks that do not store data; use 1 to indicate data blocks that store data block index tables; and use 2 to indicate data blocks that store data.

reserve [7] is a reserved field.

check_num is a check digit, which is used to determine whether the data is legal.

A data block index node is 32 bytes. If a 4T hard disk has a total of 16384 data blocks, the size of the data block index table corresponding to the hard disk is 16384 * 32 = 524288 bytes.

In order to quickly obtain the data block index table, the data block index table is fixedly stored in the data block numbered 0, and only the data block index table is stored in the data block numbered 0. Therefore, the data block numbered 0 corresponds to The data block type is 1, the other data blocks are used to store the data of each channel.

Each data block is divided into a data area and an index area, as shown in Figure 2. The data area stores data of each channel, and the index area stores an index table in the block. The index table in the block is constructed by data of a preset time length of each channel, and the preset time length can be 1 second. As shown in FIG. 3, the intra-block index table includes multiple intra-block index nodes, and each intra-block index node stores information related to one second of data. The index table in the block can accurately locate the data of a certain channel for a certain second. In order to avoid the situation where the data is written into the data area while the data is written into the index area, that is, to avoid the increase in the write position, the index area (the index table in the block) is buffered into memory. When the data area is written After that, the in-block index table in the memory is written into the index area, which is equivalent to the write operation of the data block as a sequential write.

The index table in the block records the relevant information of each second of data. The definition of the index node in the block is as follows:

typedefstruct _data_info_

{

unsigned char chn;

unsigned char hour;

unsigned char minute;

unsigned char second;

unsigned long offset;

unsigned long length;

unsigned char flag;

unsigned char reserve [3];

} data_info_t;

Among them, chn is the channel number, which indicates that the data corresponding to the index node in the block belongs to the channel number.

Hour, minute, and second are hours, minutes, and seconds, respectively, indicating that the data corresponding to the i-nodes in the block are the hours, minutes, and seconds.

offset is the data offset, which indicates the offset of the data corresponding to the index node in the block in the data block.

length is the data size, which indicates the size of the data corresponding to the inodes in the block.

flag is the index flag. If it is 0, it means idle, if it is 1, it means occupied.

reserve [3] is a reserved field.

According to the definition of the i-node within a block, the i-node within a block is 16 bytes.

For 256MB data blocks, allocate 256KB of space for the index area, then the index area can store 256 * 1024/16 = 16384 in-block inodes, that is, 16384 in-block inodes are used to index 256MB of data. Each block is averaged. The size of the data indexed by the inner i-node is 256 * 1024/16384 = 16KB, that is, the average data per second is 16KB. Since the code rate of video data or audio data is greater than 16bps, the 256KB index area is 256MB of data. The blocks are completely adequate. In addition, the 256KB index area occupies 1/1024 of the 256MB data block, which means that the space utilization rate can reach 99.9%.

In the embodiment of the present application, when data needs to be written, the data block index table is first read, and the free data block is found according to the data block index table. After finding the free data block, the data block index node corresponding to the free data block is found. The data block type in is updated from 0 to 2, and the file internal number is updated.

Step S102: Allocate a cache for the in-block index table in the free data block.

In the embodiment of the present application, before writing data to the free data block, first allocate a cache for the in-block index table in the free data block, and save the in-block index table in the cache. At the same time, the in-block index table in the cache is updated.

Step S103: The data in the data buffer area of each channel is read cyclically.

Each channel has a corresponding data buffer area, and the monitoring data of the corresponding channel is stored in the data buffer area.

Read the data in the data buffer area of each channel cyclically, and determine whether the time length of the data in the data buffer area of each channel is greater than or equal to the preset time length.

Assume that there are 4 channels, which are channels 0, 1, 2 and 3, then the data in the data buffer area of each channel is read cyclically. The data in the data buffer area of channel 0 can be read first, and then read. Data in the data buffer area of channel 1, then read the data in the data buffer area of channel 2, then read the data in the data buffer area of channel 3, and then read the data in the data buffer area of channel 0 again, In this way, the data in the data buffer area of each channel is read cyclically.

Step S104: If the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, then read the data of the preset time length in the data buffer area of the target channel and write the data of the preset time length Into the idle data block, and update the data block index node corresponding to the idle data block and the in-block index table in the cache, the target channel is any one of all channels.

The preset time length may be 1 second.

In the embodiment of the present application, if the time length of the data in the data buffer area of the target channel is greater than or equal to 1 second, then the data in the data buffer area of the target channel with a length of 1 second is read, and the 1 second is The data is written into the data area of the free data block, and the data block index node corresponding to the free data block and the in-block index table in the cache are updated.

Updating the data block index node corresponding to the idle data block may be updating the file start time, file end time, channel information, etc. of the data block index node.

Updating the intra-block index table in the cache may be to find an idle intra-block index node in the intra-block index table, that is, an intra-block index node with an index identifier of 0, and updating the idle intra-block index node according to the channel number of the target channel. Update the hour, minute, and second of the index node in the free block according to the time of the 1-second data, and update the free block according to the position where the 1-second data is written in the free data block The data offset of the inode is updated according to the data size of the 1 second data of the inode in the idle block, and the index identifier is updated to 1.

Step S105: if the free data block is full, write the index table in the block in the cache to the tail of the free data block, and continue to perform the steps of finding the free data block according to the data block index table.

In the embodiment of the present application, if the free data block is full, that is, the data area of the free data block is full, the index table in the block in the cache is written to the tail of the free data block, that is, the data of the free data block is written. Index area, and search for a new data block again, that is, return to the step of finding a free data block according to the data block index table in step S101 to perform a loop.

From the above description, it can be known that, in the embodiment of the present application, the data in the data buffer area of each channel is read cyclically, and it is determined whether the time length of the data in the data buffer area of each channel is greater than or equal to the preset time length. The data of a preset length of time is written into the free data block, and the data of multiple channels can be combined into one channel for sequential write operation, which greatly reduces the addressing time of the hard disk, reduces the write position, and improves the write speed of the SMR disk. , Which can avoid write amplification of SMR disks at the maximum speed limit; build a two-level index through the data block index table and the in-block index table in the data block. Because the index structure is small, it can complete high efficiency with less memory overhead. Data retrieval and reading of specified data; by writing a buffer to the in-block index table in the free data block before writing the data, while writing the data, the in-block index table in the cache is updated and the data is full Then, the in-block index table in the cache is written to the free data block, and the sequential writing of the data area and the index area in the data block is realized, reducing the writing position, further High write speed.

As another embodiment of the present application, the data block index node includes a file belonging stamp and a data block type;

After reading the data block index table, it also includes:

Cache the data block index table into memory.

Find the free data block according to the data block index table, and update the data block index node corresponding to the free data block, including:

Traverse the data block index table in memory to get the largest file attribution stamp, and determine the file attribution stamp for the file to be stored based on the largest file attribution stamp.

Find the free data block according to the data block type, and update the data block index node corresponding to the free data block according to the file ownership stamp of the file to be stored.

In the embodiment of the present application, after reading the data block index table, the data block index table is cached in the memory. In the subsequent steps, when the data block index table needs to be read, it can be read directly in the memory. It is faster than reading the data block index table on the hard disk.

By comparing the file attribution stamps of all data block index nodes in the data block index table, the largest file attribution stamp of the file attribution stamps of all data block inodes is obtained, and the largest file attribution stamp plus 1 is used to obtain the file attribution of the file to be stored. stamp.

The idle data block is a data block corresponding to a data block index node whose data block type is 0. After finding the free data block, the file home stamp of the data block index node corresponding to the free data block is updated to the file home stamp of the file to be stored, and the data block type of the data block index node corresponding to the free data block is updated to 2 And update the file internal number of the data block index node corresponding to the free data block, and the file internal number is updated to the file internal number of the data block index node corresponding to the previously stored data block plus 1 if it does not exist. For a stored data block, the internal number of the file is numbered from the beginning and can be updated to 1.

As another embodiment of the present application, after reading the data in the data buffer area of each channel in a loop, the method further includes:

If the time length of the data in the data buffer area of the target channel is less than the preset time length, it continues to determine whether the time length of the data in the data buffer area of the next channel is greater than or equal to the preset time length.

After updating the data block index node corresponding to the free data block and the in-block index table in the cache, the method further includes:

If the free data block is not full, the steps of reading data in the data buffer area of each channel in a loop are continued.

In the embodiment of the present application, if the time length of the data in the data buffer area of the target channel is less than 1 second, the data in the data buffer area of the next channel is continuously read, and the data in the data buffer area of the next channel is determined. Whether the length of data is greater than or equal to 1 second.

If the free data block is not full, that is, the data area of the free data block is not full, the process returns to step S103 to execute cyclically.

FIG. 4 is a schematic flowchart of a data storage method provided by another embodiment of the present application. As shown in FIG. 4, based on the above embodiment, the data storage method may further include the following steps:

Step S401: Obtain information to be queried.

The information to be queried includes the time to be queried, the time to be queried and the number of the channel to be queried, and is used to query the time between the time to be queried and the time to be queried. The data of the channel number to be queried belongs to List of file information.

Step S402: traverse the data block index table, and merge all data block index nodes with the same file ownership stamp into one file description information.

The file description information includes the file attribution stamp, the file start time, the file end time, and the channel information contained in the file. The definition is as follows:

typedefstruct _file_info_

{

unsigned long owner_stamp;

unsigned long start_time;

unsigned long end_time;

unsigned long chn_info;

} file_info_t;

Among them, owner_stamp is the file ownership stamp.

start_time is the file start time, and the file start time is the file start time of the first data block among all data blocks corresponding to the file home stamp.

end_time is the end time of the file, and the end time of the file is the end time of the last data block among all data blocks corresponding to the file home stamp.

chn_info is the channel information contained in the file, and the channel information contained in the file is the union of the channel information of all the data blocks corresponding to the file home stamp.

In the embodiment of the present application, all data block index nodes with the same home stamp are merged into one piece of file description information, that is, one file corresponds to one piece of file description information.

Step S403: Determine a query result corresponding to the query information according to the file description information.

The time between the file start time and the file end time in the file description information includes the time between the start time to be queried and the end time to be queried, and in the channel information contained in the file, the channel number to be queried corresponds to A list of file information with a bit of 1. The file information list may include content such as file ownership stamp, file start time, file end time, and channel information included in the file.

As another embodiment of the present application, the information to be queried includes a start time to be queried, an end time to be queried, and a channel number to be queried, and the file description information includes a file home stamp, a file start time, a file end time, and channel information contained in the file;

The query result corresponding to the query information is determined according to the file description information, including:

According to the file start time, file end time, and channel information contained in the file, a list of file information corresponding to the start time to be queried, the end time to be queried, and the channel number to be queried is determined.

It can be known from the foregoing description that the embodiments of the present application can query the file information list that meets the requirements according to the start time, end time and channel number to be queried.

FIG. 5 is a schematic flowchart of a data storage method provided by another embodiment of the present application. As shown in FIG. 5, based on the above embodiment, the data storage method may further include the following steps:

Step S501: Obtain a positioning time and a channel number to be read.

In the embodiment of the present application, the data of the channel number to be read after the positioning time is read.

Step S502: Traverse the data block index table to obtain the positioning data block to which the positioning time belongs.

The data block to which the positioning time belongs is referred to as a positioning data block.

In the embodiment of the present application, the data block index table is traversed, and the positioning data block to which the positioning time belongs is obtained according to the file start time and the file end time, that is, the period between the file start time and the file end time corresponding to the positioning data block includes The positioning time.

Step S503: Read the index table in the positioning block in the positioning data block, and cache the index table in the positioning block into the memory.

In the embodiment of the present application, the intra-block index table in the positioning data block is referred to as an intra-block index table. Read the index table in the positioning block, and cache the index table in the positioning block into memory.

Step S504: traverse the index table in the positioning block in memory, determine the positioning time and the index node in the positioning block where the channel number to be read is located, and determine the data to be read according to the data offset in the index node in the positioning block. position.

In the embodiment of the present application, the index node in the block where the positioning time and the channel number to be read are located is referred to as the index node in the positioning block. That is, the hour, minute and second of the index node in the positioning block is the positioning time, and the channel number of the index node in the positioning block is the number of the channel to be read.

Obtain the data offset in the index node in the positioning block, and determine the position of the data to be read in the positioning data block according to the data offset.

Step S505: Read the data to be read according to the position of the data to be read.

At the position of the data to be read, the data to be read whose size is the data size of the index node in the positioning block is read.

Step S506: In the in-block index node after the in-block index node in the in-block index table, find whether there is a target in-block index node that contains the channel number to be read.

In the embodiment of the present application, among the in-block index nodes after the in-block index node in the in-block index table, the in-block index node whose channel number is the channel number to be read is referred to as the target in-block index node.

Step S507: if there is an index node in the target block, use the index node in the target block as a new index node in the positioning block, and continue to determine the position of the data to be read according to the data offset in the index node in the positioning block. A step of.

If there is an index node in the target block, the index node in the target block is used as a new index node in the positioning block, and the method returns to step S504 to determine the position of the data to be read according to the data offset in the index node in the positioning block. The steps are cycled.

Step S508: if there is no index node in the target block, then in the data block index table, find whether a file attribution stamp is the same as the file attribution corresponding to the positioning data block, and the file internal number is the internal number of the file corresponding to the positioning data block. Increment the target data block inode.

Wherein, the file home stamp is the same as the file home stamp corresponding to the positioning data block, and the internal number of the file is the data block index node plus the internal number of the file corresponding to the positioning data block plus one as the target data block index node.

If there are no inodes in the target block, then in the data block index table, look up whether there is a target data block inode.

Step S509: if a target data block index node exists, use the data block corresponding to the target data block index node as a new positioning data block, read the new index table in the positioning block in the new positioning data block, and add the new The index table in the positioning block is cached in memory.

The intra-block index table in the new positioning data block is used as the new intra-block index table.

Step S510: traverse the new index table in the positioning block in the memory, determine the new index node in the positioning block where the channel number to be read is located, and continue to perform determination based on the data offset in the index node in the positioning block to be read Steps to get the location of the data.

In the embodiment of the present application, in the new index table within the positioning block, the index node within the block with the same channel number as the channel number to be read is used as the new index node within the positioning block, and the new index node within the positioning block is replaced. The previous index node in the positioning block returns to step S504, and the step of determining the position of the data to be read according to the data offset in the index node in the positioning block is executed cyclically.

Step S511: if the target data block index node does not exist, the data reading ends.

From the above description, it can be known that the embodiment of the present application can read data after a certain time node in a channel, and first locate the data block according to the data block index table, and then accurately locate the data through the index table in the block, which can efficiently locate Required data.

As another embodiment of the present application, the data block index node includes a file start time and a file end time;

Traverse the data block index table to obtain the positioning data block to which the positioning time belongs, including:

Traverse the data block index table, and obtain the positioning data block to which the positioning time belongs according to the file start time and the file end time.

It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

FIG. 6 is a schematic block diagram of a data storage system according to an embodiment of the present application. For convenience of explanation, only a part related to the embodiment of the present application is shown.

In the embodiment of the present application, the data storage system 6 includes:

The idle data block search module 61 is configured to read a data block index table, find a free data block according to the data block index table, and update a data block index node corresponding to the idle data block. The data block index table includes multiple data block index nodes;

A cache allocation module 62, configured to allocate a cache for an in-block index table in an idle data block;

A cyclic reading module 63, configured to cyclically read data in a data buffer area of each channel;

The data writing module 64 is configured to read data of a predetermined time length in the data buffer area of the target channel if the time length of the data in the data buffer area of the target channel is greater than or equal to a preset time length, and The length of data is written into the idle data block, and the data block index node corresponding to the idle data block and the in-block index table in the cache are updated, and the target channel is any one of all channels;

The index table writing module 65 is configured to write an in-block index table in the cache to the tail of the free data block if the free data block is full, and continue to perform the steps of finding the free data block according to the data block index table.

Optionally, the idle data block search module 61 is further configured to cache the data block index table into memory;

The idle data block search module 61 further includes:

The file ownership stamp obtaining unit is used to traverse the data block index table in the memory to obtain the largest file ownership stamp, and determine the file ownership stamp of the file to be stored according to the largest file ownership stamp;

The idle data block searching unit is configured to find an idle data block according to the data block type, and update a data block index node corresponding to the idle data block according to a file belonging stamp of a file to be stored.

Optionally, the data storage system 6 further includes:

A judging module, configured to continue to judge whether the time length of the data in the data buffer area of the next channel is greater than or equal to the preset time length if the time length of the data in the data buffer area of the target channel is less than the preset time length;

The first loop module is configured to, if the free data block is not full, continue to perform the step of cyclically reading data in the data buffer area of each channel.

Optionally, the data block index node includes a file belonging stamp;

The data storage system 6 further includes:

A first obtaining module, configured to obtain information to be queried;

A merging module for traversing the data block index table and merging all data block index nodes with the same file ownership stamp into one file description information;

The query result obtaining module is configured to determine a query result corresponding to the query information according to the file description information.

Optionally, the information to be queried includes a start time to be queried, an end time to be queried, and a channel number to be queried, and the file description information includes a file belonging stamp, a file start time, a file end time, and channel information contained in the file;

The query result acquisition module is specifically configured to determine a list of file information corresponding to the start time, the end time of the query, and the channel number to be queried according to the file start time, the file end time, and the channel information contained in the file.

Optionally, the data storage system 6 further includes:

A second acquisition module, configured to acquire a positioning time and a channel number to be read;

A positioning data block determination module, which is used to traverse a data block index table to obtain a positioning data block to which a positioning time belongs;

The index table reading module is used to read the index table in the positioning block in the positioning data block, and cache the index table in the positioning block into memory;

The reading position determination module is used to traverse the index table in the positioning block in memory, determine the positioning time and the index node in the positioning block where the channel number to be read is located, and determine according to the data offset in the index node in the positioning block. The location of the data to be read;

A data reading module, configured to read the data to be read according to the position of the data to be read;

The index node search module is used to find whether there is a target in-block index node containing the channel number to be read in the in-block index node after the in-block index node in the index table in the positioning block;

A first processing module, configured to use the index node in the target block as a new index node in the positioning block if the index node in the target block exists, and continue to execute the determination of the to-be-read based on the data offset in the index node in the positioning block; Steps to get the location of the data;

The second processing module is configured to find, in the data block index table, if there is no inode in the target block, the file attribution stamp is the same as the file attribution corresponding to the positioning data block, and the internal number of the file corresponds to the positioning data block The internal number of the file plus the target data block index node;

A third processing module, configured to: if a target data block index node exists, use the data block corresponding to the target data block index node as a new positioning data block, and read a new index table within the positioning block in the new positioning data block; And cache the index table in the new positioning block into memory;

The second loop module is used to traverse the new index table in the positioning block in memory, determine the index node in the positioning block where the channel number to be read is located, and continue to execute the determination based on the data offset in the index node in the positioning block. Steps for the location of the data to be read;

A fourth processing module is configured to: if the target data block index node does not exist, the data reading ends.

Optionally, the data block index node includes a file start time and a file end time;

The positioning data block determination module is specifically used to traverse the data block index table and obtain the positioning data block to which the positioning time belongs according to the file start time and the file end time.

Those skilled in the art can clearly understand that, for the convenience and brevity of the description, only the above-mentioned division of functional units and modules is used as an example. In practical applications, the above functions can be allocated by different functional units according to needs. Module completion, that is, dividing the internal structure of the data storage system into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated into one processing unit, or each unit may exist separately physically, or two or more units may be integrated into one unit. The integrated unit may be hardware. It can be implemented in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For specific working processes of the units and modules in the foregoing device, reference may be made to corresponding processes in the foregoing method embodiments, and details are not described herein again.

FIG. 7 is a schematic block diagram of a terminal device according to an embodiment of the present application. As shown in FIG. 7, the terminal device 7 of this embodiment includes one or more processors 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the foregoing data storage method embodiments are implemented, for example, steps S101 to S105 shown in FIG. Alternatively, when the processor 70 executes the computer program 72, the functions of the modules / units in the foregoing embodiment of the data storage system are implemented, for example, the functions of modules 61 to 65 shown in FIG. 6.

Exemplarily, the computer program 72 may be divided into one or more modules / units, and the one or more modules / units are stored in the memory 71 and executed by the processor 70 to complete This application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the terminal device 7. For example, the computer program 72 may be divided into a free data block search module, a cache allocation module, a cyclic read module, a data write module, and an index table write module. The specific functions of each module are as follows:

For other modules or units, reference may be made to the description in the embodiment shown in FIG. 6, and details are not described herein again.

The terminal device may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The terminal device 7 includes, but is not limited to, a processor 70 and a memory 71. Those skilled in the art can understand that FIG. 7 is only an example of the terminal device, and does not constitute a limitation on the terminal device 7. It may include more or fewer components than shown in the figure, or combine some components or different components. For example, the terminal device 7 may further include an input device, an output device, a network access device, a bus, and the like.

The processor 70 may be a central processing unit (Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), and application-specific integrated circuits (Applications) Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 71 may be an internal storage unit of the terminal device, such as a hard disk or a memory of the terminal device. The memory 71 may also be an external storage device of the terminal device, for example, a plug-in hard disk, a smart memory card (Smart Media) Card, SMC), Secure Digital (SD) card, Flash Card, etc. Further, the memory 71 may further include both an internal storage unit of the terminal device and an external storage device. The memory 71 is configured to store the computer program 72 and other programs and data required by the terminal device. The memory 71 may also be used to temporarily store data that has been output or is to be output.

In the above embodiments, the description of each embodiment has its own emphasis. For a part that is not detailed or recorded in an embodiment, reference may be made to related descriptions of other embodiments.

Those of ordinary skill in the art may realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed data storage system and method may be implemented in other ways. For example, the embodiments of the data storage system described above are merely schematic. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be another division manner, such as multiple units or units. Components can be combined or integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, which may be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, this application implements all or part of the processes in the method of the above embodiment, and can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. The computer When the program is executed by a processor, the steps of the foregoing method embodiments can be implemented. The computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signals, telecommunication signals, and software distribution media. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdictions. For example, in some jurisdictions, the computer-readable medium Excludes electric carrier signals and telecommunication signals.

The above-mentioned embodiments are only used to describe the technical solution of the present application, but are not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still implement the foregoing implementations. The technical solutions described in the examples are modified, or some of the technical features are equivalently replaced; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in Within the scope of this application.

Claims

A data storage method, comprising:

Read a data block index table, find a free data block according to the data block index table, and update a data block index node corresponding to the idle data block, where the data block index table includes a plurality of data block index nodes;

Allocating a cache for an in-block index table in the idle data block;

Read the data in the data buffer area of each channel cyclically;

If the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, then read the data of the preset time length in the data buffer area of the target channel, and change the preset time length The data is written into the idle data block, and the data block index node corresponding to the idle data block and the in-block index table in the cache are updated, and the target channel is any one of all channels;

If the free data block is full, write an in-block index table in the cache to the tail of the free data block, and continue to execute the step of finding an idle data block according to the data block index table.
The data storage method according to claim 1, wherein the data block index node includes a file belonging stamp and a data block type;

After the reading the data block index table, the method further includes:

Buffering the data block index table into memory;

The searching for an idle data block according to the data block index table and updating a data block index node corresponding to the idle data block includes:

Traverse the data block index table in the memory to obtain the largest file attribution stamp, and determine the file attribution stamp of the file to be stored according to the largest file attribution stamp;

Find an idle data block according to the data block type, and update a data block index node corresponding to the idle data block according to a file ownership stamp of the file to be stored.
The data storage method according to claim 1, after the loop reading data in a data buffer area of each channel, further comprising:

If the time length of the data in the data buffer area of the target channel is shorter than the preset time length, continue to determine whether the time length of the data in the data buffer area of the next channel is greater than or equal to the preset time length;

After the updating the data block index node corresponding to the idle data block and the in-block index table in the cache, the method further includes:

If the idle data block is not full, the step of reading the data in the data buffer area of each channel in a loop is continued.
The data storage method according to claim 1, wherein the data block index node includes a file belonging stamp;

The data storage method further includes:

Obtain the information to be queried;

Traverse the data block index table, and merge all data block index nodes with the same file ownership stamp into one file description information;

A query result corresponding to the query information is determined according to the file description information.
The data storage method according to claim 4, wherein the information to be queried includes a start time to be queried, an end time to be queried, and a channel number to be queried, and the file description information includes a file belonging stamp, a file start time, File end time and channel information contained in the file;

The determining a query result corresponding to the information to be queried based on the file description information includes:

Determining a file information list corresponding to the start time to be queried, the end time to be queried, and the channel number to be queried according to the file start time, the file end time, and channel information contained in the file.
The data storage method according to claim 1, further comprising:

Get the positioning time and the channel number to be read;

Traverse the data block index table to obtain a positioning data block to which the positioning time belongs;

Reading the index table in the positioning block in the positioning data block, and buffering the index table in the positioning block into memory;

Traverse the index table in the positioning block in memory, determine the positioning time and the index node in the positioning block where the channel number to be read is located, and determine according to the data offset in the index node in the positioning block The location of the data to be read;

Reading the data to be read according to the position of the data to be read;

Searching for an in-block index node following the in-block index node in the in-block index table to find a target in-block index node that includes the channel number to be read;

If the index node in the target block exists, use the index node in the target block as a new index node in the positioning block, and continue to perform the determining based on the data offset in the index node in the positioning block. Steps to read the location of the data;

If the index node in the target block does not exist, in the data block index table, find whether a file home stamp is the same as the file home stamp corresponding to the positioning data block, and the file internal number is the positioning data block. The target data block index node corresponding to the internal number of the file plus one;

If the target data block index node exists, use the data block corresponding to the target data block index node as a new positioning data block, read a new index table in the positioning block in the new positioning data block, and Cache the index table in the new positioning block into memory;

Traversing the new index table in the positioning block in memory, determining the index node in the positioning block where the channel number to be read is located, and continuing to perform the according to the data offset in the index node in the positioning block, Steps to determine the location of the data to be read;

If the target data block index node does not exist, data reading ends.
The data storage method according to claim 6, wherein the data block index node includes a file start time and a file end time;

The traversing the data block index table to obtain a positioning data block to which the positioning time belongs includes:

Traverse the data block index table, and obtain a positioning data block to which the positioning time belongs according to the file start time and the file end time.
A data storage system, comprising:

An idle data block search module is configured to read a data block index table, find an idle data block according to the data block index table, and update a data block index node corresponding to the idle data block. The data block index table includes a plurality of Data block index node;

A cache allocation module, configured to allocate a cache for an in-block index table in the idle data block;

Cyclic reading module, which is used to read the data in the data buffer area of each channel cyclically;

A data writing module, configured to read data of a predetermined time length in the data buffer area of the target channel if the time length of the data in the data buffer area of the target channel is greater than or equal to the preset time length, Write the data of the preset time length into the idle data block, and update the data block index node corresponding to the idle data block and the in-block index table in the cache, and the target channel is the one in all channels Either

An index table writing module is configured to write an in-block index table in the cache to the tail of the idle data block if the free data block is full, and continue to execute the index table based on the data block. Steps to find free data blocks.
A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the computer program according to claims 1 to 1 when executing the computer program. Steps of the data storage method according to any one of 7.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by one or more processors, the data according to any one of claims 1 to 7 is realized Store method steps.