CN110764706B

CN110764706B - Storage system, data management method, and storage medium

Info

Publication number: CN110764706B
Application number: CN201911021948.4A
Authority: CN
Inventors: 陈白杨; 许海涛
Original assignee: TP Link Technologies Co Ltd
Current assignee: TP Link Technologies Co Ltd
Priority date: 2019-10-25
Filing date: 2019-10-25
Publication date: 2023-10-03
Anticipated expiration: 2039-10-25
Also published as: CN110764706A

Abstract

The application is suitable for the technical field of data management, and provides a storage system, a data management method and a storage medium, wherein the storage system divides a disk into a first storage area for storing video data and a second storage area for storing index data by dividing the internal structure of the storage system, and the index data is stored in advance by using an index data manager, and when the index data amount of the index data manager reaches a preset data threshold value, the index data is stored in the second storage area, so that the index data and the storage data cannot be written into the disk at the same time, the occupied disk IO is reduced, the disk performance is fully utilized, and the disk writing efficiency is improved.

Description

Storage system, data management method, and storage medium

Technical Field

The present application relates to the field of data management technologies, and in particular, to a storage system, a data management method, and a storage medium.

Background

In the prior art, a certain problem exists in directly realizing data storage in an embedded type, such as using a file system to store video data files, and the problem is that when the video data files are stored in fixed-length data files, excessive handles of the fixed-length data files can become a bottleneck of reading and writing speeds, so that the data writing speeds are influenced to a certain extent, power-off data loss is easy to occur, and serious experience degradation is caused by scattered data management. Alternatively, the use of a mixed stream bare disk for storing data has a problem in that the stored data is excessively scattered, resulting in a serious degradation of data reading performance. In addition, the use of a memory database in a storage system to store index data consumes excessive memory even if the index data is small relative to the size of video data, and a large amount of index data in memory can result in a slow restart recovery rate.

In summary, in the current storage technology, there is a problem that the data storage occupies a large amount of disk input/output (IO) terminals, resulting in slow data reading and writing speeds.

Disclosure of Invention

The embodiment of the application provides a storage system, a data management method and a storage medium, which can solve the problem of slow data reading and writing speed caused by the fact that a large amount of data storage occupies a disk input/output (IO) end in the prior art.

In a first aspect, an embodiment of the present application provides a storage system, where the storage system includes an index data manager, a data read-write module, and a disk, where the disk includes a first storage area and a second storage area;

the index data manager is connected with the second storage area, and is used for generating index data of the first storage data, generating an index data file and sending the index data file to the second storage area when the generated index data amount reaches a preset data threshold value;

the data read-write module is connected with the first storage area and is used for acquiring the data fragments of the first storage data in real time and reading and writing the data fragments into the first storage area; wherein the first storage area is divided into a plurality of storage areas.

In a second aspect, an embodiment of the present application provides a data management method, applied to a storage system, where the method includes:

acquiring a data fragment in the first stored data;

writing the data fragment into a first storage area in real time;

generating index data, and writing the index data into an index data manager; the index data is used for indexing the first storage data, and the index data manager is used for generating an index data file when the index data amount stored in the index data manager reaches a preset data threshold value;

and writing the index data file into a second storage area.

In one embodiment, the storage system includes a plurality of the first storage areas, each first storage area is divided into a plurality of storage partitions, the first storage data includes a plurality of data fragments, and the storage system further includes a client;

the writing the data segment in the first storage area in real time includes:

selecting a corresponding storage partition according to a client channel of the client;

acquiring a storable region of the storage partition;

judging whether the data quantity of a first data segment in the first storage data is smaller than the storage quantity of the storable region or not;

And if the data volume of the first data segment in the first storage data is smaller than the storage volume of the storable region, writing the first data segment into the storage partition in real time.

In an embodiment, the index data includes event index data for recording storage data information;

the generating index data, writing the index data into the index data manager, includes:

acquiring first timestamp information of a first data segment in the first storage data;

correspondingly searching tail data fragments of the stored second storage data in the storage partition;

acquiring second timestamp information of a tail data segment of the second storage data;

judging whether the first time stamp information and the second time stamp information are continuous time information or not;

if the first time stamp information and the second time stamp information are continuous time information, second event index data of the second storage data are obtained;

after all the data fragments are written into the storage partition, associating first storage data with the second event index data;

and writing the associated second event index data into the index data manager.

In an embodiment, after determining whether the first timestamp information and the second timestamp information are continuous time information, the method further includes:

if the first time stamp information and the second time stamp information are not continuous time information, writing all the data fragments of the first storage data into the current storage partition, and generating first event index data associated with the first storage data; and writing the first event index data into the index data manager.

In an embodiment, the generating index data, writing the index data into the index data manager, further includes:

after the data fragments are written into the storage partition of the first storage area in real time, partition index data are generated;

associating the storage partition with the partition index data; the partition index data is used for recording storage information of the storage partition;

and writing the partition index data into the index data manager.

In an embodiment, the index data further includes segment index data;

the generating index data, writing the index data into the index data manager, further includes:

Generating first segment index data; the first segment index data associates data segments written into the storage partition;

the first segment index data is written into the index data manager.

In one embodiment, after writing the index data file into the second storage area, the method further comprises:

after the storage system is abnormal, inquiring the storage partition written in the data segment in a first preset time;

inquiring a second storage area corresponding to the storage partition;

acquiring tail fragment index data in the storage partition, and inquiring a first target data fragment corresponding to the storage partition in the first storage area according to the tail fragment index data;

and re-mapping the index data of the first storage data after the first target data segment is generated.

In an embodiment, the index data is written into the second storage area to carry time information;

after writing the index data file into the second storage area, further comprising:

acquiring a target client, and acquiring all corresponding storage partitions according to a second preset time period and a client channel of the target client;

inquiring a second storage area corresponding to each storage partition;

Determining a target second storage area according to the event index data;

acquiring target fragment index data in the second preset time period in the target second storage area;

acquiring a second target data segment corresponding to the target segment index data;

and reading the second target data segment in turn according to the time information.

In a third aspect, an embodiment of the present application provides a terminal device, including: at least one processor, a memory and a computer program stored in the memory and executable on the at least one processor, the processor implementing the data management method according to any of the second aspects described above when the computer program is executed.

In a fourth aspect, embodiments of the present application provide a computer readable storage medium storing a computer program which, when executed by a processor, implements a data management method as in any of the second aspects above.

In a fifth aspect, an embodiment of the present application provides a computer program product, which, when run on a terminal device, causes the terminal device to perform the data management method according to any one of the second aspects above.

It will be appreciated that the advantages of the second to fifth aspects may be found in the relevant description of the first aspect, and are not described here again.

Compared with the prior art, the embodiment of the application has the beneficial effects that: the internal structure of the storage system is divided, the disk is divided into a first storage area for storing video data and a second storage area for storing index data, the index data is stored in advance by using the index data manager, and the index data is stored in the second storage area when the index data amount of the index data manager reaches a preset data threshold value, so that the index data and the storage data cannot be written into the disk at the same time, the occupied disk IO is reduced, the disk performance is fully utilized, and the disk reading and writing speed is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of a storage system according to an embodiment of the present application;

FIG. 2 is a schematic diagram of an index data manager in a storage system according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a disk in a storage system according to an embodiment of the present application;

FIG. 4 is a schematic flow chart of an implementation of a data management method according to an embodiment of the present application;

FIG. 5 is a schematic flow chart of another implementation of the data management method according to the embodiment of the present application;

FIG. 6 is a schematic flow chart of another implementation of the data management method according to the embodiment of the present application;

FIG. 7 is a schematic diagram of a data management device according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of a terminal device according to an embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

Furthermore, the terms "first," "second," "third," and the like in the description of the present specification and in the appended claims, are used for distinguishing between descriptions and not necessarily for indicating or implying a relative importance.

The embodiment of the application provides a storage system, which can be various storage devices for storing programs and data in a computer, such as various memories, such as a magnetic core memory, a magnetic bubble memory and the like; the device can also be a device for storing information by utilizing a magnetic energy mode, such as a hard disk, a floppy disk and the like; but also devices for optically storing information, such as CDs or DVDs, are not limited thereto. Referring to fig. 1, fig. 1 illustrates a structure of a storage system provided by the present application.

As shown in fig. 1 and fig. 3, a storage system provided in an embodiment of the present application includes an index data manager 2, a data read-write module 5, and a disk 3, where the disk 3 includes a first storage area 31 and a second storage area 32;

the index data manager 2 is connected to the second storage area 32, and the index data manager 2 is configured to obtain index data for generating the first storage data, and generate an index data file and send the index data file to the second storage area 32 when the generated index data amount reaches a preset data threshold;

the data read-write module 5 is connected to the first storage area 31, and the data read-write module 5 is configured to acquire a data segment of the first stored data in real time, and read and write the data segment into the first storage area 31; wherein the first storage area 31 is divided into a plurality of storage partitions 311.

In application, the index data manager 2 is configured to generate index data, where the index data indexes first storage data, and the first storage data may be video data, audio data, digital data, or the like, which is not limited thereto. Specifically, the first storage data may be specifically video data, the storage system is a memory, the memory includes an index data manager 2, a data read-write module 5 and a disk 3, where the disk 3 is internally divided into a first storage area 31 and a second storage area 32, the first storage area 31 may be specifically a data partition for storing video data, and the second storage area 32 may be specifically a database for storing index data, that is, in one disk 3, an internal corresponding one database and one data partition. Referring to fig. 3, the data partition is further divided into a plurality of storage partitions 311 with equal size, and each storage partition 311 can be positioned to be 1GB, or can be positioned to be other sizes, and the specific storage space changes with the size of the disk 3, so as to reduce the space waste of the disk 3 and improve the space utilization of the disk 3.

In application, the video segment adopts video coding and decoding standard H.264/AVC to form an image group by I frames, P frames and B frames to code an image sequence, the image sequence is stored in a unit of 2s video data (gp) segment, the size generally changes along with the code rate, the same path of video is continuously stored in one partition, when the video reaches the tail of the partition, the partition is reselected and written, the gp can be used as a data segment, and each data segment has corresponding index data.

In application, the first storage data may be transmitted by the client 1 connected to the storage system, or the storage system may include the client 1 therein, where the client 1 is a port where the storage system interacts with an external system, and the video data, the audio data, and the like are transmitted according to a client channel of the client 1, which is not limited.

In application, the data read-write module 5 is used for writing video data into the data partition in real time and sequentially, is used for continuously storing the video data, is convenient for video data management, is convenient for continuously and sequentially reading the video data during reading, and avoids the random writing of the video data by the disk IO, thereby causing the scattered storage of the same path of video data. If stored in the different first storage area 31, there arises a problem that the reading speed is slow when reading video data. The preset data threshold may be a storable data file size set by a user, or may be a storable data file size preset by a system, which is not limited.

In this embodiment, the data read-write module 5 writes the video data into the disk 3 in real time, and when the index data amount reaches the preset data threshold value, the index data manager 2 regenerates the index data file to write the index data file into the database of the disk 3, so that the index data and the storage data cannot be written into the disk at the same time, the occupied disk IO is reduced, the disk performance is fully utilized, and the writing efficiency of the disk IO is improved.

Referring to fig. 1, in an embodiment, the index data manager 2 further includes an index pre-storage area 4, where the index pre-storage area 4 is configured to receive the index data in real time, and generate an index data file to send to the second storage area 32 when the index data amount reaches a threshold value.

In application, the index data manager 2 includes an index pre-storage area 4, where the index pre-storage area 4 may be a storage system or a database in the index data manager 2, and the index pre-storage area 4 is a temporary storage area. When the index data manager 2 receives the first storage data, it generates index data and stores the index data in the index pre-storage area 4, and the data read/write module 5 reads/writes the data segments of the first storage data in real time sequence according to the index data. When the index data amount of the index pre-storage area 4 reaches a preset data threshold value, an index data file is generated and sent to the second storage area 32, so that the index data and the first storage data can be stored separately and cannot be written into a disk at the same time, the occupied disk IO is reduced, the performance of the disk 3 is fully utilized, and the writing efficiency of the disk 3 is improved.

In a specific application, the index pre-storing area 4 may be a persistent database (locksdb), the index data manager 2 first stores index data in the form of a write-ahead logging (Wal) in the index pre-storing area 4, and when the Wal size in the index pre-storing area 4 reaches a certain threshold, generates a database file from the index data in the Wal, and writes the database file in the sst format into the disk 3. When the generated data file is written into the second storage area 32 in the disk 3 by using the persistent database, the new index data file does not need to traverse the disk 3 structure again, and the step of traversing the structure of the disk 3 when the index data is written into the disk according to the conventional mode is reduced after the new index data file is written into the position of the last index data file in the second storage area 32 according to the characteristic of the persistent database, so that the data writing efficiency is greatly improved.

Referring to FIG. 2, in one embodiment, the index data manager 2 includes a partition manager 21, an event manager 22, and a fragment manager 23;

the partition manager 21 is configured to generate partition index data of the data segment after the data segment is written into the storage partition 311 of the first storage area 31 in real time.

In application, partition manager 21 is configured to generate partition index data for the first storage data, where each video data is written to a different or the same storage partition 311, and generate corresponding partition index data for distinction. Specifically, after the first storage area is divided into a plurality of storage partitions 311 with equal sizes, the partition index data needs to record the status, the size, the usage, etc. of each storage partition 311 in the first storage area 31. Such as in video data writing and video data reading, it is desirable to determine or modify the usage of the memory partition 311. The video segment data in each storage partition 311 are sequentially arranged in time, and the partition index data can be written into the partition index data in sequence by the magnetic disk 3 according to time sequence, so that the selected target is determined to be positioned in the storage partition 311 in the first storage area 31 according to the time period quickly during reading.

The event manager 22 is configured to generate first event index data associated with the first storage data after the data segment of the first storage data is written into the first storage area 31.

In application, the event manager 22 is configured to generate first event index data for the first stored data. In particular, the event index data serves as a sort or label of video data. Specifically, the first storage data has a tag a, the second storage data has a tag B, and when both are stored in one first storage area 31, the target storage data can be indexed according to the data tag.

In other applications, the event may be considered a continuous piece of video and the first event index data may be considered the first video index data. The video data of one video can correspond to a plurality of event indexes, the video data of one video is stored in a plurality of sections, and each stored section of video data corresponds to one event index data, namely the events of the same video are not overlapped. The corresponding event index data of the video data of different paths of videos are different, and the video data of different paths of videos can be simultaneously stored in different first storage areas, namely, time coincidence can exist in a plurality of time index data corresponding to different paths of videos. The time attribute is written along with the event (video), the labels (event index data) of different paths of videos (events) are not identical, but can be sequentially and incrementally arranged in time, and can be written in the disk 3 in time sequence, and video data index is carried out through the event index data, and can also be converted into video index according to time.

The fragment manager 23 is configured to generate fragment index data; the fragment index data associates data fragments that have been written into the first storage area 31.

In application, the fragment manager 23 is configured to generate fragment index data. Specifically, the segment index data is metadata of recording one divided video data, for example, the video data is encoded into a plurality of video segments with equal length, and then corresponding segment index data is generated for each video segment, wherein the segment index data contains metadata of each video segment and is used for realizing accurate searching of the stored video data.

In other applications, the video data of the same path is also ordered video segments over time, and the key words (keys) of the segment index data are also continuously increased over time, so that the key words are added in sequence and backward in the event index data or the partition index data, and therefore, the segment index data of all the video segments in the time period can be also indexed according to the time period.

In this embodiment, multiple methods may be used to index the first storage data in multiple ways by creating multiple index data, so as to reduce the difficulty of indexing, and accurately index the data segments in the first storage data.

Referring to fig. 1, in another embodiment, the storage system includes a plurality of magnetic disks 3, and each magnetic disk 3 includes a first storage area 31 and a second storage area 32.

In application, the above-mentioned one magnetic disk 3 is correspondingly divided into a first storage area 31 and a second storage area 32, where the first storage area 31 is a data partition in the magnetic disk 3, and the second storage area 32 is a database area in the magnetic disk 3, and is used for managing video data in the data partition according to index data in the database. When the system is started, index data can be directly loaded into the memory of the storage system from the magnetic disk 3 through the index pre-storage area 4 by the index data manager 2, and then the data read-write module 5 in the storage system reads video data in the magnetic disk 3 according to the index data. And the time of writing the index data and the video data into the disk 3 is not synchronous, so that when the index data is lost, the video data written into the disk 3 can be directly searched through the index data manager, and incremental recovery is initiated from the disk 3, so that the lost index data can be generated more conveniently and rapidly.

Referring to fig. 4, an embodiment of the present application further provides a data management method, applied to a storage system, where the method includes:

s101, acquiring a data fragment in the first storage data.

S102, writing the data fragment into a first storage area in real time.

S103, generating index data, and writing the index data into an index data manager; the index data is used for indexing the first storage data, and the index data manager is used for generating an index data file when the index data amount stored in the index data manager reaches a preset data threshold.

S104, writing the index data file into a second storage area.

In application, the first storage data may be video data, audio data, digital data, or the like, which is not limited. The storage system is a memory, and the memory includes an index data manager, a data read-write module and a disk, wherein the disk is internally divided into a first storage area and a second storage area, the first storage area can be specifically used for data partition for storing video data, and the second storage area can be specifically used for database for storing index data, namely, in a disk, the interior of the disk corresponds to one database and one data partition.

In application, the video data can be written into the data partition in real time according to the time sequence, the continuous storage video data can be written in real time sequence, the video data management is convenient, the continuous sequential reading of the video data is convenient during the reading, the problem that the same path of video data is scattered in storage as stored in different first storage areas due to the fact that the video data are randomly written into the video data by the magnetic disk IO is avoided, and the reading speed is low during the reading of the video data is generated. The preset data threshold may be a storable data file size set by a user, or may be a storable data file size preset by a system, which is not limited.

In other applications, the index data manager further includes an index pre-storage area, where the index pre-storage area may be a part of components in the index data manager, a storage system or a database, and the index pre-storage area is used as a temporary storage area of the index data manager. The index data manager can generate index data of the first storage data, prestore the index data in the index prestored area, generate an index data file and send the index data file to the second storage area when the index data amount in the index prestored area reaches a preset data threshold value, so that the index data and the first storage data can be stored separately and cannot be written into a disk at the same time, the occupied disk IO is reduced, the disk performance is fully utilized, and the disk read-write speed is improved.

In this embodiment, when the disk in the memory writes video data, the data read-write module performs real-time disk brushing on the video data, writes the video data into the data partition in real time, and the index data generated by the index data manager is not written into the disk simultaneously with the video data, but stores the index data into the second storage area when the index data amount of the index data manager reaches a preset data threshold value, so that the index data and the storage data cannot be written into the disk simultaneously, the occupied disk IO is reduced, the disk performance is fully utilized, and the disk writing efficiency is improved.

Referring to fig. 5, in an embodiment, the storage system includes a plurality of the first storage areas, each first storage area is divided into a plurality of storage partitions, the first storage data includes a plurality of data segments, and the storage system further includes a client; step 102, including:

s201, selecting a corresponding storage partition according to a client channel of the client.

In an application, the client is a storage client, and is configured to receive the first storage data, and select a disk where the first storage data is to be stored. When one client works, only one client channel is opened, one disk is selected for writing first storage data, one storage system can correspond to one or more client channels, a plurality of disks can be arranged in the storage system, the first storage data can be stored in the plurality of disks at the same time, but video data stored at the same time cannot be stored in the same first storage area. The selection may be performed according to whether the first storage area is in a use state, and if the first storage area is in a use state, other unused first storage areas are selected for storage. If the first storage area is in an unused state, the client may specifically select a certain storage partition in the first storage area to store video data, for example, may select according to a storage space of each storage partition in the first storage area, which is not limited.

S202, obtaining a storable region of the storage partition.

In an application, the above-mentioned storable region may be a storage space remaining after storing other video data for the storage partition, and the remaining storage space may be used to store the first storage data.

S203, judging whether the data volume of the first data segment in the first storage data is smaller than the storage volume of the storable region.

In the application, the data segments of the first storage data are written in time sequence, and the first storage data are written from the first data segments in sequence until the first storage data are written, or after the storage partition is fully written, the writing is carried out again by using another storage partition after the use. For example, referring to fig. 3, after storage partition 1 is full of first storage data, writing may continue in storage partition 2. The first data segment may be specifically 2S video data, and when the storage amount of the storable area is greater than 2S video data, the first storage data may start to be written into the storage partition, and after the storage area is full, the rest storage partitions are selected for storage.

In other embodiments, the step 203 may specifically further include:

and acquiring tail data fragments of the written second storage data in the memory partition.

And judging the tail data segment to a storable area at the tail of the first storage area.

And judging whether the first data segment in the first storage data is smaller than the storage amount of the storable region or not.

And if the first data segment in the first storage data is larger than the storage amount of the storable region, reselecting the first storage region.

In the application, the method judges whether the first storage data can be written according to the storage amount from the tail data segment of the written second storage data to the tail of the first storage area, and determines the area where the video data can be written in the first storage area. Specifically, the storage system includes a plurality of disks, such as disk 1 and disk 2, wherein disk 1 is divided into storage partition 1, storage partition 2 to storage partition N, wherein the storage partition N is a tail storage partition in the first storage area, when the first storage data is written into the storage partition N again, if the storage amount of the storable region in the storage partition is too small, if the storage system determines that the disk 1 is full of the second storage data, in order to avoid the situation that the disk is selected again for storage after part of the first storage data is stored in the disk 1, when the storage amount is determined to be too small, the storage partition of the first storage area in the other disk is selected for storage. The object is to write first stored data in a more stable sequential order without causing a problem of data dispersion.

S204, if the data volume of the first data segment in the first storage data is smaller than the storage volume of the storable region, writing the first data segment into the storage partition in real time.

In the application, the first storage data are sequentially written into the storage partition, when the first data segment is smaller than the storage capacity of the storable region, the first storage region of the rest magnetic disk is selected to write the first storage data, and the purpose is to reduce the waste of the magnetic disk space and maximally improve the utilization rate of the magnetic disk space.

Referring to fig. 6, in one embodiment, the index data includes event index data for recording storage data information; step S103 includes:

s301, acquiring first timestamp information of a first data segment in the first storage data.

S302, correspondingly searching tail data fragments of the stored second storage data in the storage partition.

S303, acquiring second timestamp information of the tail data segment of the second storage data.

S304, judging whether the first time stamp information and the second time stamp information are continuous time information or not.

S305, if the first time stamp information and the second time stamp information are continuous time information, second event index data of the second storage data is obtained.

S306, after all the data fragments are written into the storage partition, the first storage data are associated with the second event index data.

S307, writing the associated second event index data into the index data manager.

In the application, the data segments are sequentially arranged in time sequence when being acquired, so that when the data segments are acquired, time information carried by the data segments can be acquired, and whether the first stored data is required to be associated with the second event index data is determined according to whether the first time stamp information of the first data segment in the first stored data and the second time stamp information of the tail data segment of the second stored data are continuous or not.

In an application, the continuous time information may have a certain time fluctuation threshold, that is, the second timestamp information of the tail data segment of the second storage data is 9:00:01S, the first timestamp information of the first data segment in the first storage data is 9:00:01S-9:00:05S is considered as continuous time information of the first time stamp and the second time stamp information, and the storage system may be abnormal in the storage process, so that a fluctuation threshold value can be set to allow errors within a certain time. The fluctuation threshold may be any value set by the user, and whether to select new event index data or not may be selected according to a user instruction, or the new event index data may be changed at any time, which is not limited.

Referring to fig. 6, in an embodiment, after step S304, the method further includes:

s308, if the first time stamp information and the second time stamp information are not continuous time information, writing all the data fragments of the first storage data into the current first storage area, and generating first event index data associated with the first storage data; and writing the first event index data into the index pre-storage area.

In application, a first storage area can store two first storage data, namely two paths of video data, different event index data are needed to be adopted for the two paths of video data in order to distinguish the two paths of video data, and the two paths of video data are used for indexing the video data in a partitioning mode, and the data partition which is searched out simultaneously has multiple paths of video data, so that target video data can be further determined according to the event index data, and the effect of classifying or labeling different video data is achieved.

In an embodiment, step S103 further includes:

and writing the data fragments into the storage partition of the first storage area in real time, and generating partition index data.

Associating the storage partition with the partition index data; the partition index data is used for recording storage information of the storage partition.

And writing the partition index data into the index pre-storage area.

In application, the data fragments are written into the storage partition in real time, the partition index data are written into the index pre-storage area first, the data fragments and the partition index data are written into the disk at different time, and the writing efficiency of the disk is improved maximally.

The partition index data is used for recording the storage information of the first storage area, and each video data is written into different or same storage partition to generate corresponding partition index data for distinguishing. Specifically, after the first storage area is divided into a plurality of storage partitions with equal sizes, the partition index data needs to record the state, the size, the use condition and the like of each storage partition in the first storage area. Such as in video data writing and video data reading, it is necessary to determine or modify the memory partition usage. The video segment data in each storage partition are arranged continuously in time, and the partition index data can be written into the partition index data in a disk sequence by time sequencing, so that the position of the selected target in the storage partition in the first storage area can be determined quickly according to the time period during reading.

In an embodiment, the index data further includes segment index data; step S103 further includes:

the first segment index data is written into the index data manager.

In the application, the segment index data is metadata of recording one divided video data, for example, the video data is encoded into a plurality of video segments with equal length, and then corresponding segment index data is generated for each video segment, wherein the segment index data contains metadata of each video segment and is used for realizing accurate searching of the stored video data. Specifically, the video segment adopts the video coding and decoding standard H.264/AVC to form an image group by using I frames, P frames and B frames to code an image sequence, the image sequence is stored by taking 2s video data (gop) as a unit, the size generally changes along with the code rate, the same path of video is continuously stored in one partition, when the video reaches the tail of the partition, the partition is reselected and written, the rap can be used as a data segment, and the first segment index data of the data segment is generated.

In an embodiment, after step S104, further includes:

After the storage system is abnormal, inquiring the storage partition written in the data segment in a first preset time.

In application, the abnormal occurrence of the storage system can be abnormal power failure when the storage system stores video data, damage to the internal structure of the storage system, and restarting of the storage system, wherein the abnormal occurrence of the storage system can cause the loss of index data which is not written into the second storage area, namely the index data in the index pre-storage area in the index data manager can be lost, and the index data and the data fragments are stored and separated, and the data fragments are written into the data partitions in the disk in real time. Therefore, the index data in the index pre-storage area is lost, which can cause the index data in the database to be inconsistent with the video data in the data partition, and the index data in the index pre-storage area cannot be indexed after the database is restarted and loaded.

In application, the first preset time may be a preset time period of the system, or may be a time period set by a user, which is not limited. Specifically, in the first preset time period of 1min, after the storage system is abnormally powered off and restarted, the storage system records that the time point is 9:01, then storage system query 9: and writing the first storage area of the data segment in the time period of 00-9:01.

And inquiring a second storage area corresponding to the storage partition.

And acquiring tail fragment index data in the second storage area, and inquiring a first target data fragment in the corresponding storage partition according to the tail fragment index data.

In the application, the first storage area and the second storage area are areas which are divided in one-to-one mode in the magnetic disk, and the corresponding second storage area can be found by inquiring the storage partition in the first storage area. Because the storage partition is written in real time to the video data, the video data before the storage system is abnormal are written in real time, and for the second storage area, one part of the index data before the storage system is abnormal is written in the second storage area in the form of a database file, and the other part of the index data is only written in the index pre-storage area in the index data manager, namely the index data in the index pre-storage area is not written in the second storage area when the storage system is abnormal because the data file is not formed yet. Therefore, the first target data segment corresponding to the storage partition can be queried according to the tail segment index data in the second storage area, and the video data after the first target data segment are not correspondingly provided with index data.

In the application, for the video data after the first target data segment, after the index data of the portion is correspondingly established, the rest of the video data is sequentially written into the corresponding storage partition in real time again, that is, the data segment obtained by the video data is repeatedly written into the storage partition in real time, the index data is generated, the index data is written into the index pre-storage area, the index data file is generated, and the index data file is written into the database.

In this embodiment, when the storage system is abnormally restarted, the incremental recovery can be restarted from the disk only by loading the second storage area, and the video data in the first storage area does not need to be loaded, so that the storage system is more quickly loaded.

In an embodiment, the index data is written into the second storage area to carry time information; after step S104, further includes:

and acquiring the target client, and acquiring all corresponding storage partitions according to a second preset time period and a client channel of the target client.

The above-mentioned time information includes, but is not limited to, time information for writing the index data into the database, and time information for generating the index data (partition index data, fragment index data, event index data), which is not limited thereto. The second preset time period may specifically be 9:00-9:30, firstly selecting a client channel of a target client from a plurality of clients, and then acquiring 9 in the client channel by a storage system: 00-9:30, the index data queries the corresponding data segment. Or, acquiring event index data of all client channels in a second preset time period through the second preset time period, and then inquiring and reading data fragments according to the fragment index data, which is not limited.

And inquiring the second storage areas corresponding to the storage partitions.

In application, the first storage area and the second storage area described above may be in one-to-one correspondence, that is, the disk is divided into a second storage area for storing index data and a first storage area for storing video data, and the first storage area is divided into a plurality of storage areas. Thus, the corresponding second storage area may be found by querying the storage partition. If all the memory partitions come from different first memory areas respectively in the second preset time period, one corresponding second memory area can be queried according to each first memory area, so that a plurality of second memory areas can be queried according to a plurality of first memory areas.

And determining a target second storage area according to the event index data.

In application, the event index data is the event index data generated by writing video data into the storage partition, wherein each path of video data can correspond to one or more event index data. If the first storage data is stored in the first storage area, the event index data is first event index data, and for a plurality of second storage areas acquired according to the second preset time period, when the storage partition corresponding to each second storage area stores video data, the second storage area records the event index data of each video data, and the target second storage area can be further selected according to the event index data.

And acquiring target fragment index data in the second preset time period in the target second storage area.

In the application, the segment index data also carries corresponding time information when being written into the second storage area, so that the time period of the target segment index data to be read can be further selected in the selected target second storage area, and the corresponding second target data segment is acquired according to the target segment index data.

And obtaining a second target data segment corresponding to the target segment index data.

In application, when video data is written into a data partition in real time by a data segment, the data segment also carries written time information, and corresponding segment index data is written into a database to also carry corresponding time information. Thus, this 9:00-9: segment index data in the 30 time period, acquiring corresponding second target data segments according to the segment index data, and obtaining the second target data segments from 9:00 starts to read video data in this period sequentially.

In application, the video data are read and written in a storage partition in real time and in sequence, namely, the same video is continuously stored in the storage partition, and when the video data reach the tail of the storage partition, the storage partition after reselection is written in, so that the video data in the time period can be read for orderly playing, and the data reading is convenient.

In this embodiment, the first storage data is sequentially written into the first storage area, so that the first storage data is also continuously and sequentially read when being read, thereby effectively improving the reading efficiency of the continuous video. And the multi-path storage data is managed in a partitioning mode by a storage partition mode, when the first storage area is full, the continuous storage data needing to be deleted can be selected by selecting the storage partition, and the management of the storage data in the disk is facilitated. Compared with the prior art, when video data are randomly stored in a disk, when part of the video data in a first storage area are deleted, the storage space occupied by the video data originally is emptied, and when new video data are continuously written into and deleted from the data partition, a plurality of scattered storage spaces are generated, so that a larger data file is placed in a plurality of discontinuous storage spaces. When the video data are read and written, the magnetic head is required to continuously adjust the positions of the magnetic tracks so as to read and write the data at different positions, and when the large-scale program is started, a large number of small files are required to be read, and the files are not stored continuously, so that the difficulty of reading the video data is increased.

In this embodiment, the internal structure of the storage system is divided, the disk is divided into a first storage area for storing video data and a second storage area for storing index data, the index data is stored in advance by using the index data manager, and when the index data amount of the index data manager reaches a preset data threshold value, the index data is stored in the second storage area, so that the index data and the stored data cannot be written into the disk at the same time, the disk IO is reduced, the disk performance is fully utilized, and the disk writing efficiency is improved.

Referring to fig. 7, the present application further provides a data management apparatus 100, including a first acquisition module 10, a first writing module 20, a first generating module 30, and a second writing module 40.

The first acquiring module 10 is configured to acquire a data segment in the first stored data.

The first writing module 20 is configured to write the data fragment in the first storage area in real time.

The first generation module 30 is configured to generate index data, and write the index data into an index data manager; the index data is used for indexing the first storage data, and the index data manager is used for generating an index data file when the index data amount stored in the index data manager reaches a preset data threshold.

The second writing module 40 is configured to write the index data file into the second storage area.

In one embodiment, the storage system includes a plurality of the first storage areas, each first storage area is divided into a plurality of storage partitions, the first storage data includes a plurality of data fragments, and the storage system further includes a client; the first writing module 20 is further configured to:

acquiring a storable region of the storage partition;

and if the data volume of the first data segment in the first storage data is smaller than the storage volume of the storable region, writing the first data segment into the first storage region in real time, and writing the rest data segments into the storage region in real time sequentially.

In an embodiment, the index data includes event index data for recording storage data information; the first generation module 30 is also for:

after all the data fragments are written into the storage area, associating first storage data with the second event index data;

and writing the associated second event index data into the index data manager.

In an embodiment, the first generating module 30 is further configured to:

if the first time stamp information and the second time stamp information are not continuous time information, generating first event index data associated with the first storage data after writing all the data fragments of the first storage data into the current first storage area; and writing the first event index data into the index data manager.

In an embodiment, the first generating module 30 is further configured to:

and writing the partition index data into the index data manager.

In an embodiment, the index data further includes segment index data; the first generation module is also for:

generating first segment index data; the first segment index data associates data segments written into the first storage area;

the first segment index data is written into the index data manager.

In an embodiment, the data management apparatus further includes a third writing module, a first querying module, a second obtaining module, and a second generating module.

And the third writing module is used for inquiring the storage partition written in the data segment in the first preset time after the storage system is abnormal.

The first query module is used for querying a second storage area corresponding to the storage partition.

The second acquisition module is used for acquiring the tail segment index data in the second storage area, and inquiring the first target data segment corresponding to the storage partition in the first storage area according to the tail segment index data.

The second generation module is used for re-correspondingly generating index data of the first storage data after the first target data segment.

In an embodiment, the writing of the index data into the second storage area carries time information, and the data management device further includes a third acquisition module, a second query module, a first determination module, a fourth acquisition module, a fifth acquisition module, and a first reading module.

The third acquisition module is used for acquiring the target client and acquiring all corresponding storage partitions according to the second preset time period and the client channels of the target client.

And the second query module is used for querying a second storage area corresponding to each storage partition.

The first determining module is used for determining a target second storage area according to the event index data.

The fourth obtaining module is configured to obtain, in the target second storage area, target fragment index data in the second preset time period.

The fifth acquisition module is used for acquiring a second target data segment corresponding to the target segment index data;

the first reading module is used for sequentially reading the second target data fragments according to the time information.

The embodiment of the application also provides a terminal device, which comprises: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, which when executed by the processor performs the steps of any of the various method embodiments described above.

Embodiments of the present application also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements steps for implementing the various method embodiments described above.

Embodiments of the present application provide a computer program product which, when run on a mobile terminal, causes the mobile terminal to perform steps that enable the implementation of the method embodiments described above.

Fig. 8 is a schematic diagram of a terminal device 50 according to an embodiment of the present application. As shown in fig. 8, the terminal device 50 of this embodiment includes: a processor 503, a memory 501 and a computer program 502 stored in said memory 501 and executable on said processor 503. The processor 503, when executing the computer program 502, implements the steps in the above-described respective method embodiments, such as steps S101 to S104 shown in fig. 4. Alternatively, the processor 503 may perform the functions of the modules/units in the above-described device embodiments when executing the computer program 502.

Illustratively, the computer program 502 may be partitioned into one or more modules/units that are stored in the memory 501 and executed by the processor 503 to accomplish the present application. The one or more modules/units may be a series of computer program instruction segments capable of performing the specified functions, which instruction segments are used for describing the execution of the computer program 502 in the terminal device 50. For example, the computer program 502 may be divided into a first acquisition module, a first writing module, a first generating module, and a second writing module, where each module specifically functions as follows:

acquiring a data fragment in the first stored data;

writing the data fragment into a first storage area in real time;

and writing the index data file into a second storage area.

The terminal device 50 may be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The terminal device may include, but is not limited to, a processor 503, a memory 501. It will be appreciated by those skilled in the art that fig. 8 is merely an example of the terminal device 50 and is not limiting of the terminal device 50, and may include more or fewer components than shown, or may combine certain components, or different components, e.g., the terminal device may also include input and output devices, network access devices, buses, etc.

The processor 503 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 501 may be an internal storage unit of the terminal device 50, for example, a hard disk or a memory of the terminal device 50. The memory 501 may also be an external storage device of the terminal device 50, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the terminal device 50. In one embodiment, the memory 501 may also include both internal storage units and external storage devices of the terminal device 50. The memory 501 is used for storing the computer program and other programs and data required by the terminal device. The memory 501 may also be used to temporarily store data that has been output or is to be output.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims

1. A storage system, wherein the storage system comprises an index data manager, a data read-write module and a disk, wherein the index data manager comprises an event manager;

the disk comprises a first storage area and a second storage area;

the data read-write module is connected with the first storage area and is used for acquiring the data fragments of the first storage data in real time and reading and writing the data fragments into the first storage area; wherein the first storage area is divided into a plurality of storage areas;

The event manager is configured to generate first event index data associated with the first storage data after the data segment of the first storage data is written into the first storage area;

the event manager is further configured to obtain first timestamp information of a first data segment in the first stored data; correspondingly searching tail data fragments of the stored second storage data in the storage partition; acquiring second timestamp information of a tail data segment of the second storage data; judging whether the first time stamp information and the second time stamp information are continuous time information or not; if the first time stamp information and the second time stamp information are continuous time information, second event index data of the second storage data are obtained; and after all the data fragments are written into the storage partition, the first storage data are associated with the second event index data and stored.

2. The storage system of claim 1, wherein said index data manager further comprises an index pre-store for receiving said index data in real time and generating an index data file for transmission to said second store when said index data amount reaches a threshold.

3. The storage system of claim 1, wherein the index data manager further comprises a partition manager and a fragment manager;

the partition manager is configured to generate partition index data of the data segment after the data segment is written into the storage partition of the first storage area in real time;

the fragment manager is used for generating fragment index data; the fragment index data associates data fragments that have been written into the first storage area.

4. A data management method, characterized in that the method is applied to a storage system, the storage system comprises a plurality of first storage areas, each first storage area is divided into a plurality of storage partitions, and first storage data comprises a plurality of data fragments; the method comprises the following steps:

acquiring a data fragment in the first stored data;

writing the data fragment into a first storage area in real time;

generating index data, and writing the index data into an index data manager; the index data is used for indexing the first storage data, and the index data manager is used for generating an index data file when the index data amount stored in the index data manager reaches a preset data threshold value; the index data comprises event index data, and the event index data is used for recording and storing data information;

Writing the index data file into a second storage area;

and writing the associated second event index data into the index data manager.

5. The data management method of claim 4, wherein the storage system further comprises a client;

the writing the data segment in the first storage area in real time includes:

Acquiring a storable region of the storage partition;

6. The data management method according to claim 4, wherein after said determining whether the first time stamp information and the second time stamp information are continuous time information, further comprising:

7. The data management method according to claim 5, wherein the generating index data, writing the index data in the index data manager, further comprises:

and writing the partition index data into the index data manager.

8. The data management method according to any one of claims 5 to 7, wherein the index data further includes fragment index data;

the first segment index data is written into the index data manager.

9. The data management method according to claim 8, wherein after writing the index data file into the second storage area, further comprising:

inquiring a second storage area corresponding to the storage partition;

acquiring tail fragment index data in the second storage area, and inquiring a first target data fragment corresponding to the storage partition in the first storage area according to the tail fragment index data;

10. The data management method according to claim 5, wherein the index data written to the second storage area carries time information;

inquiring a second storage area corresponding to each storage partition;

determining a target second storage area according to the event index data;

11. A computer readable storage medium storing a computer program, which when executed by a processor implements the method of any one of claims 4 to 10.