CN116455920A

CN116455920A - Data storage method, system, computer equipment and storage medium

Info

Publication number: CN116455920A
Application number: CN202310386225.4A
Authority: CN
Inventors: 夏华丽; 郭军; 张志勇
Original assignee: Inspur Jinan data Technology Co ltd
Current assignee: Inspur Jinan data Technology Co ltd
Priority date: 2023-04-07
Filing date: 2023-04-07
Publication date: 2023-07-18

Abstract

The invention relates to the technical field of storage, and particularly discloses a data storage method, a system, computer equipment and a storage medium, wherein the method comprises the following steps: starting a timer of a node in the cluster according to a preset condition; monitoring whether the timer has timed out; responding to the timeout of the timer, initiating election based on the node with the timeout of the timer, and voting the node initiating the election based on other nodes in the cluster and the node initiating the election; and determining whether the node initiating the election can become a main center service based on the voting result so as to bear data storage service based on the main center service. By the scheme of the invention, high-availability data storage is realized in the server cluster.

Description

Data storage method, system, computer equipment and storage medium

Technical Field

The present invention relates to the field of storage technologies, and in particular, to a data storage method, a data storage system, a computer device, and a storage medium.

Background

With the rapid development of security technology, security monitoring gradually evolves towards the direction of digitalization, networking and intellectualization, and accordingly, the pressure of storing massive video data is brought forward, and the traditional single-machine storage system has poor performance in the aspects of capacity, capacity growth speed, data backup, data security and the like, so that a distributed storage system realized by cluster application and network technology is promoted, and therefore, higher requirements are put forward on capacity, read-write performance, reliability, expansibility and the like of storage equipment. The real-time video storage in the security monitoring industry is to record video streams generated by a video camera for 7 x 24 hours without interruption so as to carry out subsequent video on-demand, downloading, calculation and other applications on the video streams, and based on the characteristics of the industry, a video storage system is generally divided into a storage service layer and a storage data layer in the industry, the storage service layer is responsible for storage management of video stream data, bears the service function of video storage and generates and manages metadata related to video services, the storage data layer is abutted and resources provided by the storage data layer are distributed, and relevant information of the data storage data layer is recorded so as to effectively read the data in the follow-up. The storage data layer is used for providing physical storage space for data, ensuring uniform distribution of the storage space for the data, providing safety protection for stored data, ensuring IO (Input Output) performance of data reading and writing, and generally comprises a storage resource pool provided by a physical disk, a file system, block equipment, object storage, a third party storage system and the like. Because the storage service layer is required to support the video service function and ensure the characteristics of the storage system, and is also required to be effectively combined with the storage data layer, how to realize the efficient storage service layer becomes an important problem to be solved by security monitoring manufacturers.

In the prior art, one implementation scheme is to perform clustered management on NVR (Network Video Recorder, i.e., network video storage), where the system includes a network encoder, a plurality of network video storages (i.e., NVR) and an NVR manager, where the network encoder, the NVR manager and each NVR are connected to an IP (Internet Protocol internet protocol) network, and each NVR supports a network storage protocol on which the network encoder directly accesses audio and video data, so as to directly store the audio and video data encoded and output by the network encoder on the NVR; the NVR manager comprises a mapping module and a monitoring module, wherein the mapping module stores the mapping relation between the network encoder and each NVR, and the monitoring module monitors and manages audio and video data received by each NVR; each NVR receives the audio and video data encoded by the network encoder through the IP network under the control of the NVR manager, performs video recording, and responds to the access request of the client or the network encoder, queries or plays back or stores video recording data. The implementation scheme couples the video application service with the storage system, can not virtualize and manage storage resources of all NVRs, has the problem of single point of failure, can not provide a high-availability storage system, and is unsatisfactory in reliability, dilatancy and compatibility.

In some optimization schemes, a technology separates a video application service from a storage system, a storage node only processes the video storage service and analyzes a video recording plan issued to the storage node, and video streams are acquired from front-end equipment corresponding to the video recording plan and stored. This approach also does not isolate streaming traffic from the storage system nor does it provide an efficient high availability video storage system.

In the NVR cluster implementation scheme or the optimization scheme, the streaming service function is not isolated from the storage system, all types of video service metadata are generated and managed by each storage node service, all service processing is undertaken, the request of a client is responded, the use of storage resources also greatly limits the compatibility of the storage system, and a high-availability, easily-expandable and well-compatible distributed video storage system cannot be realized. The following problems are common in the current implementation schemes:

1) After the management node fails, the data writing operation of the cluster is abnormal, and video is lost.

2) The metadata generation and management mode causes that when video searching is carried out, searching is carried out on all storage nodes, the returned results of all storage nodes are combined and then returned to a caller, and second-level return of video searching cannot be achieved.

3) The coupling of the streaming service and the storage system limits service failover and the use of storage space, such as the failure of a certain storage node, the streaming service and the storage service of the failed node can be switched to other storage nodes, and load balancing is performed on the streaming service and the storage service after the recovery of the failed node, which can result in the waste of the underwritten physical storage space occupied by certain channels in the storage period.

4) The management and use of storage resources greatly limits the compatibility of the storage system and cannot dynamically adapt to multiple types of storage data layers.

Disclosure of Invention

In view of this, the invention provides a data storage method, a system, a computer device and a storage medium, which implement a main selection algorithm to ensure that main service can be selected to bear the operation of the cluster in any scene, and separate video storage business from video data reading and writing to bear by different service processes, so as to solve the problems in usability, expansibility, compatibility and the like in the prior art.

Based on the above object, an aspect of the embodiments of the present invention provides a data storage method, which specifically includes the following steps:

starting a timer of a node in the cluster according to a preset condition;

Monitoring whether the timer has timed out;

responding to the timeout of the timer, initiating election based on the node with the timeout of the timer, and voting the node initiating the election based on other nodes in the cluster and the node initiating the election;

and determining whether the node initiating the election can become a main center service based on the voting result so as to bear data storage service based on the main center service.

In some embodiments, voting the election-initiating node based on other nodes in the cluster and the election-initiating node itself comprises:

adding 1 to the period of the node initiating the election, sending a voting instruction to the other nodes based on the period of the node initiating the election, and voting the node initiating the election;

and receiving voting results returned by the other nodes based on the voting instructions.

In some embodiments, after receiving the voting instruction, the other node further includes:

voting the node initiating the election based on the period of the node initiating the election and the period of the other nodes, and returning the voting result to the node initiating the election; or alternatively, the first and second heat exchangers may be,

and responding to the faults of the other nodes, and returning voting results of network connection errors to the node initiating the election.

In some embodiments, voting the election-initiating node based on its tenure with the tenure of the other nodes themselves comprises:

comparing the period of the node initiating the election with the period of the other node itself;

if the period of the node initiating the election is smaller than the period of the other nodes, refusing to vote on the node initiating the election;

if the period of the node initiating the election is larger than the period of the other nodes, voting is carried out on the node initiating the election;

if the period of the node initiating the election is equal to the period of the other node, voting the node initiating the election based on the time of the node initiating the election and the time of the other node initiating the election.

In some implementations, determining whether the election-initiating node can become a primary hub service based on the voting results includes:

judging whether the number of surviving nodes in the cluster is more than half of the total node number based on the voting result;

if the number of the surviving nodes is greater than half of the total number of nodes, obtaining the number of votes obtained by the nodes initiating the election based on the voting result, and determining whether the nodes initiating the election can become a main center service based on the number of votes obtained by the nodes initiating the election and the total number of nodes;

If the number of the surviving nodes is less than half of the total node number, acquiring the node ID of the node initiating the election, and determining whether the node initiating the election can become a main center service based on the node ID.

In some embodiments, determining whether the election-initiating node can become a primary hub service based on the number of votes obtained by the election-initiating node and a total number of nodes comprises:

judging whether the number of votes obtained by the nodes initiating election exceeds half of the total node number;

if the number of votes obtained by the node initiating the election exceeds half of the total node number, determining that the node initiating the election becomes a main center service;

and if the number of votes obtained by the node initiating the election does not exceed half of the total number of nodes, returning to the step of initiating the election by the node based on the timeout of the timer in response to the timeout of the timer.

In some implementations, determining whether the election-initiating node can become a primary hub service based on the node ID includes:

determining that the election-initiating node is a primary hub service in response to the node ID being a minimum of surviving nodes;

and returning to the step of initiating election by the node based on the timer timeout in response to the node ID not being the minimum of surviving nodes.

In some implementations, the nodes include a node state that is any one of a follower, a candidate, or a leader.

In some embodiments, the node state of the node initiating the election is a candidate, the node state of the node where the primary hub service is located is a leader, and the node states of the remaining nodes are followers.

In some embodiments, the timer is used to count the time waiting to receive a leader heartbeat notification;

the method further comprises the steps of:

a heartbeat notification is sent to other nodes based on the node timing that becomes the primary hub service.

In some embodiments, the data storage services include any one or more of cluster management, load balancing, global metadata management, video retrieval, video locking, and time period elimination detection.

In some implementations, assuming the cluster management based on the primary hub service includes: managing and monitoring the state and resource information of all node services in the cluster based on the main center service;

in some implementations, global metadata management includes node information metadata and channel information metadata, assuming the global metadata management based on the primary hub service includes: and in response to the starting of the main center service, loading the node information metadata and the channel information metadata from a center metadata base into a memory of a node where the main center service is located, and periodically and permanently updating the node information metadata and the channel information metadata in the memory into the center metadata base.

In some embodiments, undertaking the video retrieval based on the primary central service includes: receiving a channel video search instruction, and inquiring a time period list meeting search conditions from channel information metadata based on the channel video search instruction;

in some embodiments, assuming the video lock based on the primary central service comprises: receiving a video locking instruction, inquiring all time periods conforming to a locking time range from channel information metadata based on the video locking instruction, setting the state of the time periods as locking, and sending a command for locking the current time period to node service where the time periods are located;

in some embodiments, undertaking the period of time elimination detection based on the primary central service includes: and traversing the channel information metadata periodically based on the main center service to detect and judge a time period list of each channel, and determining whether to eliminate the time period based on a detection judging result and the state of the time period.

In another aspect of the embodiment of the present invention, there is also provided a data storage system, including: a central service configured to perform the steps of:

Starting a timer of a node where the center service is located in the cluster according to a preset condition;

monitoring whether the timer has timed out;

In yet another aspect of the embodiment of the present invention, there is also provided a computer apparatus, including: at least one processor; and a memory storing a computer program executable on the processor, which when executed by the processor, performs the steps of the method as above.

In yet another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium storing a computer program which, when executed by a processor, implements the method steps as described above.

The invention has at least the following beneficial technical effects: the invention realizes a high availability, high performance and easy expansion data storage scheme, separates the video storage service from the data read-write to bear by different service processes, uses the main selection algorithm to select the main center service to perform unified channel management, channel information metadata management, cluster management, load balancing, video searching, video locking/unlocking, video data elimination and other functions, and can be applied to the use scene of security monitoring service. Compared with the prior art, the method has the following advantages:

1. The performance, video searching only needs the main center service to search the time period list cached in the memory, the searching in a long time range can also achieve second-level return, the data reading and writing is directly interacted with the node service, no transit and bottleneck exist, and network interaction is not needed between the video stream and the storage device under most conditions;

2. availability, because for each node in the system cluster, the system is a storage system, and supports all functional characteristics of the system, and can process all services, the system has dynamic fault switching;

3. the method has the advantages that the method is expandable, each node can process all services, and load balancing considers two dimensions of flow and storage space, so that node adding and deleting operations can be performed under the condition that the services are not interrupted;

4. the method has the advantages that the cost is low, when a certain channel applies for writing nodes, the block files which are not fully written in the channel are preferentially written in, and then new block files are distributed to the channel, so that each channel only occupies one block file which is not fully written in most cases, and space waste is avoided.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other embodiments may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a block diagram of one embodiment of a data storage method provided by the present invention;

FIG. 2 is a flowchart illustrating an embodiment of a node status switching method according to the present invention;

FIG. 3 is a flow chart of an embodiment of an election algorithm provided by the present invention;

FIG. 4 is a schematic diagram of an embodiment of a channel, block file and time period provided by the present invention;

FIG. 5 is a schematic diagram illustrating a data storage system according to an embodiment of the present invention;

FIG. 6 is a schematic diagram illustrating a data storage system according to another embodiment of the present invention;

FIG. 7 is a schematic diagram illustrating an embodiment of a data storage system according to the present invention;

FIG. 8 is a flowchart of a data writing method according to an embodiment of the present invention;

FIG. 9 is a flowchart of a data reading method according to an embodiment of the present invention;

FIG. 10 is a flowchart of an embodiment of a data locking/unlocking method according to the present invention;

FIG. 11 is a flow chart of an embodiment of a data elimination method according to the present invention;

FIG. 12 is a schematic diagram of a computer device according to an embodiment of the present invention;

fig. 13 is a schematic structural diagram of an embodiment of a computer readable storage medium according to the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention will be described in further detail with reference to the accompanying drawings.

It should be noted that, in the embodiments of the present invention, all the expressions "first" and "second" are used to distinguish two entities with the same name but different entities or different parameters, and it is noted that the "first" and "second" are only used for convenience of expression, and should not be construed as limiting the embodiments of the present invention, and the following embodiments are not described one by one.

Based on the above object, a first aspect of the embodiments of the present invention proposes an embodiment of a data storage method. As shown in fig. 1, it includes the steps of:

s10, starting a timer of a node in the cluster according to a preset condition;

s20, monitoring whether the timer is overtime;

s30, responding to the overtime of the timer, initiating election by the node based on the overtime of the timer, and voting the node initiating the election based on other nodes in the cluster and the node initiating the election;

and S40, determining whether the node initiating the election can become a main center service or not based on the voting result so as to bear data storage service based on the main center service.

Each server node runs a central service, so that the problems of instantaneous traffic congestion, data consistency and the like of a certain node load during repeated creation and load balancing of channels can be avoided, each server node is started for the first time, the central service on each server node selects a main central service through a main selection algorithm to bear the data storage service function, and the storage service function can comprise the functions of load balancing, node monitoring, fault switching and the like. When the main service or the server node where the main service is located is down, the central service running on other server nodes can reselect one main service to bear all data storage service functions through the main selection algorithm, so that the high availability of data storage is realized.

The method and the device can be applied to the technical field of security monitoring and are used for realizing high availability of video stream storage service, but are not limited to the method and the device.

When the method is applied to the technical field of security monitoring, based on the real-time video storage characteristic of the security monitoring industry, the video service is isolated from the video stream data read-write, and the video service is effectively combined with a storage data layer, so that a high-performance, high-reliability and high-availability video storage method is provided.

Specifically, in step S10, the timer is used to count the time of waiting for the heartbeat notification of the leader, and the preset condition may be the preset starting time of the timer, and the starting time of each node is different, so that only one node at the same time point is ensured to initiate election, and errors are avoided.

In step S20, it is monitored whether the time of the timer waiting for the heartbeat notification of the leader has expired.

In step S30, the timer waits for the timeout of the heartbeat notification of the leader, and the node that the timer times out initiates the election, and votes the node that initiates the election based on other nodes in the cluster and the node itself that initiates the election, and the other nodes can select the vote or reject the vote, and return the voting result to the node that initiates the election.

In step S40, after receiving the voting result, the node initiating the election can determine whether it can serve as the main center.

The principal selection algorithm mentioned in the present invention has been described above in connection with specific embodiments, it being understood that the embodiments described herein are for illustration and explanation of the present invention only and are not intended to limit the present invention.

In step S40, if it is determined that the node initiating election cannot become the service of the main center, the process returns to step S20 to wait for the node initiating election with the timeout of the other timer. If the node initiating the election can be determined to be the main center service, the node initiating the election is set to be the main center service, and video storage service is carried out based on the main center service.

Through the scheme, the normal operation of the cluster is ensured by selecting the main algorithm and one main center service in the server cluster, and the high availability of video storage is realized.

the method further comprises the steps of:

The following optional main functions of the invention are set forth in connection with yet another specific embodiment, it being understood that the embodiment described herein is merely illustrative and explanatory of the invention, and is not intended to limit the invention.

1) Summary of the principal selection algorithm

The main selection algorithm can be divided into three scenarios: the single node center service, the number of the survival nodes is more than half of the total node number, and the number of the survival nodes is less than or equal to half of the total node number.

Different scenarios the processing logic of the selection master algorithm is different. The central service of the single node is the main central service without electing; the processing logic with the number of the movable joints being more than half of the total number of the nodes is that the node with the number of the votes which is received first and is more than half of the total number of the nodes serves as a main center; the processing logic with the number of the movable joints less than or equal to half of the total number of the nodes is that the node with the smallest node ID (Identity document, identity) serves as a main center.

The condition for the election initiated by the election main algorithm may be the first time after the storage system of the node is deployed. The condition that the election main algorithm initiates the election can also be that the main center service does not exist currently or the main center service fails currently, if the main center service does not exist currently, the election operation can not be performed even if the main center service only exists in the current cluster, and the service of the cluster continues to normally run.

2) Implementation scheme of main selection algorithm

The concept related to each node in the main selection algorithm comprises a node ID, a timeout interval (timeout interval), a term (term), a startup election time (election time) and a node state.

The node IDs are identified by the IP address of the server where the central service node is located as unique identifiers, and the starting parameters of each central service node can set the current central service node ID and other central service node ID lists in the cluster.

The time-out time interval comprises a time-out interval for waiting for the notification of the main center service and an election time-out time interval, the election time-out time interval of each center service node is different, and the interval value is a random value within a certain range.

The period is a condition basis for electing the main center service, the period is 0 when all nodes are initially started, 1 is added to the period of a certain node when the node starts a new round of election, and the probability that the node with the larger period becomes the main center service is higher.

The starting election time is also a condition basis for electing the main center service, and under the condition that the tenure period is the same, the earlier the starting election time is, the greater the possibility that the node becomes the main center service is.

The node states include a follower, a candidate and a leader, and the state of the central service node may be one of the three states, and when the central service node is the state of the leader, the central service node becomes the main central service. The switching logic for the three states is shown in fig. 2.

3) Implementation steps of election algorithm

The implementation of the election algorithm is described in conjunction with fig. 3;

each node runs a central service, and when the node is started, the current node ID and other node IDs in the cluster are obtained according to the set starting parameters;

starting a timer waiting for the heartbeat notification of the leader, wherein the time for starting the timer is different for each node, so that only one node initiates election at the same time point;

The node waiting for overtime initiates a round of election, and the node can carry out own tenure +1;

setting the state of the self as a candidate state, concurrently sending voting instructions to other nodes, and casting a vote for the self;

the other nodes will compare the period of the initiated voting node with the period of the self-initiated voting after receiving the voting instruction, if the period of the self-initiated voting is smaller than the period of the self-initiated voting, if the period of the self-initiated voting is larger than the period of the self-initiated voting, if the period of the self-initiated voting is equal to the period of the self-initiated voting, the period of the self-initiated voting is compared with the period of the self-initiated voting, if the period of the self-initiated voting is earlier than the period of the self-initiated voting, the voting is refused later than the period of the self-initiated voting, a result is returned, and if the node fault returns a network connection error;

the node initiating the election receives the voting result and judges whether the number of the votes exceeds half of the total number of the nodes if the number of the stored nodes is more than half of the total number of the nodes, the nodes which wait for overtime return to initiate a round of election if the number of the stored nodes exceeds half of the total number of the nodes and become a leader if the number of the votes exceeds half of the total number of the nodes and the number of the nodes does not exceed the total number of the nodes; if the number of the stored nodes is less than half of the total number of the nodes, judging whether the IP address of the node is the minimum value of the surviving nodes, if the IP address is the minimum value, the node becomes a leader, if the IP address is not the minimum value, returning the node waiting for overtime to initiate a round of election;

A node serving as a leader periodically transmits a heartbeat notification to other nodes;

the other nodes set their own state as follower after receiving the heartbeat notification from the leader.

In this embodiment, the provided central service selection main algorithm supports selection of main central services in three scenes of single central service node, survival central service node number greater than total central service node number, survival central service node number less than total central service node number, and the like, and can ensure that a main central service exists in a cluster to perform normal operation of the cluster.

The data storage method implemented by the embodiment can separate different processes from the video storage service and the data read-write service to bear, the separated central service process is a high-availability cluster architecture, global channel information metadata is generated and managed by one main central service, all service operations except the data read-write operation are uniformly processed, and the separated node service is used for processing the data read-write operation.

In some implementations, assuming the cluster management based on the primary hub service includes: and managing and monitoring the state and resource information of all node services in the cluster based on the main center service.

Specifically, the main center service manages all node services in the cluster, and monitors the state and resource information of the node services. After the node service is started, registering with the main center service, synchronizing the information such as the node address of the node where the node service is located, the total number of block files, the number of idle block files and the like to the main center service, and after the main center service receives the registered and synchronized node information of the node service, setting the state of the node to be on line and updating the node information metadata of the node. In the subsequent operation process, the node service can send keep-alive heartbeat to the main center service at regular time, if the main center service does not receive the keep-alive heartbeat sent by the node service on a certain node in a time interval exceeding the time interval, the state of the node is set to be in an off-line state, and after the node service fault is monitored, the data writing operation of the fault node service can be switched to other normal node services for carrying out.

The node service is specially used for bearing the data read-write service, and separates the video storage service from the data read-write service through the center service and the node service, so that the high-availability, high-performance and high-expansion video storage is realized.

In one embodiment, the cluster performance is linearly related to the sum of the performance of each node in an ideal state. For example, the IO, capacity of a storage cluster of 10 nodes is the sum of the I/O and capacity of 10 nodes in the cluster. However, since load balancing cannot achieve an absolute equilibrium state, the space utilization rate is partially reduced when the number of nodes increases, and the overall performance is difficult to achieve linear growth due to the influence of external environments such as a network. At this time, load balancing and consistency of cluster data can be uniformly performed through the main center service.

The basis for load balancing of the main center service is as follows:

1) And taking the number of idle blocks of the node as a judging basis of the load.

2) And preferentially distributing the SDK channel writing application to node services running on the same server node to write data, and avoiding network transmission of data among the server nodes as much as possible.

3) If one channel occupies one or more block files which are not fully written because of node faults, the block files which are not fully written are preferably fully written when the channel applies for writing the nodes, so that space waste is avoided, and other channels cannot be used after one block file is occupied by a certain channel.

The primary hub service may take any of the three logic described above when performing load balancing.

In some embodiments, the priorities of the three load balancing logics may also be set, preferably, the priorities of the two latter load balancing logics are set to be higher than the priorities of the first load balancing logic, and the load balancing logics with the same level of the two latter priorities are selected based on specific application scenarios.

Specifically, in the present invention, 4 types of metadata are defined, which are node information, channel information, block file information, time period information, etc., respectively, and the storage structure of the metadata is defined as a Key-Value (Key Value pair) pattern.

The node information mainly describes the information related to the size of the storage resources allocated to the node service and the node service state, and is used for carrying out the processing related to the node service state monitoring and the storage resource load balancing.

Channel information, block file information, and time period information are metadata related to a video service, and are related to each other. The minimum unit of the metadata is only up to the time period, no frame data packet or metadata of the frame data exists, and no index file of the frame data exists, and only block files of the frame data are stored. In order to avoid the data reading performance being low caused by the fact that the data is positioned on the data block file in a plurality of jumping mode, a key frame information list contained in the time period is recorded in the time period information. The channel information and the block file information are in one-to-many relationship, and the block file information and the time period information are also in one-to-many relationship. Each channel can use a plurality of block files, each block file only belongs to one channel, and when one block file is occupied by a certain channel, the block file cannot be used by other channels. A block file contains 1 or more time periods, which only exist on a block file and cannot exist across the block file, and the specific relationship diagram is shown in fig. 4.

The specific structure of the node information metadata and the channel information metadata is described below in connection with a specific embodiment, and it should be understood that the embodiments described herein are for illustration and explanation of the present invention only and are not intended to limit the present invention.

Node information: storing a node name (HostName) in the Key; the Value stores the node address, the node state (online and offline), the total number of block files and the number of idle block files.

Channel information: a channel identifier (ChannelName) is stored in the Key, and is transmitted by a user and ensures uniqueness, and is generally a channel ID or a device ID; the Value stores the cycle type, cycle period, occupied block file list, block ID, node address, unwritten block file list, block ID, node address, time zone list, time zone ID, start time, end time, video type identification, node address, time zone status (normal, expired, locked).

The node information and the channel information are global metadata of the storage system and are stored in a cluster of a central metadata base. The unified operation is carried out by the main center service, metadata of node information and channel information are loaded into the memory from the central database when the main center service is started, and the node information and the channel information metadata in the memory are permanently updated into clusters of the central metadata database in a running process.

The following description of the node information metadata management and the channel information metadata management, respectively, is made in connection with two specific embodiments, it being understood that the embodiments described herein are for illustration and explanation of the present invention only and are not intended to limit the present invention thereto.

Example 1: node information metadata management

After the node service is started, registering the node service with a main center service, reporting the total number of block files and the number of idle block files managed by the node service to the main center service, generating or updating node information metadata of the node service by the main center service, keeping alive the node information metadata of the node service in a heartbeat mode with the main center service all the time in the running process of the node service, and reporting the node information metadata of the node service to the main center service in real time when the total number of block files and the number of idle block files managed by the node service change.

The main center service monitors the online state of each node service in the cluster, and when the state of a certain node service changes, the main center service updates the state value in the node information metadata of the node service.

Example 2: channel information metadata management

When a certain channel selects a new block file to write data, the selected block file related information is synchronized to the main center service to update the channel information metadata of the channel, and in the process of data writing, the related information of a time period is synchronized to the main center service to update the channel information metadata of the channel.

And after the block file occupied by a certain channel is eliminated, the related information of the block file is synchronized to the main center service to update the channel information metadata of the channel.

In some embodiments, undertaking the video retrieval based on the primary central service includes: and receiving a channel video search instruction, and inquiring a time period list meeting search conditions from channel information metadata based on the channel video search instruction.

Specifically, when receiving a video command for searching a certain channel, the main center service queries a time slot list in metadata of the channel information according to conditions such as search start time, search end time and video mark, judges according to the information of the time slot list and search conditions, returns the list of time slots meeting the search conditions to the SDK, and the SDK directly communicates with a node service on a server node where the time slots are located according to time slot IDs, node addresses, start time and end time in the time slot list to read data. Because the video retrieval only needs to retrieve to the main center service, and the main center service caches the metadata in the memory, the operation only needs to search the memory, and even if the video retrieval in a long time range can return in seconds.

The SDK (Software Development Kit ) provides the management platform with an API (Application Programming Interface, application program interface) for use, simplifies packaging of management platform use functions, and finally provides the static/dynamic link library for use by the management platform.

In some embodiments, assuming the video lock based on the primary central service comprises: receiving a video locking instruction, inquiring all time periods conforming to a locking time range from channel information metadata based on the video locking instruction, setting the state of the time periods as locking, and sending a command for locking the current time period to node service where the time periods are located.

Specifically, the minimum unit of video locking is a time period, so when locking data in a certain time range, all time periods covered in the time range can be locked, the locking of the time period can directly influence the elimination of a block file, and if only one time period is in a locking state on each block file, the block file cannot be eliminated unless the block file can be normally eliminated after the time period is unlocked.

The video locking is processed by a main center service, when the main center service receives a video locking instruction and a video of a certain channel time range is to be locked, channel information metadata of the channel is searched according to the locked time range, states of all time periods conforming to the time range are set to be locked, an instruction for locking the current time period is sent to a node service where the time period is located, and the node service sets the state of the time period to be locked after receiving the locking instruction.

The video unlocking instruction is also processed by the main center service, when the main center service receives the video unlocking a certain channel time range, the channel information metadata of the channel is searched according to the time range, the locking state of all time periods meeting the time range is cleared, the instruction for unlocking the current time period is sent to the node service where the time period is located, and the node service receives the unlocking instruction to set the state of the time period to be normal.

The video search is locked, and the same flow as the video search is added with the judgment of whether the time slot meeting the condition is locked or not in the search condition.

Specifically, if the video data is not locked, the video data cannot exist permanently, and has a certain storage period, the storage period of the video data can be set for each channel, the video data can be eliminated after exceeding the storage period, and the minimum unit of eliminating the video data is also a time period, namely, all the video data in the current time period exceeds the storage period and can be eliminated. The video elimination of the invention is divided into time period elimination and block file elimination, the main center service is responsible for the elimination of the time period, the main center service can traverse channel information metadata periodically, detect and judge a time period list of each channel, if the ending time of a certain time period exceeds a storage period and the state of the time period is not locked, the state of the time period is set as eliminated, and an instruction for setting the state of the time period as eliminated is sent to the node service where the time period is located. Block file elimination is based on node service implementation.

The embodiment realizes a high-availability, high-performance and easily-extensible data storage method, separates video storage service from data read-write to bear different service processes, uses a main selection algorithm to select main center service to perform unified channel management, channel information metadata management, cluster management, load balancing, video searching, video locking/unlocking, video data elimination and other functions, and can be applied to security monitoring service use scenes. Compared with the prior art, the method has the following advantages:

1. the performance, video searching only needs the main center service to search the time period list cached in the memory, the searching in a long time range can also achieve second-level return, the data reading and writing is directly interacted with the node service, no transit and bottleneck exist, and network interaction is not needed between the video stream and the storage device under most conditions.

2. Availability, because it is a storage system itself for each node in the system cluster, supporting all the functional features of the system, and capable of handling all traffic, dynamic failover is provided.

3. And the expandability is realized, each node can process all services, and the load balancing considers two dimensions of traffic and storage space, so that the node adding and deleting operation can be performed under the condition that the services are not interrupted.

Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 5, an embodiment of the present invention further provides a data storage system, including: a central service (in the present invention, the central service may also be referred to as a central service module) 110, said central service 110 being configured to perform the steps of:

monitoring whether the timer has timed out;

In a specific embodiment, the data storage system of the invention can be used for storing real-time video in the security monitoring field, and provides a high-availability implementation scheme for solving the problems in usability, expansibility, compatibility and the like of the prior art.

As shown in fig. 6, the data storage system includes 5 service modules, which are respectively an SDK, a center service, a center metadata base, a node service, and a local node metadata base. Each storage node runs the service modules, and for each single storage node server, the storage node server is a relatively independent storage system, and supports all functional characteristics of the system and can process any operation of the SDK.

The SDK provides an API for the management platform to use, simplifies and encapsulates the use functions of the management platform, and finally provides a static/dynamic link library for the management platform to use.

The center service has the main functions of high availability of service, cluster management, load balancing, channel management, service processing and the like. The system is used for bearing all node services in the management system, uniformly distributing storage resources provided by the storage data layer, supporting a plurality of management platforms to use the system, avoiding the management platform from repeatedly creating channels in the system, only processing instruction operation and not processing data read-write operation. The central service cluster selects a main service to complete the born function through an election algorithm, and the main service generates and manages global metadata of node information and channel information in the system, so that the data consistency of the global metadata is ensured.

The central metadata base is global metadata managed by the central service and is stored by a database cluster, only one main central service is used for operating the database, and the database cluster ensures the data consistency in the database and the high availability of the database cluster.

The node service is used for synchronizing the current node information to the center service, managing time period, managing block files, reading and writing data and the like. The management platform directly reads and writes data to the node service, and the data reading and writing performance bottleneck can not occur due to no transit service. The node services on the node servers in the system are mutually independent, and have no master-slave division, each node service only self-generates and self-distributes storage resources allocated to the node server and only manages block file information and time slot information metadata of the current node, and because the block file information and the time slot information metadata are frequently updated during data writing, in order to avoid updating metadata in a database of all node services in the system, each node service stores the block file information and the time slot information metadata of the current node into a local node metadata database running on the current node server.

The local node metadata base is a single point database for storing block file information and time period information metadata generated and managed by the node service. The local node metadata databases running on all node servers in the system are mutually independent, do not form a cluster, and only perform communication interaction with the node service running on the current node server.

The data storage system provides a unified interface docking management platform upwards, the function of streaming from the front-end equipment is also encapsulated in the SDK, the function is called by a service process of the management platform, different storage data layers are adapted downwards through different data system engines, and a specific logic diagram is shown in fig. 7.

In some embodiments, the functionality of the data storage system is described in connection with FIGS. 6, 8, 9, 10, and 11. It should be understood that the embodiments described herein are only used for illustrating and explaining the present invention, and are not intended to limit the present invention.

1) Description of the data write procedure with reference to FIGS. 6 and 8

In an actual application scene, channels are in one-to-one correspondence with the image pickup devices, when a certain image pickup device writes data into the data storage system for the first time, when a certain channel is created to the main center service through the SDK, the main center service generates channel information metadata of the channel, if the channel information metadata of the channel already exists, the main center service returns error information existing in the SDK channel, and the fact that the channels in the cluster cannot be repeatedly created is ensured.

The specific data writing process is as follows:

under the condition that data writing is started or the fact that the current block is fully written is judged, the management platform applies for writing nodes to the main center service through the SDK;

the management platform directly performs continuous data stream writing to node service interaction of the writing nodes distributed to the main center service through the SDK;

after receiving the data, the node service judges whether a new block file needs to be selected for writing;

if a new block file needs to be selected, selecting the block file from the free block file list, and generating a new time period;

if the new block file does not need to be selected, selecting the block file currently occupied by the channel, and judging whether a new time period needs to be generated or not;

generating new time period metadata if the generation is needed, and updating the current time period information if the generation is not needed;

writing the data into a physical storage unit mapped by the block file, and updating the block file information and time period information metadata in a database after successful writing;

and the loop execution management platform directly performs continuous data stream writing and subsequent steps to node service interaction of the writing nodes distributed to the main center service through the SDK.

2) The data reading flow will be described with reference to fig. 6 and 9

The management platform enumerates channels to the main center service through the SDK, and if the management platform definitely determines that the channel data is to be read, the operation can be omitted;

the management platform sends a video searching command to the main center service through the SDK so as to perform video searching;

the main center service queries all time period lists of the channel according to the retrieval conditions of the retrieval command and returns all time period lists meeting the conditions;

the management platform selects a certain time period from the time period list through the SDK and directly reads data from the node service where the certain time period is located;

after receiving the data reading instruction, the node service loads the corresponding time period information metadata from the database into the memory;

positioning the offset of the nearest key frame on the block file according to the read starting time;

positioning the data to a storage position of the data according to the offset to continuously read the data;

the data reading and the following operations are cyclically performed until the reading of all the data is completed.

3) The video locking/unlocking process will be described with reference to fig. 6 and 10

The management platform performs video locking or unlocking operation through the SDK interface, namely, sends a video locking or unlocking command to the main center service through the SDK interface;

The main center service finds out all time periods covered by the time range according to the locking time range/unlocking time range;

the main center service sends a locking/unlocking instruction to the node service where the time period meeting the condition is located;

the node service updates the metadata of the time period after receiving the locking/unlocking instruction;

after receiving the successful execution return result of the node service, the main center service updates the channel information metadata;

and the main center service is circularly executed to send a locking/unlocking instruction and the subsequent operation to the node service where the time period meeting the condition is located until the locking/unlocking is completed at all times, and then a successful operation result is returned.

4) Description will be given of the data elimination flow with reference to fig. 6 and 11

a) Master center service period detection

The main center service detects channel information metadata at regular time, and detects and judges the time slot list of each channel

Judging whether the time period is locked or not, if so, returning to the step of detecting channel information metadata at regular time by the main center service, and if not, executing the next step;

if the ending time of the time period exceeds the storage period, the main center service sends a elimination instruction of the time period to the node service where the time period is located;

The node serves to update the state of the time period and updates metadata of the time period to the database;

the main center service clears the time slot information from the channel information metadata, updates the metadata of the channel to the database, and continues to circularly execute the operation.

b) Node service block file detection

The node service detects all non-idle block files at regular time, judges whether the block files are being read or written, if so, detects the next block file, if not, executes the next step;

detecting the states of all time periods of the block files, detecting whether the states of the time periods are locked, if so, executing the step of the node service timing detection of all non-idle block files, and if not, executing the next step;

detecting whether the state of all time periods is the elimination state, if not, executing the step of periodically detecting all non-idle block files by the node service, and if so, executing the next step;

recovering the block file as an idle block, and deleting metadata of all time periods on the block file;

updating block file information and time period information metadata to a database, and serving synchronization information to a center;

the above operation steps are cyclically performed.

In this embodiment, the process of data elimination is divided into two parts, namely, time period elimination and block file elimination and recovery, where the time period elimination is processed by the main center service and the block file elimination and recovery is processed by the node service, and this scheme is completed under the condition that the resource occupation is negligible.

The embodiment realizes a high-availability, high-performance and easily-expandable data storage system, separates video storage service from data read-write to bear different service processes, uses a main selection algorithm to select main center service to perform unified channel management, channel information metadata management, cluster management, load balancing, video searching, video locking/unlocking, video data elimination and other functions, and can be applied to security monitoring service use scenes. Compared with the prior art, the method has the following advantages:

According to another aspect of the present invention, as shown in fig. 12, according to the same inventive concept, an embodiment of the present invention further provides a computer device 30, in which the computer device 30 includes a processor 310 and a memory 320, the memory 320 storing a computer program 321 executable on the processor, and the processor 310 executing the steps of the method as above.

The memory is used as a non-volatile computer readable storage medium, and can be used for storing non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions/modules corresponding to the data storage method in the embodiments of the present application. The processor executes various functional applications of the system and data processing, i.e., implements the data storage methods of the method embodiments described above, by running non-volatile software programs, instructions, and modules stored in the memory.

The memory may include a memory program area and a memory data area, wherein the memory program area may store an operating system, at least one application program required for a function; the storage data area may store data created according to the use of the system, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory optionally includes memory remotely located relative to the processor, the remote memory being connectable to the local module through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 13, an embodiment of the present invention further provides a computer-readable storage medium 40, the computer-readable storage medium 40 storing a computer program 410 that when executed by a processor performs the above method.

Finally, it should be noted that, as will be appreciated by those skilled in the art, all or part of the procedures in implementing the methods of the embodiments described above may be implemented by a computer program for instructing relevant hardware, and the program may be stored in a computer readable storage medium, and the program may include the procedures of the embodiments of the methods described above when executed. The storage medium of the program may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (RAM), or the like. The computer program embodiments described above may achieve the same or similar effects as any of the method embodiments described above.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. The foregoing embodiment of the present invention has been disclosed with reference to the number of embodiments for the purpose of description only, and does not represent the advantages or disadvantages of the embodiments. Furthermore, although elements of the disclosed embodiments may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.

It should be understood that as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

Those of ordinary skill in the art will appreciate that: the above discussion of any embodiment is merely exemplary and is not intended to imply that the scope of the disclosure of embodiments of the invention, including the claims, is limited to such examples; combinations of features of the above embodiments or in different embodiments are also possible within the idea of an embodiment of the invention, and many other variations of the different aspects of the embodiments of the invention as described above exist, which are not provided in detail for the sake of brevity. Therefore, any omission, modification, equivalent replacement, improvement, etc. of the embodiments should be included in the protection scope of the embodiments of the present invention.

Claims

1. A method of data storage, comprising:

starting a timer of a node in the cluster according to a preset condition;

monitoring whether the timer has timed out;

2. The method of claim 1, wherein voting the election-initiating node based on other nodes in the cluster and the election-initiating node itself comprises:

3. The method of claim 2, wherein the other nodes, upon receiving the voting instructions, further comprise:

voting the node initiating the election based on the period of the node initiating the election and the period of the other nodes, and returning the voting result to the node initiating the election; or (b)

4. A method according to claim 3, wherein voting the election-initiating node based on its tenninal period and the tenninal period of the other nodes themselves comprises:

5. The method of claim 1, wherein determining whether the election-initiating node can become a primary hub service based on the voting results comprises:

6. The method of claim 5, wherein determining whether the election-initiating node can become a primary hub service based on the number of votes obtained by the election-initiating node and a total number of nodes comprises:

7. The method of claim 5, wherein determining whether the election-initiating node can become a primary hub service based on the node ID comprises:

8. The method of claim 1, wherein the node comprises a node state, the node state being any one of a follower, a candidate, or a leader.

9. The method of claim 8, wherein the node state of the election initiating node is a candidate, the node state of the node where the primary hub service resides is a leader, and the node states of the remaining nodes are followers.

10. The method of claim 1, wherein the timer is used to count the time waiting to receive a leader heartbeat notification;

the method further comprises the steps of:

11. The method of claim 1, wherein the data storage services include any one or more of cluster management, load balancing, global metadata management, video retrieval, video lock-out, and time period elimination detection.

12. The method of claim 11, wherein assuming the cluster management based on the primary hub service comprises: managing and monitoring the state and resource information of all node services in the cluster based on the main center service;

Global metadata management including node information metadata and channel information metadata, assuming the global metadata management based on the primary hub service includes: and in response to the starting of the main center service, loading the node information metadata and the channel information metadata from a center metadata base into a memory of a node where the main center service is located, and periodically and permanently updating the node information metadata and the channel information metadata in the memory into the center metadata base.

13. The method of claim 11, wherein undertaking the video retrieval based on the primary central service comprises: receiving a channel video search instruction, and inquiring a time period list meeting search conditions from channel information metadata based on the channel video search instruction;

bearing the video lock based on the primary central service includes: receiving a video locking instruction, inquiring all time periods conforming to a locking time range from channel information metadata based on the video locking instruction, setting the state of the time periods as locking, and sending a command for locking the current time period to node service where the time periods are located;

Assuming the period of time elimination detection based on the primary central service includes: and traversing the channel information metadata periodically based on the main center service to detect and judge a time period list of each channel, and determining whether to eliminate the time period based on a detection judging result and the state of the time period.

14. A data storage system, comprising: a central service configured to perform the steps of:

monitoring whether the timer has timed out;

15. A computer device, comprising:

at least one processor; and

a memory storing a computer program executable on the processor, wherein the processor performs the steps of the method of any one of claims 1 to 13 when the program is executed.

16. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor performs the steps of the method according to any one of claims 1 to 13.