CN115904805A

CN115904805A - Data backup method and device, electronic equipment and storage medium

Info

Publication number: CN115904805A
Application number: CN202211404484.7A
Authority: CN
Inventors: 江文龙; 牛小龙; 罗心; 周明伟
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2022-11-10
Filing date: 2022-11-10
Publication date: 2023-04-04

Abstract

The present application relates to the field of data storage technologies, and in particular, to a data backup method and apparatus, an electronic device, and a storage medium, so as to improve data backup efficiency. The method comprises the following steps: acquiring storage node information of data to be stored; storing data to be stored into corresponding storage nodes according to the storage node information, wherein the length of the data to be stored each time is not more than a preset length, and the following processes are executed each time the storage of set times is completed: judging whether the length of the data to be backed up is not less than a preset threshold value or not; if the length of the data to be backed up is not smaller than the preset threshold value, a data backup instruction is sent to the associated cluster through the management device corresponding to the storage cluster, so that the associated cluster backs up the data to be backed up. According to the data backup method and device, when the length of the data to be backed up is not smaller than the preset threshold value, the associated cluster is informed to back up the data, so that real-time backup in the data storage process can be achieved, and the data backup efficiency is improved.

Description

Data backup method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a data backup method and apparatus, an electronic device, and a storage medium.

Background

In recent years, with the rapid development of information technology, huge-scale data is generated in many fields such as scientific computing and business computing, and the data volume is still increasing at a relatively fast speed. Meanwhile, data backup needs to be carried out on the data in order to protect the data in the cloud storage system, so that the data can be found back in time when the data is damaged or lost.

In the related art, file data is stored in a cloud storage system, and then the stored file data is detected at regular time, so that the file data which is not backed up is backed up. However, when data backup is performed in the above manner, all file data need to be detected at regular time to determine the file data that is not backed up, the amount of data that needs to be processed is huge, the data backup efficiency is low, and real-time backup of the file data cannot be performed. Therefore, how to improve the data backup efficiency is a problem to be solved urgently.

Disclosure of Invention

The embodiment of the application provides a data backup method and device, electronic equipment and a storage medium, which are used for improving data backup efficiency.

The first data backup method provided by the embodiment of the application comprises the following steps:

acquiring storage node information of the data to be stored, wherein the storage node information comprises at least one storage node corresponding to the data to be stored, and the storage node belongs to a storage cluster;

storing the data to be stored into corresponding storage nodes according to the storage node information, wherein the length of the data to be stored each time is not more than a preset length;

the following process is executed every time the storage of the set number of times is completed:

judging whether the length of data to be backed up is not less than a preset threshold value, wherein the data to be backed up comprises data which is stored in a storage node and is not backed up in the data to be stored;

if the length of the data to be backed up is not less than a preset threshold value, sending a data backup instruction to an associated cluster through a management device corresponding to the storage cluster so that the associated cluster backs up the data to be backed up, wherein the associated cluster and the storage cluster belong to the same cloud storage system.

Based on the above manner, when the data to be stored is stored in the corresponding storage node, each time the storage of the set number of times is completed, whether the length of the data to be backed up is not less than the preset threshold value is judged, if yes, a data backup instruction is sent to the associated cluster through the management device, so that the associated cluster backs up the data to be backed up, and therefore the data can be backed up in real time in the data storage process, the data backup efficiency is improved, and the data security is further improved.

In an optional implementation manner, before the obtaining storage node information of data to be stored, the method further includes:

dividing data to be stored into a plurality of sub-storage data, and generating verification data based on each sub-storage data; the check data is used for representing the incidence relation among the sub-storage data;

and taking each sub-storage data and the check data as the data to be stored.

Based on the mode, when any one of the sub-storage data is damaged or lost, the sub-storage data can be obtained according to the remaining sub-storage data and the verification data, and the safety of the data to be stored can be effectively improved.

In an alternative embodiment, the method further comprises:

if the length of the data to be backed up is smaller than the preset threshold value, judging whether data which are not stored in the corresponding storage nodes exist in the data to be stored;

if the data which are not stored in the corresponding storage nodes do not exist in the data to be stored, the management equipment informs the associated cluster to back up the data to be backed up.

In an optional implementation manner, the sending, by the management device corresponding to the storage cluster, the data backup instruction to the associated cluster includes:

sending a data backup request to the management device, so that the management device generates a data backup instruction based on the data backup request, and sends the data backup instruction to the associated cluster, wherein the data backup instruction includes the data length of the data to be backed up and the position of the data to be stored.

Based on the mode, the data backup instruction is sent to the associated cluster so that the associated cluster performs data backup, real-time backup in the process of writing data can be achieved, and data backup efficiency is improved.

The second data backup method provided by the embodiment of the application comprises the following steps:

receiving a data backup instruction; the data backup instruction is sent by management equipment corresponding to a storage cluster when a client determines that the length of data to be backed up is not less than a preset threshold value after finishing storage for a set number of times, the data to be backed up comprises data which is stored in a storage node and is not backed up in the data to be stored, the storage node belongs to the storage cluster, and the associated cluster and the storage cluster belong to the same cloud storage system;

and backing up the data to be backed up based on the data backup instruction.

In an optional implementation manner, the backing up the data to be backed up based on the data backup instruction includes:

analyzing the data backup instruction to obtain the data length of the data to be backed up and the position of the data to be stored;

determining at least one storage node corresponding to the data to be backed up based on the data length, the position and the storage node information of the data to be stored;

and reading the data to be backed up from each storage node, and storing the data to be backed up to the local.

The first data backup device provided in the embodiment of the present application includes:

the acquisition unit is used for acquiring storage node information of the data to be stored, wherein the storage node information comprises at least one storage node corresponding to the data to be stored, and the storage node belongs to a storage cluster;

the storage unit is used for storing the data to be stored into the corresponding storage node according to the storage node information, wherein the length of the data to be stored each time is not more than a preset length;

a sending unit, configured to execute the following processes each time the storage of the set number of times is completed:

if the length of the data to be backed up is not smaller than a preset threshold value, sending a data backup instruction to an associated cluster through a management device corresponding to the storage cluster so that the associated cluster backs up the data to be backed up, wherein the associated cluster and the storage cluster belong to the same cloud storage system.

In an optional implementation manner, the apparatus further includes a dividing unit, configured to:

dividing data to be stored into a plurality of sub-storage data, and generating verification data based on each sub-storage data; the verification data is used for representing the incidence relation among the sub-storage data;

and taking each piece of sub-storage data and the check data as the data to be stored.

In an optional implementation manner, the apparatus further includes a determining unit, configured to:

if the data which are not stored in the corresponding storage nodes do not exist in the data to be stored, notifying an associated cluster through a management device to back up the data to be backed up.

In an optional implementation manner, the sending unit is specifically configured to:

sending a data backup request to the management device, so that the management device generates a data backup instruction based on the data backup request, and sends the data backup instruction to the associated cluster, wherein the data backup instruction includes the data length of the data to be backed up and the position of the data to be stored. Optionally, the first processing unit is specifically configured to:

the second data backup device provided in the embodiment of the present application includes:

the receiving unit is used for receiving a data backup instruction; the data backup instruction is sent by management equipment corresponding to a storage cluster when a client determines that the length of data to be backed up is not less than a preset threshold value after finishing storage for a set number of times, the data to be backed up comprises data which is stored in a storage node and is not backed up in the data to be stored, the storage node belongs to the storage cluster, and the associated cluster and the storage cluster belong to the same cloud storage system;

and the backup unit is used for backing up the data to be backed up based on the data backup instruction.

In an optional implementation manner, the backup unit is specifically configured to:

An electronic device provided in an embodiment of the present application includes a processor and a memory, where the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of any one of the data backup methods.

An embodiment of the present application provides a computer-readable storage medium, which includes a computer program, and when the computer program runs on an electronic device, the computer program is configured to enable the electronic device to execute any of the steps of the data backup method described above.

An embodiment of the present application provides a computer program product, which includes a computer program, the computer program being stored in a computer-readable storage medium; when the processor of the electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any of the data backup methods described above.

Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

fig. 1 is an alternative schematic diagram of an application scenario in an embodiment of the present application;

FIG. 2 is a flowchart illustrating an implementation of an image processing method according to an embodiment of the present disclosure;

fig. 3 is a schematic diagram of a storage node information acquisition method in an embodiment of the present application;

FIG. 4 is a diagram illustrating a data storage method according to an embodiment of the present application;

fig. 5 is a schematic flowchart of an OP subscription method in an embodiment of the present application;

fig. 6 is a schematic structural diagram of a cloud storage system in an embodiment of the present application;

fig. 7 is a schematic flowchart of a data backup method in an embodiment of the present application;

FIG. 8 is a flowchart illustrating another implementation of a data backup method in an embodiment of the present application;

FIG. 9 is a schematic flowchart of another data backup method in the embodiment of the present application;

fig. 10 is a schematic structural diagram of a data backup device in an embodiment of the present application;

fig. 11 is a schematic structural diagram of another data backup device in the embodiment of the present application;

fig. 12 is a schematic diagram of a hardware component of an electronic device to which an embodiment of the present application is applied;

fig. 13 is a schematic diagram of a hardware component structure of another electronic device to which the embodiment of the present application is applied.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments, but not all embodiments, of the technical solutions of the present application. All other embodiments obtained by a person skilled in the art without any inventive step based on the embodiments described in the present application are within the scope of the protection of the present application.

Some concepts related to the embodiments of the present application are described below.

The cloud storage system comprises: the system is a functional system which combines a network technology, a distributed file system, a metadata management service for managing file metadata information, a directory server service for managing file directory information, storage devices of different storage types and application software into a distributed cluster through a network, and provides data storage and service data access. In the embodiment of the present application, data backup in a video cloud storage system is mainly taken as an example for description.

Metadata: the management device is a device for maintaining metadata information of Data written in a storage node by a client, and may also be referred to as a metadata management Module (MDS).

Operation (Operation, OP): in the video cloud storage system, a client performs operations such as file creation, file opening and file closing to the video cloud storage system, and records an operation log in an MDS, where the operation log records a current operation serial number and operation detailed information, so that such operations are regarded as an OP.

Associating the clusters: the number of the associated clusters is one or more, and a plurality of the associated clusters can simultaneously back up the data to be backed up, so that the safety of the data is further improved.

Operation Log file (Edit Log): all operation records of the client record an operation log at the MDS, the operation records are firstly stored in the memory, and when the cluster subscribing the OP operation (namely the associated cluster) acquires and analyzes the OP successfully, the MDS serializes the OP records in the memory to an operation log file.

Log playback: returning to a certain time point according to the operation log file of the user, and analyzing all the operations after the time point one by one.

The preferred embodiments of the present application will be described below with reference to the accompanying drawings of the specification, it should be understood that the preferred embodiments described herein are merely for illustrating and explaining the present application, and are not intended to limit the present application, and that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.

Fig. 1 is a schematic view of an application scenario in the embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.

In the embodiment of the present application, the terminal device 110 includes, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a desktop computer, an e-book reader, an intelligent voice interaction device, an intelligent household appliance, a vehicle-mounted terminal, and other devices; the terminal device 110 may be installed with a client related to data backup, where the client may be software (e.g., a browser, camera software, etc.), or a web page, an applet, etc., and the server 120 is a background server corresponding to the software, the web page, the applet, etc., or a server specially used for data backup, which is not limited in this application. The server 120 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.

It should be noted that, the data backup method in the embodiment of the present application may be executed by an electronic device, and the electronic device may be the server 120 or the terminal device 110, that is, the method may be executed by the server 120 or the terminal device 110 alone, or may be executed by both the server 120 and the terminal device 110. For example, when the server 120 and the terminal device 110 are executed together, the terminal device 110 obtains storage node information of data to be stored, stores the data to be stored in a corresponding storage node according to the storage node information, and executes the following processes each time the storage of the set number of times is completed: judging whether the length of the data to be backed up is not less than a preset threshold, if the length of the data to be backed up is not less than the preset threshold, sending a data backup instruction to the associated cluster through the management device corresponding to the storage cluster, receiving the data backup instruction by the server 120, and backing up the data to be backed up based on the data backup instruction.

The terminal device 110 and the server 120 may be included in the cloud storage system, or may be independent of the cloud storage system, and are used as devices specially used for data backup, which is not specifically limited herein.

In an alternative embodiment, terminal device 110 and server 120 may communicate via a communication network.

In an alternative embodiment, the communication network is a wired network or a wireless network.

It should be noted that, the illustration shown in fig. 1 is only an example, and the number of the terminal devices and the servers is not limited in practice, and is not specifically limited in the embodiment of the present application.

In the embodiment of the application, when the number of the servers is multiple, the multiple servers can be combined into a block chain, and the servers are nodes on the block chain; the data backup method disclosed in the embodiment of the present application, wherein the data to be stored may be stored in a block chain.

In addition, the embodiment of the application can be applied to various scenes, including not only a data backup scene, but also scenes such as but not limited to cloud technology, artificial intelligence, intelligent transportation, driving assistance and the like.

The data backup method provided by the exemplary embodiment of the present application is described below with reference to the accompanying drawings in conjunction with the application scenarios described above, and it should be noted that the application scenarios described above are only shown for the convenience of understanding the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.

Referring to fig. 2, an implementation flowchart of a data backup method provided in the embodiment of the present application is shown, and a specific implementation flow of the method includes the following steps S21 to S22:

s21: acquiring storage node information of data to be stored;

before data storage, a client needs to acquire a storage location of the data to be stored, storage node information is used for representing the storage location of the data to be stored, and includes at least one storage node corresponding to the data to be stored, the storage node belongs to a storage cluster, the storage cluster includes a plurality of storage nodes, and for example, the storage node information may be: and storing the data to be stored into the data node 1, the data node 2 and the data node 3.

The method includes the steps that after a Data Node (DN) is on line, the Data Node (DN) is registered to an MDS, the MDS allocates an Identity (ID) to each DN, when a client side obtains storage Node information, as shown in figure 3, the client side applies for an object (object) to the MDS, the MDS returns the object applied by the client side to the client side, wherein the object comprises a plurality of blocks (blocks), each block records the storage Node ID and the block ID in which Data to be stored are to be written, and the client side writes the Data to be stored into the corresponding storage Node according to the storage Node ID and the block ID.

Before step S21, in order to improve the security of the data to be stored, the data to be stored may be divided so that the data to be stored can be stored in different storage nodes, in an optional implementation, the data to be stored is divided into a plurality of sub-storage data, and the check data is generated based on each sub-storage data; and taking each piece of sub-storage data and the verification data as data to be stored.

When any sub-storage data is damaged or lost, the sub-storage data can be obtained according to the remaining sub-storage data and the check data, so that the safety of the data to be stored can be effectively improved. When data to be stored is divided, N + M, that is, N data blocks (sub-storage data) and M parity blocks (parity data), are divided according to a set Erasure Code (EC), which is mainly described in the embodiment of the present application by taking N as 4 and M as 1 as examples, for example, a 4M file object (data to be stored may be a file) is divided into 4 data blocks and one parity block according to 4+1, the size of each block is 1M, and as shown in fig. 4, each data block (object fragmentation) parity block may be written on different storage nodes respectively.

S22: storing the data to be stored into the corresponding storage nodes according to the storage node information, and executing the following processes every time the storage of the set times is completed: judging whether the length of the data to be backed up is not less than a preset threshold value or not; and if the length of the data to be backed up is not less than the preset threshold value, sending a data backup instruction to the associated cluster through the management equipment corresponding to the storage cluster so that the associated cluster backs up the data to be backed up.

The data length of each storage is not larger than the preset length, the data to be backed up comprises data which is stored in the storage nodes in the data to be stored and is not backed up, and the associated cluster and the storage cluster belong to the same cloud storage system. Taking the preset length of 1280K and the preset threshold of 10M as an example, when the length of the data stored each time is 1280K, after the 8 th storage is completed, the length of the data to be backed up is 10M and is equal to the preset threshold, and a data backup instruction is sent to the associated cluster through the management device corresponding to the storage cluster. It should be noted that the length of the data stored each time is not fixed, and is not greater than the preset length. The data backup instruction may be an OP, and the management device generates an update file length OP and pushes the update file length OP to the associated cluster for data backup.

Specifically, when writing data to be stored, a client initializes a write cache, the default write cache is 8 (N + M) × piece, when the data to be stored is a large file, the size of a block (piece) is 32K, the default write cache is 1280K according to a 4+1ec rule, the write cache is divided into a plurality of stripes, the size of each stripe is (N + M) × piece size, that is, 160K, after the data to be stored is written into the cache, the data of a full stripe in the cache is asynchronously written into a storage node of the cloud storage system, the data length (preset length) written into the storage node each time is not greater than 1280K, and meanwhile, whether the data length which is currently written into the cloud storage system and is not backed up is not less than a preset threshold or not is judged, and if not less than the preset threshold, a data backup instruction is sent to an associated cluster through a Management Device (MDS) corresponding to the storage cluster.

It should be noted that the granularity of the preset threshold is adjustable, the default size is 10M, the dynamic loading script is executed by modifying the Update Op Len configuration item in the MDS configuration file, the client program is triggered to dynamically load the configuration item, the size of the preset threshold can be changed, and the file length is updated in stages to adjust the data backup speed.

In the embodiment of the application, a plurality of update file lengths OP are added between the file creating OP and the file closing OP, the granularity of the update file lengths OP is adjustable and is dynamically loaded, the service is not sensed, adjustable parameters are set in an MDS configuration file, and the data backup rate can be adjusted according to an actual service scene.

Based on the mode, in the field of video cloud storage, aiming at the characteristic that real-time audio and video files are large, from the structural angle of the files, a threshold value of the write-in data length is set, the length OP operation of the update files is established, the file length is updated in a staged mode, real-time backup is carried out on the real-time written audio and video files, the purpose of writing while backup is achieved, and the real-time backup efficiency is improved.

In an optional implementation manner, the sending, by the management device corresponding to the storage cluster, the data backup instruction to the associated cluster in step S22 may be implemented as:

and sending the data backup request to the management equipment so that the management equipment generates a data backup instruction based on the data backup request, and sending the data backup instruction to the associated cluster, wherein the data backup instruction comprises the data length of the data to be backed up and the position of the data to be backed up in the data to be stored.

Specifically, the client sends a data backup request (update file length OP request) to the management device, the storage cluster may provide a subscription service, the association cluster subscription service sends a subscription OP request to the storage cluster subscription service, where the subscription OP request is used for subscribing to the update file length OP request, and after receiving the subscription OP request, the storage cluster stores the subscription OP request in the memory and writes the subscription OP request into the database, so that the management device queries which clusters subscribe to the update file length OP request. When the storage device receives an update file length OP request, field information such as a file ID (identity), a bucket ID (bucket ID), an offset of the data to be backed up in the data to be stored, a data length of the data to be backed up and the like of the data to be backed up is analyzed from the update file length OP request, the management device updates metadata information of the data to be stored through the field information, creates an update file length OP and stores the update file length OP in a memory of a log management module, the log management module finds cluster information subscribing the update file length OP from the subscription service memory and sends the update file length OP to a cluster (namely an associated cluster) subscribing the update file length OP.

The update file length OP is used to notify the associated cluster to perform data backup, and may be called a data backup instruction, and a format of the update file length OP may be: op Type (operation Type): op Update File Len; bucket ID:2022397707666073560; file ID (file ID): 4503599627370498; offset (offset): 1265465564; len (data length): 4194304; EC (redundancy rule): 51. it should be noted that the update file length OP is only an example, and actually, the update file length OP may also be in other formats, which is not limited herein.

Based on the mode, the length OP of the updated file is newly added and serialized into the EditLog, the OP records the storage position information and the real-time file length of the current file, the background thread sends the unprocessed updated OP to the associated cluster so as to inform the associated cluster to analyze the OP read file, after the associated cluster obtains the length OP of the updated file, the information such as the file ID, the current length of the file, the block ID of the file, the ID of the storage node and the like in the OP is analyzed, and the file is read from the storage node and written into the associated cluster according to the information.

Referring to fig. 5, it is a schematic flowchart of an OP subscription method in an embodiment of the present application, including the following steps:

s501, the subscription service of the associated cluster sends a subscription OP request to the subscription service of the storage cluster;

s502, the subscription service of the storage cluster analyzes the subscription OP request, stores the subscription OP request in the content and writes the subscription OP request into the database.

In an optional implementation manner, if the length of the data to be backed up is smaller than a preset threshold, whether data which is not stored in a corresponding storage node exists in the data to be stored is judged; and if the data which are not stored in the corresponding storage nodes do not exist in the data to be stored, the management equipment informs the associated cluster to back up the data to be backed up.

Specifically, in the process of writing the data to be stored into the storage node, if the length of the data to be stored is smaller than the preset threshold and there is no data which is not stored into the corresponding storage node, it indicates that the data to be stored has been written into the storage node, the client closes the file after the data is written, updates the file length when the file is closed, sends the closed file OP to the associated cluster, and notifies the associated cluster to back up the data to be backed up.

In the embodiment of the application, when the data to be stored is stored in the corresponding storage node, each time the storage of the set number of times is completed, whether the length of the data to be backed up is not less than a preset threshold value is judged, if yes, a data backup instruction is sent to the associated cluster through the management device, so that the associated cluster backs up the data to be backed up, the data can be backed up in real time in the data storage process, the data backup efficiency is improved, and the data safety is further improved.

Referring to fig. 6, which is a schematic structural diagram of a cloud storage system in an embodiment of the present application, the cloud storage system includes a client, a cluster a (storage cluster), and a cluster B (association cluster), where the cluster a includes:

a metadata management module: the system comprises a data storage node, an MDS (data management system) and an operation log file, wherein the data storage node is used for maintaining metadata information of data (namely data to be stored) written into the storage node by a client, and when a cluster (namely an associated cluster) subscribing the OP operation acquires and analyzes an OP successfully, the MDS serializes OP records in an internal memory to the operation log file for logging;

a storage node: the client side is used for storing the data to be stored, and writing the data to be stored into a magnetic disk of the storage node;

and (3) subscribing for services: receiving a subscription OP request sent by a cluster B, and pushing an update file length OP to the cluster B;

a log management module: recording key operations of a client, such as file OP creation, file length OP updating, file OP login closing and the like.

The cluster B includes:

and (3) subscribing for services: sending a subscription OP request to the cluster A, subscribing the update file length OP of the cluster A

A log analysis module: analyzing the length OP of the update file pushed from the cluster A to the cluster B to obtain information such as offset, length, file ID, bucket ID and the like of the data to be backed up;

a data reading module: the cluster B acquires object information of the data to be backed up from the cluster A and reads the data from the corresponding storage node;

a data storage module: and writing the read data to be backed up to the local.

In fig. 6, after a client acquires storage node information of data to be stored from a metadata management module of a cluster a (storage cluster), the data to be stored is written into a corresponding storage node, when the length of the data to be backed up is not less than a preset threshold, the client triggers an update file length OP request to be sent to the metadata management module, the metadata management module queries information of an associated cluster, a log association module sends the update file length OP to the associated cluster, a log analysis module of the associated cluster analyzes the update file length OP, a data reading module obtains object information of the data to be backed up from the metadata management module according to the analyzed information, reads the data from the corresponding storage node, and stores the data in the data storage module to complete data backup. In addition, the cluster A also comprises a standby node which is used for synchronizing the metadata and improving the data security.

It should be noted that only one storage cluster and one association cluster are shown in fig. 6, actually, the number of the storage clusters and the number of the association clusters are not unique, and one storage cluster may correspond to multiple association clusters to perform data backup on the data to be stored together, which is not specifically limited herein.

Referring to fig. 7, a schematic flowchart of a data backup method in an embodiment of the present application is shown, which includes the following steps:

s701: the client writes the data to be stored into a cache;

s702: the client writes the full stripe data into the storage nodes every time the client stores the full stripe data;

s703: the client judges whether the length of the data to be backed up is not less than a preset threshold value, if so, the step S704 is executed, and if not, the step S702 is executed;

s704: the client sends an OP request for updating the file length to the MDS;

s705: the MDS analyzes the update file length OP request, creates an update file length OP and stores the update file length OP in the memory;

s706: and the log management module acquires the associated cluster from the subscription service and sends the update file length OP to the associated cluster.

Referring to fig. 8, an implementation flowchart of a data backup method provided in the embodiment of the present application, taking an execution subject as a server as an example, includes the following steps S801 to S802:

s801: receiving a data backup instruction;

the data backup instruction is sent by management equipment corresponding to the storage cluster when the length of the data to be backed up is determined to be not less than a preset threshold value after the client finishes the storage for the set times, the data to be backed up comprises data which is stored in the storage nodes in the data to be stored and is not backed up, the storage nodes belong to the storage cluster, and the association cluster and the storage cluster belong to the same cloud storage system;

s802: and backing up the data to be backed up based on the data backup instruction.

Specifically, the association cluster analyzes and updates a file length OP (namely a data backup instruction), analyzes information such as an operation type, a data offset, a data length, a bucket ID and a file ID of the OP, queries whether the file exists according to the bucket ID and the file ID obtained by the analysis to a directory service, returns the file information including information such as a file type, a file length and an EC type if the file exists, determines that the file exists later, carries the file ID to the MDS to obtain object information of 0-4G, obtains inode file objects of the file according to the file ID, maintains object information and the number of objects of the file in the inode file objects, obtains objects in 0-4G according to the object ID, each object includes the object ID, the number of blocks and all block information, stores information such as which storage node the current block is stored and the size of the block, sends the object information to the association cluster, initializes the association cluster, constructs association cluster storage nodes, reads data read by initializing the object, and reads data read according to the number of the object read by the association cluster. When the associated cluster reads a file, firstly, a file read pointer is searched (seek) to the position of data offset, then a stripe in which the current offset is located is calculated according to the offset, an object is obtained according to the offset of the stripe in the file, the object comprises a plurality of blocks, the object read data is converted into the plurality of blocks, the object read data is connected with a storage node, the read block data is stored in the stripe, after the associated cluster reads the data successfully, the data in the stripe is written into a local storage node through a data storage module, an executed OP result is sent to an MDS, the OP operation is serialized into an EditLog by the MDS, and the OP in an MDS memory is deleted.

In the embodiment of the application, based on a video cloud storage distributed architecture, after a client acquires storage node information from an MDS, data is written into a storage node, the length of a file is updated after data with a certain length is written into the storage node, an OP operation log corresponding to a write-in operation is generated, the granularity of the written-in file length is adjustable, the OP operation log is subscribed by an associated cluster, the file information is analyzed, the data with the corresponding length is read from the storage node through asynchronous operation and written into a local, and the purpose of dynamically backing up the data in real time is achieved.

In addition, in order to prevent the associated cluster from occupying too large bandwidth for reading data from the storage cluster and affecting normal service writing of the storage cluster, the storage cluster may limit the bandwidth traffic of the associated cluster for reading data, the default size is 50M, and the size of the limited traffic may be adjusted in the data storage node.

Based on the mode, the read flow of the associated cluster from the storage nodes in the storage cluster is limited, the disk pressure is reduced, the writing IO of the normal streaming media service is basically not influenced, the read flow threshold is adjustable in the configuration file, the adjustment can be carried out according to the actual service scene, the flexibility of dynamic configuration is enhanced, and the reliability and the stability of the cloud storage system are improved

Referring to fig. 9, which is a schematic flowchart of another data backup method in the embodiment of the present application, the method includes the following steps:

s901: a log association module of the storage cluster sends the length OP of the update file to an association cluster;

s902: analyzing and updating the file length OP by the associated cluster to obtain file information, and obtaining object information from the storage cluster according to the file information;

s903: storing the cluster check file information and sending the object information to the associated cluster;

s904: associating the cluster initialization object and initializing a read cache;

s905: the association cluster shifts out the current read pointer seek, and calculates the corresponding relation between the object and the stripe according to the shift and the data length;

s906: the associated cluster reads the block data and stores the block data into the stripe according to the block information and the corresponding relation in the object;

s907: the associated cluster writes the data in the stripe to the local data storage node.

Based on the mode, one file in the storage cluster is written into different disks of the storage nodes, if the storage nodes have faults, the file cannot be read and cannot be recovered, and the storage cluster can read data with fixed offset and fixed length from the associated cluster according to the offset and the length of the damaged data to recover

Based on the same inventive concept, the embodiment of the application also provides a data backup device. As shown in fig. 10, which is a schematic diagram of a data backup apparatus 1000, the data backup apparatus may include:

an obtaining unit 1001, configured to obtain storage node information of data to be stored, where the storage node information includes at least one storage node corresponding to the data to be stored, and the storage node belongs to a storage cluster;

the storage unit 1002 is configured to store data to be stored into a corresponding storage node according to the storage node information, where a length of the data stored each time is not greater than a preset length;

a sending unit 1003, configured to execute the following procedure every time the storage of the set number of times is completed:

judging whether the length of the data to be backed up is not less than a preset threshold value, wherein the data to be backed up comprises data which is stored in a storage node and is not backed up in the data to be stored;

if the length of the data to be backed up is not less than the preset threshold value, sending a data backup instruction to the associated cluster through the management device corresponding to the storage cluster so that the associated cluster backs up the data to be backed up, wherein the associated cluster and the storage cluster belong to the same cloud storage system.

In an optional implementation, the apparatus further comprises a dividing unit 1004 configured to:

and taking each piece of sub-storage data and the verification data as data to be stored.

In an optional implementation, the apparatus further includes a determining unit 1005 configured to:

if the length of the data to be backed up is smaller than a preset threshold value, judging whether the data to be stored has data which is not stored in a corresponding storage node;

and if the data which are not stored in the corresponding storage nodes do not exist in the data to be stored, the management equipment informs the association cluster to back up the data to be backed up.

In an optional embodiment, the sending unit 1003 is specifically configured to:

and sending the data backup request to the management equipment so that the management equipment generates a data backup instruction based on the data backup request, and sending the data backup instruction to the associated cluster, wherein the data backup instruction comprises the data length of the data to be backed up and the position of the data to be stored. Optionally, the first processing unit is specifically configured to:

based on the same inventive concept, the embodiment of the application also provides a data backup device. Fig. 11 is a schematic diagram of a data backup apparatus 1100, which may include:

a receiving unit 1101 for receiving a data backup instruction; the data backup instruction is sent by management equipment corresponding to the storage cluster when the length of the data to be backed up is determined to be not less than a preset threshold value after the client finishes the storage for the set times, the data to be backed up comprises data which is stored in the storage nodes in the data to be stored and is not backed up, the storage nodes belong to the storage cluster, and the association cluster and the storage cluster belong to the same cloud storage system;

the backup unit 1102 is configured to backup the data to be backed up based on the data backup instruction.

In an alternative embodiment, the backup unit 1102 is specifically configured to:

analyzing the data backup instruction to obtain the data length of the data to be backed up and the position of the data to be backed up in the data to be stored;

and reading the data to be backed up from each storage node, and storing the data to be backed up to the local. For convenience of description, the above parts are separately described as modules (or units) according to functional division. Of course, the functionality of the various modules (or units) may be implemented in the same one or more pieces of software or hardware when implementing the present application.

As will be appreciated by one skilled in the art, aspects of the present application may be embodied as a system, method or program product. Accordingly, various aspects of the present application may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.

The electronic equipment is based on the same inventive concept as the method embodiment, and the embodiment of the application also provides the electronic equipment. In one embodiment, the electronic device may be a server, such as server 120 shown in FIG. 1. In this embodiment, the electronic device may be configured as shown in fig. 12, and include a memory 1201, a communication module 1203, and one or more processors 1202.

A memory 1201 for storing computer programs executed by the processor 1202. The memory 1201 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, a program required for running an instant messaging function, and the like; the storage data area can store various instant messaging information, operation instruction sets and the like.

Memory 1201 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1201 may also be a non-volatile memory (non-volatile memory), such as a read-only memory (rom), a flash memory (flash memory), a hard disk (HDD) or a SolID State Drive (SSD); or the memory 1201 is any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 1201 may be a combination of the above memories.

The processor 1202 may include one or more Central Processing Units (CPUs), a digital processing unit, and the like. A processor 1202, configured to implement the data backup method when calling the computer program stored in the memory 1201.

The communication module 1203 is used for communicating with the terminal device and other servers.

In the embodiment of the present application, the specific connection medium between the memory 1201, the communication module 1203 and the processor 1202 is not limited. In fig. 12, the memory 1201 and the processor 1202 are connected by a bus 1204, the bus 1204 is depicted by a thick line in fig. 12, and the connection manner between other components is merely illustrative and not limited. The bus 1204 may be divided into an address bus, a data bus, a control bus, and the like. For ease of description, only one thick line is depicted in fig. 12, but only one bus or one type of bus is not depicted.

The memory 1201 stores a computer storage medium, and the computer storage medium stores computer-executable instructions for implementing the data backup method according to the embodiment of the present application. The processor 1202 is configured to execute the data backup method described above, as shown in fig. 2 or fig. 8.

In another embodiment, the electronic device may also be other electronic devices, such as the terminal device 110 shown in fig. 1. In this embodiment, the structure of the electronic device may be as shown in fig. 13, including: a communication assembly 1310, a memory 1320, a display unit 1330, a camera 1340, a sensor 1350, an audio circuit 1360, a bluetooth module 1370, a processor 1380, and the like.

The communication component 1310 is for communicating with a server. In some embodiments, a circuit Wireless FIDelity (WiFi) module may be included, the WiFi module belongs to a short distance Wireless transmission technology, and the electronic device may help a user to send and receive information through the WiFi module.

Memory 1320 may be used to store software programs and data. The processor 1380 performs various functions of the terminal device 110 and data processing by executing software programs or data stored in the memory 1320. The memory 1320 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Memory 1320 stores an operating system that enables terminal device 110 to operate. The memory 1320 in the present application may store an operating system and various application programs, and may also store a computer program for executing the data backup method according to the embodiment of the present application.

The display unit 1330 may also be used to display a Graphical User Interface (GUI) of information input by or provided to the user and various menus of the terminal device 110. Specifically, display unit 1330 may include a display screen 1332 disposed on a front surface of terminal device 110. The display 1332 may be configured in the form of a liquid crystal display, a light emitting diode, or the like. The display unit 1330 may be configured to display a data backup user interface in the embodiment of the present application, and the like.

The display unit 1330 may also be configured to receive input numeric or character information and generate signal input related to user settings and function control of the terminal device 110, and specifically, the display unit 1330 may include a touch screen 1331 disposed on the front surface of the terminal device 110 and configured to collect touch operations by a user thereon or nearby, such as clicking a button, dragging a scroll box, and the like.

Touch screen 1331 may cover display screen 1332, or touch screen 1331 and display screen 1332 may be integrated to implement the input and output functions of terminal device 110, and after integration, the touch screen may be referred to as a touch display screen for short. The display unit 1330 may display the application programs and the corresponding operation steps.

The camera 1340 may be used to capture still images, and the user may post comments on the images taken by the camera 1340 through the application. The number of the cameras 1340 may be one or more. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxIDe-semiconductor (CMOS) phototransistor. The light sensing elements convert the light signals into electrical signals, which are then passed to a processor 1380 for conversion into digital image signals.

The terminal device may further comprise at least one sensor 1350, such as an acceleration sensor 1351, a distance sensor 1352, a fingerprint sensor 1353, a temperature sensor 1354. The terminal device may also be configured with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, motion sensor, and the like.

The audio circuit 1360, speaker 1361, microphone 1362 may provide an audio interface between the user and the terminal device 110. The audio circuit 1360 may transmit the electrical signal converted from the received audio data to the speaker 1361, and the electrical signal is converted into a sound signal by the speaker 1361 and output. Terminal device 110 may also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1362 converts the collected sound signal into an electrical signal, converts the electrical signal into audio data after being received by the audio circuit 1360, and then outputs the audio data to the communication module 1310 to be transmitted to, for example, another terminal device 110, or outputs the audio data to the memory 1320 for further processing.

The bluetooth module 1370 is used for information interaction with other bluetooth devices having a bluetooth module through a bluetooth protocol. For example, the terminal device may establish a bluetooth connection with a wearable electronic device (e.g., a smart watch) that is also equipped with a bluetooth module through the bluetooth module 1370, so as to perform data interaction.

The processor 1380 is a control center of the terminal device, connects various parts of the entire terminal device using various interfaces and lines, performs various functions of the terminal device and processes data by operating or executing software programs stored in the memory 1320 and calling data stored in the memory 1320. In some embodiments, processor 1380 may include one or more processing units; the processor 1380 may also integrate an application processor, which primarily handles operating systems, user interfaces, application programs, and the like, and a baseband processor, which primarily handles wireless communications. It will be appreciated that the baseband processor may not be integrated into the processor 1380. The processor 1380 in the present application may run an operating system, an application, a user interface display, and a touch response, as well as the data backup method of the embodiments of the present application. Additionally, a processor 1380 is coupled to the display unit 1330.

In some possible embodiments, the aspects of the data backup method provided in the present application may also be implemented in the form of a program product including a computer program for causing an electronic device to perform the steps in the data backup method according to various exemplary embodiments of the present application described above in this specification when the program product is run on the electronic device, for example, the electronic device may perform the steps as shown in fig. 2 or fig. 8.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The program product of embodiments of the present application may employ a portable compact disc read only memory (CD-ROM) and include a computer program, and may be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with a command execution system, apparatus, or device.

Readable signal media may include a propagated data signal with a readable computer program embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with a command execution system, apparatus, or device.

The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer programs for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer program may execute entirely on the consumer electronic device, partly on the consumer electronic device, as a stand-alone software package, partly on the consumer electronic device and partly on a remote electronic device, or entirely on the remote electronic device or server. In the case of remote electronic devices, the remote electronic devices may be connected to the consumer electronic device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external electronic device (for example, through the internet using an internet service provider).

It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. Indeed, the features and functions of two or more of the units described above may be embodied in one unit, according to embodiments of the application. Conversely, the features and functions of one unit described above may be further divided into embodiments by a plurality of units.

Further, while the operations of the methods of the present application are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having a computer-usable computer program embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A method for backing up data, the method comprising:

2. The method of claim 1, wherein before the obtaining storage node information of data to be stored, further comprising:

dividing data to be stored into a plurality of sub-storage data, and generating check data based on each sub-storage data; the check data is used for representing the incidence relation among the sub-storage data;

3. The method of claim 1, wherein the method further comprises:

if the data which are not stored in the corresponding storage nodes do not exist in the data to be stored, the associated cluster is notified to back up the data to be backed up through the management equipment.

4. The method of any one of claims 1 to 3, wherein sending the data backup instruction to the associated cluster through the management device corresponding to the storage cluster comprises:

5. A data backup method is applied to an associated cluster, and comprises the following steps:

and backing up the data to be backed up based on the data backup instruction.

6. The method of claim 5, wherein backing up the data to be backed up based on the data backup instructions comprises:

7. A data backup apparatus, comprising:

8. A data backup apparatus, comprising:

the receiving unit is used for receiving a data backup instruction; the data backup instruction is sent by a management device corresponding to a storage cluster when a client determines that the length of data to be backed up is not less than a preset threshold value after finishing storage for a set number of times, the data to be backed up comprises data which is stored in storage nodes in the data to be stored and is not backed up, the storage nodes belong to the storage cluster, and the associated cluster and the storage cluster belong to the same cloud storage system;

9. An electronic device, characterized in that it comprises a processor and a memory, wherein the memory stores a computer program which, when executed by the processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that it comprises a computer program for causing an electronic device to carry out the steps of the method according to any one of claims 1 to 6, when said computer program is run on said electronic device.

11. A computer program product, comprising a computer program stored in a computer readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the method of any of claims 1-6.