WO2020048488A1 - 一种数据存储方法及存储装置 - Google Patents
一种数据存储方法及存储装置 Download PDFInfo
- Publication number
- WO2020048488A1 WO2020048488A1 PCT/CN2019/104417 CN2019104417W WO2020048488A1 WO 2020048488 A1 WO2020048488 A1 WO 2020048488A1 CN 2019104417 W CN2019104417 W CN 2019104417W WO 2020048488 A1 WO2020048488 A1 WO 2020048488A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- stored
- security level
- storage
- equal parts
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
- G06F3/0623—Securing storage systems in relation to content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
Definitions
- the present application relates to the field of data storage technology, and in particular, to a data storage method and a storage device.
- Video cloud storage technology is a technology for data cloud storage.
- the cloud storage server can receive data from the front-end camera equipment and store the data.
- Current cloud storage servers usually store only one copy when storing data.
- EC erasure code, erasure code
- EC technology is a data encoding technology. It can add N original data, increase M data, and restore any original N data from N + M data to original data. That is, if there are any M or less The data is invalid, and the original data can still be restored from the remaining N copies.
- the cloud storage server uses EC technology to store data, specifically, after receiving data from the front-end camera device, the received data can be divided into N shares according to a preset security level N + M value, and then these N shares The data generates M copies of verification data, and then stores the N + M pieces of data into N + M storage nodes respectively.
- the data sent by the front-end camera equipment usually includes real-time video data, alarm video data, and smart stream data, where the real-time video data is video data recorded by the camera device under normal conditions; the alarm video data is recorded by the camera device under emergency conditions
- Intelligent stream data is data obtained by the camera device after the video data is lightly calculated, and contains the feature information of the subject in the video, such as the characteristics of the person ’s gender and clothing color in the video.
- Current cloud storage servers generally encode and store data sent by the same camera device with the same security level.
- the purpose of the embodiments of the present application is to provide a data storage method and a storage device, so as to further ensure that when some storage nodes of a cloud storage server fail, important data stored therein can be restored.
- an embodiment of the present application provides a data storage method, which is applied to a cloud storage server.
- the method includes:
- the method further includes:
- the determining the data type of the data to be stored includes:
- determining the security level of the data to be stored according to a preset corresponding relationship between different data types and different security levels includes:
- determining that the security level of the data to be stored is N + M, where N and M are both integers greater than 1;
- the security level of the data to be stored is N + (M + 2).
- generating the verification data that conforms to the security level of the data to be stored according to the determined security level of the data to be stored includes:
- the security level of the data to be stored is N + (M + 2)
- the data to be stored is divided into N equal parts of the original data, and verification data of M + 2 equal parts is generated.
- the storing the data to be stored and the verification data to N + M storage nodes respectively includes:
- N N + M
- M M
- the method further includes:
- Generate storage result information for the data to be stored where the storage result information includes: identification information that the data to be stored has been successfully stored, and a storage node name for storing the data to be stored.
- an embodiment of the present application provides a data storage device, which is applied to a cloud storage server.
- the device includes:
- a first determining module configured to determine a data type of data to be stored
- a second determining module configured to determine a security level of the data to be stored according to a preset corresponding relationship between different data types and different security levels
- a first generating module configured to generate check data that meets the security level of the data to be stored according to the determined security level of the data to be stored;
- the storage module is configured to store the data to be stored and the verification data to N + m storage nodes, respectively, where the number of N + m is consistent with the security level of the data to be stored.
- the apparatus further includes:
- the receiving module is configured to receive data to be stored sent by the front-end camera device.
- the first determining module is specifically configured to:
- the second determining module is specifically configured to:
- determining that the security level of the data to be stored is N + M, where N and M are both integers greater than 1;
- the security level of the data to be stored is N + (M + 2).
- the first generating module is specifically configured to:
- the security level of the data to be stored is N + (M + 2)
- the data to be stored is divided into N equal parts of original data, and verification data of M + 2 equal parts is generated.
- the storage module is specifically configured to:
- the apparatus further includes:
- a second generating module is configured to generate storage result information for the data to be stored, where the storage result information includes identification information that the data to be stored has been successfully stored, and storage for storing the data to be stored Node name.
- an embodiment of the present application provides a cloud storage server, including a processor and a machine-readable storage medium, where the machine-readable storage medium stores machine-executable instructions that can be executed by the processor.
- the processor is prompted by the machine-executable instructions to implement the method steps of the data storage method provided by the first aspect of the embodiments of the present application.
- an embodiment of the present application provides a computer-readable storage medium.
- the computer-readable storage medium stores a computer program, and the computer program is executed by a processor.
- the data storage provided in the first aspect of the embodiment of the present application Method steps of the method.
- an embodiment of the present application provides an application program for executing at runtime: the data storage method provided in the first aspect of the embodiment of the present application.
- the cloud storage server determines the data type of the data to be stored, it can determine the corresponding security level of the data to be stored according to the preset corresponding relationship between different data types and different security levels, so that Generate verification data that meets the security level of the data to be stored.
- the data to be stored is coded with EC technology, more copies of the verification data can be generated, and the data to be stored and the generated multiple copies of the verification data are separated.
- N + m storage nodes to further ensure that when some storage nodes fail, the data with a higher security level can still be restored by the data stored in the storage nodes that have not failed, thereby further Keep your important data safe.
- it is not necessary to achieve all the advantages described above at the same time.
- FIG. 1a is a schematic flowchart of a data storage method according to an embodiment of the present application.
- FIG. 1b is a schematic flowchart of step S104 in the embodiment of the present application.
- FIG. 2 is another schematic flowchart of a data storage method according to an embodiment of the present application.
- FIG. 3 is a schematic flowchart of a specific implementation of a data storage method according to an embodiment of the present application.
- FIG. 4 is a schematic structural diagram of a data storage device according to an embodiment of the present application.
- FIG. 5 is another schematic structural diagram of a data storage device according to an embodiment of the present application.
- FIG. 6 is a schematic structural diagram of a cloud storage server according to an embodiment of the present application.
- network RAID Redundant Arrays of Independent Disks
- network RAID can protect user data across nodes on the basis of saving user costs.
- the implementation of network RAID is mainly based on EC technology.
- critical data such as real-time video data, alarm video data, and intelligent streaming data, the security is consistent with ordinary data.
- the security level of the data is N + M
- the number of storage nodes that are simultaneously offline in the cluster is greater than M
- the real-time streaming data, intelligent data, The alarm data and so on are all unreadable, and at this time, if you want to view the recording of the alarm event under the front-end camera device, you will not be able to view it, or you need to view the smart data under the front-end camera device.
- a data storage method determines when a cloud storage server receives data sent from a front-end camera device, determines the data type of the received data, and according to the corresponding relationship between different data types and different security levels To determine the security level corresponding to the data, so as to generate check data that conforms to the security level of the data to be stored.
- a cloud storage server receives data sent from a front-end camera device, determines the data type of the received data, and according to the corresponding relationship between different data types and different security levels To determine the security level corresponding to the data, so as to generate check data that conforms to the security level of the data to be stored.
- an embodiment of the present application provides a data storage method, which is applied to a cloud storage server.
- the process may include the following steps:
- the cloud storage server After the cloud storage server receives the data to be stored, it can determine the data type of the received data. Specifically, it can determine that the data type is real-time video data, alarm video data, or intelligent stream data.
- the real-time video data may be a video data stream recorded by the front-end camera device under normal conditions;
- the alarm video data may be video data recorded by the front-end camera device in an emergency situation, for example, a moving object suddenly appears in the shooting range of the current camera device
- the intelligent stream data may be data obtained by the front-end camera device after performing light calculation on the video data, and including the characteristic information of the shooting object in the video.
- the front-end camera device can have data processing capabilities, so that it can obtain, for example, feature information of the photographic subject, such as information such as the gender and clothing color of the person.
- the intelligent stream data received by the cloud storage server can be data in the format of JSON (JavaScript Object Notation, a lightweight array) string.
- the data to be stored received by the cloud storage server may carry different data type identifiers, so that the cloud storage server may determine the data type of the data to be stored according to the data type identifier. .
- the data type identifier is 1, it can be determined that the data type of the data to be stored is real-time video data; when the data type identifier is 2, it can be determined that the data type of the data to be stored is alarm video data; when the data When the type identifier is 3, it can be determined that the data type of the data to be stored is intelligent stream data.
- the front-end camera device may send data to be stored in the form of a data packet. By adding the foregoing data type identification field to the header of the data packet, after the cloud storage server receives the data to be stored, the cloud storage server may identify the field according to the data type therein. Determine the data type of the received data to be stored.
- S102 Determine the security level of the data to be stored according to the preset correspondence between different data types and different security levels.
- the security level of a data can be expressed as the N + M value of the data, where N can refer to the number of original data obtained after dividing a data into N parts, and M can refer to the N original data The number of M pieces of verification data to be generated. It is redundant data obtained after calculation by algorithms in EC technology. Where N and M are integers greater than one.
- a correspondence relationship between different data types and different security levels is preset, and this correspondence relationship may be stored in a cloud storage server.
- the security levels of the real-time video data, the alarm video data, and the intelligent stream data are set to be sequentially increased.
- the security level of real-time video data can be preset to N + M
- the security level of alarm video data can be preset to N + (M + 1)
- the security level of intelligent stream data can be preset to N + (M + 2)
- the server can determine the security level of the data to be stored through a preset correspondence.
- the security levels of the above three data types may also be preset to other values, as long as the security levels of real-time video data, alarm video data, and intelligent stream data are sequentially increased.
- step S102 may specifically include:
- the security level of the data to be stored is N + (M + 2).
- the security level of the real-time video data, alarm video data, and intelligent stream data determined by the cloud storage server increases in order, indicating that the server can receive different types of data to be stored sent by the front-end camera device, and can pass different Security level, when some storage nodes of the cloud storage server fail, to further ensure that these data can be restored and recovered.
- the number of copies of the verification data to be generated by the data to be stored may be determined according to the determined security level, and the data to be stored is encoded to generate the data to be stored
- the check data of the data that is, the check data conforming to the security level of the data to be stored.
- Data coding technology based on EC technology, it can convert N original data, increase M data, and restore any original N data from N + M data to original data, so it can be based on the security of the data to be stored. Level to generate verification data that meets the security level of the data to be stored.
- step S104 may specifically include:
- the security level of the data to be stored is N + M
- the data to be stored is divided into N equal parts of the original data, and verification data of M equal parts is generated.
- the security level of the data to be stored is N + (M + 1)
- the data to be stored is divided into N equal parts of the original data, and verification data of M + 1 equal parts is generated.
- the security level of the data to be stored is N + (M + 2)
- the data to be stored is divided into N equal parts of the original data, and verification data of M + 2 equal parts is generated.
- a larger number of verification data can be generated. For example, for the alarm video data with security level N + (M + 1), you can further set the security level to N + (M + 2), and the number of copies of the generated verification data is M + 2; for the security level, For the intelligent stream data of N + (M + 2), the security level can be further set to N + (M + 3), and the number of copies of the generated verification data is M + 3.
- Data coding technology based on EC technology, it can convert N original data, increase M data, and restore any original N data from N + M data to original data, so it can be based on the security level of the data to be stored. Generate verification data that meets the security level of the data to be stored, and store the data and verification data in a number of storage nodes that are consistent with the security level of the data to be stored, that is, to N + m storage nodes respectively.
- the data to be stored may be divided into multiple copies of original data, and the generated verification data that meets the security level of the data to be stored may also be multiple copies.
- the data to be stored and the verification data may be stored in a storage node of the cloud storage server.
- the above-mentioned multiple copies of the original data and multiple copies of the verification data can be stored in the same number of storage nodes as the sum of their numbers, and even when some storage nodes fail, they can still pass through the storage nodes that have not failed The data stored in is restored.
- the sum of the number of copies of the original data and the verification data is the security level of the data, that is, the number of storage nodes used to store the data to be stored is consistent with the security level of the data to be stored.
- step S104 may specifically include:
- m is the number of storage nodes required to verify the data.
- the security level of the data to be stored is N + (M + 1)
- the number of original data divided by the data to be stored is N
- the number of copies of the verification data generated by the data to be stored is (M + 1) copies
- the security level of the data to be stored is N + (M + 2)
- the number of original data divided by the data to be stored is N
- the number of copies of the verification data generated by the data to be stored is (M + 2) copies
- the cloud storage server determines the data type of the data to be stored, it can determine the corresponding security level of the data to be stored according to the preset corresponding relationship between different data types and different security levels, so that Generate verification data that meets the security level of the data to be stored.
- the data to be stored is coded with EC technology, more copies of the verification data can be generated, and the data to be stored and the generated multiple copies of the verification data are separated.
- N + m storage nodes to further ensure that when some storage nodes fail, the data with a higher security level can still be restored by the data stored in the storage nodes that have not failed, thereby further Keep your important data safe.
- FIG. 2 is another schematic flowchart of a data storage method according to an embodiment of the present application. The process is applied to a cloud storage server and includes the following steps:
- the cloud storage server can receive the data to be stored sent by the front-end camera device, and store the received data to be stored in its own storage node.
- the cloud storage server can also receive data from different front-end camera devices.
- S203 Determine the security level of the data to be stored according to the preset correspondence between different data types and different security levels.
- steps S202 to S205 are performed with reference to the processes of S101 to S104 in FIG. 1a, and details are not described herein again.
- S206 Generate storage result information for the data to be stored, where the storage result information includes an identifier of whether the data to be stored was successfully stored, and a name of a storage node for storing the data to be stored.
- the cloud storage server may generate identification information for the data to be stored successfully. Exemplarily, the information may be identified as "successfully stored”.
- the cloud storage server can also record the storage location of the stored data and its verification data, that is, which storage nodes are located, and the names of these storage nodes, so as to facilitate data query based on the storage location.
- the stored data refers to the storage to be stored.
- the stored data corresponding to the data.
- a data storage method provided in the embodiment of the present application after the data to be stored sent by the front-end camera device is successfully stored, information about a storage result for the data to be stored can be generated, thereby facilitating data query according to the storage location and thus facilitating Query whether the data to be stored has been successfully stored, and query the storage location of the stored data in the cloud storage server.
- a cloud storage server is provided with a video access module (AccessService) for receiving data sent from a front-end camera device;
- Encoding unit Encoder
- Encoder is used to perform EC encoding on the data and send the encoded data to the storage node;
- the method includes the following steps:
- the front-end device sends data to be stored to the video access module.
- the front-end device can actively push data to the video access module, or it can send data to the video access module according to the acquisition request of the video access module.
- the data can include data such as real-time video data, alarm video data, or intelligent streaming data.
- the video access module After receiving the data to be stored, the video access module determines a data type of the received data, for example, real-time video data, alarm video data, or intelligent stream data.
- a data type of the received data for example, real-time video data, alarm video data, or intelligent stream data.
- the video access module obtains preset correspondences between different data types and different security levels, thereby determining the security level of the data to be stored.
- the video access module sends the data to be stored and the security level corresponding to the data to be stored to the encoding calculation unit.
- the encoding calculation unit encodes the data to be stored, and generates corresponding verification data that meets the security level of the data to be stored.
- the encoding calculation unit sends the encoded data to a different data storage server.
- the data storage server After the data storage server stores the encoded data, it returns the storage result information to the encoding calculation unit.
- the encoding calculation unit returns the foregoing storage result information to the video access module.
- a data storage method provided in the embodiment of the present application.
- a cloud storage server receives data sent from a front-end camera device, it can determine a security level corresponding to the data to be stored according to different data types of the data, so as to generate data consistent with the data to be stored.
- Security-level check data In this way, when the data is coded using EC technology, more copies of the check data can be generated, and then the data divided into multiple original data and the generated multiple check data are stored separately.
- FIG. 4 is a schematic structural diagram of a data storage device according to an embodiment of the present application, including:
- the first determining module 401 is configured to determine a data type of data to be stored.
- the second determining module 402 is configured to determine a security level of data to be stored according to preset correspondences between different data types and different security levels.
- the first generating module 403 is configured to generate verification data that conforms to the security level of the data to be stored according to the determined security level of the data to be stored.
- the storage module 404 is configured to store the data and the verification data to N + m storage nodes, respectively, where the number of N + m is consistent with the security level of the data to be stored.
- the first determining module 401 is specifically configured to determine a data type of the data to be stored according to a data type identifier in the data to be stored.
- the second determining module 402 is specifically configured to:
- the security level of the data to be stored is N + M, where N and M are both integers greater than 1.
- the security level of the data to be stored is N + (M + 2).
- the first generating module 403 is specifically configured to:
- the security level of the data to be stored is N + M
- the data to be stored is divided into N equal parts of the original data, and verification data of M equal parts is generated
- the security level of the data to be stored is N + (M + 2)
- the data to be stored is divided into N equal parts of the original data, and verification data of M + 2 equal parts is generated.
- the storage module 404 is specifically configured to:
- N + M the security level of the data to be stored is N + M
- the cloud storage server determines the data type of the data to be stored, it can determine the corresponding security level of the data to be stored according to the preset corresponding relationship between different data types and different security levels, so that Generate verification data that meets the security level of the data to be stored.
- the data to be stored is coded with EC technology, more copies of the verification data can be generated, and the data to be stored and the generated multiple copies of the verification data are separated.
- N + m storage nodes to further ensure that when some storage nodes fail, the data with a higher security level can still be restored by the data stored in the storage nodes that have not failed, thereby further Keep your important data safe.
- FIG. 5 is another schematic diagram of the structure of the data storage device according to the embodiment of the present application, including:
- the receiving module 501 is configured to receive data to be stored sent by a front-end camera device.
- the first determining module 401 is configured to determine a data type of data to be stored.
- the second determining module 402 is configured to determine a security level of data to be stored according to preset correspondences between different data types and different security levels.
- the first generating module 403 is configured to generate verification data that conforms to the security level of the data to be stored according to the determined security level of the data to be stored.
- the storage module 404 is configured to store the data and the verification data to N + m storage nodes, respectively, where the number of N + m is consistent with the security level of the data to be stored.
- the second generating module 502 is configured to generate storage result information for the data to be stored, where the storage result information includes identification information that the data to be stored has been successfully stored, and a storage node name for storing the data to be stored.
- a data storage device provided in the embodiment of the present application can generate storage result information for the data to be stored after the data to be stored sent by the front-end camera device is successfully stored, thereby facilitating querying whether the data to be stored has been successfully stored. And query the storage location of the stored data in the cloud storage server.
- the server 600 includes a processor 601 and a machine-readable storage medium 602.
- the machine-readable storage medium stores a machine executable executable by the processor. Instructions, the processor is caused by machine-executable instructions to implement the following steps:
- the data to be stored and the verification data are respectively stored to N + m storage nodes, and the number of N + m is consistent with the security level of the data to be stored.
- a cloud storage server provided in the embodiment of the present application. After determining the data type of the data to be stored, the cloud storage server can determine the security level corresponding to the data to be stored according to preset correspondences between different data types and different security levels, thereby Generate verification data that meets the security level of the data to be stored. In this way, when the data to be stored is coded with EC technology, more copies of the verification data can be generated, and the data to be stored and the generated multiple copies of the verification data are separated. Store to N + m storage nodes to further ensure that when some storage nodes fail, the data with a higher security level can still be restored by the data stored in the storage nodes that have not failed, thereby further Keep your important data safe.
- the machine-readable storage medium mentioned in the foregoing server may include Random Access Memory (RAM), and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
- the storage medium may also be at least one storage device located far from the foregoing processor.
- the above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like; and a digital signal processor (DSP) Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
- CPU central processing unit
- NP network processor
- DSP digital signal processor
- ASIC Application Specific Integrated Circuit
- FPGA Field-Programmable Gate Array
- An embodiment of the present application further provides a computer-readable storage medium, and a computer program is stored in the computer-readable storage medium to perform the following steps:
- the data to be stored and the verification data are respectively stored to N + m storage nodes, and the number of N + m is consistent with the security level of the data to be stored.
- the cloud storage server after the cloud storage server determines the data type of the data to be stored, it can determine the corresponding security level of the data to be stored according to the preset corresponding relationship between different data types and different security levels.
- the cloud storage server determines the corresponding security level of the data to be stored according to the preset corresponding relationship between different data types and different security levels.
- the data is stored in N + m storage nodes respectively, to further ensure that when some storage nodes fail, the stored data with a higher security level can still be restored through the data stored in the storage nodes that have not failed, So as to further ensure the security of important data.
- the embodiment of the present application further provides a computer program product, which is used to execute the following steps when it runs on a computer:
- the data to be stored and the verification data are respectively stored to N + m storage nodes, and the number of N + m is consistent with the security level of the data to be stored.
- the cloud storage server determines the data type of the data to be stored, it can determine the corresponding security level of the data to be stored according to preset correspondences between different data types and different security levels, thereby generating Verification data that conforms to the security level of the data to be stored.
- the data to be stored is encoded with EC technology, more copies of the verification data can be generated, and the to-be-stored data and the generated multiple copies of the verification data are stored separately.
- N + m storage nodes to further ensure that when some storage nodes fail, the data with a higher security level can still be restored from the data stored in the storage nodes that did not fail, thereby further ensuring Security of important data.
- the device, the cloud storage server, and the storage medium in the embodiments of the present application are a device, a cloud storage server, and a storage medium to which the foregoing data storage method is applied, and all embodiments of the foregoing data storage method are applicable to the device, Both cloud storage servers and storage media can achieve the same or similar beneficial effects.
Abstract
Description
Claims (17)
- 一种数据存储方法,其特征在于,应用于云存储服务器,所述方法包括:确定待存储数据的数据类型;根据预设的不同数据类型与不同安全级别的对应关系,确定所述待存储数据的安全级别;根据所确定的所述待存储数据的安全级别,生成符合所述待存储数据安全级别的校验数据;将所述待存储数据和所述校验数据分别存储至N+m个存储节点,其中,N+m数量与所述待存储数据的安全级别一致。
- 根据权利要求1所述的数据存储方法,其特征在于,所述确定待存储数据的数据类型之前,所述方法还包括:接收由前端摄像设备发送的待存储数据。
- 根据权利要求1所述的数据存储方法,其特征在于,所述确定待存储数据的数据类型,包括:根据所述待存储数据中的数据类型标识,确定所述待存储数据的数据类型。
- 根据权利要求1所述的数据存储方法,其特征在于,所述根据预设的不同数据类型与不同安全级别的对应关系,确定所述待存储数据的安全级别,包括:当所述待存储数据的数据类型为实时视频数据时,确定所述待存储数据的安全级别为N+M,其中N和M均为大于1的整数;当所述待存储数据的数据类型为告警视频数据时,确定所述待存储数据的安全级别为N+(M+1);当所述待存储数据的数据类型为智能流数据时,确定所述待存储数据的安全级别为N+(M+2)。
- 根据权利要求4所述的数据存储方法,其特征在于,所述根据所确定的所述待存储数据的安全级别,生成符合所述待存储数据安全级别的校验数据,包括:当所述待存储数据的安全级别为N+M时,将所述待存储数据划分为N等份的原始数据,并生成M等份的校验数据;当所述待存储数据的安全级别为N+(M+1)时,将所述待存储数据划分为N等份的原始数据,并生成M+1等份的校验数据;当所述待存储数据的安全级别为N+(M+2)时,将所述待存储数据划分为N等份的原始数据,并生成M+2等份的校验数据。
- 根据权利要求5所述的数据存储方法,其特征在于,所述将所述待存储数据和所述校验数据分别存储至N+m个存储节点,包括:当所述待存储数据的安全级别为N+M时,将经EC纠删码编码后的N等份原始数据和M等份校验数据,分别存储至N+M个存储节点中,其中M=m;当所述待存储数据的安全级别为N+(M+1)时,将经EC编码后的N等份原始数据和M+1等份校验数据,分别存储至N+(M+1)个存储节点中,其中M+1=m;当所述待存储数据的安全级别为N+(M+2)时,将经EC编码后的N等份原始数据和M+2等份校验数据,分别存储至N+(M+2)个存储节点中,其中M+2=m。
- 根据权利要求1或2所述的数据存储方法,其特征在于,所述将所述待存储数据和所述校验数据分别存储至N+m个存储节点之后,所述方法还包括:生成针对所述待存储数据的存储结果信息,所述存储结果信息中包括:所述待存储数据已成功存储的标识信息,以及用于存储所述待存储数据的存储节点名称。
- 一种数据存储装置,其特征在于,应用于云存储服务器,所述装置包括:第一确定模块,用于确定待存储数据的数据类型;第二确定模块,用于根据预设的不同数据类型与不同安全级别的对应关系,确定所述待存储数据的安全级别;第一生成模块,用于根据所确定的所述待存储数据的安全级别,生成符合所述待存储数据安全级别的校验数据;存储模块,用于将所述待存储数据和所述校验数据分别存储至N+m个存储节点,其中,N+m数量与所述待存储数据的安全级别一致。
- 根据权利要求8所述的数据存储装置,其特征在于,所述装置还包括:接收模块,用于接收由前端摄像设备发送的待存储数据。
- 根据权利要求8所述的数据存储装置,其特征在于,所述第一确定模块,具体用于:根据所述待存储数据中的数据类型标识,确定所述待存储数据的数据类型。
- 根据权利要求8所述的数据存储装置,其特征在于,所述第二确定模块,具体用于:当所述待存储数据的数据类型为实时视频数据时,确定所述待存储数据的安全级别为N+M,其中N和M均为大于1的整数;当所述待存储数据的数据类型为告警视频数据时,确定所述待存储数据的安全级别为N+(M+1);当所述待存储数据的数据类型为智能流数据时,确定所述待存储数据的安全级别为N+(M+2)。
- 根据权利要求11所述的数据存储装置,其特征在于,所述第一生成模块,具体用于:当所述待存储数据的安全级别为N+M时,将所述待存储数据划分为N等份的原始数据,并生成M等份的校验数据;当所述待存储数据的安全级别为N+(M+1)时,将所述待存储数据划分为N等份的原始数据,并生成M+1等份的校验数据;当所述待存储数据的安全级别为N+(M+2)时,将所述待存储数据划分为N等份的原始数据,并生成M+2等份的校验数据。
- 根据权利要求12所述的数据存储装置,其特征在于,所述存储模块,具体用于:当所述待存储数据的安全级别为N+M时,将经EC纠删码编码后的N等份原始数据和M等份校验数据,分别存储至N+m个存储节点中,其中M=m;当所述待存储数据的安全级别为N+(M+1)时,将经EC编码后的N等份原始数据和M+1等份校验数据,分别存储至N+(M+1)个存储节点中,其中M+1=m;当所述待存储数据的安全级别为N+(M+2)时,将经EC编码后的N等份原始数据和M+2等份校验数据,分别存储至N+(M+2)个存储节点中,其中M+2=m。
- 根据权利要求8或9所述的数据存储装置,其特征在于,所述装置还包括:第二生成模块,用于生成针对所述待存储数据的存储结果信息,所述存储结果信息中包括:所述待存储数据已成功存储的标识信息,以及用于存储所述待存储数据的存储节点名称。
- 一种云存储服务器,其特征在于,包括处理器和机器可读存储介质,所述机器可读存储介质存储有能够被所述处理器执行的机器可执行指令,所述处理器被所述机器可执行指令促使:实现权利要求1-7任一项所述的方法步骤。
- 一种计算机可读存储介质,其特征在于,所述计算机可读存储介质内存储有计算机程序,所述计算机程序被处理器执行时实现权利要求1-7任一项所述的方法步骤。
- 一种应用程序,其特征在于,用于在运行时执行:权利要求1-7任一项所述的方法步骤。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201811025881.7 | 2018-09-04 | ||
CN201811025881.7A CN110874185B (zh) | 2018-09-04 | 2018-09-04 | 一种数据存储方法及存储装置 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020048488A1 true WO2020048488A1 (zh) | 2020-03-12 |
Family
ID=69716015
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/104417 WO2020048488A1 (zh) | 2018-09-04 | 2019-09-04 | 一种数据存储方法及存储装置 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN110874185B (zh) |
WO (1) | WO2020048488A1 (zh) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112558875A (zh) * | 2020-12-14 | 2021-03-26 | 北京百度网讯科技有限公司 | 数据校验方法、装置、电子设备及存储介质 |
CN113590042A (zh) * | 2021-07-29 | 2021-11-02 | 杭州宏杉科技股份有限公司 | 一种数据保护存储方法、装置及设备 |
CN114218619A (zh) * | 2021-12-20 | 2022-03-22 | 昊链(中山)科技有限责任公司 | 数据安全保护方法、装置、介质及电子设备 |
CN115085993A (zh) * | 2022-06-08 | 2022-09-20 | 国汽智控(北京)科技有限公司 | 数据校验方法、装置及域控制器 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103034567A (zh) * | 2012-12-06 | 2013-04-10 | 华为技术有限公司 | 发现并修复损坏数据的装置和方法 |
CN104967691A (zh) * | 2015-07-08 | 2015-10-07 | 浪潮(北京)电子信息产业有限公司 | 一种分布式存储控制方法及系统 |
US20160274980A1 (en) * | 2015-03-20 | 2016-09-22 | Electronics And Telecommunications Research Institute | Distributed file system |
CN107341070A (zh) * | 2017-06-30 | 2017-11-10 | 长江大学 | 一种基于纠删码的随机写方法及系统 |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2006119941A (ja) * | 2004-10-22 | 2006-05-11 | Hitachi Ltd | 動画像蓄積方法 |
CN101783795B (zh) * | 2009-12-25 | 2013-02-13 | 天柏宽带网络技术(北京)有限公司 | 一种安全等级认证的方法和系统 |
CN104023246B (zh) * | 2014-04-28 | 2018-01-30 | 深圳英飞拓科技股份有限公司 | 一种视频数据私有云存储系统和视频数据私有云存储方法 |
CN104657681B (zh) * | 2015-03-13 | 2018-11-06 | 深圳酷派技术有限公司 | 一种数据存储方法及装置 |
CN106713940B (zh) * | 2015-08-04 | 2019-12-20 | 杭州海康威视数字技术股份有限公司 | 视频流的存储方法、读取方法及装置 |
EP3430515B1 (en) * | 2016-03-15 | 2021-09-22 | Datomia Research Labs Ou | Distributed storage system data management and security |
CN110597453A (zh) * | 2018-06-13 | 2019-12-20 | 杭州海康威视系统技术有限公司 | 一种云存储系统中的视频数据存储方法及装置 |
-
2018
- 2018-09-04 CN CN201811025881.7A patent/CN110874185B/zh active Active
-
2019
- 2019-09-04 WO PCT/CN2019/104417 patent/WO2020048488A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103034567A (zh) * | 2012-12-06 | 2013-04-10 | 华为技术有限公司 | 发现并修复损坏数据的装置和方法 |
US20160274980A1 (en) * | 2015-03-20 | 2016-09-22 | Electronics And Telecommunications Research Institute | Distributed file system |
CN104967691A (zh) * | 2015-07-08 | 2015-10-07 | 浪潮(北京)电子信息产业有限公司 | 一种分布式存储控制方法及系统 |
CN107341070A (zh) * | 2017-06-30 | 2017-11-10 | 长江大学 | 一种基于纠删码的随机写方法及系统 |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN112558875A (zh) * | 2020-12-14 | 2021-03-26 | 北京百度网讯科技有限公司 | 数据校验方法、装置、电子设备及存储介质 |
CN113590042A (zh) * | 2021-07-29 | 2021-11-02 | 杭州宏杉科技股份有限公司 | 一种数据保护存储方法、装置及设备 |
CN113590042B (zh) * | 2021-07-29 | 2024-03-19 | 杭州宏杉科技股份有限公司 | 一种数据保护存储方法、装置及设备 |
CN114218619A (zh) * | 2021-12-20 | 2022-03-22 | 昊链(中山)科技有限责任公司 | 数据安全保护方法、装置、介质及电子设备 |
CN115085993A (zh) * | 2022-06-08 | 2022-09-20 | 国汽智控(北京)科技有限公司 | 数据校验方法、装置及域控制器 |
Also Published As
Publication number | Publication date |
---|---|
CN110874185A (zh) | 2020-03-10 |
CN110874185B (zh) | 2021-12-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020048488A1 (zh) | 一种数据存储方法及存储装置 | |
US9542404B2 (en) | Subpartitioning of a namespace region | |
US20160006461A1 (en) | Method and device for implementation data redundancy | |
WO2020151323A1 (zh) | 一种基于数据分片的数据存储方法、设备及介质 | |
WO2018040589A1 (zh) | 一种基于分布式存储系统的数据处理方法及存储设备 | |
US9483482B2 (en) | Partitioning file system namespace | |
KR102167613B1 (ko) | 메시지 푸시 방법 및 장치 | |
WO2019237774A1 (zh) | 一种云存储系统中的视频数据存储方法及装置 | |
CN110389859B (zh) | 用于复制数据块的方法、设备和计算机程序产品 | |
CN111046310A (zh) | 页面处理方法、装置、服务器及计算机可读存储介质 | |
WO2018233630A1 (zh) | 故障发现 | |
WO2019205788A1 (zh) | 数据存储方法、存储服务器及云存储系统 | |
CN112069169B (zh) | 区块数据存储方法、装置、电子设备及可读存储介质 | |
TWI663515B (zh) | 用於物聯網設備於資料中心備份的分散式重複資料刪除儲存系統及其達成分散式重複資料刪除方法 | |
US20160080462A1 (en) | Data processing method and apparatus in service-oriented architecture system, and the service-oriented architecture system | |
CN109165112B (zh) | 一种元数据集群的故障恢复方法、系统及相关组件 | |
US20170116061A1 (en) | Mechanism for Running Diagnostic Rules Across Multiple Nodes | |
CN107516048B (zh) | 一种控制分布式文件系统中文件访问的方法与设备 | |
KR20160042933A (ko) | 데이터 피드 관리 | |
US10169440B2 (en) | Synchronous data replication in a content management system | |
CN111211993A (zh) | 流式计算的增量持久化方法及其装置 | |
US10951732B2 (en) | Service processing method and device | |
US8095980B2 (en) | Detecting malicious behavior in data transmission of a de-duplication system | |
WO2019072088A1 (zh) | 一种文件管理方法、文件管理装置、电子设备及存储介质 | |
US11704296B2 (en) | Data management system, data management apparatus, and non-transitory computer readable medium storing data management program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19858154 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19858154 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19858154 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 04.02.2022) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 19858154 Country of ref document: EP Kind code of ref document: A1 |