CN111339192A - Distributed edge computing data storage system - Google Patents
Distributed edge computing data storage system Download PDFInfo
- Publication number
- CN111339192A CN111339192A CN202010107732.6A CN202010107732A CN111339192A CN 111339192 A CN111339192 A CN 111339192A CN 202010107732 A CN202010107732 A CN 202010107732A CN 111339192 A CN111339192 A CN 111339192A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- file
- edge computing
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 42
- 238000003860 storage Methods 0.000 claims abstract description 121
- 238000012545 processing Methods 0.000 claims abstract description 22
- 238000007726 management method Methods 0.000 claims abstract description 12
- 238000009826 distribution Methods 0.000 claims abstract description 6
- 238000000034 method Methods 0.000 claims description 11
- 230000004044 response Effects 0.000 claims description 5
- 238000011144 upstream manufacturing Methods 0.000 claims 1
- 238000005516 engineering process Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000004880 explosion Methods 0.000 description 2
- 239000003999 initiator Substances 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000008263 repair mechanism Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000011010 flushing procedure Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000033001 locomotion Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011112 process operation Methods 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Abstract
The invention provides a distributed edge computing data storage system, which comprises distributed edge computing equipment for processing and storing data, terminal equipment for providing data for the distributed edge computing equipment and a data storage system, wherein the distributed edge computing equipment is used for processing and storing the data; wherein the data storage system comprises at least an object storage model and a storage system architecture; the object storage model is used for dividing user data into a group of small objects on the bottom layer, the group of small objects are uniformly and uniformly distributed on each equipment node of the whole cluster according to a certain algorithm, and when the user uses the storage model, the small objects are spliced into complete user data by a storage system and are provided for the user to use; the storage system architecture at least comprises a client, a storage interface, storage management, storage distribution, data storage and physical equipment. The invention can realize that the transaction processing is divided into each edge node for processing, thereby leading the system to be more efficient and easy to manage, accelerating the processing and transmitting speed of the data and reducing the delay.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a distributed edge computing data storage system.
Background
The edge computing means that an open platform with network, computing, storage and application core capabilities is fused at the edge side of a network close to an object or a data source, so that edge intelligent services are provided nearby, and the key requirements of industry digitization on aspects of agile connection, real-time service, data optimization, application intelligence, safety, privacy protection and the like are met.
According to the international telecommunication union, telecommunication standards office, ITU-T, research report, by 2020, each person will produce 1.7MB of data per second, and the shipment of IoT wearable devices will reach 2.37 billion. IDC also promulgated relevant predictions that by 2018, 50% of internet of things networks would face network bandwidth limitations, 40% of data would need to be analyzed, processed and stored at the network edge, and by 2025, this figure would exceed 50%.
The united states deploys more than 3000 million surveillance cameras, generating large volumes of video data over 40 hundred million hours per week. The field of the internet of things has massive terminal equipment, and if data generated by the equipment are gathered together, the data are astronomical numbers.
The analysis and storage of massive data pose a great challenge to network bandwidth, and the birth of edge computing is to solve the problem.
1. Distributed and low latency computing
Cloud computing is often not the optimal strategy and computing needs to be performed closer to the data source. This advantage can be extended to any Web-based application: APPs, including fourqure and google now, respond faster and therefore become increasingly popular with mobile users. This means that edge computations can be used to improve service at edge nodes closer to the user.
Many data streams are generated by edge devices, but with "distant" cloud computing processing and analysis, it is not possible to make real-time decisions. For example, for a visual service using a wearable camera, the response time needs to be between 25ms and 50ms, and the use of cloud computing causes a serious delay; for example, the real-time performance of detection, control and execution of an industrial system is high, the real-time performance of part of scenes is required to be within 10ms, and if all data analysis and control logics are realized in the cloud, the service requirements are difficult to meet; in addition, multimedia applications that generate huge data streams, such as videos or network games based on a cloud platform, rely on cloud computing to cause problems similar to long waiting time for players, and cannot meet the needs of users.
As an advantageous addition to cloud computing, edge nodes (e.g., routers or base stations closest to edge devices) may be utilized to reduce network latency.
2. Overriding resource limitations of terminal devices
The hardware conditions of the user terminal (e.g., a smartphone) are relatively limited compared to the servers of the data center. These terminal devices obtain data input in the form of text, audio, video, gestures or motions, but due to the limitations of middleware and hardware, the terminal devices cannot perform complex analysis and the execution process is very power consuming. Therefore, it is usually necessary to send data to the cloud, perform processing and calculation, and then return meaningful information to the terminal through a relay.
However, not all data from the end devices need to be performed by cloud computing, and the data may be filtered or analyzed at the edge nodes using spare computing resources suitable for data management tasks.
3. Sustainable energy consumption
A great deal of research has shown that cloud computing consumes enormous amounts of energy, and the amount of energy consumed by a data center in the next decade may be 3 times that consumed today. As more and more applications are moved to the cloud, energy demands are increasing or even becoming unmet. Therefore, it is very urgent to adopt a calculation strategy for maximizing energy efficiency.
The basic information acquisition and processing of some embedded small-sized devices can be completely finished at the end, namely, after the mobile phone sensor transmits data to the gateway, the data is filtered and processed through edge calculation, and each piece of original data is not required to be transmitted to the cloud, so that a large amount of energy cost is saved.
4. Coping with data explosion and network traffic pressure
The number of edge devices is speeding up, and by 2018, one third of the world's population will own smartphones or wearable devices, and by 2020, these devices will generate 43 trillion GB of data. Processing these data requires further expansion of the data center, again drawing a wide focus on network traffic pressure.
By performing data analysis on the edge device, data explosion can be effectively dealt with, and the flow pressure of the network is relieved. The edge computing can shorten the response time of the device and reduce the data traffic from the device to the cloud data center, so as to more effectively allocate resources in the network.
5. Intelligent computing
Not only consumer-grade internet of things terminals, but also edge computing will play an important role in industrial applications. The computation may be performed hierarchically, using resources at the far end of the network. For example, a typical production pipeline may filter data generated on a device, perform a portion of the analysis work on edge nodes that transmit the data, and then perform more complex computational tasks through the cloud. The edge nodes can enhance the computing capacity of the data center by sharing part of tasks of cloud computing.
The business process optimization, operation and maintenance automation and business innovation drive the business to move towards intelligence, and the edge side intelligence can bring remarkable efficiency improvement and cost advantage. In fact, edge computing is not strange to those working in industrial automation. For example, in the control system based on PLC, DCS, industrial personal computer and industrial network, which is currently and generally used, the computing resources located at the bottom layer and embedded in the device are more or less edge computing resources.
At present, the information of the metallurgical enterprises with the scale above is already made to be quite effective, but the shortage is exactly that the end intelligence is. Metallurgical data often presents integrity and consistency problems, commonly referred to as "dirty" data. The problem of poor solution can cause very big puzzlement for energy management and intelligent management link. The edge calculation plays an important role in the method and becomes an effective supplement of the industrial Internet of things technology.
Disclosure of Invention
The technical problem to be solved by the embodiments of the present invention is to provide a distributed edge computing data storage system, which solves the technical problems of system reliability, availability and access efficiency.
In one aspect of the present invention, a distributed edge computing data storage system is provided, including:
the system comprises distributed edge computing equipment for processing and storing data, terminal equipment for providing data for the distributed edge computing equipment and a data storage system;
wherein the data storage system comprises at least an object storage model and a storage system architecture;
the object storage model is used for dividing user data into a group of small objects on the bottom layer, the group of small objects are uniformly and uniformly distributed on each equipment node of the whole cluster according to a certain algorithm, and when the user uses the storage model, the small objects are spliced into complete user data by a storage system and are provided for the user to use;
the storage system architecture at least comprises a client, a storage interface, storage management, storage distribution, data storage and physical equipment.
Further, the distributed edge computing device at least comprises an edge distributed computing node, and is used for receiving resources of an edge computing API routing center node, computing service scheduling, micro-service software distribution packet pushing, distributed data resource storage, computing or storage service response.
Further, the edge of the edge computing device is any computing and network resource between the data source and the cloud computing center path, the downlink data of the edge in the edge computing device is cloud service, and the uplink data of the edge in the edge computing device is internet of everything service.
Further, the terminal device at least comprises a terminal device data operation frame, a cloud computing system and an edge computing API routing center node; the cloud computing system consists of a large number of cloud servers and is used for providing services for a large number of users; the edge computing API routing center node is used for receiving network access registration data of the edge computing equipment; the cloud server stores data in a distributed storage mode and is used for storing the data in a redundant storage mode.
Further, when the terminal device uploads a file to be stored to the cloud server, the terminal device is connected to a storage gateway and an object storage device through an external access network, the storage gateway and the object storage device are connected to a remote backup server through an internal storage network, and the remote backup server is connected to the cloud server through a WAN;
when the terminal equipment downloads the storage file to the cloud server, the terminal equipment is connected to the storage gateway and the object storage equipment through an external access network, and the storage gateway and the object storage equipment are connected to the remote backup server through an internal storage network.
Further, when the terminal device uploads the file to be stored to the cloud server, the terminal device sends self identification information to the cloud server, the cloud server returns a matching result, if the matching is successful, the next operation is performed, and if the matching is failed, the operation cannot be stopped;
the terminal equipment selects a file to be stored of a user, and simultaneously selects an encryption algorithm from an encryption algorithm library of the cloud server according to the matching success message of the cloud server;
the method comprises the steps that terminal equipment sends a file storage request to a cloud server, wherein the file storage request at least comprises a file to be stored, file meta-information and an encryption algorithm, and the file meta-information at least comprises a file name, a file type and a file size.
Further, the cloud server performs file encryption or file decryption through a DES encryption algorithm or an RSA encryption algorithm.
Further, the data storage system stores write-in data through distributed blocks, specifically, the write operation of the terminal device is fragmented into a plurality of 4K blocks and issued to the storage system architecture, the data is written to the NVRAM of the primary storage node corresponding to the corresponding volume in the storage system architecture at first, and then copied to the secondary storage node NVRAM corresponding to the volume in the storage system architecture as a second copy, and after the copy is completed, the cluster directly returns write-in completion information of the terminal device.
Further, when the data storage system reads data, when the data reading fails, the system judges the error type, if the data reading is wrong in the disk sector, the system can automatically read the data from the copy stored by other nodes, and rewrite the copy data to the wrong node in the hard disk sector, so that the total number of the data copies is not reduced, and the data consistency between the copies is ensured.
In summary, the embodiment of the invention has the following beneficial effects:
the distributed edge computing data storage system provided by the invention adopts an expandable system structure, utilizes a plurality of storage servers to share the storage load, and utilizes the position server to position the storage information, thereby not only improving the reliability, the availability and the access efficiency of the system, but also being easy to expand. The storage space with the maximum of more than 100PB has excellent performance; the interface is elastically expanded and enriched; simple management, unified view and unified management. The distributed edge computing data storage system is a result of combination of database technology and network technology, and in the big data era, the increase of data types and quantity makes the distributed database become a core technology for data storage and processing. Data is stored on multiple computers, and distributed database operations are not limited to a single machine, but allow transactional transactions to be performed on multiple machines, thereby improving the performance of database access. The transactions originally performed by the core nodes are divided into edge nodes for processing and are arranged close to the end users, so that the system is more efficient and easy to manage. The edge node is closer to the user terminal device, so that the data processing and transmitting speed can be increased and the delay can be reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is within the scope of the present invention for those skilled in the art to obtain other drawings based on the drawings without inventive exercise.
FIG. 1 is a block diagram of a distributed edge computing data storage system provided by the present invention.
Fig. 2 is a schematic diagram of a network structure of an embodiment of the distributed edge computing data storage system provided in the present invention.
Fig. 3 is a schematic flow chart of a method for uploading a file to be stored in the distributed edge computing data storage system according to the present invention.
Fig. 4 is a schematic flowchart of a method for downloading a storage file in a distributed edge computing data storage system according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail with reference to the accompanying drawings.
Fig. 1 is a schematic diagram of an embodiment of a distributed edge computing data storage system according to the present invention. In this embodiment, a distributed edge computing data storage system includes:
the system comprises distributed edge computing equipment for processing and storing data, terminal equipment for providing data for the distributed edge computing equipment and a data storage system;
in a specific embodiment, the distributed edge computing device at least includes an edge distributed computing node, configured to receive resources of an edge computing API routing center node, compute service scheduling, micro-service software distribution package pushing, distributed data resource storage, compute or store service response; the downlink data of the edge in the edge computing device represents cloud services, the uplink data represents internet of everything services, and the edge of the edge computing device refers to any computing and network resource between paths from a data source to a cloud computing center.
In a specific embodiment, the terminal device at least comprises a terminal device data operation framework, a cloud computing system and an edge computing API routing center node; the cloud computing system consists of a large number of cloud servers and is used for providing services for a large number of users; the edge computing API routing center node is used for receiving network access registration data of the edge computing equipment; the cloud server stores data in a distributed storage mode and is used for storing the data in a redundant storage mode; the cloud server encrypts or decrypts the file through a DES encryption algorithm or an RSA encryption algorithm; the cloud server can be a Tencent cloud server, a Baidu cloud server, an Ali cloud server or Huacheng cloud servers and other cloud servers.
Specifically, as shown in fig. 2, when the terminal device uploads a file to be stored to the cloud server, the terminal device (S3/WebDAV/Client, CIFS/NFS-Client, iSCSI/FC-Initiator) is connected to a storage Gateway (GW) and an Object Storage Device (OSD) through an external access network, the storage Gateway (GW) and the Object Storage Device (OSD) are connected to a remote backup server (RRS) through an internal storage network, and the remote backup server (RRS) is connected to the cloud server through a WAN; when the terminal device (S3/WebDAV/Client, CIFS/NFS-Client, iSCSI/FC-Initiator) downloads the storage file to the cloud server, the terminal device is connected to a storage Gateway (GW) and an Object Storage Device (OSD) through an external access network, and the storage Gateway (GW) and the Object Storage Device (OSD) are connected to a remote backup server (RRS) through an internal storage network.
Specifically, as shown in fig. 3, when the terminal device uploads a file to be stored to the cloud server, the terminal device sends its own identification information to the cloud server, the cloud server returns a matching result, if the matching is successful, the next operation is performed, and if the matching is failed, the operation cannot be stopped; the terminal equipment selects a file to be stored of a user, and simultaneously selects an encryption algorithm from an encryption algorithm library of the cloud server according to the matching success message of the cloud server; the method comprises the steps that terminal equipment sends a file storage request to a cloud server, wherein the file storage request at least comprises a file to be stored, file meta-information and an encryption algorithm, and the file meta-information at least comprises a file name, a file type and a file size.
Specifically, as shown in fig. 4, when the terminal device downloads the storage file to the cloud server, the terminal device matches the cloud server, sends its identification information to the cloud server, waits for the cloud server to give a matching result, performs the next operation if the matching is successful, and cannot continue to operate if the matching is failed; the method comprises the steps that the terminal equipment obtains file meta information of a file to be downloaded from a user; the terminal equipment sends a downloading request to the cloud server, wherein the downloading request comprises file meta information of a file to be downloaded.
Wherein the data storage system comprises at least an object storage model and a storage system architecture;
the object storage model is used for dividing user data into a group of small objects on the bottom layer, the group of small objects are uniformly and uniformly distributed on each equipment node of the whole cluster according to a certain algorithm, and when the user uses the storage model, the small objects are spliced into complete user data by a storage system and are provided for the user to use;
the storage system architecture at least comprises a client, a storage interface, storage management, storage distribution, data storage and physical equipment; the storage interface comprises AmazonS3, WebDAV, NFS, iSCSI, OpenStack-Swift, CIFS and FC/FCoE; the storage management comprises policy management, monitoring/reporting and automatic repair; the storage distribution comprises object distribution, object replication and object rebalancing; the data storage comprises data deduplication, data compression and SSD acceleration.
Specifically, the data storage system stores write-in data through distributed blocks, specifically, the write operation of the terminal device is fragmented into a plurality of 4K blocks and issued to the storage system architecture, the data is firstly written to an NVRAM of a primary storage node corresponding to a corresponding volume in the storage system architecture, and then copied to a secondary storage node NVRAM corresponding to the volume in the storage system architecture as a second copy, and after the copy is completed, the cluster directly returns write-in completion information of the terminal device; after the write-in of the return terminal equipment is completed, the storage node in the storage system architecture also performs a data falling operation of flushing the data in the NVRAM into a rear-end SSD hard disk, and finally the data storage process is completed; the data storage system is provided with a read repair mechanism, when data reading fails, the system judges the error type, if the data reading is wrong in a disk sector, the system can automatically read data from the copy stored by other nodes, and rewrite the copy data to the wrong node of the hard disk sector, so that the total number of data copies is not reduced, and the data consistency among the copies is ensured; in the read repair mechanism, the number of copies of N-data; w-update data is the number of nodes that need to be guaranteed to write; r-the number of nodes to be read when reading data; if W + R > N, the written node and the read node are overlapped, then the consistency is strong; for example, for a typical one-master-one-backup synchronous replication distributed storage system, N is 2, W is 2, and R is 1, the data read from the master or slave is consistent. If W + R < ═ N, weak consistency is achieved; for example, for a primary-secondary asynchronous replicated distributed storage, N is 2, W is 1, and R is 1, if the read is a secondary replica, the data that has been updated by the primary replica may not be read, and thus the dirty data is read and therefore weakly consistent; for distributed storage systems, in order to ensure high availability, N > is generally set to 3, and a read at the primary copy is forced, and a distributed storage system, which is also generally called, uses a strong consistency principle.
The central idea of the edge calculation in the invention is to divide the transaction processing originally performed by the core node into each edge node for processing and to arrange close to the terminal user, thereby making the system more efficient and easy to manage. Because the edge node is closer to the user terminal device, the processing and transmitting speed of the data can be accelerated, and the delay is reduced; the distributed edge computing data storage system is a result of combination of database technology and network technology, and in the big data era, the increase of data types and quantity makes the distributed database become a core technology for data storage and processing. Distributed database operations are not limited to a single machine, but allow transactional transactions to be performed on multiple machines, thereby improving the performance of database access.
Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
In summary, the embodiment of the invention has the following beneficial effects:
the distributed edge computing data storage system provided by the invention adopts an expandable system structure, utilizes a plurality of storage servers to share the storage load, and utilizes the position server to position the storage information, thereby not only improving the reliability, the availability and the access efficiency of the system, but also being easy to expand. The storage space with the maximum of more than 100PB has excellent performance; the interface is elastically expanded and enriched; simple management, unified view and unified management. The distributed edge computing data storage system is a result of combination of database technology and network technology, and in the big data era, the increase of data types and quantity makes the distributed database become a core technology for data storage and processing. Data is stored on multiple computers, and distributed database operations are not limited to a single machine, but allow transactional transactions to be performed on multiple machines, thereby improving the performance of database access. The transactions originally performed by the core nodes are divided into edge nodes for processing and are arranged close to the end users, so that the system is more efficient and easy to manage. The edge node is closer to the user terminal device, so that the data processing and transmitting speed can be increased and the delay can be reduced.
While the invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not to be limited to the disclosed embodiment, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (9)
1. A distributed edge computing data storage system, comprising: the system comprises distributed edge computing equipment for processing and storing data, terminal equipment for providing data for the distributed edge computing equipment and a data storage system;
wherein the data storage system comprises at least an object storage model and a storage system architecture;
the object storage model is used for dividing user data into a group of small objects on the bottom layer, the group of small objects are uniformly and uniformly distributed on each equipment node of the whole cluster according to a certain algorithm, and when the user uses the storage model, the small objects are spliced into complete user data by a storage system and are provided for the user to use;
the storage system architecture at least comprises a client, a storage interface, storage management, storage distribution, data storage and physical equipment.
2. The system of claim 1, wherein the distributed edge computing device comprises at least an edge distributed computing node to receive resources of an edge computing API route center node, compute service scheduling, micro-service software distribution package pushing, distributed data resource deposit, compute, or store service responses.
3. The system of claim 2, wherein the edge of the edge computing device is any computing and network resource between a data source and a cloud computing center path, the downstream data of the edge in the edge computing device is cloud services, and the upstream data of the edge in the edge computing device is internet of everything services.
4. The system of claim 1, wherein the end device comprises at least an end device data manipulation framework, a cloud computing system, and an edge computing API routing hub node; the cloud computing system consists of a large number of cloud servers and is used for providing services for a large number of users; the edge computing API routing center node is used for receiving network access registration data of the edge computing equipment; the cloud server stores data in a distributed storage mode and is used for storing the data in a redundant storage mode.
5. The system of claim 4, wherein when the terminal device uploads the file to be stored to the cloud server, the terminal device is connected to a storage gateway and an object storage device through an external access network, the storage gateway and the object storage device are connected to a remote backup server through an internal storage network, and the remote backup server is connected to the cloud server through a WAN;
when the terminal equipment downloads the storage file to the cloud server, the terminal equipment is connected to the storage gateway and the object storage equipment through an external access network, and the storage gateway and the object storage equipment are connected to the remote backup server through an internal storage network.
6. The system of claim 5, wherein when the terminal device uploads the file to be stored to the cloud server, the terminal device sends its own identification information to the cloud server, the cloud server returns a matching result, if the matching is successful, the next operation is performed, and if the matching is unsuccessful, the operation cannot be stopped;
the terminal equipment selects a file to be stored of a user, and simultaneously selects an encryption algorithm from an encryption algorithm library of the cloud server according to the matching success message of the cloud server;
the method comprises the steps that terminal equipment sends a file storage request to a cloud server, wherein the file storage request at least comprises a file to be stored, file meta-information and an encryption algorithm, and the file meta-information at least comprises a file name, a file type and a file size.
7. The method of claim 6, wherein the cloud server performs file encryption or file decryption by a DES encryption algorithm or a RSA encryption algorithm.
8. The system of claim 1, wherein the data storage system stores write-in data through distributed blocks, and specifically, the write-in operation of the terminal device is fragmented into a plurality of blocks of 4K and issued to the storage system architecture, the data is first written to the NVRAM of the primary storage node corresponding to the corresponding volume in the storage system architecture, and then copied to the secondary storage node NVRAM corresponding to the volume in the storage system architecture as a second copy, and after the copying is completed, the cluster directly returns write-in completion information of the terminal device.
9. The method as claimed in claim 8, wherein when the data storage system reads data, the system determines the type of error when the data reading fails, and if the data reading is a disk sector reading error, the system will automatically read data from the copy stored in other nodes, and rewrite the copy data to the node with the disk sector error, so as to ensure that the total number of data copies is not reduced and the data consistency between the copies is ensured.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010107732.6A CN111339192A (en) | 2020-02-21 | 2020-02-21 | Distributed edge computing data storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010107732.6A CN111339192A (en) | 2020-02-21 | 2020-02-21 | Distributed edge computing data storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111339192A true CN111339192A (en) | 2020-06-26 |
Family
ID=71185477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010107732.6A Pending CN111339192A (en) | 2020-02-21 | 2020-02-21 | Distributed edge computing data storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111339192A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108121511A (en) * | 2017-12-21 | 2018-06-05 | 北京黑螺技术有限公司 | Data processing method, device and equipment in a kind of distributed edge storage system |
| CN111949217A (en) * | 2020-08-21 | 2020-11-17 | 广东韶钢松山股份有限公司 | Super-fusion all-in-one machine and software definition storage SDS processing method and system thereof |
| CN112600924A (en) * | 2020-12-15 | 2021-04-02 | 四川长虹电器股份有限公司 | IoT terminal data storage system, method, computer device and storage medium |
| CN112671847A (en) * | 2020-12-11 | 2021-04-16 | 周口师范学院 | Content storage and transmission joint planning method based on heterogeneous wireless cellular network |
| CN113391647A (en) * | 2021-07-20 | 2021-09-14 | 中国人民解放军国防科技大学 | Multi-unmanned aerial vehicle edge computing service deployment and scheduling method and system |
| CN114442912A (en) * | 2020-10-30 | 2022-05-06 | 潘塔安全系统公司 | Method and apparatus for distributed data storage |
| CN114500073A (en) * | 2022-02-11 | 2022-05-13 | 浪潮云信息技术股份公司 | User data cut-over method and system supporting privacy protection in cloud storage system |
| CN114584397A (en) * | 2022-04-28 | 2022-06-03 | 睿至科技集团有限公司 | Distributed cloud data centralized processing method and system |
| WO2023093079A1 (en) * | 2021-11-26 | 2023-06-01 | 浪潮通信信息系统有限公司 | Consistency check method and apparatus for distributed edge cloud edge nodes |
| CN116974827A (en) * | 2023-09-25 | 2023-10-31 | 北京装库创意科技有限公司 | Cloud big data storage management method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102136003A (en) * | 2011-03-25 | 2011-07-27 | 上海交通大学 | Large-scale distributed storage system |
| CN102307221A (en) * | 2011-03-25 | 2012-01-04 | 国云科技股份有限公司 | Cloud storage system and implementation method thereof |
| US20140181021A1 (en) * | 2012-12-21 | 2014-06-26 | Zetta, Inc. | Back up using locally distributed change detection |
| CN106657267A (en) * | 2016-11-15 | 2017-05-10 | 华中科技大学 | Cloud storage system based on edge server |
| CN109542457A (en) * | 2018-11-21 | 2019-03-29 | 四川长虹电器股份有限公司 | A kind of system and method for the Distributed Application distribution deployment of edge calculations network |
| CN109815371A (en) * | 2019-01-30 | 2019-05-28 | 无锡华云数据技术服务有限公司 | Video monitoring data stores access system |
| CN109887098A (en) * | 2019-02-13 | 2019-06-14 | 浙江传媒学院 | A kind of web AR data presentation mode based on distributed computing |
-
2020
- 2020-02-21 CN CN202010107732.6A patent/CN111339192A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102136003A (en) * | 2011-03-25 | 2011-07-27 | 上海交通大学 | Large-scale distributed storage system |
| CN102307221A (en) * | 2011-03-25 | 2012-01-04 | 国云科技股份有限公司 | Cloud storage system and implementation method thereof |
| US20140181021A1 (en) * | 2012-12-21 | 2014-06-26 | Zetta, Inc. | Back up using locally distributed change detection |
| CN106657267A (en) * | 2016-11-15 | 2017-05-10 | 华中科技大学 | Cloud storage system based on edge server |
| CN109542457A (en) * | 2018-11-21 | 2019-03-29 | 四川长虹电器股份有限公司 | A kind of system and method for the Distributed Application distribution deployment of edge calculations network |
| CN109815371A (en) * | 2019-01-30 | 2019-05-28 | 无锡华云数据技术服务有限公司 | Video monitoring data stores access system |
| CN109887098A (en) * | 2019-02-13 | 2019-06-14 | 浙江传媒学院 | A kind of web AR data presentation mode based on distributed computing |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108121511A (en) * | 2017-12-21 | 2018-06-05 | 北京黑螺技术有限公司 | Data processing method, device and equipment in a kind of distributed edge storage system |
| CN111949217A (en) * | 2020-08-21 | 2020-11-17 | 广东韶钢松山股份有限公司 | Super-fusion all-in-one machine and software definition storage SDS processing method and system thereof |
| CN114442912A (en) * | 2020-10-30 | 2022-05-06 | 潘塔安全系统公司 | Method and apparatus for distributed data storage |
| CN114442912B (en) * | 2020-10-30 | 2024-03-29 | 潘塔安全系统公司 | Method and apparatus for distributed data storage |
| CN112671847A (en) * | 2020-12-11 | 2021-04-16 | 周口师范学院 | Content storage and transmission joint planning method based on heterogeneous wireless cellular network |
| CN112600924B (en) * | 2020-12-15 | 2022-02-22 | 四川长虹电器股份有限公司 | IoT terminal data storage system, method, computer device and storage medium |
| CN112600924A (en) * | 2020-12-15 | 2021-04-02 | 四川长虹电器股份有限公司 | IoT terminal data storage system, method, computer device and storage medium |
| CN113391647A (en) * | 2021-07-20 | 2021-09-14 | 中国人民解放军国防科技大学 | Multi-unmanned aerial vehicle edge computing service deployment and scheduling method and system |
| WO2023093079A1 (en) * | 2021-11-26 | 2023-06-01 | 浪潮通信信息系统有限公司 | Consistency check method and apparatus for distributed edge cloud edge nodes |
| CN114500073A (en) * | 2022-02-11 | 2022-05-13 | 浪潮云信息技术股份公司 | User data cut-over method and system supporting privacy protection in cloud storage system |
| CN114500073B (en) * | 2022-02-11 | 2024-04-12 | 浪潮云信息技术股份公司 | User data cutting method and system supporting privacy protection in cloud storage system |
| CN114584397A (en) * | 2022-04-28 | 2022-06-03 | 睿至科技集团有限公司 | Distributed cloud data centralized processing method and system |
| CN114584397B (en) * | 2022-04-28 | 2022-07-29 | 睿至科技集团有限公司 | Distributed cloud data centralized processing method and system |
| CN116974827A (en) * | 2023-09-25 | 2023-10-31 | 北京装库创意科技有限公司 | Cloud big data storage management method and system |
| CN116974827B (en) * | 2023-09-25 | 2024-01-16 | 北京装库创意科技有限公司 | Cloud big data storage management method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111339192A (en) | Distributed edge computing data storage system | |
| US9983825B2 (en) | Efficient data volume replication for block-based storage | |
| US9792060B2 (en) | Optimized write performance at block-based storage during volume snapshot operations | |
| US10719260B2 (en) | Techniques for storing and retrieving data from a computing device | |
| CA2940328C (en) | Reducing data volume durability state for block-based storage | |
| CN108280080B (en) | Data synchronization method and device and electronic equipment | |
| US20230087447A1 (en) | Data migration method and device | |
| CN107832423B (en) | File reading and writing method for distributed file system | |
| CN114201421B (en) | Data stream processing method, storage control node and readable storage medium | |
| CN104735110A (en) | Metadata management method and system | |
| US9083725B2 (en) | System and method providing hierarchical cache for big data applications | |
| CN106873902B (en) | File storage system, data scheduling method and data node | |
| CN113411363A (en) | Uploading method of image file, related equipment and computer storage medium | |
| CN107798063A (en) | Snap processing method and snapshot processing unit | |
| CN110784498B (en) | Personalized data disaster tolerance method and device | |
| EP3349416B1 (en) | Relationship chain processing method and system, and storage medium | |
| CN109992447B (en) | Data copying method, device and storage medium | |
| WO2024021470A1 (en) | Cross-region data scheduling method and apparatus, device, and storage medium | |
| CN115695049A (en) | CDN metadata distribution method and device | |
| WO2022121387A1 (en) | Data storage method and apparatus, server, and medium | |
| CN115238006A (en) | Retrieval data synchronization method, device, equipment and computer storage medium | |
| CN115695587A (en) | Service data processing system, method, device and storage medium | |
| CN105446651B (en) | Memory mirror method and system and data sending device and data sink | |
| CN116760850B (en) | Data processing method, device, equipment, medium and system | |
| WO2023109554A1 (en) | Data processing method for distributed system, and system, node and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200626 |