CN117176744A - Internet of things real-time data layered storage system and method based on distributed digital base - Google Patents
Internet of things real-time data layered storage system and method based on distributed digital base Download PDFInfo
- Publication number
- CN117176744A CN117176744A CN202311072854.6A CN202311072854A CN117176744A CN 117176744 A CN117176744 A CN 117176744A CN 202311072854 A CN202311072854 A CN 202311072854A CN 117176744 A CN117176744 A CN 117176744A
- Authority
- CN
- China
- Prior art keywords
- data
- data center
- slave
- center
- internet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000011084 recovery Methods 0.000 claims abstract description 16
- 230000001360 synchronised effect Effects 0.000 claims description 23
- 238000013500 data storage Methods 0.000 claims description 14
- 230000003068 static effect Effects 0.000 claims description 14
- 238000009434 installation Methods 0.000 claims description 6
- 238000007689 inspection Methods 0.000 claims 1
- 230000006855 networking Effects 0.000 claims 1
- 238000005265 energy consumption Methods 0.000 abstract description 5
- 239000010410 layer Substances 0.000 description 44
- 230000005540 biological transmission Effects 0.000 description 12
- 238000004891 communication Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 6
- 230000004044 response Effects 0.000 description 5
- 238000007405 data analysis Methods 0.000 description 3
- 239000007787 solid Substances 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005538 encapsulation Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002346 layers by function Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides an Internet of things real-time data layered storage system and method based on a distributed digital base, comprising the following steps: a master data center and a plurality of slave data centers; and uploading the data of the Internet of things equipment to the nearest slave data center, and storing the total data of the master data center and backing up the data of the slave data center through a data synchronization strategy. The method and the device can improve the rate of reporting the equipment data in the urban level internet of things application; the energy consumption of uploading equipment data in the urban cascade application is reduced; the efficiency of reading the data of the client is improved; the security of the data center is improved, and the disaster recovery capability of the data is improved.
Description
Technical Field
The application relates to the technical field of Internet of things engineering, in particular to an Internet of things real-time data layered storage system and method based on a distributed digital base.
Background
Along with the development of the Internet of things engineering and the promotion of the digital transformation process of the smart city, a large number of Internet of things sensing devices are deployed at all corners of the city and used for collecting multidimensional data, and urban-level and even urban-group-level Internet of things application becomes an industry trend. The application relates to tens of millions of or even hundreds of millions of Internet of things sensing devices, the physical address distribution area of the devices relates to large range span, and how to efficiently manage real-time data generated by the Internet of things devices and the Internet of things devices, so that the problems of high data quality and low delay are solved.
The data reported by the Internet of things equipment has the characteristics of mass, time sequence, diversity and real-time performance, the traditional database storage is difficult to meet the requirements of reading and writing and efficient management, and a data center needs to be constructed for unified management of the data. Meanwhile, for the application of the Internet of things of the city group level, the distribution range of equipment is wide, and the average distance from the equipment to the data center is very large. A large distance causes two problems: the terminal equipment has large energy consumption and large data transmission delay. If a traditional single data center is used for receiving data, the overall performance is greatly affected.
In order to solve the problem that in the application of the urban mass internet of things, a single data center cannot efficiently manage equipment data, the application provides a layered storage solution based on a distributed digital base, and solves the problem that the distance between equipment and the data center is long by setting a mode of a master data center and a plurality of slave data centers serving as nodes for data receiving and independent storage; meanwhile, a data synchronization queue mode is designed, so that data mutual backup of the slave data center and data synchronization of the master data center are realized.
Patent document CN112947848A (application No. 202010080358.5) discloses a low-cost and high-performance data center, which adopts layered heterogeneous storage, and includes at least three groups of servers, and three types of data with different corresponding access frequencies are provided for three storage media. The three groups of servers are connected to each other by a network. The group server is operated in a pseudo single-level storage unit mode by using a solid state disk constructed by a plurality of level storage units as a storage medium. The patent is suitable for a low-cost small-sized data center, and can ensure that data with high frequency access can be responded quickly by adopting different storage media for data with different access frequencies through layered storage. The application sets up master-slave data center to access and plan data according to the layering mode of the whole structure for the large-scale data center with wide coverage area.
Patent document CN109951370B (application number: 201711394815.2) discloses a hierarchical interconnection method and device for multiple data centers, wherein the method comprises: session management, data analysis and encapsulation, state monitoring and metadata management are carried out through a communication protocol of the data analysis layer; session management, data analysis and encapsulation, cluster monitoring and task metadata management are carried out through a communication protocol of a computing engine layer; performing cross-cluster scheduling of tasks through a communication protocol of a resource management layer; and performing cross-domain reading and writing of the remote big data clusters through a communication protocol of the file storage layer so as to remotely read data in file storage of other big data clusters. The patent divides the data center into functional layers which are not communicated, and establishes a certain communication rule to ensure the communication between different layers, thereby realizing the cross-domain communication and resource sharing of the data center. The application mainly aims to solve the problem that the Internet of things transmission protocol cannot efficiently and rapidly transmit data to the data center because of overlarge geographic range coverage, and sets a plurality of slave data centers for receiving equipment data in a certain range according to the geographic range.
Patent document CN116303766a (application number 202310127621.5) discloses a data synchronization method, a device, a storage medium and computer equipment, and relates to the technical field of data synchronization. According to the method, through related technologies such as data synchronization, different copying schemes are selected from clusters to cross-domain data centers according to different service characteristics, a synchronization strategy is adopted, the operation mode of the multi-activity data center is a premise and a basis for researching remote data synchronization among the multi-data centers, various multi-data center construction modes are provided by analyzing the construction mode of the existing multi-data center, the multi-data center operation mode of a bank financial service information system is constructed, the synchronization of heterogeneous databases can not be well supported aiming at the existing remote data synchronization efficiency, corresponding SQL is captured and sent to the heterogeneous databases to be executed through setting a capturer in a driving program, the mechanism optimizes the existing data synchronization flow, reduces the performance influence caused by network delay, and further effectively supports the synchronization among the heterogeneous databases. The patent sends the sql statement to the corresponding offsite database for execution by setting the sql capturer, and then returns the execution result. By the method, the overhead of data transmission is reduced. The method for synchronizing data in the application uses a synchronous queue to carry out batch synchronization, and by setting certain batch conditions (such as time, data quantity and data number), when the data accumulation reaches the conditions, the synchronous operation is triggered, which is essentially different from the method for searching patents.
Disclosure of Invention
Aiming at the defects in the prior art, the application aims to provide an Internet of things real-time data layered storage system and method based on a distributed digital base.
The application provides an Internet of things real-time data layered storage system based on a distributed digital base, which comprises the following components: a master data center and a plurality of slave data centers;
and uploading the data of the Internet of things equipment to the nearest slave data center, and storing the total data of the master data center and backing up the data of the slave data center through a data synchronization strategy.
Preferably, the master data center is a center point position of a polygon composed by taking each slave data center as a node; the distances between the master data center and each slave data center are basically equal; the sum of the distances of the respective slave data centers and the master data center is minimized.
Preferably, the primary data center stores full data and the data backup of the data center adopts: carrying out data synchronization between the data centers in a synchronous queue mode;
the master data center is used as a full-volume data storage center, and data of the slave data centers are synchronized in real time;
and the slave data centers are mutually provided with data synchronization according to the nearest neighbor non-synchronization principle, and for one slave data center, the neighbor node which is nearest to and is not used as the other slave data center backup is searched and used as the object of data backup.
Preferably, a buffer layer is arranged at the data center node;
the cache layer is a layer structure above the original data storage layer of the data center, and the hardware adopts solid-state hardware, so that the high-efficiency reading and writing performance is improved; the caching layer is used for caching static resource data and hot data;
the static resource data comprises equipment basic information and equipment installation address information;
the hot data is the data with highest access frequency, and the replacement strategy is the latest and longest non-access strategy; for hot data of the cache region, the latest access time of each group of data is marked, and the latest and longest non-access strategy is used for replacing new data.
Preferably, each data center provides an external data receiving inlet, and the internet of things device directly registers with the nearest slave data center and uploads data.
Preferably, when a user initiates a data access request, judging the data center where the requested resource is located, and selecting one data center closest to the requested place in physical distance to provide service; when the request arrives at the data center, whether the cache layer can hit or not is preferentially searched, if so, the data is directly returned, and if not, the data is searched for the storage layer data and returned to the user.
Preferably, when the main data center fails, firstly searching for the condition of lost data by a global data checking mode; then, positioning the slave data center where the lost data is located, and synchronizing the data in the slave data center to the master data center when the master data center has a data recovery condition; when the slave data center fails, the backup slave data center is searched through the backup information table, and when the slave data center has the data recovery condition, the slave backup slave data center recovers the data to the slave data center.
The application provides a real-time data layering storage method of the Internet of things based on a distributed digital base, which comprises the following steps:
step S1: uploading the data of the Internet of things equipment to a slave data center closest to the Internet of things equipment;
step S2: and storing the total data of the main data center and backing up the data of the sub data center through a data synchronization strategy.
Preferably, the master data center is a center point position of a polygon composed by taking each slave data center as a node; the distances between the master data center and each slave data center are basically equal; the sum of the distances between each slave data center and the master data center is minimum;
the main data center stores full data and the data backup of the data center adopts the following steps: carrying out data synchronization between the data centers in a synchronous queue mode;
the master data center is used as a full-volume data storage center, and data of the slave data centers are synchronized in real time;
and the slave data centers are mutually provided with data synchronization according to the nearest neighbor non-synchronization principle, and for one slave data center, the neighbor node which is nearest to and is not used as the other slave data center backup is searched and used as the object of data backup.
Preferably, a buffer layer is arranged at the data center node;
the cache layer is a layer structure above the original data storage layer of the data center, and the hardware adopts solid-state hardware, so that the high-efficiency reading and writing performance is improved; the caching layer is used for caching static resource data and hot data;
the static resource data comprises equipment basic information and equipment installation address information;
the hot data is the data with highest access frequency, and the replacement strategy is the latest and longest non-access strategy; for hot data of the cache region, marking the latest access time of each group of data, and replacing new data by using the latest longest non-access strategy;
each data center provides an external data receiving inlet, and the Internet of things equipment directly registers the nearest slave data center and uploads data;
when a user initiates a data access request, judging the data center where the requested resource is located, and selecting one data center closest to the requested place in physical distance to provide service; when the request arrives at the data center, preferentially searching whether the cache layer can hit, if so, directly returning the data, and if not, searching the storage layer data and returning the data to the user;
when the main data center fails, firstly searching for the condition of lost data in a global data checking mode; then, positioning the slave data center where the lost data is located, and synchronizing the data in the slave data center to the master data center when the master data center has a data recovery condition; when the slave data center fails, the backup slave data center is searched through the backup information table, and when the slave data center has the data recovery condition, the slave backup slave data center recovers the data to the slave data center.
Compared with the prior art, the application has the following beneficial effects:
1. the application can improve the efficiency of data transmission of the equipment and increase the safety of data storage of the system; meanwhile, the response efficiency of the data is improved through the design of a latest access strategy from the data center and an efficient cache layer;
2. the method solves the problems of wide application geographic range coverage, high information transmission energy consumption and low efficiency of the Internet of things equipment of the city group level by arranging a plurality of slave data centers; meanwhile, the data is synchronized and backed up in a mode of synchronizing the queues and batch synchronization, so that the redundancy and safety of the data are ensured;
3. in order to improve the access efficiency of the data center, the application reduces the delay of data inquiry and data transmission by selecting the data center and the high-efficiency cache layer which are closest in geographic position and meet the conditions;
4. the method and the device can improve the rate of reporting the equipment data in the urban level internet of things application; the energy consumption of uploading equipment data in the urban cascade application is reduced; the efficiency of reading the data of the client is improved; the security of the data center is improved, and the disaster recovery capability of the data is improved.
Drawings
Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
fig. 1 is a schematic diagram of a hierarchy of a data center.
Fig. 2 is a flow chart of a data center write strategy.
FIG. 3 is a flow chart of a data center response strategy.
Detailed Description
The present application will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present application, but are not intended to limit the application in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present application.
The application of the urban mass internet of things has wide coverage geographical range and large number of cities, and if only a single data center is arranged, the data uploading of the internet of things equipment is performed because the internet of things equipment is far away from the data center, and the energy consumption and the delay are high during data transmission. The application provides a distributed digital base-based real-time data hierarchical storage system and a distributed digital base-based real-time data hierarchical storage method for the Internet of things, which are used for improving the efficiency of equipment data transmission and the safety of system data storage in the application of the Internet of things of an urban group level; meanwhile, the data response efficiency is improved through the design of the access strategy of the data center and the high-efficiency cache layer recently.
Example 1
According to the application, as shown in fig. 1 to 3, the internet of things real-time data layered storage system based on the distributed digital base comprises: setting a master data center and a plurality of slave data centers, uploading equipment data to the data center closest to the master data center, and realizing the data backup of the master data center for storing the total data and the multiple data centers through a data synchronization strategy; setting a high-efficiency cache layer at a data center node, and caching static data and hot data with high access frequency; when providing data service, the request of user application data is provided by the server which can meet the condition and is closest to the condition, and the search of the cache data is preferentially carried out, the cache data is not hit, and then the search of the data in the data center is carried out.
More specifically, the setting a master multi-slave data center adopts:
the rule set from the data center is as follows: the application of the urban Internet of things is based on geographic positions, and a data center is arranged according to a spatial range. Taking the delay of data transmission into consideration, planning a data center according to the principle that a slave data center is arranged for 100 kilometers on a square circle.
The main data center setting rule adopts: the main data center is arranged at the central position of the whole city group area, namely, the central point of the polygon formed by taking each slave data center as a node. In this way, the distances between the master data center and the respective slave data centers are substantially equal, and the sum of the distances between the slave data centers and the master data center is minimized, i.e., the overall distance of data transmission is minimized.
More specifically, the data synchronization strategy is used for realizing the storage of the full data of the main data center and the data backup of the sub data center, and the following steps are adopted:
the synchronous mutual backup of the data mainly comprises the full-scale synchronization of the master data center and the mutual backup of the slave data center, and under the synchronization rule, each data realizes three backups of the original slave data center S1, the backup slave data center S2 and the master data center S, thereby ensuring the redundancy and the safety of the data.
The master data center is used as a full-scale data storage center of the whole system, and the data of the slave data centers need to be synchronized in real time. And carrying out data synchronization according to a nearest neighbor non-synchronization principle from the data center data mutual backup, namely searching neighbor nodes which are nearest to one data center and are not backed up as other data centers as data backup objects.
The data synchronization between the data centers is performed in a synchronous queue mode, and specifically comprises the following steps:
module 1: when the data arrives at the slave data center S1, simultaneously entering a synchronous queue;
module 2: synchronizing data in the queue for batch synchronization, wherein three modes of data synchronization according to a certain time interval, reaching a specified data quantity level and reaching a specified data number can be set;
module 3: the data in the message queue is firstly processed to a certain extent, and multiple modifications of the same data are combined into one piece, for example, one field str in a certain record is provided with two modification records, one is increased by 3, one is increased by 5, and the field value can be combined to be increased by 8;
module 4: and synchronizing the data to the master data center, backing up the data to the backup slave data center, and deleting the data in the synchronous queue after the two-step synchronization is completed.
Specifically, the setting of the high-efficiency cache layer at the data center node adopts: the data center node adopts layered storage and comprises a data storage layer and a high-efficiency cache layer;
the data storage layer:
the data center consists of a plurality of data storage nodes, the storage hardware adopts a mechanical hard disk, and full data accessed into the data center is stored;
the high-efficiency cache layer:
the buffer memory area is a layer of structure above the original data storage layer of the data center, and the hardware adopts a solid state disk to provide high-efficiency reading and writing performance. The caching layer is mainly used for caching static resource data and hot data.
The static resource data includes device basic information, device installation address information, and the like.
The hot data is the data with highest access frequency, and the replacement policy is the latest and longest non-access policy. For hot data of the cache region, the latest access time of each group of data is marked, and new data is replaced by using the latest and longest access strategy.
Specifically, the data writing rule employs:
each data center provides an external data receiving inlet, and the equipment directly registers with the nearest data center and uploads the data.
Specifically, the data response rule employs:
when a user initiates a data access request, it is determined which of the data centers the requested resource is located in, and three data centers meeting the conditions are generally available: a master data center S for storing the total data, a slave data center S1 for data access and a data center S2 for data backup; then, one service is selected from among 3 data centers satisfying the condition, which is closest to the request place in physical distance. When the request arrives at the data center, whether the cache layer can hit or not is preferentially searched, if so, the data is directly returned, and if not, the data is searched for the storage layer data and returned to the user.
Specifically, the data disaster recovery adopts:
when the main data center S fails, firstly searching for the condition of lost data in a global data checking mode; then, the slave data center S1 where the lost data is located, and when the master data center has the data recovery condition, the data in S1 is synchronized to S.
When a failure occurs in the slave data center S1, the backup slave data center S2 is found out by the backup information table, and when S1 has a data recovery condition, the slave data center S2 recovers the data to S1.
Example 2
Example 2 is a preferred example of example 1
The application provides a real-time data layering storage method of the Internet of things based on a distributed digital base, which comprises the following steps: and setting a master data center and a plurality of slave data centers, uploading the equipment data to the data center closest to the master data center, and realizing the data backup of the master data center and the data center through a data synchronization strategy.
Setting a high-efficiency cache layer at a data center node, and caching static data and hot data with high access frequency;
when providing data service, the request of user application data is provided by the server which can meet the condition and is closest to the condition, and the search of the cache data is preferentially carried out, the cache data is not hit, and then the search of the data in the data center is carried out.
More specifically, the setting a master multi-slave data center adopts:
the rule set from the data center is as follows: the application of the urban Internet of things is based on geographic positions, and a data center is arranged according to a spatial range. Taking the delay of data transmission into consideration, planning a data center according to the principle that a slave data center is arranged for 100 kilometers on a square circle.
The main data center setting rule adopts: the main data center is arranged at the central position of the whole city group area, namely, the central point of the polygon formed by taking each slave data center as a node. In this way, the distances between the master data center and the respective slave data centers are substantially equal, and the sum of the distances between the slave data centers and the master data center is minimized, i.e., the overall distance of data transmission is minimized.
More specifically, the data synchronization strategy is used for realizing the storage of the full data of the main data center and the data backup of the sub data center, and the following steps are adopted:
the synchronous mutual backup of the data mainly comprises the full-scale synchronization of the master data center and the mutual backup of the slave data center, and under the synchronization rule, each data realizes three backups of the original slave data center S1, the backup slave data center S2 and the master data center S, thereby ensuring the redundancy and the safety of the data.
The master data center is used as a full-scale data storage center of the whole system, and the data of the slave data centers need to be synchronized in real time. And carrying out data synchronization according to a nearest neighbor non-synchronization principle from the data center data mutual backup, namely searching neighbor nodes which are nearest to one data center and are not backed up as other data centers as data backup objects.
The data synchronization between the data centers is carried out by adopting a synchronous queue mode, and the specific steps are as follows:
step 1: when the data arrives at the slave data center S1, simultaneously entering a synchronous queue;
step 2: synchronizing data in the queue for batch synchronization, wherein three modes of data synchronization according to a certain time interval, reaching a specified data quantity level and reaching a specified data number can be set;
step 3: the data in the message queue is firstly processed to a certain extent, and multiple modifications of the same data are combined into one piece, for example, one field str in a certain record is provided with two modification records, one is increased by 3, one is increased by 5, and the field value can be combined to be increased by 8;
step 4: and synchronizing the data to the master data center, backing up the data to the backup slave data center, and deleting the data in the synchronous queue after the two-step synchronization is completed.
Specifically, the setting of the high-efficiency cache layer at the data center node adopts: the data center node adopts layered storage and comprises a data storage layer and a high-efficiency cache layer;
the data storage layer:
the data center consists of a plurality of data storage nodes, the storage hardware adopts a mechanical hard disk, and full data accessed into the data center is stored;
the high-efficiency cache layer:
the buffer memory area is a layer of structure above the original data storage layer of the data center, and the hardware adopts a solid state disk to provide high-efficiency reading and writing performance. The caching layer is mainly used for caching static resource data and hot data.
The static resource data includes device basic information, device installation address information, and the like.
The hot data is the data with highest access frequency, and the replacement policy is the latest and longest non-access policy. For hot data of the cache region, the latest access time of each group of data is marked, and new data is replaced by using the latest and longest access strategy.
Specifically, the data writing rule employs:
each data center provides an external data receiving inlet, and the equipment directly registers with the nearest data center and uploads the data.
Specifically, the data response rule employs:
when a user initiates a data access request, it is determined which of the data centers the requested resource is located in, and three data centers meeting the conditions are generally available: a master data center S for storing the total data, a slave data center S1 for data access and a data center S2 for data backup; then, one service is selected from among 3 data centers satisfying the condition, which is closest to the request place in physical distance. When the request arrives at the data center, whether the cache layer can hit or not is preferentially searched, if so, the data is directly returned, and if not, the data is searched for the storage layer data and returned to the user.
Specifically, the data disaster recovery adopts:
when the main data center S fails, firstly searching for the condition of lost data in a global data checking mode; then, the slave data center S1 where the lost data is located, and when the master data center has the data recovery condition, the data in S1 is synchronized to S.
When a failure occurs in the slave data center S1, the backup slave data center S2 is found out by the backup information table, and when S1 has a data recovery condition, the slave data center S2 recovers the data to S1.
Those skilled in the art will appreciate that the application provides a system and its individual devices, modules, units, etc. that can be implemented entirely by logic programming of method steps, in addition to being implemented as pure computer readable program code, in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Therefore, the system and various devices, modules and units thereof provided by the application can be regarded as a hardware component, and the devices, modules and units for realizing various functions included in the system can also be regarded as structures in the hardware component; means, modules, and units for implementing the various functions may also be considered as either software modules for implementing the methods or structures within hardware components.
The foregoing describes specific embodiments of the present application. It is to be understood that the application is not limited to the particular embodiments described above, and that various changes or modifications may be made by those skilled in the art within the scope of the appended claims without affecting the spirit of the application. The embodiments of the application and the features of the embodiments may be combined with each other arbitrarily without conflict.
Claims (10)
1. The utility model provides a thing networking real-time data layering memory system based on distributed digital base which characterized in that includes: a master data center and a plurality of slave data centers;
and uploading the data of the Internet of things equipment to the nearest slave data center, and storing the total data of the master data center and backing up the data of the slave data center through a data synchronization strategy.
2. The distributed digital base-based internet of things real-time data hierarchical storage system according to claim 1, wherein the master data center is a center point position of a polygon composed of each slave data center as a node; the distances between the master data center and each slave data center are basically equal; the sum of the distances of the respective slave data centers and the master data center is minimized.
3. The internet of things real-time data hierarchical storage system based on a distributed digital base according to claim 1, wherein the master data center stores full data and the data backup of the data center adopts: carrying out data synchronization between the data centers in a synchronous queue mode;
the master data center is used as a full-volume data storage center, and data of the slave data centers are synchronized in real time;
and the slave data centers are mutually provided with data synchronization according to the nearest neighbor non-synchronization principle, and for one slave data center, the neighbor node which is nearest to and is not used as the other slave data center backup is searched and used as the object of data backup.
4. The distributed digital base-based internet of things real-time data hierarchical storage system according to claim 1, wherein a cache layer is arranged at a data center node;
the cache layer is a layer structure above an original data storage layer of the data center, and the hardware adopts solid-state hardware to provide high-efficiency reading and writing performance; the caching layer is used for caching static resource data and hot data;
the static resource data comprises equipment basic information and equipment installation address information;
the hot data is the data with highest access frequency, and the replacement strategy is the latest and longest non-access strategy; for hot data of the cache region, the latest access time of each group of data is marked, and the latest and longest non-access strategy is used for replacing new data.
5. The internet of things real-time data hierarchical storage system based on the distributed digital base according to claim 1, wherein each data center provides an external data receiving inlet, and the internet of things equipment directly registers with the nearest slave data center and uploads data.
6. The internet of things real-time data hierarchical storage system based on the distributed digital base according to claim 1, wherein when a user initiates a data access request, the data center where the requested resource is located is judged, and one data center closest to the requested location is selected to provide service; when the request arrives at the data center, whether the cache layer can hit or not is preferentially searched, if so, the data is directly returned, and if not, the data is searched for the storage layer data and returned to the user.
7. The internet of things real-time data hierarchical storage system based on the distributed digital base according to claim 1, wherein when the main data center fails, the lost data is searched by means of global data inspection; then, positioning the slave data center where the lost data is located, and synchronizing the data in the slave data center to the master data center when the master data center has a data recovery condition; when the slave data center fails, the backup slave data center is searched through the backup information table, and when the slave data center has the data recovery condition, the slave backup slave data center recovers the data to the slave data center.
8. The Internet of things real-time data layered storage method based on the distributed digital base is characterized by comprising the following steps of:
step S1: uploading the data of the Internet of things equipment to a slave data center closest to the Internet of things equipment;
step S2: and storing the total data of the main data center and backing up the data of the sub data center through a data synchronization strategy.
9. The internet of things real-time data hierarchical storage method based on the distributed digital base according to claim 8, wherein the master data center is a center point position of a polygon formed by taking each slave data center as a node; the distances between the master data center and each slave data center are basically equal; the sum of the distances between each slave data center and the master data center is minimum;
the main data center stores full data and the data backup of the data center adopts the following steps: carrying out data synchronization between the data centers in a synchronous queue mode;
the master data center is used as a full-volume data storage center, and data of the slave data centers are synchronized in real time;
and the slave data centers are mutually provided with data synchronization according to the nearest neighbor non-synchronization principle, and for one slave data center, the neighbor node which is nearest to and is not used as the other slave data center backup is searched and used as the object of data backup.
10. The internet of things real-time data layered storage method based on the distributed digital base according to claim 8, wherein a cache layer is arranged at a data center node;
the cache layer is a layer structure above an original data storage layer of the data center, and the hardware adopts solid-state hardware to provide high-efficiency reading and writing performance; the caching layer is used for caching static resource data and hot data;
the static resource data comprises equipment basic information and equipment installation address information;
the hot data is the data with highest access frequency, and the replacement strategy is the latest and longest non-access strategy; for hot data of the cache region, marking the latest access time of each group of data, and replacing new data by using the latest longest non-access strategy;
each data center provides an external data receiving inlet, and the Internet of things equipment directly registers the nearest slave data center and uploads data;
when a user initiates a data access request, judging the data center where the requested resource is located, and selecting one data center closest to the requested place in physical distance to provide service; when the request arrives at the data center, preferentially searching whether the cache layer can hit, if so, directly returning the data, and if not, searching the storage layer data and returning the data to the user;
when the main data center fails, firstly searching for the condition of lost data in a global data checking mode; then, positioning the slave data center where the lost data is located, and synchronizing the data in the slave data center to the master data center when the master data center has a data recovery condition; when the slave data center fails, the backup slave data center is searched through the backup information table, and when the slave data center has the data recovery condition, the slave backup slave data center recovers the data to the slave data center.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311072854.6A CN117176744A (en) | 2023-08-23 | 2023-08-23 | Internet of things real-time data layered storage system and method based on distributed digital base |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311072854.6A CN117176744A (en) | 2023-08-23 | 2023-08-23 | Internet of things real-time data layered storage system and method based on distributed digital base |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117176744A true CN117176744A (en) | 2023-12-05 |
Family
ID=88944057
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311072854.6A Pending CN117176744A (en) | 2023-08-23 | 2023-08-23 | Internet of things real-time data layered storage system and method based on distributed digital base |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117176744A (en) |
-
2023
- 2023-08-23 CN CN202311072854.6A patent/CN117176744A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103268318B (en) | A kind of distributed key value database system of strong consistency and reading/writing method thereof | |
| CN109241161B (en) | Meteorological data management method | |
| JP6044539B2 (en) | Distributed storage system and method | |
| CA2930101C (en) | Partition-based data stream processing framework | |
| CA2929776C (en) | Client-configurable security options for data streams | |
| CN103116661B (en) | A kind of data processing method of database | |
| US9794135B2 (en) | Managed service for acquisition, storage and consumption of large-scale data streams | |
| CN102855239B (en) | A kind of distributed geographical file system | |
| CN104965850B (en) | A kind of database high availability implementation method based on open source technology | |
| CN103379159B (en) | A kind of method that distributed Web station data synchronizes | |
| US20150134795A1 (en) | Data stream ingestion and persistence techniques | |
| CN103138912B (en) | Method of data synchronization and system | |
| CN105138615A (en) | Method and system for building big data distributed log | |
| CN111078667B (en) | Data migration method and related device | |
| CN101997823A (en) | Distributed file system and data access method thereof | |
| CN109815026A (en) | Electric power time series database based on distributed component | |
| Malensek et al. | Galileo: A framework for distributed storage of high-throughput data streams | |
| CN103399894A (en) | Distributed transaction processing method on basis of shared storage pool | |
| CN113168404A (en) | System and method for replicating data in a distributed database system | |
| WO2019197918A1 (en) | Fault-tolerant federated distributed database | |
| CN104281980B (en) | Thermal power generation unit remote diagnosis method and system based on Distributed Calculation | |
| CN109739435A (en) | File storage and update method and device | |
| CN113449065A (en) | Data deduplication-oriented decentralized storage method and storage device | |
| Duan et al. | Scalable data resilience for in-memory data staging | |
| CN107566341B (en) | Data persistence storage method and system based on federal distributed file storage system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |