CN116781699B - Data communication method and system based on distributed edge computing - Google Patents
Data communication method and system based on distributed edge computing Download PDFInfo
- Publication number
- CN116781699B CN116781699B CN202311041669.0A CN202311041669A CN116781699B CN 116781699 B CN116781699 B CN 116781699B CN 202311041669 A CN202311041669 A CN 202311041669A CN 116781699 B CN116781699 B CN 116781699B
- Authority
- CN
- China
- Prior art keywords
- data
- edge
- processing
- node
- edge node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000004891 communication Methods 0.000 title claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 133
- 238000004364 calculation method Methods 0.000 claims abstract description 14
- 238000007781 pre-processing Methods 0.000 claims abstract description 11
- 230000006798 recombination Effects 0.000 claims abstract description 9
- 238000005215 recombination Methods 0.000 claims abstract description 9
- 230000003139 buffering effect Effects 0.000 claims abstract description 8
- 230000008439 repair process Effects 0.000 claims abstract description 8
- 239000012634 fragment Substances 0.000 claims description 26
- 230000008569 process Effects 0.000 claims description 23
- 230000005540 biological transmission Effects 0.000 claims description 15
- 229920006395 saturated elastomer Polymers 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 10
- 239000000872 buffer Substances 0.000 claims description 8
- 238000009826 distribution Methods 0.000 claims description 6
- 230000001502 supplementing effect Effects 0.000 claims description 5
- 238000013467 fragmentation Methods 0.000 claims description 4
- 238000006062 fragmentation reaction Methods 0.000 claims description 4
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000011084 recovery Methods 0.000 claims description 3
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 230000008521 reorganization Effects 0.000 claims description 2
- 230000006870 function Effects 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 5
- 238000007796 conventional method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000011265 semifinished product Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/36—Flow control; Congestion control by determining packet size, e.g. maximum transfer unit [MTU]
- H04L47/365—Dynamic adaptation of the packet size
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Abstract
The application discloses a data communication method and a system based on distributed edge calculation, which relate to the field of data communication and comprise the following steps: acquiring data acquired by a sensor, and preprocessing the acquired data; adding description fields of data through the identification of edge nodes, and performing secondary processing and slicing processing on the edge nodes; if the data volume is too large to enable the edge node to be capable of processing, distributing the node task to other nodes, and carrying out uploading after revising the summarized content; buffering and repairing information received by a data receiving end, carrying out slicing recombination on the buffered and repaired data, and carrying out secondary repair on the recombined data to serve as output data; and feeding back to the cloud; and the cloud end adjusts the data processing of the edge node through the feedback result. The efficiency of data communication is improved, the stability and the reliability of the data communication are ensured, and the integrity and the accuracy of the data are improved.
Description
Technical Field
The application relates to the technical field of data communication, in particular to a data communication method and system based on distributed edge computing.
Background
In the 21 st century, along with the development of technologies such as internet of things, artificial intelligence and 5G, the scale and speed of data generation have been explosively increased, and particularly in the environment of internet of things, a large number of devices and sensors continuously generate a large amount of data. However, uploading all of these data to the cloud for processing may bring serious problems of bandwidth pressure, delay, potential safety hazard, etc.
In this regard, distributed edge computation has evolved. Edge computing is a computing architecture whose core idea is to transfer computing tasks from the cloud to devices at the network edge, so-called edge nodes, which may be internet of things gateways, enterprise servers, even unmanned vehicles, etc. The method can effectively reduce network delay, improve data processing efficiency and better protect data privacy.
However, the distributed nature of edge computation also presents new challenges, especially in terms of data communications. Because the edge nodes are widely distributed, the network environment is complex and changeable, and how to effectively, quickly and safely complete the data exchange between the edge nodes and the data center becomes an important problem to be solved. There is a need to devise a data communication method and system that can address these challenges.
Therefore, the data communication method and system based on the distributed edge calculation have important practical significance and value. The method can help enterprises and organizations to better process and utilize large-scale data, thereby improving the business efficiency, opening new business modes and even promoting the development of society and economy.
Disclosure of Invention
The present application has been made in view of the above-described problems.
Therefore, the technical problems solved by the application are as follows: the existing data communication method of distributed edge computing has the problem that breakdown easily occurs when data is large, and how to solve the problem of distribution of edge node tasks.
In order to solve the technical problems, the application provides the following technical scheme: a data communication method based on distributed edge computing, comprising: acquiring data acquired by a sensor, and preprocessing the acquired data; transmitting the preprocessed data to an edge node, adding a description field of the data through the identification function of the edge node, and performing secondary processing and fragmentation processing on the edge node; analyzing the data subjected to the secondary processing and the slicing processing through the edge node, and uploading analysis results to a cloud end and a data receiving end; if the data volume is too large, the edge node can not process, the node task is distributed to other nodes, the processing results are summarized, and the summarized content is modified again and then uploaded; buffering and repairing information received by a data receiving end, carrying out slicing recombination on the buffered and repaired data, and carrying out secondary repair on the recombined data to serve as output data; evaluating the integrity and accuracy of the output data by comparing the output data with the description field, and feeding back to the cloud; the cloud end adjusts the data processing of the edge node through the feedback result, and if the output result is inaccurate due to excessive operation, the processing capacity of the edge node is reduced.
As a preferred embodiment of the data communication method based on distributed edge computing according to the present application, the method further comprises: the preprocessing comprises filtering out the data information of the repeated content at the sensor and transmitting the processed data to the edge node; the secondary processing includes cleaning the data at the edge node, removing invalid and redundant data, and correcting the primarily identified error data.
As a preferred embodiment of the data communication method based on distributed edge computing according to the present application, the method further comprises: the edge node further comprises a node matching step according to the data volume and the distribution position of the sensors, wherein the node matched with the sensors is the edge node of the sensors.
And carrying out data slicing processing on the edge nodes, inserting and processing data edge node numbers and content description information of the data into the data content, dividing the inserted data into a plurality of data fragments, adding identifiable connection sequence characters before and after the fragments, and connecting the data fragments according to the connection sequence characters after receiving information by a receiving end.
When the data slicing is completed and the sliced data is accumulated to a preset number of slices, starting data transmission;
the slice size S is:
S = min(D, NT, PT)
wherein S represents the size of the slice, D represents the data size, N represents the network bandwidth, T represents the transmission time of the target, and P represents the processing capability of the edge node.
As a preferred embodiment of the data communication method based on distributed edge computing according to the present application, the method further comprises: establishing a coordinate system by taking all areas of data acquisition as coordinate origins, assigning numbers to each edge node in a coordinate form, taking nodes which are necessary to be used for data communication as conventional edge nodes, and setting a special edge node at the middle position of each four conventional edge nodes adjacent to each other; each regular edge node can communicate with 4 special edge nodes around the edge except the regional edge of the whole data acquisition region, and each special edge node can communicate with 4 regular edge nodes.
The special edge node does not participate in the processing of the data when the conventional edge node can process the data.
The node tasks comprise that when the data processed by the conventional edge nodes reach saturation, the special edge nodes intervene in the processed data; the function of processing data of 1/4 kinds of intermediate conventional edge nodes is recorded in 4 adjacent special edge nodes of the conventional edge nodes respectively; and when the special edge node intervenes, matching the data processing capacity with the highest functional operation load of the conventional edge node with the special edge node, and if one node intervenes and cannot be in a saturated state, sequentially matching the special edge node intervenes with the processing capacity according to the functional operation load of the conventional edge node from high to low.
As a preferred embodiment of the data communication method based on distributed edge computing according to the present application, the method further comprises: the node task further comprises the step of scheduling a part which cannot be processed by the data processing task of the node if the intervention of the special edge node is still in a saturated state, and scheduling the data processing task of the node by utilizing the conventional and special edge nodes with the same processing capacity; when scheduling, if the scheduled node is processing data, the original task of the scheduled node is preferentially processed, and if the original task of the scheduled node is also in a saturated state, the edge node is reselected.
After the scheduled node task is completed, summarizing the processing result to a primary edge node scheduled by the task, checking the data processing result through the primary edge node, and if the checking is correct, sending the processing result through the edge node for processing the data; if the exception exists in the check, the recovery scheduling task simultaneously limits the uploading processing result of the edge node for processing the data, and the original edge node carries out uploading after revising the summarized content.
As a preferred embodiment of the data communication method based on distributed edge computing according to the present application, the method further comprises: the segmentation and recombination comprises the steps that a receiving unit buffers and restores information received by a data receiving end, the segments are collected through a buffer zone, and the collected segments are connected with data segments according to the sequence characters.
Logically supplementing and repairing the recombined data, checking the recombined output data according to the content description information of the data inserted in the data content, and approving the data content if the recombined data accords with the content description information; if the recombined data does not accord with the content description information, the integrity and the accuracy of the output data are not approved, the non-conforming data content is marked, and the non-conforming data content and the serial number of the edge node outputting the non-approved output data are fed back to the cloud.
As a preferred embodiment of the data communication method based on distributed edge computing according to the present application, the method further comprises: the cloud derives and recalculates the original data of the edge node corresponding to the number through the feedback result, and feeds back the recalculated result to the receiving end; according to the recalculation process of the cloud to the original data, updating the data processing mode of the edge node corresponding to the serial number, and if the output result is inaccurate due to excessive operation, reducing the processing capacity of the edge node; if the calculation process is incorrect, updating the calculation process according to the cloud computing; after updating the data processing mode of the edge node, the data processing modes of the 4 special edge nodes adjacent to the updated edge node are synchronized.
Another object of the present application is to provide a data communication system based on distributed edge computing, which includes an acquisition unit, acquires data acquired by a sensor, performs preprocessing on the acquired data, and transmits information to an edge computing module; the edge computing module is used for adding description fields of data through the identification function of the edge nodes, performing secondary processing and slicing processing on the edge nodes, performing analysis on the data after the secondary processing and slicing processing through the edge nodes, and uploading analysis results to the cloud end and the data receiving end; if the data volume is too large to enable the edge node to be capable of processing, distributing node tasks to other nodes, summarizing processing results, and carrying out uploading after revising summarized contents; the data receiving module is used for receiving the information of the edge computing module, buffering and repairing the information received by the data receiving end, carrying out fragment recombination on the buffered and repaired data, and carrying out secondary repair on the recombined data to be used as output data; evaluating the integrity and accuracy of the output data by comparing the output data with the description field, and feeding back to the cloud; and the cloud end adjusts the data processing of the edge node through the feedback result.
A computer device, comprising: a memory and a processor; the memory stores a computer program characterized in that: the processor, when executing the computer program, implements the steps of a data communication method based on distributed edge computing as described above.
A computer-readable storage medium having stored thereon a computer program, characterized by: the computer program when executed by a processor implements the data communication method steps based on distributed edge computing as described above.
The application has the beneficial effects that: according to the data communication method based on distributed edge calculation, the data is preprocessed, processed secondarily and processed in the slicing mode at the edge nodes, so that the data quantity to be uploaded can be reduced, the influence of network quality on data transmission is reduced, and the data communication efficiency is improved. When the data volume is too large or the network environment is not good, the size of the data fragments and the strategy of sending can be dynamically adjusted so as to adapt to different network environments and data volumes and ensure the stability and reliability of data communication. The data receiving end can effectively solve the problems of data loss, disorder and the like in network transmission by buffering, repairing and slicing recombination of the received data, and improves the integrity and accuracy of the data. The designed node task allocation mechanism is used for inserting the special edge nodes into the processed data when the data processed by the conventional edge nodes reach saturation, so that the processing capacity of the whole system is effectively improved, and the stability and the efficiency of data processing are ensured.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:
FIG. 1 is a general flow chart of a data communication method based on distributed edge computing according to a first embodiment of the present application;
FIG. 2 is a diagram showing distribution of regular edge nodes and special edge nodes in a data communication method based on distributed edge computation according to a second embodiment of the present application;
fig. 3 is a comparison chart of updating efficiency and quality of an edge node algorithm in a data communication method based on distributed edge computing according to a second embodiment of the present application.
Detailed Description
So that the manner in which the above recited objects, features and advantages of the present application can be understood in detail, a more particular description of the application, briefly summarized above, may be had by reference to the embodiments, some of which are illustrated in the appended drawings. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application, but the present application may be practiced in other ways other than those described herein, and persons skilled in the art will readily appreciate that the present application is not limited to the specific embodiments disclosed below.
Further, reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic can be included in at least one implementation of the application. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.
While the embodiments of the present application have been illustrated and described in detail in the drawings, the cross-sectional view of the device structure is not to scale in the general sense for ease of illustration, and the drawings are merely exemplary and should not be construed as limiting the scope of the application. In addition, the three-dimensional dimensions of length, width and depth should be included in actual fabrication.
Also in the description of the present application, it should be noted that the orientation or positional relationship indicated by the terms "upper, lower, inner and outer", etc. are based on the orientation or positional relationship shown in the drawings, are merely for convenience of describing the present application and simplifying the description, and do not indicate or imply that the apparatus or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present application. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
The terms "mounted, connected, and coupled" should be construed broadly in this disclosure unless otherwise specifically indicated and defined, such as: can be fixed connection, detachable connection or integral connection; it may also be a mechanical connection, an electrical connection, or a direct connection, or may be indirectly connected through an intermediate medium, or may be a communication between two elements. The specific meaning of the above terms in the present application will be understood in specific cases by those of ordinary skill in the art.
Example 1
Referring to fig. 1, for one embodiment of the present application, there is provided a data communication method based on distributed edge computing, including:
s1: and acquiring data acquired by the sensor, and preprocessing the collected data.
Further, preprocessing includes filtering out duplicate data information at the sensor and transmitting the processed data to the edge node.
It is known that preprocessing at the sensor is only simple to perform, since the computational power on the sensor is not strong. The primary purpose of the sensor is to collect data, which may be digital or image data. The data are filtered, repeated content is filtered, the small data size can be guaranteed, and data transmission can be smoother.
S2: and transmitting the preprocessed data to an edge node, adding a description field of the data through the identification function of the edge node, and performing secondary processing and fragmentation processing on the edge node.
Analyzing the data subjected to the secondary processing and the slicing processing through the edge node, and uploading analysis results to a cloud end and a data receiving end; if the data volume is too large, the edge node can not process, the node task is distributed to other nodes, the processing results are summarized, and the summarized content is modified again and uploaded.
Further, the secondary processing includes cleaning the data at the edge node, removing invalid and redundant data, and correcting the primarily identified erroneous data. The edge node further comprises the step of matching the nodes according to the data quantity and the distribution positions of the sensors, wherein the nodes matched with the sensors are edge nodes of the sensors. And carrying out data slicing processing on the edge nodes, inserting and processing data edge node numbers and content description information of the data into the data content, dividing the inserted data into a plurality of data fragments, adding identifiable connection sequence characters before and after the fragments, and connecting the data fragments according to the connection sequence characters after receiving information by a receiving end.
It is to be noted that the edge node processes data, has a relatively strong computing power, and performs secondary processing on the data at the edge node, so that the data quality can be improved. The correction of the initially identified error data means that the obviously abnormal content in the data information or the image information is corrected. For example, when the image information is processed, the image content is deviated to break the original graphic outline, and the deviation position is repaired. Inserting processing data edge node numbers and content description information of data into data content to update algorithm providing basis of the nodes when processing errors are recognized in the later period; meanwhile, the content of the data is summarized, so that the accuracy of the data can be evaluated in time during the subsequent integration of the fragments. If the content description information description data does not have too high or too low data, abnormal data of the last received data can be locked quickly; if the content description information describes that the image is an image in a daytime mall, it is wrong if the received image is a black day or outdoors.
When the data slicing is completed and the sliced data is accumulated to a preset number of slices, starting data transmission; if the network quality is reduced, the preset number of the transmitting fragments is reduced in an equal ratio according to the network speed reduction ratio; determining the appropriate tile size is an important issue. Too large a slice may cause network congestion and a delay in the processing of the slice, and too small a slice may increase network overhead and complexity of data management. Each time of transmission has a quota, when the set number is reached, the stability of the data quantity can be ensured by starting the data transmission, and the size of the data transmission can be ensured to be supported by the network speed according to the number setting of the network speed reduction ratio and the like. The fragment size may also be dynamically adjusted according to network conditions, processing power of the edge nodes, and data type.
The slice size S is:
S = min(D, NT, PT)
wherein S represents the size of the slice, D represents the data size, N represents the network bandwidth, T represents the transmission time of the target, and P represents the processing capability of the edge node. This ensures that neither the bandwidth of the network nor the processing power of the nodes is exceeded, while the transmission time target is met. Encoding the data slices may increase the fault tolerance of the system. Some common coding schemes, such as Reed-Solomon coding, may be used. The encoding may add redundant information to the data slices so that in the event of a missing portion of the data slices, the complete data can still be recovered.
The method is characterized in that a coordinate system is established by taking all the data acquisition areas as the origin of coordinates, each edge node is given a number in the form of coordinates, the nodes which are necessary to be used for data communication are taken as conventional edge nodes, and a special edge node is arranged at the middle position of each four conventional edge nodes which are adjacent to each other; each conventional edge node can communicate with 4 surrounding special edge nodes except the regional edges of all the data acquisition regions, and each special edge node can communicate with 4 conventional edge nodes; according to the coordinate numbers, the edge node positions can be accurately locked.
When the special edge node can process data at the conventional edge node, the special edge node does not participate in the data processing; the node tasks comprise that when the data processed by the conventional edge nodes reach saturation, the special edge nodes intervene in processing the data; the function of processing data of 1/4 kinds of intermediate conventional edge nodes is recorded in 4 adjacent special edge nodes of the conventional edge nodes respectively; and when the special edge node intervenes, matching the data processing capacity with the highest functional operation load of the conventional edge node with the special edge node, and if one node intervenes and cannot be in a saturated state, sequentially matching the special edge node intervenes with the processing capacity according to the functional operation load of the conventional edge node from high to low.
It should be noted that, the function of each conventional edge node is divided into 4 parts and stored in 4 special edge nodes respectively, and one special edge node can store 1/4 functions of 4 conventional edge nodes, so that the total amount of data that each edge node can process is ensured to be not much different, and the computing capability of the node is fully utilized. The functions of the nodes are divided into 4 parts and stored respectively, so that when a special edge node is damaged or fails, the other 3 edge nodes can be involved in processing data. When a special edge node is damaged or fails, the node becomes a saturated state during intervention processing, so that the data volume which can be processed by the node can be ensured, and the node cannot collapse when the data volume is large.
It is to be noted that if the intervention of the special edge node is still in a saturated state, the part of the data processing task of the node which cannot be processed is scheduled, and the data processing task of the node is scheduled by using the conventional and special edge nodes with the same processing capacity; when scheduling, if the scheduled node is processing data, the original task of the scheduled node is preferentially processed, and if the original task of the scheduled node is also in a saturated state, the edge node is reselected; the task amount is reduced through the scheduling of the node tasks, and the tasks are reasonably processed through the scheduling of the nodes with the same processing capacity.
After the scheduled node task is completed, summarizing the processing result to a primary edge node scheduled by the task, checking the data processing result through the primary edge node, and if the checking is correct, sending the processing result through the edge node for processing the data; if the exception exists in the check, the recovery scheduling task simultaneously limits the uploading processing result of the edge node for processing the data, and the original edge node carries out uploading after revising the summarized content.
It will be appreciated that if the task scheduling of the original node a is that of the node B, the data processed by the node B does not meet the criteria, and a failure is identified by the node a, it is indicated that the node B is unsuitable for processing the content, and the data content of the "semi-finished product" processed by the node a is reprocessed or modified continuously, so that the final processing unit of the data becomes a, and the node a needs to be responsible for outputting the content.
S3: buffering and repairing information received by a data receiving end, carrying out slicing recombination on the buffered and repaired data, and carrying out secondary repair on the recombined data to serve as output data; evaluating the integrity and accuracy of the output data by comparing the output data with the description field, and feeding back to the cloud; the cloud end adjusts the data processing of the edge node through the feedback result, and if the output result is inaccurate due to excessive operation, the processing capacity of the edge node is reduced.
The fragment reorganization comprises that a receiving unit buffers and restores information received by a data receiving end, fragments are collected through a buffer zone, and the collected fragments are connected with the data fragments according to the sequence characters. The receiving end needs to maintain a buffer for the received data fragments. This buffer needs to be able to accommodate a sufficient number of data slices and to be able to sort according to the metadata of the data slices. Logically supplementing and repairing the recombined data, checking the recombined output data according to the content description information of the data inserted in the data content, and approving the data content if the recombined data accords with the content description information; if the recombined data does not accord with the content description information, the integrity and the accuracy of the output data are not approved, the non-conforming data content is marked, and the non-conforming data content and the serial number of the edge node outputting the non-approved output data are fed back to the cloud.
It is noted that when successive data slices are received, data merging is required. The goal of merging is to reassemble the fragmented data into the original data format, e.g., a complete file or data stream. Data merging needs to deal with the problem of fragment loss and out-of-order. For lost data fragmentation, a retransmission request needs to be sent to a corresponding edge node; for out-of-order data slicing, reordering according to its slicing order is required. And carrying out logic supplement and data restoration on the recombined data, supplementing and restoring the content according to the description field when the recombined data does not accord with the description of the content description field, for example, describing as a face photo of a person, and supplementing and restoring the missing ear according to the description when the recombined image data lacks one ear. According to conventional logic, another part is reproduced with reference to a single ear appearing in the image information.
The cloud derives and recalculates the original data of the edge node corresponding to the number through the feedback result, and feeds back the recalculated result to the receiving end; according to the recalculation process of the cloud to the original data, updating the data processing mode of the edge node corresponding to the serial number, and if the output result is inaccurate due to excessive operation, reducing the processing capacity of the edge node; if the calculation process is incorrect, updating the calculation process according to the cloud computing; after updating the data processing mode of the edge node, the data processing modes of the 4 special edge nodes adjacent to the updated edge node are synchronized.
It is known that the computing capacity and learning capacity of the cloud are higher than those of the edge nodes, so that the process is led into the edge nodes through a cloud updating algorithm and an analysis process to update the edge nodes. And meanwhile, the corresponding algorithm of the special edge node is updated, so that the consistency of the algorithm and an output result can be ensured during the intervention data processing.
The embodiment also provides a data communication system based on distributed edge calculation, which comprises an acquisition unit, an edge calculation module, a data receiving module and a cloud.
Specifically, the acquisition unit is used for acquiring data acquired by the sensor, preprocessing the acquired data and transmitting information to the edge calculation module; the edge computing module adds description fields of data through the identification function of edge nodes, performs secondary processing and slicing processing on the edge nodes, performs analysis on the data after the secondary processing and slicing processing through the edge nodes, and uploads analysis results to a cloud end and a data receiving end; if the data volume is too large, the edge node can not process, the node task is distributed to other nodes, the processing results are summarized, and the summarized content is modified again and then uploaded; the data receiving module is used for receiving the information of the edge computing module, buffering and repairing the information received by the data receiving end, carrying out fragment recombination on the buffered and repaired data, and carrying out secondary repair on the recombined data to be used as output data; evaluating the integrity and accuracy of the output data by comparing the output data with the description field, and feeding back to the cloud; and the cloud end adjusts the data processing of the edge node through the feedback result.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile memory may include read only memory, magnetic tape, floppy disk, flash memory, optical memory, high density embedded nonvolatile memory, resistive memory, magnetic memory, ferroelectric memory, phase change memory, graphene memory, and the like. Volatile memory can include random access memory, external cache memory, or the like. By way of illustration, and not limitation, RAM can take many forms, such as static random access memory or dynamic random access memory. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like.
The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
Example 2
Referring to fig. 2 and 3, for one embodiment of the present application, a data communication method based on distributed edge computing is provided, and in order to verify the beneficial effects of the present application, scientific demonstration is performed through economic benefit computing and simulation experiments.
First, fig. 2 is a diagram showing a distribution diagram of regular edge nodes and special edge nodes in a data communication method based on distributed edge calculation, and it can be seen that each regular edge node has a connection with surrounding 4 special edge nodes, and each special edge node has a connection with surrounding 4 regular edge nodes. Where triangles represent special edge nodes and circles represent regular edge nodes.
Table 1 shows the results of the present application compared with the conventional method, and the test data obtained by a plurality of simulation tests.
Table 1 simulation test results table
It can be seen that the application is significantly higher than the conventional method in terms of data transmission accuracy and is maintained at a stable level; the data transmission speed is also faster than the conventional method. The application can rapidly and accurately perform data communication.
Fig. 3 is a comparison of the efficiency and quality of the algorithm update at the edge node of the present application with the conventional method. Wherein the data quality and the efficiency of the algorithm update are embodied in a ratio to the reference value. According to the method, the algorithm updating speed of the nodes is higher through feedback and cloud updating, and the data quality can be ensured to be maintained at a higher level all the time.
It should be noted that the above embodiments are only for illustrating the technical solution of the present application and not for limiting the same, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present application may be modified or substituted without departing from the spirit and scope of the technical solution of the present application, which is intended to be covered in the scope of the claims of the present application.
Claims (7)
1. A data communication method based on distributed edge computing, comprising:
acquiring data acquired by a sensor, and preprocessing the acquired data;
transmitting the preprocessed data to an edge node, adding a description field of the data through the identification function of the edge node, and performing secondary processing and fragmentation processing on the edge node;
analyzing the data subjected to the secondary processing and the slicing processing through the edge node, and uploading analysis results to a cloud end and a data receiving end; if the data volume is too large, the edge node can not process, the node task is distributed to other nodes, the processing results are summarized, and the summarized content is modified again and then uploaded;
buffering and repairing information received by a data receiving end, carrying out slicing recombination on the buffered and repaired data, and carrying out secondary repair on the recombined data to serve as output data; evaluating the integrity and accuracy of the output data by comparing the output data with the description field, and feeding back to the cloud; the cloud end adjusts the data processing of the edge nodes through the feedback result;
the edge nodes comprise a coordinate system established by taking all the data acquisition areas as the origin of coordinates, each edge node is given a number in the form of coordinates, the nodes which are necessary to be used for data communication are taken as conventional edge nodes, and a special edge node is arranged at the middle position of each four conventional edge nodes which are adjacent to each other; each conventional edge node can communicate with 4 surrounding special edge nodes except the regional edges of all the data acquisition regions, and each special edge node can communicate with 4 conventional edge nodes;
the special edge node does not participate in the data processing when the conventional edge node can process the data;
the node tasks comprise that when the data processed by the conventional edge nodes reach saturation, the special edge nodes intervene in the processed data; the function of processing data of 1/4 kinds of intermediate conventional edge nodes is recorded in 4 adjacent special edge nodes of the conventional edge nodes respectively; and when the special edge node is in the medium, the special edge node is matched according to the data processing capacity with the highest functional operation load of the conventional edge node, and if one special edge node is in the medium, the conventional edge node is still in a saturated state, and the special edge node with the processing capacity is matched in sequence from high to low according to the functional operation load of the conventional edge node.
2. The distributed edge computing-based data communication method of claim 1, wherein: the preprocessing comprises filtering out the data information of the repeated content at the sensor and transmitting the processed data to the edge node;
the secondary processing includes cleaning the data at the edge node, removing invalid and redundant data, and correcting the primarily identified error data.
3. The distributed edge computing-based data communication method of claim 2, wherein: the edge nodes further comprise nodes matched according to the data volume and the distribution positions of the sensors, and the nodes matched with the sensors are edge nodes of the sensors;
performing data slicing processing on the edge nodes, inserting and processing data edge node numbers and content description information of data into data content, dividing the inserted data into a plurality of data fragments, adding identifiable connection sequence characters before and after the fragments, and connecting the data fragments according to the connection sequence characters after receiving information by a receiving end;
when the data slicing is completed and the sliced data is accumulated to a preset number of slices, starting data transmission;
the slice size S is:
S = min(D, NT, PT)
wherein S represents the size of the slice, D represents the data size, N represents the network bandwidth, T represents the transmission time of the target, and P represents the processing capability of the edge node.
4. A data communication method based on distributed edge computing as defined in claim 3, wherein: the node task further comprises the step of scheduling a part which cannot be processed by the data processing task of the node if the intervention of the special edge node is still in a saturated state, and scheduling the data processing task of the node by utilizing the conventional and special edge nodes with the same processing capacity; when scheduling, if the scheduled node is processing data, the original task of the scheduled node is preferentially processed, and if the original task of the scheduled node is also in a saturated state, the edge node is reselected;
after the scheduled node task is completed, summarizing the processing result to a primary edge node of task scheduling, checking the data processing result through the primary edge node, and if the checking is correct, sending the processing result through the edge node for processing the data; if the exception exists in the check, the recovery scheduling task simultaneously limits the uploading processing result of the edge node for processing the data, and the original edge node carries out uploading after revising the summarized content.
5. The distributed edge computing-based data communication method of claim 4, wherein: the fragment reorganization comprises the steps that a receiving unit buffers and repairs information received by a data receiving end, fragments are collected through a buffer zone, and the collected fragments are connected with data fragments according to the sequence characters;
logically supplementing and repairing the recombined data, checking the recombined output data according to the content description information of the data inserted in the data content, and approving the data content if the recombined data accords with the content description information; if the recombined data does not accord with the content description information, the integrity and the accuracy of the output data are not approved, the non-conforming data content is marked, and the number of the edge node of the processing data inserted in the data content is fed back to the cloud.
6. The distributed edge computing-based data communication method of claim 5, wherein: the cloud derives and recalculates the original data of the edge node corresponding to the number through the feedback result, and feeds back the recalculated result to the receiving end;
according to the recalculation process of the cloud to the original data, updating the data processing mode of the edge node corresponding to the serial number, and if the output result is inaccurate due to excessive operation, reducing the processing capacity of the edge node; if the calculation process is incorrect, updating the calculation process according to the cloud computing;
after updating the data processing mode of the edge node, the data processing modes of the 4 special edge nodes adjacent to the updated edge node are synchronized.
7. A data communication system employing the distributed edge computing-based data communication method of any of claims 1-6, wherein:
the acquisition unit acquires data acquired by the sensor, performs preprocessing on the acquired data, and transmits information to the edge calculation module;
the edge computing module is used for adding description fields of data through the identification function of the edge nodes, performing secondary processing and slicing processing on the edge nodes, performing analysis on the data after the secondary processing and slicing processing through the edge nodes, and uploading analysis results to the cloud end and the data receiving end; if the data volume is too large, the edge node can not process, the node task is distributed to other nodes, the processing results are summarized, and the summarized content is modified again and then uploaded; the edge nodes comprise a coordinate system established by taking all the data acquisition areas as the origin of coordinates, each edge node is given a number in the form of coordinates, the nodes which are necessary to be used for data communication are taken as conventional edge nodes, and a special edge node is arranged at the middle position of each four conventional edge nodes which are adjacent to each other; each conventional edge node can communicate with 4 surrounding special edge nodes except the regional edges of all the data acquisition regions, and each special edge node can communicate with 4 conventional edge nodes; the special edge node does not participate in the data processing when the conventional edge node can process the data; the node tasks comprise that when the data processed by the conventional edge nodes reach saturation, the special edge nodes intervene in the processed data; the function of processing data of 1/4 kinds of intermediate conventional edge nodes is recorded in 4 adjacent special edge nodes of the conventional edge nodes respectively; when a special edge node is in-between, matching the special edge node according to the data processing capacity with the highest functional operation load of the conventional edge node, and if one special edge node is in-between, the conventional edge node is still in a saturated state, and then the special edge node with the processing capacity is matched in sequence from high to low according to the functional operation load of the conventional edge node;
the data receiving module is used for receiving the information of the edge computing module, buffering and repairing the information received by the data receiving end, carrying out fragment recombination on the buffered and repaired data, and carrying out secondary repair on the recombined data to be used as output data; evaluating the integrity and accuracy of the output data by comparing the output data with the description field, and feeding back to the cloud;
and the cloud end adjusts the data processing of the edge node through the feedback result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311041669.0A CN116781699B (en) | 2023-08-18 | 2023-08-18 | Data communication method and system based on distributed edge computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311041669.0A CN116781699B (en) | 2023-08-18 | 2023-08-18 | Data communication method and system based on distributed edge computing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116781699A CN116781699A (en) | 2023-09-19 |
| CN116781699B true CN116781699B (en) | 2023-10-27 |
Family
ID=87993388
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311041669.0A Active CN116781699B (en) | 2023-08-18 | 2023-08-18 | Data communication method and system based on distributed edge computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116781699B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117097592B (en) * | 2023-10-20 | 2023-12-15 | 南京科控奇智能科技有限公司 | Edge computing gateway based on cloud computing |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112511586A (en) * | 2020-10-21 | 2021-03-16 | 中国铁道科学研究院集团有限公司通信信号研究所 | High-speed railway intelligent traffic scheduling safety card control system based on cloud edge cooperation |
| CN115604353A (en) * | 2022-10-27 | 2023-01-13 | 广西电网有限责任公司（Cn） | Data processing method and system in power monitoring system and computer equipment |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111641520B (en) * | 2020-04-30 | 2022-11-18 | 深圳精匠云创科技有限公司 | Edge computing node device |
| US20230021868A1 (en) * | 2021-07-23 | 2023-01-26 | Ancestry.Com Operations Inc. | Data-sharding for efficient record search |
-
2023
- 2023-08-18 CN CN202311041669.0A patent/CN116781699B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112511586A (en) * | 2020-10-21 | 2021-03-16 | 中国铁道科学研究院集团有限公司通信信号研究所 | High-speed railway intelligent traffic scheduling safety card control system based on cloud edge cooperation |
| CN115604353A (en) * | 2022-10-27 | 2023-01-13 | 广西电网有限责任公司（Cn） | Data processing method and system in power monitoring system and computer equipment |
Non-Patent Citations (1)
| Title |
|---|
| 区块链存储：技术与挑战;蔡振华;林嘉韵;刘芳;;网络与信息安全学报(第05期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116781699A (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2019238109A1 (en) | Fault root cause analysis method and apparatus | |
| CN116781699B (en) | Data communication method and system based on distributed edge computing | |
| US9983941B2 (en) | Method and apparatus for recovering data | |
| WO2021003822A1 (en) | Data storage and recovery method and apparatus, and computer device | |
| CN103916483A (en) | Self-adaptation data storage and reconstruction method for coding redundancy storage system | |
| CN111831223B (en) | Fault-tolerant coding method, device and system for improving expandability of data deduplication system | |
| JP5254443B2 (en) | Surveillance method used for communication system images or multimedia video images | |
| US11467905B1 (en) | Stripe merging method and system based on erasure codes | |
| CN112131609A (en) | Merkle tree-based electric energy quality data exchange format file integrity verification method and system | |
| CN110543472B (en) | Data reconciliation method and related device | |
| CN103838649A (en) | Method for reducing calculation amount in binary coding storage system | |
| CN115408350A (en) | Log compression method, log recovery method, log compression device, log recovery device, computer equipment and storage medium | |
| US10229142B2 (en) | Method and system for handling binary large objects | |
| CN116192154B (en) | Data compression and data decompression method and device, electronic equipment and chip | |
| CN116955355A (en) | Block data processing method and device and electronic equipment | |
| CN116760661A (en) | Data storage method, apparatus, computer device, storage medium, and program product | |
| CN116226139A (en) | Distributed storage and processing method and system suitable for large-scale ocean data | |
| CN116303789A (en) | Parallel synchronization method and device for multi-fragment multi-copy database and readable medium | |
| CN107612554B (en) | Data compression processing method | |
| US20220245097A1 (en) | Hashing with differing hash size and compression size | |
| CN103092716A (en) | Method and device for obtaining erasure code parameters | |
| CN109992475B (en) | Log processing method, server and storage medium | |
| CN112365247A (en) | Block chain storage optimization method based on remainder system and RAFT algorithm | |
| CN112131291A (en) | JSON data-based structured analysis method, device, equipment and storage medium | |
| CN112199529A (en) | Picture processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |