CN115022045A - Data processing method and system based on edge cloud - Google Patents
Data processing method and system based on edge cloud Download PDFInfo
- Publication number
- CN115022045A CN115022045A CN202210624353.3A CN202210624353A CN115022045A CN 115022045 A CN115022045 A CN 115022045A CN 202210624353 A CN202210624353 A CN 202210624353A CN 115022045 A CN115022045 A CN 115022045A
- Authority
- CN
- China
- Prior art keywords
- data
- operation data
- edge cloud
- job
- validity verification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 14
- 230000005540 biological transmission Effects 0.000 claims abstract description 89
- 238000012795 verification Methods 0.000 claims abstract description 72
- 238000009499 grossing Methods 0.000 claims abstract description 51
- 238000012545 processing Methods 0.000 claims abstract description 21
- 238000000034 method Methods 0.000 claims abstract description 20
- 230000001105 regulatory effect Effects 0.000 claims abstract description 9
- 238000001514 detection method Methods 0.000 claims description 18
- 238000004458 analytical method Methods 0.000 claims description 10
- 230000007246 mechanism Effects 0.000 claims description 9
- 238000012544 monitoring process Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000001276 controlling effect Effects 0.000 claims description 6
- 230000008030 elimination Effects 0.000 claims description 2
- 238000003379 elimination reaction Methods 0.000 claims description 2
- 238000010200 validation analysis Methods 0.000 claims 1
- 208000032370 Secondary transmission Diseases 0.000 abstract description 5
- 238000011161 development Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 230000003111 delayed effect Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 229910002056 binary alloy Inorganic materials 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0209—Architectural arrangements, e.g. perimeter networks or demilitarized zones
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
- H04L63/0236—Filtering by address, protocol, port number or service, e.g. IP-address or URL
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention relates to the technical field of data processing, in particular to a data processing method and system based on an edge cloud. The method replaces the existing information comparison method with the validity verification of the operation data, so that on one hand, the data security of the edge cloud is ensured, and on the other hand, the efficiency of the data validity verification is improved. In addition, data validity verification based on the exponential smoothing algorithm can effectively avoid data which is unfavorable for practical development when lawless persons utilize vulnerability propagation in data transmission. The transmission source and the transmission mode of the illegal operation data are recorded and stored, so that the photos, the characters and the numbers can be inquired at any time, the illegal data are eliminated, the secondary transmission of the illegal data is prevented, and the data transmission safety is effectively enhanced. On the basis, the redundancy of the operation data is eliminated, the operation data is regulated and controlled, a proper port is distributed to the operation data, and the data legality verification efficiency can be further improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data processing method and system based on an edge cloud.
Background
Edge cloud computing, edge cloud for short, is based on the core of cloud computing technology and the ability of edge computing, constructs a cloud computing platform on edge infrastructure, forms an elastic cloud platform with comprehensive capabilities of computing, network, storage, and the like at an edge position, forms an end-to-end technical architecture of 'cloud edge end three-body cooperation' with a central cloud and an internet of things terminal, reduces response delay, reduces cloud pressure, reduces bandwidth cost, and provides cloud services such as whole network scheduling, computing power distribution, and the like by putting the work of network forwarding, storage, computing, intelligent data analysis, and the like at the edge.
In the prior art, a patent with publication number CN113572821A discloses an edge cloud node task cooperative processing method and system, including: constructing an edge cloud node pool comprising a mobile edge cloud and a network edge cloud, and acquiring a time delay topology and edge cloud node resources of the mobile edge cloud and the network edge cloud; receiving a terminal service; according to the terminal service attribute, the service processing time delay of the edge cloud node, the node resource and the service processing benefit, the optimal edge cloud node or the optimal edge cloud node group in the edge cloud node pool is selected, so that the terminal service processing is completed, the node resources of the edge cloud and the network edge cloud are moved in a comprehensive mode, the optimal edge cloud node or the optimal node group is selected based on the service processing time delay of the edge cloud node, the node resource, the service processing benefit and the like, and the above patent can effectively utilize the edge cloud resource and meet the requirements of user service requirements and cost reduction and efficiency improvement of operators.
With the continuous development of cloud technology, the existing edge cloud is basically perfect, and can well perform cooperative processing on cloud computing, reduce response time delay and lighten cloud pressure. However, the edge cloud applications are gradually increased, and the defects of the edge cloud are also developed, wherein especially, the safety hazard exists in the data execution and data transmission of the edge cloud. Since most of data received by the edge cloud is from the outside, the legitimacy of the data and the legitimacy of data transmission cannot be guaranteed, and therefore before data is calculated, the edge cloud needs to verify the legitimacy of the data to confirm the type, source and authority of the data. The existing legitimacy verification generally adopts an information comparison method to compare non-uniform format data such as photos, characters, data and the like one by one and judge whether the data is legal, and the comparison method not only occupies large system resources and has low efficiency, but also has the problems of loss of transmitted data and the like. The existing edge cloud cannot quickly verify the legality of massive and non-uniform data, so that the data processing efficiency is low, and the problem is one of the problems to be solved urgently in the technical field of data processing. At present, a data processing method and system based on edge cloud and capable of improving the validity verification efficiency of edge cloud data are needed.
Disclosure of Invention
The invention aims to overcome at least one defect of the prior art, and provides a data processing method and a data processing system based on edge cloud, which are used for solving the problem of low legal verification efficiency of edge cloud data.
The technical scheme adopted by the invention is as follows:
an edge cloud-based data processing method, comprising:
receiving operation data transmitted by a central cloud;
carrying out validity verification on the operation data;
if the operation data do not pass the validity verification, the operation data transmitted by the central cloud are received again;
if the operation data passes the validity verification, analyzing the operation data, and calculating the data volume and the predicted transmission time of the operation data;
removing redundancy of the operation data according to the analysis result of the operation data;
compressing and decomposing the job data after the redundancy is removed, and regulating and controlling the decomposed job data to a corresponding port according to the data volume and the predicted transmission time of the job data;
and transmitting the operation data positioned on the port to a terminal server.
As a further aspect of the present invention, the verifying the validity of the job data includes:
performing data validity verification on the operation data based on an exponential smoothing algorithm;
if the operation data does not pass the data validity verification, judging that the operation data does not pass the data validity verification, deleting the operation data, and storing a transmission record of the operation data;
if the operation data passes the data validity verification, the transmission validity verification is carried out on the operation data;
if the operation data does not pass the transmission validity verification, judging that the operation data does not pass the validity verification, and returning the operation data to a sending place;
and if the operation data passes the transmission validity verification, judging that the operation data passes the validity verification.
As a further aspect of the present invention, the data validity verification of the job data based on the exponential smoothing algorithm includes:
establishing a detection model based on an exponential smoothing algorithm, and setting rule data; the rule data are data in accordance with a legality rule;
calculating the job data and the rule data by using the detection model;
and comparing the calculation results of the job data and the rule data so as to verify the data validity of the job data.
As a further aspect of the present invention, the formula of the exponential smoothing algorithm is:
Y t+1 ＝αX t +α(1-α)X t-1 +α(1-α) 2 X t-2 +…+α(1-α) n X t-n
wherein, Y t+1 The predicted value of the operation/rule data is shown as alpha, a weight coefficient is shown as n, a smoothing index is shown as t, the number of observation periods of the operation/rule data is shown as t, and the observed value of the operation/rule data is shown as X.
As a further aspect of the present invention, after transmitting the job data located at the port to the terminal server, the method further includes:
judging whether the terminal server receives the operation data;
if the terminal server does not finish receiving the operation data within the expected transmission time, the operation data is sent to the terminal server again;
and if the terminal server has received the operation data within the expected transmission time, finishing the operation data transmission.
An edge cloud based data processing system comprising:
the edge cloud input module is used for receiving operation data transmitted by the center cloud;
the edge cloud monitoring module is used for verifying the validity of the operation data;
the edge cloud analysis module is used for analyzing the job data passing the validity verification and calculating the data volume and the predicted transmission time of the job data;
the edge cloud computing module is used for eliminating redundancy of the operation data according to the analysis result of the operation data;
the edge cloud regulation and control module is used for compressing and decomposing the job data after the redundancy is removed, and regulating and controlling the decomposed job data to corresponding ports according to the data volume and the predicted transmission time of the job data;
and the edge cloud output module is used for transmitting the operation data positioned at the port to a terminal server.
As a further aspect of the present invention, the edge cloud monitoring module includes:
the data legality unit is used for verifying the data legality of the operation data based on an exponential smoothing algorithm;
a record elimination unit configured to delete the job data that fails the data validity verification and save a transmission record of the job data;
a transmission legality unit for performing transmission legality verification on the job data passing the data legality verification;
and the data return unit is used for returning the job data which fails to pass the transmission validity verification to the sending part.
As a further aspect of the present invention, the data legality unit includes:
the initialization mechanism is used for establishing a detection model based on an exponential smoothing algorithm and setting rule data; the rule data are data in accordance with a legality rule;
a detection model mechanism for calculating the operation data and the rule data by using the detection model;
and the data comparison mechanism is used for comparing the calculation results of the operation data and the rule data so as to verify the data validity of the operation data.
As a further aspect of the present invention, the formula of the exponential smoothing algorithm is:
Y t+1 ＝αX t +α(1-α)X t-1 +α(1-α) 2 X t-2 +…+α(1-α) n X t-n
wherein, Y t+1 And alpha is a weight coefficient, n is a smoothing index, t is the number of observation periods of the operation/rule data, and X is an observation value of the operation/rule data.
As a further aspect of the present invention, the method further includes:
and the edge cloud delay module is used for retransmitting the operation data which is not received by the terminal server within the expected transmission time.
Compared with the prior art, the invention has the beneficial effects that: according to the scheme, the validity verification of the operation data replaces the existing information comparison method, so that on one hand, the data security of the edge cloud is guaranteed, and on the other hand, the data validity verification efficiency is improved. In addition, data validity verification based on the exponential smoothing algorithm can effectively avoid data which is unfavorable for practical development when lawless persons utilize vulnerability propagation in data transmission. The transmission source and the transmission mode of the illegal operation data are recorded and stored, so that the photos, the characters and the numbers can be inquired at any time, the illegal data are eliminated, the secondary transmission of the illegal data is prevented, and the data transmission safety is effectively enhanced. On the basis, the redundancy of the operation data is eliminated, the operation data is regulated and controlled, a proper port is distributed to the operation data, and the data legality verification efficiency can be further improved.
Drawings
FIG. 1 is a flow chart of a method of the present invention;
FIG. 2 is a schematic diagram of an edge cloud of the present invention;
FIG. 3 is a schematic diagram of the system of the present invention.
Detailed Description
The drawings are only for purposes of illustration and are not to be construed as limiting the invention. For a better understanding of the following embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
Examples
As shown in fig. 1, the present embodiment provides a data processing method based on an edge cloud, including:
s10, receiving operation data transmitted by a center cloud;
as shown in fig. 2, in the present embodiment, the center cloud is communicatively connected to the edge cloud, and the edge cloud is communicatively connected to the terminal server. The center cloud transmits the operation data to the edge cloud, and the edge cloud processes the operation data and provides the processed operation data to the terminal server. The operation data is a computer instruction, and the edge cloud is used for executing the operation data to provide forwarding, storing and computing services for the user. The central cloud comprises an internet data center and the edge cloud comprises a domain controller. The domain controller comprises a domain name resolver and a domain name server, and the internet data center is in communication connection with the domain name resolver and the domain name server respectively. And the terminal server is respectively in communication connection with the domain name resolver and the domain name server.
The edge cloud also includes DNS and dynamic DNS. The DNS is respectively in communication connection with the domain name resolver and the domain name server, and the DNS analyzes the domain name resolver and the domain name server. The dynamic DNS is in communication connection with the domain name resolver and the domain name server, and the dynamic DNS analyzes the domain name resolver and the domain name server.
The operation data sent by the user passes through the internet data center, and at the moment, the center cloud carries out primary cloud computing on the operation data. After the operation data are simply calculated by the central cloud, the operation data pass through the domain controller, at the moment, a domain name resolver and a domain name server in the domain controller analyze an access IP address by using a DNS (domain name system) and a dynamic DNS (domain name system), so that the terminal server further processes the transmission data.
S20, performing validity verification on the operation data;
the validity verification comprises: data validity verification and transmission validity verification. Whether the data is legal or not is judged, and the legality of the data transmission process need to be judged in a grading manner. Data types include, but are not limited to, photographs, text, and numbers. The existing judgment method is to compare photos, characters and numbers with data stored in edge cloud so as to distinguish illegal data. If the edge cloud uses the existing judging mode to compare and identify the data in the non-uniform format such as the photos, the characters, the numbers and the like one by one, the operation of the identification can occupy a large amount of computing resources, and the edge cloud server can not process massive comparison data in time. To this end, the present invention employs an exponential smoothing algorithm to solve this problem.
As a further aspect of the present invention, the verifying the validity of the job data includes:
performing data validity verification on the operation data based on an exponential smoothing algorithm;
if the operation data does not pass the data validity verification, judging that the operation data does not pass the data validity verification, deleting the operation data, and storing a transmission record of the operation data;
if the operation data passes the data validity verification, the transmission validity verification is carried out on the operation data;
if the operation data does not pass the transmission validity verification, judging that the operation data does not pass the validity verification, and returning the operation data to a sending place;
and if the operation data passes the transmission validity verification, judging that the operation data passes the validity verification.
In the process of validity verification, for the fact that the operation data are legal but the operation data transmission process is illegal, the edge cloud returns the operation data to the user where the data are sent, and the data are sent again until the data transmission process is transmitted through a legal channel, data transmission success is displayed, and the data which are not beneficial to practical development are effectively prevented from being transmitted by lawbreakers through holes in the data transmission. The transmission source and the transmission mode of the illegal operation data are recorded and stored so as to inquire the photo, the characters and the numbers at any time, eliminate the illegal data, prevent the secondary transmission of the illegal data and further enhance the safety of data transmission.
As a further scheme of the invention, the data validity verification is carried out on the operation data based on the exponential smoothing algorithm, and the method comprises the following steps:
establishing a detection model based on an exponential smoothing algorithm; and setting rule data; the rule data are data in accordance with a legality rule;
the exponential smoothing algorithm is a medium-short term time series data trend prediction algorithm, is compatible with the advantages of full-term averaging and moving average, and is characterized in that different weights are given to past observed values, namely the weight of a more recent observed value is larger than that of a more distant observed value. According to different smoothing times, the exponential smoothing algorithm is divided into a first exponential smoothing method, a second exponential smoothing method, a third exponential smoothing method and the like. The basic ideas of the exponential smoothing algorithm are as follows: the predicted value is a weighted sum of previous observations, with different weights given to different data, with new data given more weight and old data given less weight.
The formula of the exponential smoothing algorithm is as follows:
Y t+1 ＝αX t +α(1-α)X t-1 +α(1-α) 2 X t-2 +…+α(1-α) n X t-n (1.1)
wherein, Y t+1 The predicted value of the operation/rule data is shown as alpha, a weight coefficient is shown as n, a smoothing index is shown as t, the number of observation periods of the operation/rule data is shown as t, and the observed value of the operation/rule data is shown as X.
The exponential smoothing value of any period is the weighted average of the actual observed value of the period and the exponential smoothing value of the previous period, the formula (1.1) is deduced, and smoothing is performed again on the basis of the first exponential smoothing to obtain a secondary exponential smoothing formula:
in the formulae (1.2) and (1.3),
is the first smoothed value for the t-th period,
is the second smoothed value of period t, X t The t-th job/rule observed value is t 1, 2, 3, …, n (n is the number of original data).
The parameters of the quadratic exponential smoothing formula include:
calculating the job data and the rule data by using the detection model;
as a preferable scheme of the present embodiment, job data of different formats input to the detection model, that is, photographs, characters, numbers, and the like, are sequentially developed in binary. The expanded job data is then divided and arranged, and each segment is marked according to a time sequence. Setting the observation period number of the expanded operation data as t, and setting the observation value of each observation period of the operation data as X t 、X t-1 、X t-2 …、X t-n . Data other than the observed values are ignored, thereby reducing the amount of data that needs to be compared. And carrying out unequal weight processing according with the actual situation on the data at different times, and dividing the data into different smoothing grades according to the smoothing times. Substituting the observed value and the observed period of the operation data into a detection model established based on an exponential smoothing algorithm to calculate a corresponding quadratic exponential smoothing parameter alpha t And b t 。
And similarly, converting the rule data into a binary system, sequentially expanding the binary system, segmenting and arranging the expanded rule data, and marking each segment according to the time sequence. The observation period number of the expanded rule data is set to t (the observation period number of the rule data is consistent with the observation period number of the operation data), and the observation value of each observation period of the rule data is set to P t 、P t-1 、P t-2 …、P t-n . Data other than the observed values are ignored, thereby reducing the amount of data that needs to be compared. And carrying out unequal weight processing according with the actual situation on the data at different times, and dividing the data into different smoothing grades according to the smoothing times. Substituting the regular data observation value and the regular data observation period number into a detection model established based on an exponential smoothing algorithm to calculate a corresponding quadratic exponential smoothing parameter Q alpha t And Qb t 。
And comparing the calculation results of the job data and the rule data so as to verify the data validity of the job data.
Respectively combine the parameters alpha t And parameter Q alpha t Parameter b t And parameter Qb t And (6) carrying out comparison. If the parameter α t And parameter Q alpha t Are coincident, and parameter b t Root of HeshenNumber Qb t And if the operation data of the part corresponding to the proof parameter is the same as the rule data and accords with the validity rule, judging that the part of the operation data passes the data validity verification. If the parameter α t And parameter Q alpha t Inconsistency, or/and parameter b t And parameter Qb t If the operation data is inconsistent with the rule data, the operation data corresponding to the parameters is different from the rule data, and the validity rule is not met, the operation data of the part is judged not to pass the data validity verification.
As a preferable aspect of this embodiment, the observation values extracted from the job data/rule data, the front part and the middle part of the extracted and expanded job data/rule data, or the middle part and the last part of the extracted and expanded job data/rule data are compared, and it is only necessary to ensure that the observation period t of the job data and the observation period t of the rule data are consistent regardless of the length of the data. And comparing the plurality of parameters of the operation data and the rule data for a plurality of times, and judging that the whole operation data passes the data validity verification as long as 80% of comparison results are consistent.
S30, if the operation data do not pass the validity verification, re-receiving the operation data transmitted by the central cloud;
s40, if the operation data pass the validity verification, analyzing the operation data, and calculating the data volume and the predicted transmission time of the operation data;
specifically, the analysis of the job data obtains the type, substantive content, and resource requirements of the job data. According to the type, the essential content and the resource requirement of the data operation, the operation data are deployed to the corresponding virtual machine to be executed, so that the execution efficiency is improved, the data volume and the predicted transmission time of the operation data are calculated, parameters are provided for subsequent port regulation and control and delayed sending, and the phenomena that the data transmission is invalid and the data transmission cannot be completed in time are effectively prevented. For example: the expected transmission time is set to be 10S for the job data of 20MB, if the transmission time exceeds the expected transmission time, the transmission is regarded as failure, and the edge cloud delays the secondary transmission of the job data after the failure of the transmission, so that the failure of data transmission is prevented.
S50, eliminating redundancy of the operation data according to the analysis result of the operation data;
specifically, the edge cloud analyzes the essence content from the job data, and eliminates the error and repeated parts of the essence content, thereby effectively shortening the data processing time. For example: and a plurality of groups of repeated photos appearing in the transmission data are analyzed and calculated to keep the photos meeting the transmission requirement, and the photos obtained by illegal ways are automatically removed in the transmission process.
S60, compressing and decomposing the job data after the redundancy is removed, and regulating and controlling the decomposed job data to corresponding ports according to the data volume and the predicted transmission time of the job data;
specifically, the operation data after being decomposed is reasonably regulated and controlled, and the operation data is dispatched to an idle/proper port for transmission, so that the phenomenon of data congestion during the transmission process of the operation data can be reduced, and the data circulation is facilitated. For example: the operation data is 6, the three ports are reasonably distributed, the output port starts to transmit the next data, and the transmission efficiency is effectively improved. Another example is: and the decomposed operation data is transmitted by using the idle port of the edge cloud, so that the occupation of a transmission channel in the transmission process is reduced.
And S70, transmitting the operation data positioned at the port to a terminal server.
Judging whether the terminal server receives the operation data;
if the terminal server does not finish receiving the operation data within the expected transmission time, the operation data is sent to the terminal server again;
and if the terminal server receives the operation data in the estimated transmission time, finishing the operation data transmission.
As a preferred scheme of this embodiment, in the process of transmitting executed job data to the terminal server, the edge cloud determines validity of the data by monitoring whether the terminal server receives the job data.
And after the terminal server receives the operation data, the edge cloud executes the data effectively, and the operation process is finished. When the job data has been transmitted to the terminal server but has not been received after exceeding the expected transmission time, the edge cloud sends an instruction that the job data has not been received to the output port. At this moment, the output port starts to perform secondary transmission on the data which is stored in the edge cloud and is transmitted in a delayed mode, and therefore the phenomenon that the data is lost in the transmission process is effectively prevented. In addition, the data transmitted in a delayed mode are automatically sent through the instruction, the problem that in the prior art, a data sending party and a data receiving party know the same data information asynchronously is solved, and data transmission in the edge cloud is facilitated. After the terminal server receives the operation data, the edge cloud receives a feedback instruction of the terminal server, the delayed transmission operation data is eliminated, and the defect of data redundancy of the edge cloud is effectively avoided.
As shown in fig. 3, the present embodiment provides a data processing system based on an edge cloud, including:
the edge cloud input module is used for receiving operation data transmitted by the center cloud;
the edge cloud monitoring module is used for verifying the validity of the operation data;
as a further aspect of the present invention, the edge cloud monitoring module includes:
the data legality unit is used for verifying the data legality of the operation data based on an exponential smoothing algorithm;
as a further aspect of the present invention, the data legality unit includes:
the initialization mechanism is used for establishing a detection model based on an exponential smoothing algorithm and setting rule data; the rule data are data in accordance with a legality rule;
a detection model mechanism for calculating the operation data and the rule data by using the detection model;
and the data comparison mechanism is used for comparing the calculation results of the operation data and the rule data so as to verify the data validity of the operation data.
A destruction recording unit configured to delete the job data that fails the data validity verification and save a transmission record of the job data;
a transmission legality unit for performing transmission legality verification on the job data passing the data legality verification;
and the data returning unit is used for returning the job data which fails to pass the transmission validity verification to the sending part.
As a further aspect of the present invention, the formula of the exponential smoothing algorithm is:
Y t+1 ＝αX t +α(1-α)X t-1 +α(1-α) 2 X t-2 +…+α(1-α) n X t-n
wherein Y is t+1 And alpha is a weight coefficient, n is a smoothing index, t is the number of observation periods of the operation/rule data, and X is an observation value of the operation/rule data.
The edge cloud analysis module is used for analyzing the job data passing the validity verification and calculating the data volume and the predicted transmission time of the job data;
the edge cloud computing module is used for eliminating redundancy of the operation data according to the analysis result of the operation data;
the edge cloud regulation and control module is used for compressing and decomposing the job data after the redundancy is removed, and regulating and controlling the decomposed job data to corresponding ports according to the data volume and the predicted transmission time of the job data;
and the edge cloud output module is used for transmitting the operation data positioned at the port to a terminal server.
And the edge cloud delay module is used for retransmitting the operation data which is not received by the terminal server within the expected transmission time.
It should be understood that the above-mentioned embodiments of the present invention are only examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific embodiments of the present invention. Any modification, equivalent replacement, and improvement made within the spirit and principle of the claims of the present invention should be included in the protection scope of the claims of the present invention.
Claims (10)
1. A data processing method based on edge cloud is characterized by comprising the following steps:
receiving operation data transmitted by a central cloud;
carrying out validity verification on the operation data;
if the operation data do not pass the validity verification, the operation data transmitted by the central cloud are received again;
if the operation data passes the validity verification, analyzing the operation data, and calculating the data volume and the predicted transmission time of the operation data;
eliminating redundancy of the operation data according to the analysis result of the operation data;
compressing and decomposing the job data after the redundancy is eliminated, and regulating and controlling the decomposed job data to a corresponding port according to the data volume and the predicted transmission time of the job data;
and transmitting the operation data positioned at the port to a terminal server.
2. The data processing method based on the edge cloud as claimed in claim 1, wherein the validation of the validity of the job data comprises:
performing data validity verification on the operation data based on an exponential smoothing algorithm;
if the operation data does not pass the data validity verification, judging that the operation data does not pass the data validity verification, deleting the operation data, and storing a transmission record of the operation data;
if the operation data passes the data validity verification, the transmission validity verification is carried out on the operation data;
if the operation data does not pass the transmission validity verification, judging that the operation data does not pass the validity verification, and returning the operation data to a sending part;
and if the operation data passes the transmission validity verification, judging that the operation data passes the validity verification.
3. The data processing method based on the edge cloud as claimed in claim 2, wherein the data validity verification of the job data based on the exponential smoothing algorithm comprises:
establishing a detection model based on an exponential smoothing algorithm, and setting rule data; the rule data are data in accordance with a legality rule;
calculating the job data and the rule data by using the detection model;
and comparing the calculation results of the job data and the rule data so as to verify the data validity of the job data.
4. The data processing method based on the edge cloud as claimed in claim 2, wherein the formula of the exponential smoothing algorithm is as follows:
Y t+1 ＝αX t +α(1-α)X t-1 +α(1-α) 2 X t-2 +…+α(1-α) n X t-n
wherein, Y t+1 The predicted value of the operation/rule data is shown as alpha, a weight coefficient is shown as n, a smoothing index is shown as t, the number of observation periods of the operation/rule data is shown as t, and the observed value of the operation/rule data is shown as X.
5. The data processing method based on the edge cloud according to claim 1, wherein after the job data at the port is transmitted to the terminal server, the method further comprises:
judging whether the terminal server receives the operation data;
if the terminal server does not receive the operation data within the estimated transmission time, the operation data is sent to the terminal server again;
and if the terminal server has received the operation data within the expected transmission time, finishing the operation data transmission.
6. An edge cloud based data processing system, comprising:
the edge cloud input module is used for receiving operation data transmitted by the center cloud;
the edge cloud monitoring module is used for verifying the validity of the operation data;
the edge cloud analysis module is used for analyzing the job data passing the validity verification and calculating the data volume and the predicted transmission time of the job data;
the edge cloud computing module is used for eliminating redundancy of the operation data according to the analysis result of the operation data;
the edge cloud regulation and control module is used for compressing and decomposing the job data after the redundancy is removed, and regulating and controlling the decomposed job data to corresponding ports according to the data volume and the predicted transmission time of the job data;
and the edge cloud output module is used for transmitting the operation data positioned at the port to a terminal server.
7. An edge cloud based data processing system according to claim 6, wherein said edge cloud monitoring module comprises:
the data legality unit is used for verifying the data legality of the operation data based on an exponential smoothing algorithm;
a record elimination unit configured to delete the job data that fails the data validity verification and save a transmission record of the job data;
a transmission legality unit for performing transmission legality verification on the job data passing the data legality verification;
and the data return unit is used for returning the job data which fails to pass the transmission validity verification to the sending part.
8. An edge cloud based data processing system according to claim 7, wherein said data lawful unit comprises:
the initialization mechanism is used for establishing a detection model based on an exponential smoothing algorithm and setting rule data; the rule data are data in accordance with a legality rule;
a detection model mechanism for calculating the operation data and the rule data by using the detection model;
and the data comparison mechanism is used for comparing the calculation results of the operation data and the rule data so as to verify the data validity of the operation data.
9. The edge cloud-based data processing system of claim 7, wherein the formula of the exponential smoothing algorithm is:
Y t+1 ＝αX t +α(1-α)X t-1 +α(1-α) 2 X t-2 +…+α(1-α) n X t-n
wherein, Y t+1 And alpha is a weight coefficient, n is a smoothing index, t is the number of observation periods of the operation/rule data, and X is an observation value of the operation/rule data.
10. An edge cloud based data processing system according to claim 6, further comprising:
and the edge cloud delay module is used for retransmitting the operation data which is not received by the terminal server within the expected transmission time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210624353.3A CN115022045B (en) | 2022-06-02 | 2022-06-02 | Data processing method and system based on edge cloud |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210624353.3A CN115022045B (en) | 2022-06-02 | 2022-06-02 | Data processing method and system based on edge cloud |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115022045A true CN115022045A (en) | 2022-09-06 |
| CN115022045B CN115022045B (en) | 2023-09-19 |
Family
ID=83072792
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210624353.3A Active CN115022045B (en) | 2022-06-02 | 2022-06-02 | Data processing method and system based on edge cloud |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115022045B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB0411588D0 (en) * | 2004-05-24 | 2004-06-23 | Toshiba Res Europ Ltd | Data transmission method |
| US20120303776A1 (en) * | 2011-05-27 | 2012-11-29 | James Michael Ferris | Methods and systems for data compliance management associated with cloud migration events |
| CN109714370A (en) * | 2019-03-07 | 2019-05-03 | 四川长虹电器股份有限公司 | A kind of implementation method based on http protocol end Yunan County full communication |
| US20200004503A1 (en) * | 2017-03-17 | 2020-01-02 | Mitsubishi Electric Corporation | Information processing device, information processing method, and computer readable medium |
| WO2020207264A1 (en) * | 2019-04-08 | 2020-10-15 | 阿里巴巴集团控股有限公司 | Network system, service provision and resource scheduling method, device, and storage medium |
| CN112187798A (en) * | 2020-09-28 | 2021-01-05 | 安徽大学 | Bidirectional access control method and system applied to cloud-side data sharing |
| CN112543187A (en) * | 2020-11-26 | 2021-03-23 | 齐鲁工业大学 | Industrial Internet of things safety data sharing method based on edge block chain |
| CN114238323A (en) * | 2021-12-11 | 2022-03-25 | 宜昌优智科技有限公司 | Internet of things data collection, cleaning, rating, transmission and storage method |
-
2022
- 2022-06-02 CN CN202210624353.3A patent/CN115022045B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB0411588D0 (en) * | 2004-05-24 | 2004-06-23 | Toshiba Res Europ Ltd | Data transmission method |
| US20120303776A1 (en) * | 2011-05-27 | 2012-11-29 | James Michael Ferris | Methods and systems for data compliance management associated with cloud migration events |
| US20200004503A1 (en) * | 2017-03-17 | 2020-01-02 | Mitsubishi Electric Corporation | Information processing device, information processing method, and computer readable medium |
| CN109714370A (en) * | 2019-03-07 | 2019-05-03 | 四川长虹电器股份有限公司 | A kind of implementation method based on http protocol end Yunan County full communication |
| WO2020207264A1 (en) * | 2019-04-08 | 2020-10-15 | 阿里巴巴集团控股有限公司 | Network system, service provision and resource scheduling method, device, and storage medium |
| CN112187798A (en) * | 2020-09-28 | 2021-01-05 | 安徽大学 | Bidirectional access control method and system applied to cloud-side data sharing |
| CN112543187A (en) * | 2020-11-26 | 2021-03-23 | 齐鲁工业大学 | Industrial Internet of things safety data sharing method based on edge block chain |
| CN114238323A (en) * | 2021-12-11 | 2022-03-25 | 宜昌优智科技有限公司 | Internet of things data collection, cleaning, rating, transmission and storage method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115022045B (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111401903B (en) | Block chain message processing method, device, computer and readable storage medium | |
| CN110704518B (en) | Business data processing method and device based on Flink engine | |
| JP6587330B2 (en) | Random forest model training method, electronic apparatus, and storage medium | |
| CN109343942B (en) | Task scheduling method based on edge computing network | |
| CN113315700A (en) | Computing resource scheduling method, device and storage medium | |
| CN104901989B (en) | A kind of Site Service offer system and method | |
| CN110134350B (en) | Print file segmentation and transmission method | |
| CN112053092B (en) | Work order processing method, device, readable medium and equipment | |
| CN110011930A (en) | The load-balancing method and device of multi-joint alliance's chain in a kind of block chain | |
| CN112543145A (en) | Method and device for selecting communication path of equipment node for sending data | |
| CN112583715A (en) | Equipment node connection adjustment method and device | |
| CN116506953A (en) | Multi-channel switching method, system and medium applied to intelligent communication system | |
| CN115189910A (en) | Network digital twin-based deliberate attack survivability evaluation method | |
| WO2022257366A1 (en) | Network slice self-optimization method, base station, and storage medium | |
| CN109740789B (en) | Wiring management method, device, equipment and storage medium | |
| CN115022205A (en) | Cross-network data transmission method applied to high-concurrency scene of massive terminals | |
| CN116367223B (en) | XR service optimization method and device based on reinforcement learning, electronic equipment and storage medium | |
| CN115022045A (en) | Data processing method and system based on edge cloud | |
| CN114070855B (en) | Resource allocation method, resource allocation device, resource allocation system, and storage medium | |
| CN117040837B (en) | Business risk processing method combining artificial intelligence | |
| CN116599862B (en) | Communication method, analysis network element and communication system | |
| CN113746899B (en) | Cloud platform access method and device | |
| CN117596126B (en) | Monitoring method for high-speed network abnormality in high-performance cluster | |
| CN117290099B (en) | Computing resource adjustment method and device for cloud platform | |
| CN111049919B (en) | User request processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20220906 Assignee: Shanghai Haoyun Changsheng Data Service Co.,Ltd. Assignor: LIANTONG (GUANGDONG) INDUSTRY INTERNET Co.,Ltd. Contract record no.: X2024980000732 Denomination of invention: A data processing method and system based on edge cloud Granted publication date: 20230919 License type: Common License Record date: 20240118 |