CN115202867A - Data processing method and system based on edge calculation - Google Patents
Data processing method and system based on edge calculation Download PDFInfo
- Publication number
- CN115202867A CN115202867A CN202210647720.1A CN202210647720A CN115202867A CN 115202867 A CN115202867 A CN 115202867A CN 202210647720 A CN202210647720 A CN 202210647720A CN 115202867 A CN115202867 A CN 115202867A
- Authority
- CN
- China
- Prior art keywords
- data
- edge
- task
- calculation
- stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004364 calculation method Methods 0.000 title claims abstract description 57
- 238000003672 processing method Methods 0.000 title claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 61
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000001914 filtration Methods 0.000 claims description 29
- 230000006870 function Effects 0.000 claims description 20
- 239000013598 vector Substances 0.000 claims description 14
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000013075 data extraction Methods 0.000 claims description 6
- 238000009826 distribution Methods 0.000 claims description 4
- 239000011159 matrix material Substances 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 9
- 238000004590 computer program Methods 0.000 description 7
- 238000007781 pre-processing Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000003860 storage Methods 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 239000000523 sample Substances 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/502—Proximity
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method and a system based on edge calculation, wherein the method comprises the following steps: the terminal equipment acquires data, generates a sampled data sample and sends the sampled data sample to the edge server; the edge server receives a data sample to be processed, calculates and classifies the data sample, filters out a data set needing to be processed and generates a corresponding data processing matrix; the edge server distributes a corresponding data processing matrix for the edge equipment based on the processing capacity of the edge equipment; and the edge equipment performs calculation task processing according to the data processing matrix distributed by the edge server. According to the method and the device, the data samples are identified and processed at the edge server, so that the data needing to be further processed by the edge device are filtered, and the edge calculation efficiency is improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data processing method and system based on edge calculation.
Background
With the development of the current internet of things technology, the data volume requested by a user shows an exponential growth trend, and great challenges are brought to the traditional cloud computing technology, so that edge computing is produced at the same time. The edge computing technology is to provide edge intelligent services nearby by a distributed open platform fusing network, computing, storage and application core capabilities at the edge side of a network close to an object or a data source. The edge computing platform is connected between the control end computing platform or the cloud computing platform and the equipment end, and can directly conduct corresponding processing on data acquired by the terminal, so that the data processing process becomes safer and more efficient.
However, the traditional edge computing technology has a great defect in data processing, data of the internet of things is increased explosively at present, some redundant data do not meet the processing requirements of users, and due to the limitation of computing power of edge nodes, great delay is generated in the processing of the redundant data, great equipment power consumption is brought, and the requirements of real-time processing and analysis of some data cannot be met.
Disclosure of Invention
The purpose of the invention is as follows: aiming at the defects of the data processing technology in the traditional edge calculation mode, the invention provides a data processing method based on edge calculation, which can realize real-time processing and analysis of data and improve the performance of edge calculation
The invention also provides a data processing system based on edge calculation and computer equipment.
The technical scheme is as follows: according to a first aspect of the present invention, a data processing method based on edge calculation includes the steps of:
the terminal equipment acquires data and sends the acquired data stream to an edge server;
the edge server carries out the following preprocessing on the received data stream: data marking is carried out on the data stream to obtain a data identifier
Identifying generated data according to task requirements
Matching, filtering the data sets W (i) meeting the task requirements, and applying data disturbance change to all the data sets W (i) to generate a final data set R (t);
the edge server generates a calculation task according to the data set R (t), and allocates a corresponding calculation task for the edge equipment based on the processing capacity of the edge equipment;
and the edge equipment carries out calculation task processing according to the distributed calculation tasks.
Preferably, the data stream is subjected to data marking to obtain data identification
The method comprises the following steps: inputting the ith terminal equipment into the jth data stream of the edge server by using a data characteristic conversion function dia function
Performing characteristic conversion, and performing orthogonal operation on the data stream subjected to the dia function and the data identification constant to generate the identification of the jth data stream of the terminal device i
Preferably, the generated data are identified according to task requirements
Matching is carried out, and the step of filtering the data set W (i) meeting the task requirement comprises the following steps:
generating a corresponding demand vector S based on task demands, and calculating data identification through a calculation formula based on vector included angle
Similarity to the demand vector S
Will be provided with
And filtering a data stream W (i) meeting the task requirement of each terminal device by using an input data filtering function LMD, wherein the similarity calculation formula is as follows:
preferably, the data filtering function establishes a processing queue for the data stream identifier of each terminal device in a sliding filtering manner, stores data meeting requirements according to the similarity of the data stream identifier of each terminal device, and removes data identifiers not meeting requirements from the processing queue to obtain a filtered data set W (i) of the terminal device i.
Preferably, the applying the data perturbation variation to all the data sets W (i) to generate the final data set R (t) comprises:
R(t)＝N t W(i)
wherein, N t Representing the multiple of data perturbation interference applied at time t.
Preferably, the method further comprises: and inputting the generated data set R (t) into a graphical interface to obtain a global data view which changes along with time.
According to a second aspect of the invention, an edge-computation-based data processing system comprises:
the terminal equipment is used for acquiring data and sending the acquired data stream to the edge server;
the edge server is used for preprocessing the received data stream as follows: data marking is carried out on the data stream to obtain a data identifier
Identifying generated data according to task requirements
Matching is carried out, a data set W (i) meeting the task requirement is filtered, and the task is matchedApplying data disturbance change to a data set W (i) to generate a final data set R (t), generating a calculation task according to the data set R (t), and distributing a corresponding calculation task for the edge equipment based on the processing capacity of the edge equipment;
and the edge equipment is used for processing the calculation task according to the distributed calculation task.
According to a third aspect of the present invention, a data processing method performed by an edge server, comprises the steps of:
performing data marking on input data stream to obtain data identification
Identifying generated data according to task requirements
Matching is carried out, and a data set W (i) meeting the task requirement is filtered;
applying data perturbation variation to all the data sets W (i) to generate a final data set R (t);
and generating a computing task according to the data set R (t), and distributing the corresponding computing task to the edge device based on the processing capacity of the edge device.
According to a fourth aspect of the present invention, there is provided a computer apparatus comprising:
a data identification module for marking the input data stream to obtain data identification
A data filtering module for marking the generated data according to task requirements
Matching is carried out, and a data set W (i) meeting the task requirement is filtered;
the data extraction module is used for applying data disturbance change to all the data sets W (i) to generate a final data set R (t);
and the data distribution module is used for generating a calculation task according to the data set R (t) and distributing a corresponding calculation task to the edge equipment based on the processing capacity of the edge equipment.
According to a fifth aspect of the present invention, there is provided a computer apparatus comprising:
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured for execution by the one or more processors, which when executed by the processors implement the steps of the data processing method of claim 8.
Has the advantages that: the invention adopts a data identification processing method at the edge server to quickly calculate and classify the perception data of the terminal equipment, filters out a data set which needs to be further calculated and processed by the edge node in advance, generates a corresponding data processing matrix, and redistributes the data processing matrix to the edge node for processing, thereby reducing the processing of redundant data and reducing the power consumption of the equipment. In addition, the calculation result obtained by the self-adaptive algorithm can be directly visualized, and a background management end can conveniently and directly monitor data change.
Drawings
FIG. 1 is a diagram of an edge computation based data processing system architecture of the present invention;
FIG. 2 is a general flowchart of a data processing method based on edge calculation according to the present invention;
fig. 3 is a flow chart of the preprocessor for data identification on the edge server according to the present invention.
Detailed Description
The technical scheme of the invention is further explained by combining the attached drawings.
As shown in FIG. 1, the data processing system architecture based on edge calculation of the present invention is mainly divided into three layers: the smart phone comprises a cloud computing layer, an edge computing layer and a terminal layer, wherein the bottom layer is the terminal layer and comprises various types of terminal equipment, such as a smart phone, a tablet computer, a portable computer, a Personal Digital Assistant (PDA), an intelligent wearable device, a game console, a netbook, a super book and the like, and the equipment is already deployed in various intelligent scenes and can acquire various types of data such as images and videos in real time. And after the data are collected by each terminal device, corresponding data streams can be generated, the data streams can be uploaded to an edge server of an edge computing layer, and the data streams are distributed to corresponding edge computing node devices by the server according to a data identification processing algorithm for processing.
The middle layer is an edge computing layer and is composed of edge devices and edge servers, wherein the edge devices, edge nodes and edge node devices are used interchangeably herein and refer to the intermediate devices which comprise routers, gateways, switches, access points and the like and provide terminal access network functions, and the devices are used as nodes of edge computing and perform computing tasks with self computing power according to the distribution of the edge servers. The edge server is a bridge between the cloud server and the terminal equipment, receives data acquired by each terminal equipment, performs data preprocessing by the self-adaptive data processing method provided by the invention, and distributes corresponding computing tasks to each edge node equipment, thereby reducing the equipment load of each edge node. And forwarding the residual data which cannot be processed by the edge computing layer to the cloud computing layer for processing.
The top layer is a cloud computing layer and consists of high-end servers and a data center. The cloud computing layer has strong computing and storing capacity, is responsible for computing and storing residual data which cannot be processed by the edge computing layer, stores data processing results, and can be called or retrieved by accessing a data center of the cloud computing layer according to requirements.
Fig. 2 shows a data processing method based on edge computing according to the present invention, which improves data processing efficiency by deploying a self-adaptive preprocessing method in an edge server based on the above system. Referring to fig. 2, the method includes:
step one, after a user initiates a data processing requirement, a request data packet is sent to an edge server, the edge server analyzes data of the request data packet and receives a user task requirement, terminal equipment acquires data such as images and videos according to the user requirement, and after each equipment acquires the data, a corresponding data stream is generated and sent to the edge server of an edge computing layer.
Step two, the edge server receives the data sample to be processed, and pre-processes the data sample according to the adaptive data processing method provided by the invention, and the pre-processing method is as shown in fig. 3, and specifically comprises the following steps:
a data identification step: let X denote the data set input to the edge server, k denote the number of terminal devices collecting the data, F denote the number of data streams generated by each terminal device,
the method comprises the steps of representing the jth data stream of an ith terminal device input edge server, taking lambda as a data identification constant which is a fixed value, representing a data characteristic conversion function by dia, vectorizing the input data stream, simultaneously simply screening the data stream, and carrying out orthogonal operation on the sampled data vectors of all devices passing through the dia function and the data identification constant, thereby carrying out data marking on the data stream generated by a sampling device and generating a data stream identification corresponding to each terminal device
Formalized as follows:
the data identification step can further regularly process complicated data streams, and is convenient for calling and retrieving data in subsequent steps, so that the defect of disordered results of a traditional data processing algorithm is overcome.
And (3) data filtering: because the data collected by the terminal equipment are more, the data flow entering the edge equipment cannot be guaranteed to be the calculation data meeting the requirements, the data which do not meet the calculation task can be removed in advance through the data filtering step, and the calculation pressure of the edge equipment is reduced. After a user initiates a data processing request, a request packet is sent to the edge server, the edge server parses the data of the request packet, receives the user task request, parses the user request (e.g.,whether to process data in a picture format, an audio format or other formats) and generate a corresponding demand vector S containing the types of all data streams required by the user, calculate a data identifier based on a calculation formula of a vector angle
Similarity to the demand vector S
Equivalent to a match between the incoming data stream and the user's requirements, and then will
The input data filter function LMD filters out the data stream W (i) that each terminal device meets the task requirements. The calculation method of data filtering is shown in the following two formulas:
in the embodiment of the invention, a sliding filtering mode is adopted, a processing queue is established for a data stream mark of each terminal device, data meeting the requirement is stored according to the similarity of the data stream mark of each terminal device, and data marks not meeting the requirement are directly removed from the processing queue to obtain a data set W (i) filtered by each group of terminal devices.
A data extraction step: because interference such as network environment and the like can have certain disturbance in the acquisition and uploading process of the terminal equipment, and the data disturbance can bring change of certain data quantity, a certain error can exist in a data set obtained by filtering, an additional calculation task is added, and in the data extraction step, the data fluctuation interference multiple N is defined in the invention t N represents a difference in data amount change before and after data disturbanceThe data set can be obtained by a measurement module arranged on the edge device and multiplied by a data fluctuation interference multiple N before the filtered data set t And removing the influence of data disturbance, and extracting a final data set R (t) as shown in the following formula:
where k is the number of terminal devices generating the data stream.
Data visualization step: and inputting the extracted data set R (t) into a graphical interface GUI (graphical user interface), so as to obtain a global data view which changes along with time, and achieve the purpose of visualizing data.
And step three, filtering redundant data which do not meet task requirements and interference data in the transmission process in advance through a preprocessing flow, storing the extracted data set by the edge server to generate a corresponding calculation task, deploying a corresponding probe in an edge calculation system network to measure transmission delay and memory of each edge node, obtaining the calculation power of each edge device according to the measurement result uploaded by the edge server, splitting the extracted data set according to the calculation power, distributing the corresponding calculation task to each node, and uploading the processed data and the residual data which cannot be processed to the cloud calculation layer by each node.
And step four, the cloud server further calculates and processes residual data which cannot be processed by the edge calculation layer, stores the data processing results of all edge nodes and completes all data processing tasks.
The invention adopts a data identification processing method at the edge server to quickly calculate and classify the perception data of the terminal equipment, filters out a data set which needs to be further calculated and processed by the edge node in advance, generates a corresponding data processing matrix, and redistributes the data processing matrix to the edge node for processing, thereby reducing the processing of redundant data and reducing the power consumption of the equipment.
The present invention also provides a computer apparatus comprising:
a data identification module for performing data marking on the input data streamRecording to obtain data identification
A data filtering module for marking the generated data according to task requirements
Matching is carried out, and a data set W (i) meeting the task requirement is filtered;
the data extraction module is used for applying data disturbance change to all the data sets W (i) to generate a final data set R (t);
and the data distribution module generates a calculation task according to the data set R (t), and distributes a corresponding calculation task for the edge equipment based on the processing capacity of the edge equipment.
Further, the data identification module specifically includes: a feature conversion unit for inputting the ith terminal device into the jth data stream of the edge server by using the data feature conversion function dia function
Carrying out feature conversion; and the arithmetic unit is used for carrying out orthogonal operation on the data stream subjected to the dia function and the data identification constant to generate the identification of the jth data stream of the terminal device i
The data filtering module specifically comprises:
the demand vector determining unit is used for generating a corresponding demand vector S based on the task demand;
a similarity calculation unit for calculating data identification by calculation formula based on vector angle
Similarity to the demand vector S
The calculation formula is as follows:
a data filtering unit for filtering the data
The input data filtering function LMD filters out the data stream W (i) that each terminal device meets the task requirements,
and the data filtering function of the data filtering unit adopts a sliding filtering mode to establish a processing queue for the data stream identifier of each terminal device, stores the data meeting the requirements according to the similarity of the data stream identifier of each terminal device, and removes the data identifier not meeting the requirements from the processing queue to obtain a filtered data set W (i) of the terminal device i.
The data extraction module applies the data perturbation variation to all the data sets W (i) to generate a final data set R (t) comprising:
R(t)＝N t W(i)
wherein, N t Representing the multiple of data perturbation interference applied at time t.
Preferably, the apparatus further comprises a visualization module for inputting the generated data set R (t) into a graphical interface to obtain a time-varying global data view.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Claims (10)
1. A data processing method based on edge calculation, comprising the steps of:
the terminal equipment acquires data and sends the acquired data stream to an edge server;
the edge server preprocesses the received data stream as follows: data marking is carried out on the data stream to obtain a data identifier
Identifying generated data according to task requirements
Matching, filtering the data sets W (i) meeting the task requirements, and applying data disturbance change to all the data sets W (i) to generate a final data set R (t);
the edge server generates a calculation task according to the data set R (t), and allocates a corresponding calculation task to the edge equipment based on the processing capacity of the edge equipment;
and the edge equipment processes the calculation task according to the distributed calculation task.
2. The method of claim 1, wherein data marking is performed on the data stream to obtain a data identifier
The method comprises the following steps: inputting the ith terminal equipment into the jth data stream of the edge server by using a data characteristic conversion function dia function
Performing characteristic conversion, and performing orthogonal operation on the data stream subjected to the dia function and the data identification constant to generate the identification of the jth data stream of the terminal device i
3. The method of claim 1, wherein the generated data is identified according to task requirements
Matching is carried out, and the step of filtering out the data set W (i) meeting the task requirement comprises the following steps:
generating a corresponding demand vector S based on task demands, and calculating data identification through a calculation formula based on vector included angle
Similarity to the demand vector S
Will be provided with
And filtering a data stream W (i) meeting the task requirement of each terminal device by using an input data filtering function LMD, wherein the similarity calculation formula is as follows:
4. the method according to claim 3, wherein the data filtering function uses a sliding filtering method to establish a processing queue for the data stream identifier of each terminal device, store the data meeting the requirement according to the similarity of the data stream identifier of each terminal device, and remove the data identifier not meeting the requirement from the processing queue to obtain the filtered data set W (i) of the terminal device i.
5. The method of claim 1, wherein applying the data perturbation variation to all the data sets W (i) to generate the final data set R (t) comprises:
R(t)＝N t W(i)
wherein, N t Representing the multiple of data perturbation interference applied at time t.
6. The method of claim 1, further comprising: and inputting the generated data set R (t) into a graphical interface to obtain a global data view which changes along with time.
7. A data processing system based on edge computing, comprising:
the terminal equipment is used for acquiring data and sending the acquired data stream to the edge server;
edge server for received dataThe stream is pretreated as follows: data marking is carried out on the data stream to obtain a data identifier
Identifying generated data according to task requirements
Matching, filtering out data sets W (i) meeting task requirements, applying data disturbance change to all the data sets W (i) to generate a final data set R (t), generating a calculation task according to the data set R (t), and distributing corresponding calculation tasks to the edge equipment based on the processing capacity of the edge equipment;
and the edge equipment is used for processing the calculation task according to the distributed calculation task.
8. A data processing method performed by an edge server, comprising the steps of:
performing data marking on input data stream to obtain data identification
Identifying generated data according to task requirements
Matching is carried out, and a data set W (i) meeting the task requirement is filtered;
applying data perturbation variation to all the data sets W (i) to generate a final data set R (t);
and generating a computing task according to the data set R (t), and distributing the corresponding computing task to the edge equipment based on the processing capacity of the edge equipment.
9. A computer device, comprising:
a data identification module for marking the input data stream to obtain data identification
A data filtering module for marking the generated data according to task requirements
Matching is carried out, and a data set W (i) meeting the task requirement is filtered;
the data extraction module is used for applying data disturbance change to all the data sets W (i) to generate a final data set R (t);
and the data distribution module generates a calculation task according to the data set R (t), and distributes a corresponding calculation task for the edge equipment based on the processing capacity of the edge equipment.
10. A computer device, comprising:
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored in the memory and configured for execution by the one or more processors, the programs when executed by the processors implement the steps of the data processing method as claimed in claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210647720.1A CN115202867A (en) | 2022-06-09 | 2022-06-09 | Data processing method and system based on edge calculation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210647720.1A CN115202867A (en) | 2022-06-09 | 2022-06-09 | Data processing method and system based on edge calculation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115202867A true CN115202867A (en) | 2022-10-18 |
Family
ID=83575352
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210647720.1A Pending CN115202867A (en) | 2022-06-09 | 2022-06-09 | Data processing method and system based on edge calculation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115202867A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110391958A (en) * | 2019-08-15 | 2019-10-29 | 北京中安智达科技有限公司 | A kind of pair of network encryption flow carries out feature extraction automatically and knows method for distinguishing |
| CN111565225A (en) * | 2020-04-27 | 2020-08-21 | 银河水滴科技（北京）有限公司 | Figure action track determination method and device |
| US20210096911A1 (en) * | 2020-08-17 | 2021-04-01 | Essence Information Technology Co., Ltd | Fine granularity real-time supervision system based on edge computing |
| CN112801136A (en) * | 2020-12-31 | 2021-05-14 | 广州技象科技有限公司 | Internet of things gateway data processing method and device with feature identification |
| CN114003596A (en) * | 2021-11-16 | 2022-02-01 | 国家工业信息安全发展研究中心 | Multi-source heterogeneous data processing system and method based on industrial system |
-
2022
- 2022-06-09 CN CN202210647720.1A patent/CN115202867A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110391958A (en) * | 2019-08-15 | 2019-10-29 | 北京中安智达科技有限公司 | A kind of pair of network encryption flow carries out feature extraction automatically and knows method for distinguishing |
| CN111565225A (en) * | 2020-04-27 | 2020-08-21 | 银河水滴科技（北京）有限公司 | Figure action track determination method and device |
| US20210096911A1 (en) * | 2020-08-17 | 2021-04-01 | Essence Information Technology Co., Ltd | Fine granularity real-time supervision system based on edge computing |
| CN112801136A (en) * | 2020-12-31 | 2021-05-14 | 广州技象科技有限公司 | Internet of things gateway data processing method and device with feature identification |
| CN114003596A (en) * | 2021-11-16 | 2022-02-01 | 国家工业信息安全发展研究中心 | Multi-source heterogeneous data processing system and method based on industrial system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020078385A1 (en) | Data collecting method and apparatus, and storage medium and system | |
| CN111858869B (en) | Data matching method and device, electronic equipment and storage medium | |
| CN113360554A (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN104391879B (en) | The method and device of hierarchical clustering | |
| CN112036577A (en) | Method and device for application machine learning based on data form and electronic equipment | |
| CN115118653A (en) | Real-time service traffic classification method and system based on multi-task learning | |
| CN105488406A (en) | Similar malicious sample file matching method and system based on feature vector | |
| CN111091106A (en) | Image clustering method and device, storage medium and electronic device | |
| CN114817243A (en) | Method, device and equipment for establishing database joint index and storage medium | |
| CN111931679A (en) | Action recognition method, device, equipment and storage medium | |
| CN116662875A (en) | Interface mapping method and device | |
| CN117097578B (en) | Network traffic safety monitoring method, system, medium and electronic equipment | |
| CN111737371B (en) | Data flow detection classification method and device capable of dynamically predicting | |
| CN112822121A (en) | Traffic identification method, traffic determination method and knowledge graph establishment method | |
| CN115202867A (en) | Data processing method and system based on edge calculation | |
| CN106407126B (en) | Processing method, device and the server of coverage data | |
| CN113762049A (en) | Content identification method and device, storage medium and terminal equipment | |
| CN117527805A (en) | Multi-mode data access optimization method and system based on server cluster | |
| CN110909072B (en) | Data table establishment method, device and equipment | |
| CN116010447A (en) | Load balancing method and device for optimizing heterogeneous database user query | |
| CN114863153A (en) | Image similarity data cleaning method and system based on deep learning | |
| Takano et al. | Privacy-protective distributed machine learning using rich clients | |
| CN114548229A (en) | Training data augmentation method, device, equipment and storage medium | |
| CN113946717A (en) | Sub-map index feature obtaining method, device, equipment and storage medium | |
| CN112765010A (en) | Method, device, equipment and storage medium for centralized management of service parameters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |