CN110750510A - Internet of things data processing system and method - Google Patents
Internet of things data processing system and method Download PDFInfo
- Publication number
- CN110750510A CN110750510A CN201910981448.9A CN201910981448A CN110750510A CN 110750510 A CN110750510 A CN 110750510A CN 201910981448 A CN201910981448 A CN 201910981448A CN 110750510 A CN110750510 A CN 110750510A
- Authority
- CN
- China
- Prior art keywords
- data
- cluster
- data processing
- processing
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Abstract
The invention discloses a data processing system and a data processing method of an Internet of things, which relate to the field of network data calculation, and are used for counting data acquired by monitoring and distributing the data to a preprocessing cluster for preprocessing the data, classifying the preprocessed data and distributing the classified data to different production ends of a production end cluster for data processing, enabling the data processed by the production end cluster to flow into a queue cluster for caching, and then processing the data of a consumption end cluster to finish the data processing process; the method divides the data processing process into a plurality of processing units, monitors and deals with the data, preprocesses the data, processes the data at a production end and processes the data at a consumption end, so that the data calculation conditions can be flexibly configured; on the premise of meeting the data processing requirements of most terminals, the universal processing logic can be fully packaged, clustering and distribution are supported, and the problem that developers cannot concentrate on own services is solved.
Description
Technical Field
The invention discloses a data processing system and method of the Internet of things, and relates to the field of network data calculation.
Background
With the development of the internet of things technology, data interaction between various terminal devices and a server becomes a main melody for network computing. At present, the data processing process of the Internet of things mainly comprises two steps: the method comprises the steps of firstly, collecting network data for different terminals and preprocessing the network data, and secondly, dispersing the data into different queues and different topics according to different data types for processing. The problems in the above two steps are: (1) the method comprises the steps that protocols supported by terminal equipment produced by different manufacturers are different, great development cost is caused by protocol analysis, (2) a data processing process comprises a general processing logic, developers cannot concentrate on development of own services, (3) the data processing process is complex, the developers need to use the existing network communication technology to disperse the processing process, unnecessary development cost is caused, and (4) configuration items in the data processing process use configuration files or hard codes, flexible configuration is not carried out by using a Web management program, and adaptability is poor.
The invention discloses a data processing system and a method of the Internet of things, which are used for converging network data from different terminals, carrying out cluster processing and distributed processing, dividing the data processing process into a plurality of processing units, wherein each processing unit encapsulates general processing logic and can reserve an interface needing personalized processing; after data flow in, the data can be filtered, monitored, responded, preprocessed, processed by a production end and processed by a consumption end; in the data interaction process, the preprocessing, the production end processing and the consumption end processing can be carried out according to the early warning prompt in the monitoring and coping stage and the number of the elastic telescopic nodes.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides an Internet of things data processing system and method, aiming at ensuring that most terminal data processing requirements are met, packaging general processing logic, supporting clustering and distribution and solving the problem that developers are not easy to concentrate on own services.
The specific scheme provided by the invention is as follows:
a data processing method of the Internet of things comprises the following steps:
the data of the terminal device is acquired,
the data obtained by statistical monitoring is distributed to a preprocessing cluster for preprocessing the data,
the preprocessed data are distributed to different production ends of the production end cluster for data processing after being classified,
and the data processed by the production end cluster flows into the queue cluster for caching, and then the data processing of the consumption end cluster is carried out, so that the data processing process is completed.
The method comprises the steps of judging whether the acquired data meet filtering conditions or not after the data of the terminal equipment are acquired, filtering the data meeting the filtering conditions, and otherwise, directly carrying out data statistics monitoring.
The method obtains the early warning index through comprehensive analysis of effective data flow and data processing indexes, defines the early warning condition according to the early warning index, achieves the early warning condition when counting monitoring data, and rebounds the data.
The data after pretreatment in the method are classified according to a dispersion mode, a balance strategy and data type comprehensive judgment, and are distributed to different generation ends in a production end cluster according to classification to be subjected to data processing.
According to the method, the number of the upper line and the lower line of the nodes in the production end cluster and the consumption end cluster is controlled according to the rebound data condition.
An internet of things data processing system comprises an acquisition module, a statistical monitoring module and a data processing module, wherein the data processing module comprises a preprocessing module, a production end processing module and a consumption end processing module,
the acquisition module acquires data of the terminal device,
the statistic monitoring module counts and monitors the acquired data and distributes the data to the preprocessing cluster controlled by the preprocessing module for preprocessing the data,
the preprocessed data are distributed to different production ends of the production end cluster controlled by the production end processing module after being classified for data processing,
and the data processed by the production end cluster flows into the queue cluster for caching, and then the data processing process is completed through the data processing of the consumption end cluster controlled by the consumption end processing module.
The system comprises an acquisition module, a data processing module and a data processing module, wherein the acquisition module acquires data of the terminal equipment and then judges whether the acquired data meet filtering conditions or not, the data meeting the filtering conditions are filtered, and otherwise, data statistics monitoring is directly carried out.
The system is characterized in that a statistical monitoring module obtains an early warning index through comprehensive analysis of effective data flow and data processing indexes, defines an early warning condition according to the early warning index, achieves the early warning condition when monitoring data are counted, and rebounds the data.
The system is characterized in that a preprocessing module classifies preprocessed data according to a dispersion mode, a balance strategy and data type comprehensive judgment, and distributes the data to different generation ends in a production end cluster according to classification for data processing.
And the statistical monitoring module in the system controls the number of the upper line and the lower line of the nodes in the production end cluster and the consumption end cluster according to the rebound data condition.
The invention has the advantages that:
the invention provides a data processing method of an Internet of things, which is characterized in that data obtained by monitoring is counted and distributed to a preprocessing cluster for preprocessing of the data, the preprocessed data are distributed to different production ends of a production end cluster for data processing after being classified, the data processed by the production end cluster flow into a queue cluster for caching, then the data processing of a consumption end cluster is carried out, and the data processing process is completed; the method divides the data processing process into a plurality of processing units, monitors and deals with the data, preprocesses the data, processes the data at a production end and processes the data at a consumption end, so that the data calculation conditions can be flexibly configured; on the premise of meeting the data processing requirements of most terminals, the universal processing logic can be fully packaged, clustering and distribution are supported, and the problem that developers cannot concentrate on own services is solved.
Drawings
FIG. 1 is a schematic flow diagram of the process of the present invention.
Detailed Description
The invention provides a data processing method of the Internet of things, which comprises the following steps:
the data of the terminal device is acquired,
the data obtained by statistical monitoring is distributed to a preprocessing cluster for preprocessing the data,
the preprocessed data are distributed to different production ends of the production end cluster for data processing after being classified,
and the data processed by the production end cluster flows into the queue cluster for caching, and then the data processing of the consumption end cluster is carried out, so that the data processing process is completed.
Meanwhile, an Internet of things data processing system corresponding to the method is provided, which comprises an acquisition module, a statistic monitoring module and a data processing module, wherein the data processing module comprises a preprocessing module, a production end processing module and a consumption end processing module,
the acquisition module acquires data of the terminal device,
the statistic monitoring module counts and monitors the acquired data and distributes the data to the preprocessing cluster controlled by the preprocessing module for preprocessing the data,
the preprocessed data are distributed to different production ends of the production end cluster controlled by the production end processing module after being classified for data processing,
and the data processed by the production end cluster flows into the queue cluster for caching, and then the data processing process is completed through the data processing of the consumption end cluster controlled by the consumption end processing module.
The present invention is further described below in conjunction with the following figures and specific examples so that those skilled in the art may better understand the present invention and practice it, but the examples are not intended to limit the present invention.
When the method is used for processing the data of the Internet of things, a Web application program can be used for managing the data processing process, various parameters in the data processing process are configured and stored in a database, the database supports the running of the Web application program, data generated by the Web application program and calculation data in a cache can be stored in the database, meanwhile, configuration information needs to be frequently read from the cache in the data processing process, and the configuration information in the database can be refreshed in real time or at regular time into the cache; meanwhile, the invention can also utilize a network communication framework to receive data and coordinate the work of internal programs, the network communication framework is used as an entrance of terminal equipment data, provides thread groups, receives the terminal equipment data, and provides communication services such as cross-server communication and the like;
by using the method of the invention to acquire data from the terminal equipment through the network communication framework,
the data obtained by statistical monitoring is distributed to a preprocessing cluster for preprocessing the data,
the preprocessed data are distributed to different production ends of the production end cluster for data processing after being classified,
the data processed by the production end cluster flows into the queue cluster cache, the queue cluster cache is convenient for the consumption end cluster to use in the Web background program, so that the consumption end node in the consumption end cluster has enough time to process the data, and is convenient for other statistical analysis programs to pull the data from the queue,
and the consumption end node in the consumption end cluster acquires the data to be processed from the queue cluster cache to complete the data processing process.
In the process, after the data of the terminal equipment is acquired, whether the acquired data meets the filtering condition can be judged, the data meeting the filtering condition is filtered, otherwise, the data statistical monitoring can be directly carried out,
when statistical monitoring is carried out, early warning indexes can be obtained through comprehensive analysis of effective data flow and data processing indexes, early warning conditions are defined according to the early warning indexes, the early warning conditions are met, rebound data are carried out, a default strategy of the rebound data is to receive one filter for the same type of data, and the number of the upper line and the lower line of nodes in a production end cluster and a consumption end cluster is controlled by a Web application program according to the rebound data condition.
Based on the above process, the process of data processing by the method of the present invention is described by taking the data reported by the shared automobile as an example:
s1: the shared automobile is used as a terminal to report data, the data is reported once every 15 seconds,
s2: loading filter conditions from the buffer for data filtering after receiving the data,
s3: counting and monitoring the filtered effective data flow, controlling the data processing thread number according to the monitoring result, executing once per minute by default,
s4: comprehensively analyzing the early warning index through the effective data flow and the data processing index of each processing unit; if the early warning condition is met, rebounding data is started, and early warning prompts can be sent to related responsible persons; the default strategy for bouncing data is to receive one filter for the same type of data; the early warning prompt has the functions of enabling related personnel to check the statistical result in the frame calculation process, finding a bottleneck and increasing the number of nodes;
s5: the monitored data distribution flows into a preprocessing cluster, and the data analysis is started aiming at different terminal protocols, wherein the default strategy of the distribution is polling,
s6: the preprocessed data are comprehensively judged according to a dispersion mode, a balance strategy and a data type and then are sent to different production end nodes in a production end cluster for processing, namely a producer in figure 1,
s7: the producer subdivides the data into different topic treatments,
s8: the data processed by the producer flows into the queue cluster buffer,
s9: and pulling the data from the queue cluster by the nodes of the consumer-side cluster, namely the consumer or a third-party statistical analysis program and the like, and further processing the data.
When the system is used for processing the data of the Internet of things, the statistical monitoring module can manage the data processing process by using the Web application program, configure various parameters in the data processing process and store the parameters into the database, the database supports the operation of the Web application program, data generated by the Web application program and calculation data in the cache can be stored into the database, meanwhile, configuration information needs to be frequently read from the cache in the data processing process, and the configuration information in the database can be refreshed into the cache in real time or at regular time; meanwhile, the invention can also utilize a network communication framework to receive data and coordinate the work of internal programs, the network communication framework is used as an entrance of terminal equipment data, provides thread groups, receives the terminal equipment data, and provides communication services such as cross-server communication and the like;
the acquisition module of the system of the invention is utilized to acquire data from the terminal equipment through the network communication framework,
the statistic monitoring module counts and monitors the acquired data and distributes the data to the preprocessing cluster controlled by the preprocessing module for preprocessing the data,
the preprocessed data are distributed to different production ends of the production end cluster controlled by the production end processing module after being classified for data processing,
the data processed by the production end cluster flows into the queue cluster cache, the queue cluster cache is convenient for the consumption end cluster to use in the Web background program, so that the consumption end node in the consumption end cluster has enough time to process the data, and is convenient for other statistical analysis programs to pull the data from the queue,
and the consumption end processing module controls consumption end nodes in the consumption end cluster to acquire the data to be processed from the queue cluster cache to complete the data processing process.
In the process, the acquisition module can judge whether the acquired data meets the filtering condition after acquiring the data of the terminal equipment, and filter the data meeting the filtering condition, otherwise, the data can be directly subjected to data statistical monitoring,
when the statistics monitoring module carries out statistics monitoring, early warning indexes can be obtained through comprehensive analysis of effective data flow and data processing indexes, early warning conditions are defined according to the early warning indexes, the early warning conditions are met, rebound data are carried out, a default strategy of the rebound data is to receive one filter for the same type of data, and the number of the upper line and the lower line of nodes in a production end cluster and a consumption end cluster is controlled by a Web application program according to the rebound data.
Based on the above process, the process of the system of the present invention for data processing is described by taking the data reported by the shared automobile as an example:
s1: the shared automobile is used as a terminal to report data, the data is reported once every 15 seconds,
s2: the acquisition module loads the filtering conditions from the buffer for data filtering after receiving the data,
s3: the statistical monitoring module carries out statistical monitoring on the filtered effective data flow, controls the size of the data processing thread number according to the monitoring result, defaults to execute once per minute,
s4: the statistical monitoring module comprehensively analyzes the early warning index through the effective data flow and the data processing index of each processing unit; if the early warning condition is met, rebounding data is started, and early warning prompts can be sent to related responsible persons; the default strategy for bouncing data is to receive one filter for the same type of data; the early warning prompt has the functions of enabling related personnel to check the statistical result in the frame calculation process, finding a bottleneck and increasing the number of nodes;
s5: the monitored data distribution flows into a preprocessing cluster controlled by a preprocessing module, and the data analysis is started aiming at different terminal protocols, wherein the default strategy of the distribution is polling,
s6: the preprocessed data are comprehensively judged according to the dispersion mode, the balance strategy and the data type and then are sent to different production end nodes in a production end cluster controlled by a production end processing module for processing, namely a producer in figure 1,
s7: the producer subdivides the data into different topic treatments,
s8: the data processed by the producer flows into the queue cluster buffer,
s9: the nodes of the consumer cluster controlled by the consumer processing module, namely consumers or third party statistical analysis programs and the like pull data from the queue cluster to further process the data.
The above-mentioned embodiments are merely preferred embodiments for fully illustrating the present invention, and the scope of the present invention is not limited thereto. The equivalent substitution or change made by the technical personnel in the technical field on the basis of the invention is all within the protection scope of the invention. The technologies involved except the technical scheme of the invention are the prior art, and the skilled person in the art selects and uses the specific method or tool which can complete the specific implementation mode of the invention in the prior art, and the creative labor and the use obstacle are not needed, so the description is not repeated. The protection scope of the invention is subject to the claims.
Claims (10)
1. A data processing method of the Internet of things is characterized by comprising the following steps:
the data of the terminal device is acquired,
the data obtained by statistical monitoring is distributed to a preprocessing cluster for preprocessing the data,
the preprocessed data are distributed to different production ends of the production end cluster for data processing after being classified,
and the data processed by the production end cluster flows into the queue cluster for caching, and then the data processing of the consumption end cluster is carried out, so that the data processing process is completed.
2. The method as claimed in claim 1, wherein the data of the terminal device is acquired and then whether the acquired data meets the filtering condition is determined, the data meeting the filtering condition is filtered, otherwise, the data statistical monitoring is directly performed.
3. The method as claimed in claim 1 or 2, wherein the early warning index is obtained by analyzing the effective data flow and the data processing index, the early warning condition is defined according to the early warning index, and the early warning condition is reached when the monitoring data is counted, and the data is rebounded.
4. The method as claimed in claim 3, wherein the preprocessed data are classified according to a dispersion mode, a balance strategy and a data type comprehensive judgment, and are distributed to different generation terminals in the production terminal cluster according to the classification for data processing.
5. The method as claimed in claim 4, wherein the number of on/off lines of the nodes in the producer and consumer clusters is controlled based on the bounce data.
6. An Internet of things data processing system is characterized by comprising an acquisition module, a statistical monitoring module and a data processing module, wherein the data processing module comprises a preprocessing module, a production end processing module and a consumption end processing module,
the acquisition module acquires data of the terminal device,
the statistic monitoring module counts and monitors the acquired data and distributes the data to the preprocessing cluster controlled by the preprocessing module for preprocessing the data,
the preprocessed data are distributed to different production ends of the production end cluster controlled by the production end processing module after being classified for data processing,
and the data processed by the production end cluster flows into the queue cluster for caching, and then the data processing process is completed through the data processing of the consumption end cluster controlled by the consumption end processing module.
7. The system as claimed in claim 6, wherein the obtaining module determines whether the obtained data satisfies the filtering condition after obtaining the data of the terminal device, and filters the data satisfying the filtering condition, otherwise, performs the data statistical monitoring directly.
8. The system as claimed in claim 6 or 7, wherein the statistical monitoring module obtains the early warning index by comprehensively analyzing the effective data flow and the data processing index, defines the early warning condition according to the early warning index, and performs rebound data when the early warning condition is reached during the statistics of the monitored data.
9. The system as claimed in claim 8, wherein the preprocessing module classifies the preprocessed data according to a dispersion mode, a balance strategy and a data type comprehensive judgment, and distributes the data to different generation terminals in the production terminal cluster according to the classification for data processing.
10. The system of claim 9, wherein the statistics monitoring module controls the number of on/off lines of the nodes in the producer and consumer clusters based on the bounce data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910981448.9A CN110750510A (en) | 2019-10-16 | 2019-10-16 | Internet of things data processing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910981448.9A CN110750510A (en) | 2019-10-16 | 2019-10-16 | Internet of things data processing system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110750510A true CN110750510A (en) | 2020-02-04 |
Family
ID=69278357
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910981448.9A Pending CN110750510A (en) | 2019-10-16 | 2019-10-16 | Internet of things data processing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110750510A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130159938A1 (en) * | 2011-12-16 | 2013-06-20 | Microsoft Corporation | Gesture inferred vocabulary bindings |
| CN104980515A (en) * | 2015-06-26 | 2015-10-14 | 浪潮(北京)电子信息产业有限公司 | Method and apparatus for distributing and processing messages in cloud storage systems |
| CN106528792A (en) * | 2016-11-10 | 2017-03-22 | 福州智永信息科技有限公司 | Big data acquisition and high-speed processing method and system based on multi-layer caching mechanism |
| CN107040608A (en) * | 2017-05-19 | 2017-08-11 | 宁波绮耘软件股份有限公司 | A kind of data processing method and system |
| CN108599992A (en) * | 2018-03-21 | 2018-09-28 | 四川斐讯信息技术有限公司 | A kind of data processing system and method |
-
2019
- 2019-10-16 CN CN201910981448.9A patent/CN110750510A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130159938A1 (en) * | 2011-12-16 | 2013-06-20 | Microsoft Corporation | Gesture inferred vocabulary bindings |
| CN104980515A (en) * | 2015-06-26 | 2015-10-14 | 浪潮(北京)电子信息产业有限公司 | Method and apparatus for distributing and processing messages in cloud storage systems |
| CN106528792A (en) * | 2016-11-10 | 2017-03-22 | 福州智永信息科技有限公司 | Big data acquisition and high-speed processing method and system based on multi-layer caching mechanism |
| CN107040608A (en) * | 2017-05-19 | 2017-08-11 | 宁波绮耘软件股份有限公司 | A kind of data processing method and system |
| CN108599992A (en) * | 2018-03-21 | 2018-09-28 | 四川斐讯信息技术有限公司 | A kind of data processing system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105978728B (en) | The monitoring method of intelligent monitor system and operational indicator | |
| CN110912773B (en) | Cluster monitoring system and monitoring method for multiple public cloud computing platforms | |
| CN106453125B (en) | A kind of remote service calling SiteServer LBS based on real time load rate | |
| CN106209482A (en) | A kind of data center monitoring method and system | |
| CN109271243B (en) | Cluster task management system | |
| CN116668380B (en) | Message processing method and device of convergence diverter equipment | |
| CN107168853A (en) | A kind of server performance information acquisition method, system and substrate control manager | |
| CN105243499A (en) | Order distribution method and system | |
| CN108108296A (en) | A kind of cloud test method, server and client | |
| CN107085579A (en) | A kind of data acquisition distribution method and device | |
| CN107675308A (en) | A kind of single ingot monitor control system based on PLC | |
| CN107395687A (en) | Monitoring method, device, system and the air-conditioning of equipment | |
| CN107886429A (en) | Credit service management method and system | |
| CN105872082A (en) | Fine-grained resource response system based on load balancing algorithm of container cluster | |
| CN110750510A (en) | Internet of things data processing system and method | |
| CN102523107B (en) | The method and device of balanced network management system service end and client computing pressure | |
| CN106446289A (en) | Information inquiry method and device based on Pinpoint | |
| CN106445709A (en) | Method and system for invoking servers in distributed manner | |
| CN112988360A (en) | Task distribution system based on big data analysis | |
| CN109542848A (en) | A kind of vehicle-mounted media file fast scanning method and system | |
| CN110988584A (en) | Intelligent monitoring and automatic control system for electric power system | |
| CN112785276B (en) | Engineering consultation management system and method | |
| CN109873708A (en) | A kind of assets portrait method clustered based on traffic characteristic and kmeans | |
| CN206962848U (en) | A kind of information O＆M device for mass data | |
| CN110445711A (en) | A kind of data traffic monitoring system based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200204 |