CN110737654A - Flink-based Internet of things equipment behavior analysis method - Google Patents
Flink-based Internet of things equipment behavior analysis method Download PDFInfo
- Publication number
- CN110737654A CN110737654A CN201910991363.9A CN201910991363A CN110737654A CN 110737654 A CN110737654 A CN 110737654A CN 201910991363 A CN201910991363 A CN 201910991363A CN 110737654 A CN110737654 A CN 110737654A
- Authority
- CN
- China
- Prior art keywords
- equipment
- flink
- data
- equipment behavior
- machine learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Probability & Statistics with Applications (AREA)
- Quality & Reliability (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Fuzzy Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides Flink-based Internet of things equipment behavior analysis methods, which belong to the technical field of Internet of things, the invention collects equipment data based on an Internet of things platform and sends the equipment data to an MQTT, the Flink acquires the equipment data from the MQTT, the data is firstly cleaned to obtain required characteristic values, then a machine learning algorithm is used for calculating the obtained characteristic values to obtain an equipment behavior model, finally, the equipment behavior model is analyzed, screened and aggregated through a time window mechanism of the Flink to obtain an equipment behavior analysis result, and the equipment behavior analysis result is stored in an HDFS and dynamically displayed in real time.
Description
Technical Field
The invention relates to the technology of Internet of things, relates to Flink, hadoop and other related components, and applies machine learning and other related technologies, in particular to methods for analyzing behaviors of Internet of things equipment based on Flink.
Background
Flink is an data stream processing-oriented distributable open source computing framework, supports high-throughput, low-delay and high-performance stream processing, supports stream processing and window processing with event time (event time) semantics, enables the result of stream computing to be more accurate, and particularly supports highly flexible window operation under the condition that events arrive out of order or delay.
In recent years, the technology of the internet of things is rapidly developed, and the technology of the internet of things not only causes the fourth industrial revolution, but also has a profound influence on the basic state of human society such as agriculture, industry, service industry and the like, so that the production and life style of the whole human society are revolutionized.
The generation of the general Internet of things platform can greatly reduce the threshold for developing Internet of things application, the traditional Internet of things application development is shifted to the platform, and with the development of machine learning technology in recent years, potential commercial value behind the data is mined by analyzing mass equipment data.
Disclosure of Invention
In order to solve the technical problems, the invention provides Flink-based Internet of things equipment behavior analysis methods, and the analysis efficiency of equipment behaviors is improved.
The technical scheme of the invention is as follows:
Flink-based Internet of things equipment behavior analysis method,
the method comprises the following steps:
1) cleaning and screening the equipment data through the Flink;
2) establishing an equipment behavior model by using the characteristic value of the equipment through a machine learning algorithm;
3) and dynamic real-time updating is realized through counting and analyzing results of the Flink time window.
In a step further , the method includes,
data of equipment are collected and sent to the MQTT based on the Internet of things platform, the flink acquires the equipment data from the MQTT, and the required characteristic value is acquired through data cleaning.
In a step further , the method includes,
and calculating the obtained characteristic values by using a machine learning algorithm to obtain an equipment behavior model. The machine learning algorithm processes data by adopting a three-layer artificial neural network method.
In a step further , the method includes,
and finally, analyzing, screening and aggregating the equipment behavior model through a time window of the flink to obtain an equipment behavior analysis result, storing the equipment behavior analysis result into the HDFS and dynamically displaying the equipment behavior analysis result in real time.
In a step further , the method includes,
wherein the characteristic value is determined by the actual demand.
And finally, the formed equipment behavior analysis result is used for equipment fault prediction, equipment performance analysis and equipment personalized service.
And a further step is carried out,
the method comprises the following specific steps:
step 1) formulating corresponding characteristic values and label rules according to actual requirements, and determining equipment behavior data to be acquired;
step 2), behavior data of the Internet of things platform collecting equipment;
step 3), sending the device behavior data stream of the Internet of things platform to Flink through MQTT for processing, processing the Flink SQL to obtain a characteristic value required by a machine learning algorithm, and delivering the characteristic value to the machine learning algorithm carried on the Flink for processing;
step 4), processing data by a machine learning algorithm by adopting a three-layer artificial neural network method, and generating an equipment behavior model in the learning process;
step 5) analyzing and aggregating the result of the equipment behavior model by the equipment behavior model generated by the machine learning algorithm through the Flink according to a label rule formulated according to actual requirements, and labeling the equipment;
step 6), storing the analysis and aggregation results into the HDFS, and dynamically displaying the analysis and aggregation results in real time through equipment and a label through a time window of the Flink;
and 7) the result of the equipment behavior analysis can be used for user behavior analysis, equipment fault prediction and equipment personalized service.
The invention has the advantages that
1) The efficiency of equipment behavior analysis is improved.
2) The effectiveness of equipment behavior analysis is improved.
3) Real-time dynamic updating is realized.
Drawings
FIG. 1 is a schematic workflow diagram of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer and more complete, the technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all embodiments of the present invention, and all other embodiments obtained by a person of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the protection scope of the present invention.
The Flink-based Internet of things equipment behavior analysis method includes the steps of collecting equipment data based on an Internet of things platform and sending the equipment data to an MQTT, obtaining equipment data from the MQTT by the Flink, firstly carrying out data cleaning to obtain required characteristic values, then calculating the obtained characteristic values by using a machine learning algorithm to obtain an equipment behavior model, finally analyzing, screening and aggregating the equipment behavior model through a time window mechanism of the Flink to obtain an equipment behavior analysis result, and storing the equipment behavior analysis result in an HDFS (Hadoop distributed File System) and dynamically displaying the equipment behavior analysis result in real time.
Wherein the characteristic value is determined by the actual demand. The finally formed equipment behavior analysis result can be used for equipment fault prediction, equipment performance analysis, equipment personalized service and the like.
The specific operation steps are as follows:
step 1) formulating corresponding characteristic values and label rules according to actual requirements, and determining equipment behavior data to be acquired;
step 2), behavior data of the Internet of things platform collecting equipment;
step 3) sending the device behavior data stream of the Internet of things platform to Flink through MQTT for processing, for example, the device behavior data is a data stream in a JSON format, and obtaining a characteristic value required by a machine learning algorithm after Flink SQL is processed, and sending the characteristic value to the machine learning algorithm loaded on Flink for processing;
and 4) processing the data by adopting a three-layer artificial neural network method in the machine learning algorithm, and generating an equipment behavior model in the learning process.
And 5) analyzing and aggregating the result of the equipment behavior model by the equipment behavior model generated by the machine learning algorithm through the Flink according to a label rule formulated according to actual requirements, and labeling the equipment.
And 6) storing the analysis and aggregation results into the HDFS, and dynamically displaying the analysis and aggregation results in real time through equipment and a label through a time window of the Flink.
And 7) the result of the equipment behavior analysis can be used for user behavior analysis, equipment fault prediction, equipment personalized service and the like.
The above description is only a preferred embodiment of the present invention, and is only used to illustrate the technical solutions of the present invention, and not to limit the protection scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (8)
1, A Flink-based Internet of things equipment behavior analysis method, which is characterized in that,
the method comprises the following steps:
1) cleaning and screening the equipment data through the Flink;
2) establishing an equipment behavior model by using the characteristic value of the equipment through a machine learning algorithm;
3) and dynamic real-time updating is realized through counting and analyzing results of the Flink time window.
2. The method of claim 1,
data of equipment are collected and sent to the MQTT based on the Internet of things platform, the flink acquires the equipment data from the MQTT, and the required characteristic value is acquired through data cleaning.
3. The method of claim 2,
and calculating the obtained characteristic values by using a machine learning algorithm to obtain an equipment behavior model.
4. The method of claim 2,
the machine learning algorithm processes data by adopting a three-layer artificial neural network method.
5. The method according to claim 3 or 4,
and finally, analyzing, screening and aggregating the equipment behavior model through a time window of the flink to obtain an equipment behavior analysis result, storing the equipment behavior analysis result into the HDFS and dynamically displaying the equipment behavior analysis result in real time.
6. The method of claim 5,
wherein the characteristic value is determined by the actual demand.
7. The method of claim 6,
and finally, the formed equipment behavior analysis result is used for equipment fault prediction, equipment performance analysis and equipment personalized service.
8. The method of claim 7,
the method comprises the following specific steps:
step 1) formulating corresponding characteristic values and label rules according to actual requirements, and determining equipment behavior data to be acquired;
step 2), behavior data of the Internet of things platform collecting equipment;
step 3), sending the device behavior data stream of the Internet of things platform to Flink through MQTT for processing, processing the Flink SQL to obtain a characteristic value required by a machine learning algorithm, and delivering the characteristic value to the machine learning algorithm carried on the Flink for processing;
step 4), processing data by a machine learning algorithm by adopting a three-layer artificial neural network method, and generating an equipment behavior model in the learning process;
step 5) analyzing and aggregating the result of the equipment behavior model by the equipment behavior model generated by the machine learning algorithm through the Flink according to a label rule formulated according to actual requirements, and labeling the equipment;
step 6), storing the analysis and aggregation results into the HDFS, and dynamically displaying the analysis and aggregation results in real time through equipment and a label through a time window of the Flink;
and 7) analyzing the equipment behavior, wherein the result of the equipment behavior analysis is used for user behavior analysis, equipment fault prediction and equipment personalized service.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910991363.9A CN110737654A (en) | 2019-10-18 | 2019-10-18 | Flink-based Internet of things equipment behavior analysis method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910991363.9A CN110737654A (en) | 2019-10-18 | 2019-10-18 | Flink-based Internet of things equipment behavior analysis method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110737654A true CN110737654A (en) | 2020-01-31 |
Family
ID=69269228
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910991363.9A Pending CN110737654A (en) | 2019-10-18 | 2019-10-18 | Flink-based Internet of things equipment behavior analysis method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737654A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111538881A (en) * | 2020-04-16 | 2020-08-14 | 广东好太太科技集团股份有限公司 | Activity degree analysis method and equipment based on behavior data and storage medium |
| CN111935226A (en) * | 2020-07-08 | 2020-11-13 | 上海微亿智造科技有限公司 | Method and system for realizing streaming computing by supporting industrial data |
| CN113783931A (en) * | 2021-08-02 | 2021-12-10 | 中企云链（北京）金融信息服务有限公司 | Internet of things data aggregation and analysis method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170083378A1 (en) * | 2015-09-18 | 2017-03-23 | Salesforce.Com, Inc. | Managing processing of long tail task sequences in a stream processing framework |
| CN107315810A (en) * | 2017-06-27 | 2017-11-03 | 济南浪潮高新科技投资发展有限公司 | A kind of internet of things equipment behavior portrait method |
| CN109618308A (en) * | 2018-12-28 | 2019-04-12 | 济南浪潮高新科技投资发展有限公司 | A method of internet of things data is handled based on Spark Streaming |
| CN109951463A (en) * | 2019-03-07 | 2019-06-28 | 成都古河云科技有限公司 | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column |
| CN110275899A (en) * | 2019-04-18 | 2019-09-24 | 智链万源(北京)数字科技有限公司 | Internet of things data method for stream processing, system and device |
-
2019
- 2019-10-18 CN CN201910991363.9A patent/CN110737654A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170083378A1 (en) * | 2015-09-18 | 2017-03-23 | Salesforce.Com, Inc. | Managing processing of long tail task sequences in a stream processing framework |
| CN107315810A (en) * | 2017-06-27 | 2017-11-03 | 济南浪潮高新科技投资发展有限公司 | A kind of internet of things equipment behavior portrait method |
| CN109618308A (en) * | 2018-12-28 | 2019-04-12 | 济南浪潮高新科技投资发展有限公司 | A method of internet of things data is handled based on Spark Streaming |
| CN109951463A (en) * | 2019-03-07 | 2019-06-28 | 成都古河云科技有限公司 | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column |
| CN110275899A (en) * | 2019-04-18 | 2019-09-24 | 智链万源(北京)数字科技有限公司 | Internet of things data method for stream processing, system and device |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111538881A (en) * | 2020-04-16 | 2020-08-14 | 广东好太太科技集团股份有限公司 | Activity degree analysis method and equipment based on behavior data and storage medium |
| CN111538881B (en) * | 2020-04-16 | 2024-02-09 | 广东好太太科技集团股份有限公司 | Activity analysis method, equipment and storage medium based on behavior data |
| CN111935226A (en) * | 2020-07-08 | 2020-11-13 | 上海微亿智造科技有限公司 | Method and system for realizing streaming computing by supporting industrial data |
| CN111935226B (en) * | 2020-07-08 | 2021-06-08 | 上海微亿智造科技有限公司 | Method and system for realizing streaming computing by supporting industrial data |
| CN113783931A (en) * | 2021-08-02 | 2021-12-10 | 中企云链（北京）金融信息服务有限公司 | Internet of things data aggregation and analysis method |
| CN113783931B (en) * | 2021-08-02 | 2023-07-25 | 中企云链（北京）金融信息服务有限公司 | Data aggregation and analysis method for Internet of things |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110737654A (en) | Flink-based Internet of things equipment behavior analysis method | |
| CN107315810B (en) | Internet of things equipment behavior portrait method | |
| CN109976997B (en) | Test method and device | |
| US20150278706A1 (en) | Method, Predictive Analytics System, and Computer Program Product for Performing Online and Offline Learning | |
| KR102236302B1 (en) | Method for providing big data-based artificial intelligence integration platform service | |
| CN111209258A (en) | Tax end system log real-time analysis method, equipment, medium and system | |
| US10843084B2 (en) | Method and system for gathering time-varying metrics | |
| CA2957454A1 (en) | Interrelation of multiple data streams | |
| CN110363427A (en) | Model quality evaluation method and apparatus | |
| US20210049414A1 (en) | Deep graph de-noise by differentiable ranking | |
| CN106610932A (en) | Corpus processing method and device and corpus analyzing method and device | |
| US20190197578A1 (en) | Generating significant performance insights on campaigns data | |
| CN111242658A (en) | Information sharing reward method and device and computer readable storage medium | |
| CN111782611A (en) | Prediction model modeling method, device, equipment and storage medium | |
| EP3975075A1 (en) | Runtime estimation for machine learning data processing pipeline | |
| CN110457895A (en) | A kind of PC application program violation content monitoring method and device | |
| CN108345620B (en) | Brand information processing method, brand information processing device, storage medium and electronic equipment | |
| CN107644042B (en) | Software program click rate pre-estimation sorting method and server | |
| KR102559950B1 (en) | An AI-based optimal advertising recommendation system | |
| CN112231344A (en) | Real-time stream data query method and device | |
| CN115619475A (en) | Commodity recommendation method, commodity recommendation system and related devices | |
| CN114444602A (en) | Method and system for automatically constructing anomaly detection model | |
| Sudan et al. | Prediction of success and complex event processing in E-learning | |
| CN113051479A (en) | File processing and recommendation information generation method, device, equipment and storage medium | |
| US20200175125A1 (en) | Event-triggered analysis for high fidelity simulation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200131 |