CN116455930A - Intelligent Internet of things exploring type data analysis focus correlation technology application system - Google Patents
Intelligent Internet of things exploring type data analysis focus correlation technology application system Download PDFInfo
- Publication number
- CN116455930A CN116455930A CN202310278190.2A CN202310278190A CN116455930A CN 116455930 A CN116455930 A CN 116455930A CN 202310278190 A CN202310278190 A CN 202310278190A CN 116455930 A CN116455930 A CN 116455930A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- access
- management
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 18
- 238000005516 engineering process Methods 0.000 title description 17
- 238000000034 method Methods 0.000 claims abstract description 32
- 230000008447 perception Effects 0.000 claims description 53
- 238000007726 management method Methods 0.000 claims description 37
- 238000012545 processing Methods 0.000 claims description 34
- 230000002159 abnormal effect Effects 0.000 claims description 32
- 238000006243 chemical reaction Methods 0.000 claims description 29
- 238000003860 storage Methods 0.000 claims description 26
- 238000013523 data management Methods 0.000 claims description 25
- 238000004458 analytical method Methods 0.000 claims description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 17
- 238000007781 pre-processing Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 15
- 238000000605 extraction Methods 0.000 claims description 14
- 230000009467 reduction Effects 0.000 claims description 14
- 238000013500 data storage Methods 0.000 claims description 13
- 238000011161 development Methods 0.000 claims description 10
- 238000005070 sampling Methods 0.000 claims description 10
- 238000004040 coloring Methods 0.000 claims description 8
- 238000001914 filtration Methods 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 7
- 230000005540 biological transmission Effects 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 6
- 238000002955 isolation Methods 0.000 claims description 6
- 230000005856 abnormality Effects 0.000 claims description 5
- 230000006870 function Effects 0.000 claims description 5
- 238000012550 audit Methods 0.000 claims description 4
- 238000009826 distribution Methods 0.000 claims description 4
- 230000004927 fusion Effects 0.000 claims description 4
- 238000007689 inspection Methods 0.000 claims description 4
- 238000013475 authorization Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 claims description 3
- 238000012937 correction Methods 0.000 claims description 2
- 230000007613 environmental effect Effects 0.000 claims description 2
- 238000012360 testing method Methods 0.000 claims description 2
- 238000012795 verification Methods 0.000 claims description 2
- 230000001953 sensory effect Effects 0.000 claims 1
- 238000013461 design Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 8
- 230000006399 behavior Effects 0.000 description 6
- 238000010276 construction Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 238000011985 exploratory data analysis Methods 0.000 description 5
- 230000006872 improvement Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000008520 organization Effects 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 230000010354 integration Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000008030 elimination Effects 0.000 description 2
- 238000003379 elimination reaction Methods 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000002485 combustion reaction Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000002269 spontaneous effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
- G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
- G16Y20/00—Information sensed or collected by the things
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/66—Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Medical Informatics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides an intelligent Internet of things exploring type data analysis focus correlation technique application system, and provides an intelligent Internet of things exploring type data analysis focus correlation technique.
Description
Technical Field
The invention belongs to the application field of physical Internet of things such as intelligent camps and intelligent parks, and particularly relates to an intelligent Internet of things exploration type data analysis focus correlation technology application system.
Background
At present, mass sensing equipment is deployed on the Internet of things terminal of an intelligent barrack for sensing the conditions of people, vehicles, objects and environment of the barrack, and the Internet of things sensing data of the barrack also changes from generation to development to sensing, utilization, linkage and deep mining from variable to variable sensing data life cycle. The development of information technology promotes the cloud brain computing power of a barrage information system to be developed into a cloud plus side mixed mode from cloud computing. How to effectively utilize the edge computing technology to improve the computing power of the intelligent barrage terminal, and solve the problems of terminal data processing and response of the Internet of things, which are the problems needing to be focused.
In order to solve the problem of acquisition access, the problems of communication interfaces, protocols, data formats and the like of diversified heterogeneous devices are solved in the related research of intelligent barrage information system construction while mass data is increased, certain data complexity is brought in the acquisition stage, and a general gateway provides a plurality of solutions for providing unified service data for an application system in the process of shielding heterogeneous perception data. However, along with the improvement of computing power and the access of data, how to effectively respond to the change of data at the edge of the construction of the intelligent barrage information system, coordinate the effective cooperation among equipment, sensors and the like, describe and process the intelligent barrage service sensing the edge from the aspects of communication between information flow and respondents, activities of respondents, time limit and rules of activities and the like, and are technical difficulties.
Disclosure of Invention
The invention aims to: aiming at the terminal data acquisition, storage and processing processes with various types and different protocol formats in the current barrage environment, the industry has fully compatible the data of various sources through gateway design by taking the construction of a unified standard as a starting point. But the problems of integration and association of mass sensing data, situation sensing linkage, data security, reliability and intelligent fine management are not thoroughly solved. The invention provides an intelligent Internet of things exploring type data analysis focus correlation technology application system, which can be combined with the current existing data acquisition access method to guide the existing intelligent gateway to change from acquisition and convergence data which are only transmitted through data to exploring type data focus extraction and built-in rule correlation feedback modes in the intelligent Internet of things environment of the intelligent camping area, so that the intelligent reconstruction of the intelligent camping area is smoothly upgraded.
The system comprises a heterogeneous protocol unified access and data conversion module, a perception data calculation module, a perception data storage module, a perception data management and control module, a front-end alarm module and a linkage feedback module;
the heterogeneous protocol unified access and data conversion module is used for realizing different protocol data access including WIFI access, WG access, network port access, serial port access and WAPI access by means of communication protocol format analysis, format conversion and data fusion for various Internet of things sensing devices; and through low-code assembly operation, input transmission channel template configuration operation is carried out on the provided unified data compiling operation panel, so that conversion between various different protocol data and data formats of a perception layer is realized.
The perception data computing module comprises a data preprocessing module, a data exchange module and a task scheduling processing module, wherein the data preprocessing module is used for preprocessing data, the data preprocessing comprises data reduction, data combination, data sampling, data dimension reduction, feature subset selection, discretization, binarization and attribute transformation, and finally preprocessed data is obtained.
The data exchange module is used for completing access service, access control service, message conversion service, data routing service and management service; the access service is mapped to each protocol, realizes the access control service of the data through buffer priority setting and authority setting, and comprises a sensitive security verification mechanism, source security control (unknown source), frequency security control (abnormal data access frequency such as the condition of too frequent burst transmission and the like), finishes the unified processing of a message format through the message conversion service, and realizes the data distribution channel of marking the data and the routing of a receiving object through the data routing service and the management service.
The task scheduling processing module is used for completing task scheduling work by adopting a timing strategy and supervising the running condition of the task; the task scheduling work comprises data acquisition, data calculation and analysis and finally data output.
The perception data storage module is used for storing structured and unstructured data and streaming data by providing a storage mode of regional differentiation.
The perception data management and control module is used for uniformly checking the quality of data by taking focus extraction as a target, wherein the checking comprises data checking, checking and testing, evaluating the integrity, accuracy and reliability of the data, synchronously analyzing the storage structure of the data, eliminating abnormal data, auditing the behavior of the perception data, wherein the perception data comprises environment data, equipment data, sensor data, position data, user behavior data and running state data, carrying out authorization of different types and different authority levels on the data, such as enhancing DAC efficiency of intelligent contracts, writing programs through computer programming languages, automatically uploading contract generation to a blockchain network without a third party, automatically executing the programs and randomly checking to complete conditions, continuously executing conditions, directly starting penalty conditions if the conditions are violated, automatically processing the data by the perception data management and control module, uniformly managing the data resources, including uniformly counting and metering the data quantity, comprehensively processing the focus extraction technology by the heterogeneous perception data, inquiring the relevant context, extracting possible influence factors and actual influence factors, carrying out uniform topic management algorithm and analysis on the data by using the analysis of the influence on the abnormal situation, and the analysis of the data by using the map information, and analyzing the abnormal situation.
The front-end warning module carries out front-end warning prompt through preset warning rules and comparison of data difference conditions, and comprises real-time data boundary crossing warning, data change trend warning, deviation normal range warning, abnormal data value warning, repeated data value warning, fault data value warning and data missing warning, and according to preset linkage feedback rules, feedback flow arrangement and decision modules, front-end warning information is transmitted to a background system to form event-driven linkage feedback, wherein the event-driven linkage feedback comprises warning triggering linkage, timing linkage, data value linkage and characteristic value linkage;
the linkage feedback module is used for rule configuration management, event flow arrangement, decision module and log management.
The data reduction realizes redundancy elimination by defining a redundancy field, data merging merges data within the constant time of the quantized value through the change of a time tag and the quantized value, data sampling is performed by setting a sampling mark point, data complexity is reduced by defining a data attention dimension, a feature subset selection provides a selection means by presetting a feature subset experience value to give a plan, a feature generation result is formed, and then data is sorted through discretization, binarization and attribute transformation.
The perception data management and control module executes the following steps:
step 1, the perception data management and control module analyzes and forms an abnormal characteristic area through an Isolation Forest algorithm and an improved K-means algorithm, and specifically comprises the following steps:
step 1-1, the Isolation Forest algorithm adopts a binary tree to segment data, and the depth of the data point in the binary tree reflects the discrete degree of the data in the set, which specifically comprises the following steps:
constructing a binary tree and combining the binary tree into a forest: for a given perceived dataset D, a property X is randomly selected 1 And at the attribute X 1 Randomly selecting a value Y between the maximum value and the minimum value, dividing the data smaller than the value Y in the data set D into the left branch of the binary tree, dividing the data larger than or equal to the value Y into the right branch, and repeating the steps until the data set has one or more than two identical records or the binary tree reaches a limited height;
calculating an abnormality coefficient: since outliers are rare for datasets, it is determined whether a data point x is outlier by the path h (x) of the root and leaf nodes, using the following formula:
where Score (x, n) is the anomaly coefficient of data point x in a binary tree of n samples; e (h (x)) represents the average path length of data x in each tree, and C (n) is a correction value, and is generally expressed as: c (n) =2h (n-1) - (2 (n-1)/n), wherein the function H (k) =lnk+δh, δ= 0.5772156649 is a euler constant;
step 1-2, the improved K-means algorithm comprises the following steps: after obtaining the abnormality degree coefficient of each sample, selecting the required cluster number K and an outlier filtering proportion r, wherein the outlier filtering proportion refers to the percentage of the number of filtered data to the total number of the samples, namely, clustering is carried out after the outlier of which the number is required to be filtered by using an improved K-means algorithm, and the default is 0.1;
and selecting an initial clustering center by combining the average interpolation method thought through the value range of the statistical data in each dimension, namely uniformly selecting initial values in the data set as much as possible, and reducing the influence of boundary values on the selection of the initial clustering center. For a value range of [ i, j ]]According to the required cluster number k, applying a formula Calculating an initial cluster center, wherein K n Representing the cluster center, T i Representing the lower limit value of the attribute T j An upper limit value representing the attribute T; obtain high-quality initial value K n Then, classifying the filtered abnormal samples into a residual sample set, and iterating for more than two times until the function converges to finally obtain an abnormal characteristic region;
step 2, coloring the abnormal characteristic region, searching a scene context according to coloring characteristics, analyzing the development trend of related parameters, wherein the scene context comprises related equipment, management objects and a region, the development trend of the related parameters refers to variable values or attribute values in a data set, the variable values or attribute values are represented in different colors in a chart, related information in the data set is reflected, such as red to represent abnormality, the abnormal processing grade is associated according to red RGB (red, green and blue) color, parameters of the related equipment are called for the condition of over-high temperature near spontaneous combustion, information of the data context such as abnormal equipment and the abnormal region where the equipment is located is found, and the information is aggregated into information submitting alarm and early warning processing of the physical world;
and step 3, analyzing and extracting by utilizing the abnormal characteristic region and the development trend to obtain a focus, comparing with a historical case, and acquiring a relevant data result.
The perception data storage module is used for marking data according to a storage mode of strategically configuring structured and unstructured data in a list, storing the structured data in a lightweight database according to a layering system of original data (initially collected unprocessed data), reorganized data (data after preliminary arrangement according to rules) and preprocessed data (data after processing according to rules or models), and completing file storage according to a transparent transmission unstructured data strategy, namely without analyzing compressed data.
The system is based on the fact that an edge computing technology improves computing power in an Internet of things sensing environment to achieve exploring type data analysis, and utilizes two key technologies of a heterogeneous sensing data comprehensive processing focus extraction technology and a data dynamic rule linkage technology to respectively obtain sensing data which is processed through data screening and filtering and data preliminary processing to form front-end treatment, and finally, intelligent linkage application is achieved in the two processes of front-end alarming and linkage feedback. The invention has good expansibility and universality, is simple in deployment, upgrading and operation, maximally preposes intelligent application in the environment of the Internet of things, improves the data value efficiency, and simultaneously provides a realization reference for the artificial intelligence technology in the real service scene.
The beneficial effects of the invention are as follows: the invention provides an intelligent Internet of things exploring type data analysis focus correlation technique, which takes an edge end exploring type data analysis technique as a guide, realizes the integrated use problems of unified access and data conversion of a heterogeneous protocol, calculation of sensing data, storage of sensing data, management and control of sensing data, front-end alarming, linkage feedback and the like, and provides a heterogeneous sensing data comprehensive processing focus extraction technique and a data dynamic rule linkage technique. The intelligent migration method has the advantages of dynamic expansion and intelligent migration, and has important prospects in the intelligent construction process of intelligent barracks.
Drawings
The foregoing and/or other advantages of the invention will become more apparent from the following detailed description of the invention when taken in conjunction with the accompanying drawings and detailed description.
Fig. 1 is a structural diagram of the present invention.
Fig. 2 is a flow chart of heterogeneous protocol unified access and data conversion module operation.
FIG. 3 is a flow chart of the sense data calculation module.
FIG. 4 is a flow chart of the sense data management module operation.
FIG. 5 is a front end alert and coordinated feedback workflow diagram.
Fig. 6 is a flow chart of the operation of the present invention.
Detailed Description
The invention relates to a support system design by adopting an edge-end exploring type data analysis technology, a heterogeneous sensing data comprehensive processing focus extraction technology and a data dynamic rule linkage technology, and particularly provides an intelligent Internet of things exploring type data analysis focus correlation technology application system, which comprises a heterogeneous protocol unified access and data conversion module, a sensing data calculation module, a sensing data storage module, a sensing data management and control module, a front-end alarm module and a linkage feedback module, as shown in fig. 1.
The heterogeneous protocol unified access and data conversion module is used for realizing different protocol data access including WiFi access, WG access, internet access serial port access and WAPI access through communication protocol format analysis, format conversion and data fusion for various sensing devices (such as a temperature and humidity sensor, a vibration sensor, a mobile sensing device, beidou equipment and positioning equipment) of the Internet of things; and the system is also used for providing a unified data compiling configuration template in a low-code mode, realizing unified conversion between various different protocol data and data formats of the perception layer, shielding the heterogeneous space of the bottom layer and realizing unified management and support of the upper layer. As a general method, the invention is compatible with future unpredictable incremental protocol content definition pluggable data acquisition standard protocol based on exploratory data analysis thought, and protocol convention comprises information such as data format, protocol type, data organization condition and the like. Meanwhile, for massive data of the Internet of things, raw data which are not processed comprise a large amount of invalid data, and the data are required to be filtered before reaching storage in the invention, so that the basic problems of data loss, data value repetition, data format adjustment according to access protocol requirements and the like are mainly solved.
The perception data computing module comprises a data preprocessing module, a data exchange module and a task scheduling processing module, wherein the data preprocessing module is used for preprocessing data, the data preprocessing comprises data reduction, data combination, data sampling, data dimension reduction, feature subset selection, discretization, binarization and attribute transformation, and finally preprocessed data is obtained;
the data exchange module is used for completing access service, access control service, message conversion service, data routing service and management service; the access service is mapped to each protocol, realizes the access control of data through buffer priority setting and authority setting, completes the unified processing of message formats through the message conversion service, and realizes the data distribution channel of marking data labels and the routing of receiving objects through the data routing service and the management service. The data labels are calibrated mainly according to the service, such as temperature, electromagnetic monitoring, video and the like, and channel distribution is carried out according to the calibrated digital labels.
The task scheduling processing module is used for completing task scheduling work by adopting a timing strategy and supervising the running condition of the task; the task scheduling work comprises data acquisition and data calculation analysis (namely layering treatment is carried out on temperature values according to different tasks such as temperature data acquisition tasks under the condition of unchanged temperature, redundant data are removed, an optimization model is a layering concept, for example, the algorithm only submits the values in a configured variable floating range, the number value on a network is greatly reduced), and finally the data after the structuring is output;
the perception data storage module is used for storing the structured and unstructured data and streaming data (comprising massive perception data) in various forms by providing a differentiated storage mode. After input collection and analysis such as mass internet of things environment perception, data are stored and processed according to a certain organization form and rule, so that effective data management is realized, and the perceived data logically has a hierarchical system of layer-by-layer connection: bits, characters, data elements, records, files, databases. According to the above division, the records of the perception data storage module are stored as logically related data element combinations; storing the file as a collection of records; the invention relates to a storage mode for gateway configuration lightweight database completion structured data. Other unstructured data is stored in the form of files that are strategically disregarded by the front-end.
And the perception data storage module stores the structured data in a lightweight database according to a layered system according to a strategically configured structured and unstructured data storage mode, and completes file storage according to a transparent unstructured data strategy.
The perception data management and control module is used for uniformly checking the quality of data through a heterogeneous perception data comprehensive processing focus extraction technology, auditing the behavior of the perception data, authorizing the data with different types and different authority levels, and uniformly managing the data resources, including uniformly counting and measuring the data quantity. The invention designs a mode of combining overview and detail aiming at focus extraction, and inquires related context in a database through abnormal point discovery, extracts influencing factors and equipment object elements and performs unified thematic management. Specifically, the working process is as follows, 1) firstly, feature extraction is performed through heterogeneous sensing data received by a data management and control module, and an abnormal feature area is formed. 2) And (3) marking the abnormal characteristic region data by coloring, and searching scene contexts such as associated equipment, management objects, belonging region information and the like according to the coloring characteristics. 3) And analyzing and extracting by utilizing the abnormal characteristic region and the development trend to obtain a focus, comparing the focus with a historical case, and acquiring a relevant data result.
The front-end warning module carries out front-end warning prompt through a preset warning rule and a comparison data difference value condition, for example, the intelligent Internet of things gives out corresponding warning prompts such as sound and light points; and according to a preset linkage feedback rule, a feedback flow arrangement and decision module, transmitting front-end alarm information to a background system is completed, and event-driven linkage feedback is formed. 1. Linkage feedback rules, i.e. which data should be associated with which events or data changes; the feedback flow arrangement is the event response to the alarm; the decision module is used for processing the decision of whether to alarm;
the linkage feedback module is used for rule configuration management, event flow arrangement, decision module and log management; the front-end sensing information is mainly transferred to a background service system, and event linkage feedback driven by events is completed through the module.
The data reduction realizes redundancy elimination by defining a redundancy field, data merging merges data within the constant time of the quantized value through the change of a time tag and the quantized value, data sampling is performed by setting a sampling mark point, data complexity is reduced by defining a data attention dimension, a feature subset selection provides a selection means by presetting a feature subset experience value to give a plan, a feature generation result is formed, and then data is sorted through discretization, binarization and attribute transformation.
The perception data management and control module executes the following steps:
and step 1, analyzing and forming an abnormal characteristic region by a perception data management and control module through an Isolation Forest algorithm and an improved K-means algorithm. The method comprises the steps of carrying out a first treatment on the surface of the
Step 2, coloring the abnormal characteristic region, searching scene contexts such as information of associated equipment, management objects, affiliated regions and the like according to coloring characteristics, and analyzing development trends of related parameters;
and step 3, analyzing and extracting by utilizing the abnormal characteristic region and the development trend to obtain a focus, comparing with a historical case, and acquiring a relevant data result.
The perception data storage module marks the data according to a strategic storage mode of configuring structured and unstructured data in a list, stores the structured data in a lightweight database according to a layering system of original data, reorganized data and preprocessed data, and completes file storage according to a transparent unstructured data strategy, namely, without analyzing compressed data.
Aiming at the edge computing mode, the problems of terminal data acquisition, storage and processing with various equipment objects and different protocol formats in the intelligent military camp IOT perception environment, mass perception data integration and association problems, situation perception linkage, data safety and reliability and intelligent camp fine management are solved. The research utilizes the exploratory data analysis thought to focus the processing and feedback of front-end data. The invention discloses an implementation method of an exploration type data analysis focus correlation technique based on an intelligent gateway of the Internet of things, which designs the intelligent gateway design for realizing cooperative data comprehensive control, data acquisition treatment focus extraction and 'neuron-like' type correlation feedback, provides an application scene construction method of an edge computing technique at a perception end of the Internet of things, an implementation method of a heterogeneous perception data comprehensive processing focus extraction technique and an implementation method of a data dynamic rule linkage technique, provides a reference basis for improving comprehensive management capability of intelligent camping, solving mass perception data integration and correlation problems, and lays an application foundation of exploration type data analysis in an intelligent mode.
The intelligent gateway management method can be used for the data refinement management of the Internet of things in the intelligent transformation process in the Internet of things environment of the barrage, can provide a technical means for intelligent camping environment management to prompt the back-end event sensitivity linkage from traditional rough mass data to scientific front-end perception data time repayment focusing based on an intelligent gateway application system in an edge computing scene, and can also be used for design guidance of newly-built intelligent gateways of the Internet of things and design reference of intelligent camping edge environment architecture.
In order to solve the three core problems of edge computing force improvement, heterogeneous data access, internet of things edge perception data focusing and data analysis result linkage, the invention researches related internet of things gateways and edge equipment, relies on intelligent integrated 'neuron-like' gateway equipment of intelligent barrage internet of things in an edge computing mode, and innovatively designs a technology implementation method in 3 layers of data communication transmission convergence, data management analysis and processing feedback. And (3) giving a model of sensor information acquisition, data processing, fusion analysis comparison, rule matching and behavior rule learning and an information processing chain visual angle on feedback by referring to a Lawson off-duty process model in general. Performing exploratory data analysis on collected data at a convergence layer; and at a front-end data analysis and management layer, the internet of things perceives data management and control, uniformly checks the quality of the data, audits the behavior of the perceiving data, authorizes the data in different types and different authority levels, forms a data focus, uniformly manages data resources, and uniformly counts and measures the data quantity. In the intelligent feedback layer, the rudder gives out a processing mode through two visual angles of front-end alarming and back-end service linkage, and finally, the processing method of the whole flow is realized, and the traditional working mechanism of the original gateway access equipment of the Internet of things is changed.
Examples
The embodiment provides an intelligent Internet of things exploring type data analysis focus correlation technology application system, which comprises the following steps:
(1) The heterogeneous protocol unified access and data conversion module (including the data conversion interface) operates as shown in fig. 2. Firstly, each sensing device facing the intelligent barrage at the sensing layer comprises a temperature and humidity sensor, a vibration sensor, a mobile sensing device, a Beidou device, a positioning device and the like, and for different types of devices and sensors, RFID access, WIFI access, WG access, network access, serial port access, WAPI access and the like are respectively realized at an access layer (the RFID access adapter, the WG access adapter, the network access adapter, the serial port access adapter, the WAPI access adapter are accessed through a unified data access interface); then, according to the compatible protocol insertion interface defined by the module, describing the data format, protocol type, data organization condition and the like of the protocol contract content to finish the acquisition and access of data; then, carrying out preliminary filtration on the accessed massive Internet of things original data to finish filtration (data missing cleaning and data deduplication cleaning), wherein the steps are divided into 2 aspects, namely deleting unrecorded information and inserting and calculating a missing value; 2 is to remove the data repetition value; finally, the protocol data and the data formats of various different sensing devices are converted, and the data is submitted according to the access protocol requirements.
(2) The operation of the perceptual data calculation module is shown in fig. 3. Preprocessing heterogeneous protocol sensing data at a data sensing front end, preprocessing by adopting modes of data reduction, data merging, data sampling, data dimension reduction, feature subset selection, feature generation, discretization, binarization, attribute conversion and the like aiming at different data, completing model selection through a data preprocessing configuration table, and then performing data access, access control, data conversion, data routing and data management, wherein further, the method can comprise data aggregation, data preprocessing, data access, access control, data conversion, data routing, data management and scheduling task monitoring; for example, temperature and humidity sensing data is oriented, and data reduction is carried out on two aspects of sample number and variable reduction respectively. The data exchange engine component is positioned in the core area of the gateway and is used for completing access service, access control service, message conversion service, data routing service, management service and the like according to the bus idea. The task scheduling processing component is mainly used for regularly running tasks from data acquisition to data calculation and analysis and finally to output task scheduling, and monitoring the running condition of the tasks at any time.
(3) The sensing data storage module is mainly used for storing structured and unstructured data and streaming data, and comprises a large amount of sensing data in various forms. After input collection and analysis such as mass internet of things environment perception, data are stored and processed according to a certain organization form and rule, so that effective data management is realized, and the perceived data logically has a hierarchical system of layer-by-layer connection: bits, characters, data elements, records, files, databases. According to the above division, the record of the storage module is stored as a logically related data element combination; storing the file as a collection of records; the invention relates to a storage mode for gateway configuration lightweight database completion structured data. Other unstructured data is stored in the form of files that are strategically disregarded by the front-end.
(4) The operation mode of the sensing data management and control module is shown in fig. 4. The internet of things perception data management and control module collects perception data through the gateway, uniformly supervises the quality of the data, audits the behavior of the perception data, authorizes the data in different types and different authority levels, and can uniformly manage the data resources, including data quality inspection, data audit, data authorization, resource management and data metering. The invention designs a mode of combining overview and detail aiming at focus extraction, and inquires related context in a database through abnormal point discovery, extracts influence factors and equipment object elements, and performs unified thematic management to obtain historical data information (similar situation statistics). Sensing data by utilizing a solution Forest algorithm and an improved K-means algorithm, analyzing to form an abnormal characteristic region, performing influence analysis, and obtaining minutiae based on environmental condition information (time, place, data value and time interval).
(5) The working modes of the front end alarm module and the linkage feedback module are shown in fig. 5. The feedback of the front-end alarm module is implemented in 4 aspects of real-time early warning prediction, real-time comparison alarm, monitoring management and alarm strategy management respectively. And inputting the early warning strategy into a network preposition unit (comprising n perception objects), inputting linkage rules (including strong association and weak association of rules) when the linkage rules are managed, obtaining various alarms, and then arranging the alarm input event flow (triggering and feedback). And carrying out front-end warning prompt on the intelligent Internet of things according to the preset early warning rules, abnormal difference values of comparison data and the like, and giving out corresponding warning prompts such as acousto-optic and the like.
(6) The linkage feedback module is mainly used for completing the functions of rule configuration management, event flow arrangement, decision module, log management and the like. The front-end sensing information is mainly transferred to a background service system, and event linkage feedback driven by events is completed through the module.
(7) The intelligent internet of things exploratory data analysis focus correlation technique workflow is shown in fig. 6: firstly judging whether gateway data are heterogeneous, if yes, carrying out data conversion, otherwise carrying out acquisition and import, then judging whether sensing calculation is needed, if yes, carrying out data calculation and storage, otherwise, carrying out data calculation and storage after submitting sensing data, then carrying out data management and control, judging whether correlation exists, if yes, carrying out front-end alarming and linkage, and otherwise, ending the workflow.
Comparison with other methods:
at computation time, the present invention only needs to migrate data computation or storage to the network edge in the vicinity of the end user. Thus, multiple computing nodes distributed over a network may offload computing pressure from a centralized data center, which may significantly reduce latency in message exchanges. Document [1] (Liu Yuancong ] mobile communication 2016 (040) 022:71-74) based on the design and research of an intelligent home gateway system of the Internet of things discloses the intelligent home gateway system of the Internet of things, and data information transmission and control between the Internet of things and the Ethernet are realized through protocol conversion and establishment of unified instructions and standards between different types of perception networks. Although the heterogeneous protocol unified access and data conversion are used in the document [1] and terminal data with various types and different protocol formats can be collected, stored and processed, the real-time response of the internet of things equipment is not considered, and the gateway can not respond in real time because the network bandwidth is occupied when the terminal equipment is more. Document [2] (Zhang Fujun; baobibo. A protocol self-adaptive Internet of things gateway system: china, CN201910598594.3[ P ]. 2019.07.04) discloses a protocol self-adaptive Internet of things gateway system, and compared with other intelligent Internet of things gateway systems, the invention solves the problems of differential management, autonomous collaborative management and equipment resource sharing of various equipment, and simultaneously integrates data computing processing capacity of edge measurement, thereby realizing Internet platform construction, but completely not considering the global situation of sensing data of the Internet of things and the abnormity and trend of important attention of users.
TABLE 1
In a specific implementation, the application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the computer program can run the invention content of the intelligent internet of things exploring type data analysis focus correlation technology application system and part or all of the steps in each embodiment. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random-access memory (random access memory, RAM), or the like.
It will be apparent to those skilled in the art that the technical solutions in the embodiments of the present invention may be implemented by means of a computer program and its corresponding general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied essentially or in the form of a computer program, i.e. a software product, which may be stored in a storage medium, and include several instructions to cause a device (which may be a personal computer, a server, a single-chip microcomputer MUU or a network device, etc.) including a data processing unit to perform the methods described in the embodiments or some parts of the embodiments of the present invention.
The invention provides an intelligent Internet of things exploratory data analysis focus correlation technology application system, which has a plurality of methods and approaches for realizing the technical scheme, the above description is only a preferred embodiment of the invention, and it should be noted that, for a person skilled in the art, a plurality of improvements and modifications can be made without departing from the principle of the invention, and the improvements and modifications should be regarded as the protection scope of the invention. The components not explicitly described in this embodiment can be implemented by using the prior art.
Claims (10)
1. The intelligent Internet of things exploring type data analysis focus correlation technique application system is characterized by comprising a heterogeneous protocol unified access and data conversion module, a perception data calculation module, a perception data storage module, a perception data management and control module, a front-end alarm module and a linkage feedback module;
the heterogeneous protocol unified access and data conversion module is used for realizing different protocol data access including WIFI access, WG access, network port access, serial port access and WAPI access by means of communication protocol format analysis, format conversion and data fusion for various Internet of things sensing devices; and through low-code assembly operation, input transmission channel template configuration operation is carried out on the provided unified data compiling operation panel, so that conversion between various different protocol data and data formats of a perception layer is realized.
2. The system of claim 1, wherein the perceived data calculation module comprises a data preprocessing module, a data exchange module, and a task scheduling processing module, wherein the data preprocessing module is configured to perform data preprocessing, and the data preprocessing comprises data reduction, data merging, data sampling, data dimension reduction, feature subset selection, discretization, binarization, and attribute transformation, and finally obtains preprocessed data.
3. The system of claim 2, wherein the data exchange module is configured to perform access services, access control services, message conversion services, data routing services, and management services; the access service is mapped to each protocol, realizes the access control service of the data through buffer priority setting and authority setting, comprises a sensitive security verification mechanism, source security control and frequency security control, completes unified processing of message formats through the message conversion service, and realizes the data distribution channel of marking data labels and the routing of receiving objects through the data routing service and the management service.
4. A system according to claim 3, wherein the task scheduling processing module is configured to perform tasks for task scheduling using a timing strategy and to supervise task operation; the task scheduling work comprises data acquisition, data calculation and analysis and finally data output.
5. The system of claim 4, wherein the awareness data storage module is configured to store structured, unstructured data and streaming data by providing a differentiated storage means.
6. The system of claim 5, wherein the perception data management and control module is configured to perform centralized inspection on quality of data with focus extraction as a target, perform centralized inspection including data inspection, checksum test, evaluate integrity, accuracy and reliability of data, synchronously analyze a storage structure of data, reject abnormal data, audit behavior of perception data, perform authorization of different types and different authority levels on the perception data including environmental data, equipment data, sensor data, location data, user behavior data and operation status data, perform centralized management on data resources, perform centralized counting and metering on data volume, perform centralized thematic management by applying heterogeneous perception data comprehensive processing focus extraction technology, query related context in a database, extract possible influencing factors and actual object elements, analyze perception data by using an Isolation Forest algorithm and an improved K-means algorithm, form an abnormal feature area, perform influence analysis, and obtain influence conditions of time, place and data value by analysis.
7. The system of claim 6, wherein the front-end alarm module performs front-end alarm prompt through preset early warning rules and comparison of data difference conditions, and the front-end alarm prompt comprises real-time data cross-boundary alarm, data change trend alarm, deviation from normal range alarm, abnormal data value alarm, repeated data value alarm, fault data value alarm and data missing alarm, and the front-end alarm information is transmitted to a background system according to preset linkage feedback rules, feedback flow arrangement and decision module to form event-driven linkage feedback, wherein the event-driven linkage feedback comprises alarm triggering linkage, timing linkage, data value linkage and characteristic value linkage;
the linkage feedback module is used for rule configuration management, event flow arrangement, decision module and log management.
8. The system of claim 7, wherein the data reduction is performed by defining redundant fields, the data combination combines data within a constant amount of time of the quantized values through time labels and changes in the quantized values, the data sampling is performed by setting sampling mark points, the data dimension reduction is performed by defining data dimension of interest to reduce data complexity, the feature subset selection is performed by presetting feature subset empirical values to give a scheme to provide a selection means, feature generation results are formed, and then the data is consolidated through discretization and binarization, attribute transformation.
9. The system of claim 8, wherein the sensory data management module performs the steps of:
step 1, the perception data management and control module analyzes and forms an abnormal characteristic area through an Isolation Forest algorithm and an improved K-means algorithm, and specifically comprises the following steps:
step 1-1, the Isolation Forest algorithm adopts a binary tree to segment data, and the depth of the data point in the binary tree reflects the discrete degree of the data in the set, which specifically comprises the following steps:
constructing a binary tree and combining the binary tree into a forest: for a given perceived dataset D, a property X is randomly selected 1 And at the attribute X 1 Randomly selecting a value Y between the maximum value and the minimum value, dividing the data smaller than the value Y in the data set D into the left branch of the binary tree, dividing the data larger than or equal to the value Y into the right branch, repeating the steps until the data set has only oneThe record of the strip or more than two identical records, or binary tree reaches the limit height;
calculating an abnormality coefficient: judging whether a data point x is an abnormal point or not through paths h (x) of the root node and the leaf node, and adopting the following formula:
where Score (x, n) is the anomaly coefficient of data point x in a binary tree of n samples; e (h (x)) represents the average path length of data x in each tree, C (n) is a correction value, and C (n) is given by: c (n) =2h (n-1) - (2 (n-1)/n), where the function H (k) =lnk+δh, δ is a euler constant;
step 1-2, the improved K-means algorithm comprises the following steps: after obtaining the abnormality degree coefficient of each sample, selecting the required cluster number k and an outlier filtering proportion r, wherein the outlier filtering proportion refers to the percentage of the number of filtered data to the total number of the samples;
through the value range of the statistical data in each dimension, the initial clustering center selection is carried out by combining the idea of the average interpolation method, and the value range is [ i, j ]]According to the required cluster number k, applying a formula Calculating an initial cluster center, wherein K n Representing the cluster center, T i Representing the lower limit value of the attribute T j An upper limit value representing the attribute T; obtain high-quality initial value K n Then, classifying the filtered abnormal samples into a residual sample set, and iterating for more than two times until the function converges to finally obtain an abnormal characteristic region;
step 2, coloring the abnormal characteristic region, searching a scene context according to the coloring characteristic, and analyzing the development trend of related parameters, wherein the scene context comprises associated equipment, management objects and a region;
and step 3, analyzing and extracting by utilizing the abnormal characteristic region and the development trend to obtain a focus, comparing with a historical case, and acquiring a relevant data result.
10. The system of claim 9, wherein the perception data storage module configures the storage mode of the structured and unstructured data in the list according to the strategic, tags the data, stores the structured data in a lightweight database according to a hierarchical system of the original data, the reorganized data and the preprocessed data, and completes file storage according to the transparent unstructured data strategy without analyzing the compressed data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310278190.2A CN116455930A (en) | 2023-03-21 | 2023-03-21 | Intelligent Internet of things exploring type data analysis focus correlation technology application system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310278190.2A CN116455930A (en) | 2023-03-21 | 2023-03-21 | Intelligent Internet of things exploring type data analysis focus correlation technology application system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116455930A true CN116455930A (en) | 2023-07-18 |
Family
ID=87122973
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310278190.2A Pending CN116455930A (en) | 2023-03-21 | 2023-03-21 | Intelligent Internet of things exploring type data analysis focus correlation technology application system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116455930A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117033143A (en) * | 2023-10-08 | 2023-11-10 | 常州瑞阳液压成套设备有限公司 | Intelligent monitoring data transmission system and method based on running state of big data |
-
2023
- 2023-03-21 CN CN202310278190.2A patent/CN116455930A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117033143A (en) * | 2023-10-08 | 2023-11-10 | 常州瑞阳液压成套设备有限公司 | Intelligent monitoring data transmission system and method based on running state of big data |
| CN117033143B (en) * | 2023-10-08 | 2024-01-26 | 常州瑞阳液压成套设备有限公司 | Intelligent monitoring data transmission system and method based on running state of big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102130783B (en) | Intelligent alarm monitoring method of neural network | |
| CN109587125B (en) | Network security big data analysis method, system and related device | |
| CN109542967A (en) | Smart city data-sharing systems and method based on XBRL standard | |
| CN112181960B (en) | Intelligent operation and maintenance framework system based on AIOps | |
| CN112332981B (en) | Data processing method and device | |
| CN106484709A (en) | A kind of auditing method of daily record data and audit device | |
| CN112016828B (en) | Industrial equipment health management cloud platform architecture based on streaming big data | |
| CN116455930A (en) | Intelligent Internet of things exploring type data analysis focus correlation technology application system | |
| CN113420009B (en) | Electromagnetic data analysis device, system and method based on big data | |
| CN109460393B (en) | Big data-based visual system for pre-inspection and pre-repair | |
| CN114153920A (en) | Big data edge platform and method | |
| CN110390020A (en) | The modeling method of semantic gateway and semantic gateway | |
| CN112328585A (en) | Data processing method and device | |
| CN115391444A (en) | Heterogeneous data acquisition and interaction method, device, equipment and storage medium | |
| Li et al. | A study of Big Data-based employees’ public opinion system construction | |
| CN113688284B (en) | Processing method and device of Internet of vehicles service data, electronic equipment and storage medium | |
| CN115604013B (en) | Industrial data interaction platform and interaction method | |
| US11836637B2 (en) | Construction method of human-object-space interaction model based on knowledge graph | |
| CN111143616A (en) | Video image data processing method | |
| CN114757448B (en) | Manufacturing inter-link optimal value chain construction method based on data space model | |
| CN111291029B (en) | Data cleaning method and device | |
| CN115310774A (en) | Method and system for sensing environmental resources of operation site | |
| CN113934796A (en) | Database subsystem for underground water application service system and data query method | |
| Dai | A Data Management Strategy for Property Management Information System Based on the Internet of Things. | |
| CN113269247B (en) | Training method and device for complaint early warning model, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |