CN112650889A - Method and system for constructing enterprise safety, environmental protection and security protection monitoring data warehouse - Google Patents
Method and system for constructing enterprise safety, environmental protection and security protection monitoring data warehouse Download PDFInfo
- Publication number
- CN112650889A CN112650889A CN202011580765.9A CN202011580765A CN112650889A CN 112650889 A CN112650889 A CN 112650889A CN 202011580765 A CN202011580765 A CN 202011580765A CN 112650889 A CN112650889 A CN 112650889A
- Authority
- CN
- China
- Prior art keywords
- data
- enterprise
- real
- time
- original
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 60
- 238000000034 method Methods 0.000 title claims abstract description 35
- 230000007613 environmental effect Effects 0.000 title claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 53
- 238000013500 data storage Methods 0.000 claims description 22
- 230000006399 behavior Effects 0.000 claims description 11
- 238000004458 analytical method Methods 0.000 claims description 9
- 238000005265 energy consumption Methods 0.000 claims description 9
- 238000010801 machine learning Methods 0.000 claims description 9
- 238000007619 statistical method Methods 0.000 claims description 9
- 238000013075 data extraction Methods 0.000 claims description 7
- 238000010276 construction Methods 0.000 claims description 6
- 238000007405 data analysis Methods 0.000 claims description 5
- 238000003908 quality control method Methods 0.000 claims description 4
- 238000004519 manufacturing process Methods 0.000 description 34
- 238000004364 calculation method Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 239000003344 environmental pollutant Substances 0.000 description 7
- 231100000719 pollutant Toxicity 0.000 description 7
- 238000003860 storage Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000007726 management method Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000005065 mining Methods 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000015572 biosynthetic process Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013144 data compression Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 231100000279 safety data Toxicity 0.000 description 1
- 230000035807 sensation Effects 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
- 230000000391 smoking effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Alarm Systems (AREA)
Abstract
The invention discloses a method and a system for constructing a plurality of bins of enterprise safety, environmental protection and security monitoring data, wherein the method comprises the following steps: collecting an original data source, wherein the original data source comprises log data, service handling data, fire-fighting video monitoring data and flow discharge data of an enterprise; removing null values and dirty data in the original data source from the original data source, and processing data exceeding a limit range in the original data source to form data detail; respectively summarizing and storing the data detail, wherein the data detail comprises data required by low time delay and data required by high time delay, and the method comprises the following steps: performing offline batch processing on the data with the low time delay requirement to obtain offline data; processing the data with high time delay requirement in real time to obtain real-time data; and responding to business requirements of the enterprise application, and outputting offline data or real-time data. The invention solves the problems that enterprises are safe, environment-friendly and large in data volume and store in a classified manner, and respond in time according to the time delay requirements of enterprise business.
Description
Technical Field
The invention relates to the technical field of enterprise safety, environmental protection and security monitoring data processing, in particular to a method and a system for constructing a plurality of bins of enterprise safety, environmental protection and security monitoring data.
Background
Fire safety and production behavior requirements of enterprises in the production and manufacturing process are a problem which needs to be concerned urgently. Safe production is an important index for judging whether the enterprises reach the standard or not. Safety and fire-fighting capacity of enterprises in the production and manufacturing process must meet certain standards. The safety production not only ensures the stable operation of enterprises in the production and manufacturing process, but also ensures the personal safety of operators of the enterprises. Avoiding the casualties and the economic loss. The temperature, humidity and smoke sensation in the production process of an enterprise are monitored, and the fire hazard and fire prevention accidents can not happen in production. The action of operators in the production process is monitored through videos, whether safety helmets are worn or not is judged, calls and gathering are carried out in production, and safety accidents are reduced. The pollution emission monitoring of enterprises is a key element for ensuring the environmental protection production of the enterprises. Monitoring the discharge flow, COD, PH value and energy consumption, and reducing the pollution to the natural environment. With the continuous requirement on industrial informatization, the manufacturing industry gradually changes from the traditional production mode, and the manufacturing industry has the following characteristics:
1. the industrial informatization degree is higher and higher, and a plurality of works can be completed through a computer without the prior manual operation.
2. The development of big data has leap development on the storage and processing capacity of the data, the network is continuously updated, and the real-time data stream acquisition and transmission are greatly influenced.
3. The development of video monitoring and cameras leads to higher and higher video definition; and with the appearance of the AI camera, identifying the field environment by using an image identification technology and monitoring a series of behaviors in the production process.
4. The enterprises greatly promote environmental protection and safety monitoring in the production and manufacturing process, utilize the pollutant discharge of the informatization monitoring enterprises and the safety production behavior monitoring of the production field, give an alarm in time, and reduce the pollution to the nature, casualties and economic loss of the enterprises.
Although the industrial informatization technology is continuously developed, the enterprise still has some problems in the safety and environmental protection production monitoring process:
1. various fire safety data monitoring methods are real-time, the interval time of data acquisition is short, the generated data volume is huge, the load of network transmission is large, and the data delay is high.
2. How to store data is very important, the traditional mysql, oracle and time sequence database have certain limitation on data query and data write-in, when the data storage capacity is huge, the data query speed is greatly influenced, the corresponding speed of the system is influenced, and the current production condition of an enterprise cannot be observed in time.
3. Data are all gathered together, but the system has a plurality of service functions, the requirements of each service on the data are different, a program is required to process the data first and then display the data, and the response speed of the system can also generate great influence.
4. As the running time of the system becomes longer, the data storage becomes larger and larger, the data is abnormal and disordered, and the rules and the values contained in the data are covered. Mining and analysis of the data becomes exceptionally difficult and can even lead to erroneous decisions.
The prior art scheme is as follows:
1. the method includes the steps that a traditional relational database is used for storing data such as mysql, oracle, and meanwhile, the data are processed through writing storage processes
2. The time sequence database is used to solve the problems of time sequence data real-time performance, high data generation frequency and huge data volume.
3. And the server and network resources are continuously increased, the load capacity and the response speed of the system are solved, and the stable and effective transmission of data is ensured.
The prior art has the following defects:
1. the traditional relational database has certain limit on reading and writing data, the pressure of the database is increased in the storage process, and when the data volume is too large, the processing speed becomes very slow, thus seriously affecting various service functions of the system.
2. Although the real-time database is greatly improved in real-time data processing capacity compared with traditional relational data, the system is still greatly influenced when the data volume is large, and meanwhile, some real-time databases are slow in concurrent query and some real-time databases are poor in writing capacity.
3. The maintenance cost and resources of enterprises are continuously increased, and the economic benefit of manufacturing enterprises is greatly influenced. And resources are not efficiently utilized.
Disclosure of Invention
The invention aims to solve the technical problems that the prior art is inconvenient to store and slow to process huge security data of an enterprise and cannot respond to various service data requirements of the enterprise in time, and provides a method and a system for constructing a digital warehouse of enterprise security, environmental protection and security monitoring data, so that the problems that the huge data volume of the enterprise is classified and stored and the enterprise responds in time according to the time delay requirements of the enterprise service are solved.
The invention is realized by the following technical scheme:
a method for constructing a plurality of warehouses of enterprise safety, environmental protection and security monitoring data comprises the following steps: step S1: and acquiring an original data source, wherein the original data source comprises log data, service handling data, fire-fighting video monitoring data and flow discharge data of an enterprise. Step S2: removing null values and dirty data in the original data source from the original data source, and processing data exceeding a limit range in the original data source to form data details; step S3: respectively summarizing and storing the data detail, wherein the data detail comprises data with a low time delay requirement and data with a high time delay requirement, and the method comprises the following substeps: performing offline batch processing on the data with the low time delay requirement to obtain offline data; processing the data with high time delay requirement in real time to obtain real-time data; step S4: and responding to business requirements of enterprise applications, and outputting the offline data or the real-time data.
The invention aims at the information processing requirement of enterprise safety, environmental protection and security monitoring data, extracts and classifies the data before storing the original data, and carries out off-line processing or real-time processing on the data aiming at the characteristic that different data have different time-delay requirements. The invention solves the problem of huge data volume storage in enterprise safety and environmental protection. And a data bin is established, so that the problem of slow data query and data transmission is solved. And processing and analyzing the data by utilizing a big data technology, and classifying and storing the data aiming at different services. The problem of the delay of data is solved, the data can be displayed with low delay, and enterprises can know the discharge of pollutants and the safety problem of a production field in time. The data storage and processing capacity is greatly improved, and quick and timely response can be achieved according to business requirements of an enterprise application level.
Further, the real-time data is subscribed and published through a message queue.
Further, data quality governance is performed on all the processing procedures from step S1 to step S4, and the data quality governance includes: and checking the accuracy and the uniqueness of the data, and removing redundant repeated data.
Further, the integrity and the reasonableness of the data processed in the step S2 are verified.
Further, the original data source and the data detail are consistent in data granularity.
Further, the step S2 includes the following steps: the business handling data is extracted and processed through ETL, and the formed data detail comprises enterprise information, equipment information, personnel information and camera information; the log data are collected through the flash, and the generated data detail comprises collected log data and system log data; the fire video monitoring and flow discharge data are processed through edges, and formed data detail comprises pictures, videos, humidity, temperature, flow and energy consumption.
The invention also discloses a warehouse construction system for enterprise safety, environmental protection and security monitoring data, which comprises an original data layer, a data detail layer, a data storage analysis layer, a data application layer and a data quality control layer, wherein the original data layer is used for storing data; the original data layer: the system comprises a data storage module, a data processing module and a data processing module, wherein the data storage module is used for storing an original data source, and the original data source comprises log data, service handling data, fire-fighting video monitoring data and flow discharge data of an enterprise; the data detail layer: the data extraction module is used for storing data details which are original data sources after data extraction; the data storage analysis layer: the data detail is respectively subjected to summarizing processing to obtain offline data and real-time data, and the offline data and the real-time data are stored; the data application layer: the system is used for responding to business requirements of enterprises and displaying the offline data or the real-time data; the data quality treatment layer is as follows: the method is used for checking the accuracy and the uniqueness of the data and removing redundant repeated data.
Further, the data application layer comprises statistical analysis, machine learning and real-time business.
Further, the statistical analysis comprises the total number of enterprises, the total number of equipment, flow discharge and sensor alarm times; the machine learning comprises behavior pictures, equipment current signals, human face pictures and equipment vibration signals; the real-time service comprises user information, enterprise information, basic authority data and a picture identification result.
Further, the data detail layer comprises: enterprise information, device information, personnel information, camera information, acquisition logs, system logs, pictures, temperature, video, humidity, flow and energy consumption.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the problems of fire safety monitoring, production behavior safety monitoring and pollution emission data storage are solved.
2. A data warehouse for fire safety monitoring, production behavior safety monitoring and pollution emission data is established, effective management of the data is achieved, data quality is effectively improved, and uniqueness, consistency, stability and relevance of the data are guaranteed. Provides a good basis for mining analysis data.
3. And establishing a plurality of theme domains for fire safety monitoring, production behavior safety monitoring, pollution discharge and system service data, wherein the theme domains correspond to a plurality of service modules respectively. The pertinence and the specificity of data are maintained; various discrete data and real-time data are formed, and the query capability and the intuition of the data are improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the principles of the invention. In the drawings:
FIG. 1 is a diagram of a data architecture according to the present invention;
FIG. 2 is a diagram of a data detail formation process;
FIG. 3 is a flow diagram of data storage analysis;
FIG. 4 is a schematic diagram of data application in a service;
FIG. 5 is a general flowchart of example 2;
FIG. 6 is a schematic flow chart of processing data of example 2.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to examples and accompanying drawings, and the exemplary embodiments and descriptions thereof are only used for explaining the present invention and are not meant to limit the present invention.
Example 1
This embodiment 1 is a method and a system for constructing a plurality of warehouses of enterprise security, environmental protection and security monitoring data, wherein the method includes the following steps:
1. collecting an original data source, wherein the original data source comprises log data, service handling data, fire-fighting video monitoring data and flow discharge data of an enterprise;
2. removing null values and dirty data in the original data source from the original data source, and processing data exceeding a limit range in the original data source to form data detail; checking the integrity and reasonableness of the data detail; the data detail is consistent with the original data source in data granularity;
business handling data is extracted and processed through ETL, and formed data detail comprises enterprise information, equipment information, personnel information and camera information; collecting log data through flash, wherein the generated data detail comprises collected log data and system log data; the fire video monitoring and the flow discharge data are processed through edges, and the formed data detail comprises pictures, videos, humidity, temperature, flow and energy consumption.
3. Respectively summarizing and storing the data detail, wherein the data detail comprises data required by low time delay and data required by high time delay, and the method comprises the following steps: performing offline batch processing on the data with the low time delay requirement to obtain offline data; processing the data with high time delay requirement in real time to obtain real-time data;
4. responding to business requirements of enterprise application, and outputting offline data or real-time data; real-time data is subscribed and published through a message queue.
This embodiment 1 has the information-based processing demand to enterprise safety ring protects and security protection monitored control data, carries out data extraction and classification before storing raw data, has the characteristics of different time ductility demands to different data, carries out off-line processing or real-time processing with data. This embodiment 1 has solved the storage problem of the huge data volume of enterprise's safety ring protection. And a data bin is established, so that the problem of slow data query and data transmission is solved. And processing and analyzing the data by utilizing a big data technology, and classifying and storing the data aiming at different services. The problem of the delay of data is solved, the data can be displayed with low delay, and enterprises can know the discharge of pollutants and the safety problem of a production field in time. The data storage and processing capacity is greatly improved, and quick and timely response can be achieved according to business requirements of an enterprise application level.
In this embodiment, data quality control is required to be performed in all processing procedures, and the data quality control includes: the accuracy and the uniqueness of the data are checked, redundant repeated data are removed, and the like.
The system of embodiment 1 includes an original data layer, a data detail layer, a data storage analysis layer, a data application layer, and a data quality management layer; original data layer: the system comprises a data storage module, a data processing module and a data processing module, wherein the data storage module is used for storing an original data source, and the original data source comprises log data of an enterprise, service handling data, fire-fighting video monitoring data and flow discharge data; data detail layer: the data extraction module is used for storing data details which are original data sources after data extraction; data storage analysis layer: the data detail collection device is used for respectively collecting and processing the data detail to obtain off-line data and real-time data, and the off-line data and the real-time data are stored; a data application layer: the system is used for responding to business requirements of enterprises and displaying off-line data or real-time data; data quality governance layer: the method is used for checking the accuracy and the uniqueness of the data and removing redundant repeated data. The data application layer includes statistical analysis, machine learning, and real-time traffic. The statistical analysis comprises the total number of enterprises, the total number of equipment, flow discharge and sensor alarm times; the machine learning comprises behavior pictures, equipment current signals, face pictures and equipment vibration signals; the real-time service comprises user information, enterprise information, basic authority data and a picture identification result. The data detail layer comprises: enterprise information, device information, personnel information, camera information, acquisition logs, system logs, pictures, temperature, video, humidity, flow and energy consumption.
Example 2
The embodiment 2 is a data warehouse construction method for enterprise safety, environmental protection and security monitoring data, the core functions of which are mainly divided into an original data layer, a data detail layer, a data storage and analysis layer, data application and data quality management, and the four parts are shown in fig. 1.
The original data layer stores original data, directly loads original logs, service data, video data and pollutant data, keeps original appearance of the data and does not perform any treatment.
The structure and granularity of the data detail layer data are kept consistent with those of the original data layer. But the data of each data source is stored in one data source through the data acquisition system, and meanwhile, the null value and dirty data of the original data source are removed, and the data exceeding the limit range are processed; and checking the integrity and reasonableness of the data. The data detail layer comprises log data of the system, system service handling data, fire-fighting video monitoring data and environment-friendly data. The formation process as detailed in fig. 2 data:
the business transaction data of the enterprise is extracted and processed by the ETL to form enterprise information, equipment information, personnel information, camera information and other data. And collecting the log data through the flash to generate collected log data and system log data. The fire video monitoring and flow discharge data are processed by the edge to form data such as pictures, videos, humidity, temperature, flow and energy consumption.
Data storage and analysis, as shown in fig. 3, the data storage and analysis process is to perform data processing on data details, calculate and summarize to form processing data, and store or analyze the processing data. The data summarization calculation is divided into two modes; offline batch calculation and real-time calculation. And processing the detail data into light summarized data or high summarized data through offline batch calculation to form an offline data summarized data set. For data with higher delay requirement, the data can be subscribed and published through a message queue in a real-time stream calculation mode, and summarized data is formed through real-time calculation and provided to a data application layer.
The offline batch computation has low requirement on the time delay of data, for example, a machine learning service, needs to train a model by using historical data, and the training process is performed offline. And the detail data is subjected to summary calculation or processing to form a coarse-grained data set, and the data sets comprise structured data, unstructured data and semi-structured data. The summarized data has the main function of providing the services for query operation, and only the query efficiency of the data needs to be concerned, so that the data is generally stored in a database with higher query efficiency.
The real-time data calculation has higher requirement on the time delay of the data, and meanwhile, the data acquisition frequency is high and the data volume is large in the industrial environment. High-speed writing, fast query, stability and extremely strong data compression capability are required. Because the flow discharge data collected by the sensor are all provided with time stamps and are generated according to the time sequence, the detail data are stored in the time sequence database after being processed by the computing platform.
The data application layer is the last layer at which data interacts directly with the business application. Details of the data application layer are shown in fig. 4.
The data is divided into three subject domains, statistical analysis, machine learning, real-time business. The statistical analysis mainly provides data support for various statistical reports, and the statistical analysis is data summarized at a higher degree. Such as enterprise equipment quantity statistics reports, flow discharge total statistics, sensor alarm quantity statistics, enterprise video quantity change reports, and the like.
The machine learning subject domain mainly provides training data basis for each artificial intelligence model. Including behavioral picture data (including smoking, without safety helmet, making phone calls, gathering, putting on shelf), human face pictures, device current signal data, device vibration signal data, etc. These types of data provide training data sets for 4 large models such as a face recognition model, equipment state monitoring, equipment health diagnosis and the like.
The real-time data topic domain mainly provides data basis for inquiry and transaction of each real-time service. Such as processing the identification result, user registration, user permission change, video monitoring result processing, etc.
Data quality governance runs through the entire number of bins. The accuracy of the data is verified, and data errors caused by artificial reasons are avoided. And checking the uniqueness of the data, removing redundant data and repeating the data. Checking the consistency of the data, for example: inconsistent naming, inconsistent data structures, inconsistent constraint rules. Data entity inconsistencies, for example: inconsistent data encoding, inconsistent naming and meaning, inconsistent classification levels, and inconsistent life cycles. Checking the relevance of data, for example: and the functional relation, the correlation coefficient, the main foreign key relation, the index relation and the like avoid influencing the result of data analysis due to the lack of the data relation.
This embodiment 2 has solved the data storage problem of fire safety monitoring, production action safety monitoring and pollutant discharge and has solved the fire safety monitoring on the industrial production, and production action monitoring and pollutant discharge are huge data volume storage problem. A data warehouse for fire safety monitoring, production behavior safety monitoring and pollution emission data is established to realize effective management of the data, the data quality is effectively improved, and the uniqueness, consistency, stability and relevance of the data are ensured. Provides a good basis for mining analysis data. And establishing a plurality of theme domains for fire safety monitoring, production behavior safety monitoring, pollution discharge and system service data, wherein the theme domains correspond to a plurality of service modules respectively. The pertinence and the specificity of data are maintained; various mild summary and high summary data are formed, and the query capability and the intuition of the data are improved.
The general flow of this example 2 is shown in fig. 5: extracting data of each data source (service data, video monitoring data, log data and environmental protection data) and extracting detail data into a data detail library (ODS). The detail data is processed by data, and the data processing is mainly divided into two types; the first type is calculation of historical data, such as (total number of enterprises, total number of personnel, number of video categories, number of pictures, total pollutant emission, total energy consumption and the like), and the historical detail data is calculated off line through a big data platform, and the processed data is stored in a database; the second type is real-time data, such as real-time resource consumption, business handling, video monitoring and the like, the big data is used for calculating the platform in real time, and the processed summary data is stored. The application module sends a data request to the real-time data modules of the offline data collection module, the offline data module sends processed data to the application module according to the data request content of the application module, and the application module performs visual display on the obtained processed data; the application module and the real-time data module subscribe and publish the messages, the application module sends related message requests needing subscription to the real-time data module, the real-time data module publishes the processing data to the application module in a message mode according to the subscription content, and the application module carries out related business processing according to the message content.
The processing data flow is shown in fig. 6: the data processing flow is divided into two types, historical detail data are subjected to Spark batch processing, the data are summarized and calculated in a multi-task mode, if a certain task fails, a failure reason is returned, the next task is continuously executed, and offline summarized data are formed and stored in a database. And (3) processing the real-time data through a Flink streaming process, performing processing calculation on the real-time data stream, and completing storage and consumption of the data through a kafka message system by the real-time stream.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method for constructing a plurality of bins of enterprise safety, environmental protection and security monitoring data is characterized by comprising the following steps:
step S1: collecting an original data source, wherein the original data source comprises log data, service handling data, fire-fighting video monitoring data and flow discharge data of an enterprise;
step S2: removing null values and dirty data in the original data source from the original data source, and processing data exceeding a limit range in the original data source to form data details;
step S3: respectively summarizing and storing the data detail, wherein the data detail comprises data with a low time delay requirement and data with a high time delay requirement, and the method comprises the following substeps:
performing offline batch processing on the data with the low time delay requirement to obtain offline data;
processing the data with high time delay requirement in real time to obtain real-time data;
step S4: and responding to business requirements of enterprise applications, and outputting the offline data or the real-time data.
2. The method for constructing the enterprise security, environmental protection and security monitoring data warehouse of claim 1, wherein the real-time data is subscribed and published through a message queue.
3. The method for constructing the warehouse of the enterprise safety, environmental protection and security monitoring data as claimed in claim 1, wherein the data quality governance is performed on all the processing procedures from step S1 to step S4, and the data quality governance comprises: and checking the accuracy and the uniqueness of the data, and removing redundant repeated data.
4. The method for constructing the enterprise safety, environmental protection and security monitoring data warehouse according to claim 1, wherein the integrity and reasonableness of the data processed in the step S2 are verified.
5. The enterprise security, environmental protection and security monitoring data warehouse construction method according to claim 1, wherein the original data source and the data detail are consistent in data granularity.
6. The method for constructing the enterprise security, environmental protection and security monitoring data warehouse of claim 1, wherein the step S2 includes the following steps:
the business handling data is extracted and processed through ETL, and the formed data detail comprises enterprise information, equipment information, personnel information and camera information;
the log data are collected through the flash, and the generated data detail comprises collected log data and system log data;
the fire video monitoring and flow discharge data are processed through edges, and formed data detail comprises pictures, videos, humidity, temperature, flow and energy consumption.
7. A warehouse construction system for enterprise safety, environmental protection and security monitoring data is characterized by comprising an original data layer, a data detail layer, a data storage and analysis layer, a data application layer and a data quality control layer;
the original data layer: the system comprises a data storage module, a data processing module and a data processing module, wherein the data storage module is used for storing an original data source, and the original data source comprises log data, service handling data, fire-fighting video monitoring data and flow discharge data of an enterprise;
the data detail layer: the data extraction module is used for storing data details which are original data sources after data extraction;
the data storage analysis layer: the data detail is respectively subjected to summarizing processing to obtain offline data and real-time data, and the offline data and the real-time data are stored;
the data application layer: the system is used for responding to business requirements of enterprises and displaying the offline data or the real-time data;
the data quality treatment layer is as follows: the method is used for checking the accuracy and the uniqueness of the data and removing redundant repeated data.
8. The enterprise security, environmental protection and security monitoring data warehouse building system of claim 7, wherein the data application layer comprises statistical analysis, machine learning and real-time business.
9. The enterprise security, environmental protection and security monitoring data counting warehouse construction system of claim 8, wherein the statistical analysis comprises total number of enterprises, total number of devices, discharge of flow and sensor alarm times;
the machine learning comprises behavior pictures, equipment current signals, human face pictures and equipment vibration signals;
the real-time service comprises user information, enterprise information, basic authority data and a picture identification result.
10. The enterprise security, environmental protection and security monitoring data warehouse construction system of claim 7, wherein the data detail layer comprises: enterprise information, device information, personnel information, camera information, acquisition logs, system logs, pictures, temperature, video, humidity, flow and energy consumption.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011580765.9A CN112650889A (en) | 2020-12-28 | 2020-12-28 | Method and system for constructing enterprise safety, environmental protection and security protection monitoring data warehouse |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011580765.9A CN112650889A (en) | 2020-12-28 | 2020-12-28 | Method and system for constructing enterprise safety, environmental protection and security protection monitoring data warehouse |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112650889A true CN112650889A (en) | 2021-04-13 |
Family
ID=75363574
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011580765.9A Pending CN112650889A (en) | 2020-12-28 | 2020-12-28 | Method and system for constructing enterprise safety, environmental protection and security protection monitoring data warehouse |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112650889A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656370A (en) * | 2021-08-16 | 2021-11-16 | 南方电网数字电网研究院有限公司 | Data processing method and device for power measurement system and computer equipment |
| CN114637786A (en) * | 2022-03-14 | 2022-06-17 | 徐工汉云技术股份有限公司 | Vehicle working hour off-line calculation method and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107515927A (en) * | 2017-08-24 | 2017-12-26 | 深圳市云房网络科技有限公司 | A kind of real estate user behavioural analysis platform |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
| US20180218051A1 (en) * | 2017-02-02 | 2018-08-02 | Cvs Pharmacy, Inc. | Data Analysis Reporting Tool |
| CN108365971A (en) * | 2018-01-10 | 2018-08-03 | 深圳市金立通信设备有限公司 | Daily record analytic method, equipment and computer-readable medium |
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108768741A (en) * | 2018-06-08 | 2018-11-06 | 上海新炬网络技术有限公司 | A kind of network management data interacted system based on big data |
| CN110362622A (en) * | 2019-07-22 | 2019-10-22 | 江苏满运软件科技有限公司 | Real-time stream processing system, method, equipment and storage medium based on real-time number storehouse |
| CN111274255A (en) * | 2020-01-20 | 2020-06-12 | 拉扎斯网络科技（上海）有限公司 | Service data monitoring method and system, monitoring architecture, equipment and storage medium |
| CN112000636A (en) * | 2020-08-31 | 2020-11-27 | 民生科技有限责任公司 | User behavior statistical analysis method based on Flink streaming processing |
-
2020
- 2020-12-28 CN CN202011580765.9A patent/CN112650889A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180218051A1 (en) * | 2017-02-02 | 2018-08-02 | Cvs Pharmacy, Inc. | Data Analysis Reporting Tool |
| CN107515927A (en) * | 2017-08-24 | 2017-12-26 | 深圳市云房网络科技有限公司 | A kind of real estate user behavioural analysis platform |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
| CN108365971A (en) * | 2018-01-10 | 2018-08-03 | 深圳市金立通信设备有限公司 | Daily record analytic method, equipment and computer-readable medium |
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108768741A (en) * | 2018-06-08 | 2018-11-06 | 上海新炬网络技术有限公司 | A kind of network management data interacted system based on big data |
| CN110362622A (en) * | 2019-07-22 | 2019-10-22 | 江苏满运软件科技有限公司 | Real-time stream processing system, method, equipment and storage medium based on real-time number storehouse |
| CN111274255A (en) * | 2020-01-20 | 2020-06-12 | 拉扎斯网络科技（上海）有限公司 | Service data monitoring method and system, monitoring architecture, equipment and storage medium |
| CN112000636A (en) * | 2020-08-31 | 2020-11-27 | 民生科技有限责任公司 | User behavior statistical analysis method based on Flink streaming processing |
Non-Patent Citations (2)
| Title |
|---|
| 张轶: "RDB平台和大数据平台混搭式的数据中心设计", 《计算机技术与发展》 * |
| 李明桂 等: "基于大数据的安全事件挖掘框架", 《通信技术》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656370A (en) * | 2021-08-16 | 2021-11-16 | 南方电网数字电网研究院有限公司 | Data processing method and device for power measurement system and computer equipment |
| CN113656370B (en) * | 2021-08-16 | 2024-04-30 | 南方电网数字电网集团有限公司 | Data processing method and device for electric power measurement system and computer equipment |
| CN114637786A (en) * | 2022-03-14 | 2022-06-17 | 徐工汉云技术股份有限公司 | Vehicle working hour off-line calculation method and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107256219B (en) | Big data fusion analysis method applied to mass logs of automatic train control system | |
| CN111885040A (en) | Distributed network situation perception method, system, server and node equipment | |
| CN111162949A (en) | Interface monitoring method based on Java byte code embedding technology | |
| CN108021809A (en) | A kind of data processing method and system | |
| CN110866642A (en) | Security monitoring method and device, electronic equipment and computer readable storage medium | |
| CN106844161A (en) | Abnormal monitoring and Forecasting Methodology and system in a kind of carrier state stream calculation system | |
| CN112650889A (en) | Method and system for constructing enterprise safety, environmental protection and security protection monitoring data warehouse | |
| CN111046022A (en) | Database auditing method based on big data technology | |
| CN112527886A (en) | Data warehouse system based on urban brain | |
| CN109308290B (en) | Efficient data cleaning and converting method based on CIM | |
| CN111581056B (en) | Software engineering database maintenance and early warning system based on artificial intelligence | |
| CN113918412A (en) | Real-time abnormal log analysis method and system | |
| CN112052134A (en) | Service data monitoring method and device | |
| CN110149303B (en) | Party-school network security early warning method and early warning system | |
| CN114443437A (en) | Alarm root cause output method, apparatus, device, medium, and program product | |
| CN110661999A (en) | Video monitoring system based on big data | |
| CN111414355A (en) | Offshore wind farm data monitoring and storing system, method and device | |
| CN113986656B (en) | Power grid data safety monitoring system based on data center platform | |
| CN115470279A (en) | Data source conversion method, device, equipment and medium based on enterprise data | |
| CN115309724A (en) | Data warehouse design and data analysis acceleration system for industrial analysis | |
| Chen et al. | Research on equipment situation display based on multi-source data fusion | |
| CN116701525A (en) | Early warning method and system based on real-time data analysis and electronic equipment | |
| CN114049036A (en) | Data computing platform, method, device and storage medium | |
| CN112784129A (en) | Pump station equipment operation and maintenance data supervision platform | |
| Wang | Research on public opinion and early warning analysis model of network emergencies based on decision tree |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210413 |
|
| RJ01 | Rejection of invention patent application after publication |