CN112417141A - Domestic industrial control system curve data query processing method - Google Patents
Domestic industrial control system curve data query processing method Download PDFInfo
- Publication number
- CN112417141A CN112417141A CN202011316070.XA CN202011316070A CN112417141A CN 112417141 A CN112417141 A CN 112417141A CN 202011316070 A CN202011316070 A CN 202011316070A CN 112417141 A CN112417141 A CN 112417141A
- Authority
- CN
- China
- Prior art keywords
- data
- curve
- retrieval
- abnormal
- display
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/34—Browsing; Visualisation therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/335—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/338—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Feedback Control In General (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a curve data query processing method for a domestic industrial control system, which comprises user-defined retrieval condition setting, multithreading synchronous incremental data retrieval, abnormal data detection and normal working condition data sampling optimization display. Specifically, a user firstly inputs a retrieval condition on a data retrieval interface; then, according to retrieval conditions input by a user, establishing a data index table through a TF-IDF algorithm, and utilizing multithreading synchronous increment for query; then, abnormal data detection is carried out through the Isolation Forest algorithm; finally, dynamically shifting increment to optimize curve drawing effect through a local sampling algorithm; according to the invention, a data query method is optimized from massive field industrial control data, and abnormal data detection and optimized analysis screening display are carried out on the acquired data, so that data concerned by a user can be displayed quickly, efficiently and accurately, the system operation pressure is reduced, the data display efficiency is improved, and curve data of real field operation conditions can be displayed quickly.
Description
Technical Field
The invention belongs to the technical field of curve query of industrial control systems, and particularly relates to a curve data query processing method of a domestic industrial control system.
Background
With the continuous progress of the information technology level, the data analysis requirement in the field of domestic industrial control is continuously developed, and the operation data curve of the field equipment needs to be timely and efficiently displayed in front of a user. At present, a mainstream industrial control system collects real-time data of industrial field equipment through a network for storage, and a curve of the industrial control system mostly adopts a full data drawing method, namely, for each data query, data query in a specified time period is carried out in full quantity and is drawn on the curve. Due to the improvement of the field hardware level, the real-time data sampling interval is continuously shortened, millisecond-level data storage is achieved at present, and the annual data storage capacity of a single measuring point of the system is in the order of tens of millions. While the current data retrieval mode provides great pressure to a system database, long-delay, stuck and other reactions also occur in curve drawing, and the data analysis efficiency of a user and the safety and stability of the system are greatly influenced.
Disclosure of Invention
In order to overcome the problems in the prior art, the invention aims to provide a curve data query processing method for a domestic industrial control system, which is used for carrying out multithreading optimization from data query and data processing and improving the analysis efficiency of a user and the running speed of the system.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a curve data query processing method for a domestic industrial control system comprises the following steps:
1) the method comprises the following steps that a user inputs measuring point query condition information in a custom retrieval interface, and the query condition information comprises the following steps: inquiring information such as time interval, measuring point type, statistical type and the like, and adopting intervals.
2) Establishing a data retrieval table from massive industrial data by using measuring point query condition information input in the step 1) and a Term Frequency-Inverse text Frequency (TF-IDF) algorithm, and quickly retrieving curve data related to a given query condition by using a multi-thread synchronous increment retrieval method;
3) detecting abnormal data by using the curve data inquired in the step 2) through an Isolation Forest algorithm, calculating an influence factor of the abnormal data, and eliminating the abnormal data from the curve data inquired in the step 2) to obtain curve data under a normal working condition;
4) optimizing the curve data under the normal working condition obtained in the step 3), and then visually displaying, wherein the optimization principle is mainly to perform local feature display on a curve consisting of a large number of data points, and the local feature display is realized by setting a historical time interval and a sampling interval; and the layered visualization effect is realized by setting dynamic offset for the overlapping curves.
In the step 1), a user-defined condition retrieval interface is provided, the user is supported to set retrieval time, a statistical data display mode is provided, the user can configure point picking, mean value, maximum value and minimum value modes for retrieval display, and the selection of the type of the measuring point is provided for display.
In step 2), a data retrieval table is established through a TF-IDF mathematical function, so that the retrieval speed is improved in the actual data retrieval process, and the method specifically comprises the following steps:
TF-IDF(x)＝TF(x)*IDF(x)
wherein N represents the total number of texts in the corpus, N (x) represents the total number of texts in the corpus containing the word x, idf (x) represents the frequency of the word x in all texts, TF represents the frequency of each word in the texts, and TF (x) represents the frequency of the word x in the current text.
In step 3), establishing an Isolation Forest algorithm function model, performing detection analysis on abnormal data, and calculating influence factors of the abnormal data, wherein the influence factors are as follows:
wherein E (h (y)) represents the average of the path lengths of the data y in the multiple binary trees, ψ represents the number of samples of the training samples of a single binary tree, and C (ψ) represents the average path length of the binary tree constructed with the ψ number of samples; the threshold value of the abnormal point can be artificially set, source (y) represents the influence factor of the data y, when the influence factor of the source (y) is greater than the threshold value, the abnormal point is classified, and when the influence factor of the source (y) is less than the threshold value, the abnormal point is regarded as a non-abnormal point.
In the step 4), for curve data under normal working conditions, due to the huge data amount, data distortion and low curve resolution can be caused if the curve data are all displayed; therefore, in the step 4), a historical data statistical analysis function is introduced, and curve drawing is performed on the measuring point values to be displayed by setting the sampling interval time of the data points and the data display time period so as to adapt to the screen resolution; in addition, aiming at the numerical value superposition condition of different measuring points, step 4) introduces dynamic offset, and different measuring point values of numerical superposition are multiplied by different weights, so that the trend on the drawing area forms the condition of layered display.
In the above scheme of the invention: the normal operation condition data are displayed in an optimized mode, specifically, for normal operation condition data, the condition that the data size is too large is displayed, and corresponding sampling intervals are automatically generated according to the time intervals of the displayed data, so that the drawn curve is adaptive to the screen resolution, and the condition that the trend is blocked and distorted under the condition of large data size is avoided. For example, if a specified drawing area shows up to 3600 station values, and if station data with a drawing time interval of 8 hours is required, the system automatically generates an adoption interval of 8 seconds. In addition, for the measurement point drawing with completely overlapped numerical values, which is often the case of drawing switch measurement points, the invention introduces a dynamic offset method and realizes the effect of automatic layered display in a drawing area by multiplying the overlapped measurement point values by different weights.
According to the method, a data query method is optimized from massive field industrial control data, and the obtained data is optimized, analyzed, screened and displayed, so that the data concerned by a user can be displayed quickly, efficiently and accurately, the system operation pressure is reduced, the data display efficiency is improved, and curve data of real field operation conditions can be displayed quickly.
Drawings
Fig. 1 is a schematic diagram of a curve data query processing method of a domestic industrial control system according to the present invention.
Detailed Description
The present invention will be better understood and implemented by those skilled in the art by the following detailed description of the technical solution of the present invention with reference to the accompanying drawings and specific examples, which are not intended to limit the present invention.
As shown in fig. 1, the method for inquiring and processing curve data of a domestic industrial control system of the present invention includes the following steps:
1) the method comprises the following steps that a user inputs measuring point query condition information in a custom retrieval interface, and the query condition information comprises the following steps: inquiring information such as time interval, measuring point type, statistical type and the like, and adopting intervals.
2) Establishing a data retrieval table from massive industrial data by using measuring point query condition information input in the step 1) and a Term Frequency-Inverse text Frequency (TF-IDF) algorithm, and quickly retrieving curve data related to a given query condition by using a multi-thread synchronous increment retrieval method;
3) detecting abnormal data by using the curve data inquired in the step 2) through an Isolation Forest algorithm, calculating an influence factor of the abnormal data, and eliminating the abnormal data from the curve data inquired in the step 2) to obtain curve data under a normal working condition;
4) optimizing the curve data under the normal working condition obtained in the step 3), and then visually displaying, wherein the optimization principle is mainly to perform local feature display on a curve consisting of a large number of data points, and the local feature display is realized by setting a historical time interval and a sampling interval; and the layered visualization effect is realized by setting dynamic offset for the overlapping curves.
In the step 1), a user-defined condition retrieval interface is provided, the user is supported to set retrieval time, a statistical data display mode is provided, the user can configure point picking, mean value, maximum value and minimum value modes for retrieval display, and the selection of the type of the measuring point is provided for display. For example, the user may select the station information showing only the switching value or the analog value or select the station showing the switching value and the analog value in a mixed manner.
In step 2), a data retrieval table is established through a TF-IDF mathematical function, so that the retrieval speed is improved in the actual data retrieval process, and the method specifically comprises the following steps:
TF-IDF(x)＝TF(x)*IDF(x)
wherein N represents the total number of texts in the corpus, N (x) represents the total number of texts in the corpus containing the word x, idf (x) represents the frequency of the word x in all texts, TF represents the frequency of each word in the texts, and TF (x) represents the frequency of the word x in the current text.
In step 3), establishing an Isolation Forest algorithm function model, performing detection analysis on abnormal data, and calculating influence factors of the abnormal data, wherein the influence factors are as follows:
wherein E (h (y)) represents the average of the path lengths of the data y in the multiple binary trees, ψ represents the number of samples of the training samples of a single binary tree, and C (ψ) represents the average path length of the binary tree constructed with the ψ number of samples; the threshold value of the abnormal point can be artificially set, source (y) represents the influence factor of the data y, when the influence factor of the source (y) is greater than the threshold value, the abnormal point is classified, and when the influence factor of the source (y) is less than the threshold value, the abnormal point is regarded as a non-abnormal point.
In the step 4), for curve data under normal working conditions, due to the huge data amount, data distortion and low curve resolution can be caused if the curve data are all displayed; therefore, in the step 4), a historical data statistical analysis function is introduced, and curve drawing is performed on the measuring point values to be displayed by setting the sampling interval time of the data points and the data display time period so as to adapt to the screen resolution; for example, a drawing area is specified to display 3600 measurement point values at most, so that a sampling interval is automatically generated for different time intervals, so that the drawn curve avoids the blockage and data distortion. In addition, aiming at the numerical value superposition condition of different measuring points, step 4) introduces dynamic offset, and different measuring point values of numerical superposition are multiplied by different weights, so that the trend on the drawing area forms the condition of layered display.
By implementing the method, the large-capacity curve data points can be quickly subjected to data query, and the curve data points obtained by query are analyzed to show abnormal points and normal trend points, so that invalid points can be efficiently screened out, the system operation efficiency and the curve display effect are greatly improved, the working efficiency of monitoring personnel is remarkably improved, and the system operation pressure is reduced.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (5)
1. A curve data query processing method for a domestic industrial control system is characterized by comprising the following steps:
1) the method comprises the following steps that a user inputs measuring point query condition information in a custom retrieval interface, and the query condition information comprises the following steps: inquiring information such as time interval, measuring point type, statistical type and the like, and adopting intervals.
2) Establishing a data retrieval table from massive industrial data by using the measuring point query condition information input in the step 1) through a word frequency-inverse text frequency TF-IDF algorithm, and quickly retrieving curve data related to the given query condition by using a multi-thread synchronous incremental retrieval method;
3) detecting abnormal data by using the curve data inquired in the step 2) through an Isolation Forest algorithm, calculating an influence factor of the abnormal data, and eliminating the abnormal data from the curve data inquired in the step 2) to obtain curve data under a normal working condition;
4) optimizing the curve data under the normal working condition obtained in the step 3), and then visually displaying, wherein the optimization principle is mainly to perform local feature display on a curve consisting of a large number of data points, and the local feature display is realized by setting a historical time interval and a sampling interval; and the layered visualization effect is realized by setting dynamic offset for the overlapping curves.
2. The curve data query processing method of the domestic industrial control system according to claim 1, wherein: in the step 1), a user-defined condition retrieval interface is provided, the user is supported to set retrieval time, a statistical data display mode is provided, the user can configure point picking, mean value, maximum value and minimum value modes for retrieval display, and the selection of the type of the measuring point is provided for display.
3. The curve data query processing method of the domestic industrial control system according to claim 1, wherein: in step 2), a data retrieval table is established through a TF-IDF mathematical function, so that the retrieval speed is improved in the actual data retrieval process, and the method specifically comprises the following steps:
TF-IDF(x)＝TF(x)*IDF(x)
wherein N represents the total number of texts in the corpus, N (x) represents the total number of texts in the corpus containing the word x, idf (x) represents the frequency of the word x in all texts, TF represents the frequency of each word in the texts, and TF (x) represents the frequency of the word x in the current text.
4. The curve data query processing method of the domestic industrial control system according to claim 1, wherein: in step 3), establishing an Isolation Forest algorithm function model, performing detection analysis on abnormal data, and calculating influence factors of the abnormal data, wherein the influence factors are as follows:
wherein E (h (y)) represents the average of the path lengths of the data y in the multiple binary trees, ψ represents the number of samples of the training samples of a single binary tree, and C (ψ) represents the average path length of the binary tree constructed with the ψ number of samples; the threshold value of the abnormal point can be artificially set, source (y) represents the influence factor of the data y, when the influence factor of the source (y) is greater than the threshold value, the abnormal point is classified, and when the influence factor of the source (y) is less than the threshold value, the abnormal point is regarded as a non-abnormal point.
5. The curve data query processing method of the domestic industrial control system according to claim 1, wherein: in the step 4), for curve data under normal working conditions, due to the huge data amount, data distortion and low curve resolution can be caused if the curve data are all displayed; therefore, in the step 4), a historical data statistical analysis function is introduced, and curve drawing is performed on the measuring point values to be displayed by setting the sampling interval time of the data points and the data display time period so as to adapt to the screen resolution; in addition, aiming at the numerical value superposition condition of different measuring points, step 4) introduces dynamic offset, and different measuring point values of numerical superposition are multiplied by different weights, so that the trend on the drawing area forms the condition of layered display.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011316070.XA CN112417141B (en) | 2020-11-22 | 2020-11-22 | Domestic industrial control system curve data query processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011316070.XA CN112417141B (en) | 2020-11-22 | 2020-11-22 | Domestic industrial control system curve data query processing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112417141A true CN112417141A (en) | 2021-02-26 |
| CN112417141B CN112417141B (en) | 2023-05-16 |
Family
ID=74777124
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011316070.XA Active CN112417141B (en) | 2020-11-22 | 2020-11-22 | Domestic industrial control system curve data query processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112417141B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113536050A (en) * | 2021-07-06 | 2021-10-22 | 贵州电网有限责任公司 | Distribution network monitoring system curve data query processing method |
| CN113568961A (en) * | 2021-09-26 | 2021-10-29 | 西安热工研究院有限公司 | Curve generation method and equipment for selecting measuring points through time marks and integrating alarm information |
| CN117792960A (en) * | 2024-02-23 | 2024-03-29 | 中国电子科技集团公司第三十研究所 | Historical flow statistics method and device based on domestic multi-core processor |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100332503A1 (en) * | 2009-06-30 | 2010-12-30 | Brad Buckley | System and Method for Using an Exemplar Document to Retrieve Relevant Documents from an Inverted Index of a Large Corpus |
| CN101982820A (en) * | 2010-11-22 | 2011-03-02 | 北京航空航天大学 | Curve display and inquiry method for large data quantity |
| CN103955192A (en) * | 2014-04-29 | 2014-07-30 | 安徽中科大国祯信息科技有限责任公司 | Curve report data sampling method for sewage disposal plant |
| CN103955508A (en) * | 2014-04-29 | 2014-07-30 | 安徽中科大国祯信息科技有限责任公司 | Method for layering, storing and inquiring data of curve chart of project control system |
| TW201727514A (en) * | 2015-10-14 | 2017-08-01 | 薩摩林根 沙庫那拿加 | Apparatus and method for displaying multi-format data in a 3D visualization space |
-
2020
- 2020-11-22 CN CN202011316070.XA patent/CN112417141B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100332503A1 (en) * | 2009-06-30 | 2010-12-30 | Brad Buckley | System and Method for Using an Exemplar Document to Retrieve Relevant Documents from an Inverted Index of a Large Corpus |
| CN101982820A (en) * | 2010-11-22 | 2011-03-02 | 北京航空航天大学 | Curve display and inquiry method for large data quantity |
| CN103955192A (en) * | 2014-04-29 | 2014-07-30 | 安徽中科大国祯信息科技有限责任公司 | Curve report data sampling method for sewage disposal plant |
| CN103955508A (en) * | 2014-04-29 | 2014-07-30 | 安徽中科大国祯信息科技有限责任公司 | Method for layering, storing and inquiring data of curve chart of project control system |
| TW201727514A (en) * | 2015-10-14 | 2017-08-01 | 薩摩林根 沙庫那拿加 | Apparatus and method for displaying multi-format data in a 3D visualization space |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113536050A (en) * | 2021-07-06 | 2021-10-22 | 贵州电网有限责任公司 | Distribution network monitoring system curve data query processing method |
| CN113536050B (en) * | 2021-07-06 | 2023-12-01 | 贵州电网有限责任公司 | Distribution network monitoring system curve data query processing method |
| CN113568961A (en) * | 2021-09-26 | 2021-10-29 | 西安热工研究院有限公司 | Curve generation method and equipment for selecting measuring points through time marks and integrating alarm information |
| CN117792960A (en) * | 2024-02-23 | 2024-03-29 | 中国电子科技集团公司第三十研究所 | Historical flow statistics method and device based on domestic multi-core processor |
| CN117792960B (en) * | 2024-02-23 | 2024-04-30 | 中国电子科技集团公司第三十研究所 | Historical flow statistics method and device based on domestic multi-core processor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112417141B (en) | 2023-05-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112417141A (en) | Domestic industrial control system curve data query processing method | |
| CN109711659B (en) | Yield improvement management system and method for industrial production | |
| CN108199795B (en) | A kind of monitoring method and device of equipment state | |
| CN106446181A (en) | Big data processing and remote monitoring system | |
| CN114997313B (en) | Abnormality detection method for ocean on-line monitoring data | |
| CN113420162A (en) | Equipment operation chain state monitoring method based on knowledge graph | |
| CN117194919A (en) | Production data analysis system | |
| CN116777114A (en) | Visual production management method for discrete workshops | |
| CN112015711A (en) | Model monitoring method and device and electronic equipment | |
| CN115983582A (en) | Data analysis method and energy consumption management system | |
| CN112434063B (en) | Method for processing monitoring data based on time sequence database | |
| CN115185938A (en) | Photoelectric detection data analysis method, device and storage medium | |
| CN112433909A (en) | Processing method of real-time monitoring data based on kafka | |
| CN110956340A (en) | Engineering test detection data management early warning decision method | |
| CN118536410B (en) | Big data driven modeling-based energy consumption optimization decision analysis method and system | |
| CN111178790B (en) | Electric power detecting system | |
| CN116910681B (en) | Electric power data analysis method and system based on Internet of things | |
| CN116307407B (en) | Enterprise data visualization processing system and method based on cloud computing | |
| CN117114911B (en) | Automatic statistical settlement method and system for financial statement | |
| KR101511991B1 (en) | Apparatus and method for processing of plant filed data | |
| CN116975041B (en) | AB experiment shunting and analyzing system | |
| CN118800346B (en) | Australian nut protein extraction process optimization method and system | |
| Li et al. | Research on Data Cleaning Method of Metal Material Corrosion Fatigue Test Data | |
| CN117421342A (en) | Abnormal number re-opening risk level determining method and device and electronic equipment | |
| CN115438060A (en) | Method and system for constructing component digital fault knowledge base |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |