CN111309561B - Method and device for monitoring state of big data system - Google Patents
Method and device for monitoring state of big data system Download PDFInfo
- Publication number
- CN111309561B CN111309561B CN202010121017.8A CN202010121017A CN111309561B CN 111309561 B CN111309561 B CN 111309561B CN 202010121017 A CN202010121017 A CN 202010121017A CN 111309561 B CN111309561 B CN 111309561B
- Authority
- CN
- China
- Prior art keywords
- data
- information
- network transmission
- time
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Abstract
The invention discloses a method and a device for monitoring the state of a big data system, which belongs to the technical field of big data systems, and the monitoring method comprises the following steps: step one: the system is accessed with big data and data information is collected; step two: splitting the collected data information according to the time point; step three: performing time point comparison according to the split data; step four: correcting the data of the compared information; step five: combining the corrected data, and then performing data delay elimination; step six: the data eliminating delay is output to the system state monitoring system, the data splitting is carried out on the collected data system state information, the split data is compared, after the data is corrected, the corrected data is substituted into the current time point to be synchronized, and then the corrected data is output to the system state monitoring system, so that the delay effect of system state monitoring under big data is effectively solved, and the loss caused by data updating errors is prevented.
Description
Technical Field
The invention relates to the technical field of big data systems, in particular to a method and a device for monitoring the state of a big data system.
Background
Big data, IT industry terminology, refers to a data set that cannot be captured, managed and processed with conventional software tools within a certain time frame, is a massive, high growth rate and diversified information asset that requires a new processing mode to have stronger decision making, insight discovery and process optimization capabilities.
In the existing big data-based system, high delay is usually existed for system state monitoring, so that data updating is wrong, and economic loss is caused when serious.
Disclosure of Invention
This section is intended to outline some aspects of embodiments of the invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section as well as in the description summary and in the title of the application, to avoid obscuring the purpose of this section, the description summary and the title of the invention, which should not be used to limit the scope of the invention.
The present invention has been made in view of the above-mentioned and/or existing problems with big data system status monitoring.
It is therefore an object of the present invention to provide a method and apparatus for big data system status monitoring that can reduce the delay caused by big data system status monitoring and reduce the occurrence of loss due to update errors.
In order to solve the technical problems, according to one aspect of the present invention, the following technical solutions are provided:
a method for big data system status monitoring, the monitoring method comprising:
step one: the system is accessed with big data and data information is collected;
step two: splitting the collected data information according to time points, wherein the time point splitting is specifically to time mark the collected data information; according to the setting information, carrying out average splitting on the marked time, wherein the setting information is mainly a time splitting point;
step three: performing time point comparison according to the split data, wherein the time point comparison is to perform coincidence ratio comparison on the split time information, and detecting whether the same data exist in different time points or not;
step four: carrying out data correction on the compared information, wherein the data correction is specifically to change the same data generated in the step three;
step five: merging corrected data, and then carrying out data delay elimination, wherein the data merging is carried out after the data in the split time information are corrected, overlapping merging is carried out, corrected data information is generated, and the data delay elimination in the step five is that the corrected data information is synchronous with the current time;
step six: and outputting the data with the delay eliminated to a system state monitoring system.
As a preferred embodiment of the method for monitoring the status of a big data system according to the present invention, the method comprises: the specific method for accessing the data in the first step is to connect the system with a network through a network transmission device, and the collected data information in the first step is specifically system state information directly obtained after networking.
As a preferred embodiment of the apparatus for monitoring the status of a big data system according to the present invention, the apparatus comprises: the monitoring device comprises a network transmission main system, a network transmission subsystem, a time comparison system, a data correction system, a data delay elimination system and a system state monitoring system, wherein the network transmission main system is connected with the network transmission subsystem, the network transmission subsystem comprises a network transmission subsystem a and a network transmission subsystem b, the output of the network transmission subsystem a and the output of the network transmission subsystem b are connected with the time comparison system, the output of the time comparison system is connected with the data correction system, the output of the data correction system is connected with the data delay elimination system, and the output of the data delay elimination system is connected with the system state monitoring system.
Compared with the prior art: in the existing system based on big data, high delay is usually existed for system state monitoring, so that data updating is wrong, economic loss is caused when serious, in the method, data splitting is carried out on collected data system state information, split data is compared, whether the same data in different time exists or not is detected, whether correction is needed or not is selected according to data detection results, after the correction is carried out on the data, the corrected data are substituted into a current time point for synchronization, and then the corrected data are output to a system state monitoring system, the delay effect of system state monitoring under the big data is effectively solved, and loss caused by data updating errors is prevented.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following detailed description of the embodiments of the present invention will be given with reference to the accompanying drawings, which are to be understood as merely some embodiments of the present invention, and from which other drawings can be obtained by those skilled in the art without inventive faculty. Wherein:
FIG. 1 is a schematic diagram of a method and apparatus for big data system status monitoring according to the present invention.
Detailed Description
In order that the above objects, features and advantages of the invention will be readily understood, a more particular description of the invention will be rendered by reference to the appended drawings.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways other than those described herein, and persons skilled in the art will readily appreciate that the present invention is not limited to the specific embodiments disclosed below.
Next, the present invention will be described in detail with reference to the drawings, wherein the sectional view of the device structure is not partially enlarged to general scale for the convenience of description, and the drawings are only examples, which should not limit the scope of the present invention. In addition, the three-dimensional dimensions of length, width and depth should be included in actual fabrication.
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
The invention provides a method for monitoring the state of a big data system, which comprises the following steps:
step one: the system is accessed with big data and data information is collected;
step two: splitting the collected data information according to the time point;
step three: performing time point comparison according to the split data;
step four: correcting the data of the compared information;
step five: combining the corrected data, and then performing data delay elimination;
step six: and outputting the data with the delay eliminated to a system state monitoring system.
The specific method for accessing the data in the first step is to connect the system with a network through a network transmission device, and the collected data information in the first step is specifically system state information directly obtained after networking.
The specific steps of splitting the time points in the second step are as follows:
step one: the collected data information is marked with time;
step two: and carrying out average splitting on the marked time according to the setting information, wherein the setting information is mainly a time splitting point.
And step three, comparing the time points, namely, comparing the split time information in a coincidence degree, and detecting whether the same data exist in different time points or not.
The data correction in the fourth step is specifically to change the same data generated in the third step.
And the data in the fifth step is merged into overlapped merging after the data in the split time information is corrected, corrected data information is generated, and the data delay elimination in the fifth step is specifically that the corrected data information is synchronized with the current time.
Referring to fig. 1, the monitoring device includes a network transmission main system, a network transmission subsystem, a time comparison system, a data correction system, a data delay elimination system and a system state monitoring system, wherein the network transmission main system is connected with the network transmission subsystem, the network transmission subsystem includes a network transmission subsystem a and a network transmission subsystem b, the output of the network transmission subsystem a and the output of the network transmission subsystem b are connected with the time comparison system, the output of the time comparison system is connected with the data correction system, the output of the data correction system is connected with the data delay elimination system, the output of the data delay elimination system is connected with the system state monitoring system specifically, the network transmission main system is used for collecting data information in the first step, the network transmission subsystem is used for splitting the collected data information according to time points in the second step, the time comparison system is used for carrying out time point comparison according to the split data in the third step, the data correction system is used for carrying out data correction on the compared information in the fourth step, the data delay elimination system is used for merging corrected data in the fifth step, and then the data delay elimination system is connected with the state monitoring system, and the state monitoring system is output to the state monitoring system.
Although the invention has been described hereinabove with reference to embodiments, various modifications thereof may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In particular, the features of the disclosed embodiments may be combined with each other in any manner as long as there is no structural conflict, and the exhaustive description of these combinations is not given in this specification merely for the sake of omitting the descriptions and saving resources. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
Claims (3)
1. A method for big data system status monitoring, characterized by: the monitoring method comprises the following steps:
step one: the system is accessed with big data and data information is collected;
step two: splitting the collected data information according to time points, wherein the time point splitting is specifically to time mark the collected data information; according to the setting information, carrying out average splitting on the marked time, wherein the setting information is mainly a time splitting point;
step three: performing time point comparison according to the split data, wherein the time point comparison is to perform coincidence ratio comparison on the split time information, and detecting whether the same data exist in different time points or not;
step four: carrying out data correction on the compared information, wherein the data correction is specifically to change the same data generated in the step three;
step five: merging corrected data, and then carrying out data delay elimination, wherein the data merging is carried out after the data in the split time information are corrected, overlapping merging is carried out, corrected data information is generated, and the data delay elimination in the step five is that the corrected data information is synchronous with the current time;
step six: and outputting the data with the delay eliminated to a system state monitoring system.
2. A method for big data system status monitoring according to claim 1, characterized by: the specific method for accessing the data in the first step is to connect the system with a network through a network transmission device, and the collected data information in the first step is specifically system state information directly obtained after networking.
3. A monitoring device for a method according to any one of claims 1-2, characterized in that: the monitoring device comprises a network transmission main system, a network transmission subsystem, a time comparison system, a data correction system, a data delay elimination system and a system state monitoring system, wherein the network transmission main system is connected with the network transmission subsystem, the network transmission subsystem comprises a network transmission subsystem a and a network transmission subsystem b, the output of the network transmission subsystem a and the output of the network transmission subsystem b are connected with the time comparison system, the output of the time comparison system is connected with the data correction system, the output of the data correction system is connected with the data delay elimination system, and the output of the data delay elimination system is connected with the system state monitoring system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010121017.8A CN111309561B (en) | 2020-02-26 | 2020-02-26 | Method and device for monitoring state of big data system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010121017.8A CN111309561B (en) | 2020-02-26 | 2020-02-26 | Method and device for monitoring state of big data system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111309561A CN111309561A (en) | 2020-06-19 |
| CN111309561B true CN111309561B (en) | 2023-04-28 |
Family
ID=71153108
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010121017.8A Active CN111309561B (en) | 2020-02-26 | 2020-02-26 | Method and device for monitoring state of big data system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111309561B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112688925A (en) * | 2020-12-17 | 2021-04-20 | 崔强 | Enterprise storage system state monitoring method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH10214221A (en) * | 1997-01-31 | 1998-08-11 | Hitachi Ltd | Controller and memory system |
| JP2010190688A (en) * | 2009-02-17 | 2010-09-02 | Beckman Coulter Inc | Monitor terminal device and management system |
| WO2014146554A1 (en) * | 2013-03-22 | 2014-09-25 | 南京南瑞继保电气有限公司 | Method and system for packet synchronization on process layer network of intelligent substation |
| CN104269937A (en) * | 2014-10-23 | 2015-01-07 | 国家电网公司 | Operation monitoring system for distributed photovoltaic power station |
| CN106600447A (en) * | 2015-10-14 | 2017-04-26 | 山东鲁能智能技术有限公司 | Transformer station inspection robot centralized monitoring system big data cloud analysis method |
| CN108365905A (en) * | 2018-01-29 | 2018-08-03 | 中国科学院国家授时中心 | A kind of national standard time restoration methods based on satellite common vision data real-time exchange |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2003333584A (en) * | 2002-05-16 | 2003-11-21 | Fujitsu Ltd | Supervisory system |
| US10565048B2 (en) * | 2017-12-01 | 2020-02-18 | Arista Networks, Inc. | Logic buffer for hitless single event upset handling |
-
2020
- 2020-02-26 CN CN202010121017.8A patent/CN111309561B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH10214221A (en) * | 1997-01-31 | 1998-08-11 | Hitachi Ltd | Controller and memory system |
| JP2010190688A (en) * | 2009-02-17 | 2010-09-02 | Beckman Coulter Inc | Monitor terminal device and management system |
| WO2014146554A1 (en) * | 2013-03-22 | 2014-09-25 | 南京南瑞继保电气有限公司 | Method and system for packet synchronization on process layer network of intelligent substation |
| CN104269937A (en) * | 2014-10-23 | 2015-01-07 | 国家电网公司 | Operation monitoring system for distributed photovoltaic power station |
| CN106600447A (en) * | 2015-10-14 | 2017-04-26 | 山东鲁能智能技术有限公司 | Transformer station inspection robot centralized monitoring system big data cloud analysis method |
| CN108365905A (en) * | 2018-01-29 | 2018-08-03 | 中国科学院国家授时中心 | A kind of national standard time restoration methods based on satellite common vision data real-time exchange |
Non-Patent Citations (2)
| Title |
|---|
| 何小利 ; 宋钰 ; .基于分布式技术的机房空调联网监控优化.计算机仿真.2016,(03),全文. * |
| 王德兵 ; .工业数字视频监控系统在石化行业安全领域的应用.中国工程咨询.2012,(09),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111309561A (en) | 2020-06-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111309561B (en) | Method and device for monitoring state of big data system | |
| CN107040395A (en) | A kind of processing method of warning information, device and system | |
| CN113723452A (en) | Large-scale anomaly detection system based on KPI clustering | |
| CN114020581A (en) | Alarm correlation method based on topological optimization FP-Growth algorithm | |
| CN111858677A (en) | Local cache data access method, device, equipment and storage medium | |
| CN112988892B (en) | Distributed system hot spot data management method | |
| CN106407233A (en) | A data processing method and apparatus | |
| CN105007200B (en) | The analysis method and system of network packet | |
| CN105607606A (en) | Data acquisition device and data acquisition method based on double-mainboard framework | |
| CN112395360B (en) | Data synchronization method, device, apparatus and medium based on non-relational database | |
| EP3793171B1 (en) | Message processing method, apparatus, and system | |
| CN104735097A (en) | Information collecting method and system | |
| CN110166295B (en) | Method for judging whether network topology supports Byzantine fault tolerance or not | |
| CN114138330A (en) | Code clone detection optimization method and device based on knowledge graph and electronic equipment | |
| CN110569231B (en) | Data migration method, device, equipment and medium | |
| CN111917826A (en) | PBFT consensus algorithm based on block chain intellectual property protection | |
| CN113961547A (en) | Method and system for analyzing quality of operator configuration data | |
| CN111698321A (en) | Internet of things equipment data synchronization method and device and control center | |
| CN112231368A (en) | Unary linear regression analysis method based on steel production big data | |
| KR102281765B1 (en) | Method and apparatus for adaptive synchronization of blockchain | |
| CN111026689A (en) | Redundancy voting consistency method for multi-redundancy flight control computer | |
| CN114894140B (en) | Method, device, equipment and medium for measuring interval thickness of three-dimensional model | |
| CN111935153B (en) | CAN bus-based target message extraction method and device and storage medium | |
| CN113485973B (en) | Data synchronization method and device | |
| CN117632773B (en) | Data interaction anomaly detection method and detection port based on computer terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |