CN111309561A - Method and device for monitoring state of big data system - Google Patents
Method and device for monitoring state of big data system Download PDFInfo
- Publication number
- CN111309561A CN111309561A CN202010121017.8A CN202010121017A CN111309561A CN 111309561 A CN111309561 A CN 111309561A CN 202010121017 A CN202010121017 A CN 202010121017A CN 111309561 A CN111309561 A CN 111309561A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- network transmission
- time
- big data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Abstract
The invention discloses a method and a device for monitoring the state of a big data system, belonging to the technical field of big data systems, wherein the monitoring method comprises the following steps: the method comprises the following steps: the system accesses big data and collects data information; step two: splitting the collected data information according to time points; step three: comparing time points according to the split data; step four: carrying out data correction on the compared information; step five: merging the corrected data, and then performing data delay elimination; step six: the data that the elimination postpones can be exported system status monitoring system, through carrying out the data split to the data system status information who collects, compare the data of split again, after the data is corrected, substitute the data of correcting at present time point and synchronize, export again to system status monitoring system in, the effectual delay effect of system status monitoring under the big data of having solved prevents that the data update mistake from causing the loss and taking place.
Description
Technical Field
The invention relates to the technical field of big data systems, in particular to a method and a device for monitoring the state of a big data system.
Background
Big data, an IT industry term, refers to a data set that cannot be captured, managed, and processed with a conventional software tool within a certain time range, and is a massive, high-growth-rate, and diversified information asset that needs a new processing mode to have stronger decision-making power, insight discovery power, and process optimization capability.
In the existing big data-based system, high delay exists for system state monitoring, which causes errors in data updating and economic loss in severe cases.
Disclosure of Invention
This section is for the purpose of summarizing some aspects of embodiments of the invention and to briefly introduce some preferred embodiments. In this section, as well as in the abstract and the title of the invention of this application, simplifications or omissions may be made to avoid obscuring the purpose of the section, the abstract and the title, and such simplifications or omissions are not intended to limit the scope of the invention.
The present invention has been made in view of the above and/or other problems with existing big data system status monitoring.
Therefore, an object of the present invention is to provide a method and an apparatus for big data system status monitoring, which can reduce the delay caused by big data system status monitoring and reduce the loss caused by update error.
To solve the above technical problem, according to an aspect of the present invention, the present invention provides the following technical solutions:
a method for monitoring the state of a big data system comprises the following steps:
the method comprises the following steps: the system accesses big data and collects data information;
step two: splitting the collected data information according to time points;
step three: comparing time points according to the split data;
step four: carrying out data correction on the compared information;
step five: merging the corrected data, and then performing data delay elimination;
step six: and outputting the data for eliminating the delay to a system state monitoring system.
As a preferred aspect of the method for monitoring the status of the big data system according to the present invention, wherein: the specific method of data access in the first step is to connect the system with the network through a network transmission device, and the collected data information in the first step is specifically system state information directly obtained after networking.
As a preferred aspect of the method for monitoring the status of the big data system according to the present invention, wherein: the time point splitting in the second step comprises the following specific steps:
the method comprises the following steps: carrying out time marking on the collected data information;
step two: and averaging and splitting the marked time according to the setting information, wherein the setting information is mainly a time splitting point.
As a preferred aspect of the method for monitoring the status of the big data system according to the present invention, wherein: and comparing the time points in the third step, namely, comparing the contact ratio of the split time information, and detecting whether the same data exists in different time points.
As a preferred aspect of the method for monitoring the status of the big data system according to the present invention, wherein: the data correction in the fourth step is to change the same data generated in the third step.
As a preferred aspect of the method for monitoring the status of the big data system according to the present invention, wherein: and the data merging in the step five is to perform overlapping merging after correcting the data in the split time information to generate corrected data information, and the data delay elimination in the step five is specifically to perform synchronization between the corrected data information and the current time.
As a preferable aspect of the apparatus for monitoring status of a big data system according to the present invention, wherein: the monitoring device comprises a network transmission main system, a network transmission subsystem, a time comparison system, a data correction system, a data delay elimination system and a system state monitoring system, wherein the network transmission main system is connected with the network transmission subsystem, the network transmission subsystem comprises a network transmission subsystem a and a network transmission subsystem b, the outputs of the network transmission subsystem a and the network transmission subsystem b are connected with the time comparison system, the output of the practice comparison system is connected with the data correction system, the output of the data correction system is connected with the data delay elimination system, and the output of the data delay elimination system is connected with the system state monitoring system.
Compared with the prior art: in the existing big data-based system, aiming at system state monitoring, higher delay often exists, which causes errors in data updating and economic loss when the data updating is serious.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail with reference to the accompanying drawings and detailed embodiments, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise. Wherein:
fig. 1 is a schematic structural diagram of a method and an apparatus for monitoring a status of a big data system according to the present invention.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways than those specifically described herein, and it will be apparent to those of ordinary skill in the art that the present invention may be practiced without departing from the spirit and scope of the present invention, and therefore the present invention is not limited to the specific embodiments disclosed below.
Next, the present invention will be described in detail with reference to the drawings, wherein for convenience of illustration, the cross-sectional view of the device structure is not enlarged partially according to the general scale, and the drawings are only examples, which should not limit the scope of the present invention. In addition, the three-dimensional dimensions of length, width and depth should be included in the actual fabrication.
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The invention provides a method for monitoring the state of a big data system, which comprises the following steps:
the method comprises the following steps: the system accesses big data and collects data information;
step two: splitting the collected data information according to time points;
step three: comparing time points according to the split data;
step four: carrying out data correction on the compared information;
step five: merging the corrected data, and then performing data delay elimination;
step six: and outputting the data for eliminating the delay to a system state monitoring system.
The data access method in the first step is to connect the system with the network through a network transmission device, and the collected data information in the first step is system state information directly obtained after networking.
The time point splitting in the second step comprises the following specific steps:
the method comprises the following steps: carrying out time marking on the collected data information;
step two: and averaging and splitting the marked time according to the setting information, wherein the setting information is mainly a time splitting point.
And comparing the time points in the third step, namely, comparing the contact ratio of the split time information, and detecting whether the same data exists in different time points.
Wherein, the data correction in the fourth step is to change the same data generated in the third step.
And the data merging in the step five is to perform superposition merging after correcting the data in the split time information to generate corrected data information, and the data delay elimination in the step five is specifically to perform synchronization between the corrected data information and the current time.
A device for monitoring the state of a big data system, please refer to FIG. 1, which comprises a network transmission main system, a network transmission subsystem, a time comparison system, a data correction system, a data delay elimination system and a system state monitoring system, wherein the network transmission main system is connected with the network transmission subsystem, the network transmission subsystem comprises a network transmission subsystem a and a network transmission subsystem b, the network transmission subsystem a and the network transmission subsystem b output and connect with the time comparison system, the practice comparison system output and connect with the data correction system, the data correction system output and connect with the data delay elimination system, the data delay elimination system output and connect with the system state monitoring system specifically, the network transmission main system is used for collecting data information in the first step, the network transmission subsystem is used for splitting the collected data information according to time points in the second step, the time comparison system is used for performing time point comparison according to the split data in the step three, the data correction system is used for performing data correction on the compared information in the step four, the data delay elimination system is used for merging the corrected data in the step five and then performing data delay elimination, and the system state monitoring system is an output end and outputs the data with the delay eliminated to the system state monitoring system.
While the invention has been described above with reference to an embodiment, various modifications may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In particular, the various features of the disclosed embodiments of the invention may be used in any combination, provided that no structural conflict exists, and the combinations are not exhaustively described in this specification merely for the sake of brevity and resource conservation. Therefore, it is intended that the invention not be limited to the particular embodiments disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
Claims (7)
1. A method for big data system condition monitoring, characterized by: the monitoring method comprises the following steps:
the method comprises the following steps: the system accesses big data and collects data information;
step two: splitting the collected data information according to time points;
step three: comparing time points according to the split data;
step four: carrying out data correction on the compared information;
step five: merging the corrected data, and then performing data delay elimination;
step six: and outputting the data for eliminating the delay to a system state monitoring system.
2. The method for big data system condition monitoring according to claim 1, wherein: the specific method of data access in the first step is to connect the system with the network through a network transmission device, and the collected data information in the first step is specifically system state information directly obtained after networking.
3. The method for big data system condition monitoring according to claim 1, wherein: the time point splitting in the second step comprises the following specific steps:
the method comprises the following steps: carrying out time marking on the collected data information;
step two: and averaging and splitting the marked time according to the setting information, wherein the setting information is mainly a time splitting point.
4. The method for big data system condition monitoring according to claim 1, wherein: and comparing the time points in the third step, namely, comparing the contact ratio of the split time information, and detecting whether the same data exists in different time points.
5. The method for big data system condition monitoring according to claim 1, wherein: the data correction in the fourth step is to change the same data generated in the third step.
6. The method for big data system condition monitoring according to claim 1, wherein: and the data merging in the step five is to perform overlapping merging after correcting the data in the split time information to generate corrected data information, and the data delay elimination in the step five is specifically to perform synchronization between the corrected data information and the current time.
7. A condition monitoring device for big data system according to any of claims 1-6, characterized in that: the monitoring device comprises a network transmission main system, a network transmission subsystem, a time comparison system, a data correction system, a data delay elimination system and a system state monitoring system, wherein the network transmission main system is connected with the network transmission subsystem, the network transmission subsystem comprises a network transmission subsystem a and a network transmission subsystem b, the outputs of the network transmission subsystem a and the network transmission subsystem b are connected with the time comparison system, the output of the practice comparison system is connected with the data correction system, the output of the data correction system is connected with the data delay elimination system, and the output of the data delay elimination system is connected with the system state monitoring system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010121017.8A CN111309561B (en) | 2020-02-26 | 2020-02-26 | Method and device for monitoring state of big data system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010121017.8A CN111309561B (en) | 2020-02-26 | 2020-02-26 | Method and device for monitoring state of big data system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111309561A true CN111309561A (en) | 2020-06-19 |
| CN111309561B CN111309561B (en) | 2023-04-28 |
Family
ID=71153108
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010121017.8A Active CN111309561B (en) | 2020-02-26 | 2020-02-26 | Method and device for monitoring state of big data system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111309561B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112688925A (en) * | 2020-12-17 | 2021-04-20 | 崔强 | Enterprise storage system state monitoring method |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH10214221A (en) * | 1997-01-31 | 1998-08-11 | Hitachi Ltd | Controller and memory system |
| US20030214400A1 (en) * | 2002-05-16 | 2003-11-20 | Fujitsu Limited | Monitoring system realizing high performance with reduced processing loads |
| JP2010190688A (en) * | 2009-02-17 | 2010-09-02 | Beckman Coulter Inc | Monitor terminal device and management system |
| WO2014146554A1 (en) * | 2013-03-22 | 2014-09-25 | 南京南瑞继保电气有限公司 | Method and system for packet synchronization on process layer network of intelligent substation |
| CN104269937A (en) * | 2014-10-23 | 2015-01-07 | 国家电网公司 | Operation monitoring system for distributed photovoltaic power station |
| CN106600447A (en) * | 2015-10-14 | 2017-04-26 | 山东鲁能智能技术有限公司 | Transformer station inspection robot centralized monitoring system big data cloud analysis method |
| CN108365905A (en) * | 2018-01-29 | 2018-08-03 | 中国科学院国家授时中心 | A kind of national standard time restoration methods based on satellite common vision data real-time exchange |
| US20190171508A1 (en) * | 2017-12-01 | 2019-06-06 | Arista Networks, Inc. | Logic buffer for hitless single event upset handling |
-
2020
- 2020-02-26 CN CN202010121017.8A patent/CN111309561B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH10214221A (en) * | 1997-01-31 | 1998-08-11 | Hitachi Ltd | Controller and memory system |
| US20030214400A1 (en) * | 2002-05-16 | 2003-11-20 | Fujitsu Limited | Monitoring system realizing high performance with reduced processing loads |
| JP2010190688A (en) * | 2009-02-17 | 2010-09-02 | Beckman Coulter Inc | Monitor terminal device and management system |
| WO2014146554A1 (en) * | 2013-03-22 | 2014-09-25 | 南京南瑞继保电气有限公司 | Method and system for packet synchronization on process layer network of intelligent substation |
| CN104269937A (en) * | 2014-10-23 | 2015-01-07 | 国家电网公司 | Operation monitoring system for distributed photovoltaic power station |
| CN106600447A (en) * | 2015-10-14 | 2017-04-26 | 山东鲁能智能技术有限公司 | Transformer station inspection robot centralized monitoring system big data cloud analysis method |
| US20190171508A1 (en) * | 2017-12-01 | 2019-06-06 | Arista Networks, Inc. | Logic buffer for hitless single event upset handling |
| CN108365905A (en) * | 2018-01-29 | 2018-08-03 | 中国科学院国家授时中心 | A kind of national standard time restoration methods based on satellite common vision data real-time exchange |
Non-Patent Citations (2)
| Title |
|---|
| 何小利;宋钰;: "基于分布式技术的机房空调联网监控优化" * |
| 王德兵;: "工业数字视频监控系统在石化行业安全领域的应用" * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112688925A (en) * | 2020-12-17 | 2021-04-20 | 崔强 | Enterprise storage system state monitoring method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111309561B (en) | 2023-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103713981A (en) | Database server performance detection and early warning method | |
| CN106656712A (en) | Bus abnormality processing method and robot controller | |
| CN111309561A (en) | Method and device for monitoring state of big data system | |
| CN106875084A (en) | Patrol and examine later stage task creation method and system | |
| CN114429256A (en) | Data monitoring method and device, electronic equipment and storage medium | |
| CN112988892B (en) | Distributed system hot spot data management method | |
| CN105306262B (en) | Anomaly detection method based on power system protocol | |
| US10509712B2 (en) | Methods and systems to determine baseline event-type distributions of event sources and detect changes in behavior of event sources | |
| CN112436962B (en) | Block chain consensus network dynamic expansion method, electronic device, system and medium | |
| EP2509265A1 (en) | Access protection device for an automation network | |
| CN112532467A (en) | Method, device and system for realizing fault detection | |
| US11940890B2 (en) | Timing index anomaly detection method, device and apparatus | |
| EP4044061A1 (en) | Speed segmentation method and apparatus, and electronic device and storage medium | |
| JP2019169877A (en) | Monitoring system, monitoring method, and computer program | |
| CN112511337B (en) | Block chain consensus network self-recovery method, electronic device, system and storage medium | |
| CN110166295B (en) | Method for judging whether network topology supports Byzantine fault tolerance or not | |
| CN110244563B (en) | Neural network internal model controller model mismatch identification and online updating method | |
| CN110908823A (en) | Operation maintenance method and device for big data cluster | |
| JP2008217280A (en) | Manufacturing method of product, and process management program | |
| CN114894140B (en) | Method, device, equipment and medium for measuring interval thickness of three-dimensional model | |
| CN111917826A (en) | PBFT consensus algorithm based on block chain intellectual property protection | |
| CN112532495A (en) | Vehicle-mounted CAN bus delay optimization method | |
| CN116488724B (en) | Optical fiber communication test method, medium and system using same | |
| CN107562553B (en) | Data center management method and equipment | |
| EP4332776A1 (en) | Network fault analysis method and apparatus, and device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |