CN109684159A - Method for monitoring state, device, equipment and the storage medium of distributed information system - Google Patents
Method for monitoring state, device, equipment and the storage medium of distributed information system Download PDFInfo
- Publication number
- CN109684159A CN109684159A CN201811049716.5A CN201811049716A CN109684159A CN 109684159 A CN109684159 A CN 109684159A CN 201811049716 A CN201811049716 A CN 201811049716A CN 109684159 A CN109684159 A CN 109684159A
- Authority
- CN
- China
- Prior art keywords
- parameter
- data
- state parameter
- monitoring
- information system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 119
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012546 transfer Methods 0.000 claims abstract description 57
- 230000005540 biological transmission Effects 0.000 claims abstract description 34
- 230000008569 process Effects 0.000 claims abstract description 16
- 230000004044 response Effects 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims description 28
- 238000004458 analytical method Methods 0.000 claims description 22
- 238000012806 monitoring device Methods 0.000 claims description 11
- 241000208340 Araliaceae Species 0.000 claims description 3
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 claims description 3
- 235000003140 Panax quinquefolius Nutrition 0.000 claims description 3
- 235000008434 ginseng Nutrition 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 9
- 230000008859 change Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 5
- 238000007726 management method Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
Abstract
The present invention discloses method for monitoring state, device, equipment and the storage medium of a kind of distributed information system, this method comprises: monitoring host computer is instructed in response to the condition monitoring received, preset data acquisition script is called to press the current state parameter of default dimension acquisition distributed information system；Data transfer components, which are stated, reads current state parameter in monitoring host computer, carry out initialization process to current state parameter and obtain init state parameter；In the data acquisition instruction for receiving the transmission of data search component, record data search component sends the sending time of data acquisition instruction；Then parameter is carried out to sending time to redefine, and when completion parameter redefines, init state parameter is sent to data search component to save, by the acquisition for carrying out state parameter by default dimension thus according to preset data acquisition script, to realize the various dimensions monitoring to distributed information system operating status, the information security and structure safety of distributed information system ensure that.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of method for monitoring state of distributed information system, dress
It sets, equipment and storage medium.
Background technique
Kafka is that a kind of distributed post of high-throughput subscribes to message system, it can handle the net of consumer's scale
Everything flow data in standing.The prior art to this distributed information system of Kafka carry out operating status monitoring when,
KafkaOffsetMonitor (a kind of monitoring tools for version of the increasing income) Lai Shixian provided often by web-based management platform,
But the defect of this kind of monitoring tools, which is it only, has the function of data monitoring and user can not be according to self-demand to monitoring parameter
Carry out it is customized, monitoring dimension it is single, additionally due to it is with a Java archive file packet (Java Archive, JAR)
Form operation, and then cause it that can not be suitable for the monitoring demand of extensive Kafka cluster.Therefore, how to distributed message
The operating status of system carries out the monitoring of various dimensions, becomes a urgent problem to be solved.
Above content is only used to facilitate the understanding of the technical scheme, and is not represented and is recognized that above content is existing skill
Art.
Summary of the invention
The main purpose of the present invention is to provide a kind of method for monitoring state of distributed information system, device, equipment
And storage medium, it is intended to which solving the prior art effectively can not carry out various dimensions monitoring to distributed information system, monitor dimension
Single technical problem.
To achieve the above object, the present invention provides a kind of method for monitoring state of distributed information system, the methods
The following steps are included:
Monitoring host computer is instructed in response to the condition monitoring received, and preset data acquisition script is called to press default dimension acquisition
The current state parameter of distributed information system；
Data transfer components read the current state parameter from the monitoring host computer, to the current state parameter into
Row initialization process obtains init state parameter；
The data transfer components record the data in the data acquisition instruction for receiving the transmission of data search component
Searching component sends the sending time of the data acquisition instruction；
The data transfer components carry out parameter to the sending time and redefine, and redefine completing the parameter
When, the init state parameter is sent to the data search component and is saved.
Preferably, the preset data acquisition script includes: data decryptor script and format analysis processing script；The monitoring master
Machine is instructed in response to the condition monitoring received, is called preset data acquisition script to press default dimension and is acquired distributed information system
Current state parameter the step of, comprising:
Monitoring host computer is instructed in response to the condition monitoring received, and the data decryptor script is called to acquire by default dimension
The reset condition parameter of distributed information system；
The monitoring host computer calls the format analysis processing script to be formatted processing to the reset condition parameter, obtains
Current state parameter.
Preferably, the monitoring host computer calls the format analysis processing script to be formatted place to the reset condition parameter
The step of reason, acquisition current state parameter, comprising:
The monitoring host computer obtains corresponding acquisition time and the distribution when acquiring the reset condition parameter and disappears
The corresponding system banner of breath system；
The system banner and the acquisition time are added to institute by the format analysis processing script by the monitoring host computer
It states in reset condition parameter, obtains the current state parameter.
Preferably, the data transfer components read the current state parameter from the monitoring host computer, work as to described
Preceding state parameter carries out the step of initialization process obtains init state parameter, comprising:
Data transfer components monitor script by preset parameter and read the current state parameter from the monitoring host computer；
The data transfer components are formatted the acquisition time in the current state parameter, obtain initial
Change state parameter.
Preferably, the data transfer components redefine sending time progress parameter, and complete the parameter
When redefining, the init state parameter is sent to the step of data search component is saved, comprising:
When the data transfer components call preset data filter inserts to extract target from the init state parameter
Between parameter, the object time parameter be the formatted time parameter of the acquisition time；
The data transfer components according to the target date field for including in the object time parameter to the transmission when
Between carry out date field replacement, with realize the parameter of the sending time is redefined；
The init state parameter is sent to described by the data transfer components when the completion parameter redefines
Data search component is saved.
Preferably, the data transfer components send out the init state parameter when the completion parameter redefines
The step of being saved, is sent to the data search component, comprising:
The data transfer components are when the completion parameter redefines, in the corresponding database of the data search component
In search whether exist include the target date field target index information；
The init state parameter is sent to by the data transfer components when finding the target index information
The data search component is believed so that the init state parameter is saved to the target to index by the data search component
Cease corresponding storage region.
Preferably, the data transfer components are corresponding in the data search component when the completion parameter redefines
Database in search whether exist include the target date field target index the step of after, the method is also wrapped
It includes:
The data transfer components send the init state parameter when not finding the target index information
To the data search component；
The data search component establishes institute according to the target date field for including in the init state parameter
State the index information between target date field and the init state parameter.
In addition, to achieve the above object, the present invention also proposes a kind of state monitoring device of distributed information system, described
Device includes: monitoring module, data transmission module and data search module；
Wherein, the monitoring module, for calling preset data to acquire script when receiving condition monitoring instruction
By the current state parameter of default dimension acquisition distributed information system；
The data transmission module carries out the current state parameter initial for reading the current state parameter
Change handles to obtain init state parameter；
The data transmission module, for recording institute in the data acquisition instruction for receiving the transmission of data search module
State the sending time that data search module sends the data acquisition instruction；
The data transmission module is redefined for carrying out parameter to the sending time, and completes the parameter weight
When definition, the init state parameter is sent to the data search module and is saved.
In addition, to achieve the above object, the present invention also proposes a kind of status monitor device of distributed information system, described
Equipment includes: memory, processor and is stored in the distributed message that can be run on the memory and on the processor
The State Monitoring Program of system, the State Monitoring Program of the distributed information system are arranged for carrying out distribution as described above
The step of method for monitoring state of formula message system.
In addition, to achieve the above object, the present invention also proposes a kind of storage medium, stores and be distributed on the storage medium
It is realized such as when the State Monitoring Program of the State Monitoring Program of formula message system, the distributed information system is executed by processor
The step of method for monitoring state of distributed information system described above.
Monitoring host computer of the present invention is instructed in response to the condition monitoring received, calls preset data acquisition script by default dimension
The current state parameter of degree acquisition distributed information system；Data transfer components, which are stated, reads current state parameter in monitoring host computer,
Initialization process is carried out to current state parameter and obtains init state parameter；In the data for receiving the transmission of data search component
When acquisition instruction, record data search component sends the sending time of data acquisition instruction；Then parameter is carried out to sending time
It redefines, and when completion parameter redefines, init state parameter is sent to data search component and is saved, due to being
Script is acquired according to preset data and carries out the acquisition of state parameter by default dimension, to realize to distributed information system
The various dimensions of operating status monitor, and ensure that the information security and structure safety of distributed information system.
Detailed description of the invention
Fig. 1 is the status monitor device of the distributed information system for the hardware running environment that the embodiment of the present invention is related to
Structural schematic diagram；
Fig. 2 is the flow diagram of the method for monitoring state first embodiment of distributed information system of the present invention；
Fig. 3 is the flow diagram of the method for monitoring state second embodiment of distributed information system of the present invention；
Fig. 4 is the flow diagram of the method for monitoring state 3rd embodiment of distributed information system of the present invention；
Fig. 5 is the flow diagram of the method for monitoring state fourth embodiment of distributed information system of the present invention；
Fig. 6 is the structural block diagram of the state monitoring device first embodiment of distributed information system of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
It should be appreciated that described herein, specific examples are only used to explain the present invention, is not intended to limit the present invention.
Referring to Fig.1, Fig. 1 is the state of the distributed information system for the hardware running environment that the embodiment of the present invention is related to
Monitoring device structural schematic diagram.
As shown in Figure 1, the status monitor device of the distributed information system may include: processor 1001, such as center
Processor (Central Processing Unit, CPU), communication bus 1002, user interface 1003, network interface 1004 are deposited
Reservoir 1005.Wherein, communication bus 1002 is for realizing the connection communication between these components.User interface 1003 may include
Display screen (Display), input unit such as keyboard (Keyboard), optional user interface 1003 can also include having for standard
Line interface, wireless interface.Network interface 1004 optionally may include standard wireline interface and wireless interface (such as Wireless Fidelity
(WIreless-FIdelity, WI-FI) interface).Memory 1005 can be the random access memory (Random of high speed
Access Memory, RAM) memory, be also possible to stable nonvolatile memory (Non-Volatile Memory,
), such as magnetic disk storage NVM.Memory 1005 optionally can also be the storage device independently of aforementioned processor 1001.
It will be understood by those skilled in the art that structure shown in Fig. 1 does not constitute the state to distributed information system
The restriction of monitoring device may include perhaps combining certain components or different components than illustrating more or fewer components
Arrangement.
As shown in Figure 1, as may include operating system, data storage mould in a kind of memory 1005 of storage medium
Block, network communication module, Subscriber Interface Module SIM and distributed information system State Monitoring Program.
In the status monitor device of distributed information system shown in Fig. 1, network interface 1004 is mainly used for and network
Server carries out data communication；User interface 1003 is mainly used for carrying out data interaction with user；Distributed message system of the present invention
Processor 1001, memory 1005 in the status monitor device of system can be set the condition monitoring in distributed information system and set
In standby, the status monitor device of the distributed information system calls the distribution stored in memory 1005 by processor 1001
The State Monitoring Program of formula message system, and execute the condition monitoring side of distributed information system provided in an embodiment of the present invention
Method.
The embodiment of the invention provides a kind of method for monitoring state of distributed information system, are this hair referring to Fig. 2, Fig. 2
The flow diagram of the method for monitoring state first embodiment of bright distributed information system.
In the present embodiment, the method for monitoring state of the distributed information system the following steps are included:
Step S10: monitoring host computer is instructed in response to the condition monitoring received, calls preset data acquisition script by default
The current state parameter of dimension acquisition distributed information system；
It should be noted that the monitoring host computer, which can be, carries the master that distributed post subscribes to message system Kafka
Machine equipment is also installed with a Java administration extensions component (Java Management under normal conditions in the host equipment
Extensions, JMX), for monitoring the operating status of Kafka.The default dimension can be staff write it is described
Preset data acquires the data acquisition dimension defined when script, and the default dimension includes but is not limited to system operation in the present embodiment
Situation, resource distribution situation etc.；The current state parameter can be the running state data of distributed information system, such as be
System memory configurations parameter, instantaneous amount of access, access response speed and message throughput etc..
In the concrete realization, monitoring host computer first opens the JMX group in the condition monitoring instruction for receiving user's transmission
Then part calls preset data acquisition script to parse the JMX component, to acquire distributed information system different dimensions
Current state parameter.
Step S20: data transfer components read the current state parameter from the monitoring host computer, to the current shape
State parameter carries out initialization process and obtains init state parameter；
It should be noted that data transfer components described in the present embodiment and following each embodiments can be ELK log point
The Logstash component in platform (hereinafter referred to as ELK platform) is analysed, which is usually used in doing day in log analysis platform or system
Will acquires equipment.Initialization process, that is, the data transfer components are by calling filter inserts Filter to flow through currently to handle
The event information of Logstash, such as addition field, removal field, conversion field type, pass through regular expression cutting data
Deng, naturally it is also possible to different data processings is carried out according to condition judgement.
It should be understood that the ELK platform is by Elasticsearch, Logstash, tri- kinds of Open-Source Tools of Kibana are taken
Log Analysis System made of building, wherein Elasticsearch is an open source distributed search engine, provides collection, divides
Analysis, storing data three zones；Logstash be then mainly used for the collection of log, analysis, filtering log tool, support big
The data acquiring mode of amount；Kibana can be Web circle for the log analysis close friend that Logstash and ElasticSearch are provided
Face can help to summarize, analyze and search for significant data log.Logstash in ELK platform is applied to point by the present embodiment
In the condition monitoring of cloth distribution subscription message system Kafka, the timeliness of monitoring can be effectively improved and supported extensive
The data monitoring of Kafka cluster.
In the concrete realization, data transfer components read distributed post from monitoring host computer and subscribe to the current of message system
Then state parameter carries out initialization process to the current state parameter by preset filter inserts and obtains init state
Parameter.
Step S30: the data transfer components are in the data acquisition instruction for receiving the transmission of data search component, record
The data search component sends the sending time of the data acquisition instruction；
It should be noted that data search component described in the present embodiment and following each embodiments can be in ELK platform
Elasticsearch component, Elasticsearch is the distributed multi-user ability based on RESTful web interface
Full-text search engine.Elasticsearch component can obtain data and be saved from Logstash component under normal circumstances, together
When it can will get time of related data as the actual acquisition time of the data from Logstash component, but in view of number
Data search component Elasticsearch is sent to data transfer components Logstash according in instantaneity the present embodiment of acquisition
Sending time when data acquisition instructs carries out the acquisition time of data acquisition as it.
In the concrete realization, data transfer components are in the data acquisition instruction for receiving the transmission of data search component, note
It records data search component and sends corresponding instruction sending time when data acquisition instruction.
Step S40: the data transfer components carry out parameter to the sending time and redefine, and complete the parameter
When redefining, the init state parameter is sent to the data search component and is saved.
It should be understood that, it is contemplated that data search component ElasticSearch is obtained from data transfer components Logstash
When taking Various types of data, can practical generation time using time when itself collecting the data as the data it is (such as distributed
The practical generation time of distribution subscription message system Kafka a certain moment corresponding running state parameter A is 2018-05-11
16:30:30, if the running state parameter A time that collects of data search component ElasticSearch is 2018-05-12
10:30:30, then data search component ElasticSearch will join 2018-05-12 10:30:30 as the operating status
The practical generation time of number A).Therefore number is collected according to itself in order to avoid data search component ElasticSearch is subsequent
According to when acquisition time establish acquisition time and data between index distortion, it is necessary to the sending time carry out the time
Parameter redefines, to guarantee the accuracy and reliability of the index finally established.
In the concrete realization, data transfer components can be by calling preset data filter inserts (such as Filter plug-in unit) from institute
Extraction object time parameter in init state parameter is stated, then according to the target date field pair for including in object time parameter
Sending time carries out date field replacement, and init state parameter is sent to data search component again when replacing and completing and is carried out
It saves.
The present embodiment monitoring host computer is instructed in response to the condition monitoring received, calls preset data acquisition script by default
The current state parameter of dimension acquisition distributed information system；Data transfer components, which are stated, reads current state ginseng in monitoring host computer
Number carries out initialization process to current state parameter and obtains init state parameter；Receiving the transmission of data search component
When data acquisition instructs, record data search component sends the sending time of data acquisition instruction；Then sending time is carried out
Parameter redefines, and when completion parameter redefines, init state parameter is sent to data search component and is saved, by
The acquisition of state parameter is carried out by default dimension thus according to preset data acquisition script, to realize to distributed message
The various dimensions of system running state monitor, and ensure that the information security and structure safety of distributed information system.
With reference to Fig. 3, Fig. 3 is the process signal of the method for monitoring state second embodiment of distributed information system of the present invention
Figure.
Based on above-mentioned first embodiment, in the present embodiment, the preset data acquisition script includes: data decryptor script
With format analysis processing script, correspondingly the step S10 is specifically included:
Step S101: monitoring host computer is instructed in response to the condition monitoring received, calls the data decryptor script by pre-
If the reset condition parameter of dimension acquisition distributed information system；
It should be noted that staff needs to write data acquisition foot according to monitoring demand before executing this step
This, the preset data acquires script in the present embodiment can include: data decryptor script " KafkaDataMonitor.java "
With format analysis processing script " kafka_mon.sh ", wherein the data decryptor script obtains distribution by parsing JMX component
The running state parameter at message system current time, i.e., the described reset condition parameter.
In the concrete realization, monitoring host computer first opens the JMX group in the condition monitoring instruction for receiving user's transmission
Then part calls data decryptor script to parse the JMX component, to acquire the original of distributed information system different dimensions
Beginning state parameter.
Step S102: the monitoring host computer calls the format analysis processing script to be formatted the reset condition parameter
Processing obtains current state parameter.
It should be noted that monitoring host computer gets the original shape of distributed information system by the data decryptor script
After state parameter, distributed message system in reset condition parameter combination that the format analysis processing script write in advance will acquire can be passed through
It unites after the system banner (such as systematic name, number) and the acquisition time of parameter of Kafka, is processed into JS object numbered musical notation
Then data, that is, the current state parameter of (JavaScript Object Notation, JSON) format is worked as by described in again
Preceding state parameter is sent to 1234 network ports of the local loopback interface 127.0.0.1 of monitoring host computer, in order to Logstash
Component obtains the current state parameter by monitoring 1234 network port.
In the concrete realization, the monitoring host computer obtains acquisition time when acquiring the reset condition parameter and described
The corresponding system banner of distributed information system；By the format analysis processing script by the system banner and the acquisition time
It is added in the reset condition parameter, obtains the current state parameter.
The present embodiment monitoring host computer is instructed in response to the condition monitoring received, and data decryptor script is called to press default dimension
The reset condition parameter of distributed information system is acquired, then the reset condition parameter is carried out by format analysis processing script again
It formats processing and obtains current state parameter, it is same guaranteeing to acquire distributed information system running state parameter various dimensions
When, by the data format of unified each reset condition parameter, improves subsequent data transmission component and current state parameter is carried out
The treatment effeciency of data processing, at the same by reset condition parameter group close the system mark of monitored distributed information system
Acquisition time when knowledge and reset condition parameter, to provide convenience for the subsequent operation maintenance to distributed information system.
With reference to Fig. 4, Fig. 4 is the process signal of the method for monitoring state 3rd embodiment of distributed information system of the present invention
Figure.
Based on the various embodiments described above, in the present embodiment, the step S20 be may particularly include:
Step S201: data transfer components are read from the monitoring host computer described current by preset parameter monitoring script
State parameter；
In the concrete realization, data transfer components Logstash is monitored by calling the parameter write in advance to monitor script
1234 network ports of the local loopback interface 127.0.0.1 of the monitoring host computer are to obtain the current state parameter.
Step S202: the data transfer components carry out format to the acquisition time in the current state parameter
Change, obtains init state parameter.
In this step, data transfer components Logstash will be worked as by the Filter filter inserts for calling itself to configure
The institute of the string format (e.g., 2018-05-11 16:30:30.830) or U nix timestamp format that include in preceding state parameter
It states the date field that acquisition time is formatted as ISO8601 time format, obtains init state parameter, such as by Beijing time
On May 11st, 2018, in 30 minutes and 30 seconds at 4 points in afternoon, be converted into the date field of ISO8601 time format then are as follows: 2018-05-
11T16:30:30+08:00 or 20180511T163030+08.
In the concrete realization, data transfer components analyze monitoring host computer by filter inserts from current state parameter and adopt
Collect the acquisition time when reset condition parameter, then the acquisition time is formatted into the date word of object time format
It is put into@timestamp field after section, to obtain init state parameter.
The present embodiment data transfer components monitor script by preset parameter and read current state parameter from monitoring host computer,
Acquisition time in current state parameter is formatted, init state parameter is obtained, ensure that collected distribution
The timeliness of the state parameter of message system
With reference to Fig. 5, Fig. 5 is the process signal of the method for monitoring state fourth embodiment of distributed information system of the present invention
Figure.
Based on the various embodiments described above, the step S40 be may particularly include in the present embodiment:
Step S401: the data transfer components call preset data filter inserts to mention from the init state parameter
Object time parameter is taken, the object time parameter is the formatted time parameter of the acquisition time；
It should be understood that ElasticSearch is the full-text search engine based on Lucene frame, it can be by all documents
Information be written in the data structure of index (Index), index foundation is mapping relations between word and document in index,
Under normal conditions in order to convenient for the operating status of each timing node of staff's subsequent query distributed information system ginseng
Number, ElasticSearch component when establishing index, can with date field and the corresponding init state parameter of date field it
Between mapping relations establish index, but in view of data search component ElasticSearch is from data transfer components
When obtaining Various types of data in Logstash, building maths modec can be carried out with time corresponding date field when itself collecting the data
Segmentum intercalaris when index between field and data really cannot reflect that distributed information system is each so as to cause the index of foundation
The running state parameter of point, so take the mode redefined to time parameter come such issues that solve in the present embodiment.
In the concrete realization, data transfer components Logstash calls preset data filter inserts (Filter plug-in unit) from obtaining
When extracting the corresponding time parameter, the i.e. target of formatted acquisition time in the init state parameter got
Between parameter.
Step S402: the data transfer components are according to the target date field for including in the object time parameter to institute
It states sending time and carries out date field replacement, the parameter of the sending time is redefined with realizing；
It should be understood that the sending time, that is, data search component ElasticSearch acquires init state parameter
When default acquisition time, in order to avoid data search component ElasticSearch is first as this using the default acquisition time
The practical generation time of beginningization state parameter needs to redefine the sending time, so-called to redefine i.e. according to extraction
The target date field for including in object time parameter out carries out date field replacement to the sending time, such as data are searched
The corresponding date field of default acquisition time (the i.e. described sending time) of rope component ElasticSearch is 2018-05-
11T16:30:30+08:00, but monitoring host computer this acquisition init state parameter time corresponding date field be 2018-
05-11T16:29:12+08:00, just need to redefine date field " 2018-05-11T16:30:30+08:00 " at this time for
Date field " 2018-05-11T16:29:12+08:00 " guarantees the accuracy of the index finally established.
Step S403: the data transfer components are when the completion parameter redefines, by the init state parameter
The data search component is sent to be saved.
In the concrete realization, data transfer components Logstash when redefining, will initialize shape in deadline parameter
State parameter is sent to data search component ElasticSearch and is saved.
Further, data transfer components Logstash can also be searched when the completion parameter redefines in the data
It is searched whether in the corresponding database of rope component in the presence of the target index information for including the target date field；It is finding
When the target index information, the init state parameter is sent to the data search component, so that the data are searched
The init state parameter is saved storage region corresponding to the target index information by rope component；Correspondingly, it is not looking into
When finding the target index information, the init state parameter is sent to the data search component；The data are searched
Rope component establishes the target date field and institute according to the target date field for including in the init state parameter
State the index information between init state parameter.
Further, in the present embodiment, data search component ElasticSearch is establishing init state parameter pair
After the index answered, Kibana component in ELK platform can according to data search component ElasticSearch foundation rope
Draw, it is (a certain in original state parameter such as using time parameter as X-axis according to monitoring dimension set by user in display interface
Design parameter is Y-axis) show the situation of change of specific init state parameter, the personnel of working at the same time can also be pre-configured with
One monitoring threshold value, so that Kibana component is monitoring that the corresponding numerical value of target monitoring parameter in original state parameter is more than to be somebody's turn to do
Switch fault report is generated when monitoring threshold value, and the switch fault report is sent to relevant staff or is alerted in the display interface
Prompt.
When the present embodiment data transfer components call preset data filter inserts to extract target from init state parameter
Between parameter, date field replacement is carried out to sending time to realize pair according to the target date field for including in object time parameter
The parameter of sending time redefines, complete parameter redefine when, by init state parameter be sent to data search component into
Row saves, and ensure that the index accuracy and reliability that data search component is established.
In addition, the embodiment of the present invention also proposes a kind of storage medium, distributed message system is stored on the storage medium
It is realized when the State Monitoring Program of the State Monitoring Program of system, the distributed information system is executed by processor as described above
Distributed information system method for monitoring state the step of.
It is the structural block diagram of the state monitoring device first embodiment of distributed information system of the present invention referring to Fig. 6, Fig. 6.
As shown in fig. 6, the state monitoring device for the distributed information system that the embodiment of the present invention proposes includes: condition monitoring
Module 601, data transmission module 602 and data search module 603；
Wherein, the monitoring module 601, for calling preset data acquisition when receiving condition monitoring instruction
Script presses the current state parameter of default dimension acquisition distributed information system；
It should be noted that the state monitoring device of the distributed information system can be and carry distributed post and order
The device of message system Kafka is read, is also installed with a Java administration extensions component (Java in the device under normal conditions
Management Extensions, JMX), for monitoring the operating status of Kafka.The default dimension can be staff
The data defined when writing preset data acquisition script acquire dimension, and the default dimension includes but unlimited in the present embodiment
In running situation, resource distribution situation etc.；The current state parameter can be the operating status of distributed information system
Data, such as Installed System Memory configuration parameter, instantaneous amount of access, access response speed and message throughput etc..
The data transmission module 602 carries out just the current state parameter for reading the current state parameter
Beginning handles to obtain init state parameter；
It should be noted that initialization process, that is, the data transmission module 602 by call filter inserts Filter come
Processing flows through the event information of current Logstash, such as addition field, removal field, conversion field type, passes through canonical table
Up to formula cutting data etc., naturally it is also possible to carry out different data processings according to condition judgement.
The data transmission module 602, when for being instructed in the data acquisition for receiving the transmission of data search module 603,
Record the sending time that the data search module 603 sends the data acquisition instruction；
In the concrete realization, data transmission module 602 is in the data acquisition instruction for receiving the transmission of data search module 603
When, record data search module 603 sends corresponding instruction sending time when data acquisition instruction.
The data transmission module 602 is redefined for carrying out parameter to the sending time, and completes the parameter
When redefining, the init state parameter is sent to the data search module 603 and is saved.
In the concrete realization, data transmission module 602 can be by calling preset data filter inserts (such as Filter plug-in unit)
Object time parameter is extracted from the init state parameter, then according to the target date word for including in object time parameter
Section carries out date field replacement to sending time, and init state parameter is sent to data search module again when replacing and completing
603 are saved.
The present embodiment by carrying out the acquisition of state parameter by default dimension thus according to preset data acquisition script, thus
The various dimensions monitoring to distributed information system operating status is realized, ensure that the information security and knot of distributed information system
Structure safety.
Based on the state monitoring device first embodiment of the above-mentioned distributed information system of the present invention, propose that the present invention is distributed
The second embodiment of the state monitoring device of message system.
In the present embodiment, the preset data acquisition script includes: data decryptor script and format analysis processing script；Accordingly
The ground monitoring module 601 is also used to call the data decryptor script to press in response to the condition monitoring instruction received
The reset condition parameter of default dimension acquisition distributed information system, calls the format analysis processing script to join the reset condition
Number is formatted processing, obtains current state parameter.
Further, the monitoring module 601 is also used to obtain corresponding when acquiring the reset condition parameter adopt
Collect time and the corresponding system banner of the distributed information system；By the format analysis processing script by the system banner
It is added in the reset condition parameter with the acquisition time, obtains the current state parameter.
Further, the data transmission module 602 is also used to monitor script from the monitoring host computer by preset parameter
It is middle to read the current state parameter；The acquisition time in the current state parameter is formatted, is obtained initial
Change state parameter.
Further, the data transmission module 602 is also used to call preset data filter inserts from the initialization shape
Object time parameter is extracted in state parameter, the object time parameter is the formatted time parameter of the acquisition time；Root
Date field replacement is carried out to the sending time according to the target date field for including in the object time parameter, with realization pair
The parameter of the sending time redefines；When the completion parameter redefines, the init state parameter is sent to institute
Data search module 603 is stated to be saved.
Further, the data transmission module 602 is also used to when the completion parameter redefines, in the data
It is searched whether in the corresponding database of search module 603 in the presence of the target index information for including the target date field；?
When finding the target index information, the init state parameter is sent to the data search module 603, so that institute
It states data search module 603 and the init state parameter is saved into storage region corresponding to the target index information.
Further, the data transmission module 602 is also used to when not finding the target index information, by institute
It states init state parameter and is sent to the data search module 603；The data search module 603 is also used to according to described first
The target date field for including in beginningization state parameter establishes the target date field and the init state parameter
Between index information.
The other embodiments or specific implementation of the state monitoring device of distributed information system of the present invention can refer to
Each method embodiment is stated, details are not described herein again.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the system that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or system institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or system.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in a storage medium
In (such as read-only memory/random access memory, magnetic disk, CD), including some instructions are used so that a terminal device (can
To be mobile phone, computer, server, air conditioner or the network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of method for monitoring state of distributed information system, which is characterized in that the described method includes:
Monitoring host computer is instructed in response to the condition monitoring received, and preset data acquisition script is called to press default dimension acquisition distribution
The current state parameter of formula message system；
Data transfer components read the current state parameter from the monitoring host computer, carry out just to the current state parameter
Beginning handles to obtain init state parameter；
The data transfer components record the data search in the data acquisition instruction for receiving the transmission of data search component
Component sends the sending time of the data acquisition instruction；
The data transfer components carry out parameter to the sending time and redefine, and when the completion parameter redefines, will
The init state parameter is sent to the data search component and is saved.
2. the method as described in claim 1, which is characterized in that the preset data acquisition script includes: data decryptor script
With format analysis processing script；
The monitoring host computer is instructed in response to the condition monitoring received, and preset data acquisition script is called to press default dimension acquisition
The step of current state parameter of distributed information system, comprising:
Monitoring host computer is instructed in response to the condition monitoring received, is called the data decryptor script to acquire by default dimension and is distributed
The reset condition parameter of formula message system；
The monitoring host computer calls the format analysis processing script to be formatted processing to the reset condition parameter, obtains current
State parameter.
3. method according to claim 2, which is characterized in that the monitoring host computer calls the format analysis processing script to described
The step of reset condition parameter is formatted processing, obtains current state parameter, comprising:
The monitoring host computer obtains corresponding acquisition time and the distributed message system when acquiring the reset condition parameter
It unites corresponding system banner；
The system banner and the acquisition time are added to the original by the format analysis processing script by the monitoring host computer
In beginning state parameter, the current state parameter is obtained.
4. method as claimed in claim 3, which is characterized in that the data transfer components read institute from the monitoring host computer
Current state parameter is stated, the step of initialization process obtains init state parameter is carried out to the current state parameter, comprising:
Data transfer components monitor script by preset parameter and read the current state parameter from the monitoring host computer；
The data transfer components are formatted the acquisition time in the current state parameter, obtain initialization shape
State parameter.
5. method as claimed in claim 4, which is characterized in that the data transfer components carry out parameter to the sending time
Redefine, and when completing the parameter and redefining, by the init state parameter be sent to the data search component into
The step of row saves, comprising:
The data transfer components call preset data filter inserts to extract object time ginseng from the init state parameter
Number, the object time parameter are the formatted time parameter of the acquisition time；
The data transfer components according to the target date field for including in the object time parameter to the sending time into
The replacement of row date field, redefines the parameter of the sending time with realizing；
The init state parameter is sent to the data when the completion parameter redefines by the data transfer components
Searching component is saved.
6. method as claimed in claim 5, which is characterized in that the data transfer components are redefined in the completion parameter
When, the init state parameter is sent to the step of data search component is saved, comprising:
The data transfer components are looked into the corresponding database of the data search component when the completion parameter is redefined
It looks for the presence or absence of the target index information for including the target date field；
The init state parameter is sent to described by the data transfer components when finding the target index information
Data search component, so that the data search component saves the init state parameter to the target index information pair
The storage region answered.
7. method as claimed in claim 6, which is characterized in that the data transfer components are redefined in the completion parameter
When, it is searched whether in the corresponding database of the data search component in the presence of the target rope for including the target date field
After the step of drawing, the method also includes:
The data transfer components are sent to institute when not finding the target index information, by the init state parameter
State data search component；
The data search component establishes the mesh according to the target date field for including in the init state parameter
Mark the index information between date field and the init state parameter.
8. a kind of state monitoring device of distributed information system, which is characterized in that described device include: monitoring module,
Data transmission module and data search module；
Wherein, the monitoring module, for calling preset data acquisition script by pre- when receiving condition monitoring instruction
If the current state parameter of dimension acquisition distributed information system；
The data transmission module carries out at initialization the current state parameter for reading the current state parameter
Reason obtains init state parameter；
The data transmission module, for recording the number in the data acquisition instruction for receiving the transmission of data search module
The sending time of the data acquisition instruction is sent according to search module；
The data transmission module is redefined for carrying out parameter to the sending time, and is redefined completing the parameter
When, the init state parameter is sent to the data search module and is saved.
9. a kind of status monitor device of distributed information system, which is characterized in that the equipment includes: memory, processor
And it is stored in the State Monitoring Program for the distributed information system that can be run on the memory and on the processor, it is described
The State Monitoring Program of distributed information system is arranged for carrying out the distributed message as described in any one of claims 1 to 7
The step of method for monitoring state of system.
10. a kind of storage medium, which is characterized in that be stored with the condition monitoring journey of distributed information system on the storage medium
It is realized as described in any one of claim 1 to 7 when the State Monitoring Program of sequence, the distributed information system is executed by processor
Distributed information system method for monitoring state the step of.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811049716.5A CN109684159A (en) | 2018-09-07 | 2018-09-07 | Method for monitoring state, device, equipment and the storage medium of distributed information system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811049716.5A CN109684159A (en) | 2018-09-07 | 2018-09-07 | Method for monitoring state, device, equipment and the storage medium of distributed information system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109684159A true CN109684159A (en) | 2019-04-26 |
Family
ID=66184494
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811049716.5A Pending CN109684159A (en) | 2018-09-07 | 2018-09-07 | Method for monitoring state, device, equipment and the storage medium of distributed information system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109684159A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110677271A (en) * | 2019-08-16 | 2020-01-10 | 平安科技（深圳）有限公司 | Big data alarm method, device, equipment and storage medium based on ELK |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070083813A1 (en) * | 2005-10-11 | 2007-04-12 | Knoa Software, Inc | Generic, multi-instance method and GUI detection system for tracking and monitoring computer applications |
| CN106815115A (en) * | 2017-01-13 | 2017-06-09 | 郑州云海信息技术有限公司 | A kind of operation condition of server monitoring system |
| CN107491375A (en) * | 2017-08-18 | 2017-12-19 | 国网山东省电力公司信息通信公司 | Equipment detection and fault early warning system and method under a kind of cloud computing environment |
| WO2018103315A1 (en) * | 2016-12-09 | 2018-06-14 | 上海壹账通金融科技有限公司 | Monitoring data processing method, apparatus, server and storage equipment |
| CN108182242A (en) * | 2017-12-28 | 2018-06-19 | 湖南大学 | A kind of indexing means for the inquiry of magnanimity multi dimensional numerical data area |
| CN108255687A (en) * | 2017-12-29 | 2018-07-06 | 五八同城信息技术有限公司 | Application program internal memory monitoring test method, device and the electronic equipment of terminal |
-
2018
- 2018-09-07 CN CN201811049716.5A patent/CN109684159A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070083813A1 (en) * | 2005-10-11 | 2007-04-12 | Knoa Software, Inc | Generic, multi-instance method and GUI detection system for tracking and monitoring computer applications |
| WO2018103315A1 (en) * | 2016-12-09 | 2018-06-14 | 上海壹账通金融科技有限公司 | Monitoring data processing method, apparatus, server and storage equipment |
| CN106815115A (en) * | 2017-01-13 | 2017-06-09 | 郑州云海信息技术有限公司 | A kind of operation condition of server monitoring system |
| CN107491375A (en) * | 2017-08-18 | 2017-12-19 | 国网山东省电力公司信息通信公司 | Equipment detection and fault early warning system and method under a kind of cloud computing environment |
| CN108182242A (en) * | 2017-12-28 | 2018-06-19 | 湖南大学 | A kind of indexing means for the inquiry of magnanimity multi dimensional numerical data area |
| CN108255687A (en) * | 2017-12-29 | 2018-07-06 | 五八同城信息技术有限公司 | Application program internal memory monitoring test method, device and the electronic equipment of terminal |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110677271A (en) * | 2019-08-16 | 2020-01-10 | 平安科技（深圳）有限公司 | Big data alarm method, device, equipment and storage medium based on ELK |
| CN110677271B (en) * | 2019-08-16 | 2022-06-24 | 平安科技（深圳）有限公司 | Big data alarm method, device, equipment and storage medium based on ELK |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104407964B (en) | A kind of centralized monitoring system and method based on data center | |
| CN109684370A (en) | Daily record data processing method, system, equipment and storage medium | |
| CN108365985A (en) | A kind of cluster management method, device, terminal device and storage medium | |
| CN107391746A (en) | Log analysis method, equipment and computer-readable recording medium | |
| CN110069464A (en) | Log analysis processing method, device, equipment and readable storage medium storing program for executing | |
| CN111752799A (en) | Service link tracking method, device, equipment and storage medium | |
| WO2021169268A1 (en) | Data processing method, apparatus and device, and storage medium | |
| CN105159964A (en) | Log monitoring method and system | |
| CN111400127B (en) | Service log monitoring method and device, storage medium and computer equipment | |
| CN112632135A (en) | Big data platform | |
| WO2019051948A1 (en) | Method, apparatus, server, and storage medium for processing monitoring data | |
| CN111259073A (en) | Intelligent business system running state studying and judging system based on logs, flow and business access | |
| CN113422794A (en) | Traffic recording and playback processing method and device and electronic equipment | |
| CN107783880A (en) | A kind of log analysis method of server system, device and server system | |
| CN111970151A (en) | Flow fault positioning method and system for virtual and container network | |
| CN102340791A (en) | System used for realizing data consistency and method thereof | |
| CN109684159A (en) | Method for monitoring state, device, equipment and the storage medium of distributed information system | |
| CN116136801B (en) | Cloud platform data processing method and device, electronic equipment and storage medium | |
| CN115766768A (en) | Method and device for designing sensing center in computational power network operating system | |
| CN110990237B (en) | Information collection system, method and storage medium | |
| CN109684158A (en) | Method for monitoring state, device, equipment and the storage medium of distributed coordination system | |
| CN114387123A (en) | Data acquisition management method | |
| CN104636372B (en) | The method and device of big data quantity processing is realized based on table | |
| CN113722135A (en) | Error log acquisition system, method, device and medium | |
| JP5802152B2 (en) | Communication network monitoring system, monitoring device, and communication network monitoring method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |