CN109815082A - KAFKA theme monitoring method, device, electronic equipment and storage medium - Google Patents

KAFKA theme monitoring method, device, electronic equipment and storage medium Download PDF

Info

Publication number
CN109815082A
CN109815082A CN201811537835.5A CN201811537835A CN109815082A CN 109815082 A CN109815082 A CN 109815082A CN 201811537835 A CN201811537835 A CN 201811537835A CN 109815082 A CN109815082 A CN 109815082A
Authority
CN
China
Prior art keywords
kafka
topics
configuration
current
theme
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201811537835.5A
Other languages
Chinese (zh)
Other versions
CN109815082B (en
Inventor
陈东杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Life Insurance Company of China Ltd
Original Assignee
Ping An Life Insurance Company of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Life Insurance Company of China Ltd filed Critical Ping An Life Insurance Company of China Ltd
Priority to CN201811537835.5A priority Critical patent/CN109815082B/en
Publication of CN109815082A publication Critical patent/CN109815082A/en
Application granted granted Critical
Publication of CN109815082B publication Critical patent/CN109815082B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Debugging And Monitoring (AREA)

Abstract

The present invention provides a kind of KAFKA theme monitoring method, device, electronic equipment and storage medium.The KAFKA theme monitoring method includes: to call kafka-topics source code using configuration script when receiving the monitoring instruction of KAFKA theme;Obtain KAFKA subject name;According to the KAFKA subject name, the kafka-topics source code is executed, obtains current KAFKA number of topics;Obtain configuration KAFKA number of topics;Compare the current KAFKA number of topics and the configuration KAFKA number of topics;When the current KAFKA number of topics is less than the configuration KAFKA number of topics, warning message is exported.The invention can avoid there is KAFKA theme deficient phenomena, realizes timely early warning, improve O&M efficiency.

Description

KAFKA theme monitoring method, device, electronic equipment and storage medium
Technical field
The present invention relates to software supervision technical fields more particularly to a kind of KAFKA theme monitoring method, device, electronics to set Standby and storage medium.
Background technique
KAFKA is that a kind of distributed post of high-throughput subscribes to message system, it can handle the net of consumer's scale Everything flow data in standing.In KAFKA, the affiliated class of data is divided using a category attribute, divides data This class, then referred to as topic.If KAFKA is seen as a database, topic be can be understood as in database One table, the name of topic are table name.
Therefore, if the topic in KAFKA is lacked, the loss of data and confusion of KAFKA will be caused, is influencing KAFKA just Often work.
Summary of the invention
In view of the foregoing, it is necessary to a kind of KAFKA theme monitoring method, device, electronic equipment and storage medium are provided, It is avoided that KAFKA theme deficient phenomena occur, realizes timely early warning, improve O&M efficiency.
A kind of KAFKA theme monitoring method, which comprises
When receiving the monitoring instruction of KAFKA theme, kafka-topics source code is called using configuration script;
Obtain KAFKA subject name;
According to the KAFKA subject name, the kafka-topics source code is executed, obtains current KAFKA number of topics;
Obtain configuration KAFKA number of topics;
Compare the current KAFKA number of topics and the configuration KAFKA number of topics;
When the current KAFKA number of topics is less than the configuration KAFKA number of topics, warning message is exported.
Preferred embodiment according to the present invention, the configuration script include kafka-run-class.sh script.
Preferred embodiment according to the present invention, after obtaining current KAFKA number of topics, the method also includes:
The current KAFKA number of topics is saved as into variable, for transferring.
Preferred embodiment according to the present invention, the acquisition configuration KAFKA number of topics include any of the following mode:
The configuration file for obtaining KAFKA, obtains the configuration KAFKA number of topics from the configuration file;Or
The history log for obtaining KAFKA obtains the KAFKA number of topics of last time operation from the history log As the configuration KAFKA number of topics.
Preferred embodiment according to the present invention, the output warning message include the combination of following one or more kinds of modes:
Alarm mail is sent to designated person;And/or
It dials the police emergency number to the designated person;And/or
Warning message is sent to the designated person;And/or
Issue alarm sound.
Preferred embodiment according to the present invention, the output warning message further include:
Obtain current time and preset alarm text;
Show the current time, the preset alarm text and the current KAFKA number of topics.
Preferred embodiment according to the present invention, the method also includes:
Pressure test is carried out to KAFKA, obtains simulation KAFKA number of topics;
Calculate the difference of the current KAFKA number of topics and the simulation KAFKA number of topics;
When the absolute value of the difference is greater than preset value, sound an alarm.
A kind of KAFKA theme monitoring device, described device include:
Call unit, for calling kafka-topics using configuration script when receiving the monitoring instruction of KAFKA theme Source code;
Acquiring unit, for obtaining KAFKA subject name;
Execution unit, for executing the kafka-topics source code, being worked as according to the KAFKA subject name Preceding KAFKA number of topics;
The acquiring unit is also used to obtain configuration KAFKA number of topics;
Comparison unit, for comparing the current KAFKA number of topics and the configuration KAFKA number of topics;
Output unit, for when the current KAFKA number of topics is less than the configuration KAFKA number of topics, output to be alarmed Information.
Preferred embodiment according to the present invention, the configuration script include kafka-run-class.sh script.
Preferred embodiment according to the present invention, described device further include:
Storage unit, for after obtaining current KAFKA number of topics, the current KAFKA number of topics to be saved as variable, For transferring.
Preferred embodiment according to the present invention, the acquiring unit obtain configuration KAFKA number of topics and include any of the following Mode:
The configuration file for obtaining KAFKA, obtains the configuration KAFKA number of topics from the configuration file;Or
The history log for obtaining KAFKA obtains the KAFKA number of topics of last time operation from the history log As the configuration KAFKA number of topics.
Preferred embodiment according to the present invention, the output unit output warning message include following one or more kinds of modes Combination:
Alarm mail is sent to designated person;And/or
It dials the police emergency number to the designated person;And/or
Warning message is sent to the designated person;And/or
Issue alarm sound.
Preferred embodiment according to the present invention, the output unit export warning message further include:
Obtain current time and preset alarm text;
Show the current time, the preset alarm text and the current KAFKA number of topics.
Preferred embodiment according to the present invention, described device further include:
Test cell obtains simulation KAFKA number of topics for carrying out pressure test to KAFKA;
Computing unit, for calculating the difference of the current KAFKA number of topics and the simulation KAFKA number of topics;
Issue unit, for sounding an alarm when the absolute value of the difference is greater than preset value.
A kind of electronic equipment, the electronic equipment include:
Memory stores at least one instruction;And
Processor executes the instruction stored in the memory to realize the KAFKA theme monitoring method.
A kind of computer readable storage medium is stored at least one instruction, institute in the computer readable storage medium At least one instruction is stated to be executed by the processor in electronic equipment to realize the KAFKA theme monitoring method.
As can be seen from the above technical solutions, the present invention can be when receiving the monitoring instruction of KAFKA theme, using configuration Script calls kafka-topics source code, and combines the KAFKA subject name obtained, executes the source kafka-topics generation Code further obtains configuration KAFKA number of topics, and work as described in comparison to quickly and accurately obtain current KAFKA number of topics Preceding KAFKA number of topics and the configuration KAFKA number of topics, when the current KAFKA number of topics is less than the configuration KAFKA theme It when number, sounds an alarm, to avoid the occurrence of KAFKA theme deficient phenomena, realizes timely early warning, improve O&M efficiency.
Detailed description of the invention
Fig. 1 is the flow chart of the preferred embodiment of KAFKA theme monitoring method of the present invention.
Fig. 2 is the functional block diagram of the preferred embodiment of KAFKA theme monitoring device of the present invention.
Fig. 3 is the structural schematic diagram of the electronic equipment for the preferred embodiment that the present invention realizes KAFKA theme monitoring method.
Main element symbol description
Electronic equipment 1
Memory 12
Processor 13
KAFKA theme monitoring device 11
Call unit 110
Acquiring unit 111
Execution unit 112
Comparison unit 113
Output unit 114
Storage unit 115
Test cell 116
Computing unit 117
Issue unit 118
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, right in the following with reference to the drawings and specific embodiments The present invention is described in detail.
As shown in Figure 1, being the flow chart of the preferred embodiment of KAFKA theme monitoring method of the present invention.According to different need It asks, the sequence of step can change in the flow chart, and certain steps can be omitted.
The KAFKA theme monitoring method is applied in one or more electronic equipment, and the electronic equipment is a kind of Can be according to the instruction for being previously set or storing, the automatic equipment for carrying out numerical value calculating and/or information processing, hardware include but Be not limited to microprocessor, specific integrated circuit (Application Specific Integrated Circuit, ASIC), can Program gate array (Field-Programmable Gate Array, FPGA), digital processing unit (Digital Signal Processor, DSP), embedded device etc..
The electronic equipment can be any electronic product that human-computer interaction can be carried out with user, for example, personal meter Calculation machine, tablet computer, smart phone, personal digital assistant (PersonalDigital Assistant, PDA), game machine, friendship Mutual formula Web TV (Internet Protocol Television, IPTV), intellectual wearable device etc..
The electronic equipment can also include the network equipment and/or user equipment.Wherein, the network equipment includes, but It is not limited to single network server, the server group of multiple network servers composition or based on cloud computing (Cloud Computing the cloud being made of a large amount of hosts or network server).
Network locating for the electronic equipment include but is not limited to internet, wide area network, Metropolitan Area Network (MAN), local area network, it is virtual specially With network (Virtual Private Network, VPN) etc..
S10, when receiving the monitoring instruction of KAFKA theme, the electronic equipment calls kafka- using configuration script Topics source code.
In at least one embodiment of the present invention, KAFKA is that a kind of distributed post of high-throughput subscribes to message system System, it can handle the everything flow data in the website of consumer's scale.In KAFKA, come using a category attribute The affiliated class for dividing data, divides this class of data, then referred to as topic theme.If KAFKA is seen as a data Library, then topic theme can be understood as a table in database, and topic subject name is table name.
Therefore, if the topic in KAFKA is lacked, the loss of data and confusion of KAFKA will be caused, is influencing KAFKA just Often work.
In at least one embodiment of the present invention, the electronic equipment, which receives the monitoring of KAFKA theme and instructs, includes, but not It is limited to the combination of following one or more:
(1) the KAFKA theme that the electronic equipment receives user's triggering monitors instruction.
Specifically, the user can trigger the KAFKA theme monitoring instruction, the triggering by clicking trigger key Key can be virtual key or physical button etc..
The user can also input corresponding phonetic order, be instructed using monitoring as the KAFKA theme, the present invention is not Limitation.
Either, the user can also input finger print information, to trigger described KAFKA theme monitoring instruction etc..
(2) signal that the electronic equipment triggers the KAFKA theme monitoring instruction every prefixed time interval is received.
Specifically, the prefixed time interval can be configured by the electronic equipment, can also by the user into Row custom-configures, and the present invention does not limit.
Such as: the prefixed time interval can be every 1 minute etc..
(3) receive the electronic equipment configuration triggers the signal of the KAFKA theme monitoring instruction in preset time.
Specifically, the preset time can be configured by the electronic equipment, can also be carried out certainly by the user Definition configuration, the present invention do not limit.
Such as: the preset time can be 10 points of the morning of December 1 etc..
(4) the KAFKA theme monitoring instruction that the electronic equipment triggers under configuration status is received.
Specifically, the configuration status includes, but are not limited to the state etc. of loss of data in KAFKA.
In at least one embodiment of the present invention, the electronic equipment is receiving the KAFKA theme monitoring instruction Afterwards, the identity of user can also be authenticated, such as: the electronic equipment can be by the identity information of the user (as used Name in an account book, password etc.) it is matched with the identity information of storage, wherein user corresponding to the identity information of the storage has Supervision authority.
Preferably, the configuration script includes kafka-run-class.sh script.
Specifically, the kafka-run-class.sh script is joined for calling TopicCommand class, and according to corresponding Number executes specified function, for example, in the present embodiment, the kafka-run-class.sh script is described for calling Kafka-topics source code, to get current KAFKA number of topics.
Further, the electronic equipment calls institute by the configuration script kafka-run-class.sh script It states kafka-topics source code, and by executing the kafka-topics source code, further gets and be currently running KAFKA number of topics, i.e., the described current KAFKA number of topics.
S11, the electronic equipment obtain KAFKA subject name.
In at least one embodiment of the present invention, according to the KAFKA subject name, the electronic equipment is available To the topic theme corresponding to the KAFKA subject name, and further count the current KAFKA number of topics.
Specifically, the electronic equipment acquisition KAFKA subject name includes, but are not limited to any one following mode:
(1) electronic equipment obtains the configuration file of KAFKA, and the KAFKA theme is obtained from the configuration file Title.
Specifically, when installing KAFKA, while the configuration file of KAFKA is saved.
It specifically, include KAFKA cluster name, the performance parameter of KAFKA, configuration KAFKA theme in the configuration file Number, KAFKA subject name etc..
(2) electronic equipment obtains the history log of KAFKA, and last time fortune is obtained from the history log Capable subject name is as the KAFKA subject name.
By above embodiment, it can more meet the currently used demand of user.
S12, the electronic equipment execute the kafka-topics source code, obtain according to the KAFKA subject name Current KAFKA number of topics.
In at least one embodiment of the present invention, the electronic equipment is according to the KAFKA subject name, and combines The parameters such as zookeeperIP (zookeeper Internet Protocol), port numbers, execute the source kafka-topics Code.
Such as: the electronic equipment executes the kafka-topics source code/wls/elis_ after filling parameter racs_kafka/bin/kafka-topics.sh--list--zookeeper30.16.4.197:2181,30.16.4.208: 2181,30.16.4.219:2181/kafka|egrep"T-HADOOP-MSG-R2P1|T-SKYNET-DATA-R1P6|T- SKYNET-DATA-R1P6-TAG | T-SKYNET-DATA-TAG-R1P6 | _ _ consumer_offsets "-c obtains described work as Preceding KAFKA number of topics.Wherein, the zookeeperIP is 30.16.4.197, and the port numbers are 2181, the KAFKA master Autograph is known as T-HADOOP-MSG-R2P1, T-SKYNET-DATA-R1P6 etc..
Preferably, after obtaining current KAFKA number of topics, the method also includes:
The current KAFKA number of topics is saved as variable by the electronic equipment, for transferring.
Under normal conditions, the current KAFKA number of topics can be stored in the running log of the KAFKA, but be worked as When the running log runs up to certain volume, the speed of service of the KAFKA will seriously affect, therefore, in the KAFKA Running log is periodically executed delete operation.
It is directed to above situation, in order to avoid subsequent the case where can not getting current KAFKA number of topics generation, institute It states electronic equipment and the current KAFKA number of topics is saved as into variable, transferred at any time with facilitating, and have more flexibility.
S13, the electronic equipment obtain configuration KAFKA number of topics.
In at least one embodiment of the present invention, the configuration KAFKA number of topics is the master that the KAFKA should have Inscribe number.
Specifically, the acquisition configuration KAFKA number of topics includes, but are not limited to any one following mode:
(1) electronic equipment obtains the configuration file of KAFKA, and the configuration KAFKA is obtained from the configuration file Number of topics.
Specifically, when installing KAFKA, while the configuration file of KAFKA is saved.
It wherein, include KAFKA cluster name, the performance parameter of KAFKA, configuration KAFKA number of topics in the configuration file Deng.
(2) electronic equipment obtains the history log of KAFKA, and last time fortune is obtained from the history log Capable KAFKA number of topics is as the configuration KAFKA number of topics.
Specifically, the KAFKA number of topics that the electronic equipment runs last time is as the configuration KAFKA number of topics, energy It is enough more to meet current monitoring demand.
S14, the electronic equipment compare the current KAFKA number of topics and the configuration KAFKA number of topics.
In at least one embodiment of the present invention, described current when deficient phenomena is not present in the KAFKA theme KAFKA number of topics should be consistent with the configuration KAFKA number of topics.
Therefore, in the present embodiment, the electronic equipment is by the current KAFKA number of topics and the configuration KAFKA master Topic number compares, so that it is determined that whether the current KAFKA number of topics is normal.
S15, when the current KAFKA number of topics is less than the configuration KAFKA number of topics, the electronic equipment output report Alert information.
In at least one embodiment of the present invention, when the current KAFKA number of topics is equal to the configuration KAFKA theme When number, the electronic equipment can determine that the current KAFKA number of topics is normal, and the electronic equipment is without alarm.
And when the current KAFKA number of topics is less than the configuration KAFKA number of topics, described in the electronic equipment determination KAFKA theme missing, the electronic equipment export warning message.
Specifically, the output warning message includes, but are not limited to the combination of following one or more kinds of modes:
(1) electronic equipment sends alarm mail to designated person.
(2) electronic equipment is dialed the police emergency number to the designated person.
(3) electronic equipment sends warning message to the designated person.
(4) alarm sound is issued.
Specifically, the designated person includes, but are not limited to the combination of following one or more: operation maintenance personnel, project Manager, developer etc..
By above embodiment, the electronic equipment can be realized timely early warning, and by configuring different alarm sides Formula effectively improves alarm efficiency.
Further, the electronic equipment exports warning message further include:
The electronic equipment obtains current time and preset alarm text, shows the current time, the preset alarm Text and the current KAFKA number of topics.
Specifically, the preset alarm text can be custom-configured by user, and the present invention does not limit, for example, institute Stating preset alarm text may include " kafka topic is less than 5 " etc..
By above embodiment, the electronic equipment can show simultaneously event occur time and details (such as: Described current KAFKA number of topics etc.), so that related personnel is become more apparent upon warning reason, and thus handled in time.
Preferably, the method also includes:
The electronic equipment carries out pressure test to KAFKA, obtains simulation KAFKA number of topics, and calculates described current The difference of KAFKA number of topics and the simulation KAFKA number of topics, when the absolute value of the difference is greater than preset value, the electricity Sub- equipment sounds an alarm.
Specifically, the pressure test refers to, by simulating the actual use scene of the electronic equipment, to realize to institute State the simulation test of the actual motion index of electronic equipment.
Specifically, the preset value can assist the case where determining KAFKA number of topics as the benchmark compared, such as: institute Stating preset value can be 3 etc..
Illustrated when actual conditions and analog case difference are larger there may be problem by above embodiment, it is described The timely early warning of electronic equipment, to reduce risk.
In conclusion the present invention can call kafka- using configuration script when receiving the monitoring instruction of KAFKA theme Topics source code, and combine the KAFKA subject name obtained, executes the kafka-topics source code, thus quickly, it is quasi- Really obtain current KAFKA number of topics, further obtain configuration KAFKA number of topics, and compare the current KAFKA number of topics and The configuration KAFKA number of topics is sounded an alarm when the current KAFKA number of topics is less than the configuration KAFKA number of topics, To avoid the occurrence of KAFKA theme deficient phenomena, realizes timely early warning, improve O&M efficiency.
As shown in Fig. 2, being the functional block diagram of the preferred embodiment of KAFKA theme monitoring device of the present invention.The KAFKA Theme monitoring device 11 includes call unit 110, acquiring unit 111, execution unit 112, comparison unit 113, output unit 114, storage unit 115, test cell 116, computing unit 117 and issue unit 118.So-called module/the unit of the present invention is Refer to that one kind can be performed by processor 13, and the series of computation machine program segment of fixed function can be completed, is stored in In memory 12.In the present embodiment, it will be described in detail in subsequent embodiment about the function of each module/unit.
When receiving the monitoring instruction of KAFKA theme, call unit 110 calls the source kafka-topics using configuration script Code.
In at least one embodiment of the present invention, KAFKA is that a kind of distributed post of high-throughput subscribes to message system System, it can handle the everything flow data in the website of consumer's scale.In KAFKA, come using a category attribute The affiliated class for dividing data, divides this class of data, then referred to as topic theme.If KAFKA is seen as a data Library, then topic theme can be understood as a table in database, and topic subject name is table name.
Therefore, if the topic in KAFKA is lacked, the loss of data and confusion of KAFKA will be caused, is influencing KAFKA just Often work.
In at least one embodiment of the present invention, the electronic equipment, which receives the monitoring of KAFKA theme and instructs, includes, but not It is limited to the combination of following one or more:
(1) the KAFKA theme that the electronic equipment receives user's triggering monitors instruction.
Specifically, the user can trigger the KAFKA theme monitoring instruction, the triggering by clicking trigger key Key can be virtual key or physical button etc..
The user can also input corresponding phonetic order, be instructed using monitoring as the KAFKA theme, the present invention is not Limitation.
Either, the user can also input finger print information, to trigger described KAFKA theme monitoring instruction etc..
(2) signal that the electronic equipment triggers the KAFKA theme monitoring instruction every prefixed time interval is received.
Specifically, the prefixed time interval can be configured by the electronic equipment, can also by the user into Row custom-configures, and the present invention does not limit.
Such as: the prefixed time interval can be every 1 minute etc..
(3) receive the electronic equipment configuration triggers the signal of the KAFKA theme monitoring instruction in preset time.
Specifically, the preset time can be configured by the electronic equipment, can also be carried out certainly by the user Definition configuration, the present invention do not limit.
Such as: the preset time can be 10 points of the morning of December 1 etc..
(4) the KAFKA theme monitoring instruction that the electronic equipment triggers under configuration status is received.
Specifically, the configuration status includes, but are not limited to the state etc. of loss of data in KAFKA.
In at least one embodiment of the present invention, the electronic equipment is receiving the KAFKA theme monitoring instruction Afterwards, the identity of user can also be authenticated, such as: the electronic equipment can be by the identity information of the user (as used Name in an account book, password etc.) it is matched with the identity information of storage, wherein user corresponding to the identity information of the storage has Supervision authority.
Preferably, the configuration script includes kafka-run-class.sh script.
Specifically, the kafka-run-class.sh script is joined for calling TopicCommand class, and according to corresponding Number executes specified function, for example, in the present embodiment, the kafka-run-class.sh script is described for calling Kafka-topics source code, to get current KAFKA number of topics.
Further, the call unit 110 is called by the configuration script kafka-run-class.sh script The kafka-topics source code, and by executing the kafka-topics source code, it further gets and is currently running KAFKA number of topics, i.e., the described current KAFKA number of topics.
Acquiring unit 111 obtains KAFKA subject name.
In at least one embodiment of the present invention, according to the KAFKA subject name, the acquiring unit 111 can be with The topic theme corresponding to the KAFKA subject name is got, and further counts the current KAFKA number of topics.
Specifically, the acquisition of the acquiring unit 111 KAFKA subject name includes, but are not limited to any one following mode:
(1) acquiring unit 111 obtains the configuration file of KAFKA, and the KAFKA master is obtained from the configuration file Autograph claims.
Specifically, when installing KAFKA, while the configuration file of KAFKA is saved.
It specifically, include KAFKA cluster name, the performance parameter of KAFKA, configuration KAFKA theme in the configuration file Number, KAFKA subject name etc..
(2) acquiring unit 111 obtains the history log of KAFKA, from obtaining in the history log The subject name of secondary operation is as the KAFKA subject name.
By above embodiment, it can more meet the currently used demand of user.
Execution unit 112 executes the kafka-topics source code according to the KAFKA subject name, obtains current KAFKA number of topics.
In at least one embodiment of the present invention, the execution unit 112 is according to the KAFKA subject name, and ties The parameters such as zookeeperIP (zookeeper Internet Protocol), port numbers are closed, the kafka-topics is executed Source code.
Such as: the execution unit 112 executes the kafka-topics source code/wls/elis_ after filling parameter racs_kafka/bin/kafka-topics.sh--list--zookeeper30.16.4.197:2181,30.16.4.208: 2181,30.16.4.219:2181/kafka|egrep"T-HADOOP-MSG-R2P1|T-SKYNET-DATA-R1P6|T- SKYNET-DATA-R1P6-TAG | T-SKYNET-DATA-TAG-R1P6 | _ _ consumer_offsets "-c obtains described work as Preceding KAFKA number of topics.Wherein, the zookeeperIP is 30.16.4.197, and the port numbers are 2181, the KAFKA master Autograph is known as T-HADOOP-MSG-R2P1, T-SKYNET-DATA-R1P6 etc..
Preferably, after obtaining current KAFKA number of topics, the method also includes:
The current KAFKA number of topics is saved as variable by storage unit 115, for transferring.
Under normal conditions, the current KAFKA number of topics can be stored in the running log of the KAFKA, but be worked as When the running log runs up to certain volume, the speed of service of the KAFKA will seriously affect, therefore, in the KAFKA Running log is periodically executed delete operation.
It is directed to above situation, in order to avoid subsequent the case where can not getting current KAFKA number of topics generation, institute It states electronic equipment and the current KAFKA number of topics is saved as into variable, transferred at any time with facilitating, and have more flexibility.
The acquiring unit 111 obtains configuration KAFKA number of topics.
In at least one embodiment of the present invention, the configuration KAFKA number of topics is the master that the KAFKA should have Inscribe number.
Specifically, the acquisition of the acquiring unit 111 configuration KAFKA number of topics includes, but are not limited to any one following side Formula:
(1) acquiring unit 111 obtains the configuration file of KAFKA, and the configuration is obtained from the configuration file KAFKA number of topics.
Specifically, when installing KAFKA, while the configuration file of KAFKA is saved.
It wherein, include KAFKA cluster name, the performance parameter of KAFKA, configuration KAFKA number of topics in the configuration file Deng.
(2) acquiring unit 111 obtains the history log of KAFKA, from obtaining in the history log The KAFKA number of topics of secondary operation is as the configuration KAFKA number of topics.
Specifically, the KAFKA number of topics that the acquiring unit 111 runs last time as the configuration KAFKA number of topics, Current monitoring demand can more be met.
Comparison unit 113 compares the current KAFKA number of topics and the configuration KAFKA number of topics.
In at least one embodiment of the present invention, described current when deficient phenomena is not present in the KAFKA theme KAFKA number of topics should be consistent with the configuration KAFKA number of topics.
Therefore, in the present embodiment, the comparison unit 113 is by the current KAFKA number of topics and the configuration KAFKA Number of topics compares, so that it is determined that whether the current KAFKA number of topics is normal.
When the current KAFKA number of topics is less than the configuration KAFKA number of topics, output unit 114 exports alarm signal Breath.
In at least one embodiment of the present invention, when the current KAFKA number of topics is equal to the configuration KAFKA theme When number, the output unit 114 can determine that the current KAFKA number of topics is normal, and the output unit 114 is without report It is alert.
And when the current KAFKA number of topics is less than the configuration KAFKA number of topics, the output unit 114 determines The KAFKA theme missing, the output unit 114 export warning message.
Specifically, the output of output unit 114 warning message includes, but are not limited to following one or more kinds of modes Combination:
(1) output unit 114 sends alarm mail to designated person.
(2) output unit 114 is dialed the police emergency number to the designated person.
(3) output unit 114 sends warning message to the designated person.
(4) output unit 114 issues alarm sound.
Specifically, the designated person includes, but are not limited to the combination of following one or more: operation maintenance personnel, project Manager, developer etc..
By above embodiment, the output unit 114 can be realized timely early warning, and by configuring different alarms Mode effectively improves alarm efficiency.
Further, the output unit 114 exports warning message further include:
The output unit 114 obtains current time and preset alarm text, shows the current time, the default report Alert text and the current KAFKA number of topics.
Specifically, the preset alarm text can be custom-configured by user, and the present invention does not limit, for example, institute Stating preset alarm text may include " kafka topic is less than 5 " etc..
By above embodiment, the output unit 114 can show the time and details that event occurs simultaneously (such as: the current KAFKA number of topics), makes related personnel become more apparent upon warning reason, and thus handled in time.
Preferably, the method also includes:
Test cell 116 carries out pressure test to KAFKA, obtains simulation KAFKA number of topics, and computing unit 117 calculates institute The difference for stating current KAFKA number of topics and the simulation KAFKA number of topics, when the absolute value of the difference is greater than preset value, Issue unit 118 sounds an alarm.
Specifically, the pressure test refers to, by simulating the actual use scene of the electronic equipment, to realize to institute State the simulation test of the actual motion index of electronic equipment.
Specifically, the preset value can assist the case where determining KAFKA number of topics as the benchmark compared, such as: institute Stating preset value can be 3 etc..
Illustrated when actual conditions and analog case difference are larger there may be problem by above embodiment, it is described The timely early warning of electronic equipment, to reduce risk.
In conclusion the present invention can call kafka- using configuration script when receiving the monitoring instruction of KAFKA theme Topics source code, and combine the KAFKA subject name obtained, executes the kafka-topics source code, thus quickly, it is quasi- Really obtain current KAFKA number of topics, further obtain configuration KAFKA number of topics, and compare the current KAFKA number of topics and The configuration KAFKA number of topics is sounded an alarm when the current KAFKA number of topics is less than the configuration KAFKA number of topics, To avoid the occurrence of KAFKA theme deficient phenomena, realizes timely early warning, improve O&M efficiency.
As shown in figure 3, being that the present invention realizes that the structure of electronic equipment of the preferred embodiment of KAFKA theme monitoring method is shown It is intended to.
The electronic equipment 1 be it is a kind of can according to the instruction for being previously set or store, automatic progress numerical value calculating and/or The equipment of information processing, hardware include but is not limited to microprocessor, specific integrated circuit (Application Specific Integrated Circuit, ASIC), programmable gate array (Field-Programmable Gate Array, FPGA), number Word processing device (Digital Signal Processor, DSP), embedded device etc..
The electronic equipment 1, which can also be but not limited to any one, to pass through keyboard, mouse, remote controler, touching with user The modes such as template or voice-operated device carry out the electronic product of human-computer interaction, for example, personal computer, tablet computer, smart phone, Personal digital assistant (PersonalDigital Assistant, PDA), game machine, Interactive Internet TV (Internet Protocol Television, IPTV), intellectual wearable device etc..
The electronic equipment 1 can also be that the calculating such as desktop PC, notebook, palm PC and cloud server are set It is standby.
Network locating for the electronic equipment 1 include but is not limited to internet, wide area network, Metropolitan Area Network (MAN), local area network, it is virtual specially With network (Virtual Private Network, VPN) etc..
In one embodiment of the invention, the electronic equipment 1 includes, but are not limited to memory 12, processor 13, And it is stored in the computer program that can be run in the memory 12 and on the processor 13, such as KAFKA theme prison Control program.
It will be understood by those skilled in the art that the schematic diagram is only the example of electronic equipment 1, not structure paired electrons The restriction of equipment 1 may include perhaps combining certain components or different components, example than illustrating more or fewer components Such as described electronic equipment 1 can also include input-output equipment, network access equipment, bus.
Alleged processor 13 can be central processing unit (Central Processing Unit, CPU), can also be Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit (Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field- Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic, Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor Deng the processor 13 is arithmetic core and the control centre of the electronic equipment 1, entire using various interfaces and connection The various pieces of electronic equipment 1, and execute the operating system of the electronic equipment 1 and types of applications program, the program of installation Code etc..
The processor 13 executes the operating system of the electronic equipment 1 and the types of applications program of installation.The place Reason device 13 executes the application program to realize the step in above-mentioned each KAFKA theme monitoring method embodiment, such as Fig. 1 institute Step S10, S11, S12, S13, S14, the S15 shown.
Alternatively, the processor 13 realizes each module in above-mentioned each Installation practice/mono- when executing the computer program The function of member, such as: when receiving the monitoring instruction of KAFKA theme, kafka-topics source code is called using configuration script; Obtain KAFKA subject name;According to the KAFKA subject name, the kafka-topics source code is executed, is obtained current KAFKA number of topics;Obtain configuration KAFKA number of topics;Compare the current KAFKA number of topics and the configuration KAFKA number of topics; When the current KAFKA number of topics is less than the configuration KAFKA number of topics, warning message is exported.
Illustratively, the computer program can be divided into one or more module/units, one or more A module/unit is stored in the memory 12, and is executed by the processor 13, to complete the present invention.It is one Or multiple module/units can be the series of computation machine program instruction section that can complete specific function, the instruction segment is for retouching State implementation procedure of the computer program in the electronic equipment 1.For example, the computer program can be divided into tune It is single with unit 110, acquiring unit 111, execution unit 112, comparison unit 113, output unit 114, storage unit 115, test Member 116, computing unit 117 and issue unit 118.
The memory 12 can be used for storing the computer program and/or module, the processor 13 by operation or The computer program and/or module being stored in the memory 12 are executed, and calls the data being stored in memory 12, Realize the various functions of the electronic equipment 1.The memory 12 can mainly include storing program area and storage data area, In, storing program area can application program needed for storage program area, at least one function (such as sound-playing function, image Playing function etc.) etc.;Storage data area, which can be stored, uses created data (such as audio data, phone directory according to mobile phone Deng) etc..In addition, memory 12 may include high-speed random access memory, it can also include nonvolatile memory, such as firmly Disk, memory, plug-in type hard disk, intelligent memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) block, flash card (Flash Card), at least one disk memory, flush memory device or other volatile solid-states Part.
The memory 12 can be the external memory and/or internal storage of electronic equipment 1.Further, described Memory 12 can be the circuit with store function for not having physical form in integrated circuit, such as RAM (Random-Access Memory, random access memory), FIFO (First In First Out) etc..Alternatively, the memory 12 is also possible to Memory with physical form, such as memory bar, TF card (Trans-flash Card).
If the integrated module/unit of the electronic equipment 1 is realized in the form of SFU software functional unit and as independent Product when selling or using, can store in a computer readable storage medium.Based on this understanding, the present invention is real All or part of the process in existing above-described embodiment method, can also instruct relevant hardware come complete by computer program At the computer program can be stored in a computer readable storage medium, which is being executed by processor When, it can be achieved that the step of above-mentioned each embodiment of the method.
Wherein, the computer program includes computer program code, and the computer program code can be source code Form, object identification code form, executable file or certain intermediate forms etc..The computer-readable medium may include: can Carry any entity or device, recording medium, USB flash disk, mobile hard disk, magnetic disk, CD, computer of the computer program code Memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium etc..It should be noted that the computer-readable medium The content for including can carry out increase and decrease appropriate according to the requirement made laws in jurisdiction with patent practice, such as in certain departments Method administrative area does not include electric carrier signal and telecommunication signal according to legislation and patent practice, computer-readable medium.
In conjunction with Fig. 1, the memory 12 in the electronic equipment 1 stores multiple instruction to realize a kind of KAFKA theme The multiple instruction can be performed to realize in monitoring method, the processor 13: when receiving the monitoring instruction of KAFKA theme, Kafka-topics source code is called using configuration script;Obtain KAFKA subject name;According to the KAFKA subject name, hold The row kafka-topics source code, obtains current KAFKA number of topics;Obtain configuration KAFKA number of topics;It compares described current KAFKA number of topics and the configuration KAFKA number of topics;When the current KAFKA number of topics is less than the configuration KAFKA number of topics When, export warning message.
Preferred embodiment according to the present invention, the configuration script include kafka-run-class.sh script.
Preferred embodiment according to the present invention, the processor 13 also execute multiple instruction and include:
The current KAFKA number of topics is saved as into variable, for transferring.
Preferred embodiment according to the present invention, the processor 13 also execute multiple instruction and include:
The configuration file for obtaining KAFKA, obtains the configuration KAFKA number of topics from the configuration file;Or
The history log for obtaining KAFKA obtains the KAFKA number of topics of last time operation from the history log As the configuration KAFKA number of topics.
Preferred embodiment according to the present invention, the processor 13 also execute multiple instruction and include:
Alarm mail is sent to designated person;And/or
It dials the police emergency number to the designated person;And/or
Warning message is sent to the designated person;And/or
Issue alarm sound.
Preferred embodiment according to the present invention, the processor 13 also execute multiple instruction and include:
Obtain current time and preset alarm text;
Show the current time, the preset alarm text and the current KAFKA number of topics.
Preferred embodiment according to the present invention, the processor 13 also execute multiple instruction and include:
Pressure test is carried out to KAFKA, obtains simulation KAFKA number of topics;
Calculate the difference of the current KAFKA number of topics and the simulation KAFKA number of topics;
When the absolute value of the difference is greater than preset value, sound an alarm.
Specifically, the processor 13 can refer to the concrete methods of realizing of above-metioned instruction related in Fig. 1 corresponding embodiment The description of step, this will not be repeated here.
In several embodiments provided by the present invention, it should be understood that disclosed system, device and method can be with It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the module It divides, only a kind of logical function partition, there may be another division manner in actual implementation.
The module as illustrated by the separation member may or may not be physically separated, aobvious as module The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple In network unit.Some or all of the modules therein can be selected to realize the mesh of this embodiment scheme according to the actual needs 's.
It, can also be in addition, each functional module in each embodiment of the present invention can integrate in one processing unit It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list Member both can take the form of hardware realization, can also realize in the form of hardware adds software function module.
It is obvious to a person skilled in the art that invention is not limited to the details of the above exemplary embodiments, Er Qie In the case where without departing substantially from spirit or essential attributes of the invention, the present invention can be realized in other specific forms.
Therefore, in all respects, the present embodiments are to be considered as illustrative and not restrictive, this The range of invention is indicated by the appended claims rather than the foregoing description, it is intended that the equivalent requirements of the claims will be fallen in All changes in meaning and scope are included in the present invention.Any attached associated diagram label in claim should not be considered as limit Claim involved in making.
Furthermore, it is to be understood that one word of " comprising " does not exclude other units or steps, odd number is not excluded for plural number.In system claims The multiple units or device of statement can also be implemented through software or hardware by a unit or device.Second equal words are used It indicates title, and does not indicate any particular order.
Finally it should be noted that the above examples are only used to illustrate the technical scheme of the present invention and are not limiting, although reference Preferred embodiment describes the invention in detail, those skilled in the art should understand that, it can be to of the invention Technical solution is modified or equivalent replacement, without departing from the spirit and scope of the technical solution of the present invention.

Claims (10)

1. a kind of KAFKA theme monitoring method, which is characterized in that the described method includes:
When receiving the monitoring instruction of KAFKA theme, kafka-topics source code is called using configuration script;
Obtain KAFKA subject name;
According to the KAFKA subject name, the kafka-topics source code is executed, obtains current KAFKA number of topics;
Obtain configuration KAFKA number of topics;
Compare the current KAFKA number of topics and the configuration KAFKA number of topics;
When the current KAFKA number of topics is less than the configuration KAFKA number of topics, warning message is exported.
2. KAFKA theme monitoring method as described in claim 1, which is characterized in that the configuration script includes kafka- Run-class.sh script.
3. KAFKA theme monitoring method as described in claim 1, which is characterized in that after obtaining current KAFKA number of topics, The method also includes:
The current KAFKA number of topics is saved as into variable, for transferring.
4. KAFKA theme monitoring method as described in claim 1, which is characterized in that the acquisition configures KAFKA number of topics packet Include any one following mode:
The configuration file for obtaining KAFKA, obtains the configuration KAFKA number of topics from the configuration file;Or
The history log for obtaining KAFKA obtains the KAFKA number of topics conduct of last time operation from the history log The configuration KAFKA number of topics.
5. KAFKA theme monitoring method as described in claim 1, which is characterized in that the output warning message includes following The combination of one or more kinds of modes:
Alarm mail is sent to designated person;And/or
It dials the police emergency number to the designated person;And/or
Warning message is sent to the designated person;And/or
Issue alarm sound.
6. KAFKA theme monitoring method as described in claim 1, which is characterized in that the output warning message further include:
Obtain current time and preset alarm text;
Show the current time, the preset alarm text and the current KAFKA number of topics.
7. KAFKA theme monitoring method as described in claim 1, which is characterized in that the method also includes:
Pressure test is carried out to KAFKA, obtains simulation KAFKA number of topics;
Calculate the difference of the current KAFKA number of topics and the simulation KAFKA number of topics;
When the absolute value of the difference is greater than preset value, sound an alarm.
8. a kind of KAFKA theme monitoring device, which is characterized in that described device includes:
Call unit, for calling the source kafka-topics generation using configuration script when receiving the monitoring instruction of KAFKA theme Code;
Acquiring unit, for obtaining KAFKA subject name;
Execution unit obtains current for executing the kafka-topics source code according to the KAFKA subject name KAFKA number of topics;
The acquiring unit is also used to obtain configuration KAFKA number of topics;
Comparison unit, for comparing the current KAFKA number of topics and the configuration KAFKA number of topics;
Output unit, for exporting warning message when the current KAFKA number of topics is less than the configuration KAFKA number of topics.
9. a kind of electronic equipment, which is characterized in that the electronic equipment includes:
Memory stores at least one instruction;And
Processor executes the instruction stored in the memory to realize as claimed in any of claims 1 to 7 in one of claims KAFKA theme monitoring method.
10. a kind of computer readable storage medium, it is characterised in that: be stored at least one in the computer readable storage medium A instruction, at least one described instruction are executed by the processor in electronic equipment to realize such as any one of claim 1 to 7 The KAFKA theme monitoring method.
CN201811537835.5A 2018-12-15 2018-12-15 KAFKA theme monitoring method and device, electronic equipment and storage medium Active CN109815082B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811537835.5A CN109815082B (en) 2018-12-15 2018-12-15 KAFKA theme monitoring method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811537835.5A CN109815082B (en) 2018-12-15 2018-12-15 KAFKA theme monitoring method and device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN109815082A true CN109815082A (en) 2019-05-28
CN109815082B CN109815082B (en) 2023-04-11

Family

ID=66602035

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811537835.5A Active CN109815082B (en) 2018-12-15 2018-12-15 KAFKA theme monitoring method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN109815082B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113608896A (en) * 2021-08-12 2021-11-05 重庆紫光华山智安科技有限公司 Method, system, medium and terminal for dynamically switching data stream
CN114253807A (en) * 2021-12-20 2022-03-29 深圳前海微众银行股份有限公司 Alarm information notification method and device

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140289231A1 (en) * 2013-03-25 2014-09-25 Salesforce.Com, Inc. Combining topic suggestions from different topic sources to assign to textual data items
CN106778033A (en) * 2017-01-10 2017-05-31 南京邮电大学 A kind of Spark Streaming abnormal temperature data alarm methods based on Spark platforms
CN107391279A (en) * 2017-07-31 2017-11-24 山东浪潮云服务信息科技有限公司 A kind of message queue container creation method, device and message queue container
CN107590210A (en) * 2017-08-25 2018-01-16 咪咕互动娱乐有限公司 A kind of data processing method, device, system and computer-readable recording medium
US20180032582A1 (en) * 2016-07-28 2018-02-01 International Business Machines Corporation Cross object synchronization
US20180063055A1 (en) * 2016-08-26 2018-03-01 Linkedin Corporation Repartitioning a topic in a publish-subscribe message system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140289231A1 (en) * 2013-03-25 2014-09-25 Salesforce.Com, Inc. Combining topic suggestions from different topic sources to assign to textual data items
US20180032582A1 (en) * 2016-07-28 2018-02-01 International Business Machines Corporation Cross object synchronization
US20180063055A1 (en) * 2016-08-26 2018-03-01 Linkedin Corporation Repartitioning a topic in a publish-subscribe message system
CN106778033A (en) * 2017-01-10 2017-05-31 南京邮电大学 A kind of Spark Streaming abnormal temperature data alarm methods based on Spark platforms
CN107391279A (en) * 2017-07-31 2017-11-24 山东浪潮云服务信息科技有限公司 A kind of message queue container creation method, device and message queue container
CN107590210A (en) * 2017-08-25 2018-01-16 咪咕互动娱乐有限公司 A kind of data processing method, device, system and computer-readable recording medium

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
ERNEST.WU: "Kafka常用命令之kafka-topics.sh", 《HTTPS://BLOG.CSDN.NET/QQ_29116427/ARTICLE/DETAILS/80202392?》 *
ZFSZHANGYUAN: "kafka单机重启topic丢失问题排查", 《HTTPS://BLOG.CSDN.NET/ZFSZHANGYUAN/ARTICLE/DETAILS/53389916》 *
苏洛尔寂: "Hadoop环境搭建笔记整理(六)——初识Kafka以及本阶段总结", 《HTTPS://ZHUANLAN.ZHIHU.COM/P/27851350》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113608896A (en) * 2021-08-12 2021-11-05 重庆紫光华山智安科技有限公司 Method, system, medium and terminal for dynamically switching data stream
CN113608896B (en) * 2021-08-12 2023-09-08 重庆紫光华山智安科技有限公司 Method, system, medium and terminal for dynamically switching data streams
CN114253807A (en) * 2021-12-20 2022-03-29 深圳前海微众银行股份有限公司 Alarm information notification method and device
CN114253807B (en) * 2021-12-20 2023-04-07 深圳前海微众银行股份有限公司 Alarm information notification method and device

Also Published As

Publication number Publication date
CN109815082B (en) 2023-04-11

Similar Documents

Publication Publication Date Title
CN109634818A (en) Log analysis method, system, terminal and computer readable storage medium
CN111885040A (en) Distributed network situation perception method, system, server and node equipment
CN109359026A (en) Log reporting method, device, electronic equipment and computer readable storage medium
CN109669837A (en) Equipment state method for early warning, system, computer installation and readable storage medium storing program for executing
CN109086182A (en) The method and terminal device of database auto-alarming
CN112070416B (en) AI-based RPA flow generation method, apparatus, device and medium
CN108595316A (en) Life cycle management method, manager, equipment and the medium of Distributed Application
CN111427748A (en) Task warning method, system, equipment and storage medium
CN105227405A (en) Method for supervising and system
CN106254137A (en) The alarm root-cause analysis system and method for supervisory systems
CN101277218B (en) Dynamic analysis system and method for network alarm
CN109800269A (en) Data managing method, device, computer equipment and storage medium
CN109542737A (en) Platform alert processing method, device, electronic device and storage medium
CN109815082A (en) KAFKA theme monitoring method, device, electronic equipment and storage medium
CN109800124A (en) CPU usage monitoring method, device, electronic equipment and storage medium
CN109284331A (en) Accreditation information acquisition method, terminal device and medium based on business datum resource
CN111914859A (en) Service multiplexing method, computing device and computer readable storage medium
CN109240916A (en) Information output controlling method, device and computer readable storage medium
CN109766236A (en) KAFKA message queue number monitoring method, device, electronic equipment and storage medium
CN114756301B (en) Log processing method, device and system
CN109889399A (en) RocketMQ client connection number monitoring method, device, electronic equipment and storage medium
CN110362294A (en) Development task executes method, apparatus, electronic equipment and storage medium
Zhang et al. Role of subgraphs in epidemics over finite-size networks under the scaled SIS process
CN105245380B (en) Message propagation mode identification method and device
CN114390015A (en) Data pushing system, method and equipment based on object model and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant