CN110493071A - Message system resources balance device, method and apparatus - Google Patents
Message system resources balance device, method and apparatus Download PDFInfo
- Publication number
- CN110493071A CN110493071A CN201810463407.6A CN201810463407A CN110493071A CN 110493071 A CN110493071 A CN 110493071A CN 201810463407 A CN201810463407 A CN 201810463407A CN 110493071 A CN110493071 A CN 110493071A
- Authority
- CN
- China
- Prior art keywords
- kafka
- cluster
- target
- index
- engine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 26
- 238000012544 monitoring process Methods 0.000 claims abstract description 25
- 238000005192 partition Methods 0.000 claims description 47
- 230000005856 abnormality Effects 0.000 claims description 22
- 238000001514 detection method Methods 0.000 claims description 22
- 238000004891 communication Methods 0.000 claims description 6
- 230000001052 transient effect Effects 0.000 claims description 6
- 230000002159 abnormal effect Effects 0.000 claims description 3
- 238000007689 inspection Methods 0.000 claims 1
- 238000005516 engineering process Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 230000036541 health Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000009434 installation Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000005303 weighing Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 230000007474 system interaction Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the present invention provides a kind of message system resources balance device, method and apparatus, and balancer includes load monitoring engine, analysis engine and operation enforcement engine；Load monitoring engine is used to acquire Kafka index from Kafka cluster, and Kafka index is sent to analysis engine；Analysis engine is used for the soft target according to Kafka index, the rigid target of Kafka cluster and Kafka cluster, generate the first equalization scheme for corresponding to all subscribers, and the first equalization scheme is sent to operation enforcement engine, wherein, rigid target is the target that several Kafka clusters must be realized, soft target is the target that several Kafka clusters are preferably realized；Enforcement engine is operated, for generating several operational orders according to the first equalization scheme, and several operational orders are sent to Kafka cluster so that Kafka cluster carries out resources balance according to operational order.The present invention can execute the operation of resource automatic equalization, promote the performance and user experience of message system using hard, soft target as core foundation.
Description
Technical field
The present embodiments relate to field of cloud computer technology, more particularly, to a kind of message system resources balance device,
Method and apparatus.
Background technique
As cloud computing is in the deep application of all trades and professions, application system, tenant's scale it is increasing and to system it
Between interaction real-time, platform data and application performance data consumption requirement of real-time it is also higher and higher, especially in telecommunications, finance
Field, core business system operate in cloud platform, for the platform data and applied business performance of large-scale application system
Real-time property is put forward higher requirements, in this regard, cloud management platform is also faced with new challenges: Floor layer Technology is complicated, subsystem is numerous
Between more, system interaction frequently, data volume it is big.Important set of the message system as asynchronous interactive between each subsystem of cloud pipe platform
Part is responsible for platform real time data, historical data, data interaction etc. between operational indicator real time data, historical data and platform
Function.
With the Quan Naguan of the resources such as IaaS, PaaS and SaaS, by pipe platform, increasing by pipe operation system scale,
Message system cluster needs to handle daily hundreds of thousands item even up to a million message, so large-scale size of message, any one
The Message Processing of message system service (Broker) fails, and can all lead to our whole cluster load imbalances, or even cause " snow
Collapse " effect.Thus will cause platform real time data can not acquire, update；Resource (container, virtual machine, physical machine, database etc.)
It can not apply open-minded；Historical data, operational indicator data can not be by consumption etc. in time, and what these problems will have a direct impact on platform can
Use by property and tenant perceives.Therefore, be to guarantee that message system cluster is balanced and availability, require a great deal of time and
Energy redistributes subregion, again balanced copy.
These problems reflect the data demand that message system processing capacity at this stage is unable to satisfy explosive increase.
In view of the above-mentioned problems, at present common processing technique have it is following several: the first be preset it is sufficient amount of
Message subject subregion, but this method cannot achieve partitioning balance.Second is that different types of message data is passed through difference
Theme is distinguished to meet business demand；It needs to create many themes, therefore additional management cost can be brought.And for number in real time
According to the consumption of, historical data, information consumption process can only be increased by manually to meet, quantity few time reduce information consumption into
Journey, this mode cannot achieve resource automatic dynamic extension (scaling), and resource utilization is low.
In summary, existing technology, which has the drawback that, presets each message firstly the need of to message system
Topic partition size and each Broker resource utilization are unable to real-time perfoming adjustment, be easy to cause message copy uneven
Weighing apparatus.And real-time performance data can not be obtained at the first time.
Summary of the invention
In view of the problems of the existing technology, the embodiment of the present invention provides a kind of message system resources balance device, method
And equipment.
The embodiment of the present invention provides a kind of message system resources balance device, including load monitoring engine, analysis engine with
And operation enforcement engine；The load monitors engine, for acquiring Kafka index from Kafka cluster, and by the Kafka
Index is sent to the analysis engine, wherein the Kafka index includes load and the partition information of the Kafka cluster；Institute
Analysis engine is stated, for according to the soft of the rigid target of the Kafka index, the Kafka cluster and the Kafka cluster
Target generates the first equalization scheme for corresponding to all subscribers, and first equalization scheme is sent to the operation and is held
Row engine, wherein the hardness target is the target that several described Kafka clusters must be realized, if the soft target is
The target that the dry Kafka cluster is preferably realized；The operation enforcement engine, for being generated according to first equalization scheme
Several operational orders, and by several described operational orders be sent to the Kafka cluster for the Kafka cluster according to
The operational order carries out resources balance.
The embodiment of the present invention provides a kind of message system resources balance method, comprising: load monitors engine from Kafka cluster
Middle acquisition Kafka index, and the Kafka index is sent to the analysis engine, wherein the Kafka index includes institute
State load and the partition information of Kafka cluster；The analysis engine is according to the Kafka index, the hardness of the Kafka cluster
The soft target of target and the Kafka cluster generates the first equalization scheme for corresponding to all subscribers, and by described first
Equalization scheme is sent to operation enforcement engine, wherein the hardness target is the mesh that several described Kafka clusters must be realized
Mark, the soft target are the target that several described Kafka clusters are preferably realized；The operation enforcement engine is according to described
One equalization scheme generates several operational orders, and several described operational orders are sent to the Kafka cluster for institute
It states Kafka cluster and resources balance is carried out according to the operational order.
The embodiment of the present invention provides a kind of message system resources balance equipment, comprising: at least one processor；And with institute
State at least one processor of processor communication connection, in which: the memory is stored with the journey that can be executed by the processor
Sequence instruction, the processor call described program instruction to be able to carry out above-mentioned equalization methods.
The embodiment of the present invention provides a kind of non-transient computer readable storage medium, the non-transient computer readable storage
Medium storing computer instruction, the computer instruction make the computer execute above-mentioned equalization methods.
Message system resources balance device provided in an embodiment of the present invention, method and apparatus pass through raw in analysis engine
, can be using rigid target and soft target as core foundation at the first equalization scheme, the operation of Dynamic Execution resource automatic equalization has
The utilization rate for imitating equalization server resource, promotes the performance of message system, greatly improves the High Availabitity and robustness of message system；
Improve the displaying real-time of operational indicator data and platform performance data, the true health condition and Real-time Feedback for reflecting system
Requiring as a result, meeting tenant platform and the effective of business datum for business operation, promotes user experience.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is this hair
Bright some embodiments for those of ordinary skill in the art without creative efforts, can be with root
Other attached drawings are obtained according to these attached drawings.
Fig. 1 is the Kafka message system framework figure in the embodiment of the present invention；
Fig. 2 is the module map of message system resources balance Installation practice of the present invention；
Fig. 3 is the another message system resources balance apparatus module figure in the embodiment of the present invention；
Fig. 4 is the flow chart of message system resources balance embodiment of the method for the present invention；
Fig. 5 is the block schematic illustration of the message system resources balance equipment in the embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without creative efforts, shall fall within the protection scope of the present invention.
Fig. 1 is the Kafka message system framework figure in the embodiment of the present invention, referring to FIG. 1, the embodiment of the present invention is to be based on
Kafka message system including producer Producer, subscriber Consumer and cluster, wherein cover in Kafka cluster
Several themes Topic, every Topic include several subregions Partition, include several realities in the Kafka cluster
It include several subregion copies in example Broker, every Broker.
It should be noted that several in the embodiment of the present invention refer to one or more.
Fig. 2 is the module map of message system resources balance Installation practice of the present invention, as shown in Fig. 2, including load monitoring
Engine 201, analysis engine 202 and operation enforcement engine 203；The load monitors engine 201, for from Kafka cluster
Kafka index is acquired, and the Kafka index is sent to the analysis engine 202, wherein the Kafka index includes institute
State load and the partition information of Kafka cluster；The analysis engine 202, for according to the Kafka index, the Kafka collection
The rigid target of group and the soft target of the Kafka cluster generate the first equalization scheme for corresponding to all subscribers, and will
First equalization scheme is sent to the operation enforcement engine 203, wherein the hardness target is several described Kafka
The target that cluster must be realized, the soft target are the target that several described Kafka clusters are preferably realized；The operation is held
Row engine 203, for generating several operational orders according to first equalization scheme, and will several operational orders hair
It send to the Kafka cluster so that the Kafka cluster carries out resources balance according to the operational order.
It should be noted that load monitoring engine 201 is connected with analysis engine 202, and analysis is drawn in the balancer
202 are held up to be connected with operation enforcement engine 203.
Specifically, in the analysis engine 202, the rigid target of appearance and soft target difference are as follows:
1, rigid target:
1) subregion copy configuration must be that rack is identifiable.Identical partitions copy is put on different racks, in this way when
When some rack is shut down, any subregion at most has a copy and loses.This helps to control failure boundary, and makes Kafka more
Add stalwartness.
2) resource utilization of Broker must be in predefined threshold value.
Even if 3) all subregion copies on Broker all become leader, network can not be allowed to utilize beyond fixed in advance
The capacity of justice.In this case, all subscriber's flows will be redirected to the Broker, this will lead to higher than usually
Network utilization.
2, soft target:
1) resource utilization as consistent as possible is realized between all Broker.
2) the uniform byte input rate of leader subregion is realized between Broker.
3) subregion of specific subject is evenly distributed on all Broker.
4) subregion copy is evenly distributed on all Broker.
It should be noted that there are priority between each subscriber, and balancer is in descending order of priority
Guarantee each rigid target and soft target.
Further, the workflow of the message system resources balance device of the present embodiment is as described below: load monitoring is drawn
It holds up 201 and acquires Kafka index from Kafka cluster, and the Kafka index is sent to the analysis engine 202.Analysis is drawn
202 are held up by the Kafka index and the rigid target and soft target of the Kafka cluster, generates and corresponds to all order
The first equalization scheme of reader, and first equalization scheme is sent to the operation enforcement engine 203.The operation executes
Engine 203 generates several operational orders according to first equalization scheme, and several described operational orders are sent to institute
Kafka cluster is stated so that the Kafka cluster carries out resources balance according to the operational order.
Further, the first equalization scheme generated in analysis engine 202 is to need to meet whole rigid targets.
Message system resources balance device provided in an embodiment of the present invention, by generating first in analysis engine 202
Weighing apparatus scheme, can be using rigid target and soft target as core foundation, the operation of Dynamic Execution resource automatic equalization, efficient balance clothes
The utilization rate for device resource of being engaged in, promotes the performance of message system, greatly improves the High Availabitity and robustness of message system；Raising business
The displaying real-time of achievement data and platform performance data, the true health condition and Real-time Feedback business operation for reflecting system
The effective of platform and business datum is required as a result, meeting tenant, promote user experience.
Based on the above embodiment, the information apparatus further includes abnormality detection engine, for being drawn according to load monitoring
The Kafka index for holding up 201 transmissions, detects the exception in the Kafka cluster, obtains abnormality detection result, and
The abnormality detection result is sent to the analysis engine 202；It is corresponding, it is described according to the Kafka index, it is described
The soft target of the rigid target of Kafka cluster and the Kafka cluster generates the first equilibrium side for corresponding to all subscribers
Case specifically includes: according to the Kafka index, the rigid target of the abnormality detection result and the Kafka cluster and soft
Property target, generate correspond to all subscribers the first equalization scheme.
Specifically, it will appear various exceptions in Kafka cluster, by abnormality detection engine to different in Kafka cluster
It is often detected, and will test result and be sent to the analysis engine 202.The first equilibrium side is generated in analysis engine 202 in this way
Also it can take into account the testing result when case.
It should be noted that testing result reflects the abnormal conditions in Kafka cluster, can illustrate are as follows: non-empty
XxBroker peels off.
Based on the above embodiment, the Kafka index sent according to the load monitoring engine 201, to described
Exception in Kafka cluster is detected, and is obtained abnormality detection result, is specifically included: monitoring that engine 201 is sent out according to the load
The Kafka index sent, to the non-empty Broker in the Kafka cluster peel off and the Kafka cluster in violate institute
The case where stating rigid target and the soft target is detected, and abnormality detection result is obtained.
It should be noted that the present embodiment specifically refers to, abnormal conditions include that non-empty Broker peels off and Kafka collection
The case where rigid target and the soft target are violated in group.
Non-empty Broker, which peels off, will lead to subregion number of copies deficiency.Because this may send out during normal cluster recovery
It is raw, so abnormality detection engine provides the configurable grace period in trigger notice and before repairing cluster.
Message system resources balance device provided in an embodiment of the present invention according to Broker failure and violates mesh by setting
Active is attempted to violate by automatically analyzing workload and executing the first equalization scheme to solve target by mark, solves existing skill
With the rapid growth of portfolio in art, needs to handle GB even TB data volume, Message Processing daily and significant delays, platform occur
It can not be presented to tenant in real time with business datum, it is thus possible to cause platform and operation system failure to occur, cannot generate in time
The problem of alarm.The present invention is improved the business throughput of message system using subregion automatic equalization, reduces data delay, and industry is promoted
Business system user experience.
Based on the above embodiment, described that Kafka index is acquired from Kafka cluster, and the Kafka index is sent to
The analysis engine 202, wherein the Kafka index includes load and the partition information of the Kafka cluster, is specifically included:
Kafka index is acquired from Kafka cluster, cluster work load model is generated by the Kafka index, and by the cluster
Work load model is sent to the analysis engine 202 and the abnormality detection engine；Wherein, the Kafka index includes
Kafka cluster overall load information, Kafka clustering performance achievement data, the quantity of the clustered partition Kafka, the clustered partition Kafka
Distributed intelligence and the clustered partition Kafka data balancing information；The cluster work load model includes the disk of Kafka cluster
Utilization rate, the cpu busy percentage of Kafka cluster and the byte input and output rate of Kafka cluster.
It should be noted that the present embodiment refers to, it is negative that load monitoring engine 201 is based on the generation cluster work of Kafka index
Model is carried, analysis engine 202 and abnormality detection engine are the cluster work load models for receiving load monitoring engine 201 and sending
, cluster work load model reflects the information such as load and the work of Kafka cluster.
Message system resources balance device provided in an embodiment of the present invention loads monitoring engine 201 by setting and is based on
Kafka index generates cluster work load model, and the first equalization scheme ultimately generated is enabled to accurately to be applicable in.
Based on the above embodiment, the analysis engine 202 is also used to through the Kafka index and the Kafka
The rigid target and soft target of cluster generate when the Kafka cluster violates any rigid target and correspond to all order
The second equalization scheme of reader, and second equalization scheme is sent to the operation enforcement engine 203.
It should be noted that the present embodiment is to violate the premise of rigid target in the first equalization scheme of previous execution
Under, second equalization scheme is regenerated, herein, violating rigid target is detected by abnormality detection engine.
Based on the above embodiment, the Kafka cluster carries out resources balance according to the operational order, comprising: described
Kafka cluster carries out the dilatation of the clustered partition Kafka, the clustered partition Kafka reassignment and Kafka collection according to the operational order
At least one of group's subregion copy reassignment, so that the Kafka cluster resource is balanced.
It should be noted that the instruction operation of several operational orders in the embodiment of the present invention includes Kafka cluster point
At least one of area's dilatation, the clustered partition Kafka reassign and the clustered partition Kafka copy reassigns, and are a kind of preferred
Scheme, it includes other schemes for indicating operation that the present invention, which also protects,.
Message system resources balance device provided in an embodiment of the present invention will be operated by setting operation enforcement engine 203
Instruction is sent in the Kafka cluster to carry out the dilatation of the clustered partition Kafka, the clustered partition Kafka to the Kafka cluster
At least one of reassignment and the clustered partition Kafka copy reassignment, so that resources balance, can be realized Kafka and disappear
The resource automatic equalization of breath system, enhances the High Availabitity of data, avoids loss of data, to reach highest business throughput, subtract
Minority promotes operation system user experience according to blocking time.
Based on the above embodiment, the balancer further includes data collector, and the data collector includes
Workload collector, Metrics collector and Partition collector；The Workload collector is described for acquiring
Kafka cluster overall load information；The Metrics collector is for acquiring Kafka clustering performance achievement data；It is described
Partition collector be used to acquire the quantity of the clustered partition Kafka, the clustered partition Kafka distributed intelligence and
The clustered partition the Kafka data balancing information.
It should be noted that the data collector in the present embodiment is to acquire the Kafka index proposed in above-described embodiment
's.
Further, the data collector proposed in the embodiment of the present invention includes Workload collector, Metrics collection
Device and Partition collector, are a kind of preferable cases, and the present invention also protects the scheme including other kinds of collectors.
Message system resources balance device provided in an embodiment of the present invention includes Workload by setting data collector
Collector, Metrics collector and Partition collector can acquire more accurate Kafka index, greatly improve
The each Broker resource utilization of message system.
As a preferred embodiment, Fig. 3 is the another message system resources balance apparatus module in the embodiment of the present invention
Figure, as shown in figure 3, including load monitoring engine 201, analysis engine 202 and operation enforcement engine 203；The load monitoring
The Kafka index for acquiring Kafka index from Kafka cluster, and is sent to the analysis engine by engine 201
202, wherein the Kafka index includes load and the partition information of the Kafka cluster；The analysis engine 202, is used for
According to the Kafka index, the soft target of the rigid target of the Kafka cluster and the Kafka cluster, generation corresponds to
The first equalization scheme of all subscribers, and first equalization scheme is sent to the operation enforcement engine 203, wherein
The hardness target is the target that several described Kafka clusters must be realized, the soft target is several described Kafka
The target that cluster is preferably realized；The operation enforcement engine 203, for generating several operations according to first equalization scheme
Instruction, and several described operational orders are sent to the Kafka cluster so that the Kafka cluster refers to according to the operation
It enables and carries out resources balance.
It further include abnormality detection engine, the Kafka index for being sent according to the load monitoring engine 201 is right
Exception in the Kafka cluster is detected, and abnormality detection result is obtained, and the abnormality detection result is sent to described
Analysis engine 202；It is corresponding, it is described according to the Kafka index, the rigid target of the Kafka cluster and the Kafka collection
The soft target of group generates the first equalization scheme for corresponding to all subscribers, specifically includes: according to the Kafka index, institute
It states the rigid target and soft target of abnormality detection result and the Kafka cluster, generates and correspond to the of all subscribers
One equalization scheme.
The balancer further includes data collector, and the data collector includes Workload collector, Metrics
Collector and Partition collector；The Workload collector is for acquiring the Kafka cluster overall load information；
The Metrics collector is for acquiring Kafka clustering performance achievement data；The Partition collector is for acquiring institute
State quantity, the clustered partition Kafka distributed intelligence and the clustered partition the Kafka data balancing of the clustered partition Kafka
Information.
Message system resources balance device provided in an embodiment of the present invention, by introducing data collector, load monitoring is drawn
201, abnormality detection engine, analysis engine 202 and operation enforcement engine 203 are held up, can be realized message system subregion, subregion copy
And Broker resource automatic equalization, Broker server resource utilization rate is improved, the arrangement achieves the height between each engine
Degree decoupling, meets the requirement of distributed deployment, and can precisely judge that each Broker resource makes by comprehensive collector
With situation, operation system is operated in the environment of optimum performance, promotes operation system stability and user experience.This
Invention implement in balancer make that Kafka message system is reliable, has strong processing capacity and high-throughput, meet constantly
The business demand of the cloud pipe platform of development.
Based on the above embodiment, Fig. 4 is the flow chart of message system resources balance embodiment of the method for the present invention, such as Fig. 4 institute
Show, comprising: S401, load monitoring engine 201 acquire Kafka index from Kafka cluster, and the Kafka index sent
To the analysis engine 202, wherein the Kafka index includes load and the partition information of the Kafka cluster；S402, institute
Analysis engine 202 is stated according to the soft mesh of the Kafka index, the rigid target of the Kafka cluster and the Kafka cluster
Mark generates the first equalization scheme for corresponding to all subscribers, and first equalization scheme is sent to operation enforcement engine
203, wherein the hardness target is the target that several described Kafka clusters must be realized, the soft target is several
The target that the Kafka cluster is preferably realized；S403, the operation enforcement engine 203 are generated according to first equalization scheme
Several operational orders, and by several described operational orders be sent to the Kafka cluster for the Kafka cluster according to
The operational order carries out resources balance.
Based on the above embodiment, Fig. 5 is the block schematic illustration of the message system resources balance equipment in the embodiment of the present invention.
Referring to FIG. 5, the embodiment of the present invention provides a kind of message system resources balance equipment, comprising: processor (processor) 510,
Communication interface (Communications Interface) 520, memory (memory) 530 and bus 540, wherein processor
510, communication interface 520, memory 530 completes mutual communication by bus 540.Processor 510 can call memory
Logical order in 530, to execute following method, comprising: load monitoring engine 201 acquires Kafka from Kafka cluster and refers to
Mark, and the Kafka index is sent to the analysis engine 202, wherein the Kafka index includes the Kafka cluster
Load and partition information；The analysis engine 202 is according to the Kafka index, the rigid target of the Kafka cluster and institute
It states the soft target of Kafka cluster, generates the first equalization scheme for corresponding to all subscribers, and by first equalization scheme
It is sent to operation enforcement engine 203, wherein the hardness target is the target that several described Kafka clusters must be realized, institute
Stating soft target is the target that several described Kafka clusters are preferably realized；The operation enforcement engine 203 is according to described first
Equalization scheme generates several operational orders, and several described operational orders are sent to the Kafka cluster for described
Kafka cluster carries out resources balance according to the operational order.
Based on the above embodiment, the embodiment of the present invention discloses a kind of computer program product, the computer program product
Including the computer program being stored in non-transient computer readable storage medium, the computer program includes program instruction,
When described program instruction is computer-executed, computer is able to carry out equalization methods provided by above method embodiment, example
Such as include: that load monitoring engine 201 acquire Kafka index from Kafka cluster, and described in the Kafka index is sent to
Analysis engine 202, wherein the Kafka index includes load and the partition information of the Kafka cluster；The analysis engine
202 according to the Kafka index, the soft target of the rigid target of the Kafka cluster and the Kafka cluster, generation pair
Operation enforcement engine 203 should be sent in the first equalization scheme of all subscribers, and by first equalization scheme, wherein
The hardness target is the target that several described Kafka clusters must be realized, the soft target is several described Kafka
The target that cluster is preferably realized；The operation enforcement engine 203 generates several operational orders according to first equalization scheme,
And by several described operational orders be sent to the Kafka cluster for the Kafka cluster according to the operational order into
Row resources balance.
Based on the above embodiment, the embodiment of the present invention provides a kind of non-transient computer readable storage medium, described non-temporary
State computer-readable recording medium storage computer instruction, the computer instruction make the computer execute above method implementation
Equalization methods provided by example, for example, load monitoring engine 201 acquires Kafka index from Kafka cluster, and by institute
State Kafka index and be sent to the analysis engine 202, wherein the Kafka index include the Kafka cluster load and
Partition information；The analysis engine 202 is according to the Kafka index, the rigid target of the Kafka cluster and the Kafka
The soft target of cluster generates the first equalization scheme for corresponding to all subscribers, and first equalization scheme is sent to
Operate enforcement engine 203, wherein the hardness target is the target that several described Kafka clusters must be realized, described soft
Target is the target that several described Kafka clusters are preferably realized；The operation enforcement engine 203 is according to the first equilibrium side
Case generates several operational orders, and several described operational orders are sent to the Kafka cluster for the Kafka collection
Group carries out resources balance according to the operational order.
Those of ordinary skill in the art will appreciate that: realize that above equipment embodiment or embodiment of the method are only schematic
, wherein the processor and the memory can be physically separate component may not be it is physically separated, i.e.,
It can be located in one place, or may be distributed over multiple network units.It can select according to the actual needs therein
Some or all of the modules achieves the purpose of the solution of this embodiment.Those of ordinary skill in the art are not paying creative labor
In the case where dynamic, it can understand and implement.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment can
It realizes by means of software and necessary general hardware platform, naturally it is also possible to pass through hardware.Based on this understanding, on
Stating technical solution, substantially the part that contributes to existing technology can be embodied in the form of software products in other words, should
Computer software product may be stored in a computer readable storage medium, such as USB flash disk, mobile hard disk, ROM/RAM, magnetic disk, CD
Deng, including some instructions use is so that a computer equipment (can be personal computer, server or the network equipment etc.)
Execute method described in certain parts of each embodiment or embodiment.
Message system resources balance device provided in an embodiment of the present invention, method and apparatus pass through raw in analysis engine
, can be using rigid target and soft target as core foundation at the first equalization scheme, the operation of Dynamic Execution resource automatic equalization has
The utilization rate for imitating equalization server resource, promotes the performance of message system, greatly improves the High Availabitity and robustness of message system；
Improve the displaying real-time of operational indicator data and platform performance data, the true health condition and Real-time Feedback for reflecting system
Requiring as a result, meeting tenant platform and the effective of business datum for business operation, promotes user experience.
Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；Although
Present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: it still may be used
To modify the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features；
And these are modified or replaceed, technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution spirit and
Range.
Claims (10)
1. a kind of message system resources balance device, which is characterized in that held including load monitoring engine, analysis engine and operation
Row engine；
The load monitors engine, for acquiring Kafka index from Kafka cluster, and the Kafka index is sent to institute
State analysis engine, wherein the Kafka index includes load and the partition information of the Kafka cluster；
The analysis engine, for according to the Kafka index, the Kafka cluster rigid target and the Kafka cluster
Soft target, generate the first equalization scheme for corresponding to all subscribers, and first equalization scheme is sent to described
Operate enforcement engine, wherein the hardness target is the target that several described Kafka clusters must be realized, the soft mesh
It is designated as the target that several described Kafka clusters are preferably realized；
The operation enforcement engine, for generating several operational orders according to first equalization scheme, and will be described several
A operational order is sent to the Kafka cluster so that the Kafka cluster carries out resources balance according to the operational order.
2. balancer according to claim 1, which is characterized in that further include abnormality detection engine, for according to
The Kafka index that load monitoring engine is sent, detects the exception in the Kafka cluster, obtains abnormality detection
As a result, and the abnormality detection result is sent to the analysis engine；
It is corresponding, it is described according to the soft of the Kafka index, the rigid target of the Kafka cluster and the Kafka cluster
Target generates the first equalization scheme for corresponding to all subscribers, specifically includes:
According to the rigid target and soft target of the Kafka index, the abnormality detection result and the Kafka cluster,
Generate the first equalization scheme for corresponding to all subscribers.
3. balancer according to claim 2, which is characterized in that the institute sent according to the load monitoring engine
Kafka index is stated, the exception in the Kafka cluster is detected, abnormality detection result is obtained, specifically includes:
According to it is described load monitoring engine send the Kafka index, to the non-empty Broker in the Kafka cluster from
The case where violating the rigid target and the soft target in group and the Kafka cluster is detected, and abnormal inspection is obtained
Survey result.
4. balancer according to claim 2, which is characterized in that it is described that Kafka index is acquired from Kafka cluster,
And the Kafka index is sent to the analysis engine, wherein the Kafka index includes the load of the Kafka cluster
And partition information, it specifically includes:
Kafka index is acquired from Kafka cluster, cluster work load model is generated by the Kafka index, and will be described
Cluster work load model is sent to the analysis engine and the abnormality detection engine；
Wherein, the Kafka index includes Kafka cluster overall load information, Kafka clustering performance achievement data, Kafka collection
Quantity, the distributed intelligence of the clustered partition Kafka and the clustered partition the Kafka data balancing information of group's subregion；The cluster work
Load module includes the disk utilization of Kafka cluster, the byte input of the cpu busy percentage and Kafka cluster of Kafka cluster
And output speed.
5. balancer according to claim 1, which is characterized in that the analysis engine is also used to through the Kafka
The rigid target and soft target of index and the Kafka cluster violate any rigid target in the Kafka cluster
When, the second equalization scheme for corresponding to all subscribers is generated, and second equalization scheme is sent to the operation and is executed
Engine.
6. balancer according to claim 1, which is characterized in that the Kafka cluster according to the operational order into
Row resources balance, comprising:
The Kafka cluster according to the operational order carry out the dilatation of the clustered partition Kafka, the clustered partition Kafka reassign with
And at least one of the clustered partition Kafka copy reassignment, so that the Kafka cluster resource is balanced.
7. balancer according to claim 4, which is characterized in that the balancer further includes data collector, institute
Stating data collector includes Workload collector, Metrics collector and Partition collector；
The Workload collector is for acquiring the Kafka cluster overall load information；The Metrics collector is used for
Acquire Kafka clustering performance achievement data；The Partition collector be used for acquire the clustered partition Kafka quantity,
The clustered partition Kafka distributed intelligence and the clustered partition the Kafka data balancing information.
8. a kind of message system resources balance method characterized by comprising
Load monitoring engine acquires Kafka index from Kafka cluster, and the Kafka index is sent to the analysis and is drawn
It holds up, wherein the Kafka index includes load and the partition information of the Kafka cluster；
The analysis engine is soft according to the Kafka index, the rigid target of the Kafka cluster and the Kafka cluster
Property target, generate correspond to all subscribers the first equalization scheme, and by first equalization scheme be sent to operation execute
Engine, wherein the hardness target is the target that several described Kafka clusters must be realized, the soft target is several
The target that a Kafka cluster is preferably realized；
The operation enforcement engine generates several operational orders according to first equalization scheme, and will several described operations
Instruction is sent to the Kafka cluster so that the Kafka cluster carries out resources balance according to the operational order.
9. a kind of message system resources balance equipment characterized by comprising
At least one processor；And
At least one processor being connect with the processor communication, in which:
The memory is stored with the program instruction that can be executed by the processor, and the processor calls described program to instruct energy
Enough execute equalization methods as claimed in claim 8.
10. a kind of non-transient computer readable storage medium, which is characterized in that the non-transient computer readable storage medium is deposited
Computer instruction is stored up, the computer instruction makes the computer execute equalization methods as claimed in claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810463407.6A CN110493071B (en) | 2018-05-15 | 2018-05-15 | Message system resource balancing device, method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810463407.6A CN110493071B (en) | 2018-05-15 | 2018-05-15 | Message system resource balancing device, method and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110493071A true CN110493071A (en) | 2019-11-22 |
| CN110493071B CN110493071B (en) | 2021-06-04 |
Family
ID=68544917
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810463407.6A Active CN110493071B (en) | 2018-05-15 | 2018-05-15 | Message system resource balancing device, method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110493071B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269661A (en) * | 2020-10-30 | 2021-01-26 | 上海哔哩哔哩科技有限公司 | Partition migration method and device based on Kafka cluster |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101414377A (en) * | 2008-10-22 | 2009-04-22 | 中国移动通信集团福建有限公司 | Method for filtrating people's livelihood information for communication vocation based on data storehouse |
| CN104794015A (en) * | 2015-04-16 | 2015-07-22 | 华中科技大学 | Real-time streaming computing flow speed perceiving elastic execution tolerant system |
| CN104836819A (en) * | 2014-02-10 | 2015-08-12 | 阿里巴巴集团控股有限公司 | Dynamic load balancing method and system, and monitoring and dispatching device |
| CN105933163A (en) * | 2016-06-24 | 2016-09-07 | 微梦创科网络科技（中国）有限公司 | Real-time distributed debugging tracking method and system |
| CN106130960A (en) * | 2016-06-12 | 2016-11-16 | 微梦创科网络科技（中国）有限公司 | Judgement system, load dispatching method and the device of steal-number behavior |
| CN106790572A (en) * | 2016-12-27 | 2017-05-31 | 广州华多网络科技有限公司 | The system and method that a kind of distributed information log is collected |
| CN106815338A (en) * | 2016-12-25 | 2017-06-09 | 北京中海投资管理有限公司 | A kind of real-time storage of big data, treatment and inquiry system |
| CN106850759A (en) * | 2016-12-31 | 2017-06-13 | 广州勤加缘科技实业有限公司 | MySQL database clustering methods and its processing system |
| CN107038071A (en) * | 2017-05-26 | 2017-08-11 | 重庆邮电大学 | A kind of flexible dispatching algorithm of Storm tasks predicted based on data flow |
| CN107103064A (en) * | 2017-04-17 | 2017-08-29 | 北京五八信息技术有限公司 | Data statistical approach and device |
| CN107145393A (en) * | 2017-04-27 | 2017-09-08 | 努比亚技术有限公司 | A kind of load regulation method, equipment and computer-readable recording medium |
| US20170272516A1 (en) * | 2016-03-17 | 2017-09-21 | International Business Machines Corporation | Providing queueing in a log streaming messaging system |
| US20180032582A1 (en) * | 2016-07-28 | 2018-02-01 | International Business Machines Corporation | Cross object synchronization |
| CN107734052A (en) * | 2017-11-02 | 2018-02-23 | 华南理工大学 | The load balancing container dispatching method that facing assembly relies on |
| US20180060143A1 (en) * | 2016-08-26 | 2018-03-01 | Vmware, Inc. | Distributed shared log storage system having an adapter for heterogenous big data workloads |
| KR101834837B1 (en) * | 2016-11-14 | 2018-03-06 | 충북대학교 산학협력단 | System of IoT sensor simulator using MQTT and KAFKA |
-
2018
- 2018-05-15 CN CN201810463407.6A patent/CN110493071B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101414377A (en) * | 2008-10-22 | 2009-04-22 | 中国移动通信集团福建有限公司 | Method for filtrating people's livelihood information for communication vocation based on data storehouse |
| CN104836819A (en) * | 2014-02-10 | 2015-08-12 | 阿里巴巴集团控股有限公司 | Dynamic load balancing method and system, and monitoring and dispatching device |
| CN104794015A (en) * | 2015-04-16 | 2015-07-22 | 华中科技大学 | Real-time streaming computing flow speed perceiving elastic execution tolerant system |
| US20170272516A1 (en) * | 2016-03-17 | 2017-09-21 | International Business Machines Corporation | Providing queueing in a log streaming messaging system |
| CN106130960A (en) * | 2016-06-12 | 2016-11-16 | 微梦创科网络科技（中国）有限公司 | Judgement system, load dispatching method and the device of steal-number behavior |
| CN105933163A (en) * | 2016-06-24 | 2016-09-07 | 微梦创科网络科技（中国）有限公司 | Real-time distributed debugging tracking method and system |
| US20180032582A1 (en) * | 2016-07-28 | 2018-02-01 | International Business Machines Corporation | Cross object synchronization |
| US20180060143A1 (en) * | 2016-08-26 | 2018-03-01 | Vmware, Inc. | Distributed shared log storage system having an adapter for heterogenous big data workloads |
| KR101834837B1 (en) * | 2016-11-14 | 2018-03-06 | 충북대학교 산학협력단 | System of IoT sensor simulator using MQTT and KAFKA |
| CN106815338A (en) * | 2016-12-25 | 2017-06-09 | 北京中海投资管理有限公司 | A kind of real-time storage of big data, treatment and inquiry system |
| CN106790572A (en) * | 2016-12-27 | 2017-05-31 | 广州华多网络科技有限公司 | The system and method that a kind of distributed information log is collected |
| CN106850759A (en) * | 2016-12-31 | 2017-06-13 | 广州勤加缘科技实业有限公司 | MySQL database clustering methods and its processing system |
| CN107103064A (en) * | 2017-04-17 | 2017-08-29 | 北京五八信息技术有限公司 | Data statistical approach and device |
| CN107145393A (en) * | 2017-04-27 | 2017-09-08 | 努比亚技术有限公司 | A kind of load regulation method, equipment and computer-readable recording medium |
| CN107038071A (en) * | 2017-05-26 | 2017-08-11 | 重庆邮电大学 | A kind of flexible dispatching algorithm of Storm tasks predicted based on data flow |
| CN107734052A (en) * | 2017-11-02 | 2018-02-23 | 华南理工大学 | The load balancing container dispatching method that facing assembly relies on |
Non-Patent Citations (4)
| Title |
|---|
| CAO NGOC NGUYEN: "Building a Kafka-Based Distributed Queue System on the Fly in a Hadoop Cluster", 《2016 IEEE 1ST INTERNATIONAL WORKSHOPS ON FOUNDATIONS AND APPLICATIONS OF SELF* SYSTEMS (FAS*W)》 * |
| 王志泳: "分布式消息系统Kafka的性能建模与优化技术研究与实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
| 王郑合: "一种优化的Kafka消费者_客户端负载均衡算法", 《计算机应用研究》 * |
| 马浩然: "基于NS3的分布式消息系统Kafka的仿真实现", 《软件》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112269661A (en) * | 2020-10-30 | 2021-01-26 | 上海哔哩哔哩科技有限公司 | Partition migration method and device based on Kafka cluster |
| CN112269661B (en) * | 2020-10-30 | 2022-04-15 | 上海哔哩哔哩科技有限公司 | Partition migration method and device based on Kafka cluster |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110493071B (en) | 2021-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107689953B (en) | Multi-tenant cloud computing-oriented container security monitoring method and system | |
| US20200117508A1 (en) | Determining an allocation of computing resources for a job | |
| CN111176852A (en) | Resource allocation method, device, chip and computer readable storage medium | |
| US20180198855A1 (en) | Method and apparatus for scheduling calculation tasks among clusters | |
| CN105049268A (en) | Distributed computing resource allocation system and task processing method | |
| CN103399781B (en) | Cloud Server and virtual machine management method thereof | |
| CN109871328B (en) | Software testing method and device | |
| CN109254836B (en) | Deadline constraint cost optimization scheduling method for priority dependent tasks of cloud computing system | |
| CN104793996A (en) | Task scheduling method and device of parallel computing equipment | |
| CN105791166B (en) | A kind of method and system of load balancing distribution | |
| CN109284229A (en) | A kind of dynamic adjusting method and relevant device based on QPS | |
| CN113255165A (en) | Experimental scheme parallel deduction system based on dynamic task allocation | |
| CN115658311A (en) | Resource scheduling method, device, equipment and medium | |
| CN106250202A (en) | A kind of cloud platform automatization opens the device and method of virtual machine | |
| CN109542874A (en) | Deriving method, guiding device, server, terminal and readable storage medium storing program for executing | |
| CN106886477B (en) | Method and device for setting monitoring threshold in cloud system | |
| CN110493071A (en) | Message system resources balance device, method and apparatus | |
| CN112732408A (en) | Method for computing node resource optimization | |
| CN109788061B (en) | Computing task deployment method and device | |
| CN105677470B (en) | Virtual machine method for scheduling task, virtual machine system | |
| CN110868330B (en) | Evaluation method, device and evaluation system for CPU resources which can be divided by cloud platform | |
| CN107273527A (en) | A kind of Hadoop clusters and distributed system | |
| CN107277143A (en) | A kind of resource matched management method and device | |
| CN115952054A (en) | Simulation task resource management method, device, equipment and medium | |
| CN114237902A (en) | Service deployment method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231218 Address after: No.19, Jiefang East Road, Hangzhou, Zhejiang Province, 310000 Patentee after: CHINA MOBILE GROUP ZHEJIANG Co.,Ltd. Patentee after: China Mobile (Zhejiang) Innovation Research Institute Co.,Ltd. Patentee after: CHINA MOBILE COMMUNICATIONS GROUP Co.,Ltd. Address before: No. 19, Jiefang East Road, Hangzhou, Zhejiang Province, 310016 Patentee before: CHINA MOBILE GROUP ZHEJIANG Co.,Ltd. Patentee before: CHINA MOBILE COMMUNICATIONS GROUP Co.,Ltd. |