CN108134814A - A kind of business data processing method and device - Google Patents
A kind of business data processing method and device Download PDFInfo
- Publication number
- CN108134814A CN108134814A CN201711205491.3A CN201711205491A CN108134814A CN 108134814 A CN108134814 A CN 108134814A CN 201711205491 A CN201711205491 A CN 201711205491A CN 108134814 A CN108134814 A CN 108134814A
- Authority
- CN
- China
- Prior art keywords
- processing unit
- service processing
- message
- threads
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
- H04L67/1008—Server selection for load balancing based on parameters of servers, e.g. available memory or workload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1031—Controlling of the operation of servers by a load balancer, e.g. adding or removing servers that serve requests
Abstract
The present invention proposes a kind of business data processing method and device, and this method includes：For any Service Processing Unit at consumer end, detect the message number subscribed in message queue corresponding with Service Processing Unit in agent side, wherein, the corresponding one or more subscription message queues of each described Service Processing Unit, the Service Processing Unit are used to handle the message in the subscription message queue；Judge whether to need the traffic handing capacity for adjusting Service Processing Unit according at least to the message number detected, if so, the traffic handing capacity of adjustment Service Processing Unit.The present invention overcomes distributed posts of the prior art to subscribe to the defects of message squeezes caused by message system.
Description
Technical field
The present invention relates to data processing field more particularly to a kind of business data processing methods and device.
Background technology
Kafka is distributed post-subscription message system.Kafka be one it is distributed, can divide, redundancy backup
Persistent log services.It is mainly for the treatment of active stream data.
In big data system, a problem can be usually encountered, entire big data is made of subsystems, and data need
The problem of high-performance, not arrheaing for low latency turning in subsystems, therefore networking complexity and complicated netting can be brought.
And Kafka can reduce system group network complexity, reduce programming complexity, subsystems are no longer to reciprocally negotiate interface, respectively
A subsystem can be inserted on socket similar to socket and equally be inserted on Kafka, and the effect of high speed data bus is undertaken by Kafka.
As shown in Figure 1, the overall architecture of Kafka is very simple, it is an explicit distributed structure/architecture, the following is in Kafka
Several basic conceptions, Topic：Refer in particular to the different classifications of the message source (feeds of messages) of Kafka processing.
Partition：The groupings of Topic physically, a Topic can be divided into multiple Partition, and each Partition is one
A orderly queue.Every message in Partition can all be assigned an orderly id.Message：Message is communication
Base unit, each Producer can issue some message to a Topic.Producers：Message and data production
Person gives out information to a Topic of Kafka.Consumers：Message and data consumer subscribe to Topics and handle its hair
The message of cloth.Broker：Caching agent, one or more server in Kafka clusters are referred to as Broker.
In Kafka, producer end Producer, agent side Broker and consumer end Consumer can have multiple.
Producer, Consumer are the interfaces for realizing Kafka registrations, and data are sent to Broker from Producer, and Broker undertakes
One intermediate buffer and the effect of distribution.Broker is registered to the Consumer in system for distribution.The effect class of Broker
It is similar to cache, i.e., the caching between active data and off-line processing system.
Kafka main features：Simultaneously high-throughput is provided for publication and subscription.It can be produced about it is reported that Kafka is per second
250000 message (memory capacity for accounting for 50MB), 550,000 message of processing per second (memory capacity for accounting for 110MB)；It can carry out persistence behaviour
Make.By message duration to hard disk, therefore available for consuming in batches；Distributed system is easy to extend to the outside.All
Producer, Broker and Consumer are distributed.It can expanding machinery without shutting down；The processed state of message is
It safeguards at Consumer ends rather than is safeguarded by server end, it can autobalance when failure；Support online and offline
Scene.
In comprising some system architectures including Kafka frameworks, the message subscribing consumption mode of existing Consumers
Progress Message Processing in specific business scenario is all coupling in, the general high concurrent processing frame of neither one comes by simple
Configuration file the server node quantity of professional ability can be realized extend automatically, therefore Message Processing can be caused to overstock, take
Being engaged in, device resource consumption is excessively high, and Message Processing is not prompt enough.
Invention content
The technical problem to be solved by the present invention is to provide a kind of business data processing method and device, overcome the prior art
In the message of distributed post-caused by subscription message system the defects of squeezing.
The technical solution adopted by the present invention is business data processing method, including：
For any Service Processing Unit at consumer end, detect subscription corresponding with Service Processing Unit in agent side and disappear
The message number in queue is ceased, wherein, each Service Processing Unit is corresponding one or more to subscribe to message queues, at business
Unit is managed for handling the message subscribed in message queue；
Judge whether to need the traffic handing capacity for adjusting Service Processing Unit according at least to the message number detected, if
It is then to adjust the traffic handing capacity of Service Processing Unit.
Further, the message number subscribed in message queue corresponding with Service Processing Unit, packet in agent side are detected
It includes：
Disappearing in message queue is subscribed to according to corresponding with Service Processing Unit in the detection cycle detection agent side of setting
Cease quantity；
Message number according to detecting judges whether the traffic handing capacity for needing to adjust Service Processing Unit, including：
In the first setting duration, if the message number detected is persistently more than or works as continuously less than Service Processing Unit
Number of threads used in preceding, then judgement need to adjust the traffic handing capacity of Service Processing Unit；First setting duration is more than
The detection cycle of setting.
Further, method further includes：In agent side is detected in subscription message queue corresponding with Service Processing Unit
Message number before, the loading configuration file in local memory；Service Processing Unit is set in configuration file and uses thread
Quantitative range；
The traffic handing capacity of Service Processing Unit is adjusted, including：
If the message number detected is persistently more than the currently used number of threads of Service Processing Unit, activate business place
The number of threads that uses of reason unit, until number of threads that Service Processing Unit uses is equal to message number, wherein, business
The upper limit for the number of threads that processing unit uses uses the maximum value of the quantitative range of thread for Service Processing Unit；
If the message number detected is shunk continuously less than the currently used number of threads of Service Processing Unit at business
The number of threads that uses of reason unit, until number of threads that Service Processing Unit uses is equal to message number, wherein, business
The lower limit for the number of threads that processing unit uses uses the minimum value of the quantitative range of thread for Service Processing Unit.
Further, if when Service Processing Unit uses a server in the second setting duration in processing message number
M is limited to, M is positive integer；It is positive integer that the number of servers upper limit N, N that Service Processing Unit uses are set in configuration file；
The traffic handing capacity of Service Processing Unit is adjusted, is further included：
If the message number increment detected in the first setting duration is more than or equal to n M, n is positive integer, then with N
For the upper limit, n+1 platform servers are used for Service Processing Unit extension；
If the message number decrement detected in the first setting duration is more than or equal to n M, then, using N as the upper limit, be
Service Processing Unit reduction uses n platform servers.
Further, method further includes：The load of server that detection Service Processing Unit uses；
Whether the load judgment of server that message number and Service Processing Unit according to detecting use needs to adjust
The traffic handing capacity of whole Service Processing Unit, if so, the traffic handing capacity of adjustment Service Processing Unit.
Further, the load judgment of server used according to the message number and Service Processing Unit that detect is
The no traffic handing capacity for needing to adjust Service Processing Unit, including：
Step A1：The server that detection cycle detection message number and Service Processing Unit according to setting use is born
It carries, if the message number detected in the first setting duration is persistently more than the currently used number of threads of Service Processing Unit
Or the load of at least one server that the Service Processing Unit detected in the first setting duration uses is more than setting
High load threshold then performs step A2；If the message number detected in the first setting duration is continuously less than business processing list
At least one clothes that the currently used number of threads of member and the Service Processing Unit detected in the first setting duration use
The load continuous of business device is less than the low-load threshold value of setting, then performs step A3；If the testing result in the first setting duration
Afore-mentioned is not met, then repeats step A1；
Step A2：It activates business the traffic handing capacity of processing unit, jump procedure A1；
Step A3：Shrink the traffic handing capacity of Service Processing Unit, jump procedure A1.
The present invention also provides a kind of service data processing apparatus, including：
Detection module, for being directed to any Service Processing Unit at consumer end, detect in agent side with business processing list
The corresponding message number subscribed in message queue of member, wherein, the corresponding one or more subscription of each Service Processing Unit
Message queue, Service Processing Unit subscribe to the message in message queue for processing；
Judgment module, for judging whether the industry for needing to adjust Service Processing Unit according at least to the message number detected
Business processing capacity, if so, calling adjustment module；
Module is adjusted, for adjusting the traffic handing capacity of Service Processing Unit.
Further, detection module is used for：According to setting detection cycle detection agent side in Service Processing Unit pair
That answers subscribes to the message number in message queue；
Judgment module is used for：First setting duration in, if the message number detected be persistently more than or continuously less than
The currently used number of threads of Service Processing Unit, then judgement need to adjust the traffic handing capacity of Service Processing Unit；The
One setting duration is more than the detection cycle of setting.
Further, device further includes：
Configuration module, in subscription message queue corresponding with Service Processing Unit in detection module detection agent side
Message number before, the loading configuration file in local memory；Service Processing Unit is set in configuration file and uses thread
Quantitative range；
Module is adjusted, is used for：
If the message number detected is persistently more than the currently used number of threads of Service Processing Unit, activate business place
The number of threads that uses of reason unit, until number of threads that Service Processing Unit uses is equal to message number, wherein, business
The upper limit for the number of threads that processing unit uses uses the maximum value of the quantitative range of thread for Service Processing Unit；
If the message number detected is shunk continuously less than the currently used number of threads of Service Processing Unit at business
The number of threads that uses of reason unit, until number of threads that Service Processing Unit uses is equal to message number, wherein, business
The lower limit for the number of threads that processing unit uses uses the minimum value of the quantitative range of thread for Service Processing Unit.
Further, if when Service Processing Unit uses a server in the second setting duration in processing message number
M is limited to, M is positive integer；It is positive integer that the number of servers upper limit N, N that Service Processing Unit uses are set in configuration file；
Module is adjusted, is additionally operable to：
If the message number increment detected in the first setting duration is more than or equal to n M, n is positive integer, then with N
For the upper limit, n+1 platform servers are used for Service Processing Unit extension；
If the message number decrement detected in the first setting duration is more than or equal to n M, then, using N as the upper limit, be
Service Processing Unit reduction uses n platform servers.
Further, detection module is additionally operable to：The load of server that detection Service Processing Unit uses；
Judgment module, the load of server for being used according to the message number and Service Processing Unit that detect are sentenced
The disconnected traffic handing capacity for whether needing to adjust Service Processing Unit.
Further, detection module, for detecting message number and Service Processing Unit according to the detection cycle of setting
The load of the server used；
Judgment module, if the message number for being detected in the first setting duration is persistently worked as more than Service Processing Unit
At least one service that number of threads used in preceding or the Service Processing Unit detected in the first setting duration use
More than the high load threshold of setting, then judgement needs to activate business the traffic handing capacity of processing unit, continues to adjust for the load of device
Use detection module；
If the message number detected in the first setting duration is continuously less than the currently used line of Service Processing Unit
The load continuous of at least one server that number of passes amount and the Service Processing Unit detected in the first setting duration use is small
In the low-load threshold value of setting, then judgement needs to shrink the traffic handing capacity of Service Processing Unit, continues to call detection module；
If the testing result in the first setting duration does not meet afore-mentioned, continue to call detection module.
Using above-mentioned technical proposal, the present invention at least has following advantages：
Business data processing method and device of the present invention, by the load to server used in Service Processing Unit with
And the corresponding message queue current message quantity of subscribing to of Service Processing Unit in distributed post-subscription message system is supervised
It surveys, extension or contraction and the expansion of lateral server node to the longitudinal thread of traffic handing capacity progress of Service Processing Unit
Exhibition is shunk.Mode in this way can make business processing more flexible, reduce server automatically when the business free time
Node, automatic expansion service device node, so as to fulfill the automatic telescopic of traffic handing capacity, avoids when heavy traffic
Distributed post-message caused by subscription message system squeezes.
Description of the drawings
Fig. 1 is the overall architecture schematic diagram of Kafka；
Fig. 2 is the business data processing method flow chart of first embodiment of the invention；
Fig. 3 is the business data processing method flow chart of second and third embodiment of the invention；
Fig. 4 is the business data processing method flow chart of fourth embodiment of the invention；
Fig. 5 is that the service data processing apparatus of fifth embodiment of the invention forms structure diagram；
Fig. 6 is the service data processing apparatus composition structure diagram of the present invention the six, the seven, eight embodiments；
Fig. 7 is that the service data processing apparatus of ninth embodiment of the invention forms structure diagram；
Fig. 8 is the management flow chart of the server of ninth embodiment of the invention.
Specific embodiment
The technological means and effect taken further to illustrate the present invention to reach predetermined purpose, below in conjunction with attached drawing
And preferred embodiment, the present invention is described in detail as after.
First embodiment of the invention, a kind of business data processing method, as shown in Fig. 2, including step in detail below：
Step S101, for any Service Processing Unit at consumer end, detect in agent side with Service Processing Unit pair
That answers subscribes to the message number in message queue；The per thread that Service Processing Unit uses can handle a piece of news, processing
The thread resources can discharge after complete.
Optionally, it in step S101, detects in agent side in subscription message queue corresponding with Service Processing Unit
Message number, including：
Disappearing in message queue is subscribed to according to corresponding with Service Processing Unit in the detection cycle detection agent side of setting
Cease quantity；
Step S102 judges whether according at least to the message number detected at the business for needing to adjust Service Processing Unit
Reason ability, if so, the traffic handing capacity of adjustment Service Processing Unit.
Optionally, in step s 102, judge whether to need to adjust Service Processing Unit according to the message number detected
Traffic handing capacity, including：
In the first setting duration, if the message number detected is persistently more than or works as continuously less than Service Processing Unit
Number of threads used in preceding, then judgement need to adjust the traffic handing capacity of Service Processing Unit；First setting duration is more than
The detection cycle of setting.
Optionally, the traffic handing capacity for adjusting Service Processing Unit includes：Extension or contraction Service Processing Unit are made
The quantity of thread and/or server.And first consider extension as possible or shrink thread used in Service Processing Unit, when logical
Thread used in crossing extension or contraction Service Processing Unit still cannot be close to the business processing energy needed for Service Processing Unit
In the case of power, consider further that extension or shrink number of servers used in Service Processing Unit.
The business data processing method of the embodiment of the present invention, by the business in distributed post-subscription message system
The corresponding message queue current message quantity of subscribing to of processing unit is monitored, to the traffic handing capacity of Service Processing Unit into
Row extension is shunk.Mode in this way can make business processing more flexible, reduce clothes automatically when the business free time
Business device node, automatic expansion service device node, so as to fulfill the automatic telescopic of traffic handing capacity, keeps away when heavy traffic
Exempt from distributed post-message caused by subscription message system to squeeze.
Second embodiment of the invention, a kind of business data processing method, as shown in figure 3, including step in detail below：
Step S201, the loading configuration file in local memory；Service Processing Unit is set in configuration file and uses line
The quantitative range of journey；
Step S202, for any Service Processing Unit at consumer end, detect in agent side with Service Processing Unit pair
That answers subscribes to the message number in message queue；
Optionally, it in step S202, detects in agent side in subscription message queue corresponding with Service Processing Unit
Message number, including：
Disappearing in message queue is subscribed to according to corresponding with Service Processing Unit in the detection cycle detection agent side of setting
Cease quantity.
Step S203 judges whether to need the business processing energy for adjusting Service Processing Unit according to the message number detected
Power, if so, the traffic handing capacity of adjustment Service Processing Unit.
Optionally, in step S203, judge whether to need to adjust Service Processing Unit according to the message number detected
Traffic handing capacity, including：
In the first setting duration, if the message number detected is persistently more than or works as continuously less than Service Processing Unit
Number of threads used in preceding, then judgement need to adjust the traffic handing capacity of Service Processing Unit；First setting duration is more than
The detection cycle of setting.
Why the embodiment of the present invention is not detecting that message number is currently made more than or less than Service Processing Unit
During number of threads just at once judgement need adjust Service Processing Unit traffic handing capacity, be because agent side in industry
The corresponding message number subscribed in message queue of processing unit be engaged in it is possible that not fluctuation quietly, if immediately following fluctuate into
The adjustment of row professional ability, adjustment excessively frequently, the resource consumption of Service Processing Unit will be increased.
Optionally, in step S203, the traffic handing capacity of Service Processing Unit is adjusted, including：
If the message number detected is persistently more than the currently used number of threads of Service Processing Unit, with quantity model
The maximum value enclosed be the upper limit, the number of threads that processing unit of activating business uses, until message number be equal to Service Processing Unit
Until the number of threads used；
If the message number detected is continuously less than the currently used number of threads of Service Processing Unit, with quantity model
The minimum value enclosed is lower limit, shrinks the number of threads that Service Processing Unit uses, until message number is equal to Service Processing Unit
Until the number of threads used.
Such as：The detection of the present embodiment and adjustment process can be according to following circulate operations：
Step A1：According to subscription message queue corresponding with Service Processing Unit in the detection cycle detection agent side of setting
In message number, if to be persistently more than Service Processing Unit currently used for the message number that detects in the first setting duration
Number of threads, then perform step A2；If the message number detected in the first setting duration is continuously less than business processing list
The currently used number of threads of member, then perform step A3；If the testing result in the first setting duration does not meet aforementioned feelings
Condition then repeats step A1；
Step A2：Using the maximum value of quantitative range as the upper limit, number of threads that processing unit of activating business uses redirects step
Rapid A1；
Step A3：Using the minimum value of quantitative range as lower limit, the number of threads that Service Processing Unit uses is shunk, redirects step
Rapid A1.
The embodiment of the present invention is roughly the same with first embodiment, and difference lies in an embodiment of the present invention provides by simple
The mode configuration service processing unit of configuration file use the quantitative range of thread, which can also configure for business
The detection of processing unit and professional ability adjustable strategies, are embodied in the concrete operations of step S202 and step S203.Separately
Outside, the embodiment of the present invention has been also provided to more detailed detection adjustment process.
The business data processing method of the embodiment of the present invention, by the business in distributed post-subscription message system
The corresponding message queue current message quantity of subscribing to of processing unit is monitored, to the traffic handing capacity of Service Processing Unit into
The extension or contraction of row longitudinal direction thread.Mode in this way can make business processing more flexible, when the business free time
Automatic reduction server node, automatic expansion service device node when heavy traffic, so as to fulfill traffic handing capacity
Automatic telescopic avoids distributed post-message caused by subscription message system and squeezes.
Third embodiment of the invention, a kind of business data processing method, as shown in figure 3, including step in detail below：
Step S201, the loading configuration file in local memory；Service Processing Unit is set in configuration file and uses line
The quantitative range of journey；
Step S202, for any Service Processing Unit at consumer end, detect in agent side with Service Processing Unit pair
That answers subscribes to the message number in message queue；
Optionally, it in step S202, detects in agent side in subscription message queue corresponding with Service Processing Unit
Message number, including：
Disappearing in message queue is subscribed to according to corresponding with Service Processing Unit in the detection cycle detection agent side of setting
Cease quantity.
Step S203 judges whether to need the business processing energy for adjusting Service Processing Unit according to the message number detected
Power, if so, the traffic handing capacity of adjustment Service Processing Unit.
Optionally, in step S203, judge whether to need to adjust Service Processing Unit according to the message number detected
Traffic handing capacity, including：
In the first setting duration, if the message number detected is persistently more than or works as continuously less than Service Processing Unit
Number of threads used in preceding, then judgement need to adjust the traffic handing capacity of Service Processing Unit；First setting duration is more than
The detection cycle of setting.
Optionally, in step S203, the traffic handing capacity of Service Processing Unit is adjusted, including：
If the message number detected is persistently more than the currently used number of threads of Service Processing Unit, with quantity model
The maximum value enclosed be the upper limit, the number of threads that processing unit of activating business uses, until message number be equal to Service Processing Unit
Until the number of threads used；
If the message number detected is continuously less than the currently used number of threads of Service Processing Unit, with quantity model
The minimum value enclosed is lower limit, shrinks the number of threads that Service Processing Unit uses, until message number is equal to Service Processing Unit
Until the number of threads used.
Such as：The detection of the present embodiment and adjustment process can be according to following circulate operations：
Step A1：According to subscription message queue corresponding with Service Processing Unit in the detection cycle detection agent side of setting
In message number, if to be persistently more than Service Processing Unit currently used for the message number that detects in the first setting duration
Number of threads, then perform step A2；If the message number detected in the first setting duration is continuously less than business processing list
The currently used number of threads of member, then perform step A3；If the testing result in the first setting duration does not meet aforementioned feelings
Condition then repeats step A1；
Step A2：Using the maximum value of quantitative range as the upper limit, number of threads that processing unit of activating business uses redirects step
Rapid A1；
Step A3：Using the minimum value of quantitative range as lower limit, the number of threads that Service Processing Unit uses is shunk, redirects step
Rapid A1.
Optionally, if Service Processing Unit uses handles the message number upper limit during server in the second setting duration
For M, M is positive integer；It is positive integer that the number of servers upper limit N, N that Service Processing Unit uses are set in configuration file；
In step S203, the traffic handing capacity of Service Processing Unit is adjusted, is further included：
If the message number increment detected in the first setting duration is more than or equal to n M, n is positive integer, then with N
For the upper limit, n+1 platform servers are used for Service Processing Unit extension；
If the message number decrement detected in the first setting duration is more than or equal to n M, then, using N as the upper limit, be
Service Processing Unit reduction uses n platform servers.
The embodiment of the present invention is roughly the same with second embodiment, and difference lies in the embodiment of the present invention is in adjustment business processing
Extension or receipts for server node used in Service Processing Unit are also provided during the traffic handing capacity of unit
Contracting, when the thread that a server can be provided has been not enough to the operation of one Service Processing Unit of support, it is necessary to expand
The server number that the Service Processing Unit uses is opened up, it is on the contrary, it is necessary to reduce.Such as：One Service Processing Unit uses
It is 100,000 to handle the message number upper limit in 3 minutes during one server, it is assumed that the current Service Processing Unit only used
One server, and the message number in currently detected agent side in subscription message queue corresponding with Service Processing Unit is
At 260000, it is necessary to increase two-server and be used for the Service Processing Unit.
The business data processing method of the embodiment of the present invention passes through the load to server used in Service Processing Unit
And the corresponding message queue current message quantity of subscribing to of Service Processing Unit in distributed post-subscription message system carries out
It monitors, the extension or contraction and lateral server node to the longitudinal thread of traffic handing capacity progress of Service Processing Unit
Extension is shunk.Mode in this way can make business processing more flexible, reduce service automatically when the business free time
Device node, automatic expansion service device node, so as to fulfill the automatic telescopic of traffic handing capacity, avoids when heavy traffic
Distributed post-message caused by subscription message system squeezes.
Fourth embodiment of the invention, a kind of business data processing method, as shown in figure 4, including step in detail below：
Step S301, the loading configuration file in local memory；Service Processing Unit is set in configuration file and uses line
The number of servers upper limit N, N that the quantitative range and Service Processing Unit of journey use are positive integer；
Step S302 for any Service Processing Unit at consumer end, detects the server that Service Processing Unit uses
Load and agent side in it is corresponding with Service Processing Unit subscribe to message queue in message number；
Optionally, in step s 302, it detects in agent side in subscription message queue corresponding with Service Processing Unit
Message number, including：
In the load of server that detection cycle detection Service Processing Unit according to setting uses and agent side with industry
The message number being engaged in the corresponding subscription message queue of processing unit.
Step S303 is according to the load judgment of server that the message number and Service Processing Unit that detect use
The no traffic handing capacity for needing to adjust Service Processing Unit, if so, the traffic handing capacity of adjustment Service Processing Unit.
Optionally, in step S303, the server that is used according to the message number and Service Processing Unit that detect
Load judgment whether need adjust Service Processing Unit traffic handing capacity, including：
Step B1：According to subscription message queue corresponding with Service Processing Unit in the detection cycle detection agent side of setting
In the load of server that uses of message number and Service Processing Unit, if the message detected in the first setting duration
Quantity is persistently more than the currently used number of threads of Service Processing Unit or the business detected in the first setting duration
The load of at least one server that processing unit uses then performs step B2 more than the high load threshold of setting；If first
The message number detected in setting duration is set continuously less than the currently used number of threads of Service Processing Unit and first
The load continuous of at least one server that the Service Processing Unit detected in timing is long uses is less than the low-load threshold of setting
Value, then perform step B3；If the testing result in the first setting duration does not meet afore-mentioned, step B1 is repeated；
Step B2：It activates business the traffic handing capacity of processing unit, jump procedure B1；
Step B3：Shrink the traffic handing capacity of Service Processing Unit, jump procedure B1.
The embodiment of the present invention is roughly the same with 3rd embodiment, and difference lies in the embodiment of the present invention will be combined in agent side
The server that the message number subscribed in message queue corresponding with Service Processing Unit and Service Processing Unit use is born
It carries, it is common to determine whether the traffic handing capacity of adjustment Service Processing Unit is needed, due to the industry of Service Processing Unit processing
The type of business is different, and logic complexity is also different, the consumption degree of server resource is also differed, server resource
Including CPU usage, memory source etc., in practical business operation, it is likely that will appear subscription corresponding with Service Processing Unit
Message number in message queue is few, but the feelings for loading excess load of certain server that Service Processing Unit uses
Condition should carry out number of servers extension according to the loading condition of server at this time；It, should when for reduction number of servers
This had both considered the situation of the message number in subscription message queue corresponding with Service Processing Unit it is further contemplated that Service Processing Unit
The loading condition of the server used to codetermine the quantity for whether needing to reduce server, can accurately just reflect in this way
The traffic handing capacity of Service Processing Unit actual needs.
The method for processing business of the embodiment of the present invention, by the load to server used in Service Processing Unit and
The corresponding message queue current message quantity of subscribing to of Service Processing Unit in distributed post-subscription message system is supervised
It surveys, extension or contraction and the expansion of lateral server node to the longitudinal thread of traffic handing capacity progress of Service Processing Unit
Exhibition is shunk.Mode in this way can make business processing more flexible, reduce server automatically when the business free time
Node, automatic expansion service device node, so as to fulfill the automatic telescopic of traffic handing capacity, avoids when heavy traffic
Distributed post-message caused by subscription message system squeezes.
Fifth embodiment of the invention, corresponding with first embodiment, the present embodiment introduces a kind of service data processing apparatus, such as
Shown in Fig. 5, including consisting of part：
1) detection module 501, for being directed to any Service Processing Unit at consumer end, detect in agent side at business
Manage the corresponding message number subscribed in message queue of unit.The per thread that Service Processing Unit uses can handle one and disappear
Breath, the thread resources can discharge after having handled.
Optionally, detection module 501 are used for：According to setting detection cycle detection agent side in Service Processing Unit
Message number in corresponding subscription message queue.
2) judgment module 502, for judging whether to need according at least to the message number detected to adjust business processing list
The traffic handing capacity of member, if so, calling adjustment module 503；
Optionally, judgment module 502 are used for：First setting duration in, if the message number detected be persistently more than or
Person needs to adjust at the business of Service Processing Unit continuously less than the currently used number of threads of Service Processing Unit, then judgement
Reason ability；First setting duration is more than the detection cycle of setting.
3) module 503 is adjusted, for adjusting the traffic handing capacity of Service Processing Unit.
Optionally, adjustment module 503 is used to extending or shrinking thread used in Service Processing Unit and/or server
Quantity.
The service data processing apparatus of the embodiment of the present invention, by the business in distributed post-subscription message system
The corresponding message queue current message quantity of subscribing to of processing unit is monitored, to the traffic handing capacity of Service Processing Unit into
Row extension is shunk.Mode in this way can make business processing more flexible, reduce clothes automatically when the business free time
Business device node, automatic expansion service device node, so as to fulfill the automatic telescopic of traffic handing capacity, keeps away when heavy traffic
Exempt from distributed post-message caused by subscription message system to squeeze.
Sixth embodiment of the invention, corresponding with second embodiment, the present embodiment introduces a kind of service data processing apparatus, such as
Shown in Fig. 6, including consisting of part：
1) configuration module 601, for the loading configuration file in local memory；Business processing list is set in configuration file
Member uses the quantitative range of thread.The per thread that Service Processing Unit uses can handle a piece of news, the line after having handled
Cheng Ziyuan can discharge.
2) detection module 602, for being directed to any Service Processing Unit at consumer end, detect in agent side at business
Manage the corresponding message number subscribed in message queue of unit.
Optionally, detection module 602 are used for：According to setting detection cycle detection agent side in Service Processing Unit
Message number in corresponding subscription message queue.
2) judgment module 603, for being judged whether to need according to the message number detected to adjust Service Processing Unit
Traffic handing capacity, if so, calling adjustment module 604；
Optionally, judgment module 603 are used for：First setting duration in, if the message number detected be persistently more than or
Person needs to adjust at the business of Service Processing Unit continuously less than the currently used number of threads of Service Processing Unit, then judgement
Reason ability；First setting duration is more than the detection cycle of setting.
Why the embodiment of the present invention is not detecting that message number is currently made more than or less than Service Processing Unit
During number of threads just at once judgement need adjust Service Processing Unit traffic handing capacity, be because agent side in industry
The corresponding message number subscribed in message queue of processing unit be engaged in it is possible that not fluctuation quietly, if immediately following fluctuate into
The adjustment of row professional ability, adjustment excessively frequently, the resource consumption of Service Processing Unit will be increased.
3) module 604 is adjusted, for adjusting the traffic handing capacity of Service Processing Unit.
Optionally, module 604 is adjusted, is used for：
If the message number detected is persistently more than the currently used number of threads of Service Processing Unit, with quantity model
The maximum value enclosed be the upper limit, the number of threads that processing unit of activating business uses, until message number be equal to Service Processing Unit
Until the number of threads used；
If the message number detected is continuously less than the currently used number of threads of Service Processing Unit, with quantity model
The minimum value enclosed is lower limit, shrinks the number of threads that Service Processing Unit uses, until message number is equal to Service Processing Unit
Until the number of threads used.
Such as：Adjustment module 604 performs following circulate operation：
Step A1：According to subscription message queue corresponding with Service Processing Unit in the detection cycle detection agent side of setting
In message number, if to be persistently more than Service Processing Unit currently used for the message number that detects in the first setting duration
Number of threads, then perform step A2；If the message number detected in the first setting duration is continuously less than business processing list
The currently used number of threads of member, then perform step A3；If the testing result in the first setting duration does not meet aforementioned feelings
Condition then repeats step A1；
Step A2：Using the maximum value of quantitative range as the upper limit, number of threads that processing unit of activating business uses redirects step
Rapid A1；
Step A3：Using the minimum value of quantitative range as lower limit, the number of threads that Service Processing Unit uses is shunk, redirects step
Rapid A1.
The embodiment of the present invention is roughly the same with the 5th embodiment, and difference lies in an embodiment of the present invention provides by simple
The mode configuration service processing unit of configuration file use the quantitative range of thread, which can also configure for business
The detection of processing unit and professional ability adjustable strategies are embodied in the specific of detection module, judgment module and adjustment module
In operation.In addition, the embodiment of the present invention has been also provided to more detailed detection adjustment process.
The service data processing apparatus of the embodiment of the present invention, by the business in distributed post-subscription message system
The corresponding message queue current message quantity of subscribing to of processing unit is monitored, to the traffic handing capacity of Service Processing Unit into
The extension or contraction of row longitudinal direction thread.Mode in this way can make business processing more flexible, when the business free time
Automatic reduction server node, automatic expansion service device node when heavy traffic, so as to fulfill traffic handing capacity
Automatic telescopic avoids distributed post-message caused by subscription message system and squeezes.
Seventh embodiment of the invention, corresponding with 3rd embodiment, the present embodiment introduces a kind of service data processing apparatus, such as
Shown in Fig. 6, including consisting of part：
1) configuration module 601, for the loading configuration file in local memory；Business processing list is set in configuration file
Member uses the quantitative range of thread.The per thread that Service Processing Unit uses can handle a piece of news, the line after having handled
Cheng Ziyuan can discharge.
2) detection module 602, for being directed to any Service Processing Unit at consumer end, detect in agent side at business
Manage the corresponding message number subscribed in message queue of unit.
Optionally, detection module 602 are used for：According to setting detection cycle detection agent side in Service Processing Unit
Message number in corresponding subscription message queue.
3) judgment module 603, for being judged whether to need according to the message number detected to adjust Service Processing Unit
Traffic handing capacity, if so, calling adjustment module 604；
Optionally, judgment module 603 are used for：First setting duration in, if the message number detected be persistently more than or
Person needs to adjust at the business of Service Processing Unit continuously less than the currently used number of threads of Service Processing Unit, then judgement
Reason ability；First setting duration is more than the detection cycle of setting.
Why the embodiment of the present invention is not detecting that message number is currently made more than or less than Service Processing Unit
During number of threads just at once judgement need adjust Service Processing Unit traffic handing capacity, be because agent side in industry
The corresponding message number subscribed in message queue of processing unit be engaged in it is possible that not fluctuation quietly, if immediately following fluctuate into
The adjustment of row professional ability, adjustment excessively frequently, the resource consumption of Service Processing Unit will be increased.
4) module 604 is adjusted, for adjusting the traffic handing capacity of Service Processing Unit.
Optionally, module 604 is adjusted, is used for：
If the message number detected is persistently more than the currently used number of threads of Service Processing Unit, with quantity model
The maximum value enclosed be the upper limit, the number of threads that processing unit of activating business uses, until message number be equal to Service Processing Unit
Until the number of threads used；
If the message number detected is continuously less than the currently used number of threads of Service Processing Unit, with quantity model
The minimum value enclosed is lower limit, shrinks the number of threads that Service Processing Unit uses, until message number is equal to Service Processing Unit
Until the number of threads used.
Such as：Adjustment module 604 performs following circulate operation：
Step A1：According to subscription message queue corresponding with Service Processing Unit in the detection cycle detection agent side of setting
In message number, if to be persistently more than Service Processing Unit currently used for the message number that detects in the first setting duration
Number of threads, then perform step A2；If the message number detected in the first setting duration is continuously less than business processing list
The currently used number of threads of member, then perform step A3；If the testing result in the first setting duration does not meet aforementioned feelings
Condition then repeats step A1；
Step A2：Using the maximum value of quantitative range as the upper limit, number of threads that processing unit of activating business uses redirects step
Rapid A1；
Step A3：Using the minimum value of quantitative range as lower limit, the number of threads that Service Processing Unit uses is shunk, redirects step
Rapid A1.
Optionally, if Service Processing Unit uses handles the message number upper limit during server in the second setting duration
For M, M is positive integer；It is positive integer that the number of servers upper limit N, N that Service Processing Unit uses are set in configuration file；
Module 604 is adjusted, is additionally operable to：
If the message number increment detected in the first setting duration is more than or equal to n M, n is positive integer, then with N
For the upper limit, n+1 platform servers are used for Service Processing Unit extension；
If the message number decrement detected in the first setting duration is more than or equal to n M, then, using N as the upper limit, be
Service Processing Unit reduction uses n platform servers.
The embodiment of the present invention is roughly the same with sixth embodiment, and difference lies in the embodiment of the present invention is in adjustment business processing
Extension or receipts for server node used in Service Processing Unit are also provided during the traffic handing capacity of unit
Contracting, when the thread that a server can be provided has been not enough to the operation of one Service Processing Unit of support, it is necessary to expand
The server number that the Service Processing Unit uses is opened up, it is on the contrary, it is necessary to reduce.Such as：One Service Processing Unit uses
It is 100,000 to handle the message number upper limit in 3 minutes during one server, it is assumed that the current Service Processing Unit only used
One server, and the message number in currently detected agent side in subscription message queue corresponding with Service Processing Unit is
At 260000, it is necessary to increase two-server and be used for the Service Processing Unit.
The service data processing apparatus of the embodiment of the present invention passes through the load to server used in Service Processing Unit
And the corresponding message queue current message quantity of subscribing to of Service Processing Unit in distributed post-subscription message system carries out
It monitors, the extension or contraction and lateral server node to the longitudinal thread of traffic handing capacity progress of Service Processing Unit
Extension is shunk.Mode in this way can make business processing more flexible, reduce service automatically when the business free time
Device node, automatic expansion service device node, so as to fulfill the automatic telescopic of traffic handing capacity, avoids when heavy traffic
Distributed post-message caused by subscription message system squeezes.
Eighth embodiment of the invention, corresponding with fourth embodiment, the present embodiment introduces a kind of service data processing apparatus, such as
Shown in Fig. 6, including consisting of part：
1) configuration module 601, for the loading configuration file in local memory；Business processing list is set in configuration file
Member is positive integer using the number of servers upper limit N, N that the quantitative range and Service Processing Unit of thread use.Business processing
The per thread that unit uses can handle a piece of news, and the thread resources can discharge after having handled.
2) detection module 602, for being directed to any Service Processing Unit at consumer end, detection Service Processing Unit uses
Server load and agent side in it is corresponding with Service Processing Unit subscribe to message queue in message number；.
Optionally, detection module 602 are used for：According to setting detection cycle detection agent side in Service Processing Unit
The load of server that message number and Service Processing Unit in corresponding subscription message queue use.
3) judgment module 603, the server that the message number and Service Processing Unit detected for basis uses
Whether load judgment needs to adjust the traffic handing capacity of Service Processing Unit, if so, calling adjustment module 604；
Optionally, judgment module 603, if the message number for being detected in the first setting duration is persistently more than business
The currently used number of threads of processing unit or in the first setting duration the Service Processing Unit that detects use to
More than the high load threshold of setting, then judgement needs to activate business the business processing energy of processing unit for the load of a few server
Power continues to call detection module 602；
If the message number detected in the first setting duration is continuously less than the currently used line of Service Processing Unit
The load continuous of at least one server that number of passes amount and the Service Processing Unit detected in the first setting duration use is small
In the low-load threshold value of setting, then judgement needs to shrink the traffic handing capacity of Service Processing Unit, continues to call detection module
602；
If the testing result in the first setting duration does not meet afore-mentioned, continue to call detection module 602.
4) module 604 is adjusted, for adjusting the traffic handing capacity of Service Processing Unit.
Optionally, the traffic handing capacity for adjusting Service Processing Unit includes：Extension or contraction Service Processing Unit are made
The quantity of thread and/or server.And first consider extension as possible or shrink thread used in Service Processing Unit, when logical
Thread used in crossing extension or contraction Service Processing Unit still cannot be close to the business processing energy needed for Service Processing Unit
In the case of power, consider further that extension or shrink number of servers used in Service Processing Unit.
The embodiment of the present invention is roughly the same with the 7th embodiment, and difference lies in the embodiment of the present invention will be combined in agent side
The server that the message number subscribed in message queue corresponding with Service Processing Unit and Service Processing Unit use is born
It carries, it is common to determine whether the traffic handing capacity of adjustment Service Processing Unit is needed, due to the industry of Service Processing Unit processing
The type of business is different, and logic complexity is also different, the consumption degree of server resource is also differed, server resource
Including CPU usage, memory source etc., in practical business operation, it is likely that will appear subscription corresponding with Service Processing Unit
Message number in message queue is few, but the feelings for loading excess load of certain server that Service Processing Unit uses
Condition should carry out number of servers extension according to the loading condition of server at this time；It, should when for reduction number of servers
This had both considered the situation of the message number in subscription message queue corresponding with Service Processing Unit it is further contemplated that Service Processing Unit
The loading condition of the server used to codetermine the quantity for whether needing to reduce server, can accurately just reflect in this way
The traffic handing capacity of Service Processing Unit actual needs.
The service data processing apparatus of the embodiment of the present invention passes through the load to server used in Service Processing Unit
And the corresponding message queue current message quantity of subscribing to of Service Processing Unit in distributed post-subscription message system carries out
It monitors, the extension or contraction and lateral server node to the longitudinal thread of traffic handing capacity progress of Service Processing Unit
Extension is shunk.Mode in this way can make business processing more flexible, reduce service automatically when the business free time
Device node, automatic expansion service device node, so as to fulfill the automatic telescopic of traffic handing capacity, avoids when heavy traffic
Distributed post-message caused by subscription message system squeezes.
Ninth embodiment of the invention, the present embodiment are on the basis of above-described embodiment, and one is introduced with reference to attached drawing 7~8
The application example of the present invention.
As shown in fig. 7, the embodiment of the present invention provides a kind of device for avoiding consumer end service process performance bottleneck, with solution
After certainly big data message desk (kafka) is subscribed to the problem of the messaging service process performance bottleneck of consumer end.By this device certainly
The automatic horizontal stroke of the dynamic Longitudinal Extension and server node that thread is carried out according to Service Processing Unit current subscription message queue depth
To extension.
After Service Processing Unit starts, the device for avoiding consumer end service process performance bottleneck of the embodiment of the present invention
Just start immediately.The corresponding software product of the device can include Service Processing Unit, can not also include Service Processing Unit.
This programme biggest advantage is, the side that Service Processing Unit passes through interface inheritance by method, that is, device that realization device provides
Formula so that Service Processing Unit has horizontal and vertical extended capability.In addition, device has general high concurrent processing knot
Structure, the server node quantity that professional ability can be realized by simple configuration file extend automatically, and Message Processing is avoided to accumulate
Pressure, the appearance for situations such as server resource exorbitant expenditure and Message Processing is not prompt enough.
Server state detection module：For detection service device present load, can server be obtained by shell modes
The information for the Service Processing Unit currently supported.For Service Processing Unit, it can obtain used in the Service Processing Unit
Server loading condition.
Message queue detection module：Depth, i.e. current subscription queue message are squeezed for detecting current subscription queue message
Quantity.
Thread pool：Realize thread dynamic expansion in individual server node.
Message subscribing module：For subscribing to message queue for Service Processing Unit from agent side, Service Processing Unit need not
Repeated encoding is uniformly processed by the device.
Server list module：Server list is provided, comprising current server cluster maximum service device number of nodes, is also used
In dynamic expansion server node.
Cache module：For the configurations in configuration file to be loaded into local memory, comprising real in the configurations
The method and common data of existing business, share use for other modules,
Server thread load detecting module：For detecting the Thread Count under Service Processing Unit.
Service Processing Unit：In a manner that method, that is, device that realization device provides is by interface inheritance, to subscribing to
Message carry out normal business processing so that Service Processing Unit has horizontal and vertical extended capability.It is not required here
High concurrent and function extending transversely are paid close attention to, developer only needs to realize for the business processing flow of message progress subscribed to i.e.
It can.
As shown in figure 8, laterally --- the management of server：
Step 1, after Service Processing Unit 1 starts, the device of consumer end service process performance bottleneck is avoided also to open therewith
It is dynamic.
Step 2, initiation message subscribing module, message subscribing module need to use caching when pulling message from agent side first
The common data of configured in advance in module, and thread process message is occupied, thread pool starts at this time, and is loaded by server thread
Detection module detects the thread service condition of Service Processing Unit 1.
Step 3, start queue depth's detection module and server state detection module.
Step 4, pass through queue depth's detection module, server state detection module and the server list in the device
Module decision whether need to extend new server node come it is extending transversely.
Step 5, it when above-mentioned decision-making module, which detects, to be needed to extend new server node, is carried out by way of script
The automatic startup of server 2- processing units 1.So as to fulfill the dynamic expansion of server node.
Similarly, when decision-making module, which detects, to be needed to reduce the server node of the processing unit 1, by way of script
The closing of server 2- Service Processing Units 1 is carried out, so as to fulfill the reasonable utilization of the resource of server node.
The server node released can also be distributed to other Service Processing Units, be deleted by script corresponding
The lock file of Program Generating, lower portion have lock file checking finger daemon to be responsible for detecting whether to stop present node process.Stop
Corresponding heap memory and cpu resource can be discharged after only, after this, you can give the resource allocation to other processing units.
The management of longitudinal direction --- -- thread：
Be limited with the minimum and maximum Thread Count for the thread pool being configured in current configuration file, carry out the extension of thread pool with
It shrinks.
The embodiment of the present invention avoids the device of consumer end service process performance bottleneck from realizing to kafka message by this
It subscribes to, operation layer only needs to realize that the automatic of message pulls according to the configuration of configuration file.
By the detection to message queue depth, the detection of server state is realized the dynamic expansion of server node, is made
Operation layer can be absorbed in business processing, without paying close attention to the complicated realization under high concurrent in terms of dynamic expansion.
By the encapsulation of the Service Processing Unit to multithreading, shielding operation layer realizes multithreading.Realize single server
The realization of high concurrent in node traffic processing unit.
By the explanation of specific embodiment, should can to the present invention for reach technological means that predetermined purpose is taken and
Effect is able to more go deep into and specific understanding, however appended diagram is only to provide reference and description and is used, and is not used for originally
Invention limits.
Claims (12)
1. a kind of business data processing method, which is characterized in that including：
For any Service Processing Unit at consumer end, detect subscription corresponding with the Service Processing Unit in agent side and disappear
The message number in queue is ceased, wherein, the corresponding one or more subscription message queues of each described Service Processing Unit, institute
Service Processing Unit is stated for handling the message in the subscription message queue；
Judge whether to need the business processing energy for adjusting the Service Processing Unit according at least to the message number detected
Power, if so, adjusting the traffic handing capacity of the Service Processing Unit.
2. business data processing method according to claim 1, which is characterized in that it is described detection agent side in the industry
The message number being engaged in the corresponding subscription message queue of processing unit, including：
Disappearing in message queue is subscribed to according to corresponding with the Service Processing Unit in the detection cycle detection agent side of setting
Cease quantity；
The message number that the basis detects judges whether to need the business processing energy for adjusting the Service Processing Unit
Power, including：
In the first setting duration, if the message number detected is persistently more than or continuously less than the business processing list
The currently used number of threads of member, then judgement need to adjust the traffic handing capacity of the Service Processing Unit；Described first
Set the detection cycle that duration is more than the setting.
3. business data processing method according to claim 2, which is characterized in that the method further includes：In detection generation
Before managing the message number in end in subscription message queue corresponding with the Service Processing Unit, load and match in local memory
Put file；The quantitative range that the Service Processing Unit uses thread is set in the configuration file；
The traffic handing capacity of the adjustment Service Processing Unit, including：
If the message number detected is persistently more than the currently used number of threads of the Service Processing Unit, institute is extended
The number of threads that Service Processing Unit uses is stated, until the number of threads that the Service Processing Unit uses is equal to the message count
Until amount, wherein, the upper limit for the number of threads that the Service Processing Unit uses is the maximum value of the quantitative range；
If the message number detected shrinks institute continuously less than the currently used number of threads of the Service Processing Unit
The number of threads that Service Processing Unit uses is stated, until the number of threads that the Service Processing Unit uses is equal to the message count
Until amount, wherein, the lower limit for the number of threads that the Service Processing Unit uses is the minimum value of the quantitative range.
4. business data processing method according to claim 3, which is characterized in that set the Service Processing Unit and use one
It is set during platform server second and the message number upper limit is handled in duration as M, M is positive integer；It is set in the configuration file
The number of servers upper limit N, N that the Service Processing Unit uses are positive integer；
The traffic handing capacity of the adjustment Service Processing Unit, further includes：
If the message number increment detected in the first setting duration is more than or equal to n M, n is positive integer, then with N
For the upper limit, n+1 platform servers are used for Service Processing Unit extension；
If the message number decrement detected in the first setting duration is more than or equal to n M, then, using N as the upper limit, be
The Service Processing Unit reduction uses n platform servers.
5. business data processing method according to claim 1, which is characterized in that the method further includes：In at least root
Before judging whether the traffic handing capacity for needing to adjust the Service Processing Unit according to the message number detected, detection
The load for the server that the Service Processing Unit uses；
Judge whether to need the business processing energy for adjusting the Service Processing Unit according at least to the message number detected
Power includes：Whether the load judgment of server that the message number and the Service Processing Unit according to detecting use
Need to adjust the traffic handing capacity of the Service Processing Unit, if so, adjusting the business processing of the Service Processing Unit
Ability.
6. business data processing method according to claim 5, which is characterized in that the message that the basis detects
Whether the load judgment of server that quantity and the Service Processing Unit use needs to adjust the Service Processing Unit
Traffic handing capacity, including：
Step A1：The server that detection cycle according to setting detects the message number and the Service Processing Unit uses
Load, if the message number that detects persistently is more than the Service Processing Unit and is currently made in the first setting duration
At least server that number of threads or the Service Processing Unit detected in the first setting duration use
Load more than setting high load threshold, then perform step A2；If the message count detected in the first setting duration
Described in amount is detected continuously less than the currently used number of threads of the Service Processing Unit and in the first setting duration
The load continuous of at least one server that Service Processing Unit uses is less than the low-load threshold value of setting, then performs step A3；
If the testing result in the first setting duration does not meet afore-mentioned, step A1 is repeated；
Step A2：Extend the traffic handing capacity of the Service Processing Unit, jump procedure A1；
Step A3：Shrink the traffic handing capacity of the Service Processing Unit, jump procedure A1.
7. a kind of service data processing apparatus, which is characterized in that including：
Detection module, for being directed to any Service Processing Unit at consumer end, detect in agent side with the business processing list
The corresponding message number subscribed in message queue of member, wherein, each described Service Processing Unit is corresponding one or more
Message queue is subscribed to, the Service Processing Unit is used to handle the message in the subscription message queue；
Judgment module, for judging whether to need according at least to the message number detected to adjust the Service Processing Unit
Traffic handing capacity, if so, call adjustment module；
Module is adjusted, for adjusting the traffic handing capacity of the Service Processing Unit.
8. service data processing apparatus according to claim 7, which is characterized in that the detection module is used for：According to setting
The message number subscribed in message queue corresponding with the Service Processing Unit in fixed detection cycle detection agent side；
The judgment module, is used for：In the first setting duration, if the message number detected is persistently more than or continues
Less than the currently used number of threads of the Service Processing Unit, then judgement needs to adjust the business of the Service Processing Unit
Processing capacity；The first setting duration is more than the detection cycle of the setting.
9. service data processing apparatus according to claim 8, which is characterized in that described device further includes：
Configuration module, for subscription message team corresponding with the Service Processing Unit in detection module detection agent side
Before message number in row, the loading configuration file in local memory；The business processing is set in the configuration file
Unit uses the quantitative range of thread；
The adjustment module, is used for：
If the message number detected is persistently more than the currently used number of threads of the Service Processing Unit, institute is extended
The number of threads that Service Processing Unit uses is stated, until the number of threads that the Service Processing Unit uses is equal to the message count
Until amount, wherein, the upper limit for the number of threads that the Service Processing Unit uses is the maximum value of the quantitative range；
If the message number detected shrinks institute continuously less than the currently used number of threads of the Service Processing Unit
The number of threads that Service Processing Unit uses is stated, until the number of threads that the Service Processing Unit uses is equal to the message count
Until amount, wherein, the lower limit for the number of threads that the Service Processing Unit uses is the minimum value of the quantitative range.
10. service data processing apparatus according to claim 9, which is characterized in that set the Service Processing Unit and use
It is set during one server second and the message number upper limit is handled in duration as M, M is positive integer；It is set in the configuration file
It is positive integer to have the number of servers upper limit N, N that the Service Processing Unit uses；
The adjustment module, is additionally operable to：
If the message number increment detected in the first setting duration is more than or equal to n M, n is positive integer, then with N
For the upper limit, n+1 platform servers are used for Service Processing Unit extension；
If the message number decrement detected in the first setting duration is more than or equal to n M, then, using N as the upper limit, be
The Service Processing Unit reduction uses n platform servers.
11. service data processing apparatus according to claim 7, which is characterized in that the detection module is additionally operable to：Inspection
Survey the load for the server that the Service Processing Unit uses；
The judgment module, for the server used according to the message number and the Service Processing Unit that detect
Load judgment whether need to adjust the traffic handing capacity of the Service Processing Unit.
12. service data processing apparatus according to claim 11, which is characterized in that the detection module, for according to
The load of server that the detection cycle of setting detects the message number and the Service Processing Unit uses；
The judgment module, if the message number for being detected in the first setting duration is persistently more than at the business
What the currently used number of threads of reason unit or the Service Processing Unit detected in the first setting duration used
The load of at least one server then judges to need to extend the business of the Service Processing Unit more than the high load threshold of setting
Processing capacity continues to call the detection module；
If the message number detected in the first setting duration is currently used continuously less than the Service Processing Unit
Number of threads and the Service Processing Unit that detects uses in the first setting duration at least server it is negative
The low-load threshold value continuously less than setting is carried, then judgement needs to shrink the traffic handing capacity of the Service Processing Unit, continues
Call the detection module；
If the testing result in the first setting duration does not meet afore-mentioned, continue to call the detection module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711205491.3A CN108134814B (en) | 2017-11-27 | 2017-11-27 | Service data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711205491.3A CN108134814B (en) | 2017-11-27 | 2017-11-27 | Service data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108134814A true CN108134814A (en) | 2018-06-08 |
| CN108134814B CN108134814B (en) | 2020-12-22 |
Family
ID=62389860
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711205491.3A Active CN108134814B (en) | 2017-11-27 | 2017-11-27 | Service data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108134814B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109117286A (en) * | 2018-07-30 | 2019-01-01 | 佛山市甜慕链客科技有限公司 | A kind of method of data collection and adjusting |
| CN109508354A (en) * | 2018-09-25 | 2019-03-22 | 许继集团有限公司 | A kind of parallel processing system (PPS) |
| CN110460534A (en) * | 2019-07-26 | 2019-11-15 | 腾讯云计算（北京）有限责任公司 | A kind of request message report method, device, equipment and storage medium |
| CN110515746A (en) * | 2019-08-22 | 2019-11-29 | 北京宝兰德软件股份有限公司 | A kind of method and device handling slow consumer |
| CN110875935A (en) * | 2018-08-30 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Message publishing, processing and subscribing method, device and system |
| CN111179080A (en) * | 2019-12-23 | 2020-05-19 | 中国建设银行股份有限公司 | Order processing method and order processing device |
| CN111625366A (en) * | 2020-06-02 | 2020-09-04 | 深圳市网是科技有限公司 | Elastic expansion service method based on release and subscription model |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092619A (en) * | 2014-07-25 | 2014-10-08 | 华为技术有限公司 | Traffic control method and device |
| CN105302890A (en) * | 2015-10-16 | 2016-02-03 | 海信集团有限公司 | Multimedia content online recommendation method and multimedia content online recommendation auxiliary method and apparatus |
| US20160154678A1 (en) * | 2014-11-28 | 2016-06-02 | Canon Kabushiki Kaisha | Reverting tightly coupled threads in an over-scheduled system |
| CN106293968A (en) * | 2016-08-04 | 2017-01-04 | 华中科技大学 | A kind of intercommunication system based on Kafka message-oriented middleware and method |
| CN106878415A (en) * | 2017-02-15 | 2017-06-20 | 阿里巴巴集团控股有限公司 | The load-balancing method and device of data consumption |
-
2017
- 2017-11-27 CN CN201711205491.3A patent/CN108134814B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104092619A (en) * | 2014-07-25 | 2014-10-08 | 华为技术有限公司 | Traffic control method and device |
| US20160154678A1 (en) * | 2014-11-28 | 2016-06-02 | Canon Kabushiki Kaisha | Reverting tightly coupled threads in an over-scheduled system |
| CN105302890A (en) * | 2015-10-16 | 2016-02-03 | 海信集团有限公司 | Multimedia content online recommendation method and multimedia content online recommendation auxiliary method and apparatus |
| CN106293968A (en) * | 2016-08-04 | 2017-01-04 | 华中科技大学 | A kind of intercommunication system based on Kafka message-oriented middleware and method |
| CN106878415A (en) * | 2017-02-15 | 2017-06-20 | 阿里巴巴集团控股有限公司 | The load-balancing method and device of data consumption |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109117286A (en) * | 2018-07-30 | 2019-01-01 | 佛山市甜慕链客科技有限公司 | A kind of method of data collection and adjusting |
| CN110875935A (en) * | 2018-08-30 | 2020-03-10 | 阿里巴巴集团控股有限公司 | Message publishing, processing and subscribing method, device and system |
| CN109508354A (en) * | 2018-09-25 | 2019-03-22 | 许继集团有限公司 | A kind of parallel processing system (PPS) |
| CN110460534A (en) * | 2019-07-26 | 2019-11-15 | 腾讯云计算（北京）有限责任公司 | A kind of request message report method, device, equipment and storage medium |
| CN110515746A (en) * | 2019-08-22 | 2019-11-29 | 北京宝兰德软件股份有限公司 | A kind of method and device handling slow consumer |
| CN111179080A (en) * | 2019-12-23 | 2020-05-19 | 中国建设银行股份有限公司 | Order processing method and order processing device |
| CN111179080B (en) * | 2019-12-23 | 2023-10-27 | 中国建设银行股份有限公司 | Order processing method and order processing device |
| CN111625366A (en) * | 2020-06-02 | 2020-09-04 | 深圳市网是科技有限公司 | Elastic expansion service method based on release and subscription model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108134814B (en) | 2020-12-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108134814A (en) | A kind of business data processing method and device | |
| CN102004670B (en) | Self-adaptive job scheduling method based on MapReduce | |
| CN106603618A (en) | Cloud platform-based application auto scaling method | |
| CN106933662A (en) | Distributed system and its dispatching method and dispatching device | |
| CN107479990A (en) | Distributed software service system | |
| CN111052086A (en) | Warm start technology of cloud hosting function | |
| CN111459665A (en) | Distributed edge computing system and distributed edge computing method | |
| CN106454354B (en) | A kind of AVS2 parallel encoding processing system and method | |
| WO2022188578A1 (en) | Method and system for multiple services to share same gpu, and device and medium | |
| CN110048907B (en) | Global flow control method and device in cluster environment | |
| CN111367693B (en) | Method, system, device and medium for scheduling plug-in tasks based on message queue | |
| CN104298565B (en) | A kind of method and device of initiating port | |
| CN110677274A (en) | Event-based cloud network service scheduling method and device | |
| CN111427751A (en) | Method and system for processing service based on asynchronous processing mechanism | |
| CN110489225A (en) | A kind of service expansion method, device and equipment based on message queue | |
| CN109445955A (en) | A kind of method of counting and number system | |
| CN104461731B (en) | Resources of virtual machine recovery method and device in a kind of dynamic resource extension | |
| CN106484459A (en) | It is applied to flow control method and the device of JavaScript | |
| CN108900607A (en) | A kind of processing method, device and the server of SMB agreement request | |
| CN101799772A (en) | Kernel dispatching method, kernel backup method and multi-core processor | |
| CN111541646A (en) | Method for enhancing security service access capability of cipher machine | |
| CN111274018A (en) | Distributed training method based on DL framework | |
| CN107911484A (en) | A kind of method and device of Message Processing | |
| CN105187518B (en) | A kind of CDN content distribution method and system | |
| CN113703982A (en) | Data consumption method, apparatus, terminal device and medium using KAFKA |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20191226 Address after: 100086, room 601-606, room 6, Pacific International Building, No. 106, Haidian District, Beijing, Zhichun Road Applicant after: Haier Youjia Intelligent Technology (Beijing) Co., Ltd. Applicant after: Qingdao Haier Science and Technology Co., Ltd. Address before: 100086, room 601-606, room 6, Pacific International Building, No. 106, Haidian District, Beijing, Zhichun Road Applicant before: Haier Youjia Intelligent Technology (Beijing) Co., Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210210 Address after: 266101 Haier Industrial Park, 1 Haier Road, Laoshan District, Shandong, Qingdao Patentee after: QINGDAO HAIER TECHNOLOGY Co.,Ltd. Patentee after: HAIER YOUJIA INTELLIGENT TECHNOLOGY (BEIJING) Co.,Ltd. Patentee after: Haier Zhijia Co.,Ltd. Address before: Room 601-606, 6 / F, Pacific International Building, 106 Zhichun Road, Haidian District, Beijing 100086 Patentee before: HAIER YOUJIA INTELLIGENT TECHNOLOGY (BEIJING) Co.,Ltd. Patentee before: QINGDAO HAIER TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |