CN108683612A - A kind of message acquisition method and device - Google Patents
A kind of message acquisition method and device Download PDFInfo
- Publication number
- CN108683612A CN108683612A CN201810496393.8A CN201810496393A CN108683612A CN 108683612 A CN108683612 A CN 108683612A CN 201810496393 A CN201810496393 A CN 201810496393A CN 108683612 A CN108683612 A CN 108683612A
- Authority
- CN
- China
- Prior art keywords
- message
- information
- queue
- message queue
- queuing middleware
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/59—Providing operational support to end devices by off-loading in the network or by emulation, e.g. when they are unavailable
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/288—Distributed intermediate devices, i.e. intermediate devices for interaction with other intermediate devices on the same level
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
- Telephonic Communication Services (AREA)
Abstract
This specification embodiment discloses a kind of method and apparatus obtaining message from Message Queuing Middleware, and the method is executed by the process for handling the message in computing platform, including：Current distribution information is obtained, current distribution information is related to the first message queue of the first quantity of the process is currently assigned to, wherein the first message queue is the message queue in the Message Queuing Middleware；Judge whether current distribution information changes compared to the original allocation information to prestore, the original allocation information is related to the second message queue of the second quantity of the process is distributed in the process initiation, wherein, the second message queue is the message queue in the Message Queuing Middleware；And in the case where the current distribution information changes compared to the original allocation information, restarted, connection is established with the first message queue with first quantity, to obtain message from the first message queue of first quantity.
Description
Technical field
This specification embodiment is related to technical field of data processing, more particularly, to one kind from Message Queuing Middleware
The method and apparatus for obtaining message.
Background technology
Currently, handling task, especially real-time task for big data, carried out usually using big data computing platform,
Such as Jstorm, Storm, Spark Streaming etc..In such processing task, usually using Message Queuing Middleware
Carry out data access.Message Queuing Middleware has been increasingly becoming the core means of IT system intercommunication.It has lower coupling,
A series of functions such as reliable delivery, broadcast, flow control, final consistency, become the main of asynchronous RPC (remote procedure call)
One of means.Currently, the message-oriented middleware of mainstream is such as including ActiveMQ, RabbitMQ, Kafka, RocketMQ.Example
Such as, in Kafka-Storm schemes, Message Queuing Middleware Kafka to the consumption node (Message Processing process) in Storm into
Row message queue poll distributes, and to ensure to evenly distribute, and each message queue is only consumed by a node, in addition,
Information consumption progress is safeguarded by upstream messages queue.However, in above-mentioned Kafka-Storm schemes, when message queue is sent out
When changing, such as when increase, reduction, change, consumption node can not perceive, also, be disappeared to safeguard by upstream messages queue
Breath consumption progress is not sufficiently stable reliably, when the special circumstances such as power-off, crash occur for Kafka middleware servers end, it is possible to
Influence the maintenance to information consumption progress.Therefore, it is necessary to it is a kind of more effectively from Message Queuing Middleware obtain message scheme,
To solve above-mentioned deficiency.
Invention content
This specification embodiment is intended to provide a kind of scheme more effectively obtaining message from Message Queuing Middleware, with solution
Deficiency certainly in the prior art.
To achieve the above object, this specification provides a kind of side obtaining message from Message Queuing Middleware on one side
Method, the method are executed by the process for handling the message in computing platform, including：
Obtain current distribution information, the current distribution information and the first of the first quantity for being currently assigned to the process
Message queue is related, wherein the first message queue is the message queue in the Message Queuing Middleware；
Judge whether the current distribution information changes compared to the original allocation information to prestore, the original allocation
Information is related to the second message queue of the second quantity of the process is distributed in the process initiation, wherein described
Two message queues are the message queues in the Message Queuing Middleware；And
In the case where the current distribution information changes compared to the original allocation information, restarted, with
First message queue with first quantity, which is established, to be connected, to disappear from the first message queue of first quantity acquisition
Breath.
In one embodiment, described from the method that Message Queuing Middleware obtains message, in the message queue
Between message queue in part there is corresponding theme, the process be arranged to from the Message Queuing Middleware at least
The corresponding at least one message queue of one specific subject obtains message, and, the current distribution information and currently described
At least one message queue is related.
In one embodiment, described from the method that Message Queuing Middleware obtains message, the process is one group
There is corresponding theme, one group of process to be set for a process in process, the message queue in the Message Queuing Middleware
It is set to from least one message queue corresponding at least one specific subject in the Message Queuing Middleware and obtains message,
The current distribution information of acquisition includes, and the first information is obtained from the Message Queuing Middleware, the first information with it is current
At least one message queue it is related, and, be based on the first information, obtain the current distribution information.
In one embodiment, believe based on described first from the method that Message Queuing Middleware obtains message described
Breath, obtaining the current distribution information includes, and according to polling algorithm, is based on the first information, obtains the current distribution letter
Breath.
In one embodiment, described from the method that Message Queuing Middleware obtains message, the process is one group
A process in process, the message queue in the Message Queuing Middleware have corresponding theme, one group of process quilt
It is set as disappearing from least one message queue acquisition corresponding at least one specific subject in the Message Queuing Middleware
Breath, the current distribution information of acquisition include：
The first information, the first information and current at least one message are obtained from the Message Queuing Middleware
Queue is related；Judge whether the first information changes compared to the second information, wherein second information in the process
At least one message queue when startup is related；And in the first information compared to the changed feelings of the second information
In condition, it is based on the first information, obtains the current distribution information.
In one embodiment, described from the method that Message Queuing Middleware obtains message, the current distribution letter
Cease the theme of each message queue, place in the first message queue of the value and first quantity that include first quantity
The mark and queue identity of server, and, the original allocation information includes the value of second quantity and second number
The mark and queue identity of the theme of each message queue, place server in the second message queue of amount.
In one embodiment, the method for obtaining message from Message Queuing Middleware further includes carrying out restarting it
Afterwards, for it is described restart before and it is described restart after the third message queue that is all connect with the process, in storage unit
The middle processing progress information for obtaining the third message queue, and according to the processing progress information, determination will be from described the
Multiple message that three message queues obtain.
In one embodiment, described from the method that Message Queuing Middleware obtains message, the processing progress letter
It ceases and is obtained from the processing progress table in storage unit, the processing progress table includes the value of following field：Theme, server mark
Know, message queue identifies and message identifier.
In one embodiment, the method for obtaining message from Message Queuing Middleware further includes, determination will be from
It is single in the storage when having handled multiple message of the acquisition after multiple message that the third message queue obtains
The processing progress table is updated in member.
In one embodiment, described from the method that Message Queuing Middleware obtains message, the storage unit is
Independently of third party's storage unit of the Message Queuing Middleware and the computing platform.
In one embodiment, described from the method that Message Queuing Middleware obtains message, the computing platform is
Kepler computing platforms.
On the other hand this specification provides a kind of device obtaining message from Message Queuing Middleware, described device is by calculating
The process for handling the message in platform is implemented, including：
First acquisition unit is configured to, and obtains current distribution information, the current distribution information and described in being currently assigned to
The first message queue of first quantity of process is related, wherein the first message queue is in the Message Queuing Middleware
Message queue；
Judging unit is configured to, and judges whether the current distribution information occurs compared to the original allocation information to prestore
Variation, the second message queue of the original allocation information and the second quantity that the process is distributed in the process initiation
It is related, wherein the second message queue is the message queue in the Message Queuing Middleware；And
Restart unit, be configured to, in the current distribution information compared to the changed feelings of original allocation information
Under condition, restarted, connection is established with the first message queue with first quantity, thus from the first of first quantity
Message queue obtains message.
In one embodiment, described from the device that Message Queuing Middleware obtains message, the process is one group
There is corresponding theme, one group of process to be set for a process in process, the message queue in the Message Queuing Middleware
It is set to from least one message queue corresponding at least one specific subject in the Message Queuing Middleware and obtains message,
The first acquisition unit further includes：First obtains subelement, is configured to, and the first letter is obtained from the Message Queuing Middleware
Breath, the first information are related to current at least one message queue；And second obtain subelement, be configured to, base
In the first information, the current distribution information is obtained.
In one embodiment, described from the device that Message Queuing Middleware obtains message, the process is one group
A process in process, the message queue in the Message Queuing Middleware have corresponding theme, one group of process quilt
It is set as disappearing from least one message queue acquisition corresponding at least one specific subject in the Message Queuing Middleware
Breath, the first acquisition unit further include：First obtains subelement, is configured to, and first is obtained from the Message Queuing Middleware
Information, the first information are related to current at least one message queue；Judgment sub-unit is configured to, described in judgement
Whether the first information changes compared to the second information, wherein second information in the process initiation described at least
One message queue correlation；And second obtain subelement, be configured to, become compared to the second information in the first information
In the case of change, it is based on the first information, obtains the current distribution information.
In one embodiment, the device that message is obtained from Message Queuing Middleware further includes second acquisition unit,
Be configured to, after being restarted, for it is described restart before and it is described restart after the third that is all connect with the process
Message queue obtains the processing progress information of the third message queue in the memory unit；And determination unit, it is configured to,
According to the processing progress information, determination will be from multiple message of third message queue acquisition.
In one embodiment, the device that message is obtained from Message Queuing Middleware further includes updating unit, configuration
For, determination will from the third message queue obtain multiple message after, in the multiple message for having handled the acquisition
When, the processing progress table is updated in the storage unit.
By obtaining scheme according to the message of this specification embodiment, can near real-time in supervisory messages queue middleware
Message queue changes, and only restarts to the process (node) that queue assignment changes, to appoint without influencing entire calculate
Business, is reduced to influence minimum.In addition, by by the processing progress information storage of message queue on SAM Stand Alone Memory, more
It is reliable and stable.
Description of the drawings
This specification embodiment is described in conjunction with the accompanying drawings, and this specification embodiment can be made clearer：
Fig. 1 shows the schematic diagram of the system 100 according to this specification embodiment；
Fig. 2 shows a kind of methods obtaining message from Message Queuing Middleware according to this specification embodiment；
Fig. 3 shows the sequence diagram of the message acquisition method according to this specification embodiment；And
Fig. 4 shows the device 400 that message is obtained according to the slave Message Queuing Middleware of this specification embodiment.
Specific implementation mode
This specification embodiment is described below in conjunction with attached drawing.
Fig. 1 shows the schematic diagram of the system 100 according to this specification embodiment.As shown in Figure 1, system 100 includes message
Queue middleware 101, computing platform 102 and storage unit 103.Message Queuing Middleware 101 is produced for receiving from message
The message of person, and the Message Processing process 12 (node 12) into computing platform 102 provides the message, is, for example, Antq,
That is the Message Queuing Middleware of a kind of suitable ant LDC frameworks of ant gold clothes, including the pull-mode message queue of distribution.
However, Message Queuing Middleware 101 is not limited to pull-mode, such as can also be push-model, point-to-point type, etc.
Deng.
Here message generally includes three theme, main body and attribute important features, and wherein theme defines the class of message
Type, for example, transaction message, click message etc., main body includes mainly the content of message and attribute is that message is arranged in the producer
Custom Attributes.Message Queuing Middleware 101 includes multiple message queues 11, and each message queue has corresponding master
Topic, the corresponding theme of message queue are identical as the theme for the message that the message queue includes.
The computing platform 102 is, for example, the computing platform Kepler of ant exploitation, is calculated including corresponding to one
Multiple consumption nodes 12 (Message Processing process) of task, consumption node 12 are used to obtain from the Message Queuing Middleware 101
Message carries out task processing and storage consumption progress.Here task processing for example, real-time calculating task, such as micro-
Risk is calculated in loan business.The computing platform 102 is not limited to Kepler, for example, it can be small-sized computing unit,
Such as can be a server (that is, a consumption node), can also be the computing platform of large size, as long as it has and task
Handle the computing capability being consistent.And the task processing is not limited to real-time calculating task, can be a plurality of types of
Task, such as asynchronous process, using decoupling, flow peak clipping, log processing.
Message Queuing Middleware 101 for example, by poll allocation algorithm will message queue corresponding with calculating task 11 it is uniform
Consumption node 12 corresponding with calculating task is distributed to, consumption node 12 obtains message from distribution to its message queue 11.Disappear
Expense node 12 periodically (such as every one minute) can judge whether the message queue sum of corresponding theme changes, and distribution
Whether change to its message queue.When distribution changes to its message queue, for example, increasing, deleting, changing
When, which is restarted, to be attached again with the message queue for distributing to it, thus from disappearing for its is distributed to
It ceases queue and receives message.However, Message Queuing Middleware 101 is not limited by polling algorithm assignment messages queue, can be
It can be to any algorithm of the message queue under consumption node equilibrium assignment same subject.
In addition, node 12 before it will obtain message from middleware 101, can read corresponding message from storage unit 103
The Message Processing progress of queue, to draw new a collection of message from the position of untreated message in message queue.And the section
After having handled this batch of message new Message Processing progress can be written in point 12 in storage unit 103.Wherein, the storage
Unit 103 can be the storage unit inside the computing platform 102, or can be the storage list outside the computing platform
Member.
It is described Fig. 2 shows a kind of method obtaining message from Message Queuing Middleware according to this specification embodiment
Method is executed by the process for handling the message in computing platform, including：
In step S21, obtain current distribution information, the current distribution information be currently assigned to the first of the process
The first message queue of quantity is related, wherein the first message queue is the message queue in the Message Queuing Middleware；
In step S22, judge whether the current distribution information changes compared to the original allocation information to prestore, institute
It is related to the second message queue of the second quantity of the process is distributed in the process initiation to state original allocation information,
In, the second message queue is the message queue in the Message Queuing Middleware；And
In step S23, in the case where the current distribution information changes compared to the original allocation information, into
Row is restarted, and connection is established with the first message queue with first quantity, thus from the first message team of first quantity
Row obtain message.
First, in step S21, obtain current distribution information, the current distribution information be currently assigned to the process
The first quantity first message queue it is related, wherein the first message queue is disappearing in the Message Queuing Middleware
Cease queue.
As described above, the Message Queuing Middleware is, for example, Antq middlewares, and the computing platform is, for example,
Kepler computing platforms.In Kepler computing platforms, such as real-time calculating task can be handled, the task for big data into
In the case of row processing, which can be distributed calculating task, i.e., on the multiple servers host of distribution
Reason.And the application example for executing the calculating task run on every host be one of the calculating task into
Journey.The process includes obtaining message from Antq, carrying out multiple mistakes such as data processing, and storage information consumption progress to the message
Journey.The process can be corresponding with the consumer in Message Queuing Middleware-computing platform system.
Above-mentioned real-time calculating task subscribes to one or more message subjects to the Antq, is arranged under each message subject
The quantity of several queues, queue determines that queue is equivalent to container according to message count, wherein being ranked sequentially multiple from message life
The message that production person's sequence obtains.
When real-time calculating task starts, i.e., in each process initiation of the real-time calculating task, each process into
The following processing of row：The entire message queue under the whole themes and the whole theme that the real-time calculating task is subscribed to is obtained, is obtained
The real-time calculating task always into number of passes；According to special algorithm, such as polling algorithm, by the entire message queue to described complete
Portion's process is polled distribution, to obtain distribution to the information of the message queue of its (process), such as the message team
The sums (i.e. described second quantity) of row, theme, the mark of place server, queue identity etc.；And with distribution to its
Message queue establishes connection, to obtain message from the message queue, while above- mentioned information being recorded as the process
Original allocation information.
In the Antq, the message queue under the theme that the calculating task is subscribed to may be according to the feelings of message write-in
Condition is changed, for example, in the case where message speedup is too fast, increases message queue number, in the case where message speedup slows down,
Reduce message queue number etc..The process can periodically (such as every one minute) be obtained about current from Antq with the meter
The information of the corresponding entire message queue of calculation task, and the current distribution information of the process is obtained according to polling algorithm, for example, working as
Sum (that is, described first quantity), theme, the mark of place server, the queue mark of message queue of the preceding distribution to the process
Know etc..
For example, in the case of as shown in Figure 1, message queue corresponding with calculating task is 4, includes in computing platform
Corresponding 3 processes (node 12).In this case, it being distributed through overpolling, first node 12 is assigned to 2 message queues,
2nd is respectively assigned to 1 message queue with third node 12.In another example, corresponding total message queue number is 2
A, corresponding process is 3, then according to polling algorithm, the 1st and the 2nd process respectively distribute a message queue, the 3rd into
Journey is unallocated message queue.That is, the minimum and maximum message queue number at most difference 1 that each course allocation arrives.From
And it ensure that equilibrium assignment of the message queue to process.In addition, according to polling algorithm, can according to process identification (PID) (node identification) come
Uniquely determine the mark of the message queue of its access.That is, the first quantity (message queue that process is currently distributed
Number) and the second quantity (the message queue number that process is distributed on startup) determined according to specific allocation algorithm, for example, being based on
Polling algorithm, according to message queue number and number of nodes, the first quantity and the second quantity can be zero, one or more.
In step S22, judge whether the current distribution information changes compared to the original allocation information to prestore, institute
It is related to the second message queue of the second quantity of the process is distributed in the process initiation to state original allocation information,
In, the second message queue is the message queue in the Message Queuing Middleware.
As described in example above, original allocation information is obtained in process initiation and is obtained when executing this method and is worked as
After preceding distribution information, the two can be compared, to judge whether current distribution information is sent out compared to original allocation information
Changing.For example, total by comparing the message queue sum in current distribution information and the message queue in original allocation information
Number, determines whether current distribution information changes.For another example respective in information and original allocation information by being based on currently distributing
The theme of message queue, the mark of place server, queue identity etc., relatively currently the message queue in distribution information with it is initial
Whether the message queue distributed in information is identical message queue, so that it is determined that currently whether distribution information changes.
In step S23, in the case where the current distribution information changes compared to the original allocation information, into
Row is restarted, and connection is established with the first message queue with first quantity, thus from the first message team of first quantity
Row obtain message.
While determining that the current distribution information of process changes by above-mentioned judgement, the process still with it is described just
The message queue recorded in the distribution information that begins is attached.And in the Kepler platforms, process can on startup with distribution
It is attached to its message queue.Therefore, can only be divided again with current in the process to realize by only restarting specific process
It is assigned to its message queue and establishes connection, and message is obtained from the message queue of the reconnect.So as to pass through the method
Message queue variation of the near real-time ground in (as described above every one minute) monitoring Antq middlewares, and only has queue assignment
The process (node) of variation restarts to connect with currently distributing to its message queue to establish, without influencing entire calculating task,
It is minimum to which influence is reduced to.
In the explanation to the above method, the computing platform is using Kepler computing platforms as an example, however, described
Computing platform is not limited to Kepler computing platforms, can be its such as Jstorm, Storm, Spark Streaming
Its big data computing platform can also be smaller computing platform, such as a server host according to business demand.
In the explanation to the above method, the Message Queuing Middleware is using Antq middlewares as an example, however, institute
State Message Queuing Middleware and be not limited to Antq middlewares, for example, can be Act iveMQ, RabbitMQ, Kafka,
Other Message Queuing Middlewares such as RocketMQ.
In the explanation to the above method, being handled in the computing platform for task is shown using the conduct of real-time calculating task
Example, however, the task be not limited to from, can be a plurality of types of tasks, such as asynchronous process, using decoupling, flow peak clipping,
Log processing etc..
In the specification to the above method, the message queue that each course allocation arrives is calculated using polling algorithm, however, right
The calculating of message queue distribution is not limited to polling algorithm, can be such as average with the algorithm of any other achievable equilibrium assignment
Allocation algorithm, mark post algorithm etc..
In one embodiment, the computing platform includes only a server host, that is to say, that for computing platform
In a task, include only a corresponding process in the computing platform.In this case, the process be set to from
At least one message queue corresponding at least one specific subject in the Message Queuing Middleware obtains message, and,
The current distribution information is related to current at least one message queue.That is, in above-mentioned real-time calculating task
In the case of, process entire message queue corresponding with whole themes that the real-time calculating task is subscribed to connects.Wherein, institute
It is at least one theme that the real-time calculating task is subscribed to state specific subject.To which the current distribution information of the process is
With the relevant information of entire message queue, for example, the sum of these message queues, theme, place server identification, queue mark
Know etc..
In one embodiment, identically as the explanation to the above method, the computing platform is executed by one group of process
Calculating task.That is, the entire message queue under whole themes that one group of process is subscribed to the calculating task is connect.It is different
It is, in this embodiment, in each process：From the Message Queuing Middleware obtain the first information, the first information with
Current at least one message queue is related；Judge whether the first information changes compared to the second information to prestore,
Wherein, second information is related at least one message queue in the process initiation；And described first
In the case where information changes compared to the second information, it is based on the first information, obtains the current distribution information.Institute
It states in the case where first information does not change compared to the second information, the method will be terminated.That is, compared to above
In description, increase current entire message queue (i.e. the first information) to the task and initial entire message queue
The comparison of (i.e. the second information), and only when the first information changes compared to second information, just carry out follow-up
Comparison to current the distribution information and original allocation information of process.To save computing resource, efficiency is improved.
In one embodiment, carry out it is above-mentioned to the restarting of process after, for it is described restart before and it is described heavy
The third message queue all being connect with the process after opening, obtain in the memory unit the processing of the third message queue into
Spend information, and the multiple message that will be obtained from the third message queue according to the processing progress information, determination.Wherein,
The processing progress information is obtained from the processing progress table in storage unit, and the processing progress table includes following field
Value：Theme, server identification, message queue mark and message identifier.In addition, determination will be from third message team
After arranging the multiple message obtained, when having handled multiple message of the acquisition, the place is updated in the storage unit
Manage program.
The storage unit can be independently of the computing platform and third party's storage unit of Message Queuing Middleware,
So that it is guaranteed that processing progress information stability is reliable.For example, in the example of above-mentioned Antq middlewares-Kepler platforms, calculate flat
Process in platform draws message by drawing-die mode from Antq middlewares.Process every time from particular message queue draw message before,
The processing progress table in storage unit can be read, which is found by theme, server identification, message queue mark
The position being shown in the table, and message identifier corresponding with the particular message queue in table is read, which is the message team
The deviant namely processing progress of row.To which the process disappears according to the processing progress from particular message queue drawing
Breath.For example, the process reads the message identifier of the particular message queue recorded in the processing progress table from storage unit
For the 100th article of message, then process pulls 10 articles of such as the 101-110 articles and disappears when drawing message from the particular message queue
Breath.Process updates the processing progress table, i.e., the processing is written in message identifier 110 after having handled 10 message
Position corresponding with the particular message queue in program.
When process is restarted, for it is described restart before and it is described restart after the third that is all connect with the process disappear
Queue is ceased, since the third message queue has been connect with the process before the restart, record has in the memory unit
The processing progress (deviant) of the message queue.Therefore, process can be obtained by reading the processing progress table in storage unit
The processing progress, and reason progress obtains message from third message queue according to this.So as to avoid missing information or repetition
Handle message.For the 4th message queue only being connect after restarting with the process, i.e. the 4th message queue is to restart
Newly-increased later, then process draws message since first message of the 4th message queue.
Fig. 3 shows the sequence diagram of the message acquisition method according to this specification embodiment.It is illustrated in figure 3 computing platform
In calculating task the case where including multiple processes, for one of process, first from Message Queuing Middleware (figure
In shown with Antq examples) obtain total message queue information currently corresponding with the calculating task.As described above, total message
The message queue that queuing message includes can correspond to one or more themes subscribed to by the calculating task, and total message
Queuing message includes the sum of corresponding message queue, place server identification, queue identity etc..Later, process is current total by this
Message queue information is compared (compared with 1) with total message queue information when prestoring in process initiation, current total to judge
Whether message queue information changes.In the case where current total message queue information changes, obtained by pre-defined algorithm
The information of the currently assigned message queue of process is taken, the pre-defined algorithm is, for example, polling algorithm, mean allocation algorithm etc..
Later, process (compares 2) compared with it currently to be distributed to distribution information of the information when process initiation to prestore, to judge process
Whether current distribution information changes.In the case where current distribution information of process changes, process is restarted, with
It establishes and connects with the message queue of the distribution after variation.
After restarting, process reads the processing progress (message offset for each message queue that it is connected from storage unit
Amount), the position of message will be read from message queue to obtain.For in the Unrecorded message queue of storage unit, that is, weighing
The message queue that Qi Qianwei is connect with the process, process read message since first message of the message queue.Then, process
A collection of message is obtained from Message Queuing Middleware according to processing progress, and handles this batch of message.Handling a collection of message
Later, process is by processing progress write storage unit.
Sequence diagram shown in Fig. 3 is exemplary, and the sequence is not limited to according to the method for this specification embodiment.For example,
In the case of the calculating task only includes a process, current total message queue that process is obtained from Message Queuing Middleware
Information is the current distribution information of process, so as to save the first two steps in figure.In addition, calculating task include it is multiple into
In the case of journey, the step of can also saving comparison 1 in figure, and directly it is compared 2.
Sequence diagram shown in Fig. 3 can be periodically executed, for example, it is per minute execute it is primary, to ensure that near real-time supervisory messages
Message queue situation of change in queue middleware, and the connection with message queue that timely updates.
Fig. 4 shows the device 400 that message is obtained according to the slave Message Queuing Middleware of this specification embodiment, the dress
It sets and is implemented by the process for handling the message in computing platform, including：
First acquisition unit 41, is configured to, and obtains current distribution information, the current distribution information be currently assigned to institute
The first message queue for stating the first quantity of process is related, wherein the first message queue is the Message Queuing Middleware
In message queue；
Judging unit 42, is configured to, and judges whether the current distribution information is sent out compared to the original allocation information to prestore
Changing, the second message team of the original allocation information and the second quantity that the process is distributed in the process initiation
Row are related, wherein the second message queue is the message queue in the Message Queuing Middleware；And
Restart unit 43, is configured to, it is changed compared to the original allocation information in the current distribution information
In the case of, restarted, connection is established with the first message queue with first quantity, to from the of first quantity
One message queue obtains message.
In one embodiment, described from the device that Message Queuing Middleware obtains message, the process is one group
There is corresponding theme, one group of process to be set for a process in process, the message queue in the Message Queuing Middleware
It is set to from least one message queue corresponding at least one specific subject in the Message Queuing Middleware and obtains message,
The first acquisition unit 41 further includes：First obtains subelement 411, is configured to, and the is obtained from the Message Queuing Middleware
One information, the first information are related to current at least one message queue；And second obtain subelement 412, match
It is set to, is based on the first information, obtain the current distribution information.
In one embodiment, described from the device that Message Queuing Middleware obtains message, the process is one group
A process in process, the message queue in the Message Queuing Middleware have corresponding theme, one group of process quilt
It is set as disappearing from least one message queue acquisition corresponding at least one specific subject in the Message Queuing Middleware
Breath, the first acquisition unit 41 further include：First obtains subelement 411, is configured to, and is obtained from the Message Queuing Middleware
The first information, the first information are related to current at least one message queue；Judgment sub-unit 413, is configured to, and sentences
Whether the first information that breaks changes compared to the second information, wherein second information and the institute in the process initiation
It is related to state at least one message queue；And second obtain subelement 412, be configured to, in the first information compared to second
In the case where information changes, it is based on the first information, obtains the current distribution information.
In one embodiment, the device 400 that message is obtained from Message Queuing Middleware further includes that second obtains list
Member 44, is configured to, after being restarted, for it is described restart before and it is described restart after all connect with the process
Third message queue obtains the processing progress information of the third message queue in the memory unit；And determination unit 45, match
It is set to, according to the processing progress information, determination will be from multiple message of third message queue acquisition.
In one embodiment, the device 400 that message is obtained from Message Queuing Middleware further includes updating unit
46, it is configured to, the more of the acquisition will be being handled after multiple message that the third message queue obtains in determination
When a message, the processing progress table is updated in the storage unit.
By obtaining scheme according to the message of this specification embodiment, can near real-time in supervisory messages queue middleware
Message queue changes, and only restarts to the process (node) that queue assignment changes, to appoint without influencing entire calculate
Business, is reduced to influence minimum.In addition, by by the processing progress information storage of message queue on SAM Stand Alone Memory, more
It is reliable and stable.
Those of ordinary skill in the art should further appreciate that, be described in conjunction with the embodiments described herein
Each exemplary unit and algorithm steps, can be realized with electronic hardware, computer software, or a combination of the two, in order to clear
Illustrate to Chu the interchangeability of hardware and software, generally describes each exemplary group according to function in the above description
At and step.These functions hold track with hardware or software mode actually, depending on technical solution specific application and set
Count constraints.Those of ordinary skill in the art can be described to be realized using distinct methods to each specific application
Function, but this realization is it is not considered that exceed scope of the present application.
The step of method described in conjunction with the examples disclosed in this document or algorithm, can hold track with hardware, processor
Software module or the combination of the two implement.Software module can be placed in random access memory (RAM), memory, read-only storage
Device (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology neck
In any other form of storage medium well known in domain.
Above-described specific implementation mode has carried out further the purpose of the present invention, technical solution and advantageous effect
It is described in detail, it should be understood that the foregoing is merely the specific implementation mode of the present invention, is not intended to limit the present invention
Protection domain, all within the spirits and principles of the present invention, any modification, equivalent substitution, improvement and etc. done should all include
Within protection scope of the present invention.
Claims (22)
1. a kind of method obtaining message from Message Queuing Middleware, the method in computing platform by being used to handle described disappear
The process of breath executes, including：
Obtain current distribution information, the first message of current the distribution information and the first quantity for being currently assigned to the process
Queue is related, wherein the first message queue is the message queue in the Message Queuing Middleware；
Judge whether the current distribution information changes compared to the original allocation information to prestore, the original allocation information
It is related to the second message queue of the second quantity of the process is distributed in the process initiation, wherein described second disappears
Breath queue is the message queue in the Message Queuing Middleware；And
In the case where the current distribution information changes compared to the original allocation information, restarted, with institute
Connection is established in the first message queue for stating the first quantity, to obtain message from the first message queue of first quantity.
2. the method according to claim 1 for obtaining message from Message Queuing Middleware, wherein among the message queue
Message queue in part has a corresponding theme, the process be arranged to from the Message Queuing Middleware at least one
The corresponding at least one message queue of a specific subject obtains message, and, the current distribution information with it is current it is described extremely
A few message queue correlation.
3. it is according to claim 1 from Message Queuing Middleware obtain message method, wherein the process be one group into
There is corresponding theme, one group of process to be set for a process in journey, the message queue in the Message Queuing Middleware
To obtain message from least one message queue corresponding at least one specific subject in the Message Queuing Middleware,
The current distribution information of acquisition includes, and the first information is obtained from the Message Queuing Middleware, the first information with
Current at least one message queue is related, and, it is based on the first information, obtains the current distribution information.
4. the method according to claim 3 for obtaining message from Message Queuing Middleware, wherein believe based on described first
Breath, obtaining the current distribution information includes, and according to polling algorithm, is based on the first information, obtains the current distribution letter
Breath.
5. it is according to claim 1 from Message Queuing Middleware obtain message method, wherein the process be one group into
There is corresponding theme, one group of process to be set for a process in journey, the message queue in the Message Queuing Middleware
It is set to from least one message queue corresponding at least one specific subject in the Message Queuing Middleware and obtains message,
Currently distribution information includes for the acquisition：
The first information, the first information and current at least one message queue are obtained from the Message Queuing Middleware
It is related；
Judge whether the first information changes compared to the second information to prestore, wherein second information with it is described into
At least one message queue when Cheng Qidong is related；And
In the case where first information changes compared to the second information, it is based on the first information, is worked as described in acquisition
Preceding distribution information.
6. the method according to claim 1 for obtaining message from Message Queuing Middleware, wherein the current distribution information
Value including first quantity and the theme of each message queue, place clothes in the first message queue of first quantity
The mark and queue identity of business device, and, the original allocation information includes the value of second quantity and second quantity
Second message queue in the theme of each message queue, place server mark and queue identity.
7. the method according to claim 1 for obtaining message from Message Queuing Middleware, further includes carrying out restarting it
Afterwards, for it is described restart before and it is described restart after the third message queue that is all connect with the process, in storage unit
The middle processing progress information for obtaining the third message queue, and according to the processing progress information, determination will be from described the
Multiple message that three message queues obtain.
8. the method according to claim 7 for obtaining message from Message Queuing Middleware, wherein the processing progress information
It is obtained from the processing progress table in storage unit, the processing progress table includes the value of following field：Theme, server mark
Know, message queue identifies and message identifier.
9. it is according to claim 8 from Message Queuing Middleware obtain message method, further include, determination will be from institute
After the multiple message for stating the acquisition of third message queue, when having handled multiple message of the acquisition, in the storage unit
The middle update processing progress table.
10. the method according to claim 7 for obtaining message from Message Queuing Middleware, wherein the storage unit is only
Stand on third party's storage unit of the Message Queuing Middleware and the computing platform.
11. the method according to claim 1 for obtaining message from Message Queuing Middleware, wherein the computing platform is
Kepler computing platforms.
12. a kind of device obtaining message from Message Queuing Middleware, described device is by described for handling in computing platform
The process of message is implemented, including：
First acquisition unit is configured to, and obtains current distribution information, the current distribution information be currently assigned to the process
The first quantity first message queue it is related, wherein the first message queue is disappearing in the Message Queuing Middleware
Cease queue；
Judging unit is configured to, and judges whether the current distribution information changes compared to the original allocation information to prestore,
The original allocation information is related to the second message queue of the second quantity of the process is distributed in the process initiation,
Wherein, the second message queue is the message queue in the Message Queuing Middleware；And
Restart unit, is configured to, in the case where the current distribution information changes compared to the original allocation information,
Restarted, connection is established with the first message queue with first quantity, thus from the first message of first quantity
Queue obtains message.
13. the device according to claim 12 for obtaining message from Message Queuing Middleware, wherein in the message queue
Between message queue in part there is corresponding theme, the process be arranged to from the Message Queuing Middleware at least
The corresponding at least one message queue of one specific subject obtains message, and, the current distribution information and currently described
At least one message queue is related.
14. the device according to claim 12 for obtaining message from Message Queuing Middleware, wherein the process is one group
There is corresponding theme, one group of process to be set for a process in process, the message queue in the Message Queuing Middleware
It is set to from least one message queue corresponding at least one specific subject in the Message Queuing Middleware and obtains message,
The first acquisition unit further includes：
First obtains subelement, is configured to, and the first information is obtained from the Message Queuing Middleware, the first information with it is current
At least one message queue it is related；And second obtain subelement, be configured to, be based on the first information, obtain institute
State current distribution information.
15. the device according to claim 14 for obtaining message from Message Queuing Middleware, wherein described second obtains son
Unit is additionally configured to, and according to polling algorithm, is based on the first information, is obtained the current distribution information.
16. the device according to claim 12 for obtaining message from Message Queuing Middleware, wherein the process is one group
A process in process, the message queue in the Message Queuing Middleware have corresponding theme, one group of process quilt
It is set as disappearing from least one message queue acquisition corresponding at least one specific subject in the Message Queuing Middleware
Breath,
The first acquisition unit further includes：
First obtains subelement, is configured to, and the first information is obtained from the Message Queuing Middleware, the first information with it is current
At least one message queue it is related；
Judgment sub-unit is configured to, and judges whether the first information changes compared to the second information to prestore, wherein described
Second information is related at least one message queue in the process initiation；And
Second obtains subelement, is configured to, and in the case where first information changes compared to the second information, is based on institute
The first information is stated, the current distribution information is obtained.
17. the device according to claim 12 for obtaining message from Message Queuing Middleware, wherein the current distribution letter
Cease the theme of each message queue, place in the first message queue of the value and first quantity that include first quantity
The mark and queue identity of server, and, the original allocation information includes the value of second quantity and second number
The mark and queue identity of the theme of each message queue, place server in the second message queue of amount.
18. the device according to claim 12 for obtaining message from Message Queuing Middleware, further includes, second obtains list
Member is configured to, after being restarted, for it is described restart before and it is described restart after all connect with the process the
Three message queues obtain the processing progress information of the third message queue in the memory unit；And determination unit, configuration
For according to the processing progress information, determination will be from multiple message of third message queue acquisition.
19. the device according to claim 18 for obtaining message from Message Queuing Middleware, wherein the processing progress letter
It ceases and is obtained from the processing progress table in storage unit, the processing progress table includes the value of following field：Theme, server mark
Know, message queue identifies and message identifier.
20. the device according to claim 19 for obtaining message from Message Queuing Middleware, further includes that updating unit is matched
It is set to, will disappear having handled the multiple of the acquisition after multiple message that the third message queue obtains in determination
When breath, the processing progress table is updated in the storage unit.
21. the device according to claim 18 for obtaining message from Message Queuing Middleware, wherein the storage unit is
Independently of third party's storage unit of the Message Queuing Middleware and the computing platform.
22. the device according to claim 12 for obtaining message from Message Queuing Middleware, wherein the computing platform is
Kepler computing platforms.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810496393.8A CN108683612B (en) | 2018-05-22 | 2018-05-22 | Message acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810496393.8A CN108683612B (en) | 2018-05-22 | 2018-05-22 | Message acquisition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108683612A true CN108683612A (en) | 2018-10-19 |
| CN108683612B CN108683612B (en) | 2022-01-18 |
Family
ID=63807616
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810496393.8A Active CN108683612B (en) | 2018-05-22 | 2018-05-22 | Message acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108683612B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143082A (en) * | 2018-11-06 | 2020-05-12 | 航天信息股份有限公司 | Method and device for reading RabbitMQ data |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8898520B1 (en) * | 2012-04-19 | 2014-11-25 | Sprint Communications Company L.P. | Method of assessing restart approach to minimize recovery time |
| US8959530B1 (en) * | 2013-05-07 | 2015-02-17 | Sprint Communications Company L.P. | Messaging middleware processing thread count based events |
| CN106549796A (en) * | 2016-09-27 | 2017-03-29 | 努比亚技术有限公司 | Resource control method and host node that a kind of firmware space is downloaded |
| CN106657349A (en) * | 2016-12-29 | 2017-05-10 | 上海理想信息产业（集团）有限公司 | Message subscription processing device, system and method |
| CN106789741A (en) * | 2016-12-26 | 2017-05-31 | 北京奇虎科技有限公司 | The consuming method and device of message queue |
| CN107423145A (en) * | 2017-07-11 | 2017-12-01 | 北京潘达互娱科技有限公司 | A kind of method and apparatus for avoiding information drop-out |
| CN107992394A (en) * | 2017-11-28 | 2018-05-04 | 顺丰科技有限公司 | Monitoring method, device, system, equipment and the storage medium of real time data processing |
-
2018
- 2018-05-22 CN CN201810496393.8A patent/CN108683612B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8898520B1 (en) * | 2012-04-19 | 2014-11-25 | Sprint Communications Company L.P. | Method of assessing restart approach to minimize recovery time |
| US8959530B1 (en) * | 2013-05-07 | 2015-02-17 | Sprint Communications Company L.P. | Messaging middleware processing thread count based events |
| CN106549796A (en) * | 2016-09-27 | 2017-03-29 | 努比亚技术有限公司 | Resource control method and host node that a kind of firmware space is downloaded |
| CN106789741A (en) * | 2016-12-26 | 2017-05-31 | 北京奇虎科技有限公司 | The consuming method and device of message queue |
| CN106657349A (en) * | 2016-12-29 | 2017-05-10 | 上海理想信息产业（集团）有限公司 | Message subscription processing device, system and method |
| CN107423145A (en) * | 2017-07-11 | 2017-12-01 | 北京潘达互娱科技有限公司 | A kind of method and apparatus for avoiding information drop-out |
| CN107992394A (en) * | 2017-11-28 | 2018-05-04 | 顺丰科技有限公司 | Monitoring method, device, system, equipment and the storage medium of real time data processing |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111143082A (en) * | 2018-11-06 | 2020-05-12 | 航天信息股份有限公司 | Method and device for reading RabbitMQ data |
| CN111143082B (en) * | 2018-11-06 | 2023-05-16 | 航天信息股份有限公司 | Method and device for reading RabbitMQ data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108683612B (en) | 2022-01-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108848039B (en) | Server, message distribution method and storage medium | |
| CN104503845B (en) | A kind of task distribution method and system | |
| CN108287756A (en) | A kind of method and device of processing task | |
| CN110213371A (en) | Information consumption method, apparatus, equipment and computer storage medium | |
| CN108733464A (en) | A kind of method and device of the scheduling scheme of determining calculating task | |
| US20170329655A1 (en) | Method and apparatus of unloading out of memory processing flow to user space | |
| CN110300188B (en) | Data transmission system, method and device | |
| CN111143039B (en) | Scheduling method and device of virtual machine and computer storage medium | |
| CN101146127A (en) | A client buffer update method and device in distributed system | |
| CN106462459A (en) | Managing metadata for distributed processing system | |
| CN108683612A (en) | A kind of message acquisition method and device | |
| JP2010128597A (en) | Information processor and method of operating the same | |
| CN110519354A (en) | A kind of distributed objects storage system and its method for processing business and storage medium | |
| CN104410511A (en) | Server management method and system | |
| CN116455830A (en) | Method for realizing high-availability distributed QOS of storage gateway | |
| KR20170031131A (en) | Stream based event processing utilizing virtual streams and processing agents | |
| CN113946362B (en) | Consumption data processing method and storage medium | |
| CN110809050A (en) | Personalized push system and method based on streaming computing | |
| CN115904669A (en) | Task scheduling method, system, electronic device and computer readable storage medium | |
| WO2022267395A1 (en) | Payment request processing method and system | |
| CN115660344A (en) | User scheduling method and device, electronic equipment and storage medium | |
| CN114860726A (en) | Database storage cold-hot separation method, device, equipment and readable storage medium | |
| CN107493338A (en) | A kind of concurrent access method suitable for hydrologic forecast information system | |
| CN106161572A (en) | The call method of a kind of service and server | |
| CN110460663A (en) | Data distributing method, device, server and storage medium between distributed node |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200925 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Innovative advanced technology Co.,Ltd. Address before: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant before: Advanced innovation technology Co.,Ltd. Effective date of registration: 20200925 Address after: Cayman Enterprise Centre, 27 Hospital Road, George Town, Grand Cayman Islands Applicant after: Advanced innovation technology Co.,Ltd. Address before: A four-storey 847 mailbox in Grand Cayman Capital Building, British Cayman Islands Applicant before: Alibaba Group Holding Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |