CN106354722A - Message processing method and device for stream-oriented computation system - Google Patents
Message processing method and device for stream-oriented computation system Download PDFInfo
- Publication number
- CN106354722A CN106354722A CN201510413095.4A CN201510413095A CN106354722A CN 106354722 A CN106354722 A CN 106354722A CN 201510413095 A CN201510413095 A CN 201510413095A CN 106354722 A CN106354722 A CN 106354722A
- Authority
- CN
- China
- Prior art keywords
- message
- write
- memory element
- unit
- service processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G06F11/1453—Management of the data involved in backup or backup restore using de-duplication of the data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/50—Queue scheduling
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer And Data Communications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a message processing method and device for a stream-oriented computation system. The method comprises the steps that 1, a message is received; 2, whether the message is stored in a first storage unit or not is judged, if yes, the message is not processed, and if not, the step 3 is executed; 3, whether a business processing result corresponding to the message exists in a second storage unit or not is judged, if yes, the step 6 is executed, and if not, the step 4 is executed; 4, business processing is conducted on the message; 5, the message and the business processing result thereof are written in the second storage unit; 6, the message is written in the first storage unit. Accordingly, it can be guaranteed that business processing is only conducted on the message in the stream-oriented computation system once, and the performance and the efficiency of the system are improved.
Description
[technical field]
The present invention relates to Computer Applied Technology field, at a kind of message of streaming computing system
Reason method and apparatus.
[background technology]
Jstorm is the real-time streaming Computational frame with reference to storm, in network io, threading model, resource
Do sustained improvement in scheduling, availability and stability, used by more and more enterprises.Streaming calculates
The rank of the current Message Processing in field is generally and processes at least one times with process at most once, needs a kind of letter badly
Single efficient mode is ensureing that message is only processed once exactly.
Jstrom has been provided for acker mechanism to realize at least processing Message Processing circle once in itself
Not it is therefore desirable to the key problem solving is, how repeated data is carried out with duplicate removal, that is, for processing
Message no longer processed.
[content of the invention]
In view of this, the invention provides a kind of message treatment method of streaming computing system and device, with
Be easy to reduce to message repeat process.
Concrete technical scheme is as follows:
The invention provides a kind of message treatment method of Stream Processing system, the method includes:
S1, reception message；
Whether s2, judge to be stored with the first memory element described message, if it is, not to this message
Processed；Otherwise, execute s3；
S3, judge in the second memory element, whether there is the corresponding service processing result of described message, such as
Fruit is to execute s6, otherwise executes s4；
S4, Business Processing is carried out to described message；
S5, described message and its service processing result are write the second memory element；
S6, by described message write the first memory element.
According to the present invention one preferred implementation, the content in described first memory element is mapped to the grand filtration of cloth
Device bloomfilter；
Also included before described s2: judge whether described message hits bloomfilter, if not, execution
Described s3；If it is, executing described s2.
According to the present invention one preferred implementation, when executing described s4, if in described second memory element
There is the messaging list with described message with identical calculations mark, then using with described message, there is phase
With the corresponding service processing result of messaging list calculating mark, Business Processing is carried out to described message；
In described s5, described message is added described messaging list, and the service processing result using s4
Update the corresponding service processing result of messaging list described in described second memory element；
When executing described s6, the described messaging list comprising described message is write the first memory element.
According to the present invention one preferred implementation, also include between described s4 and described s5: disappear described
Breath and its corresponding service processing result put into write queue；
Using the content execution described s5 and described s6 in said write queue.
According to the present invention one preferred implementation, execute described s5 and institute using the content in said write queue
State s6 to include:
S51, from said write queue take out a cache node, comprise message in one cache node
And its corresponding service processing result；
S52, by described cache node content write described second memory element；
S53, the message in described cache node is put into duplicate removal storage write queue；
S54, store write queue from duplicate removal and take out message write the first memory element.
According to the present invention one preferred implementation, also include after described s2 and between described s3:
Judge whether there is described message, such as in said write queue and the storage write queue of described duplicate removal
Fruit is then not carry out Business Processing to described message, otherwise continues executing with described s3.
According to the present invention one preferred implementation, also included before described s51 and described s52:
Judge whether there is the comprised message tool with described cache node in described duplicate removal storage write queue
There is the message of identical calculations mark, if it is, described cache node is put back to said write queue, wait
Re-execute described s51；Otherwise, continue executing with described s52.
Present invention also offers a kind of message processing apparatus of streaming computing system, this device includes:
Receiving unit, for receiving message；
First judging unit, for the described message that judges whether to be stored with the first memory element, if
Judged result is yes, no longer described message is processed；
Second judging unit, for when the judged result of described first judging unit is no, judging described the
Whether there is the corresponding service processing result of described message in two memory element；
Processing unit, for when the judged result of described second judging unit is for being, not entering to described message
Row Business Processing；When the judged result of described second judging unit is no, described message is carried out at business
Reason；
First writing unit, for writing the second memory element by the result of described message；
Second writing unit, for, after described first writing unit completes write, or sentencing described second
When the judged result of disconnected unit is no, described message is write the first memory element.
According to the present invention one preferred implementation, the content in described first memory element is mapped to
bloomfilter；
This device also includes: the 3rd judging unit, for after described receiving unit receives described message,
Judge whether described message hits bloomfilter, if not, triggering described second judging unit to execute judgement
Operation；If it is, the operation that the described first judging unit execution of triggering judges.
According to the present invention one preferred implementation, described processing unit, if single specifically for the described second storage
There is the messaging list with described message with identical calculations mark in unit, then had using with described message
There is the corresponding service processing result of messaging list of identical calculations mark, Business Processing is carried out to described message；
Described first writing unit, specifically for adding described messaging list by described message, and using described
This service processing result of processing unit updates the corresponding industry of messaging list described in described second memory element
Business result；
Described second writing unit, specifically for depositing the described messaging list write first comprising described message
Storage unit.
According to the present invention one preferred implementation, described processing unit, it is additionally operable to described message and its correspondence
Service processing result put into write queue；
Described first writing unit and described second writing unit are write using the content execution in said write queue
The operation entering.
According to the present invention one preferred implementation, described first writing unit, specifically for from said write queue
One cache node of middle taking-up, comprises message and its corresponding service processing result in one cache node,
Content in described cache node is write described second memory element, the message in described cache node is put
Enter duplicate removal storage write queue；
Described second writing unit, deposits specifically for storing to take out message in write queue and write first from duplicate removal
Storage unit.
According to the present invention one preferred implementation, this device also includes:
4th judging unit, for when the judged result of described first judging unit is no, judging said write
Queue and described go to re-write in queue whether there is described message, if it is, not to described message
Carry out Business Processing；Otherwise, the operation that the described second judging unit execution of triggering judges.
According to the present invention one preferred implementation, described first writing unit is by the content in described cache node
Before writing described second memory element, it is additionally operable to judge whether deposited in described duplicate removal storage write queue
There is the message of identical calculations mark in message comprised with described cache node, if it is, will be described slow
Deposit node and put back to said write queue, wait retrieves；Otherwise, continue executing with described cache node
Content write described second memory element operation.
As can be seen from the above technical solutions, message is single together with result corresponding write the second storage
Unit, even if message write the first memory element failure, when receiving identical message next time, due to second
There is the corresponding service processing result of this message it is not necessary to again industry be entered to this message in memory element
Business process, only by this message write the first memory element can it is ensured that message only by Business Processing once,
Improve systematic function and efficiency.
[brief description]
Fig. 1 is a process chart provided in an embodiment of the present invention；
Fig. 2 is the flow chart of message treatment method provided in an embodiment of the present invention；
Fig. 3 is the write flow process of calculating storaging unit provided in an embodiment of the present invention and duplicate removal memory element
Figure；
Fig. 4 is the structure chart of message processing apparatus provided in an embodiment of the present invention.
[specific embodiment]
In order that the object, technical solutions and advantages of the present invention are clearer, below in conjunction with the accompanying drawings and specifically
Embodiment describes the present invention.
In the present invention using two memory element: calculating storaging unit (the second memory element) and duplicate removal
Memory element (the first memory element).Wherein, calculating storaging unit is used for storing to the business of message
Reason result, duplicate removal memory element storage carried out the message of Business Processing, and was used for doing duplicate removal to avoid message
Repetition Business Processing.
Upon receiving a message, handling process can as shown in fig. 1, including below scheme:
Whether in 101, judge to be stored with duplicate removal memory element the message that this receives, if
It is no longer this message to be processed；Otherwise, 102 are executed.
In 102, Business Processing is carried out to this message.
The Business Processing that message is carried out being related in embodiments of the present invention can be needed by specific business
Ask to determine, history result may be utilized, if necessary to using history result, then in industry
When business is processed, history result can be obtained first from calculating storaging unit, then using acquisition
History result is processed to this message.Under some business demands, at business that message is carried out
Reason can be the calculating being carried out using message, for example, when message 2 is that blanket order amount to a class user is entered
Row statistics, if there is the blanket order amount that message 1 is also to a class user before this counted, that
The calculating knot of message 1 is obtained for message 2 when being calculated it is possible to from calculating storaging unit
Really, obtain message after this result of calculation being added with the order volume of a class user comprising in message 2
2 result of calculation.
In 103, service processing result is write calculating storaging unit.
In 104, this message is write duplicate removal memory element.
When there is a new message, again go to 102 to start to execute.
Said method can reduce to a certain extent to message repeat process, but if execution 103
Afterwards, system faults such as machine of delaying are occurred to lead to 104 to execute failure when 104, this will result in industry
Business result write calculating storaging unit success, but message write duplicate removal memory element failure.This feelings
Under condition, can still result in message and repeated Business Processing, thus affecting systematic function and efficiency.For example, when
When receiving this message again, judge not existing this message in duplicate removal memory element, then again to this message
Carry out Business Processing, and service processing result is write calculating storaging unit, this message write duplicate removal is deposited
Storage unit.
For above-mentioned problem, the embodiment of the present invention further provides a solution, and it executes stream
Journey can as shown in Figure 2, upon receiving a message, execution following steps:
Whether in 201, judge to be stored with duplicate removal memory element the message that this receives, if
It is no longer this message to be processed；Otherwise, 202 are executed.
In 202, judge in calculating storaging unit, whether there is this message corresponding Business Processing knot
Really, if it is, no longer this message is carried out with Business Processing, direct execution step 205；Otherwise, execute
203.
When whether there is the corresponding service processing result of this message in judging calculating storaging unit, can
Whether there is service processing result in calculating storaging unit to determine first, then pass through to judge to calculate storage
Whether there is the message receiving in unit to realize.If existed at business in calculating storaging unit
There is, in reason result and calculating storaging unit, the message that this receives it is determined that calculating storaging unit
In there is the corresponding service processing result of this message, again business need not be carried out to the message receiving
Process.
In 203, Business Processing is carried out to this message.
In 204, by corresponding with the service processing result of this message for this message write calculating storaging unit.
In 205, this message is write duplicate removal memory element.
It is exemplified below with several scenes:
Assume for the first time to receive message a, judge in duplicate removal memory element that this message and calculating is not deposited
There is not the corresponding service processing result of this message in storage unit, then Business Processing carried out to this message yet,
By the service processing result of message a and message a corresponding write calculating storaging unit, and message a is write
Enter duplicate removal memory element.
Assume the write duplicate removal memory element failure of message a.
Receive message a again, judge there is not this message in duplicate removal memory element, but calculate storage
There is the corresponding service processing result of this message a in unit, then again industry need not have been entered to this message
Business is processed, and directly message a is write duplicate removal memory element.Obviously, this mode avoids to message a
Carry out Business Processing and write calculating storaging unit again, improve systematic function and efficiency.
In addition, in order to improve efficiency, can be by the content map in duplicate removal memory element to bloomfilter
(Bloom filter), when receiving message, first determines whether whether this message hits bloomfilter,
If not, illustrating that this message does not exist in duplicate removal memory element, then in the relaying of execution flow process shown in Fig. 2
Continuous execution step 202；If hit bloomfilter, illustrate may be stored with duplicate removal memory element
This message, then continue execution step 201 in execution flow process shown in Fig. 2.
In flow process shown in above-mentioned Fig. 2, it is understood that there may be such a scene, the Business Processing of message a according to
Rely the service processing result in message b, in this case, message a and message b just have identical
Calculate key (calculate mark), such as certain message is that the order volume to user u adds up, need by
The order volume of user u comprising in this message with using the user u's of the service processing result of message before
Order volume is added up, and these message all have identical and calculate key, and this calculating key can be using use
Mark of family u etc..For this scene, in order to more accurately ensure duplicate removal, to message in 203
When carrying out Business Processing, if existed in calculating storaging unit with the message that this receives have identical
Calculate the messaging list of key, then with the message receiving, there are identical calculations using in calculating storaging unit
The corresponding service processing result of messaging list of key, carries out Business Processing to the message receiving.204
In, this message receiving is added above-mentioned messaging list, and is updated using this service processing result
The corresponding service processing result of this messaging list in calculating storaging unit.To comprise to receive in 205
The above-mentioned messaging list write duplicate removal memory element of message.
Give one example:
Assume to receive message a, judge not comprising message a in duplicate removal memory element, in calculating storaging unit
Do not comprise message a, then Business Processing is carried out to this message a, obtains service processing result yet, should
Message a and its corresponding service processing result a write calculating storaging unit, now in calculating storaging unit
Message a is only comprised in the corresponding messaging list of this service processing result a；By the write duplicate removal storage of message a
Unit.
Receive message b on this basis again, judge not comprising message b in duplicate removal memory element, calculate
Also do not comprise message b in memory element, but with message b, there is phase due to containing in calculating storaging unit
With the messaging list calculating key, therefore, offseted using the service processing result a in calculating storaging unit
Breath b carries out Business Processing, obtains service processing result b.Message b is added the upper of calculating storaging unit
State messaging list, now comprise message a and message b in messaging list, using service processing result b more
New business result a, now comprises the messaging list corresponding Business Processing knot of message a and message b
Fruit is service processing result b.Message a and message b are write duplicate removal memory element.
If receive message a, the write duplicate removal memory element failure of message a then can cause to calculate
Comprise message a in memory element, and in duplicate removal memory element, do not comprise message a.So on this basis such as
Fruit receives message b, is capable of the Business Processing of message b by above-mentioned flow process, and by message a and
Message b writes duplicate removal memory element, that is, when receiving message b, message a benefit is written with duplicate removal and deposits
Storage unit.Even if subsequently receiving message a again it is also possible to ensure this message a will not be repeated
Business Processing.
In fig. 2 after shown step 203, the service processing result of this message and message can be put
Enter to write queue.Further, since there may be the mode of multiple threads in flow-type business processing system,
Therefore, before execution step 202, can first determine whether write queue and duplicate removal storage write queue (after
Will be related in continuous description) in whether this message included, if it is, not executing 202, no longer
The message that this receives is processed；Otherwise, then execution step 202.
Specific write flow process, that is, the step 204 shown in Fig. 2 and 205 can as shown in Figure 3,
Comprise the following steps:
In 301, one cachenode (cache node) of taking-up from write queue, one
Message and the corresponding service processing result of this message is comprised, wherein message can be to disappear in cachenode
Presented in breath list.
In 302, judge whether existed and this cachenode bag in duplicate removal storage write queue
There is the message of identical calculations key containing message, if it is, this cachenode is put back to write queue,
Wait re-executes 301；Otherwise, 303 are executed.
In order to prevent message to be not yet stored into duplicate removal memory element, just be there is the message of identical calculations key to cover
Lid falls, and in this step, first judges whether existed and this cachenode in duplicate removal storage write queue
Comprised message has the message of identical calculations key, if it has, explanation has the message of identical calculations key
It is being written into duplicate removal memory element it should write new message after waiting to be written completing, therefore first
This cachenode is put back to write queue, wait retries.
In 303, by the content write calculating storaging unit in cachenode.
In 304, the message in cachenode is put into duplicate removal storage write queue.
In 305, store write queue from duplicate removal and take out message, message is write duplicate removal memory element.
It should be noted that above-mentioned write queue is actually a total queue, it is mainly used in calculating
The write of memory element, and duplicate removal storage write queue is only used for the write of duplicate removal memory element.Can see
Go out, the write of the write of calculating storaging unit and duplicate removal memory element is by the way of asynchronous, thus improving
Handling capacity.
Fig. 4 is structure drawing of device provided in an embodiment of the present invention, and this device can be arranged at flow-type business
Need in reason system in the equipment that message is processed.As shown in Figure 4, this device may include that
Receiving unit 01, the first judging unit 02, the second judging unit 03, processing unit 04, first write
Unit 05 and the second writing unit 06, can further include the 3rd judging unit 07 and the 4th judgement
Unit 08.Wherein, the major function of each component units is as follows:
Receiving unit 01 is responsible for receiving message.
Responsible this message that judges whether to be stored with duplicate removal memory element of first judging unit 02, if sentenced
Disconnected result is yes, no longer this message is processed.When the judged result of the first judging unit 02 is no,
Second judging unit 03 judges whether there is the corresponding service processing result of message in calculating storaging unit.
When the judged result of the second judging unit 03 is to be, processing unit 04 is not processed to message；?
When the judged result of two judging units 03 is no, processing unit 04 carries out Business Processing to message.
First writing unit 05 is responsible for for the service processing result of message writing calculating storaging unit.Second write
Unit 06 is after the first writing unit 05 completes write, or in the judged result of the second judging unit 03 is
When no, message is write duplicate removal memory element.
Preferably, in order to improve efficiency, can be by the content map in duplicate removal memory element to bloomfilter.
After receiving unit 01 receives message, first can judge whether this message hits by the 3rd judging unit 07
Bloomfilter, if not, illustrating that this message does not necessarily exist in duplicate removal memory element, triggering second judgement
The operation that unit 03 execution judges；If it is, this message that may be stored with explanation duplicate removal memory element, then
Trigger the operation that the first judging unit 02 execution judges.
For needing to carry out the scene of Business Processing using history service result, in order to more accurately ensure
Duplicate removal, it is preferred to use in the following manner: if existed in calculating storaging unit and the message tool receiving
There is the messaging list of identical calculations key, then processing unit 04 has identical calculations key using with this message
The corresponding service processing result of messaging list, carries out Business Processing to this message.
Now, the first writing unit 05 by this message add messaging list, and using processing unit 04 this
Service processing result update calculating storaging unit in the corresponding service processing result of messaging list.Second write
Unit 06 will comprise the messaging list write duplicate removal memory element of the message that this receives.
In addition, processing unit 04 can also be by this message (comprising the message queue of this message) and its corresponding
Service processing result puts into write queue.First writing unit 05 and the second writing unit 06 are using write team
The operation of row execution write.
Specifically, the first writing unit 05 can take out a cachenode from writing queue, one
Message (messaging list) and its corresponding service processing result is comprised, by cachenode in cachenode
In content write calculating storaging unit, by the message in cachenode put into duplicate removal storage write queue.
Second writing unit 06 stores from duplicate removal and takes out message write queue and write duplicate removal memory element.Permissible
Find out, the write of the write of calculating storaging unit and duplicate removal memory element is by the way of asynchronous, thus carrying
High-throughput.
Preferably, when the judged result of above-mentioned first judging unit 02 is no, the 4th judging unit 08 judges
Write queue and go to re-write in queue whether there is message, if it is, not processing to message；
Otherwise, the operation that the execution of triggering the second judging unit 03 judges.
Further, the content write in cachenode is being calculated storage by the first writing unit 05
Before unit, can first judge whether existed and cachenode bag in duplicate removal storage write queue
There is the message of identical calculations key containing message, if it is, cachenode is put back to write queue,
Wait retrieves；Otherwise, continue executing with and the content in cachenode is write calculating storaging unit
Operation.
It should be noted that method and apparatus provided in an embodiment of the present invention be not limited to jstorm and
Storm, goes for the Message Processing of any flow-type business processing system using similar mechanism.
It should be understood that disclosed apparatus and method in several embodiments provided by the present invention,
Can realize by another way.For example, device embodiment described above is only schematically,
For example, the division of described unit, only a kind of division of logic function, in addition actual can have when realizing
Dividing mode.
The described unit illustrating as separating component can be or may not be physically separate, make
For the part that unit shows can be or may not be physical location, you can with positioned at a place,
Or can also be distributed on multiple NEs.Can select according to the actual needs part therein or
The whole unit of person is realizing the purpose of this embodiment scheme.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit
In or unit is individually physically present it is also possible to two or more units are integrated in one
In individual unit.Above-mentioned integrated unit both can be to be realized in the form of hardware, it would however also be possible to employ hardware adds
The form of SFU software functional unit is realized.
The above-mentioned integrated unit realized in the form of SFU software functional unit, can be stored in a computer
In read/write memory medium.Above-mentioned SFU software functional unit is stored in a storage medium, including some fingers
Order is with so that a computer equipment (can be personal computer, server, or network equipment etc.)
Or processor (processor) executes the part steps of each embodiment methods described of the present invention.And it is aforementioned
Storage medium include: u disk, portable hard drive, read only memory (read-only memory, rom),
Random access memory (random access memory, ram), magnetic disc or CD etc. are various can
Medium with store program codes.
The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all this
Within the spirit of invention and principle, any modification, equivalent substitution and improvement done etc., should be included in
Within the scope of protection of the invention.
Claims (14)
1. a kind of message treatment method of streaming computing system is it is characterised in that the method includes:
S1, reception message；
Whether s2, judge to be stored with the first memory element described message, if it is, not to this message
Processed；Otherwise, execute s3；
S3, judge in the second memory element, whether there is the corresponding service processing result of described message, such as
Fruit is to execute s6, otherwise executes s4；
S4, Business Processing is carried out to described message；
S5, described message and its service processing result are write the second memory element；
S6, by described message write the first memory element.
2. method according to claim 1 is it is characterised in that content in described first memory element
It is mapped to Bloom filter bloomfilter；
Also included before described s2: judge whether described message hits bloomfilter, if not, execution
Described s3；If it is, executing described s2.
3. method according to claim 1 it is characterised in that execute described s4 when, if institute
State and in the second memory element, there is the messaging list with described message with identical calculations mark, then utilize
There is with described message the corresponding service processing result of messaging list of identical calculations mark, described message is entered
Row Business Processing；
In described s5, described message is added described messaging list, and the service processing result using s4
Update the corresponding service processing result of messaging list described in described second memory element；
When executing described s6, the described messaging list comprising described message is write the first memory element.
4. the method according to claim 1,2 or 3 is it is characterised in that in described s4 and described s5
Between also include: described message and its corresponding service processing result are put into write queue；
To the content execution described s5 and described s6 in said write queue.
5. method according to claim 4 is it is characterised in that hold to the content in said write queue
Row described s5 and described s6 includes:
S51, from said write queue take out a cache node, comprise message in one cache node
And its corresponding service processing result；
S52, by described cache node content write described second memory element；
S53, the message in described cache node is put into duplicate removal storage write queue；
S54, store write queue from duplicate removal and take out message write the first memory element.
6. method according to claim 5 it is characterised in that after described s2 and described s3 it
Front also include:
Judge whether there is described message, such as in said write queue and the storage write queue of described duplicate removal
Fruit is then not carry out Business Processing to described message, otherwise continues executing with described s3.
7. method according to claim 5 is it is characterised in that between described s51 and described s52
Also include:
Judge whether there is the comprised message tool with described cache node in described duplicate removal storage write queue
There is the message of identical calculations mark, if it is, described cache node is put back to said write queue, wait
Re-execute described s51；Otherwise, continue executing with described s52.
8. a kind of message processing apparatus of streaming computing system are it is characterised in that this device includes:
Receiving unit, for receiving message；
First judging unit, for the described message that judges whether to be stored with the first memory element, if
Judged result is yes, no longer described message is processed；
Second judging unit, for when the judged result of described first judging unit is no, judging described the
Whether there is the corresponding service processing result of described message in two memory element；
Processing unit, for when the judged result of described second judging unit is for being, not entering to described message
Row Business Processing；When the judged result of described second judging unit is no, described message is carried out at business
Reason；
First writing unit, for writing the second memory element by the service processing result of described message；
Second writing unit, for, after described first writing unit completes write, or sentencing described second
When the judged result of disconnected unit is no, described message is write the first memory element.
9. device according to claim 8 is it is characterised in that content in described first memory element
It is mapped to bloomfilter；
This device also includes: the 3rd judging unit, for after described receiving unit receives described message,
Judge whether described message hits bloomfilter, if not, triggering described second judging unit to execute judgement
Operation；If it is, the operation that the described first judging unit execution of triggering judges.
10. device according to claim 8 is it is characterised in that described processing unit, specifically for such as
There is the messaging list with described message with identical calculations mark, then in really described second memory element
Using the corresponding service processing result of messaging list with described message with identical calculations mark, disappear to described
Breath carries out Business Processing；
Described first writing unit, specifically for adding described messaging list by described message, and using described
This service processing result of processing unit updates the corresponding industry of messaging list described in described second memory element
Business result；
Described second writing unit, specifically for depositing the described messaging list write first comprising described message
Storage unit.
Device described in 11. according to Claim 8,9 or 10 it is characterised in that described processing unit,
It is additionally operable to for described message and its corresponding service processing result to put into write queue；
Described first writing unit and described second writing unit are write using the content execution in said write queue
The operation entering.
12. devices according to claim 11 it is characterised in that described first writing unit, specifically
For from said write queue take out a cache node, comprise in one cache node message and its
Corresponding service processing result, the content in described cache node is write described second memory element, by institute
State the message in cache node and put into duplicate removal storage write queue；
Described second writing unit, deposits specifically for storing to take out message in write queue and write first from duplicate removal
Storage unit.
13. devices according to claim 11 are it is characterised in that this device also includes:
4th judging unit, for when the judged result of described first judging unit is no, judging said write
Queue and described go to re-write in queue whether there is described message, if it is, not to described message
Processed；Otherwise, the operation that the described second judging unit execution of triggering judges.
14. devices according to claim 11 are it is characterised in that described first writing unit is by institute
Before stating content described second memory element of write in cache node, it is additionally operable to judge that described duplicate removal storage is write
Whether there is the message that the comprised message with described cache node has identical calculations mark in enqueue,
If it is, described cache node is put back to said write queue, wait retrieves；Otherwise, continue to hold
The content being about in described cache node writes the operation of described second memory element.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510413095.4A CN106354722B (en) | 2015-07-15 | 2015-07-15 | Message processing method and device for streaming computing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510413095.4A CN106354722B (en) | 2015-07-15 | 2015-07-15 | Message processing method and device for streaming computing system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106354722A true CN106354722A (en) | 2017-01-25 |
| CN106354722B CN106354722B (en) | 2019-12-24 |
Family
ID=57842506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510413095.4A Active CN106354722B (en) | 2015-07-15 | 2015-07-15 | Message processing method and device for streaming computing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106354722B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107943840A (en) * | 2017-10-30 | 2018-04-20 | 深圳前海微众银行股份有限公司 | Data processing method, system and computer-readable recording medium |
| CN110457305A (en) * | 2019-08-13 | 2019-11-15 | 腾讯科技（深圳）有限公司 | Data duplicate removal method, device, equipment and medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103763378A (en) * | 2014-01-24 | 2014-04-30 | 中国联合网络通信集团有限公司 | Task processing method and system and nodes based on distributive type calculation system |
| CN104615777A (en) * | 2015-02-27 | 2015-05-13 | 浪潮集团有限公司 | Method and device for real-time data processing based on stream-oriented calculation engine |
| CN104683488A (en) * | 2015-03-31 | 2015-06-03 | 百度在线网络技术（北京）有限公司 | Flow-type calculation system as well as dispatching method and dispatching device of flow-type calculation system |
-
2015
- 2015-07-15 CN CN201510413095.4A patent/CN106354722B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103763378A (en) * | 2014-01-24 | 2014-04-30 | 中国联合网络通信集团有限公司 | Task processing method and system and nodes based on distributive type calculation system |
| CN104615777A (en) * | 2015-02-27 | 2015-05-13 | 浪潮集团有限公司 | Method and device for real-time data processing based on stream-oriented calculation engine |
| CN104683488A (en) * | 2015-03-31 | 2015-06-03 | 百度在线网络技术（北京）有限公司 | Flow-type calculation system as well as dispatching method and dispatching device of flow-type calculation system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107943840A (en) * | 2017-10-30 | 2018-04-20 | 深圳前海微众银行股份有限公司 | Data processing method, system and computer-readable recording medium |
| CN107943840B (en) * | 2017-10-30 | 2022-01-11 | 深圳前海微众银行股份有限公司 | Data processing method, system and computer readable storage medium |
| CN110457305A (en) * | 2019-08-13 | 2019-11-15 | 腾讯科技（深圳）有限公司 | Data duplicate removal method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106354722B (en) | 2019-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6743141B2 (en) | Service processing method and device | |
| CN107395665A (en) | A kind of block chain service handling and business common recognition method and device | |
| US9665460B2 (en) | Detection of abnormal resource usage in a data center | |
| CN104063293B (en) | A kind of data back up method and stream calculation system | |
| CN109725899B (en) | Data stream processing method and device | |
| CN104035836B (en) | Automatic disaster-tolerant recovery method and system in cluster searching platform | |
| CN109298990A (en) | Log storing method, device, computer equipment and storage medium | |
| KR101574451B1 (en) | Imparting durability to a transactional memory system | |
| KR102024005B1 (en) | Method for processing transactions using blockchain, and transaction management server using the same | |
| CN103294558B (en) | A kind of MapReduce dispatching method supporting dynamic trust evaluation | |
| CN104156197A (en) | Microprocessor and method for operating microprocessor | |
| CN107729018A (en) | Cyclic vector method and apparatus | |
| CN108108248A (en) | A kind of CPU+GPU cluster management methods, device and equipment for realizing target detection | |
| CN106201561B (en) | The upgrade method and equipment of distributed caching cluster | |
| CN110213128A (en) | Serve port detection method, electronic equipment and computer storage medium | |
| CN109325548A (en) | Image processing method, device, electronic equipment and storage medium | |
| CN107766188A (en) | Internal-memory detection method and device in train control system | |
| CN110737504A (en) | deep learning model training fault-tolerant method, system, terminal and storage medium | |
| CN110324211A (en) | A kind of data capture method and device | |
| CN106354722A (en) | Message processing method and device for stream-oriented computation system | |
| CN108369547A (en) | The tail portion of daily record in persistence main memory | |
| CN103634167B (en) | Security configuration check method and system for target hosts in cloud environment | |
| CN108984779A (en) | Distributed file system snapshot rollback metadata processing method, device and equipment | |
| US11210757B2 (en) | GPU packet aggregation system | |
| JP6847112B2 (en) | How and devices to process data after node reboot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |