CN107194013A - Processing method and processing system based on Distributed Message Queue middleware - Google Patents
Processing method and processing system based on Distributed Message Queue middleware Download PDFInfo
- Publication number
- CN107194013A CN107194013A CN201710533747.7A CN201710533747A CN107194013A CN 107194013 A CN107194013 A CN 107194013A CN 201710533747 A CN201710533747 A CN 201710533747A CN 107194013 A CN107194013 A CN 107194013A
- Authority
- CN
- China
- Prior art keywords
- massage pattern
- cache
- information
- pattern
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Computer And Data Communications (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a kind of processing method and processing system based on Distributed Message Queue middleware, wherein, method includes：Obtain upload message and the information with massage pattern is set up according to upload message；Massage pattern progress is handled and obtains Query Information；The result inquired about according to Query Information extracts the massage pattern in the first Cache, and information encode further according to massage pattern to obtain coding information, and coding information is sent to the storage of Distributed Message Queue cluster；The result inquired about according to Query Information extracts the massage pattern in the second Cache, and corresponding coding information is obtained from Distributed Message Queue cluster further according to massage pattern and is decoded, and is consumed according to decoded result.Present invention, avoiding process of " being shaken hands " under line, when manufacturer information pattern needs change, it is not necessary to which line sends a notice consumer, just can be automatically performed the upgrading of massage pattern, mitigate regular maintenance and operation cost.
Description
Technical field
The present invention relates to message processing technique field, more particularly to a kind of processing based on Distributed Message Queue middleware
Method and processing system.
Background technology
Distributed Message Queue middleware be a kind of extensive utilization in a distributed system be used for inter-node communication it is soft
Part, is a significant components in distributed system, is mainly used to solution and gives out information, using coupling, the problems such as flow peak clipping.
Kafka, is the Distributed Message Queue middleware developed and increased income by LinkedIn companies, flat as a distributed flow media
Platform, is a main flow open source system of Distributed Message Queue, is very suitable between each component of big data system in real time
Or quasi real time communicate.
Kafka Message Queuing Middlewares include the producer, manager and consumer；When one new product business of generation
When, it is necessary to formulate a massage pattern first, and the producer can carry out encoding and then again by Kafka according to this massage pattern
Transmitting data stream.When the service of the Kafka other ends needs to consume this data flow, it is also desirable to take this massage pattern
Just can know that how decoded back data flow.
At present, massage pattern is uploaded and is registered to massage pattern service manager by the producer, and will be told under result line
Message consumer.These information are supplied to massage pattern manager, and download message pattern by consumer again, are completed on line and line
Under " shaking hands " process for being combined.In practice, with the continuous propulsion of business, massage pattern is often upgraded change, and
All these variations, which are required for both sides " to shake hands " again, could correctly handle the data flow of transmission.When the producer due to demand or
Person's other reasonses cause information pattern to need change, if do not notify consumer, consumer can be caused to occur abnormal, i.e.,
The message modified line for generating the producer has sent a notice, and consumer also needs to be supplied to massage pattern to manage these information
Person, and download message pattern.This not only brings very big burden to the regular maintenance of product and operation, and causes resource
Waste.
The content of the invention
It is an object of the invention to provide a kind of processing method and processing system based on Distributed Message Queue middleware, keep away
Exempt under line " shaking hands " process, when the producer causes information pattern to need change due to demand or other reasonses, it is not necessary to
Line sends a notice consumer, just can be automatically performed the upgrading of massage pattern, mitigates regular maintenance and operation cost.
The invention provides a kind of processing method based on Distributed Message Queue middleware, comprise the following steps：
The upload message in external message stream is obtained, information is set up according to upload message；The information includes
Massage pattern；
The massage pattern is handled, Query Information is obtained；
Inquired about according to the Query Information, the message mould in the first Cache is extracted according to Query Result
Formula, is encoded further according to the massage pattern to the information, obtains coding information, and the coding information is sent
To the storage of Distributed Message Queue cluster；
Inquired about according to the Query Information, the message mould in the second Cache is extracted according to Query Result
Formula, obtains the corresponding coding information further according to the massage pattern from the Distributed Message Queue cluster and is solved
Code, and consumed according to decoded result.
It is described to be inquired about according to the Query Information as a kind of embodiment, extract first according to Query Result
The massage pattern in Cache, comprises the following steps：
Inquired about according to the Query Information, judge whether current massage pattern is stored in the first Cache
In；
If current massage pattern is had been stored in the first Cache, the institute in the first Cache is extracted
State massage pattern；
If current massage pattern is not stored in the first Cache, according to the registration in the information
Information is uploaded the massage pattern, and the massage pattern is stored in pattern database；Meanwhile, by the message
Pattern is stored in the first Cache.
Described according to the Query Information as a kind of embodiment, that extracts in the second Cache described disappears
Breath pattern, comprises the following steps：
Inquired about according to the Query Information, judge whether current massage pattern is stored in second cache
In device；
If current massage pattern is had been stored in the second Cache, the institute in the second Cache is extracted
State massage pattern；
If current massage pattern is not stored in the second Cache, according to the Query Information and the mould
Formula database is contrasted, and the corresponding massage pattern is obtained according to result, meanwhile, it is stored in the second Cache by described
In.
It is described that the massage pattern is handled as a kind of embodiment, obtain Query Information, including following step
Suddenly：
Hashing algorithm is carried out to the massage pattern, cryptographic Hash is obtained；
Classification processing is carried out to the massage pattern, grouped data is obtained；
The cryptographic Hash and the grouped data are merged and obtain the Query Information.
The processing method based on Distributed Message Queue middleware provided as a kind of embodiment, the present invention, also
Comprise the following steps：
Before being inquired about according to the Query Information, version information and timestamp are encapsulated into the message data
In.
Accordingly, the present invention also provides a kind of processing system based on Distributed Message Queue middleware, including obtains mould
Block, operation module, extraction coding module and extraction decoder module；
The acquisition module, for obtaining the upload message in external message stream, information is set up according to upload message；
The information includes massage pattern；
The operation module, for handling the massage pattern, obtains Query Information；
The extraction coding module, for being inquired about according to the Query Information, extracts first high according to Query Result
The massage pattern in fast buffer, is encoded further according to the massage pattern to the information, obtains coding letter
Breath, and the coding information is sent to the storage of Distributed Message Queue cluster；
The extraction decoder module, for being inquired about according to the Query Information, extracts second high according to Query Result
The massage pattern in fast buffer, correspondence is obtained further according to the massage pattern from the Distributed Message Queue cluster
The coding information decoded, and consumed according to decoded result.
As a kind of embodiment, the extraction coding module, including the first judging unit, the first extraction unit and
Register memory cell；
Whether first judging unit, for being inquired about according to the Query Information, judge current massage pattern
It is stored in the first Cache；
First extraction unit, if being had been stored in for current massage pattern in the first Cache, is carried
Take the massage pattern in the first Cache；
The registration memory cell, if being not stored in for current massage pattern in the first Cache, root
The massage pattern is uploaded according to the log-on message in the information, and the massage pattern is stored in pattern count
According in storehouse；Meanwhile, the massage pattern is stored in the first Cache.
As a kind of embodiment, the extraction decoder module, including the second judging unit, the second extraction unit and
Contrast memory cell；
Whether second judging unit, for being inquired about according to the Query Information, judge current massage pattern
It is stored in second Cache；
Second extraction unit, if being had been stored in for current massage pattern in the second Cache, is carried
Take the massage pattern in the second Cache；
The contrast memory cell, if being not stored in for current massage pattern in the second Cache, root
Contrasted according to the Query Information and the pattern database, the corresponding massage pattern is obtained according to result, meanwhile, will be described
It is stored in the second Cache.
As a kind of embodiment, the operation module, including hashing algorithm unit, classification processing unit and merging
Unit；
The hashing algorithm unit, for carrying out hashing algorithm to the massage pattern, obtains cryptographic Hash；
The classification processing unit, for carrying out classification processing to the massage pattern, obtains grouped data；
The combining unit, the Query Information is obtained for the cryptographic Hash and the grouped data to be merged.
The processing system based on Distributed Message Queue middleware provided as a kind of embodiment, the present invention, also
Including package module；
The package module, for before being inquired about according to the Query Information, version information and timestamp to be sealed
It is attached in the message data.
Compared with prior art, the technical program has advantages below：
The processing method and processing system based on Distributed Message Queue middleware that the present invention is provided, by according to inquiry
The massage pattern stored in the Cache of information extraction first and the second high-speed buffer, and according in the first Cache
Massage pattern information is encoded, by obtained coding information send to Distributed Message Queue cluster store；Root
Corresponding coding information is obtained from Distributed Message Queue cluster according to the massage pattern in the second Cache to be decoded,
And consumed according to decoded result.Present invention, avoiding process of " being shaken hands " under line, when the producer is due to demand or other originals
During because causing information pattern to need change, it is not necessary to which line sends a notice consumer, just can be automatically performed the upgrading of massage pattern, subtract
Light regular maintenance and operation cost.
Brief description of the drawings
Fig. 1 is the flow signal for the processing method based on Distributed Message Queue middleware that the embodiment of the present invention one is provided
Figure；
Fig. 2 is the structural representation for the processing system based on Distributed Message Queue middleware that the embodiment of the present invention two is provided
Figure；
Fig. 3 is the structural representation of operation module in Fig. 2；
Fig. 4 is the structural representation of extraction coding module in Fig. 2；
Fig. 5 is the structural representation of extraction decoder module in Fig. 2.
In figure：100th, acquisition module；200th, operation module；210th, hashing algorithm unit；220th, classification processing unit；230、
Combining unit；300th, coding module is extracted；310th, the first judging unit；320th, the first extraction unit；330th, memory cell is registered；
400th, decoder module is extracted；410th, the second judging unit；420th, the second extraction unit；430th, memory cell is contrasted；500th, encapsulate
Module.
Embodiment
Below in conjunction with accompanying drawing, the technical characteristic above-mentioned and other to the present invention and advantage are clearly and completely described,
Obviously, described embodiment is only the section Example of the present invention, rather than whole embodiments.
Referring to Fig. 1, the processing method based on Distributed Message Queue middleware that the embodiment of the present invention one is provided, including
Following steps：
S100, the upload message obtained in external message stream, information is set up according to upload message；Information includes
Massage pattern；
S200, massage pattern is handled, obtain Query Information；
S300, inquired about according to Query Information, the massage pattern in the first Cache extracted according to Query Result,
Information is encoded further according to massage pattern, coding information is obtained, and coding information is sent to distributed message team
Row cluster is stored；
S400, inquired about according to Query Information, the massage pattern in the second Cache extracted according to Query Result,
Corresponding coding information is obtained from Distributed Message Queue cluster further according to massage pattern to be decoded, and according to decoded result
Consumed.
It should be noted that upload message is the product industry for needing to be changed or newly producing in external message stream
The information flow that business is included；And there are countless such upload messages in external message stream, upload message is according to the actual requirements
Produce, so being the real-time upload message obtained in external message stream in the present embodiment.In other embodiment, also may be used
Be according to other modes judge obtain upload message again.
Query Information is, for query messages pattern, each massage pattern to be classified, then by each message mould
Its corresponding Query Information, metadata information can be all stored in a massage pattern by formula when storage, that is to say, that
A variety of massage patterns are store in the first Cache and the second Cache of one class, are being used and current phase
During the massage pattern answered, it can be extracted and use.
The processing method based on Distributed Message Queue middleware that the present invention is provided, by extracting the according to Query Information
The massage pattern stored in one Cache and the second high-speed buffer, and the massage pattern in the first Cache
Information is encoded, obtained coding information is sent to Distributed Message Queue cluster and stored；According to second at a high speed
Massage pattern in buffer obtains corresponding coding information from Distributed Message Queue cluster and decoded, and according to decoding
As a result consumed.Present invention, avoiding process of " being shaken hands " under line, as the producer because demand or other reasonses cause information
When pattern needs change, it is not necessary to which line sends a notice consumer, the upgrading of massage pattern just can be automatically performed, mitigates regular maintenance
And operation cost.
Further, inquired about according to Query Information, the message in the first Cache is extracted according to Query Result
Pattern, comprises the following steps：
S310, inquired about according to Query Information, judge whether current massage pattern is stored in the first Cache
In；
If S320, current massage pattern are had been stored in the first Cache, the first Cache is extracted
In massage pattern；
If S330, current massage pattern are not stored in the first Cache, the note in information
Volume information is uploaded massage pattern, and massage pattern is stored in pattern database；Meanwhile, massage pattern is stored in
In first Cache.
In the present embodiment, massage pattern is stored in the first Cache among step S330 simultaneously.It is also
The massage pattern that will be stored in the first Cache comes out in advance；And one query is often carried out, no matter the current first high speed
Buffer whether there is massage pattern corresponding with Query Information, finally all can deposit massage pattern corresponding with Query Information production again
It is stored in the first Cache, and the massage pattern can be also stored in pattern database, in general in pattern database
Massage pattern finally can all be stored in the first Cache, inquiry each time is also a mistake to current message pattern
The process of filter, it is to avoid the massage pattern as storing is repeated in the first Cache；Network access can so be reduced
Number of times, mitigates system operation burden, improves operational efficiency.Massage pattern progress upload is directly realized by message according to log-on message
The automatic registration of pattern, is supported without additional programs.Need exist for explanation be in a massage pattern may also nesting may be used also
To there is other massage pattern.
Further, according to Query Information, the massage pattern in the second Cache is extracted, is comprised the following steps：
S410, inquired about according to Query Information, judge whether current massage pattern is stored in the second Cache
In；
If S420, current massage pattern are had been stored in the second Cache, the second Cache is extracted
In massage pattern；
If S430, current massage pattern are not stored in the second Cache, according to Query Information and pattern
Database is contrasted, and corresponding massage pattern is obtained according to result, meanwhile, it will be stored in the second Cache.
In the present embodiment, massage pattern is stored in the first Cache among step S430 simultaneously.It is also
The massage pattern that will be stored in the first Cache comes out in advance；Second Cache and the first Cache one
Sample, the various types of messages that itself is stored with pattern, when being inquired about, if corresponding disappear is not present in current second Cache
Breath pattern, then contrast according to Query Information and pattern database, corresponding massage pattern obtained according to result, meanwhile, it will store
In the second Cache, that is to say, that the massage pattern in pattern database finally can all be stored in the first Cache
In, inquiry each time is also the process to one of current message pattern filtering, it is to avoid repeated in the first Cache
The same massage pattern of storage；Network access number of times can be so reduced, is mitigated on system operation burden, message handling ability
Rise, the stability of a system and availability of service are improved.
Further, step S200 comprises the following steps：
S210, to massage pattern carry out hashing algorithm, obtain cryptographic Hash；
S220, to massage pattern carry out classification processing, obtain grouped data；
S230, cryptographic Hash and grouped data are merged obtain Query Information.
Here hashing algorithm can be carried out by md5, obtain the cryptographic Hash of one 128, and grouped data is pair
Massage pattern carries out what classification processing was obtained.In the present embodiment, the algorithmic procedure of cryptographic Hash and grouped data is not carried out
Limit.
Further, the processing method based on Distributed Message Queue middleware that the embodiment of the present invention one is provided also includes
Following steps：
Before being inquired about according to Query Information, version information and timestamp are encapsulated into message data.
When version information and timestamp are stored in the first Cache and the second Cache to massage pattern
Play booster action.
The concrete principle to the present invention by taking Avro and Kafka as an example is described in detail below：
Kafka Message Queuing Middlewares include the producer, manager and consumer；
Producer sends the information with massage pattern and gives Avro encoders；
Encoder is mainly responsible for encoding information by Avro, the embedded massage pattern of Avro encoders
Cache.Cache preserves all used Avro massage patterns；Before the coding, it will can currently disappear
Cease the massage pattern used in information and carry out md5 (hash) operations, take the cryptographic Hash of one 128.Then, encoder can be taken
Whether Kafka topic (grouped data) and cryptographic Hash conjunctive query local cache, see massage pattern local
Exist inside Cache.If existed, encoder can use the massage pattern taken to encode information,
And obtained decoded information is sent to Kafka clusters.If it does not, it can pass through the automatic message call moulds of HTTP POST
Formula registrar, uploads the massage pattern.Massage pattern registrar is received after log-on message, can be by this message mould
Formula and its relevant metadata information write-in massage pattern database.At the same time, this new massage pattern is loaded into encoder
Cache so that encoder can use the massage pattern taken to encode information.
Consumer is received after the information, can be decoded by decoder by Avro.In Avro decoders
Face embeds a Cache, preserves all decoders used massage pattern.During decoding, decoder can be disappeared
The cryptographic Hash of breath pattern, the massage pattern then inquired about by Kafka topic and cryptographic Hash in local cache, is checked
Whether current message pattern has been present in the Cache of decoder.If it does, encoder can use this that take to disappear
Breath pattern is decoded to information, consumes current information.If it does not, encoder can employ the Kafka certainly
Topic and cryptographic Hash simultaneously go on massage pattern registrar to obtain this new information by HTTP GET.Message mould
Formula registrar is responsible for taking out corresponding massage pattern inside massage pattern database and returns to encoder.It is same with this
When, this massage pattern can be fitted into local cache by decoder, be used for the information for consuming same massage pattern.
It is understood that, information by massage pattern after being encoded, when consuming this information,
Have to be decoded by this massage pattern.And during decoding, decoded using following algorithm：
Wherein, getSchemaByTopicAndSchema (topicA, md5) realization is as follows：
It is pointed out that each element corresponding to list here is probably a different types of sub- letter
Breath, with different sub- massage patterns.Although this scheme is illustrated based on Avro and Kafka, other message queue is used
With other massage pattern mode, the present invention is equally applicable.
Based on same inventive concept, the embodiment of the present invention also provides a kind of processing based on Distributed Message Queue middleware
System, the implementation of the system can refer to the process realization of the above method, repeat part no longer redundant later.
As shown in Fig. 2 being the processing system based on Distributed Message Queue middleware that the embodiment of the present invention two is provided
Structural representation, including acquisition module 100, operation module 200, extraction coding module 300 and extraction decoder module 400；
Acquisition module 100 is used to obtain the upload message in external message stream, and information is set up according to upload message；Disappear
Ceasing information includes massage pattern；
Operation module 200 is used to handle massage pattern, obtains Query Information；
Extracting coding module 300 is used to be inquired about according to Query Information, and the first cache is extracted according to Query Result
Massage pattern in device, is encoded further according to massage pattern to information, obtains coding information, and coding information is sent
To the storage of Distributed Message Queue cluster；
Extracting decoder module 400 is used to be inquired about according to Query Information, and the second cache is extracted according to Query Result
Massage pattern in device, obtains corresponding coding information further according to massage pattern from Distributed Message Queue cluster and is solved
Code, and consumed according to decoded result.
As shown in figure 4, being the structural representation for extracting coding module, including the first judging unit 310, the first extraction unit
320 and registration memory cell 330；
First judging unit 310 is used to be inquired about according to Query Information, judges whether current massage pattern is stored in
In first Cache；
If the first extraction unit 320 is had been stored in the first Cache for current massage pattern, extract
Massage pattern in first Cache；
If registration memory cell 330 is not stored in the first Cache for current massage pattern, basis
Log-on message in information is uploaded massage pattern, and massage pattern is stored in pattern database；Meanwhile, will
Massage pattern is stored in the first Cache.
As shown in figure 5, being the structural representation for extracting decoder module, including the second judging unit 410, the second extraction unit
420 and contrast memory cell 430；
Second judging unit 410 is used to be inquired about according to Query Information, judges whether current massage pattern is stored in
In second Cache；
If the second extraction unit 420 is had been stored in the second Cache for current massage pattern, extract
Massage pattern in second Cache；
If contrast memory cell 430 is not stored in the second Cache for current massage pattern, basis
Query Information is contrasted with pattern database, and corresponding massage pattern is obtained according to result, meanwhile, it will be stored in the second cache
In device.
As shown in figure 3, be the structural representation of operation module, including hashing algorithm unit 210, classification processing unit 220
And combining unit 230；
Hashing algorithm unit 210 is used to carry out hashing algorithm to massage pattern, obtains cryptographic Hash；
Processing unit 220 of classifying is used to carry out classification processing to massage pattern, obtains grouped data；
Combining unit 230, which is used to merge cryptographic Hash and grouped data, obtains Query Information.
Further, as shown in figure 1, the processing based on Distributed Message Queue middleware that the embodiment of the present invention one is provided
System also includes package module 500；
Package module 500 is used for before being inquired about according to Query Information, and version information and timestamp are encapsulated into and disappeared
Cease in data.
The processing system based on Distributed Message Queue middleware that the present invention is provided, by extracting the according to Query Information
The massage pattern stored in one Cache and the second high-speed buffer, and the massage pattern in the first Cache
Information is encoded, obtained coding information is sent to Distributed Message Queue cluster and stored；According to second at a high speed
Massage pattern in buffer obtains corresponding coding information from Distributed Message Queue cluster and decoded, and according to decoding
As a result consumed.Present invention, avoiding process of " being shaken hands " under line, as the producer because demand or other reasonses cause information
When pattern needs change, it is not necessary to which line sends a notice consumer, the upgrading of massage pattern just can be automatically performed, mitigates regular maintenance
And operation cost.
Although the present invention is disclosed as above with preferred embodiment, it is not for limiting the present invention, any this area
Technical staff without departing from the spirit and scope of the present invention, may be by the methods and techniques content of the disclosure above to this hair
Bright technical scheme makes possible variation and modification, therefore, every content without departing from technical solution of the present invention, according to the present invention
Any simple modifications, equivalents, and modifications made to above example of technical spirit, belong to technical solution of the present invention
Protection domain.
Claims (10)
1. a kind of processing method based on Distributed Message Queue middleware, it is characterised in that comprise the following steps：
The upload message in external message stream is obtained, information is set up according to upload message；The information includes message
Pattern；
The massage pattern is handled, Query Information is obtained；
Inquired about according to the Query Information, the massage pattern in the first Cache extracted according to Query Result,
The information is encoded further according to the massage pattern, coding information is obtained, and by the coding information send to
Distributed Message Queue cluster is stored；
Inquired about according to the Query Information, the massage pattern in the second Cache extracted according to Query Result,
The corresponding coding information is obtained from the Distributed Message Queue cluster further according to the massage pattern to be decoded, and
Consumed according to decoded result.
2. the processing method as claimed in claim 1 based on Distributed Message Queue middleware, it is characterised in that the basis
The Query Information is inquired about, and the massage pattern in the first Cache is extracted according to Query Result, including following
Step：
Inquired about according to the Query Information, judge whether current massage pattern is stored in the first Cache；
If current massage pattern is had been stored in the first Cache, that extracts in the first Cache described disappears
Breath pattern；
If current massage pattern is not stored in the first Cache, according to the log-on message in the information
The massage pattern is uploaded, and the massage pattern is stored in pattern database；Meanwhile, by the massage pattern
It is stored in the first Cache.
3. the processing method as claimed in claim 2 based on Distributed Message Queue middleware, it is characterised in that the basis
The Query Information, extracts the massage pattern in the second Cache, comprises the following steps：
Inquired about according to the Query Information, judge whether current massage pattern is stored in second Cache
In；
If current massage pattern is had been stored in the second Cache, that extracts in the second Cache described disappears
Breath pattern；
If current massage pattern is not stored in the second Cache, according to the Query Information and the pattern count
Contrasted according to storehouse, the corresponding massage pattern is obtained according to result, meanwhile, the massage pattern is stored in the second cache
In device.
4. the processing method as claimed in claim 1 based on Distributed Message Queue middleware, it is characterised in that described to institute
State massage pattern to be handled, obtain Query Information, comprise the following steps：
Hashing algorithm is carried out to the massage pattern, cryptographic Hash is obtained；
Classification processing is carried out to the massage pattern, grouped data is obtained；
The cryptographic Hash and the grouped data are merged and obtain the Query Information.
5. the processing method based on Distributed Message Queue middleware as claimed in claim 1, it is characterised in that also including with
Lower step：
Before being inquired about according to the Query Information, version information and timestamp are encapsulated into the message data.
6. a kind of processing system based on Distributed Message Queue middleware, it is characterised in that including acquisition module, operation mould
Block, extraction coding module and extraction decoder module；
The acquisition module, for obtaining the upload message in external message stream, information is set up according to upload message；It is described
Information includes massage pattern；
The operation module, for handling the massage pattern, obtains Query Information；
The extraction coding module, for being inquired about according to the Query Information, extracts first slow at a high speed according to Query Result
The massage pattern in storage, is encoded further according to the massage pattern to the information, obtains coding information, and
The coding information is sent to the storage of Distributed Message Queue cluster；
The extraction decoder module, for being inquired about according to the Query Information, extracts second slow at a high speed according to Query Result
The massage pattern in storage, corresponding institute is obtained further according to the massage pattern from the Distributed Message Queue cluster
State coding information to be decoded, and consumed according to decoded result.
7. the processing system as claimed in claim 6 based on Distributed Message Queue middleware, it is characterised in that the extraction
Coding module, including the first judging unit, the first extraction unit and registration memory cell；
First judging unit, for being inquired about according to the Query Information, judges whether current massage pattern stores
In the first Cache；
First extraction unit, if being had been stored in for current massage pattern in the first Cache, extracts
The massage pattern in one Cache；
The registration memory cell, if being not stored in for current massage pattern in the first Cache, according to institute
The log-on message stated in information is uploaded the massage pattern, and the massage pattern is stored in into pattern database
In；Meanwhile, the massage pattern is stored in the first Cache.
8. the processing system as claimed in claim 7 based on Distributed Message Queue middleware, it is characterised in that the extraction
Decoder module, including the second judging unit, the second extraction unit and contrast memory cell；
Second judging unit, for being inquired about according to the Query Information, judges whether current massage pattern stores
In second Cache；
Second extraction unit, if being had been stored in for current massage pattern in the second Cache, extracts
The massage pattern in two Caches；
The contrast memory cell, if being not stored in for current massage pattern in the second Cache, according to institute
State Query Information to contrast with the pattern database, the corresponding massage pattern is obtained according to result, meanwhile, by the storage
In the second Cache.
9. the processing system as claimed in claim 6 based on Distributed Message Queue middleware, it is characterised in that the operation
Module, including hashing algorithm unit, classification processing unit and combining unit；
The hashing algorithm unit, for carrying out hashing algorithm to the massage pattern, obtains cryptographic Hash；
The classification processing unit, for carrying out classification processing to the massage pattern, obtains grouped data；
The combining unit, the Query Information is obtained for the cryptographic Hash and the grouped data to be merged.
10. the processing system as claimed in claim 6 based on Distributed Message Queue middleware, it is characterised in that also include
Package module；
The package module, for before being inquired about according to the Query Information, version information and timestamp to be encapsulated into
In the message data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710533747.7A CN107194013B (en) | 2017-07-03 | 2017-07-03 | Processing method and processing system based on Distributed Message Queue middleware |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710533747.7A CN107194013B (en) | 2017-07-03 | 2017-07-03 | Processing method and processing system based on Distributed Message Queue middleware |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107194013A true CN107194013A (en) | 2017-09-22 |
| CN107194013B CN107194013B (en) | 2019-10-29 |
Family
ID=59881211
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710533747.7A Active CN107194013B (en) | 2017-07-03 | 2017-07-03 | Processing method and processing system based on Distributed Message Queue middleware |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107194013B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108595483A (en) * | 2018-03-13 | 2018-09-28 | 腾讯科技（深圳）有限公司 | Data processing method and relevant apparatus |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101917332A (en) * | 2010-05-24 | 2010-12-15 | 宁波东海蓝帆科技有限公司 | Message delivery method for reducing network load of message midware |
| CN104111998A (en) * | 2014-07-09 | 2014-10-22 | 江西理工大学 | Method and device for sorting coding and integrated exchange and management of heterogeneous data of enterprise |
| CN106487891A (en) * | 2016-10-11 | 2017-03-08 | 北京集奥聚合科技有限公司 | A kind of method of the process message based on kafka |
-
2017
- 2017-07-03 CN CN201710533747.7A patent/CN107194013B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101917332A (en) * | 2010-05-24 | 2010-12-15 | 宁波东海蓝帆科技有限公司 | Message delivery method for reducing network load of message midware |
| CN104111998A (en) * | 2014-07-09 | 2014-10-22 | 江西理工大学 | Method and device for sorting coding and integrated exchange and management of heterogeneous data of enterprise |
| CN106487891A (en) * | 2016-10-11 | 2017-03-08 | 北京集奥聚合科技有限公司 | A kind of method of the process message based on kafka |
Non-Patent Citations (1)
| Title |
|---|
| 陈飞: "利用SCTP对MRB消息中间件的改进", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108595483A (en) * | 2018-03-13 | 2018-09-28 | 腾讯科技（深圳）有限公司 | Data processing method and relevant apparatus |
| CN108595483B (en) * | 2018-03-13 | 2020-11-24 | 腾讯科技（深圳）有限公司 | Data processing method and related device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107194013B (en) | 2019-10-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103297807A (en) | Hadoop-platform-based method for improving video transcoding efficiency | |
| CN105187845B (en) | Apparatus for decoding video data and coding/decoding method | |
| CN110149361A (en) | A kind of Internet service operating process retrogressive method, system and device | |
| CN101299690A (en) | Method, system and apparatus for monitoring multimedia contents | |
| CN111581404B (en) | Method and system for marking image-text manuscript through whole process based on block chain technology | |
| CN105825083A (en) | Picture copyright protecting method and recognizing method and related method and system | |
| CN106487891A (en) | A kind of method of the process message based on kafka | |
| CN111985504A (en) | Copying detection method, device, equipment and medium based on artificial intelligence | |
| CN103237209A (en) | H264 video watermarking method based on regional DCT (discrete cosine transform) coefficients | |
| CN106686148A (en) | Method and system for increasing uploading speed of objects in object storage system | |
| CN107197296B (en) | A kind of HEVC parallel encoding method and system based on COStream | |
| CN107194013A (en) | Processing method and processing system based on Distributed Message Queue middleware | |
| CN107066526A (en) | A kind of network crawler system and method | |
| US20210014540A1 (en) | Method and system for codec of visual feature data | |
| CN107229516A (en) | A kind of data processing method and device | |
| CN111211887B (en) | Resource encryption method, system, device and computer readable storage medium | |
| CN106846190A (en) | A kind of intellectual property case management system | |
| CN100551066C (en) | The implementation method of encoder and adaptive arithmetic code and device | |
| CN115205758A (en) | Intelligent conversion method and device based on video and text, electronic equipment and medium | |
| WO2021135699A1 (en) | Decision scheduling customization method and device based on information flow | |
| CN102271122B (en) | A kind of rubbish contents applied to P2P networks determine method and its system | |
| CN112203117A (en) | Video source tracing and tamper-proof method based on video coder/decoder | |
| CN112631675A (en) | Workflow configuration method, device, equipment and computer readable storage medium | |
| CN113099270A (en) | File storage method, decoding method, device, storage medium and electronic device | |
| CN115734008B (en) | Method, system and medium for rapidly integrating video resources of content provider |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210120 Address after: 401120 room 310, business building a, East terminal area of Chongqing Jiangbei International Airport, Yubei District, Chongqing Patentee after: Chongqing permutation Technology Co.,Ltd. Address before: Room 501, building 19, haichuangyuan, No. 998, Wenyi West Road, Hangzhou, Zhejiang 311121 Patentee before: HANGZHOU ECREDITPAL TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |