CN107194013B - Processing method and processing system based on Distributed Message Queue middleware - Google Patents
Processing method and processing system based on Distributed Message Queue middleware Download PDFInfo
- Publication number
- CN107194013B CN107194013B CN201710533747.7A CN201710533747A CN107194013B CN 107194013 B CN107194013 B CN 107194013B CN 201710533747 A CN201710533747 A CN 201710533747A CN 107194013 B CN107194013 B CN 107194013B
- Authority
- CN
- China
- Prior art keywords
- massage pattern
- information
- cache
- pattern
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Computer And Data Communications (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a kind of processing method and processing system based on Distributed Message Queue middleware, wherein method includes: to obtain upload message to establish the information for having massage pattern according to upload message；Massage pattern is handled to obtain query information；The massage pattern in the first Cache is extracted according to the result that query information is inquired, information is encoded further according to massage pattern to obtain encoded information, and encoded information is sent to the storage of Distributed Message Queue cluster；The massage pattern in the second Cache is extracted according to the result that query information is inquired, corresponding encoded information is obtained from Distributed Message Queue cluster further according to massage pattern and is decoded, and consumed according to decoding result.The invention avoids processes of " shaking hands " under line, when manufacturer information mode needs to change, do not need line and send a notice consumer, can be automatically performed the upgrading of massage pattern, mitigate daily maintenance and operation cost.
Description
Technical field
The present invention relates to message processing technique field more particularly to a kind of processing based on Distributed Message Queue middleware
Method and processing system.
Background technique
Distributed Message Queue middleware be a kind of extensive utilization in a distributed system for the soft of inter-node communication
Part is a significant components in distributed system, is mainly used to solve to give out information, using coupling, the problems such as flow peak clipping.
Kafka is the Distributed Message Queue middleware developed and increased income by LinkedIn company, flat as a distributed flow media
Platform is a mainstream open source system of Distributed Message Queue, is very suitable between the various components of big data system in real time
Or it quasi real time communicates.
Kafka Message Queuing Middleware includes the producer, manager and consumer；When one new product business of generation
When, need to formulate a massage pattern first, and the producer according to this massage pattern can encode and then pass through Kafka again
Transmitting data stream.When the service of the Kafka other end needs to consume this data flow, it is also desirable to take this massage pattern
Just can know that how decoded back data flow.
Currently, massage pattern is uploaded and be registered to massage pattern service manager by the producer, and will be told under result line
Message consumer.These information are supplied to massage pattern manager, and download message mode again by consumer, are completed on line and line
Under " shaking hands " process for combining.In practice, with the continuous propulsion of business, massage pattern often upgrades variation, and
All these variations require both sides " shakes hands " again could correctly handle the data flow of transmission.When the producer due to demand or
Person's other reasons cause information pattern to need to change, if do not notify consumer, will lead to consumer and exception occur, i.e.,
The message modified line for generating the producer has sent a notice, and consumer also needs these information being supplied to massage pattern management
Person, and download message mode.This not only brings very big burden to the daily maintenance of product and operation, but also causes resource
Waste.
Summary of the invention
The object of the present invention is to provide a kind of processing method and processing system based on Distributed Message Queue middleware, keep away
" shaking hands " process is exempted under line, when the producer causes information pattern to need to change due to demand or other reasons, has not needed
Line sends a notice consumer, can be automatically performed the upgrading of massage pattern, mitigates daily maintenance and operation cost.
The present invention provides a kind of processing methods based on Distributed Message Queue middleware, comprising the following steps:
The upload message in external message stream is obtained, information is established according to upload message；The information includes
Massage pattern；
The massage pattern is handled, query information is obtained；
It is inquired according to the query information, the message mould in the first Cache is extracted according to query result
Formula encodes the information further according to the massage pattern, obtains encoded information, and the encoded information is sent
It is stored to Distributed Message Queue cluster；
It is inquired according to the query information, the message mould in the second Cache is extracted according to query result
Formula obtains the corresponding encoded information from the Distributed Message Queue cluster further according to the massage pattern and is solved
Code, and consumed according to decoding result.
As an embodiment, described to be inquired according to the query information, first is extracted according to query result
The massage pattern in Cache, comprising the following steps:
It is inquired according to the query information, judges whether current massage pattern is stored in the first Cache
In；
If current massage pattern has been stored in the first Cache, the institute in the first Cache is extracted
State massage pattern；
If current massage pattern is not stored in the first Cache, according to the registration in the information
Information uploads the massage pattern, and the massage pattern is stored in pattern database；Meanwhile by the message
Mode is stored in the first Cache.
As an embodiment, described according to the query information, that extracts in the second Cache described disappears
Breath mode, comprising the following steps:
It is inquired according to the query information, judges whether current massage pattern is stored in second cache
In device；
If current massage pattern has been stored in the second Cache, the institute in the second Cache is extracted
State massage pattern；
If current massage pattern is not stored in the second Cache, according to the query information and the mould
The comparison of formula database, obtains the corresponding massage pattern according to result, meanwhile, the second Cache is stored in by described
In.
As an embodiment, described that the massage pattern is handled, obtain query information, including following step
It is rapid:
Hashing algorithm is carried out to the massage pattern, obtains cryptographic Hash；
Classification processing is carried out to the massage pattern, obtains classification data；
Merge the cryptographic Hash and the classification data to obtain the query information.
As an embodiment, the processing method provided by the invention based on Distributed Message Queue middleware, also
The following steps are included:
Before being inquired according to the query information, version information and timestamp are encapsulated into the message data
In.
Correspondingly, the present invention also provides a kind of processing system based on Distributed Message Queue middleware, including obtain mould
Block, operation module extract coding module and extract decoder module；
The acquisition module establishes information according to upload message for obtaining the upload message in external message stream；
The information includes massage pattern；
The operation module obtains query information for handling the massage pattern；
It is high to extract first according to query result for being inquired according to the query information for the extraction coding module
The massage pattern in fast buffer encodes the information further according to the massage pattern, obtains coding letter
Breath, and the encoded information is sent to the storage of Distributed Message Queue cluster；
It is high to extract second according to query result for being inquired according to the query information for the extraction decoder module
The massage pattern in fast buffer is corresponded to from the Distributed Message Queue cluster further according to the massage pattern
The encoded information be decoded, and consumed according to decoding result.
As an embodiment, the extraction coding module, including the first judging unit, the first extraction unit and
Register storage unit；
Whether first judging unit judges current massage pattern for being inquired according to the query information
It is stored in the first Cache；
First extraction unit mentions if having been stored in the first Cache for current massage pattern
Take the massage pattern in the first Cache；
The registration storage unit, if being not stored in the first Cache for current massage pattern, root
The massage pattern is uploaded according to the registration information in the information, and the massage pattern is stored in pattern count
According in library；Meanwhile the massage pattern being stored in the first Cache.
As an embodiment, the extraction decoder module, including second judgment unit, the second extraction unit and
Compare storage unit；
Whether the second judgment unit judges current massage pattern for being inquired according to the query information
It is stored in second Cache；
Second extraction unit mentions if having been stored in the second Cache for current massage pattern
Take the massage pattern in the second Cache；
The comparison storage unit, if being not stored in the second Cache for current massage pattern, root
It is compared according to the query information and the pattern database, the corresponding massage pattern is obtained according to result, meanwhile, it will be described
It is stored in the second Cache.
As an embodiment, the operation module, including hashing algorithm unit, classification processing unit and merging
Unit；
The hashing algorithm unit obtains cryptographic Hash for carrying out hashing algorithm to the massage pattern；
The classification processing unit obtains classification data for carrying out classification processing to the massage pattern；
The combining unit, for merging the cryptographic Hash and the classification data to obtain the query information.
As an embodiment, the processing system provided by the invention based on Distributed Message Queue middleware, also
Including package module；
The package module, for before being inquired according to the query information, version information and timestamp to be sealed
It is attached in the message data.
Compared with prior art, the technical program has the advantage that
Processing method and processing system provided by the invention based on Distributed Message Queue middleware, by according to inquiry
The massage pattern stored in the first Cache of information extraction and the second high-speed buffer, and according in the first Cache
Massage pattern information is encoded, by obtained encoded information be sent to Distributed Message Queue cluster storage；Root
Corresponding encoded information is obtained from Distributed Message Queue cluster according to the massage pattern in the second Cache to be decoded,
And it is consumed according to decoding result.The invention avoids processes of " shaking hands " under line, when the producer is due to demand or other originals
Because not needing line and sending a notice consumer, the upgrading of massage pattern can be automatically performed, subtracted when information pattern being caused to need to change
Light daily maintenance and operation cost.
Detailed description of the invention
Fig. 1 is the process signal for the processing method based on Distributed Message Queue middleware that the embodiment of the present invention one provides
Figure；
Fig. 2 is the structural representation of the processing system provided by Embodiment 2 of the present invention based on Distributed Message Queue middleware
Figure；
Fig. 3 is the structural schematic diagram of operation module in Fig. 2；
Fig. 4 is the structural schematic diagram that coding module is extracted in Fig. 2；
Fig. 5 is the structural schematic diagram that decoder module is extracted in Fig. 2.
In figure: 100, obtaining module；200, operation module；210, hashing algorithm unit；220, classification processing unit；230,
Combining unit；300, coding module is extracted；310, the first judging unit；320, the first extraction unit；330, storage unit is registered；
400, decoder module is extracted；410, second judgment unit；420, the second extraction unit；430, storage unit is compared；500, it encapsulates
Module.
Specific embodiment
Below in conjunction with attached drawing, the technical characteristic and advantage above-mentioned and other to the present invention are clearly and completely described,
Obviously, described embodiment is only section Example of the invention, rather than whole embodiments.
Referring to Fig. 1, the processing method based on Distributed Message Queue middleware that the embodiment of the present invention one provides, including
Following steps:
Upload message in S100, acquisition external message stream, establishes information according to upload message；Information includes
Massage pattern；
S200, massage pattern is handled, obtains query information；
S300, it is inquired according to query information, the massage pattern in the first Cache is extracted according to query result,
Information is encoded further according to massage pattern, obtains encoded information, and encoded information is sent to distributed message team
The storage of column cluster；
S400, it is inquired according to query information, the massage pattern in the second Cache is extracted according to query result,
Corresponding encoded information is obtained from Distributed Message Queue cluster further according to massage pattern to be decoded, and according to decoding result
It is consumed.
It should be noted that upload message is to need to change in external message stream or a newly generated product industry
The information flow that business includes；And there are numerous such upload messages in external message stream, upload message is according to actual needs
It generates, so being the real-time upload message obtained in external message stream in this present embodiment.It, can also in other embodiments
To be to obtain upload message again according to other modes judgement.
Query information can be classified to each massage pattern for query messages mode, then by each message mould
Its corresponding query information, metadata information can be all stored in a massage pattern by formula when storage, that is to say, that
A variety of different massage patterns are store in a kind of the first Cache and the second Cache, are being used and current phase
When the massage pattern answered, use can be extracted.
Processing method provided by the invention based on Distributed Message Queue middleware, by extracting the according to query information
The massage pattern stored in one Cache and the second high-speed buffer, and according to the massage pattern in the first Cache
Information is encoded, obtained encoded information is sent to the storage of Distributed Message Queue cluster；According to the second high speed
Massage pattern in buffer obtains corresponding encoded information from Distributed Message Queue cluster and is decoded, and according to decoding
As a result it is consumed.The invention avoids processes of " shaking hands " under line, as the producer since demand or other reasons lead to information
It when mode needs to change, does not need line and sends a notice consumer, the upgrading of massage pattern can be automatically performed, mitigate daily maintenance
And operation cost.
Further, it is inquired according to query information, the message in the first Cache is extracted according to query result
Mode, comprising the following steps:
S310, it is inquired according to query information, judges whether current massage pattern is stored in the first Cache
In；
If S320, current massage pattern have been stored in the first Cache, the first Cache is extracted
In massage pattern；
If S330, current massage pattern are not stored in the first Cache, according to the note in information
Volume information uploads massage pattern, and massage pattern is stored in pattern database；Meanwhile massage pattern being stored in
In first Cache.
In this present embodiment, among step S330 simultaneously, massage pattern is stored in the first Cache.It is also
The massage pattern being stored in the first Cache is come out in advance；And every carry out one query, regardless of current first high speed
Buffer whether there is massage pattern corresponding with query information, finally all can deposit again massage pattern production corresponding with query information
It is stored in the first Cache, and the massage pattern can be also stored in pattern database, in general in pattern database
Massage pattern finally can all be stored in the first Cache, inquiry each time is also to a mistake of current message mode
The process of filter avoids and repeats to store the same massage pattern in the first Cache；Network access can be reduced in this way
Number mitigates system and runs burden, improves operational efficiency.Directly massage pattern upload according to registration information and realizes message
The automatic registration of mode is supported without additional programs.What needs to be explained here is that nested may also also may be used in a massage pattern
To there is other massage pattern.
Further, according to query information, the massage pattern in the second Cache is extracted, comprising the following steps:
S410, it is inquired according to query information, judges whether current massage pattern is stored in the second Cache
In；
If S420, current massage pattern have been stored in the second Cache, the second Cache is extracted
In massage pattern；
If S430, current massage pattern are not stored in the second Cache, according to query information and mode
Database comparison, obtains corresponding massage pattern according to result, meanwhile, it will be stored in the second Cache.
In this present embodiment, among step S430 simultaneously, massage pattern is stored in the first Cache.It is also
The massage pattern being stored in the first Cache is come out in advance；Second Cache and the first Cache one
Sample itself is stored with various types of messages mode, when being inquired, if disappeared in current second Cache there is no corresponding
Breath mode then compares according to query information and pattern database, obtains corresponding massage pattern according to result, meanwhile, it will store
In the second Cache, that is to say, that the massage pattern in pattern database finally can all be stored in the first Cache
In, inquiry each time is also the process of a filtering to current message mode, avoids and repeats in the first Cache
The same massage pattern of storage；It can reduce network access times in this way, mitigate system and run burden, on message handling ability
It rises, system stability and availability of service improve.
Further, step S200 the following steps are included:
S210, hashing algorithm is carried out to massage pattern, obtains cryptographic Hash；
S220, classification processing is carried out to massage pattern, obtains classification data；
S230, merge cryptographic Hash and classification data to obtain query information.
Here hashing algorithm can be to be carried out by md5, obtains one 128 cryptographic Hash, and classification data is pair
Massage pattern carries out what classification processing obtained.In this present embodiment, the algorithmic procedure of cryptographic Hash and classification data is not carried out
It limits.
Further, the processing method based on Distributed Message Queue middleware of the offer of the embodiment of the present invention one further includes
Following steps:
Before being inquired according to query information, version information and timestamp are encapsulated into message data.
When version information and timestamp are stored in the first Cache and the second Cache to massage pattern
Play booster action.
Concrete principle of the invention is described in detail by taking Avro and Kafka as an example below:
Kafka Message Queuing Middleware includes the producer, manager and consumer；
Producer sends the information with massage pattern and gives Avro encoder；
Encoder, which is mainly responsible for, encodes information by Avro, the embedded massage pattern of Avro encoder
Cache.Cache saves all used Avro massage patterns；Before the coding, it will can currently disappear
It ceases massage pattern used in information and carries out md5 (hash) operation, take one 128 cryptographic Hash.Then, encoder can be taken
Whether Kafka topic (classification data) and cryptographic Hash conjunctive query local cache see massage pattern in local
Exist inside Cache.If had existed, encoder will use the massage pattern taken and encode to information,
And Kafka cluster is sent by obtained decoded information.If it does not, it can call automatically message mould by HTTP POST
Formula registrar uploads the massage pattern.It, can be by this message mould after massage pattern registrar receives registration information
Massage pattern database is written in formula and its relevant metadata information.At the same time, this new massage pattern is loaded into encoder
Cache so that encoder will use the massage pattern taken and encode to information.
After consumer receives the information, it can be decoded by decoder by Avro.In Avro decoder
Face embeds a Cache, saves all decoders used massage pattern.When decoding, decoder can be disappeared
Then the cryptographic Hash of breath mode is inquired the massage pattern in local cache by Kafka topic and cryptographic Hash, is checked
Whether current message mode has existed in the Cache of decoder.Disappear if it does, encoder will use this taken
Breath mode is decoded information, consumes current information.If it does not, encoder can employ the Kafka certainly
Topic and cryptographic Hash simultaneously go on massage pattern registrar to obtain this new information by HTTP GET.Message mould
Formula registrar is responsible for taking out corresponding massage pattern inside massage pattern database and returns to encoder.It is same with this
When, this massage pattern can be fitted into local cache by decoder, and the information for consuming same massage pattern uses.
It is understood that information is after massage pattern is encoded, when consuming this information,
It has to be decoded by this massage pattern.And during decoded, it is decoded using following algorithm:
Wherein, getSchemaByTopicAndSchema (topicA, md5) is accomplished by
It should be pointed out that each element corresponding to list here may be a different types of sub- letter
Breath has different sub- massage patterns.Although this scheme is illustrated based on Avro and Kafka, other message queue is used
With other massage pattern mode, the present invention is equally applicable.
Based on the same inventive concept, the embodiment of the present invention also provides a kind of processing based on Distributed Message Queue middleware
System, the process that the implementation of the system can refer to the above method is realized, it is no longer redundant later to repeat place.
As shown in Fig. 2, being the processing system provided by Embodiment 2 of the present invention based on Distributed Message Queue middleware
Structural schematic diagram, including obtain module 100, operation module 200, extract coding module 300 and extract decoder module 400；
It obtains module 100 to be used to obtain the upload message in external message stream, information is established according to upload message；Disappear
Ceasing information includes massage pattern；
Operation module 200 obtains query information for handling massage pattern；
Coding module 300 is extracted for being inquired according to query information, the first cache is extracted according to query result
Massage pattern in device encodes information further according to massage pattern, obtains encoded information, and encoded information is sent
It is stored to Distributed Message Queue cluster；
Decoder module 400 is extracted for being inquired according to query information, the second cache is extracted according to query result
Massage pattern in device obtains corresponding encoded information from Distributed Message Queue cluster further according to massage pattern and is solved
Code, and consumed according to decoding result.
As shown in figure 4, being to extract the structural schematic diagram of coding module, including the first judging unit 310, the first extraction unit
320 and registration storage unit 330；
First judging unit 310 judges whether current massage pattern is stored in for being inquired according to query information
In first Cache；
If the first extraction unit 320 has been stored in the first Cache for current massage pattern, extract
Massage pattern in first Cache；
If registration storage unit 330 is not stored in the first Cache for current massage pattern, basis
Registration information in information uploads massage pattern, and massage pattern is stored in pattern database；Meanwhile it will
Massage pattern is stored in the first Cache.
As shown in figure 5, being to extract the structural schematic diagram of decoder module, including second judgment unit 410, the second extraction unit
420 and comparison storage unit 430；
Second judgment unit 410 judges whether current massage pattern is stored in for being inquired according to query information
In second Cache；
If the second extraction unit 420 has been stored in the second Cache for current massage pattern, extract
Massage pattern in second Cache；
If comparison storage unit 430 is not stored in the second Cache for current massage pattern, basis
Query information and pattern database compare, and obtain corresponding massage pattern according to result, meanwhile, the second cache will be stored in
In device.
As shown in figure 3, being the structural schematic diagram of operation module, including hashing algorithm unit 210, classification processing unit 220
And combining unit 230；
Hashing algorithm unit 210 is used to carry out hashing algorithm to massage pattern, obtains cryptographic Hash；
Classification processing unit 220 is used to carry out classification processing to massage pattern, obtains classification data；
Combining unit 230 is for merging cryptographic Hash and classification data to obtain query information.
Further, as shown in Figure 1, the processing based on Distributed Message Queue middleware that the embodiment of the present invention one provides
System further includes package module 500；
Package module 500 is used for before being inquired according to query information, and version information and timestamp are encapsulated into and disappeared
It ceases in data.
Processing system provided by the invention based on Distributed Message Queue middleware, by extracting the according to query information
The massage pattern stored in one Cache and the second high-speed buffer, and according to the massage pattern in the first Cache
Information is encoded, obtained encoded information is sent to the storage of Distributed Message Queue cluster；According to the second high speed
Massage pattern in buffer obtains corresponding encoded information from Distributed Message Queue cluster and is decoded, and according to decoding
As a result it is consumed.The invention avoids processes of " shaking hands " under line, as the producer since demand or other reasons lead to information
It when mode needs to change, does not need line and sends a notice consumer, the upgrading of massage pattern can be automatically performed, mitigate daily maintenance
And operation cost.
Although the invention has been described by way of example and in terms of the preferred embodiments, but it is not for limiting the present invention, any this field
Technical staff without departing from the spirit and scope of the present invention, may be by the methods and technical content of the disclosure above to this hair
Bright technical solution makes possible variation and modification, therefore, anything that does not depart from the technical scheme of the invention, and according to the present invention
Technical spirit any simple modifications, equivalents, and modifications to the above embodiments, belong to technical solution of the present invention
Protection scope.
Claims (8)
1. a kind of processing method based on Distributed Message Queue middleware, which comprises the following steps:
The upload message in external message stream is obtained, information is established according to upload message；The information includes message
Mode；
The massage pattern is handled, query information is obtained；
It is inquired according to the query information, the massage pattern in the first Cache is extracted according to query result,
The information is encoded further according to the massage pattern, obtains encoded information, and the encoded information is sent to
The storage of Distributed Message Queue cluster；
It is inquired according to the query information, the massage pattern in the second Cache is extracted according to query result,
The corresponding encoded information is obtained from the Distributed Message Queue cluster further according to the massage pattern to be decoded, and
It is consumed according to decoding result；
It is described to be inquired according to the query information, the message mould in the first Cache is extracted according to query result
Formula, comprising the following steps:
It is inquired according to the query information, judges whether current massage pattern is stored in the first Cache；
If current massage pattern has been stored in the first Cache, that extracts in the first Cache described disappears
Breath mode；
If current massage pattern is not stored in the first Cache, according to the registration information in the information
The massage pattern is uploaded, and the massage pattern is stored in pattern database；Meanwhile by the massage pattern
It is stored in the first Cache.
2. the processing method as described in claim 1 based on Distributed Message Queue middleware, which is characterized in that the basis
The query information extracts the massage pattern in the second Cache, comprising the following steps:
It is inquired according to the query information, judges whether current massage pattern is stored in second Cache
In；
If current massage pattern has been stored in the second Cache, that extracts in the second Cache described disappears
Breath mode；
If current massage pattern is not stored in the second Cache, according to the query information and the pattern count
It is compared according to library, the corresponding massage pattern is obtained according to result, meanwhile, the massage pattern is stored in the second cache
In device.
3. the processing method as described in claim 1 based on Distributed Message Queue middleware, which is characterized in that described to institute
It states massage pattern to be handled, obtains query information, comprising the following steps:
Hashing algorithm is carried out to the massage pattern, obtains cryptographic Hash；
Classification processing is carried out to the massage pattern, obtains classification data；
Merge the cryptographic Hash and the classification data to obtain the query information.
4. the processing method as described in claim 1 based on Distributed Message Queue middleware, which is characterized in that further include with
Lower step:
Before being inquired according to the query information, version information and timestamp are encapsulated into message data.
5. a kind of processing system based on Distributed Message Queue middleware, which is characterized in that including obtaining module, operation mould
Block extracts coding module and extracts decoder module；
The acquisition module establishes information according to upload message for obtaining the upload message in external message stream；It is described
Information includes massage pattern；
The operation module obtains query information for handling the massage pattern；
It is slow to extract the first high speed according to query result for being inquired according to the query information for the extraction coding module
The massage pattern in storage encodes the information further according to the massage pattern, obtains encoded information, and
The encoded information is sent to the storage of Distributed Message Queue cluster；
It is slow to extract the second high speed according to query result for being inquired according to the query information for the extraction decoder module
The massage pattern in storage obtains corresponding institute further according to the massage pattern from the Distributed Message Queue cluster
It states encoded information to be decoded, and is consumed according to decoding result；
The extraction coding module, including the first judging unit, the first extraction unit and registration storage unit；
First judging unit judges whether current massage pattern stores for being inquired according to the query information
In the first Cache；
First extraction unit extracts if having been stored in the first Cache for current massage pattern
The massage pattern in one Cache；
The registration storage unit, if being not stored in the first Cache for current massage pattern, according to institute
The registration information stated in information uploads the massage pattern, and the massage pattern is stored in pattern database
In；Meanwhile the massage pattern being stored in the first Cache.
6. the processing system as claimed in claim 5 based on Distributed Message Queue middleware, which is characterized in that the extraction
Decoder module, including second judgment unit, the second extraction unit and comparison storage unit；
The second judgment unit judges whether current massage pattern stores for being inquired according to the query information
In second Cache；
Second extraction unit extracts if having been stored in the second Cache for current massage pattern
The massage pattern in two Caches；
The comparison storage unit, if being not stored in the second Cache for current massage pattern, according to institute
It states query information and the pattern database compares, the corresponding massage pattern is obtained according to result, meanwhile, by the storage
In the second Cache.
7. the processing system as claimed in claim 5 based on Distributed Message Queue middleware, which is characterized in that the operation
Module, including hashing algorithm unit, classification processing unit and combining unit；
The hashing algorithm unit obtains cryptographic Hash for carrying out hashing algorithm to the massage pattern；
The classification processing unit obtains classification data for carrying out classification processing to the massage pattern；
The combining unit, for merging the cryptographic Hash and the classification data to obtain the query information.
8. the processing system as claimed in claim 5 based on Distributed Message Queue middleware, which is characterized in that further include envelope
Die-filling piece；
The package module, for before being inquired according to the query information, version information and timestamp to be encapsulated into
In message data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710533747.7A CN107194013B (en) | 2017-07-03 | 2017-07-03 | Processing method and processing system based on Distributed Message Queue middleware |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710533747.7A CN107194013B (en) | 2017-07-03 | 2017-07-03 | Processing method and processing system based on Distributed Message Queue middleware |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107194013A CN107194013A (en) | 2017-09-22 |
| CN107194013B true CN107194013B (en) | 2019-10-29 |
Family
ID=59881211
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710533747.7A Active CN107194013B (en) | 2017-07-03 | 2017-07-03 | Processing method and processing system based on Distributed Message Queue middleware |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107194013B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108595483B (en) * | 2018-03-13 | 2020-11-24 | 腾讯科技（深圳）有限公司 | Data processing method and related device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101917332A (en) * | 2010-05-24 | 2010-12-15 | 宁波东海蓝帆科技有限公司 | Message delivery method for reducing network load of message midware |
| CN104111998A (en) * | 2014-07-09 | 2014-10-22 | 江西理工大学 | Method and device for sorting coding and integrated exchange and management of heterogeneous data of enterprise |
| CN106487891A (en) * | 2016-10-11 | 2017-03-08 | 北京集奥聚合科技有限公司 | A kind of method of the process message based on kafka |
-
2017
- 2017-07-03 CN CN201710533747.7A patent/CN107194013B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101917332A (en) * | 2010-05-24 | 2010-12-15 | 宁波东海蓝帆科技有限公司 | Message delivery method for reducing network load of message midware |
| CN104111998A (en) * | 2014-07-09 | 2014-10-22 | 江西理工大学 | Method and device for sorting coding and integrated exchange and management of heterogeneous data of enterprise |
| CN106487891A (en) * | 2016-10-11 | 2017-03-08 | 北京集奥聚合科技有限公司 | A kind of method of the process message based on kafka |
Non-Patent Citations (1)
| Title |
|---|
| 利用SCTP对MRB消息中间件的改进;陈飞;《中国优秀硕士学位论文全文数据库 信息科技辑》;20081015(第10期);I138-63 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107194013A (en) | 2017-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103297807A (en) | Hadoop-platform-based method for improving video transcoding efficiency | |
| CA2718722A1 (en) | Modular transcoding pipeline | |
| CN109274973A (en) | Fast video coding/decoding method on embedded-type ARM platform | |
| CN111935663B (en) | Sensor data stream processing method, device, medium and electronic equipment | |
| CN113181646A (en) | Game data method and device, electronic equipment and storage medium | |
| CN103051953B (en) | One utilizes video player to carry out coding/decoding method to video file | |
| WO2023179161A1 (en) | Video frame rate control method and apparatus, and electronic device and storage medium | |
| CN107194013B (en) | Processing method and processing system based on Distributed Message Queue middleware | |
| CN107426624A (en) | Video game realization method and system, television terminal, cloud server, medium | |
| CN113961289A (en) | Data processing method, device, equipment and storage medium | |
| CN111813529B (en) | Data processing method, device, electronic equipment and storage medium | |
| US20210014540A1 (en) | Method and system for codec of visual feature data | |
| CN105323593A (en) | Multimedia transcoding scheduling method and multimedia transcoding scheduling device | |
| CN106846190A (en) | A kind of intellectual property case management system | |
| CN106791932A (en) | Distributed trans-coding system, method and its device | |
| CN111541864B (en) | Digital retina software defined camera method and system | |
| CN110012003A (en) | A kind of cloud application grabs screen method and apparatus | |
| CN105338371A (en) | Multimedia transcoding scheduling method and apparatus | |
| CN114880107A (en) | High-efficient low-cost cloud game system | |
| CN112188235B (en) | Media processing mode selection method and media processing method | |
| CN111818337A (en) | Method and system for compatible monitoring of multiple video monitoring devices | |
| CN112541391A (en) | Violation behavior identification method and system based on examination video analysis | |
| CN115734008B (en) | Method, system and medium for rapidly integrating video resources of content provider | |
| CN205430500U (en) | Multi -media transcoding dispatch device | |
| CN102271122B (en) | A kind of rubbish contents applied to P2P networks determine method and its system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20210120 Address after: 401120 room 310, business building a, East terminal area of Chongqing Jiangbei International Airport, Yubei District, Chongqing Patentee after: Chongqing permutation Technology Co.,Ltd. Address before: Room 501, building 19, haichuangyuan, No. 998, Wenyi West Road, Hangzhou, Zhejiang 311121 Patentee before: HANGZHOU ECREDITPAL TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right |