CN105786629A - Data processing method based on message queue - Google Patents
Data processing method based on message queue Download PDFInfo
- Publication number
- CN105786629A CN105786629A CN201610071604.4A CN201610071604A CN105786629A CN 105786629 A CN105786629 A CN 105786629A CN 201610071604 A CN201610071604 A CN 201610071604A CN 105786629 A CN105786629 A CN 105786629A
- Authority
- CN
- China
- Prior art keywords
- data processing
- server
- message queue
- message
- processing request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Abstract
The invention relates to the data processing technology, aims at solving the problem that the existing data processing efficiency is low. The invention provides a data processing method based on a message queue. The method comprises the following steps: firstly, a web server receiving a data processing request of a user, and sending the data processing request to a message queue server; and then the message queue server performing enqueuer processing on the data processing request, and returning response information to the user through the web server; finally, the message queue server orderly processing the data processing request according to first-in first-out principle. The method is suitable for mass data processing.
Description
Technical field
The present invention relates to data processing technique, particularly to a kind of data processing method based on message queue.
Background technology
For realizing the quick process of concurrent message, adopt the key-value data base of this advanced person of Redis, and the API of multiple language is provided.
Redis is one and increases income, advanced key-value storage, and is used for building high-performance, the perfect solution of extendible web application.
Redis inherits, from its many competitions, three main features of coming:
1) Redis data base is completely in internal memory, uses disk only for persistency；
2) comparing the storage of many key assignments data, Redis has the data type of a set of relatively horn of plenty；
3) Redis can copy data to any number of from server.
The advantage 1 of Redis) abnormal quick: the speed of Redis is very fast, and energy per second performs about 110,000 set, about 81000+ bar record per second.
2) abundant data type is supported: Redis supports the already known picture list of most developer, set, ordered set, hash data type.This make it be very easy to solve various problems, as we know which problem be can process by its data type better.
3) operation is all atomicity: all Redis operation is atom, which ensure that if the Redis server that two clients access will obtain the value after updating simultaneously.
4) practical function instrument: Redis is instrument practical more than, it is possible in multiple use such as buffer memory, message, queue uses (the primary support publish/subscribe of Redis), any of short duration data, application program, such as web application session, webpage hit count etc..
Summary of the invention
The invention aims to solve the problem that existing data-handling efficiency is low.
For reaching above-mentioned purpose, the present invention provides a kind of data processing method based on message queue, it is characterised in that comprise the steps:
Step one, Web server receive the data processing request of user, and described data processing request is sent to Message Queuing server；
Described data processing request is joined the team process by step 2, Message Queuing server, returns response message by web server to user simultaneously；
Described data processing request is processed successively by step 3, Message Queuing server according to first in first out.
Specifically, in step 2, after the process of joining the team of described data processing request is completed by Message Queuing server, return the successful response message of process by web server to user.
Specifically, in step 3, if described data are processed unsuccessfully by Message Queuing server, then return, to user, the information that process is failed by web server.
Specifically, web server returns, to user, the information that process is failed by letter in standing and/or note.
Specifically, in step 3, described data processing request is processed and includes the principle according to first in first out and store data into successively in database server by described Message Queuing server.
The invention has the beneficial effects as follows: the present invention utilizes the key-value memory property that Redis is advanced, by the mode of message queue, the data processing request of user is processed, drastically increase the efficiency that data process.
Detailed description of the invention
Hereinafter technical scheme is described in further detail.
The present invention is directed to the inefficient problem that existing database data processes, a kind of data processing method based on message queue is provided, first the method comprises the steps:, web server receives the data processing request of user, and described data processing request is sent to Message Queuing server；Then, described data processing request is joined the team process by Message Queuing server, returns response message by web server to user simultaneously；Finally, described data processing request is processed successively by Message Queuing server according to first in first out.
After web server receives the data processing request that user sends, the request that user is sent is directly stored in message queue, namely carries out enqueue operations, because redis carries out in internal memory, so can very quick.
Then can make an immediate response the request of user, such as informs the user that successful operation, and now, backstage is at this moment but without carrying out database manipulation in fact, if database manipulation failure, then send note to user, or in station, letter notifies.At this moment in Message Queuing server, the server being usually a platform independent is operated, and queue message just now is further taken out, namely dequeue operation, follow the principle of " first in first out ", deal with successively, such as preserve to data base, send mail, note, letter etc. in standing.Wanting just as you to eat Fructus Vitis viniferae, you will not eat a Fructus Vitis viniferae, wash a Fructus Vitis viniferae, and you cluster of grapes wash clean, then can eat together.Utilize asynchronous process to achieve high concurrent like this, alleviate server stress.
Claims (5)
1. based on the data processing method of message queue, it is characterised in that comprise the steps:
Step one, web server receive the data processing request of user, and described data processing request is sent to Message Queuing server；
Described data processing request is joined the team process by step 2, Message Queuing server, returns response message by web server to user simultaneously；
Described data processing request is processed successively by step 3, Message Queuing server according to first in first out.
2. the data processing method based on message queue as claimed in claim 1, it is characterised in that in step 2, after the process of joining the team of described data processing request is completed by Message Queuing server, returns the successful response message of process by web server to user.
3. the data processing method based on message queue as claimed in claim 2, it is characterised in that in step 3, if described data are processed unsuccessfully by Message Queuing server, then return, to user, the information that process is failed by web server.
4. the data processing method based on message queue as claimed in claim 2, it is characterised in that web server returns, to user, the information that process is failed by letter in standing and/or note.
5. the data processing method based on message queue as described in Claims 1-4 any one, it is characterized in that, in step 3, described data processing request is processed and includes the principle according to first in first out and store data into successively in database server by described Message Queuing server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610071604.4A CN105786629A (en) | 2016-02-02 | 2016-02-02 | Data processing method based on message queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610071604.4A CN105786629A (en) | 2016-02-02 | 2016-02-02 | Data processing method based on message queue |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105786629A true CN105786629A (en) | 2016-07-20 |
Family
ID=56403186
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610071604.4A Pending CN105786629A (en) | 2016-02-02 | 2016-02-02 | Data processing method based on message queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105786629A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106484805A (en) * | 2016-09-22 | 2017-03-08 | 联动优势电子商务有限公司 | A kind of data export method and data handling system |
| CN107169064A (en) * | 2017-05-04 | 2017-09-15 | 聚好看科技股份有限公司 | Data asynchronous processing method and device |
| CN108874513A (en) * | 2017-05-11 | 2018-11-23 | 北京京东尚科信息技术有限公司 | Handle method, system, electronic equipment and the computer-readable medium of timed task |
| CN109542643A (en) * | 2018-11-16 | 2019-03-29 | 北京金山云网络技术有限公司 | The restoration methods and device of message in a kind of OpenStack system |
| CN113114725A (en) * | 2021-03-19 | 2021-07-13 | 中新网络信息安全股份有限公司 | Multi-node data interaction system based on HTTP (hyper text transport protocol) and implementation method thereof |
| CN113391933A (en) * | 2021-06-21 | 2021-09-14 | 山东国晟华文信息技术有限公司 | Method for processing fund |
| CN113568761A (en) * | 2020-04-28 | 2021-10-29 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
| CN103312624A (en) * | 2012-03-09 | 2013-09-18 | 腾讯科技（深圳）有限公司 | Message queue service system and method |
| CN103825835A (en) * | 2013-11-29 | 2014-05-28 | 中邮科通信技术股份有限公司 | Internet high concurrency seckilling system |
| CN104598563A (en) * | 2015-01-08 | 2015-05-06 | 北京京东尚科信息技术有限公司 | High concurrency data storage method and device |
| CN104731899A (en) * | 2015-03-19 | 2015-06-24 | 浪潮通用软件有限公司 | Data high concurrency processing method based on asynchronous processing |
| CN105187327A (en) * | 2015-08-14 | 2015-12-23 | 广东能龙教育股份有限公司 | Distributed message queue middleware |
-
2016
- 2016-02-02 CN CN201610071604.4A patent/CN105786629A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103312624A (en) * | 2012-03-09 | 2013-09-18 | 腾讯科技（深圳）有限公司 | Message queue service system and method |
| CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
| CN103825835A (en) * | 2013-11-29 | 2014-05-28 | 中邮科通信技术股份有限公司 | Internet high concurrency seckilling system |
| CN104598563A (en) * | 2015-01-08 | 2015-05-06 | 北京京东尚科信息技术有限公司 | High concurrency data storage method and device |
| CN104731899A (en) * | 2015-03-19 | 2015-06-24 | 浪潮通用软件有限公司 | Data high concurrency processing method based on asynchronous processing |
| CN105187327A (en) * | 2015-08-14 | 2015-12-23 | 广东能龙教育股份有限公司 | Distributed message queue middleware |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106484805A (en) * | 2016-09-22 | 2017-03-08 | 联动优势电子商务有限公司 | A kind of data export method and data handling system |
| CN107169064A (en) * | 2017-05-04 | 2017-09-15 | 聚好看科技股份有限公司 | Data asynchronous processing method and device |
| CN108874513A (en) * | 2017-05-11 | 2018-11-23 | 北京京东尚科信息技术有限公司 | Handle method, system, electronic equipment and the computer-readable medium of timed task |
| CN109542643A (en) * | 2018-11-16 | 2019-03-29 | 北京金山云网络技术有限公司 | The restoration methods and device of message in a kind of OpenStack system |
| CN109542643B (en) * | 2018-11-16 | 2021-04-30 | 北京金山云网络技术有限公司 | Method and device for recovering messages in OpenStack system |
| CN113568761A (en) * | 2020-04-28 | 2021-10-29 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
| CN113568761B (en) * | 2020-04-28 | 2023-06-27 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
| CN113114725A (en) * | 2021-03-19 | 2021-07-13 | 中新网络信息安全股份有限公司 | Multi-node data interaction system based on HTTP (hyper text transport protocol) and implementation method thereof |
| CN113391933A (en) * | 2021-06-21 | 2021-09-14 | 山东国晟华文信息技术有限公司 | Method for processing fund |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105786629A (en) | Data processing method based on message queue | |
| US11327945B2 (en) | Method and device for storing high-concurrency data | |
| CN103927218B (en) | Event distributing method and system | |
| US20190173969A1 (en) | Push notification delivery system | |
| US8849776B2 (en) | Method and system for resolving data inconsistency | |
| CN107093138B (en) | Auction Ask-Bid System and its operation method based on distributed clog-free asynchronous message tupe | |
| CN103581230B (en) | Document transmission system and method, receiving terminal, transmitting terminal | |
| CN104700836B (en) | A kind of audio recognition method and system | |
| CN103516585B (en) | Method and system for distributing messages according to priorities | |
| CN107734004A (en) | A kind of high concurrent SiteServer LBS based on Nginx, Redis | |
| CN104125163B (en) | Data processing method and device and terminal | |
| CN102932472B (en) | Data processing method and communication system | |
| CN106170016A (en) | A kind of method and system processing high concurrent data requests | |
| EP2698730A1 (en) | Data acquisition method, device and system | |
| Lockwood et al. | Implementing ultra low latency data center services with programmable logic | |
| CN101577705A (en) | Multi-core paralleled network traffic load balancing method and system | |
| CN108881485A (en) | The method for ensureing the high concurrent system response time under big data packet | |
| CN104899161B (en) | A kind of caching method of the continuous data protection based on cloud storage environment | |
| CN106034113A (en) | Data processing method and data processing device | |
| CN104899278A (en) | Method and apparatus for generating data operation logs of Hbase database | |
| CN107562803A (en) | Data supply system and method, terminal | |
| WO2019242279A1 (en) | Message processing method and device | |
| CN108076111B (en) | System and method for distributing data in big data platform | |
| CN103067419B (en) | A kind of distributed file system and the method controlling file storage within the system | |
| JP2023031248A (en) | Edge computing network, data transmission method, apparatus, device, and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160720 |
|
| RJ01 | Rejection of invention patent application after publication |