CN106528306A - Message queue storage method - Google Patents
Message queue storage method Download PDFInfo
- Publication number
- CN106528306A CN106528306A CN201610983556.6A CN201610983556A CN106528306A CN 106528306 A CN106528306 A CN 106528306A CN 201610983556 A CN201610983556 A CN 201610983556A CN 106528306 A CN106528306 A CN 106528306A
- Authority
- CN
- China
- Prior art keywords
- message
- storage
- redis
- mysql
- internal memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a message queue storage method. The method comprises the following steps of: when a message production amount is increased, writing messages into an internal memory; when the memory space of the internal memory is up to an upper limit, storing the residual messages into an redis; when the memory space of the redis is up to an upper limit, storing the residual messages into an mysql; when memory space of the mysql is up to a peak value, storing the residual messages into a file, wherein the file can be a local file or a network file; when a message queue is consumed, decreasing the message amount in the internal memory; when the message amount in the internal memory is decreased to a threshold value, complementing the data in the redis into the internal memory and detecting a space margin in the redis; when the data amount in the redis is redced to 50% of a capacity, complenting messages in a mysql database into the redis and detecting a message margin in the mysql; and when the data in the mysql is lower than 50% of the capacity, importing the data in the file into the database in batches. According to the message queue storage method, the pressure of a storage device is decreased through multi-stage storage and consumption.
Description
Technical field
The present invention relates to field of data storage, and in particular to a kind of storage method of message queue.
Background technology
In soft project, very universal, the click of the such as keyboard of windows systems and mouse of queue technology,
The technologies such as stock's warehouse-in, outbound in the short message receiving-transmitting, store in mobile phone realize that queue technology is mainly used to realize computing unit
Decoupling, redundancy, horizontal extension, flexibly collocation and situations such as peak load shifting, for any IT architecture or various application software,
Message queue is all a vital component.Along with the arrival in big data epoch, people are for the excavation of mass data
Will be more and more with situation, the increase in productivity of a new ripple and the arrival of consumer surplus's tide are this imply that, therefore it is anxious
High Availabitity, height is needed to handle up, efficient queue scheme carrys out the development of supporting business system.
The content of the invention
The purpose of the present invention is the deficiency existed for existing technology, it is proposed that a kind of message queue of high working efficiency
Storage method.
Technical problem solved by the invention employs the following technical solutions to realize a kind of storage method of message queue, bag
Include following steps：
(1) store message
1. when message production quantity increases, in write internal memory；
2., after the amount of storage of internal memory reaches the upper limit, remaining message continues to be stored in redis；
3., after redis amount of storage reaches the upper limit, remaining message may proceed to storage in mysql；
4., after mysql amount of storage is to peaking, remaining message is stored in file, and file can be local file or net
Network file.
(2) consume message
1., when message queue is consumed, the message number in internal memory reduces；
2., after the message number in internal memory is reduced to a threshold value, a thread task is started, by the number inside redis
According to adding in internal memory, and detect the spatial margin in redis；
3., after the data volume inside the redis is reduced to the 50% of capacity, a thread task is started, by mysql data
Message in storehouse is added in redis, and detects the message surplus in mysql；
4., after the data inside the mysql are less than capacity 50%, a thread is started, by the batch data in File files
Import in database.
Described the 1st grade ' memory storage ' rank is k levels, and message number is from several k to the quantity of hundreds of k.
The redis of described level 2 cache memory supports that storage rank is M levels, and message number is up to tens of thousands of to millions of numbers
Amount.
The mysql of described 3rd level caching supports that storage rank is G levels, and message number is up to several ten million quantity.
The File of the 4th grade of described caching supports that storage rank is T levels, storage capacity and hardware configuration positive correlation.
Beneficial effects of the present invention are：A kind of storage method of message queue is proposed, development and maintenance cost is reduced, is increased
Adding system data-handling capacity, can allow more users to experience the enjoyment of big data, and may determine that the index source of user's foundation
Whether data are legal.
Description of the drawings
Fig. 1 is the system structure diagram of the present invention；
Fig. 2 is the flow chart of the present invention.
Specific embodiment
Referring to the drawings, a kind of storage method of message queue, comprises the steps：
(1) store message
1. when message production quantity increases, in write internal memory；
2., after the amount of storage of internal memory reaches the upper limit, remaining message continues to be stored in redis；
3., after redis amount of storage reaches the upper limit, remaining message may proceed to storage in mysql；
4., after mysql amount of storage is to peaking, remaining message is stored in file, and file can be local file or net
Network file.
(2) consume message
1., when message queue is consumed, the message number in internal memory reduces；
2., after the message number in internal memory is reduced to a threshold value, a thread task is started, by the number inside redis
According to adding in internal memory, and detect the spatial margin in redis；
3., after the data volume inside the redis is reduced to the 50% of capacity, a thread task is started, by mysql data
Message in storehouse is added in redis, and detects the message surplus in mysql；
4., after the data inside the mysql are less than capacity 50%, a thread is started, by the batch data in File files
Import in database.
Described the 1st grade ' memory storage ' rank is k levels, and message number is from several k to the quantity of hundreds of k.
The redis of described level 2 cache memory supports that storage rank is M levels, and message number is up to tens of thousands of to millions of numbers
Amount.
The mysql of described 3rd level caching supports that storage rank is G levels, and message number is up to several ten million quantity.
The File of the 4th grade of described caching supports that storage rank is T levels, storage capacity and hardware configuration positive correlation.
The present invention proposes a kind of storage method of message queue, and multistage storage and consumption reduce the pressure of storage device
Power, more increase handle up, the storage method of efficient message queue carrys out the development of supporting business system.
The foregoing description of the disclosed embodiments, enables professional and technical personnel in the field to realize or using the present invention.
Various modifications to these embodiments will be apparent for those skilled in the art, as defined herein
General Principle can be realized without departing from the spirit or scope of the present invention in other embodiments.Therefore, the present invention
The embodiments shown herein is not intended to be limited to, and is to fit to and principles disclosed herein and features of novelty phase one
The most wide scope for causing.
Claims (5)
1. a kind of storage method of message queue, it is characterised in that comprise the steps：
(1) store message
1. when message production quantity increases, in write internal memory；
2., after the amount of storage of internal memory reaches the upper limit, remaining message continues to be stored in redis；
3., after redis amount of storage reaches the upper limit, remaining message may proceed to storage in mysql；
4. after mysql amount of storage is to peaking, remaining message is stored in file, and file can be local file or network text
Part,
(2) consume message
1., when message queue is consumed, the message number in internal memory reduces；
2. after the message number in internal memory is reduced to a threshold value, start a thread task, the data inside redis are mended
It is charged in internal memory, and detects the spatial margin in redis；
3., after the data volume inside the redis is reduced to the 50% of capacity, a thread task is started, by mysql databases
Message add in redis, and detect the message surplus in mysql；
4. after the data inside the mysql are less than capacity 50%, start a thread, the batch data in File files is imported
In database.
2. the storage method of a kind of message queue according to claim 1, it is characterised in that：Described the 1st grade ' internal memory
Storage ' rank be k levels, message number is from several k to the quantity of hundreds of k.
3. the storage method of a kind of message queue according to claim 1, it is characterised in that：Described level 2 cache memory
Redis supports that storage rank is M levels, and message number is up to tens of thousands of to millions of quantity.
4. the storage method of a kind of message queue according to claim 1, it is characterised in that：Described 3rd level caching
Mysql supports that storage rank is G levels, and message number is up to several ten million quantity.
5. the storage method of a kind of message queue according to claim 1, it is characterised in that：The 4th grade of described caching
File supports that storage rank is T levels, storage capacity and hardware configuration positive correlation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610983556.6A CN106528306A (en) | 2016-11-08 | 2016-11-08 | Message queue storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610983556.6A CN106528306A (en) | 2016-11-08 | 2016-11-08 | Message queue storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106528306A true CN106528306A (en) | 2017-03-22 |
Family
ID=58350092
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610983556.6A Pending CN106528306A (en) | 2016-11-08 | 2016-11-08 | Message queue storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106528306A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109298957A (en) * | 2018-10-18 | 2019-02-01 | 大唐网络有限公司 | A kind of information management method with high reliability for supporting to regularly send |
| CN110333956A (en) * | 2019-05-23 | 2019-10-15 | 平安普惠企业管理有限公司 | Message storage method, device, medium and electronic equipment in message queue |
| CN110570223A (en) * | 2018-06-06 | 2019-12-13 | 北京三快在线科技有限公司 | Intelligent distribution method and system for merchant ticket stock, electronic equipment and storage medium |
| WO2021180025A1 (en) * | 2020-03-13 | 2021-09-16 | 北京金山云网络技术有限公司 | Message processing method and apparatus, electronic device and medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858735A (en) * | 2005-12-30 | 2006-11-08 | 华为技术有限公司 | Method for processing mass data |
| US20120233668A1 (en) * | 2011-03-08 | 2012-09-13 | Rackspace Us, Inc. | Pluggable Allocation in a Cloud Computing System |
| CN103390049A (en) * | 2013-07-23 | 2013-11-13 | 南京联创科技集团股份有限公司 | Method for processing high-speed message queue overflow based on memory database cache |
| CN103399856A (en) * | 2013-07-01 | 2013-11-20 | 北京科东电力控制系统有限责任公司 | Explosive type data caching and processing system for SCADA system and method thereof |
| CN103838547A (en) * | 2012-11-27 | 2014-06-04 | 中国电信股份有限公司 | Queue management method and device |
| US20140373049A1 (en) * | 2013-06-14 | 2014-12-18 | ART19, Inc. | Method And Apparatus For Demonstrating The Exposure Of Consumers To Media Content And To Advertisements Which Are Delivered Over A Network. |
| CN104965850A (en) * | 2015-04-29 | 2015-10-07 | 云南电网有限责任公司 | Database high-available implementation method based on open source technology |
| CN105138410A (en) * | 2015-08-31 | 2015-12-09 | 北京锐安科技有限公司 | Message queue achievement method and device based on disk buffer |
-
2016
- 2016-11-08 CN CN201610983556.6A patent/CN106528306A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858735A (en) * | 2005-12-30 | 2006-11-08 | 华为技术有限公司 | Method for processing mass data |
| US20120233668A1 (en) * | 2011-03-08 | 2012-09-13 | Rackspace Us, Inc. | Pluggable Allocation in a Cloud Computing System |
| CN103838547A (en) * | 2012-11-27 | 2014-06-04 | 中国电信股份有限公司 | Queue management method and device |
| US20140373049A1 (en) * | 2013-06-14 | 2014-12-18 | ART19, Inc. | Method And Apparatus For Demonstrating The Exposure Of Consumers To Media Content And To Advertisements Which Are Delivered Over A Network. |
| CN103399856A (en) * | 2013-07-01 | 2013-11-20 | 北京科东电力控制系统有限责任公司 | Explosive type data caching and processing system for SCADA system and method thereof |
| CN103390049A (en) * | 2013-07-23 | 2013-11-13 | 南京联创科技集团股份有限公司 | Method for processing high-speed message queue overflow based on memory database cache |
| CN104965850A (en) * | 2015-04-29 | 2015-10-07 | 云南电网有限责任公司 | Database high-available implementation method based on open source technology |
| CN105138410A (en) * | 2015-08-31 | 2015-12-09 | 北京锐安科技有限公司 | Message queue achievement method and device based on disk buffer |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110570223A (en) * | 2018-06-06 | 2019-12-13 | 北京三快在线科技有限公司 | Intelligent distribution method and system for merchant ticket stock, electronic equipment and storage medium |
| CN109298957A (en) * | 2018-10-18 | 2019-02-01 | 大唐网络有限公司 | A kind of information management method with high reliability for supporting to regularly send |
| CN110333956A (en) * | 2019-05-23 | 2019-10-15 | 平安普惠企业管理有限公司 | Message storage method, device, medium and electronic equipment in message queue |
| WO2021180025A1 (en) * | 2020-03-13 | 2021-09-16 | 北京金山云网络技术有限公司 | Message processing method and apparatus, electronic device and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106528306A (en) | Message queue storage method | |
| CN101686209B (en) | Method and device for storing message in message retransmission system | |
| CN101996067B (en) | Data export method and device | |
| CN104156390A (en) | Comment recommendation method and system | |
| CN102158349A (en) | Log management device and method thereof | |
| CN102479223A (en) | Data query method and system | |
| CN103186537A (en) | Data reading-writing method and device based on index sorting | |
| CN102129425A (en) | Method and device for accessing large-object set table in data warehouse | |
| CN103051701A (en) | Cache admission method and system | |
| CN106055274A (en) | Data storage method, data reading method and electronic device | |
| CN108363772A (en) | A kind of register date storage method and device based on caching | |
| CN110750372A (en) | Log system based on shared memory and log management method | |
| CN104679493A (en) | Improved method for process event handling mechanism | |
| CN102932416A (en) | Intermediate data storage method, processing method and device in information flow task | |
| CN102867023A (en) | Method for storing and reading grid data and device | |
| CN102929401A (en) | Method and device for processing input method application resource or function based on input behavior | |
| CN105574124A (en) | Data storage system based on product information | |
| CN105654880A (en) | Method of controlling scale factor and method of controlling luminance including the same | |
| CN114817458A (en) | Bid-winning item retrieval method based on funnel model and cosine algorithm | |
| CN104572643B (en) | Searching method and search engine | |
| CN105574122A (en) | Product information-based data retrieval system | |
| CN105653646A (en) | Dynamic query system and method under concurrent query condition | |
| CN103577480A (en) | Parameter division system and method, and service processing system and method | |
| Lv et al. | Classification of data granularity in data warehouse | |
| CN105117176A (en) | Method and system for data reading-writing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20170322 |