CN111949732A - System and method for real-time storage of massive concurrent messages - Google Patents
System and method for real-time storage of massive concurrent messages Download PDFInfo
- Publication number
- CN111949732A CN111949732A CN202010841877.9A CN202010841877A CN111949732A CN 111949732 A CN111949732 A CN 111949732A CN 202010841877 A CN202010841877 A CN 202010841877A CN 111949732 A CN111949732 A CN 111949732A
- Authority
- CN
- China
- Prior art keywords
- queue
- gradient
- level
- messages
- partition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 16
- 238000005192 partition Methods 0.000 claims abstract description 45
- 230000003993 interaction Effects 0.000 claims abstract description 5
- 230000008878 coupling Effects 0.000 abstract description 4
- 238000010168 coupling process Methods 0.000 abstract description 4
- 238000005859 coupling reaction Methods 0.000 abstract description 4
- 238000002955 isolation Methods 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000003313 weakening effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0668—Management of faults, events, alarms or notifications using network fault recovery by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The invention discloses a real-time storage system and a real-time storage method for massive concurrent messages, wherein the system comprises a plurality of level 1 gradient queues, and the level 1 gradient queues are used for storing camera messages of different users; each level 1 gradient queue is connected with a plurality of level 2 gradient queues, and the level 2 gradient queues are used for independently storing messages of different cameras of the same user; each grade of gradient queue comprises a micro-cluster, the micro-cluster comprises a main queue partition, the main queue partition is used for external interaction and actual storage, the main queue partition is connected with a backup queue partition, and the backup queue partition is used for starting writing of the main queue partition and reading of a backup queue. The invention can eliminate the coupling on the data, is convenient for horizontal expansion and greatly improves the stable reliability of the system.
Description
Technical Field
The invention relates to the technical field of video monitoring, in particular to a system and a method for storing massive concurrent messages in real time.
Background
In a video monitoring scene, persistent storage is required for push stream messages of a camera or viewing messages of a user, and generally, the existing technology has 3 conventional schemes, which respectively include local storage of the camera, sub-database sub-tables of the cloud and a mode of adding a database to the cloud cache to store messages in a high-concurrency scene, but the storage mode has the following defects:
1. the camera local storage may have a risk of losing messages due to equipment failure;
2. the traditional database and table division has complex operation and maintenance structure, is not beneficial to horizontal expansion and maintenance, and needs to be strongly coupled with business logic;
3. according to the caching scheme, a plurality of users at the cloud share the caching service, a single node fault can affect all other users, even a certain path of camera stream fault of a single user can affect all camera streams of the user.
Disclosure of Invention
It is an object of the present invention to provide a system and method for real-time storage of mass concurrent messages that overcomes some or all of the disadvantages of the prior art.
The invention relates to a real-time storage system for massive concurrent messages, which comprises a plurality of level 1 gradient queues, wherein the level 1 gradient queues are used for storing camera messages of different users;
each level 1 gradient queue is connected with a plurality of level 2 gradient queues, and the level 2 gradient queues are used for independently storing messages of different cameras of the same user;
each grade of gradient queue comprises a micro-cluster, the micro-cluster comprises a main queue partition, the main queue partition is used for external interaction and actual storage, the main queue partition is connected with a backup queue partition, and the backup queue partition is used for starting writing of the main queue partition and reading of a backup queue.
Furthermore, each level of gradient queues is connected with a queue compensation module, and the queue compensation module is used for balancing messages among different queues.
Further, the level 2 gradient queue may be connected with the level 3 gradient queue, the level 4 gradient queue … … N gradient queue.
Further, different users can share the level 1 gradient queue, and the level 1 gradient queue can adjust the binding relationship between the users and the level 1 gradient queue at any time.
Further, the number of backup queue partitions is at least 2.
The invention also provides a method for storing massive concurrent messages in real time, which comprises the following steps:
firstly, when massive messages are transmitted to a storage system, a level 1 gradient queue is reached first, and the level 1 gradient queue stores camera messages of different users;
the level 1 gradient queue transmits the camera information of the user to a corresponding level 2 gradient queue, and the level 2 gradient queue stores different camera messages of the same user;
and the third grade 2 gradient queue transmits the camera message stored by the third grade 2 gradient queue to the next grade 3 gradient queue and grade 4 gradient queue … … N gradient queue for isolated storage.
Further, when the level 1 gradient queue or the level 2 gradient queue is stored, the camera message is stored in the main queue partition, and the backup queue partition connected with the main queue partition can start writing in the main queue partition and reading out the backup queue.
Further, when each stage of gradient queues stores, the queue compensation module balances messages among different queues in real time.
Further, when the level 1 gradient queue is stored, users with relatively large traffic volume can be allocated with independent level 1 gradient queues, and users with small traffic volume can share the level 1 gradient queues.
Further, the level 1 gradient queue can adjust the binding relationship between the user and the level 1 gradient queue at any time.
The multi-gradient queue realizes the decoupling of the logic and physical layers between users, and can not cause the bad user experience and unavailable service due to the fault of some users or the fault of some nodes; the queue compensation and the queue micro-cluster are used for solving the problem that some nodes are off-line, messages can be automatically transferred to standby nodes, each gradient queue is a micro-cluster with more than 3 nodes, the expansion is easy, the robustness of the whole system is extremely strong, and the system can easily cope with massive concurrent messages.
The multi-gradient queue + queue compensation + queue micro-cluster provided by the invention can eliminate the coupling on data, is convenient for horizontal expansion, and greatly improves the stability and reliability of the system.
Drawings
Fig. 1 is a block diagram of a massive concurrent message real-time storage system according to an embodiment of the present invention.
Detailed Description
For a further understanding of the invention, reference should be made to the following detailed description taken in conjunction with the accompanying drawings and examples. It is to be understood that the examples are illustrative of the invention and not limiting.
As shown in fig. 1, the present embodiment provides a mass concurrent message real-time storage system, which includes a plurality of level 1 gradient queues, where the level 1 gradient queues are used to store camera messages of different users; the level 1 gradient queue can release the data coupling among different users, and can allocate independent level 1 gradient queues to some users with larger service volume according to actual requirements, while users with small service volume can share the level 1 gradient queues, and can also adjust the binding relationship between the users and the level 1 gradient queues at any time, thereby realizing the isolation and the release of the coupling of data and weakening the influence of the fault of some users on other users to the maximum extent;
in this implementation, a certain "user" camera may also be a camera under a certain "account", and in practical application, one camera may be bound under one user (or account), or there may be several cameras.
Each level-1 gradient queue is connected with a plurality of level-2 gradient queues, and the level-2 gradient queues are used for independently storing messages of different cameras of the same user, so that the camera level message decoupling and isolation are realized in the same user again, and the level-2 gradient queues realize the isolation from the user dimension to the equipment dimension at different levels;
each grade of gradient queue comprises a micro-cluster, the micro-cluster comprises a main queue partition, the main queue partition is used for external interaction and actual storage, the main queue partition is connected with a backup queue partition, and the backup queue partition is used for starting writing of the main queue partition and reading of a backup queue.
The gradient queue is a logic structure, and the partition is a physical structure, wherein the main queue partition is used as a main node of the micro cluster species and is used for carrying out external interaction and actual storage; the 2 backup queue partitions are an alternative scheme for realizing the downtime of the main queue partition, and when the message concurrency is large, the writing of the main queue partition and the reading function of the backup queue can be started to realize the read-write separation; each queue partition is an autonomous micro-cluster, and can realize read-write separation and horizontal division, thereby bringing concurrent processing capacity and high availability of massive messages for the whole system.
In this embodiment, each level of gradient queues is connected with a queue compensation module, the queue compensation module is used for balancing messages among different queues, and when some gradient queues have a fault, other queues can acquire messages from the fault queue for subsequent processing, so that the balancing and fault self-recovery capabilities of the whole system are ensured.
In this embodiment, as a more preferable implementation manner, the 2-level gradient queue may further be connected with the 3-level gradient queue and the 4-level gradient queue … … N-level gradient queue, so that as the gradient is continuously decreased, higher-dimensional isolation may be achieved.
In this embodiment, different users may share the level 1 gradient queue, and the level 1 gradient queue may adjust the binding relationship between the users and the level 1 gradient queue at any time.
In this embodiment, the number of backup queue partitions is at least 2.
The multi-gradient queue in the embodiment realizes the decoupling of the logic and physical layers between users, and can not cause the poor user experience and the unavailable service due to the faults of some users or the faults of some nodes; the queue compensation and the queue micro-cluster in the embodiment are used for solving the problem that some nodes are offline, messages can be automatically transferred to standby nodes, each gradient queue is a micro-cluster with more than 3 nodes, the expansion is easy, the robustness of the whole system is extremely strong, and the system can easily cope with massive concurrent messages.
The embodiment also provides a method for storing massive concurrent messages in real time, which comprises the following steps:
firstly, when massive messages are transmitted to a storage system, a level 1 gradient queue is reached first, and the level 1 gradient queue stores camera messages of different users;
the level 1 gradient queue transmits the camera information of the user to a corresponding level 2 gradient queue, and the level 2 gradient queue stores different camera messages of the same user;
and the third grade 2 gradient queue transmits the camera message stored by the third grade 2 gradient queue to the next grade 3 gradient queue and grade 4 gradient queue … … N gradient queue for isolated storage.
When the level 1 gradient queue or the level 2 gradient queue is stored, the camera information is stored in the main queue partition, the backup queue partition connected with the main queue partition can open the writing of the main queue partition and the reading of the backup queue.
When each grade of gradient queue is stored, the queue compensation module balances the messages among different queues in real time.
When the level 1 gradient queue is stored, users with larger traffic volume can distribute independent level 1 gradient queues, and users with smaller traffic volume can share the level 1 gradient queues.
Wherein, the level 1 gradient queue can adjust the binding relationship between the user and the level 1 gradient queue at any time.
The present invention and its embodiments have been described above schematically, without limitation, and what is shown in the drawings is only one of the embodiments of the present invention, and the actual structure is not limited thereto. Therefore, if the person skilled in the art receives the teaching, without departing from the spirit of the invention, the person skilled in the art shall not inventively design the similar structural modes and embodiments to the technical solution, but shall fall within the scope of the invention.
Claims (10)
1. A mass concurrent message real-time storage system is characterized in that: the system comprises a plurality of level 1 gradient queues, wherein the level 1 gradient queues are used for storing camera messages of different users;
each level 1 gradient queue is connected with a plurality of level 2 gradient queues, and the level 2 gradient queues are used for independently storing messages of different cameras of the same user;
each grade of gradient queue comprises a micro-cluster, the micro-cluster comprises a main queue partition, the main queue partition is used for external interaction and actual storage, the main queue partition is connected with a backup queue partition, and the backup queue partition is used for starting writing of the main queue partition and reading of a backup queue.
2. The mass concurrent message real-time storage system according to claim 1, wherein: and the gradient queue of each level is connected with a queue compensation module, and the queue compensation module is used for balancing messages among different queues.
3. The mass concurrent message real-time storage system according to claim 1, wherein: the level 2 gradient queue can be continuously connected with a level 3 gradient queue and a level 4 gradient queue … … N gradient queue.
4. The mass concurrent message real-time storage system according to claim 1, wherein: different users can share the grade 1 gradient queue, and the grade 1 gradient queue can adjust the binding relationship between the users and the grade 1 gradient queue at any time.
5. The mass concurrent message real-time storage system according to claim 1, wherein: the number of backup queue partitions is at least 2.
6. A real-time storage method for massive concurrent messages is characterized in that: the method comprises the following steps:
firstly, when massive messages are transmitted to a storage system, a level 1 gradient queue is reached first, and the level 1 gradient queue stores camera messages of different users;
the level 1 gradient queue transmits the camera information of the user to a corresponding level 2 gradient queue, and the level 2 gradient queue stores different camera messages of the same user;
and the third grade 2 gradient queue transmits the camera message stored by the third grade 2 gradient queue to the next grade 3 gradient queue and grade 4 gradient queue … … N gradient queue for isolated storage.
7. The method for storing massive concurrent messages in real time according to claim 6, wherein the method comprises the following steps: when the level 1 gradient queue or the level 2 gradient queue is stored, the camera information is stored in the main queue partition, and the backup queue partition connected with the main queue partition can open the writing of the main queue partition and the reading of the backup queue.
8. The method for storing massive concurrent messages in real time according to claim 6, wherein the method comprises the following steps: and when each stage of gradient queue is stored, the queue compensation module balances the messages among different queues in real time.
9. The method for storing massive concurrent messages in real time according to claim 6, wherein the method comprises the following steps: when the level 1 gradient queue is stored, users with larger traffic volume can be allocated with independent level 1 gradient queues, and users with smaller traffic volume can share the level 1 gradient queues.
10. The method for storing massive concurrent messages in real time according to claim 9, wherein: the level 1 gradient queue can adjust the binding relationship between the user and the level 1 gradient queue at any time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010841877.9A CN111949732A (en) | 2020-08-20 | 2020-08-20 | System and method for real-time storage of massive concurrent messages |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010841877.9A CN111949732A (en) | 2020-08-20 | 2020-08-20 | System and method for real-time storage of massive concurrent messages |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111949732A true CN111949732A (en) | 2020-11-17 |
Family
ID=73358800
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010841877.9A Pending CN111949732A (en) | 2020-08-20 | 2020-08-20 | System and method for real-time storage of massive concurrent messages |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111949732A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001045458A (en) * | 1999-07-29 | 2001-02-16 | Matsushita Electric Ind Co Ltd | Video server and method for controlling data transfer for the same |
| US20050141416A1 (en) * | 2003-12-24 | 2005-06-30 | Daisuke Kobayashi | Frame transmission device |
| CN106533916A (en) * | 2016-11-22 | 2017-03-22 | 天脉聚源(北京)传媒科技有限公司 | Live broadcast message display method and device |
| CN106603992A (en) * | 2016-12-30 | 2017-04-26 | 深圳市茁壮网络股份有限公司 | Monitoring system |
| CN106658050A (en) * | 2016-12-30 | 2017-05-10 | 北京奇虎科技有限公司 | Concurrent read-write method and device for stream data |
| CN108495082A (en) * | 2018-03-01 | 2018-09-04 | 国云科技股份有限公司 | A kind of video monitoring system based on cloud platform |
-
2020
- 2020-08-20 CN CN202010841877.9A patent/CN111949732A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001045458A (en) * | 1999-07-29 | 2001-02-16 | Matsushita Electric Ind Co Ltd | Video server and method for controlling data transfer for the same |
| US20050141416A1 (en) * | 2003-12-24 | 2005-06-30 | Daisuke Kobayashi | Frame transmission device |
| CN106533916A (en) * | 2016-11-22 | 2017-03-22 | 天脉聚源(北京)传媒科技有限公司 | Live broadcast message display method and device |
| CN106603992A (en) * | 2016-12-30 | 2017-04-26 | 深圳市茁壮网络股份有限公司 | Monitoring system |
| CN106658050A (en) * | 2016-12-30 | 2017-05-10 | 北京奇虎科技有限公司 | Concurrent read-write method and device for stream data |
| CN108495082A (en) * | 2018-03-01 | 2018-09-04 | 国云科技股份有限公司 | A kind of video monitoring system based on cloud platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2548738C (en) | Systems and methods for synchronizing data between communication devices in a networked environment | |
| US7349986B2 (en) | Method for data distribution | |
| CN102088490B (en) | Data storage method, device and system | |
| CN108306955A (en) | A kind of car-mounted terminal interconnects cluster method on a large scale | |
| CN100525378C (en) | Management method, system and device to update distributed set top box | |
| CN113157449A (en) | Real-time stream data analysis processing method based on MQTT | |
| CN106874142A (en) | A kind of real time data fault-tolerance processing method and system | |
| CN111741135B (en) | Hybrid cloud disaster recovery system and control method thereof | |
| CN112804332A (en) | Message processing system, method, device, equipment and computer readable storage medium | |
| CN111858190B (en) | Method and system for improving availability of cluster | |
| CN108390919B (en) | Message synchronization system and method for high-reliability dual-computer hot standby | |
| CN106354870B (en) | A kind of method and apparatus of data load | |
| CN101552799A (en) | Media node fault-tolerance method and device | |
| CN111949732A (en) | System and method for real-time storage of massive concurrent messages | |
| CN113489601A (en) | Anti-destruction method and device based on video networking autonomous cloud network architecture | |
| US20230403414A1 (en) | Distributed data storage for in-vehicle entertainment system | |
| CN115514651B (en) | Cloud edge data transmission path planning method and system based on software-defined stacked network | |
| CN109218386B (en) | High-availability method for managing Hadoop namespace | |
| CN114884878A (en) | MAC address synchronization method for multi-switch chip stacking in hardware learning mode | |
| CN109257403B (en) | Data storage method and device and distributed storage system | |
| CN107404511B (en) | Method and device for replacing servers in cluster | |
| CN113179228B (en) | Method, device, equipment and medium for improving switch stacking reliability | |
| WO2022205650A1 (en) | Distributed audio and video task management and distribution system, and implementation method | |
| CN115604271A (en) | Micro-service-based software and hardware complementary load balancing method | |
| CN112751789A (en) | Method and system for realizing asymmetric SDN controller cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |