CN111651631B - High concurrency video data processing method, electronic equipment, storage medium and system - Google Patents
High concurrency video data processing method, electronic equipment, storage medium and system Download PDFInfo
- Publication number
- CN111651631B CN111651631B CN202010350998.3A CN202010350998A CN111651631B CN 111651631 B CN111651631 B CN 111651631B CN 202010350998 A CN202010350998 A CN 202010350998A CN 111651631 B CN111651631 B CN 111651631B
- Authority
- CN
- China
- Prior art keywords
- video data
- server
- cache
- data
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 12
- 238000012545 processing Methods 0.000 claims abstract description 57
- 238000000034 method Methods 0.000 claims abstract description 19
- 230000007246 mechanism Effects 0.000 claims abstract description 18
- 230000005540 biological transmission Effects 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 7
- 238000005192 partition Methods 0.000 abstract description 2
- 238000012544 monitoring process Methods 0.000 description 9
- 238000007726 management method Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 238000013500 data storage Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 206010000117 Abnormal behaviour Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/71—Indexing; Data structures therefor; Storage structures
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a high concurrency video data processing method, which relates to the technical field of video data processing and comprises the following steps: receiving video data; based on the lock mechanism, video data is written into the cache server and simultaneously written into the database server through the queue server. The method adopts the atomic operation of a lock mechanism, ensures the consistency of the written data in the database server and the cache server, improves the stability of storing a large amount of video data, and solves the problem of data divergence caused by that the data in the cache server is landed to the database server in the partition time. The invention also discloses electronic equipment, a computer storage medium and a high concurrency video data processing system.
Description
Technical Field
The present invention relates to the field of video data processing technologies, and in particular, to a method, an electronic device, a storage medium, and a system for processing high-concurrency video data.
Background
The intelligent city system integrates multi-party resources such as emergency, public security traffic police, fire protection, city management and the like by means of various digital city technologies such as computer networks, mobile communication, computer communication integration, space information, grid management, city component management and the like, and realizes the exchange and sharing of various business data. The intelligent city system can effectively improve city management regulation and control and emergency handling capability of government, and provides good digitalized, intelligent and humanized working and living environments for people, wherein city safety is guaranteed in the whole intelligent city, an effective safety prevention means is a core ring of the whole city safety guarantee, and a video monitoring system is an indispensable part.
In video monitoring system project application, video data with rapid acquisition frequency and extremely large data volume is often received and stored, and high concurrent viewing requests for a large amount of video data are responded in real time. In the existing processing of video data, the cached data is cached and then landed in a database at intervals, however, the processing mode has insufficient consideration on the safety and consistency of the data, particularly when processing high-concurrency video data, once some data are lost, such as the loss of some data in the cache, the data are not landed in the database at the moment, so that the data are separated between the database and the cache, the consistency cannot be ensured, and the problem is caused when the data are recovered.
Disclosure of Invention
In order to overcome the defects of the prior art, one of the purposes of the invention is to provide a high concurrency video data processing method, which adopts a lock mechanism to control the atomic operation of writing video data into a cache server and a database server at the same time, and solves the problems of data safety and consistency.
One of the purposes of the invention is realized by adopting the following technical scheme:
receiving video data;
based on a lock mechanism, the video data is written into a cache server, and is written into a database server through a queue server.
Further, the queue server employs a multithreading mechanism, each thread for processing transactions for a set of producers and consumers, the producers receiving the video data, the consumers writing the video data to the database server.
Further, after receiving video data, detecting the data volume of the video data and the thread number of the queue server, and when the ratio of the data volume to the thread number is greater than or equal to a thread threshold value, adding threads.
Further, after receiving video data, detecting the data volume of the video data, and when the data volume is larger than a performance threshold, performing database separation, and newly adding a cache server and a database server.
Further, each piece of video data in the cache server is marked with an expiration time, and the cache server stores the video data with the current time smaller than the expiration time.
Further, each cache server maintains a cache index table, each database server maintains a database index table, the key value in each index table is the transmission timestamp of each video data, and the value in each index table is the position of each video data in the cache server or the database server.
Further, receiving a view request;
searching the cache index table according to the target transmission time, and judging whether a key value corresponding to the target transmission time is found in the cache index table;
if yes, returning target video data from the cache server;
if not, searching the database index table, returning the target video data from the database server, and writing the target video data into the cache server.
It is a second object of the present invention to provide an electronic device for performing one of the objects of the invention, comprising a processor, a storage medium and a computer program stored in the storage medium, which computer program, when being executed by the processor, is a method for processing highly concurrent video data according to one of the objects of the invention.
It is a further object of the invention to provide a computer readable storage medium storing one of the objects of the invention, on which a computer program is stored which, when being executed by a processor, achieves a high concurrence video data processing method of one of the objects of the invention.
The invention aims at providing a high concurrency video data processing system, which distributes video data received from a terminal through a load balancing server, and the processing and control server writes the video data into a cache server and a database server at the same time according to the data volume of the distributed video data, so as to solve the problems of data safety and consistency.
The fourth technical scheme adopted by the invention is as follows:
a high concurrency video data processing system, comprising: the system comprises a load balancing server, a plurality of processing and control servers, a cache server, a queue server and a database server;
the load balancing server is used for receiving video data; distributing said video data to a plurality of said processing and control servers;
the processing and control server is used for writing the distributed video data into the cache server based on a lock mechanism, and writing the distributed video data into the database server through the queue server; detecting the data quantity of the distributed video data and the thread quantity of the queue server; when the ratio of the data quantity to the thread quantity is greater than or equal to a thread threshold value, adding threads newly; and when the data volume is larger than the performance threshold, a cache server and a database server are newly added.
Compared with the prior art, the invention has the beneficial effects that:
the invention adopts the atomic operation of the lock mechanism, writes the video data into the cache server, and simultaneously writes the video data into the database server through the queue server, thereby ensuring the consistency of the written data in the database server and the cache server, improving the stability of storing a large amount of video data and solving the problem of data divergence caused by that the data in the cache server is landed to the database server in the partition time.
Drawings
FIG. 1 is a flow chart of a method for processing high concurrency video data according to a first embodiment of the present invention;
FIG. 2 is a flow chart of a method for processing high concurrency video data according to a fourth embodiment of the present invention;
fig. 3 is a block diagram of an electronic device according to a fifth embodiment of the present invention.
Detailed Description
The invention will now be described in more detail with reference to the accompanying drawings, to which it should be noted that the description is given below by way of illustration only and not by way of limitation. Various embodiments may be combined with one another to form further embodiments not shown in the following description.
Example 1
An embodiment provides a high concurrency video data processing method, which aims to control the atomic operation of writing video data into a cache server and a database server simultaneously by adopting a lock mechanism, ensure the consistency of writing data into the database server and the cache server, improve the storage stability of a large amount of video data, and solve the problem of data divergence caused by that the data in the cache server is landed to the database server in a blocking time.
Referring to fig. 1, a method for processing high concurrency video data includes the following steps:
s110, receiving video data.
In practical application of the video monitoring system project, a plurality of acquisition terminals acquire video data at high frequency at the same time, so that a large amount of video data with high concurrence of the plurality of acquisition terminals can be generated. The acquisition terminal comprises, but is not limited to, a single chip microcomputer and a microcomputer for acquiring the monitoring video data. And a large amount of video data is distributed to a plurality of processing and control servers through the load balancing server, so that the single processing and control server is prevented from processing the excessive video data, and delay and even downtime are caused. Each processing and control server receives the video data distributed by the load balancing server and processes the video data.
S120, based on a lock mechanism, video data is written into a cache server, and meanwhile, the video data is written into a database server through a queue server.
If the video data is directly landed to the database server, the connection of the database server is abnormal, so that the video data is written into the queue server, and a large amount of video data is landed to the database server by the queue server. The queue server plays a role in buffering a large amount of video data, avoids abnormal connection of the database server, and improves the storage stability of the large amount of video data.
In order to ensure that the data in the database server is consistent with the data in the cache server, code programs realized by aiming at different programming voices can be optimized, a lock mechanism can be adopted to write video data into the database server and the cache server to realize atomic operation, the video data can be ensured to be written at the same time, and if the video data fails to be written into the database server or the cache server, the other party cannot be written. When the data is lost, the database server can be used as a backup source to quickly recover the lost data, so that the storage stability of a large amount of video data is improved.
Preferably, when the video data fails to be written simultaneously, abnormal information is sent to the log server, so that the video data failing to be written can be traced and analyzed conveniently, and the stability is further improved.
Example two
The second embodiment is an improvement based on the first embodiment, and according to the data volume of the video data and the number of threads of the queue server, the threads are quantitatively increased, so that the high-efficiency floor backup of the number of video data is ensured, and the storage stability is improved.
The queue server employs a multithreading mechanism, each thread for processing a set of producer and consumer transactions, the producer receiving video data and the consumer landing the video data in the database server. When only one producer and one consumer exist in the thread, the consumer can read the data only after the producer puts the data, and the producer can produce the data after the consumer takes the data, so that the data synchronization is realized; when the producer receives the data, the consumer can not land the data, or when the consumer lands the data, the producer can not receive the data, otherwise, the producer can not access the data, and the producer can not access the data, thereby further ensuring the consistency of the data written into the queue server and the cache server and improving the storage stability.
After receiving the video data, detecting the data quantity q of the video data, the number n of producer and consumer groups (i.e. the number of threads) in the queue server, and when the ratio of the data quantity q to the number n of the producer and consumer groups is greater than or equal to the thread threshold Wt, namelyAt least->Group producer and consumer, i.e. add +.>And the threads are used for ensuring that a large amount of video data can fall to a database server in time, so that the stability of data storage is improved. The thread threshold Wt may be preset according to actual business requirements in the video surveillance system project. A large amount of video data falls to the database server through each thread in the queue server.
Example III
The third embodiment is an improvement based on the first and/or second embodiments, in which the cache server and the database server are added according to the data volume of the video data, and the capacity expansion is implemented for a large amount of video data, so that the storage stability is further improved.
The performance threshold can be preset according to actual service requirements in the video monitoring system project, after video data is received, when the data volume of the video data is larger than the performance threshold, the storage performance of the cache server and the database server for the data is reduced, at the moment, the cache server and the database server are newly added, the transverse expansion of the storage capacity is realized, then the video data is stored, the video data is stored in different caches and databases separately, the storage stability of a large number of video data can be further improved, and the quick response video searching is facilitated. In order to effectively control the cost and avoid downtime caused by overlarge data quantity stored in the cache server, preferably, each piece of video data in the cache server is marked with the expiration time, the cache server only stores the video data with the current time less than the expiration time, the latest time of caching of the monitoring video data is ensured, and when a video viewing request exists, the data can be taken from the cache server to respond quickly.
Example IV
The fourth embodiment is an improvement based on the third embodiment, and the cache index table and the database index table are searched successively according to the target transmission time, so that the high concurrent data query request can be responded in real time.
In order to further improve storage stability, each cache server maintains a cache index table, each database server maintains a database index table, a key value in each index table is a transmission timestamp of each video data, and a value in each index table is a position of each video data in the cache server or the database server. Each cache server and database server can also quickly respond to video viewing requests by maintaining respective index tables.
Referring to fig. 2, the method for processing high-concurrency video data in this embodiment includes the following steps:
s210, receiving a view request.
When a plurality of output terminals simultaneously make requests to view a large amount of video data, highly concurrent viewing requests are generated. The output terminal may be a client or web browser on a cell phone or tablet that presents the monitoring data. The high-concurrency check request is distributed to a plurality of processing and control servers through the load balancing server, so that the single processing and control server is prevented from processing a large number of check requests, and delay and even downtime are caused. Each processing and control server receives the view request distributed by the load balancing server and responds to the view request. S220, searching a cache index table according to the target transmission time, and judging whether a key value corresponding to the target transmission time is found in the cache index table.
After each video data is stored by different cache servers or database servers, in order to ensure that each video data has a unique corresponding key value, the key value of each video data adopts a coding scheme of adding a transmission time stamp and a database table sequence number and bigdata-hash, so that the target video data can be accurately searched according to the transmission time of the target video data.
And S230, if yes, returning the target video data from the cache server. If the target transmission time is in the latest period of time, the key value in the cache index table can be directly inquired to obtain the position of the target video data in the cache server, and then the target video data is returned to finish the response to the check request. Therefore, the response speed to the high concurrent viewing request can be increased, the concurrent access quantity of the terminal is greatly improved, and the requirement of subsequent business operation is met.
S240, if not, searching the database index table, returning the target video data from the database server, and writing the target video data into the cache server.
If the key value corresponding to the target transmission time is not found in the cache index table, the target transmission time is not in the latest period of time or the target video data in the cache server is accidentally lost, and the key value in the database index table is queried at the moment, so that the target video data in the database server is obtained, and the response to the viewing request is completed. And simultaneously, the target video data is rewritten into the cache server, the cache data can be directly recovered without stopping service, and the terminal can be conveniently responded for the second time.
Example five
Fig. 3 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention, where, as shown in fig. 3, the electronic device includes a processor 310, a memory 320, an input device 330 and an output device 340; the number of processors 310 in the computer device may be one or more, one processor 310 being taken as an example in fig. 3; the processor 310, the memory 320, the input device 330 and the output device 340 in the electronic device may be connected by a bus or other means, in fig. 3 by way of example.
The memory 320 is a computer readable storage medium, and may be used to store software programs, computer executable programs, and modules, such as program instructions/modules corresponding to the high concurrency video data processing method in the embodiment of the present invention. The processor 310 executes various functional applications of the electronic device and data processing by executing software programs, instructions and modules stored in the memory 320, i.e., implements the high concurrency video data processing methods of the first to fourth embodiments.
Memory 320 may include primarily a program storage area and a data storage area, wherein the program storage area may store an operating system, at least one application program required for functionality; the storage data area may store data created according to the use of the terminal, etc. In addition, memory 320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some examples, memory 320 may further include memory located remotely from processor 310, which may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 330 may be used to receive video data, view requests, and the like. The output device 340 may include a display device such as a display screen.
Example six
A sixth embodiment of the present invention also provides a storage medium containing computer executable instructions which, when executed by a computer processor, are for performing a method of high-concurrency video data processing, the method comprising:
receiving video data;
based on a lock mechanism, the video data is written into a cache server, and is written into a database server through a queue server.
Preferably, after receiving video data, detecting the data volume of the video data and the thread number of the queue server, and when the ratio of the data volume to the thread number is greater than or equal to a thread threshold value, adding threads.
Preferably, after receiving video data, detecting the data volume of the video data, and when the data volume is larger than a performance threshold, performing database separation, and newly adding a cache server and a database server.
Preferably, each cache server maintains a cache index table, each database server maintains a database index table, the key value in each index table is the transmission timestamp of each video data, and the value in each index table is the position of each video data in the cache server or the database server.
Preferably, receiving a view request;
searching the cache index table according to the target transmission time, and judging whether a key value corresponding to the target transmission time is found in the cache index table;
if yes, returning target video data from the cache server;
if not, searching the database index table, returning the target video data from the database server, and writing the target video data into the cache server.
Of course, the storage medium containing the computer executable instructions provided in the embodiments of the present invention is not limited to the method operations described above, but may also perform the related operations in the high concurrency video data processing method provided in any embodiment of the present invention.
From the above description of embodiments, it will be clear to a person skilled in the art that the present invention may be implemented by means of software and necessary general purpose hardware, but of course also by means of hardware, although in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, etc., and include several instructions for causing an electronic device (which may be a mobile phone, a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments of the present invention.
Example seven
The seventh embodiment is a high concurrency video data processing system corresponding to the high concurrency video data processing method of the above embodiment, and the processing and control server writes the video data into the cache server and the database server at the same time according to the data amount of the distributed video data, so as to solve the problems of data security and consistency, and improve the stability of storing a large amount of video data.
A high concurrency video data processing system, comprising: the system comprises a load balancing server, a plurality of processing and control servers, a cache server, a queue server and a database server.
The load balancing server is used for receiving video data; distributing video data to a plurality of processing and control servers;
the processing and control server is used for writing the distributed video data into the cache server based on the locking mechanism and writing the distributed video data into the database server through the queue server; detecting the data quantity of the distributed video data and the thread quantity of a queue server; when the ratio of the data quantity to the number of threads is greater than or equal to a thread threshold value, adding threads newly; when the data volume is greater than the performance threshold, a cache server and a database server are newly added.
The load balancing server can be connected with a plurality of acquisition terminals or output terminals, video data are obtained through the acquisition terminals in the video monitoring system, the acquisition terminals can be a single chip microcomputer, a microcomputer and the like, the output terminals can be mobile phones or clients on a tablet or web browsers for displaying the monitoring data, and when the output terminals give out data viewing requests, target video data are returned to the output terminals for displaying. The load balancing server is connected with a plurality of processing and control servers, and distributes high-concurrency video data or viewing requests to each processing and control server, so that the single processing and control server is prevented from processing overlarge data or requests, and delay and even downtime are caused. The processing and control server supports lateral expansion and can respond to requests of different magnitudes.
The processing and control server is connected with at least one cache server and is connected with at least one database server through a queue server. The cache server and the database server can be transversely expanded according to the data volume of the video data, so that the stability of storing a large amount of video data is improved.
Preferably, the high concurrency video data processing system further comprises a log server for recording abnormal behavior logs of the whole system, and the log server is connected with other servers in the system.
Preferably, each cache server maintains a cache index table, each database server maintains a database index table, the key value in each index table is the transmission timestamp of each video data, and the value in each index table is the position of each video data in the cache server or the database server. The load balancing server is used for receiving high-concurrency check requests and distributing the check requests to the plurality of processing and control servers;
the processing and control server is used for receiving the distributed viewing request;
searching a performance threshold cache index table according to the target transmission time, and judging whether a key value corresponding to the performance threshold target transmission time is found in the performance threshold cache index table;
if yes, returning target video data from the performance threshold cache server;
if not, searching the performance threshold database index table, returning the performance threshold target video data from the performance threshold database server, and writing the performance threshold target video data into the performance threshold cache server.
The processing and control server searches the cache index table and the database index table successively according to the target transmission time, and can respond to the high concurrent data query request in real time.
It should be noted that, in the above-mentioned embodiment of the method and system for processing high-concurrency video data, each server is only divided according to the functional logic, but not limited to the above-mentioned division, so long as the corresponding functions can be implemented; in addition, the specific names of the functional servers are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It will be apparent to those skilled in the art from this disclosure that various other changes and modifications can be made which are within the scope of the invention as defined in the appended claims.
Claims (8)
1. A method for processing high concurrency video data, characterized in that: the method comprises the following steps:
receiving video data;
detecting the data volume of the video data and the thread number of a queue server, and adding threads when the ratio of the data volume to the thread number is greater than or equal to a thread threshold value;
detecting the data volume of the video data, and dividing the database when the data volume is larger than a performance threshold value, and newly adding a cache server and a database server;
based on a lock mechanism, the video data is written into the cache server and is written into the database server through the queue server, the lock mechanism ensures that the video data can be written at the same time, and if the video data fails to be written into the database server or the cache server, the other party cannot be written.
2. A method of high concurrency video data processing as claimed in claim 1, wherein: the queue server employs a multithreading mechanism, each thread for processing transactions for a set of producers and consumers, the producers receiving the video data, the consumers writing the video data to the database server.
3. A method of high concurrency video data processing as claimed in claim 1, wherein: each piece of video data in the cache server marks the expiration time, and the cache server stores the video data with the current time smaller than the expiration time.
4. A method of high concurrency video data processing as claimed in any one of claims 1 to 3, wherein: each cache server maintains a cache index table, each database server maintains a database index table, the key value in each index table is the transmission time stamp of each video data, and the value in each index table is the position of each video data in the cache server or the database server.
5. A method of high concurrency video data processing as recited in claim 4, wherein:
receiving a viewing request;
searching the cache index table according to the target transmission time, and judging whether a key value corresponding to the target transmission time is found in the cache index table;
if yes, returning target video data from the cache server;
if not, searching the database index table, returning the target video data from the database server, and writing the target video data into the cache server.
6. An electronic device comprising a processor, a storage medium and a computer program stored in the storage medium, characterized in that the computer program, when executed by the processor, performs the high concurrency video data processing method of any one of claims 1 to 5.
7. A computer storage medium having a computer program stored thereon, characterized by: the computer program, when executed by a processor, implements the high concurrency video data processing method of any one of claims 1 to 5.
8. A high concurrency video data processing system, comprising: comprising the following steps: the system comprises a load balancing server, a plurality of processing and control servers, a cache server, a queue server and a database server;
the load balancing server is used for receiving video data; distributing said video data to a plurality of said processing and control servers;
the processing and control server is used for writing the distributed video data into the cache server based on a lock mechanism, and writing the distributed video data into the database server through the queue server at the same time, wherein the lock mechanism ensures that the video data can be written at the same time, and if the video data fails to be written into the database server or the cache server, the other party cannot write; detecting the data quantity of the distributed video data and the thread quantity of the queue server; when the ratio of the data quantity to the thread quantity is greater than or equal to a thread threshold value, adding threads newly; and when the data volume is larger than the performance threshold, a cache server and a database server are newly added.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010350998.3A CN111651631B (en) | 2020-04-28 | 2020-04-28 | High concurrency video data processing method, electronic equipment, storage medium and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010350998.3A CN111651631B (en) | 2020-04-28 | 2020-04-28 | High concurrency video data processing method, electronic equipment, storage medium and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111651631A CN111651631A (en) | 2020-09-11 |
| CN111651631B true CN111651631B (en) | 2023-11-28 |
Family
ID=72348174
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010350998.3A Active CN111651631B (en) | 2020-04-28 | 2020-04-28 | High concurrency video data processing method, electronic equipment, storage medium and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111651631B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113434344A (en) * | 2021-07-22 | 2021-09-24 | 咪咕数字传媒有限公司 | File storage method and device, computing equipment and computer storage medium |
| CN114461151B (en) * | 2022-04-11 | 2022-08-26 | 武汉中科通达高新技术股份有限公司 | Video storage method, computer equipment and storage medium of national standard camera |
| CN116192816A (en) * | 2022-06-01 | 2023-05-30 | 中兴通讯股份有限公司 | Data transmission method, transmission server, reception server, and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102023973A (en) * | 2009-09-14 | 2011-04-20 | 中国移动通信集团公司 | Method, device and system for synchronizing database with application cache server |
| CN102117309A (en) * | 2010-01-06 | 2011-07-06 | 卓望数码技术（深圳）有限公司 | Data caching system and data query method |
| CN102331986A (en) * | 2010-07-12 | 2012-01-25 | 阿里巴巴集团控股有限公司 | Database cache management method and database server |
| CN103076994A (en) * | 2012-12-31 | 2013-05-01 | 北京中科大洋科技发展股份有限公司 | Method for realizing off-line file writing in SAN (Storage Area Network) shared-file system |
| CN105160572A (en) * | 2015-09-30 | 2015-12-16 | 努比亚技术有限公司 | Device and method for controlling order to generate, and seckilling system |
| CN106257456A (en) * | 2016-07-08 | 2016-12-28 | 北京京东尚科信息技术有限公司 | The method of data base's stability, Apparatus and system is improved under high concurrent request |
| CN108595487A (en) * | 2018-03-14 | 2018-09-28 | 摇了购（武汉）电子商务有限公司 | The method and system of data are accessed under a kind of big data high concurrent |
| CN108664520A (en) * | 2017-04-01 | 2018-10-16 | 北京京东尚科信息技术有限公司 | Safeguard method, apparatus, electronic equipment and the readable storage medium storing program for executing of data consistency |
| CN109033292A (en) * | 2018-07-13 | 2018-12-18 | 南京邮电大学 | A kind of database distributed caching system and method |
| CN109271252A (en) * | 2018-08-29 | 2019-01-25 | 华为技术有限公司 | A kind of method and device adjusting Thread Count |
-
2020
- 2020-04-28 CN CN202010350998.3A patent/CN111651631B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102023973A (en) * | 2009-09-14 | 2011-04-20 | 中国移动通信集团公司 | Method, device and system for synchronizing database with application cache server |
| CN102117309A (en) * | 2010-01-06 | 2011-07-06 | 卓望数码技术（深圳）有限公司 | Data caching system and data query method |
| CN102331986A (en) * | 2010-07-12 | 2012-01-25 | 阿里巴巴集团控股有限公司 | Database cache management method and database server |
| CN103076994A (en) * | 2012-12-31 | 2013-05-01 | 北京中科大洋科技发展股份有限公司 | Method for realizing off-line file writing in SAN (Storage Area Network) shared-file system |
| CN105160572A (en) * | 2015-09-30 | 2015-12-16 | 努比亚技术有限公司 | Device and method for controlling order to generate, and seckilling system |
| CN106257456A (en) * | 2016-07-08 | 2016-12-28 | 北京京东尚科信息技术有限公司 | The method of data base's stability, Apparatus and system is improved under high concurrent request |
| CN108664520A (en) * | 2017-04-01 | 2018-10-16 | 北京京东尚科信息技术有限公司 | Safeguard method, apparatus, electronic equipment and the readable storage medium storing program for executing of data consistency |
| CN108595487A (en) * | 2018-03-14 | 2018-09-28 | 摇了购（武汉）电子商务有限公司 | The method and system of data are accessed under a kind of big data high concurrent |
| CN109033292A (en) * | 2018-07-13 | 2018-12-18 | 南京邮电大学 | A kind of database distributed caching system and method |
| CN109271252A (en) * | 2018-08-29 | 2019-01-25 | 华为技术有限公司 | A kind of method and device adjusting Thread Count |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111651631A (en) | 2020-09-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111651631B (en) | High concurrency video data processing method, electronic equipment, storage medium and system | |
| JP6716727B2 (en) | Streaming data distributed processing method and apparatus | |
| US5920857A (en) | Efficient optimistic concurrency control and lazy queries for B-trees and other database structures | |
| US9563426B1 (en) | Partitioned key-value store with atomic memory operations | |
| CN105138571B (en) | Distributed file system and method for storing massive small files | |
| US20190005101A1 (en) | Method and apparatus for accessing time series data in memory | |
| CN110784498B (en) | Personalized data disaster tolerance method and device | |
| CN112015820A (en) | Method, system, electronic device and storage medium for implementing distributed graph database | |
| CN103399894A (en) | Distributed transaction processing method on basis of shared storage pool | |
| US11775480B2 (en) | Method and system for deleting obsolete files from a file system | |
| CN112235396B (en) | Content processing link adjustment method, content processing link adjustment device, computer equipment and storage medium | |
| CN111526188B (en) | System and method for ensuring zero data loss based on Spark Streaming in combination with Kafka | |
| CN110866011B (en) | Data table synchronization method and device, computer equipment and storage medium | |
| CN116226139B (en) | Distributed storage and processing method and system suitable for large-scale ocean data | |
| CN111159131A (en) | Performance optimization method, device, equipment and computer readable storage medium | |
| CN107566341A (en) | A kind of data persistence storage method and system based on federal distributed file storage system | |
| CN106815318B (en) | Clustering method and system for time sequence database | |
| Wang et al. | Waterwheel: Realtime indexing and temporal range query processing over massive data streams | |
| CN116339626A (en) | Data processing method, device, computer equipment and storage medium | |
| CN106339279B (en) | Service recovery method and device | |
| CN113010373B (en) | Data monitoring method and device, electronic equipment and storage medium | |
| CN114780648A (en) | Task scheduling method, device, computer equipment, storage medium and program product | |
| CN104407918B (en) | A kind of preservation of avionics system task data and restoration methods | |
| CN111382200A (en) | Information loading method and device, server and storage medium | |
| CN117093367B (en) | Service data processing method, device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB03 | Change of inventor or designer information | ||
| CB03 | Change of inventor or designer information |
Inventor after: Ye Junyan Inventor after: Tang Zhuo Inventor after: Lu Bin Inventor after: OuYang Jiming Inventor before: Tang Zhuo Inventor before: Ye Junyan Inventor before: Lu Bin Inventor before: OuYang Jiming |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |