CN107153527B - Parallel radar data processing method based on message queue - Google Patents
Parallel radar data processing method based on message queue Download PDFInfo
- Publication number
- CN107153527B CN107153527B CN201710350927.1A CN201710350927A CN107153527B CN 107153527 B CN107153527 B CN 107153527B CN 201710350927 A CN201710350927 A CN 201710350927A CN 107153527 B CN107153527 B CN 107153527B
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- queue
- data processing
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/38—Concurrent instruction execution, e.g. pipeline, look ahead
- G06F9/3885—Concurrent instruction execution, e.g. pipeline, look ahead using a plurality of independent parallel functional units
Abstract
The invention discloses a parallel radar data processing method based on a message queue, namely, a sending end divides radar original echo data into independent data blocks according to a data frame structure according to service requirements and sends the independent data blocks to a message input queue, a processing module adopts a competition mode to preempt messages to perform data processing on the data blocks and sends results to an output queue, and a receiving end performs queue processing on the results and combines the processing results. The invention extracts each module in the radar data processing process into an independent program which operates independently, and each program only processes single data, thereby reducing the coupling degree between the modules; for a time-consuming processing flow, a plurality of processing modules can be deployed under the condition of meeting hardware requirements to accelerate the data processing process, so that the efficiency and the real-time performance of radar data processing are improved.
Description
Technical Field
The invention relates to the field of computer data processing, in particular to a parallel radar data processing method based on a message queue.
Background
The method comprises the steps of carrying out wild value elimination on original echo data, carrying out pulse pressure processing on narrow-band echo data, carrying out noise reduction, alignment, imaging and multi-target segmentation processing on broadband original echo data, storing the analyzed data, and carrying out follow-up data playback, feature extraction, target identification and the like. How to rapidly process the measured data of the radar has important significance for external field experiments and laboratory researches.
With the development of computer technology, high-performance computing is developing towards multi-CPU multi-core computing, GPU computing, cluster computing, and the like, and in order to fully exert the processing capability of a computer and improve the data processing efficiency, it is a common practice to parallelize programs. The common method for parallelization is mainly a multithreading/process mode, which needs to modify the original program processing logic, easily causes problems such as deadlock and the like, and has high programming difficulty. An Actor mode is a mathematical model that addresses distributed computing, where an Actor is a computing entity that receives and processes received messages, and is able to collaborate in parallel with other actors. Through the message queue, an application program developer can conveniently carry out quick and reliable communication with other programs through sending and receiving messages, reduce the coupling among modules and improve the reliability and stability of the programs.
Disclosure of Invention
Aiming at the existing state of the prior art, the invention provides a parallel radar data processing method based on a message queue, which makes full use of a multi-core CPU and a GPU to accelerate radar data processing, improves data processing efficiency and reduces the development difficulty of parallel programs.
The invention provides a parallel radar data processing method based on a message queue, which is improved in that a sending end divides radar original echo data into independent data blocks according to a data frame structure according to service requirements and sends the independent data blocks to a message input queue, a processing module adopts a competition mode to seize messages to perform data processing on the data blocks and sends results to an output queue, and a receiving end performs queue processing on the results and combines the processing results.
Preferably, the processing modules of the sending end and the receiving end keep the original single thread mode for data processing, and complex multi-thread development is not needed; when there are multiple senders and receivers, the system performs data processing in parallel in a producer-consumer manner.
Preferably, different message types are set, each message type corresponds to one message queue, and the processing modules of the same type only process messages of a certain specific type in the message queues; a plurality of processing modules of the same type process data in a parallelization way;
and the processing module increases and decreases according to the number of the message types.
Preferably, a retry queue is set, and data with execution exception is stored and executed again by other processing modules according to a retry strategy.
Preferably, the specific type of message includes raw data of the radar, wideband echo data, narrowband echo data, a wideband echo processing result, and a narrowband echo processing result.
Preferably, the I/O throughput rate is improved by optimizing the message queue parameters, the I/O time of the message queue is ensured to be far shorter than the processing time of the processing module, and the system can improve the processing efficiency.
In the technical scheme of the invention, each module in the radar data processing process is extracted into an independent program to run independently, and each program only processes single data, so that the coupling degree between the modules is reduced; for a time-consuming processing flow, a plurality of processing modules can be deployed under the condition of meeting hardware requirements to accelerate the data processing process, so that the efficiency and the real-time performance of radar data processing are improved.
Drawings
FIG. 1 is a flow chart illustrating a parallel program processing based on message queues according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a parallel radar data processing flow based on a message queue according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a program deployment according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a single-threaded processing time distribution according to an embodiment of the present invention;
fig. 5 is a schematic diagram of parallel processing time distribution based on message queues according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be described in further detail below with reference to the accompanying drawings by way of examples of preferred embodiments. It should be noted, however, that the numerous details set forth in the description are merely for the purpose of providing the reader with a thorough understanding of one or more aspects of the present invention, which may be practiced without these specific details.
In this embodiment, a Push-Pop mode based on a message queue is adopted, a system function is split into a plurality of independent modules, data is transmitted between the modules in a message queue mode, and the coupling degree between the modules is reduced. A schematic flow diagram of parallel radar data processing is shown in fig. 1, and the parallel radar data processing method based on the message queue includes a sending end, a message input queue, a processing module, a result output queue, and a receiving end, where the schematic flow diagram of the parallel radar data processing method based on the message queue is shown in fig. 2, the sending end divides radar original echo data into independent data blocks according to a data frame structure according to service requirements, and sends the independent data blocks to the message input queue, the processing module uses a competition mode to preempt messages to perform data processing on the data blocks, and sends results to the output queue, and the receiving end performs dequeue processing on the results, and combines the processing results.
The processing modules of the sending end and the receiving end respectively adopt a single-thread mode to process data, and problems caused by algorithm transplantation and multithreading concurrency can be reduced. When there are multiple senders and receivers, the system performs data processing in parallel in a producer-consumer manner.
In this embodiment, different message types may be set, each message type corresponds to one message queue, and the processing module of the same type only processes a certain type of message in the message queue by adopting the processes of preprocessing, analyzing and processing. Multiple processing module examples of the same type can be executed simultaneously, data can be processed in parallel, the number of the processing module examples can be increased dynamically, and the system throughput rate is improved. In this embodiment, redis is used as a message queue, the message queue does not persistently store data (store a message in a memory, and does not persistently store the message on a device such as a hard disk), and a processing module is responsible for inserting, deleting, executing, exception handling, retrying, and the like of the message; a retry queue can be set, the data with execution exception is stored, and other processing modules are selected to be executed again according to the retry strategy.
In this embodiment, because there are overheads such as I/O transmission delay, thread switching, etc., by optimizing the message queue parameters, the I/O throughput is improved, it is ensured that the I/O time of the message queue is much shorter than the processing time of the processing module, and the processing efficiency is improved.
Specifically, in this embodiment, redis is used as a message queue, a program is written in a C + + language, and is deployed on one workstation of an 8-core CPU as shown in fig. 3, if multiple machines are used, the processing modules may be deployed on different machines respectively as needed, and the machines are connected through a gigabit network.
On a workstation of an 8-core CPU, each processing module exclusively occupies one CPU core, wherein two instances are respectively deployed in a wide/narrow band feature extraction and identification program. For the conventional single-thread approach, the processing time distribution is shown in fig. 4. The program processing time distribution of the embodiment is shown in fig. 5, the total time consumption of each processing module is basically the same as that of the corresponding single-thread mode, and the processing time consumption is far lower than that of the single-thread mode due to the fact that the performance of the multi-core CPU is fully utilized. For 4GB of test data, single-threaded processing takes about 15min (depending on the algorithm selected and implementation); the processing time of the method is about 6min (wherein a certain time is occupied by disk, network I/O, switching among processes and the like), and the radar data processing speed and efficiency are greatly accelerated.
The invention supports heterogeneous computation and multi-machine parallel computation, the processing modules can run on a CPU core (exclusive) or a GPU, and the processing modules communicate with each other only through a message queue, thereby reducing coupling programs among the modules and being capable of being flexibly deployed on a plurality of machines.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be construed as the protection scope of the present invention.
Claims (6)
1. A parallel radar data processing method based on a message queue is characterized in that a sending end divides radar original echo data into independent data blocks according to a data frame structure according to business requirements and sends the independent data blocks to a message input queue, a processing module adopts a competition mode to seize messages to perform data processing on the data blocks and sends results to an output queue, and a receiving end performs queue processing on the results and combines the processing results;
the processing module of the same type only processes a certain type of message in the message queue; and a plurality of processing modules of the same type process data in a parallelization mode.
2. The parallel radar data processing method according to claim 1, wherein the processing modules of the sending end and the receiving end maintain a single thread mode for data processing; when there are multiple senders and receivers, the system performs data processing in parallel in a producer-consumer manner.
3. The parallel radar data processing method of claim 1,
and the processing module increases and decreases according to the number of the message types.
4. The parallel radar data processing method of claim 3, wherein a retry queue is provided, data of execution exception is saved, and the data is executed again by other processing modules according to a retry strategy.
5. The parallel radar data processing method of claim 1, wherein the specific type of message includes raw data of a radar, wideband echo data, narrowband echo data, a wideband echo processing result, and a narrowband echo processing result.
6. The parallel radar data processing method of claim 1, wherein I/O throughput and system execution efficiency are improved by optimizing message queue parameters.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710350927.1A CN107153527B (en) | 2017-05-17 | 2017-05-17 | Parallel radar data processing method based on message queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710350927.1A CN107153527B (en) | 2017-05-17 | 2017-05-17 | Parallel radar data processing method based on message queue |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107153527A CN107153527A (en) | 2017-09-12 |
| CN107153527B true CN107153527B (en) | 2020-10-13 |
Family
ID=59794329
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710350927.1A Active CN107153527B (en) | 2017-05-17 | 2017-05-17 | Parallel radar data processing method based on message queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107153527B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108572877A (en) * | 2018-05-10 | 2018-09-25 | 厦门南讯软件科技有限公司 | A kind of concurrent message processing method of feature based grouping |
| CN109655792B (en) * | 2018-12-12 | 2022-11-08 | 北京无线电测量研究所 | Radar data parallel processing method and device and storage medium |
| CN110413428B (en) * | 2019-08-06 | 2022-01-25 | 中国联合网络通信集团有限公司 | Account checking data processing method, electronic equipment and storage medium |
| CN112130981A (en) * | 2020-09-28 | 2020-12-25 | 天地伟业技术有限公司 | Method for improving service efficiency of internal hardware processing unit of SOC (System on chip) |
| CN113406572B (en) * | 2021-06-23 | 2022-08-26 | 四川九洲电器集团有限责任公司 | Radar parallel processing system and method, storage medium and terminal |
| CN115396387B (en) * | 2022-08-30 | 2024-04-12 | 上海航天电子通讯设备研究所 | Data recording method, device, equipment and storage medium based on VxWorks message queue |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102006625A (en) * | 2010-11-10 | 2011-04-06 | 清华大学 | System and method for reliable broadcast protocol of vehicle-mounted ad hoc network based on forward error correction |
| CN102175999A (en) * | 2011-01-21 | 2011-09-07 | 西安电子科技大学 | Spectrogram feature-based radar target high-resolution distance image identification method |
| CN103218313A (en) * | 2013-04-02 | 2013-07-24 | 杭州华三通信技术有限公司 | Method and electric device for interacting cache descriptors |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4788124B2 (en) * | 2004-09-16 | 2011-10-05 | 株式会社日立製作所 | Data processing system |
| CN101616365B (en) * | 2008-06-24 | 2012-12-05 | 中兴通讯股份有限公司 | System and method for short message retry based on parallel queues |
| CN102457442A (en) * | 2010-10-29 | 2012-05-16 | 金蝶软件（中国）有限公司 | Message sending and reading methods and devices as well as middleware system |
| WO2013097248A1 (en) * | 2011-12-31 | 2013-07-04 | 华为技术有限公司 | Distributed task processing method, device and system based on message queue |
| CN102902512B (en) * | 2012-08-31 | 2015-12-16 | 浪潮电子信息产业股份有限公司 | A kind of multi-threading parallel process method based on multi-thread programming and message queue |
| CN104142867B (en) * | 2013-05-09 | 2018-01-09 | 华为技术有限公司 | Data processing equipment and data processing method |
| CN104200540B (en) * | 2014-08-15 | 2016-08-24 | 中国人民解放军空军装备研究院雷达与电子对抗研究所 | Data recording equipment |
| CN105187533A (en) * | 2015-09-10 | 2015-12-23 | 浪潮软件股份有限公司 | Data transmission method and device |
| CN105183698B (en) * | 2015-09-23 | 2019-03-08 | 上海无线电设备研究所 | A kind of control processing system and method based on multi-core DSP |
-
2017
- 2017-05-17 CN CN201710350927.1A patent/CN107153527B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102006625A (en) * | 2010-11-10 | 2011-04-06 | 清华大学 | System and method for reliable broadcast protocol of vehicle-mounted ad hoc network based on forward error correction |
| CN102175999A (en) * | 2011-01-21 | 2011-09-07 | 西安电子科技大学 | Spectrogram feature-based radar target high-resolution distance image identification method |
| CN103218313A (en) * | 2013-04-02 | 2013-07-24 | 杭州华三通信技术有限公司 | Method and electric device for interacting cache descriptors |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107153527A (en) | 2017-09-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107153527B (en) | Parallel radar data processing method based on message queue | |
| US11003489B2 (en) | Cause exception message broadcast between processing cores of a GPU in response to indication of exception event | |
| US7526634B1 (en) | Counter-based delay of dependent thread group execution | |
| US9772879B2 (en) | System and method for isolating I/O execution via compiler and OS support | |
| US20070226696A1 (en) | System and method for the execution of multithreaded software applications | |
| US20110131425A1 (en) | Systems and methods for power management in a high performance computing (hpc) cluster | |
| EP2796991A2 (en) | Processor for batch thread processing, batch thread processing method using the same, and code generation apparatus for batch thread processing | |
| US20080288496A1 (en) | Method for implementing concurrent producer-consumer buffers | |
| WO2016041126A1 (en) | Method and device for processing data stream based on gpu | |
| CN115129480B (en) | Scalar processing unit and access control method thereof | |
| CN111443898A (en) | Method for designing flow program control software based on priority queue and finite-state machine | |
| US9170966B2 (en) | System method for deterministic message processing a direct memory access (DMA) adapter with head pointers and tail pointers for determining the in and out sequence of packets | |
| US20220300322A1 (en) | Cascading of Graph Streaming Processors | |
| CN111078289B (en) | Method for executing sub-threads of a multi-threaded system and multi-threaded system | |
| US20190310857A1 (en) | Method of Concurrent Instruction Execution and Parallel Work Balancing in Heterogeneous Computer Systems | |
| US9146886B2 (en) | Deterministic message processing in a direct memory access adapter | |
| WO2020001295A1 (en) | Client-server architecture for multicore computer system to realize single-core-equivalent view | |
| CN111797158A (en) | Data synchronization system, method and computer-readable storage medium | |
| US8375155B2 (en) | Managing concurrent serialized interrupt broadcast commands in a multi-node, symmetric multiprocessing computer | |
| CN109800064B (en) | Processor and thread processing method | |
| CN109460254B (en) | Instruction processing method, CPU (central processing unit) interactive system and consumable chip adopting system | |
| CN112579268A (en) | Efficient out-of-order unified dyeing array multi-task parallel processing method | |
| CN115964164A (en) | Computer-implemented method, hardware accelerator, and storage medium | |
| CN114880101A (en) | AI treater, electronic part and electronic equipment | |
| CN116685964A (en) | Processing method of operation acceleration, using method of operation accelerator and operation accelerator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |