CN110990163A - High-concurrency method for multi-application data processing process - Google Patents
High-concurrency method for multi-application data processing process Download PDFInfo
- Publication number
- CN110990163A CN110990163A CN201911039156.XA CN201911039156A CN110990163A CN 110990163 A CN110990163 A CN 110990163A CN 201911039156 A CN201911039156 A CN 201911039156A CN 110990163 A CN110990163 A CN 110990163A
- Authority
- CN
- China
- Prior art keywords
- data processing
- data
- application
- source data
- source
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 108
- 238000000034 method Methods 0.000 title claims abstract description 49
- 230000008569 process Effects 0.000 title claims abstract description 25
- 238000013507 mapping Methods 0.000 claims abstract description 4
- 238000013501 data transformation Methods 0.000 claims description 2
- 230000008521 reorganization Effects 0.000 claims description 2
- 230000007246 mechanism Effects 0.000 abstract description 3
- 238000003672 processing method Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 5
- 239000000284 extract Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Abstract
The invention relates to a high concurrency method for a multi-application data processing process, which comprises the following steps: applying for a source data processing queue; the application identification of the source data determines the identification number of the application identification when the application is initialized; the source data processing queue storage mode is a first-in first-out mode; detecting whether the source data processing queue has data, if so, processing the source data by the data processing interface, otherwise, waiting; when the data processing queue processes the source data, the application identifier is extracted and stored in a temporary variable, the source data is processed, and the processed source data and the application identifier stored in the temporary variable are recombined to generate target data; and extracting the application identifier of the target data, and extracting the target data according to the mapping relation between the application and the application identifier and sending the target data to the application. The data processing method and the data processing system lead the data processing flow to realize a high concurrency mechanism of data processing during multi-application through introducing the data processing queue between the service module and the data processing module, thereby improving the data processing efficiency of the multi-application.
Description
Technical Field
The invention relates to a network data communication technology, in particular to a high concurrency method for a multi-application data processing process.
Background
In our daily life, the network is a very inseparable part, and how to improve the network data processing efficiency is a key point that we need to consider in the process of realizing the network architecture, wherein the application data processing is a part for improving the network transmission efficiency.
Fig. 1 is a schematic diagram of a conventional application data processing, and as shown in fig. 1, a current general network data processing flow is as follows: sending source data to a processing interface for processing by the application; after the data processing module finishes processing, the target data is returned to the application; performing subsequent processing on the obtained target data;
the existing application data processing process is a serial flow, an application sends source data to a data processing module, and waits for the data processing module to return processed target data, the operation process is that the application monopolizes the data processing module, other applications needing data processing are in a waiting state so as to block receiving new source data, and the data processing module needs to wait until the previous application data processing is finished when receiving the new source data, so that the overall efficiency of the data processing process is reduced.
Disclosure of Invention
The invention aims to provide a high-concurrency method for a multi-application data processing process so as to solve the problem of overall efficiency of the data processing process.
The invention discloses a high concurrency method for a multi-application data processing process, which comprises the following steps: applying for a source data processing queue; the application identification of the source data determines the identification number of the application identification when the application is initialized; the source data processing queue storage mode is a first-in first-out mode; detecting whether the source data processing queue has data, if so, processing the source data by the data processing interface, otherwise, waiting; when the data processing queue processes the source data, the application identifier is extracted and stored in a temporary variable, the source data is processed, and the processed source data and the application identifier stored in the temporary variable are recombined to generate target data; and extracting the application identifier of the target data, and extracting the target data according to the mapping relation between the application and the application identifier and sending the target data to the application.
An embodiment of the method for high concurrency of multiple application data processing procedures according to the present invention is described, wherein the queue length is 1000.
According to an embodiment of the present invention, the multiple application data processing process is a high concurrency method, wherein the data is sent to the source data processing queue in the format of id + data length.
According to an embodiment of the present invention, the multi-application data processing process is a highly concurrent method, wherein processing the source data includes data reorganization and data transformation.
An embodiment of the method for high concurrency of multiple application data processing procedures according to the present invention, wherein the flag is 4 bytes, and the data length is less than or equal to 4 kbytes and greater than or equal to 4 bytes.
The data processing method and the data processing system lead the data processing flow to realize a high concurrency mechanism of data processing during multi-application through introducing the data processing queue between the service module and the data processing module, thereby improving the data processing efficiency of the multi-application.
Drawings
FIG. 1 is a schematic diagram of a conventional application data processing;
FIG. 2 is a schematic diagram of the multi-application data processing process of the present invention;
FIG. 3 is a flow diagram illustrating multi-application high concurrency data processing;
fig. 4 is a flow chart of a multi-application high-concurrency data processing method.
Detailed Description
In order to make the objects, contents, and advantages of the present invention clearer, the following detailed description of the embodiments of the present invention will be made in conjunction with the accompanying drawings and examples.
Fig. 2 is a schematic diagram of a multi-application data processing procedure of the present invention, fig. 3 is a flow chart of a multi-application high-concurrency data processing procedure, fig. 4 is a flow chart of a multi-application high-concurrency data processing procedure method, as shown in fig. 2 to 4,
applying for a source data processing queue, wherein the queue length is 1000 (the length is determined according to actual conditions);
the application identification of the source data determines the identification number (0, 1, 2.) of the application when the application is initialized, and the format of the application identification is (identification (4 bytes) + data length (less than or equal to (4K bytes) -4 bytes)) when the data is sent to a source data processing queue;
the source data processing queue storage mode is a first-in first-out mode;
the data processing interface detects whether the source data processing queue has data, if so, the data processing interface processes the source data, otherwise, the data processing interface waits;
when the data processing queue processes the source data, the application identifier is extracted and stored in a temporary variable, the source data is processed (operations such as data recombination, data conversion and the like), and the processed data and the application identifier stored in the temporary variable are recombined to generate target data;
the data distribution module extracts the application identification, and extracts the target data to send to the application by referring to the mapping relation between the previous application and the application identification.
When multiple applications need data processing, the method enables source data to be processed to be stored in a processing queue in advance, target data are generated after the data processing is completed, and a data distribution module distributes the target data, so that each application can not monopolize the data processing module when the multiple applications are processed, a concurrence mechanism of the multiple application data processing is realized, and high concurrence of the multiple application data processing process is improved.
1. The application can receive new data after storing the source data needing data processing into the processing queue.
2. If the data processing queue has data, the data processing module takes out the source data from the processing queue for data processing, and the processed data are sent to the corresponding application; the source data sending interface is a producer, the data processing module is a consumer, the producer does not need to wait for the consumer to consume to finish reproduction, the consumer cannot be occupied by a certain producer, and the producer and the consumer are mutually independent parts without mutual influence. The source data sending interface can insert data into the data processing queue all the time, the data processing module can take out the data from the data processing queue all the time to call the thread pool for processing, and even if the data processing module is busy all the time, the source data of the application is not influenced to be inserted into the data processing queue. Therefore, the source data sending interface of the application can push new data into the data processing queue without waiting for the data processing module to finish the processing of the previous application data, the data processing module can receive the data to be processed without waiting for the completion of the processing of the previous application data, and the data processing module is not monopolized by a certain application.
The above description is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, several modifications and variations can be made without departing from the technical principle of the present invention, and these modifications and variations should also be regarded as the protection scope of the present invention.
Claims (6)
1. A multi-application data processing process high concurrency method is characterized by comprising the following steps:
applying for a source data processing queue;
the application identification of the source data determines the identification number of the application identification when the application is initialized;
the source data processing queue storage mode is a first-in first-out mode;
detecting whether the source data processing queue has data, if so, processing the source data by the data processing interface, otherwise, waiting;
when the data processing interface processes the source data, the application identifier is extracted and stored in a temporary variable, the source data is processed, and the processed source data and the application identifier stored in the temporary variable are recombined to generate target data;
and extracting the application identifier of the target data, and extracting the target data according to the mapping relation between the application and the application identifier and sending the target data to the application.
2. The multi-application data processing process high concurrency method of claim 1, wherein the queue length is 1000.
3. The multi-application data processing process high concurrency method of claim 1, wherein the format is identification + data length when sending data to the source data processing queue.
4. The multi-application data processing process high concurrency method of claim 1, wherein processing source data comprises data reorganization and data transformation.
5. The multi-application data processing procedure high concurrency method of claim 1, wherein the flag is 4 bytes, and the data length is 4 kbytes or less and 4 kbytes or more.
6. The multi-application data processing process high concurrency method of claim 1, wherein the data is inserted into the source data processing queue when the source data transmit interface receives the data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911039156.XA CN110990163A (en) | 2019-10-29 | 2019-10-29 | High-concurrency method for multi-application data processing process |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911039156.XA CN110990163A (en) | 2019-10-29 | 2019-10-29 | High-concurrency method for multi-application data processing process |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110990163A true CN110990163A (en) | 2020-04-10 |
Family
ID=70082529
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911039156.XA Pending CN110990163A (en) | 2019-10-29 | 2019-10-29 | High-concurrency method for multi-application data processing process |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110990163A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858735A (en) * | 2005-12-30 | 2006-11-08 | 华为技术有限公司 | Method for processing mass data |
| CN104184635A (en) * | 2014-08-19 | 2014-12-03 | 烽火通信科技股份有限公司 | Method and device of home gateway for achieving one-to-many data communication based on Android RIL |
| CN105989179A (en) * | 2015-03-06 | 2016-10-05 | 北京邮电大学 | Financial data processing method and system |
| US20180285172A1 (en) * | 2017-03-28 | 2018-10-04 | Vmware, Inc. | Data exchange between applications |
| US10445322B1 (en) * | 2018-11-29 | 2019-10-15 | Capital One Services, Llc | Real-time processing of event-based streaming with NoSQL |
-
2019
- 2019-10-29 CN CN201911039156.XA patent/CN110990163A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1858735A (en) * | 2005-12-30 | 2006-11-08 | 华为技术有限公司 | Method for processing mass data |
| CN104184635A (en) * | 2014-08-19 | 2014-12-03 | 烽火通信科技股份有限公司 | Method and device of home gateway for achieving one-to-many data communication based on Android RIL |
| CN105989179A (en) * | 2015-03-06 | 2016-10-05 | 北京邮电大学 | Financial data processing method and system |
| US20180285172A1 (en) * | 2017-03-28 | 2018-10-04 | Vmware, Inc. | Data exchange between applications |
| US10445322B1 (en) * | 2018-11-29 | 2019-10-15 | Capital One Services, Llc | Real-time processing of event-based streaming with NoSQL |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107395565B (en) | Serial port receiving preprocessing method under fixed-length single byte format message protocol | |
| CN102262590B (en) | Method and system for rearranging request queue of hardware accelerator | |
| CN107589990B (en) | Data communication method and system based on thread pool | |
| CN101459611A (en) | Data transmission scheduling method, system and device for IP SAN storage | |
| US20030056073A1 (en) | Queue management method and system for a shared memory switch | |
| CN102195895A (en) | Message sending and receiving processing method and device | |
| CN104065588A (en) | Device for scheduling and buffering data packets and method thereof | |
| CN112632079A (en) | Data stream identification query method and device | |
| EP1569111A2 (en) | Method and object request broker for accelerating object-oriented communication | |
| CN103841181A (en) | Local area network multi-threading file transmission method | |
| CN107870982A (en) | Data processing method, system and computer-readable recording medium | |
| CN112291336B (en) | Multichannel parallel data loading method of ARINC429 network card | |
| CN103888424A (en) | Cluster-type data encryption system and data processing method thereof | |
| CN110990163A (en) | High-concurrency method for multi-application data processing process | |
| CN105357148A (en) | Method and system for preventing output message of network exchange chip from being disordered | |
| CN108830724B (en) | Resource data packet processing method and terminal equipment | |
| CN101183973A (en) | Method of implementing command line configuration distribution in distributed system | |
| CN115982091A (en) | Data processing method, system, medium and equipment based on RDMA engine | |
| CN110704212A (en) | Message processing method and device | |
| CN103220260A (en) | Method of updating data, server, client-side and system | |
| CN105306355A (en) | Transmission control protocol packet adhesion processing method and device | |
| CN113297218B (en) | Multi-system data interaction method, device and system | |
| CN102299861A (en) | Message traffic control method | |
| CN115695409A (en) | Unloading processing method for high-capacity flight parameter data and bus data | |
| CN111949601B (en) | Data storage method and device and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200410 |