CN102833336A - Data sub-packet processing method in separate distributed information acquisition and concurrent processing system - Google Patents
Data sub-packet processing method in separate distributed information acquisition and concurrent processing system Download PDFInfo
- Publication number
- CN102833336A CN102833336A CN2012103157176A CN201210315717A CN102833336A CN 102833336 A CN102833336 A CN 102833336A CN 2012103157176 A CN2012103157176 A CN 2012103157176A CN 201210315717 A CN201210315717 A CN 201210315717A CN 102833336 A CN102833336 A CN 102833336A
- Authority
- CN
- China
- Prior art keywords
- data
- thread
- packet
- data processing
- divided
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a data sub-packet processing method in a separate distributed information acquisition and concurrent processing system. The data sub-packet processing method comprises the following steps of: adding data packet identifier including a task uniqueness mark and sub-packet information for each data sub-packet when a client side decomposes an original task into a plurality of data sub-packets; transmitting the data sub-packet with the data packet identifier to a server side; opening a data receiving thread, establishing a thread pool comprising at least one data processing thread and a data distribution thread, meanwhile developing a section of dynamic memory zero BM and developing a dynamic memory zone SM for each data processing thread in the thread pool by the server side when a process is started; and distributing a corresponding data processing thread for the data sub-packet by the data distribution thread according to the data packet identifier and the state of each data processing thread, and distributing the data sub-packet of the same original task to the same data processing thread. According to the data sub-packet processing method, the data processing performance, the correctness and the stability of a mass data concurrent processing system can be obviously improved.
Description
Technical field
The present invention relates to a kind of data processing method, relate in particular in information gathering of a kind of dispersed and distributed formula and the concurrent processing system data and divide packet processing method, belong to technical field of the computer network.
Background technology
In Communication System Development based on TCP/IP; All to relate to and carry out procedure development based on the SOCKET technology; Based on the SOCKET technological development, relate to client and send data and service end reception data, and following based on the realization flow of SOCKET technological service end:
1. load socket
2. create the socket of monitoring
3. bound socket
4. monitoring socket
5. receive client data, handle the client association requests
At the 5th point, service end receives client data, generally produces a thread, passes to this new thread to the data that receive and handles.
Information gathering of dispersed and distributed formula and concurrent processing system are meant system terminal and a small amount of central processing platform that in a large-scale distributed system, relates to magnanimity; The mass data terminal is distributed in zones of different or the same area; And walk abreast to central processing platform transmission, request msg or central processing platform at one time to each system terminal request msg; Central processing platform is simultaneously to the request of data of each system terminal parallel acquisition data or a large amount of system terminals of parallel processing; Be characterized in the same time; The central processing platform parallel processing comes from the mass data of system terminal in a large number, and most typical application is that the encoding and decoding of various web servers, telecommunications backstage charge system are handled or the like.In information gathering of dispersed and distributed formula and concurrent processing system; It is inevitable and a large amount of things that exist that complete request subpackage of client sends to service end; Great amount of terminals subpackage request at random; After service end receives the random request data, the bag that very difficult differentiation the receives processing that how on earth packages, two results of so frequent appearance:
One, the client terminal quantity of subpackage transmission of same time of restriction does not allow a plurality of clients to occur subpackage simultaneously and sends data conditions;
Two, the subpackage simultaneously of a plurality of clients is sent, and the service end process is returned error result;
So just had a strong impact on accuracy, the stability of system handles data, the system that influences uses.
Summary of the invention
Technical problem to be solved by this invention is to overcome the prior art deficiency; Provide in information gathering of a kind of dispersed and distributed formula and the concurrent processing system data to divide packet processing method; Through the data that receive are at random judged; In the data dispatch thread through rationally judging and handle, to each data processing thread carry out correctly, efficient data distribution, can solve any terminal at random subpackage send data to the processing condition data of service end; Improve service end data-handling efficiency, accuracy, improve the system data handling property.
The present invention is concrete to adopt following technical scheme to solve the problems of the technologies described above.
Data are divided packet processing method in information gathering of a kind of dispersed and distributed formula and the concurrent processing system, and client is decomposed into a plurality of divided data bags with ancestral task, is sent to service end; After service end receives said a plurality of divided data bag, handle after it is merged into ancestral task;
Client is that each divided data bag adds one and comprises task uniqueness sign and the package identification that divides package informatin when ancestral task being decomposed into a plurality of divided data bag, and the divided data bag that will have package identification then is sent to service end;
Service end is when process initiation; Open a data receiving thread, and create the thread pool that comprises at least one data processing thread, and a data dispatch thread; Open up one section dynamic area BM simultaneously, and open up a dynamic area SM for each the data processing thread in the thread pool;
The Data Receiving thread receives the divided data bag and deposits it in dynamic area BM; The divided data bag among the dynamic area BM is read in the circulation of data dispatch thread; State according to package identification and each data processing thread deposits the divided data bag in corresponding dynamic memory block SM; The divided data bag that belongs to same ancestral task deposits same dynamic area SM in; And when all divided data Bao Jun of a certain ancestral task deposit same dynamic area SM in, trigger the pairing data processing thread of this dynamic area SM and merge and handle the packetized data that this dynamic area SM is stored.
Preferably; The said package identification that adds for the divided data bag is the information header that comprises task uniqueness sign, overall budget number and packet number; Wherein, The overall budget number is meant total number of all divided data bags that the ancestral task under this divided data bag is decomposed, and packet number is meant the sequence number in the divided data bag that the ancestral task of this divided data under wrapping in decompose.
Further, after service end comprises the thread pool of at least one data processing thread in establishment, for each the data processing thread in the thread pool is provided with a thread state sign and a data sign; The divided data bag among the dynamic area BM is read in said data dispatch thread circulation, and carries out handled according to the information header of divided data bag, and is specific as follows:
If the overall budget number in the information header greater than 1, packet number equals 1; Then from said thread pool, select a thread state to be masked as the data processing thread of " free time "; Put into the divided data bag dynamic area SM of this data processing thread; The Data Labels of this data processing thread is set to the task uniqueness sign in the said information header simultaneously, and the thread state sign of this data processing thread is set to " subpackage " state;
If the overall budget number in the information header greater than 1, packet number is greater than 1 and less than the bag sum; Then the thread Status Flag is in the data processing thread of " subpackage " state in thread pool; Detect the identical data processing thread of task uniqueness sign in Data Labels and the said information header, and put into the dynamic area SM of this data processing thread to the divided data bag;
If the packet sum in the information header greater than 1, the overall budget number equates with packet number; Then the thread Status Flag is in the data processing thread of " subpackage " state in thread pool; Detect the identical data processing thread of task uniqueness sign in Data Labels and the said information header; And put into the divided data bag dynamic area SM of this data processing thread; Trigger this data processing thread simultaneously and merge the packetized data that processing this dynamic area SM is stored, and the thread state of this data processing thread of resetting is masked as " free time ".
According to the inventive method, also can obtain a kind of information gathering of dispersed and distributed formula and concurrent processing system that uses above-mentioned arbitrary technical scheme.
The inventive method is through judging the data that receive at random; In the data dispatch thread, judge and processing through reasonable; To each data processing thread carry out correctly, efficient data distribution; Can solve any terminal at random subpackage send data to the processing condition data of service end, improve service end data-handling efficiency, accuracy, improve the system data handling property.
Description of drawings
Fig. 1 is the principle schematic of the inventive method;
Fig. 2 is the packetized data handling process sketch map of the inventive method.
Embodiment
Below in conjunction with accompanying drawing technical scheme of the present invention is elaborated:
In the inventive method, client is that each divided data bag adds one and comprises task uniqueness sign and the package identification that divides package informatin when ancestral task being decomposed into a plurality of divided data bag, and the divided data bag that will have package identification then is sent to service end; Service end is when process initiation; As shown in Figure 1; Open a data receiving thread, and create the thread pool that comprises at least one data processing thread, and a data dispatch thread; Open up one section dynamic area BM simultaneously, and open up a dynamic area SM for each the data processing thread in the thread pool; The Data Receiving thread receives the divided data bag and deposits it in dynamic area BM; The divided data bag among the dynamic area BM is read in the circulation of data dispatch thread; State according to package identification and each data processing thread deposits the divided data bag in corresponding dynamic memory block SM; The divided data bag that belongs to same ancestral task deposits same dynamic area SM in; And when all divided data Bao Jun of a certain ancestral task deposit same dynamic area SM in, trigger the pairing data processing thread of this dynamic area SM and merge and handle the packetized data that this dynamic area SM is stored.
Understand technical scheme of the present invention for the ease of the public, come of the present invention minute packet processing method is elaborated with a specific embodiment below, its handling process is as shown in Figure 2, may further comprise the steps:
When step 1, service end process initiation; Open a data receiving thread R; Open a data partition line journey S and open a data processing thread pool that comprises a plurality of data processing thread D; Simultaneously in program, open up one section dynamic area BM that is used to store the client data that receives, and manage dynamic area BM according to the FIFO mode; Service end is that each the data processing thread in the thread pool is provided with a thread state sign TS and a data sign DS, and a bit of dynamic area SM.
Step 2, client are decomposed into a plurality of divided data bags with ancestral task, comprise task uniqueness sign and the package identification that divides package informatin for each divided data bag adds one, and the divided data bag that will have package identification then is sent to service end successively; In this embodiment, said task uniqueness sign is made up of client sign+task number, ancestral task of each task uniqueness sign sign; Said subpackage information comprises overall budget number and packet number; Wherein, The overall budget number is meant total number of all divided data bags that the ancestral task under this divided data bag is decomposed, and packet number is meant the sequence number in the divided data bag that the ancestral task of this divided data under wrapping in decompose; Task uniqueness sign, overall budget number and packet number have constituted the uniqueness sign of divided data bag jointly.
The Data Receiving thread R of step 3, service end intercepts and receives client and passes the data of coming through socket, directly put into data storage area BM to data, returns immediately and continues to intercept the reception client data.
Dynamic area BM is read in step 4, the circulation of data dispatch thread; Obtain divided data bag D; Parse the information header of the divided data bag that reads, divided data is responsible for assigning to corresponding data processing thread according to the state of information header and each data processing thread, specific as follows:
If the overall budget number in the information header is greater than 1; And packet number equals 1; Then poll detects the thread state sign TS of the data processing thread of each thread in the thread pool, if thread state sign TS is " free time ", then puts into the divided data bag dynamic area SM of this data processing thread; The Data Labels DS that this data processing thread is set simultaneously is the task uniqueness sign in the information header of divided data bag, and the thread state sign TS that this thread is set is " subpackage " state;
If the packet sum in the information header greater than 1, packet number is greater than 1 and less than the bag sum; Then poll detects the Status Flag TS of each data processing thread in the thread pool; If thread state sign TS is " subpackage " state; The Data Labels DS that then continue to detect this thread whether with the information header of divided data bag in task uniqueness sign be complementary (promptly whether identical), like coupling, then put into the divided data bag dynamic area SM of this data processing thread;
If the packet sum greater than 1, the overall budget number equates with sequence number; Then poll detects the Status Flag TS of each data processing thread in the thread pool; If thread state sign TS is " subpackage " state, the Data Labels DS that then continue to detect this thread whether with the information header of divided data bag in task uniqueness sign be complementary; Mate, then put into the divided data bag dynamic area SM of this data processing thread, trigger this data processing thread simultaneously and begin to handle the packetized data that has received, data are merged handle, and replacement thread state sign TS is " free time " state.
Pass through said method; Service end is carried out multi-threaded parallel to the divided data bag from a large amount of clients and is handled; Can be dispensed to the dynamic area SM of same data processing thread successively from the divided data bag of same ancestral task; And after all divided data bags arrive, immediately it is merged, handles; The data processing thread of accomplishing after handling restarts new task handling, thereby has effectively improved accuracy, the stability of system handles data.
Claims (4)
1. data are divided packet processing method in dispersed and distributed formula information gathering and the concurrent processing system, and client is decomposed into a plurality of divided data bags with ancestral task, is sent to service end; After service end receives said a plurality of divided data bag, handle after it is merged into ancestral task; It is characterized in that,
Client is that each divided data bag adds one and comprises task uniqueness sign and the package identification that divides package informatin when ancestral task being decomposed into a plurality of divided data bag, and the divided data bag that will have package identification then is sent to service end;
Service end is when process initiation; Open a data receiving thread, and create the thread pool that comprises at least one data processing thread, and a data dispatch thread; Open up one section dynamic area BM simultaneously, and open up a dynamic area SM for each the data processing thread in the thread pool;
The Data Receiving thread receives the divided data bag and deposits it in dynamic area BM; The divided data bag among the dynamic area BM is read in the circulation of data dispatch thread; State according to package identification and each data processing thread deposits the divided data bag in corresponding dynamic memory block SM; The divided data bag that belongs to same ancestral task deposits same dynamic area SM in; And when all divided data Bao Jun of a certain ancestral task deposit same dynamic area SM in, trigger the pairing data processing thread of this dynamic area SM and merge and handle the packetized data that this dynamic area SM is stored.
According to claim 1 in information gathering of dispersed and distributed formula and the concurrent processing system data divide packet processing method; It is characterized in that; The said package identification that adds for the divided data bag is the information header that comprises task uniqueness sign, overall budget number and packet number; Wherein, the overall budget number is meant total number of all divided data bags that the ancestral task under this divided data bag is decomposed, and packet number is meant the sequence number in the divided data bag that the ancestral task of this divided data under wrapping in decompose.
3. divide packet processing method like data in said diffusing distributed information collection of claim 2 and the concurrent processing system; It is characterized in that; After service end comprises the thread pool of at least one data processing thread in establishment, for each the data processing thread in the thread pool is provided with a thread state sign and a data sign; The divided data bag among the dynamic area BM is read in said data dispatch thread circulation, and carries out handled according to the information header of divided data bag, and is specific as follows:
If the overall budget number in the information header greater than 1, packet number equals 1; Then from said thread pool, select a thread state to be masked as the data processing thread of " free time "; Put into the divided data bag dynamic area SM of this data processing thread; The Data Labels of this data processing thread is set to the task uniqueness sign in the said information header simultaneously, and the thread state sign of this data processing thread is set to " subpackage " state;
If the overall budget number in the information header greater than 1, packet number is greater than 1 and less than the bag sum; Then the thread Status Flag is in the data processing thread of " subpackage " state in thread pool; Detect the identical data processing thread of task uniqueness sign in Data Labels and the said information header, and put into the dynamic area SM of this data processing thread to the divided data bag;
If the packet sum in the information header greater than 1, the overall budget number equates with packet number; Then the thread Status Flag is in the data processing thread of " subpackage " state in thread pool; Detect the identical data processing thread of task uniqueness sign in Data Labels and the said information header; And put into the divided data bag dynamic area SM of this data processing thread; Trigger this data processing thread simultaneously and merge the packetized data that processing this dynamic area SM is stored, and the thread state of this data processing thread of resetting is masked as " free time ".
4. the information gathering of dispersed and distributed formula and the concurrent processing system of a use such as each said packetized data processing method of claim 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012103157176A CN102833336A (en) | 2012-08-31 | 2012-08-31 | Data sub-packet processing method in separate distributed information acquisition and concurrent processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012103157176A CN102833336A (en) | 2012-08-31 | 2012-08-31 | Data sub-packet processing method in separate distributed information acquisition and concurrent processing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102833336A true CN102833336A (en) | 2012-12-19 |
Family
ID=47336303
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012103157176A Pending CN102833336A (en) | 2012-08-31 | 2012-08-31 | Data sub-packet processing method in separate distributed information acquisition and concurrent processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102833336A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103078811A (en) * | 2013-01-31 | 2013-05-01 | 北京金和软件股份有限公司 | Network data packet out-of-order control method based on multithreaded environment |
| CN103457926A (en) * | 2013-01-17 | 2013-12-18 | 成都阜特科技股份有限公司 | Communication method for CMS front-end processor and server |
| CN103647790A (en) * | 2013-12-24 | 2014-03-19 | 常州工学院 | Extra-large file protocol analytical and statistical method |
| CN104142865A (en) * | 2014-07-18 | 2014-11-12 | 国家电网公司 | Data collecting and processing method based on thread synchronization |
| CN104572262A (en) * | 2014-12-27 | 2015-04-29 | 北京奇虎科技有限公司 | Task execution method and device |
| CN104834558A (en) * | 2015-05-19 | 2015-08-12 | 北京京东尚科信息技术有限公司 | Method and system for processing data |
| CN105260378A (en) * | 2015-09-08 | 2016-01-20 | 上海上讯信息技术股份有限公司 | Database audit method and device |
| CN108737180A (en) * | 2018-05-22 | 2018-11-02 | 北京汉能光伏投资有限公司 | Collecting method, data transmission method for uplink, data acquisition equipment and the network equipment |
| CN110855738A (en) * | 2019-09-25 | 2020-02-28 | 安徽四创电子股份有限公司 | Communication processing system for multi-source equipment |
| CN110943942A (en) * | 2019-11-13 | 2020-03-31 | 北京连山时代科技有限公司 | Multi-channel concurrent transmission method and system based on data packet numbers |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1434386A (en) * | 2003-01-25 | 2003-08-06 | 华中科技大学 | Division distributed storage method of program source based on cluster video server |
| CN101119219A (en) * | 2007-09-12 | 2008-02-06 | 中兴通讯股份有限公司 | Method and system for implementing data transmission of broadcast/multicast service |
| EP1236088B1 (en) * | 1999-09-01 | 2008-05-21 | Intel Corporation | Register set used in multithreaded parallel processor architecture |
| CN101217467A (en) * | 2007-12-28 | 2008-07-09 | 杭州华三通信技术有限公司 | An inter-core load dispensing device and method |
| CN102624889A (en) * | 2012-03-06 | 2012-08-01 | 河海大学 | Mass data concurrency processing method based on receiving and processing separation |
-
2012
- 2012-08-31 CN CN2012103157176A patent/CN102833336A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1236088B1 (en) * | 1999-09-01 | 2008-05-21 | Intel Corporation | Register set used in multithreaded parallel processor architecture |
| CN1434386A (en) * | 2003-01-25 | 2003-08-06 | 华中科技大学 | Division distributed storage method of program source based on cluster video server |
| CN101119219A (en) * | 2007-09-12 | 2008-02-06 | 中兴通讯股份有限公司 | Method and system for implementing data transmission of broadcast/multicast service |
| CN101217467A (en) * | 2007-12-28 | 2008-07-09 | 杭州华三通信技术有限公司 | An inter-core load dispensing device and method |
| CN102624889A (en) * | 2012-03-06 | 2012-08-01 | 河海大学 | Mass data concurrency processing method based on receiving and processing separation |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103457926A (en) * | 2013-01-17 | 2013-12-18 | 成都阜特科技股份有限公司 | Communication method for CMS front-end processor and server |
| CN103078811B (en) * | 2013-01-31 | 2015-12-09 | 北京金和软件股份有限公司 | A kind of based on multi-thread environment network packet out-of-order control method |
| CN103078811A (en) * | 2013-01-31 | 2013-05-01 | 北京金和软件股份有限公司 | Network data packet out-of-order control method based on multithreaded environment |
| CN103647790A (en) * | 2013-12-24 | 2014-03-19 | 常州工学院 | Extra-large file protocol analytical and statistical method |
| CN104142865A (en) * | 2014-07-18 | 2014-11-12 | 国家电网公司 | Data collecting and processing method based on thread synchronization |
| CN104572262A (en) * | 2014-12-27 | 2015-04-29 | 北京奇虎科技有限公司 | Task execution method and device |
| CN104572262B (en) * | 2014-12-27 | 2018-09-04 | 北京奇虎科技有限公司 | A kind of task executing method and device |
| CN104834558A (en) * | 2015-05-19 | 2015-08-12 | 北京京东尚科信息技术有限公司 | Method and system for processing data |
| CN104834558B (en) * | 2015-05-19 | 2018-06-01 | 北京京东尚科信息技术有限公司 | A kind of method and system of data processing |
| CN105260378A (en) * | 2015-09-08 | 2016-01-20 | 上海上讯信息技术股份有限公司 | Database audit method and device |
| CN108737180A (en) * | 2018-05-22 | 2018-11-02 | 北京汉能光伏投资有限公司 | Collecting method, data transmission method for uplink, data acquisition equipment and the network equipment |
| WO2019223065A1 (en) * | 2018-05-22 | 2019-11-28 | 北京汉能光伏投资有限公司 | Data collection method, data sending method, data collection device, and network device |
| CN110855738A (en) * | 2019-09-25 | 2020-02-28 | 安徽四创电子股份有限公司 | Communication processing system for multi-source equipment |
| CN110855738B (en) * | 2019-09-25 | 2022-06-07 | 安徽四创电子股份有限公司 | Communication processing system for multi-source equipment |
| CN110943942A (en) * | 2019-11-13 | 2020-03-31 | 北京连山时代科技有限公司 | Multi-channel concurrent transmission method and system based on data packet numbers |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102833336A (en) | Data sub-packet processing method in separate distributed information acquisition and concurrent processing system | |
| CN101656677B (en) | Message diversion processing method and device | |
| CN202455371U (en) | Multi-machine testing system | |
| CN102638487B (en) | Large-scale telemetric seismic instrument high-performance data transmission method | |
| CN109391500A (en) | A kind of configuring management method, device and equipment | |
| CN108881008A (en) | A kind of methods, devices and systems of data transmission | |
| CN101877659B (en) | Method, device and system for monitoring packet loss | |
| CN102694855A (en) | Meteorological disaster early warning information publishing system and method on basis of Beidou satellite | |
| CN106155812A (en) | Method, device, system and the electronic equipment of a kind of resource management to fictitious host computer | |
| CN105556916B (en) | The information statistical method and device of network flow | |
| CN108429701A (en) | network acceleration system | |
| CN102377650A (en) | Data transmission processing method, device and system | |
| CN102737065A (en) | Method and device for acquiring data | |
| CN105141660A (en) | Stream media management method and system | |
| CN102255942A (en) | Multimedia quantum flow and cloud processing technology and realization method | |
| CN104519129A (en) | Data transmission method, device and system | |
| CN102035738A (en) | Method and device for acquiring routing information | |
| CN104618253B (en) | A kind of transmitting message treating method and apparatus of dynamic change | |
| CN105763297A (en) | Cloud computing system-based remote data optimized transmission method and device | |
| CN104821958B (en) | Electricity consumption data packet interactive interface method based on WebService | |
| CN103441952A (en) | Network data package processing method based on multi-core or many-core embedded processor | |
| CN106209764A (en) | A kind of data transmission method based on udp protocol and system | |
| CN104202389A (en) | Monitoring method for storage space and running state in cloud environment and cloud storage system | |
| CN106850770A (en) | High-efficiency transmission method and device based on cloud computing service | |
| CN109089299A (en) | It is a kind of to allow facility registration to cloud with network method based on SmartConfig mode |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20121219 |