CN107544855A - A kind of method for quickly being analyzed and processed using multithreading and distributing data - Google Patents
A kind of method for quickly being analyzed and processed using multithreading and distributing data Download PDFInfo
- Publication number
- CN107544855A CN107544855A CN201710941400.6A CN201710941400A CN107544855A CN 107544855 A CN107544855 A CN 107544855A CN 201710941400 A CN201710941400 A CN 201710941400A CN 107544855 A CN107544855 A CN 107544855A
- Authority
- CN
- China
- Prior art keywords
- data
- terminal
- data analysis
- line journey
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a kind of method for quickly being analyzed and processed using multithreading and distributing data, first terminal security code is defined for each data terminal, for every a kind of data analysis machine defined analysis code, establish data publication queue, then data analysis main thread creates terminal queue thread according to the terminal security code of acquisition, the analyze data request content in data analysis sub-line journey, determine data analysis pattern, it is subsequently sent in data analysis machine, specific data are handled in data analysis machine and are saved in Redis, there is the terminal for changing data to send data acquisition instruction to all subscription simultaneously.The present invention takes full advantage of the advantage of multithreading, and each stage content is separated, and can allow all request quick responses and issue, and influencing each other between reducing, and greatly improves data processing, the efficiency of distribution.
Description
Technical field
The invention belongs to Redis Publish-subscribe message systems, are related to one kind using multithreading quickly analyzing and processing and distribution
The method of data.
Background technology
Redis Publish-subscribe patterns realize interprocess communication, and subscriber can subscribe to one or more channels
（channel）, and publisher can send message to specified channel, all subscribers for subscribing to sub-channel, which can receive, to disappear
Breath.However, when subscribe to and issue number it is many when, and exist complex data processing logic when, direct distribution subscription then can
It is processed the constraint of performance.
Allow all request quick responses and issue, and between reducing influencing each other to Data Analysis Services and point
Send out important in inhibiting.
The content of the invention
To ensure the promptness and concurrency of data distribution processing, and effectively improve the efficiency of data distribution processing.This
The purpose of invention is to provide a kind of method for quickly being analyzed and processed using multithreading and distributing data, can allow all requests are quick to ring
Should and it issue, and influencing each other between reduction, effectively improve the efficiency that data distribution is handled.
The purpose of the present invention is achieved through the following technical solutions：
A kind of method for quickly being analyzed and processed using multithreading and distributing data, it is characterised in that this method is as follows：
1）Terminal security code is defined for each data terminal first, is per a kind of data analysis machine defined analysis code, establishes data
Publishing queue；
2）Then data analysis main thread creates terminal queue thread according to the terminal security code of acquisition, in data analysis sub-line journey
Middle analyze data request content, determines data analysis pattern, is subsequently sent in data analysis machine；
3）Specific data are handled in data analysis machine and are saved in Redis, while have the end for changing data to all subscription
End sends data acquisition instruction.
Comprise the following steps that：
Step 1：Monitoring information passage, the solicited message from the issue of each terminal is obtained in real time；
Step 2：Terminal security code is obtained, is created and maintenance terminal queue according to terminal security code；
Step 3：Obtain data processor coding, the information content that the processing of information processing sub-line journey is issued from queue；
Step 4：Information processing sub-line journey processing data, each data processor independent process data information content；
Step 5：Information processing sub-line journey preserves data, and data result information is built into Redis according to data format requirement；
Step 6：Information processing sub-line journey issues load instruction, subscribes to terminal and receives load instruction from redis acquisition data；
Step 7：Information processing sub-line journey issues data acquisition condition information, log recording to main thread.
Present invention introduces multithreading, using producer consumer pattern.First end is defined for each data terminal
Hold safety code, be per a kind of data analysis machine defined analysis code, establish data publication queue, then data analysis main thread according to
The terminal security code of acquisition creates terminal queue thread, and analyze data request content, determines data in data analysis sub-line journey
Analytical model, it is subsequently sent in data analysis machine, specific data is handled in data analysis machine and are saved in Redis,
There is the terminal for changing data to send data acquisition instruction to all subscription simultaneously.
The present invention takes full advantage of the advantage of multithreading, and each stage content is separated, and can allow all request quick responses
And influencing each other between issue, and reduction, greatly improve data processing, the efficiency of distribution.
Brief description of the drawings
Fig. 1 is main thread workflow diagram in the present invention；
Fig. 2 is neutron thread work flow chart of the present invention；
Fig. 3 is main thread and sub-line journey message transmission schematic diagram in the present invention.
Embodiment
Below in conjunction with the accompanying drawings and it is embodied, the handling process of the present invention is further elaborated on.
A kind of method for quickly being analyzed and processed using multithreading and distributing data：First terminal is defined for each data terminal
Safety code, it is per a kind of data analysis machine defined analysis code, establishes data publication queue；Then data analysis main thread is according to obtaining
The terminal security code obtained creates terminal queue thread, the analyze data request content in data analysis sub-line journey, determines data point
Analysis pattern, is subsequently sent in data analysis machine；Specific data are handled in data analysis machine and are saved in Redis, together
When to it is all subscription have change data terminal send data acquisition instruction.
Comprise the following steps that：
Step 1：Monitoring information passage.The solicited message from the issue of each terminal is obtained in real time.
Step 2：Obtain terminal security code.Created and maintenance terminal queue according to terminal security code.
Step 3：Obtain data processor coding.The information content that the processing of information processing sub-line journey is issued from queue
Step 4：Information processing sub-line journey processing data.Each data processor independent process data information content
Step 5：Information processing sub-line journey preserves data.Data result information is built into Redis according to data format requirement.
Step 6：Information processing sub-line journey issues load instruction.Subscribe to terminal and receive load instruction from redis acquisition data
Step 7：Information processing sub-line journey issues data acquisition condition information, log recording to main thread.
Claims (2)
- A kind of 1. method for quickly being analyzed and processed using multithreading and distributing data, it is characterised in that this method is as follows：1）Terminal security code is defined for each data terminal first, is per a kind of data analysis machine defined analysis code, establishes data Publishing queue；2）Then data analysis main thread creates terminal queue thread according to the terminal security code of acquisition, in data analysis sub-line journey Middle analyze data request content, determines data analysis pattern, is subsequently sent in data analysis machine；3）Specific data are handled in data analysis machine and are saved in Redis, while have the end for changing data to all subscription End sends data acquisition instruction.
- 2. the method according to claim 1 for quickly being analyzed and processed using multithreading and distributing data, it is characterised in that tool Body step is as follows：Step 1：Monitoring information passage, the solicited message from the issue of each terminal is obtained in real time；Step 2：Terminal security code is obtained, is created and maintenance terminal queue according to terminal security code；Step 3：Obtain data processor coding, the information content that the processing of information processing sub-line journey is issued from queue；Step 4：Information processing sub-line journey processing data, each data processor independent process data information content；Step 5：Information processing sub-line journey preserves data, and data result information is built into Redis according to data format requirement；Step 6：Information processing sub-line journey issues load instruction, subscribes to terminal and receives load instruction from redis acquisition data；Step 7：Information processing sub-line journey issues data acquisition condition information, log recording to main thread.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710941400.6A CN107544855A (en) | 2017-10-11 | 2017-10-11 | A kind of method for quickly being analyzed and processed using multithreading and distributing data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710941400.6A CN107544855A (en) | 2017-10-11 | 2017-10-11 | A kind of method for quickly being analyzed and processed using multithreading and distributing data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107544855A true CN107544855A (en) | 2018-01-05 |
Family
ID=60967687
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710941400.6A Pending CN107544855A (en) | 2017-10-11 | 2017-10-11 | A kind of method for quickly being analyzed and processed using multithreading and distributing data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107544855A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112511558A (en) * | 2020-12-01 | 2021-03-16 | 河南东方世纪交通科技股份有限公司 | Electromechanical device measurement and control system based on Internet of things |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902512A (en) * | 2012-08-31 | 2013-01-30 | 浪潮电子信息产业股份有限公司 | Multi-thread parallel processing method based on multi-thread programming and message queue |
| CN104394090A (en) * | 2014-11-14 | 2015-03-04 | 北京航空航天大学 | SDN (Software Defined Networking) controller classifying network flows through DPI (Deep Packet Inspection) data package |
| CN104935658A (en) * | 2015-06-17 | 2015-09-23 | 交通银行股份有限公司 | Bank task allocation method and device |
| US9378069B2 (en) * | 2014-03-05 | 2016-06-28 | International Business Machines Corporation | Lock spin wait operation for multi-threaded applications in a multi-core computing environment |
| CN106021315A (en) * | 2016-05-05 | 2016-10-12 | 智者四海（北京）技术有限公司 | Log management method and system for application program |
| US9626220B2 (en) * | 2015-01-13 | 2017-04-18 | International Business Machines Corporation | Computer system using partially functional processor core |
-
2017
- 2017-10-11 CN CN201710941400.6A patent/CN107544855A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102902512A (en) * | 2012-08-31 | 2013-01-30 | 浪潮电子信息产业股份有限公司 | Multi-thread parallel processing method based on multi-thread programming and message queue |
| US9378069B2 (en) * | 2014-03-05 | 2016-06-28 | International Business Machines Corporation | Lock spin wait operation for multi-threaded applications in a multi-core computing environment |
| CN104394090A (en) * | 2014-11-14 | 2015-03-04 | 北京航空航天大学 | SDN (Software Defined Networking) controller classifying network flows through DPI (Deep Packet Inspection) data package |
| US9626220B2 (en) * | 2015-01-13 | 2017-04-18 | International Business Machines Corporation | Computer system using partially functional processor core |
| CN104935658A (en) * | 2015-06-17 | 2015-09-23 | 交通银行股份有限公司 | Bank task allocation method and device |
| CN106021315A (en) * | 2016-05-05 | 2016-10-12 | 智者四海（北京）技术有限公司 | Log management method and system for application program |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112511558A (en) * | 2020-12-01 | 2021-03-16 | 河南东方世纪交通科技股份有限公司 | Electromechanical device measurement and control system based on Internet of things |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106020948B (en) | A kind of process dispatch method and device | |
| CN107645562A (en) | Data transmission processing method, device, equipment and system | |
| CN106161135B (en) | Business transaction failure analysis methods and device | |
| CN106020777B (en) | A kind of data processing method, apparatus and system | |
| CN111078432B (en) | Tracking method and device for scheduling between services | |
| CN103023980A (en) | Method and system for processing user service request by cloud platform | |
| CN102007495A (en) | A method, apparatus and software for transforming a natural language request for modifying a set of subscriptions for a publish/subscribe topic string | |
| CN104734983A (en) | Scheduling system, method and device for service data request | |
| CN107544855A (en) | A kind of method for quickly being analyzed and processed using multithreading and distributing data | |
| CN113329139B (en) | Video stream processing method, device and computer readable storage medium | |
| CN109379245A (en) | A kind of wifi report form generation method and system | |
| CN110221914B (en) | File processing method and device | |
| CN102685257A (en) | Data synchronous processing system, data synchronous processing method and server system | |
| CN103905484A (en) | Method of processing hyper text transport protocol (HTTP) requests and device | |
| CN106776072A (en) | Information push method and system | |
| CN106407391B (en) | Data quality monitoring method and system | |
| CN104239537A (en) | Method for realizing generating and processing flow for large-data pre-processing text data | |
| CN114979058B (en) | CAN multi-mailbox multiplexing processing method and system | |
| CN103514036A (en) | Scheduling system and method for event trigger and batch processing | |
| CN106598721B (en) | Media asset data streaming method and device | |
| CN115994156A (en) | Method and system for real-time analysis of data streams | |
| CN114064397A (en) | Heterogeneous data source synchronization quality monitoring method and device and electronic equipment | |
| CN105095042B (en) | Management information system and its method for processing business | |
| CN112817761A (en) | Energy-saving method for enhancing cloud computing environment | |
| CN110519381B (en) | Distributed task processing method and device, terminal equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180105 |