CN111737026A - Multithreading message processing method based on lookup operation - Google Patents
Multithreading message processing method based on lookup operation Download PDFInfo
- Publication number
- CN111737026A CN111737026A CN202010469760.2A CN202010469760A CN111737026A CN 111737026 A CN111737026 A CN 111737026A CN 202010469760 A CN202010469760 A CN 202010469760A CN 111737026 A CN111737026 A CN 111737026A
- Authority
- CN
- China
- Prior art keywords
- message
- processing
- lookup
- thread pool
- message queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 78
- 238000000034 method Methods 0.000 claims abstract description 29
- 238000012216 screening Methods 0.000 claims abstract description 7
- 238000012544 monitoring process Methods 0.000 claims description 4
- 238000006467 substitution reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Multi Processors (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a multithreading message processing method based on lookup operation, which comprises the following steps: increasing a lookup message processing thread pool, and setting the minimum processing amount of the thread pool; setting an initial position, traversing the message queue, and screening out the message queue to be processed according to the minimum processing amount; averagely distributing the messages in the message queue to be processed to the threads of the thread pool; and simultaneously carrying out lookup processing operation on each thread. The method actively transfers the lookup messages which are intensively sent to the MDS to the corresponding thread pool in the message queue, and executes the lookup processing operation at the same time, thereby greatly saving the overall message processing time of the message queue.
Description
Technical Field
The invention belongs to the technical field of distributed file processing, and particularly relates to a multithreading message processing method based on lookup operation.
Background
In the current distributed file system, in many scenarios, at least one lookup operation needs to be performed before actual operations are performed, for example, a lookup operation is performed before a file is created or deleted in a certain directory to determine whether the file really exists, so as to assist an upper module in determining whether the actual operations can be performed. In all messages, the lookup operation in many scenarios is only used for acquiring the basic attribute of the file, and the file is not changed by writing, and the consistency and accuracy of data are not affected by such operation.
In practical application, such lookup operation messages are often sent to the message queue of the MDS in a short time period, and in the MDS, all messages must wait for the completion of the previous message processing before taking out the next message from the message queue for processing, so as to ensure the consistency and correctness of data. Assuming that there are n messages in the message queue to process at the same time, i.e. the time consumed by the nth message can be expressed as follows: and the consumed time n is the processing time of the message plus the processing time of the previous n-1 messages. If there is a high time-consuming message processing operation in the previous n-1 message, such as network delay, etc., this will result in high time-consuming messages in the queue behind the message, never resulting in performance fluctuations of the upper layer application many times in succession.
Disclosure of Invention
In view of the above disadvantages in the prior art, the present invention provides a multithread message processing method based on lookup operation to solve the above technical problems.
The invention provides a multithreading message processing method based on lookup operation, which comprises the following steps:
acquiring a message queue of a distributed file system which needs to perform lookup processing operation;
adding a lookup message processing thread pool in the message queue, and setting the lowest processing amount of the thread pool for processing messages;
traversing the message queue, and screening out a message queue to be processed according to the lowest processing amount;
and averagely distributing the lookup type messages in the message queue to be processed to the threads of the thread pool, and simultaneously processing the lookup type messages by a plurality of threads in the thread pool.
Further, the traversing the message queue and screening the message queue to be processed according to the minimum throughput includes:
configuring the Mth message after the current processing message as the starting position;
and traversing the current message queue from the initial position, and taking the message with the message quantity continuously belonging to the lookup type reaching the minimum processing quantity as a message queue to be processed.
Further, the method further comprises:
inserting a self-defined type message at the starting point of the message queue to be processed;
monitoring the processing state of the custom type message: when the self-defined type message is processed, the original message processing thread is suspended, and a plurality of threads in the thread pool process the lookup type message at the same time.
Further, the method further comprises:
and adding a new thread, wherein the new thread is used for processing the custom type message.
Further, the method further comprises:
and after the processing of each thread is finished, independently returning a processing result to the upper layer module.
Further, adding a lookup message processing thread pool in the message queue, including:
adding a thread pool, wherein the thread pool is used for processing the lookup type message in the message queue;
setting the default dormant state of the thread pool;
and opening the permission of processing the lookup type message in the thread execution file of all threads in the thread pool.
Further, the method further comprises:
and when all the messages in the lookup message processing thread pool are processed, the thread pool enters a dormant state.
The beneficial effect of the invention is that,
according to the multithreading message processing method based on the lookup operation, provided by the invention, the lookup messages which are intensively sent to MDS are actively transferred to the corresponding thread pool in the message queue, and the lookup processing operation is executed at the same time, so that the overall message processing time of the message queue is greatly saved; meanwhile, before the lookup processing operation is carried out, the message queue detection is actively carried out, and the high time delay caused by the lookup message which is intensively sent to the MDS in a short time is reduced and avoided, so that the purpose of improving the performance stability of the upper application is achieved, the adverse effect on normal service caused by performance shifting is avoided, and the performance stability of the distributed storage system is enhanced.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The following explains key terms appearing in the present invention.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention.
As shown in fig. 1, the method 100 includes:
and step 140, averagely distributing the lookup type messages in the message queue to be processed to the threads of the thread pool, wherein a plurality of threads in the thread pool process the lookup type messages at the same time.
Optionally, as an embodiment of the present invention, the traversing the message queue and screening the message queue to be processed according to the lowest throughput includes:
configuring the Mth message after the current processing message as the starting position;
and traversing the current message queue from the initial position, and taking the message with the message quantity continuously belonging to the lookup type reaching the minimum processing quantity as a message queue to be processed.
Optionally, as an embodiment of the present invention, the method further includes:
inserting a self-defined type message at the starting point of the message queue to be processed;
monitoring the processing state of the custom type message: when the self-defined type message is processed, the original message processing thread is suspended, and a plurality of threads in the thread pool process the lookup type message at the same time.
Optionally, as an embodiment of the present invention, the method further includes:
and adding a new thread, wherein the new thread is used for processing the custom type message.
Optionally, as an embodiment of the present invention, the method further includes:
and after the processing of each thread is finished, independently returning a processing result to the upper layer module.
Optionally, as an embodiment of the present invention, adding a lookup message processing thread pool in the message queue includes:
adding a thread pool, wherein the thread pool is used for processing the lookup type message in the message queue;
setting the default dormant state of the thread pool;
and opening the permission of processing the lookup type message in the thread execution file of all threads in the thread pool.
Optionally, as an embodiment of the present invention, the method further includes:
and when all the messages in the lookup message processing thread pool are processed, the thread pool enters a dormant state.
In order to facilitate understanding of the present invention, the following further describes a multithreading message processing method based on lookup operation, which is provided by the present invention, by using the principle of the multithreading lookup message processing method of the present invention and combining with the distributed process in the embodiment.
Specifically, the multithreading message processing method based on lookup operation includes:
s1, adding a thread pool in the MDS as a lookup message processing thread pool, wherein all threads in the thread pool can execute lookup operation of files, all threads are in a dormant state by default, and the minimum processing amount of the thread pool is set to be N;
s2, adding a new thread, configuring the Mth message after the current processing message as the initial position, traversing the current message queue from the initial position, and taking the N messages as a message queue to be processed if the current message queue traverses the N messages continuously belonging to the lookup type; if no N consecutive lookup-type messages are found in step S2, proceed to step S2 again while waiting for the next message in the message queue to be processed; the embodiment provides a specific method for actively ascertaining the lookup type message by the message queue, which automatically extracts the lookup type message according to the processing capacity of multiple threads, so that the overall speed of processing the lookup type message by the MDS is increased, and the time delay fluctuation of an upper layer application can be further reduced.
S3, inserting a self-defined type message at the beginning of the message queue to be processed; monitoring the processing state of the custom type message: when the self-defined type message is processed by the new thread, suspending the original message processing thread, and starting the lookup processing operation by the thread pool;
s4, independently returning a processing result to the upper layer module after the processing of each thread is finished;
returning a processing result to the upper layer module when one lookup message MDS is processed; the feedback to all threads is convenient, and the maintenance is convenient for users.
And S5, when all the messages in the lookup message processing thread pool are processed, the thread pool enters a dormant state.
Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (7)
1. A multithreading message processing method based on lookup operation is characterized by comprising the following steps:
acquiring a message queue of a distributed file system which needs to perform lookup processing operation;
adding a lookup message processing thread pool in the message queue, and setting the lowest processing amount of the thread pool for processing messages;
traversing the message queue, and screening out a message queue to be processed according to the lowest processing amount;
and averagely distributing the lookup type messages in the message queue to be processed to the threads of the thread pool, and simultaneously processing the lookup type messages by a plurality of threads in the thread pool.
2. The method of claim 1, wherein traversing the message queue and screening the message queue to be processed according to the lowest throughput comprises:
configuring the Mth message after the current processing message as the starting position;
and traversing the current message queue from the initial position, and taking the message with the message quantity continuously belonging to the lookup type reaching the minimum processing quantity as a message queue to be processed.
3. The method of claim 1, wherein the method further comprises:
inserting a self-defined type message at the starting point of the message queue to be processed;
monitoring the processing state of the custom type message: when the self-defined type message is processed, the original message processing thread is suspended, and a plurality of threads in the thread pool process the lookup type message at the same time.
4. The method of claim 3, wherein the method further comprises:
and adding a new thread, wherein the new thread is used for processing the custom type message.
5. The method of claim 1, wherein the method further comprises:
and after the processing of each thread is finished, independently returning a processing result to the upper layer module.
6. The method for multithreading message processing based on lookup operation as claimed in claim 1, wherein adding a lookup message processing thread pool in the message queue comprises:
adding a thread pool, wherein the thread pool is used for processing the lookup type message in the message queue;
setting the default dormant state of the thread pool;
and opening the permission of processing the lookup type message in the thread execution file of all threads in the thread pool.
7. The method of claim 1, wherein the method further comprises:
and when all the messages in the lookup message processing thread pool are processed, the thread pool enters a dormant state.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010469760.2A CN111737026A (en) | 2020-05-28 | 2020-05-28 | Multithreading message processing method based on lookup operation |
| PCT/CN2021/073445 WO2021238261A1 (en) | 2020-05-28 | 2021-01-23 | Multi-thread message processing method based on lookup operation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010469760.2A CN111737026A (en) | 2020-05-28 | 2020-05-28 | Multithreading message processing method based on lookup operation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111737026A true CN111737026A (en) | 2020-10-02 |
Family
ID=72648153
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010469760.2A Pending CN111737026A (en) | 2020-05-28 | 2020-05-28 | Multithreading message processing method based on lookup operation |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111737026A (en) |
| WO (1) | WO2021238261A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021238261A1 (en) * | 2020-05-28 | 2021-12-02 | 苏州浪潮智能科技有限公司 | Multi-thread message processing method based on lookup operation |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106802826A (en) * | 2016-12-23 | 2017-06-06 | 中国银联股份有限公司 | A kind of method for processing business and device based on thread pool |
| CN107370667A (en) * | 2017-07-31 | 2017-11-21 | 北京北信源软件股份有限公司 | Multi-threading parallel process method and apparatus, computer-readable recording medium and storage control |
| CN109753354A (en) * | 2018-11-26 | 2019-05-14 | 平安科技（深圳）有限公司 | Processing method, device and the computer equipment of Streaming Media task based on multithreading |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100962531B1 (en) * | 2007-12-11 | 2010-06-15 | 한국전자통신연구원 | Apparatus for processing multi-threading framework supporting dynamic load-balancing and multi-thread processing method using by it |
| CN105471630B (en) * | 2015-11-18 | 2019-04-19 | 武汉众邦领创技术有限公司 | North orientation system message dissemination system and method under large capacity packet delivery system |
| CN111737026A (en) * | 2020-05-28 | 2020-10-02 | 苏州浪潮智能科技有限公司 | Multithreading message processing method based on lookup operation |
-
2020
- 2020-05-28 CN CN202010469760.2A patent/CN111737026A/en active Pending
-
2021
- 2021-01-23 WO PCT/CN2021/073445 patent/WO2021238261A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106802826A (en) * | 2016-12-23 | 2017-06-06 | 中国银联股份有限公司 | A kind of method for processing business and device based on thread pool |
| CN107370667A (en) * | 2017-07-31 | 2017-11-21 | 北京北信源软件股份有限公司 | Multi-threading parallel process method and apparatus, computer-readable recording medium and storage control |
| CN109753354A (en) * | 2018-11-26 | 2019-05-14 | 平安科技（深圳）有限公司 | Processing method, device and the computer equipment of Streaming Media task based on multithreading |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021238261A1 (en) * | 2020-05-28 | 2021-12-02 | 苏州浪潮智能科技有限公司 | Multi-thread message processing method based on lookup operation |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021238261A1 (en) | 2021-12-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| CN108132837B (en) | Distributed cluster scheduling system and method | |
| CN108574645B (en) | Queue scheduling method and device | |
| CN115373835A (en) | Task resource adjusting method and device for Flink cluster and electronic equipment | |
| CN113157411B (en) | Celery-based reliable configurable task system and device | |
| CN115150471A (en) | Data processing method, device, equipment, storage medium and program product | |
| CN113014608B (en) | Flow distribution control method and device, electronic equipment and storage medium | |
| CN111737026A (en) | Multithreading message processing method based on lookup operation | |
| CN113778644A (en) | Task processing method, device, equipment and storage medium | |
| CN114697398B (en) | Data processing method, device, electronic equipment, storage medium and product | |
| CN112817687A (en) | Data synchronization method and device | |
| CN110795239A (en) | Application memory leakage detection method and device | |
| CN115658248A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN110912949A (en) | Method and device for submitting sites | |
| CN112115123B (en) | Method and apparatus for performance optimization of distributed databases | |
| CN115391361A (en) | Real-time data processing method and device based on distributed database | |
| CN114416717A (en) | Data processing method and architecture | |
| CN114756468A (en) | Test data creating method, device, equipment and storage medium | |
| CN109739883B (en) | Method and device for improving data query performance and electronic equipment | |
| CN111913996A (en) | Data processing method, device, equipment and storage medium | |
| CN114417216B (en) | Data acquisition method and device, electronic equipment and readable storage medium | |
| CN115203139A (en) | Log query method and device, electronic equipment and storage medium | |
| CN117633116A (en) | Data synchronization method, device, electronic equipment and storage medium | |
| CN112307099A (en) | Spark-based method for processing mass data in real time | |
| CN118113766A (en) | Batch data processing method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201002 |
|
| RJ01 | Rejection of invention patent application after publication |