CN112631806B - Asynchronous message arrangement and scheduling method and device, electronic equipment and storage medium - Google Patents
Asynchronous message arrangement and scheduling method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112631806B CN112631806B CN202011587499.2A CN202011587499A CN112631806B CN 112631806 B CN112631806 B CN 112631806B CN 202011587499 A CN202011587499 A CN 202011587499A CN 112631806 B CN112631806 B CN 112631806B
- Authority
- CN
- China
- Prior art keywords
- message
- messages
- primary
- original
- standard
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003860 storage Methods 0.000 title claims abstract description 53
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000012545 processing Methods 0.000 claims description 42
- 238000004806 packaging method and process Methods 0.000 claims description 28
- 230000008521 reorganization Effects 0.000 claims description 12
- 230000003111 delayed effect Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 238000005520 cutting process Methods 0.000 claims description 5
- 238000009826 distribution Methods 0.000 claims description 5
- 238000012163 sequencing technique Methods 0.000 claims description 5
- 238000012856 packing Methods 0.000 abstract description 3
- 238000005516 engineering process Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000001364 causal effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 125000001475 halogen functional group Chemical group 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a cloud storage technology, and discloses an asynchronous message arrangement and scheduling method, which comprises the following steps: obtaining an original message, arranging the original message to obtain a primary message queue, grouping and packing the primary message queue, searching a database to obtain a primary standard message, judging whether the logic relationship of the primary standard message is legal, arranging the primary standard message to obtain a secondary message queue when the logic relationship of the primary standard message is illegal, grouping and packing the secondary message queue, searching the database to obtain a secondary standard message, and logically scheduling the secondary standard message. Furthermore, the present invention relates to blockchain techniques, wherein the original message is available from a node in the blockchain. The invention also provides an asynchronous message arrangement and scheduling device, electronic equipment and a computer readable storage medium. The invention can solve the problem of disordered message scheduling sequence.
Description
Technical Field
The present invention relates to the field of cloud storage technologies, and in particular, to an asynchronous message arrangement and scheduling method, an apparatus, an electronic device, and a computer readable storage medium.
Background
The intelligent scheduling platform obtains service operation message flow of each service center by accessing each service center, and realizes real-time monitoring and scheduling of task queues of each post of the flow service. In the prior art, by arranging the service operation message flow of the service system, sequential consumption is set for messages of all posts of the same flow, so that real-time monitoring and scheduling are realized. For example, a service can create, grab and submit three message messages, and the three message messages are required to be consumed in sequence. However, with a plurality of accessed systems, the operation scene of the front end is complex, and some systems can send a plurality of operation messages of the same service in a short time, so that the sequence of message messages is disordered after the message messages reach the intelligent scheduling platform, the former message is not consumed, and the latter message begins to be consumed, thereby influencing real-time monitoring and scheduling.
Disclosure of Invention
The invention provides an asynchronous message arrangement and scheduling method, an asynchronous message arrangement and scheduling device and a computer readable storage medium, and mainly aims to solve the problem of disordered message scheduling sequence.
In order to achieve the above object, the present invention provides an asynchronous message arranging and scheduling method, including:
Acquiring an original message by using a preset distributed system, and storing the original message to a preset database and a storage system;
Arranging original messages in the storage system to form a first-level message queue;
Grouping and packaging the primary message queues, and searching the database according to the packaged message queues to obtain primary standard messages;
Judging whether the logic relationship of the primary standard message is legal, when the logic relationship of the primary standard message is legal, carrying out logic scheduling on the primary standard message, and when the logic relationship of the primary standard message is illegal, arranging the primary standard message to obtain a secondary message queue;
And grouping and packaging the secondary message queues, searching the database according to the packaged message queues to obtain secondary standard messages, and carrying out logic scheduling on the secondary standard messages.
Optionally, the obtaining the original message by using a preset distributed system and saving the original message to a preset database and a storage system includes:
Subscribing the original information of each service end in real time by using the distributed system;
Storing the subscribed original message into a preset database, and storing the message serial number, the message ID and the message expiration time in the original message into the storage system.
Optionally, the arranging the original messages in the storage system to form a first-level message queue includes:
Carrying out hash processing on the message serial number of the original message to obtain a hash value;
summarizing original messages with the same hash value to obtain task messages;
Sequencing the task messages according to the message deadline of the original message to obtain a task queue;
And executing message distribution on the task queue according to a preset message middleware, and performing decoupling, asynchronous message and flow cutting operation on the task queue to obtain the primary message queue.
Optionally, the grouping and packaging the primary message queue includes:
Performing delay processing on task messages in the primary message queue by using a preset first delay time;
and packaging and grouping the delayed task messages to obtain a plurality of work packages.
Optionally, the searching the database according to the packed message queue to obtain the primary standard message includes:
delivering the work package to a preset thread pool by using a preset reorganization executor;
And extracting the message ID of the task message in the work packet by using the thread in the thread pool, extracting the message with the same message ID in the database according to the message ID, and taking the message as a primary standard message.
Optionally, the arranging the primary standard message to obtain a secondary message queue includes:
Performing delay processing on the primary standard message according to a preset second delay time;
And arranging the primary standard messages according to the message deadline and the message serial number in the primary standard messages to obtain the secondary message queue.
Optionally, the grouping and packaging the secondary message queue, and searching the database according to the packaged message queue to obtain a secondary standard message, including:
packaging the primary standard messages in the secondary message queue to obtain a plurality of message work packages;
delivering the message work package to a preset thread pool by using a preset reorganization executor;
And extracting the message ID of the primary standard message in the message work packet by using the threads in the thread pool, and extracting the message with the same message ID in the database according to the message ID to obtain the secondary standard message.
In order to solve the above problems, the present invention further provides an asynchronous message arranging and scheduling apparatus, which includes:
The message storage module is used for acquiring an original message by using a preset distributed system and storing the original message to a preset database and a storage system;
the primary message arrangement module is used for arranging the original messages in the storage system to form a primary message queue;
the primary message query module is used for grouping and packaging the primary message queues, searching the database according to the packaged message queues and obtaining primary standard messages;
the second-level message arrangement module is used for judging whether the logic relationship of the first-level standard message is legal, carrying out logic scheduling on the first-level standard message when the logic relationship of the first-level standard message is legal, and arranging the first-level standard message when the logic relationship of the first-level standard message is illegal to obtain a second-level message queue;
And the secondary message query module is used for grouping and packaging the secondary message queues, searching the database according to the packaged message queues to obtain secondary standard messages, and carrying out logic scheduling on the secondary standard messages.
In order to solve the above-mentioned problems, the present invention also provides an electronic apparatus including:
a memory storing at least one instruction; and
And the processor executes the instructions stored in the memory to realize the asynchronous message arrangement and scheduling method.
In order to solve the above problems, the present invention also provides a computer readable storage medium having at least one instruction stored therein, the at least one instruction being executed by a processor in an electronic device to implement the above-described asynchronous message arrangement and scheduling method.
According to the invention, the primary message queue is obtained by arranging the original messages, the primary message queue is packed in a grouping way to obtain the primary standard message, when the logical relation of the primary standard message is illegal, the primary standard message is arranged to obtain the secondary message queue, the secondary message queue is packed in a grouping way to obtain the secondary standard message, and the secondary standard message is logically scheduled. The invention carries out two-stage message queue through time sequential message, and simultaneously judges whether the logic relationship of the first-stage standard message is legal or not, and also considers the logic relationship between the messages, so that the message scheduling is more accurate. Therefore, the asynchronous message arrangement and scheduling method, the asynchronous message arrangement and scheduling device, the electronic equipment and the computer readable storage medium can solve the problem of disordered message scheduling sequence.
Drawings
FIG. 1 is a flow chart of an asynchronous message arrangement and scheduling method according to an embodiment of the present invention;
FIG. 2 is a detailed flow chart of one of the steps shown in FIG. 1;
FIG. 3 is a detailed flow chart of another step of FIG. 1;
FIG. 4 is a detailed flow chart of another step of FIG. 1;
FIG. 5 is a detailed flow chart of another step of FIG. 1;
FIG. 6 is a functional block diagram of an asynchronous message arrangement and scheduling apparatus according to an embodiment of the present invention;
Fig. 7 is a schematic structural diagram of an electronic device implementing the method for arranging and scheduling asynchronous messages according to an embodiment of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The embodiment of the application provides an asynchronous message arrangement and scheduling method. The execution body of the asynchronous message arrangement and scheduling method includes, but is not limited to, at least one of a server, a terminal, and the like, which can be configured to execute the method provided by the embodiment of the application. In other words, the asynchronous message arranging and scheduling method may be performed by software or hardware installed in a terminal device or a server device, and the software may be a blockchain platform. The service end includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.
Referring to fig. 1, a flow chart of an asynchronous message arranging and scheduling method according to an embodiment of the invention is shown. In this embodiment, the method for arranging and scheduling asynchronous messages includes:
S1, acquiring an original message by using a preset distributed system, and storing the original message to a preset database and a storage system.
In the embodiment of the invention, the original message can be a business operation flow message of business ends such as branch institutions, mobile ends, enterprise direct connection ends and the like in various fields. The original message comprises information such as a message serial number, a message ID, a message expiration time, a message content and the like.
Specifically, referring to fig. 2, the S1 includes:
s10, subscribing original messages of all service ends in real time by utilizing the distributed system;
S11, storing the subscribed original message into a preset database, and storing a message serial number, a message ID and a message expiration time in the original message into the storage system.
The distributed system may be a Halo framework, SPringBOOT framework, etc., and the embodiment of the present invention acquires the original message from the service end in real time by using the Halo framework, etc., and stores the original message in a database. The storage system may be a non-relational database such as Redis (Remote Dictionary Server, remote dictionary service) and SSDB. In the embodiment of the invention, the Redis is a key-value storage system, and supports more value types including string (character string), list (linked list), set (set), hash (hash type) and the like. The preset database may be a relational database, including Oracle, SQLServer, etc.
Further, the embodiment of the invention can acquire the original message in real time through the distributed system, thereby ensuring the timeliness and the accuracy of the original message.
S2, arranging the original messages in the storage system to form a first-level message queue.
In an embodiment of the present invention, referring to fig. 3, the S2 includes:
s20, carrying out hash processing on the message serial number of the original message to obtain a hash value;
S21, summarizing original messages with the same hash value to obtain task messages;
s22, sequencing the task messages according to the message deadline of the original message to obtain a task queue;
s23, executing message distribution on the task queue according to a preset message middleware, and performing application decoupling, asynchronous message and flow cutting operation to obtain the primary message queue.
In the embodiment of the invention, the original messages of the same message serial number are summarized into the task message, so that the problem of disordered sequence of the same service serial number caused by different sending time is effectively avoided.
The Hash processing is to convert a message serial number with any length into a Hash value with a fixed length, the Hash processing is compression processing, occupation of storage space can be reduced, and meanwhile, the Hash processing is a one-way algorithm and is not easy to crack, so that safety is higher. The preset message middleware can be the currently disclosed Kafka middleware, the Kafka is an open source stream processing platform, is written by Java and Scala, and is a high-throughput distributed message system which can process all information stream data in the task queue. The primary message queue refers to MQ (Message Queue) information, which adopts a data structure of first-in first-out, places data (original message) to be transmitted in a queue, and uses a queue mechanism to realize message transmission. The term "first-in first-out" means that the information arranged in front among the queue information is preprocessed. In the embodiment of the invention, the application decoupling means that when a message is correspondingly processed by a plurality of systems or modules, the coupling between the systems or modules can be reduced by the Kafka middleware. The asynchronous message refers to the asynchronous processing of computer multithreading, and does not block the threads of the current system or module to wait for processing to complete, but allows subsequent systems or modules to operate. The flow peak clipping refers to that under the condition of high concurrency, the Kafka middleware uses MQ information as a buffer, disperses the information in the MQ information, processes the information in a period of time, and provides peak service processing capacity.
Further, in the embodiment of the invention, the primary message queue is formed by arranging the original messages, so that the original messages are processed according to the queue, and the accuracy of asynchronous message processing is improved.
S3, grouping and packaging the primary message queues, and searching the database according to the packaged message queues to obtain primary standard messages.
In the embodiment of the present invention, referring to fig. 4, the step of grouping and packaging the primary message queue includes:
s30, carrying out delay processing on the task message in the primary message queue by utilizing a preset first delay time;
S31, packaging and grouping the delayed task messages to obtain a plurality of work packages.
The preset first delay time may be 10s, because the original message of the primary message queue is from an MQ cluster, the original message received through the MQ cluster is asynchronous, and may be delayed to cause the original message not to arrive according to the time sequence of production, so that obtaining the original message with disordered sequence is avoided, the original message of the last 10 seconds is not read, which is equivalent to delaying for 10s and then processing.
In detail, the searching the database according to the packed message queue to obtain the primary standard message includes:
delivering the work package to a preset thread pool by using a preset reorganization executor;
And extracting the message ID of the task message in the work packet by using the thread in the thread pool, extracting the message with the same message ID in the database according to the message ID, and taking the message as a primary standard message.
The preset reorganization executor may be a Batch job (Batch job), and the reorganization executor is configured to call a thread in a thread pool to process the work package.
Further, in the embodiment of the invention, the primary standard message is obtained by grouping and packaging the primary message queues and searching the database according to the packaged primary message queues, so that the accuracy of message processing is improved.
S4, judging whether the logic relationship of the first-level standard message is legal or not.
In the embodiment of the invention, the logical relationship refers to a causal relationship, for example, the message A consumes according to the order of creation, grabbing and submitting, that is, the message A is necessarily created in advance and then grabbed by a teller, and the grabbing is performed and then the submitting is completed, and when the order is correct and complete, the logical relationship is legal.
And when the logic relation of the primary standard message is legal, executing S5 to carry out logic scheduling on the primary standard message.
The logic scheduling refers to sequential consumption according to causal relation, so that scheduling of the message is completed.
And when the logic relation of the primary standard message is illegal, executing S6, and arranging the primary standard message to obtain a secondary message queue.
In detail, referring to fig. 5, the S6 includes:
S60, carrying out delay processing on the primary standard message according to a preset second delay time;
S61, the primary standard messages are arranged according to the message deadline and the message serial number in the primary standard messages, and the secondary message queue is obtained.
Wherein the preset second delay time may be 30s. In the embodiment of the invention, the first-level standard message with illegal logic relationship (namely incorrect or incomplete causal relationship sequence) is subjected to delay processing, so that a more complete message is obtained.
S7, grouping and packaging the secondary message queues, searching the database according to the packaged message queues to obtain secondary standard messages, and carrying out logic scheduling on the secondary standard messages.
In detail, the step of grouping and packaging the secondary message queues, and searching the database according to the packaged message queues to obtain secondary standard messages comprises the following steps:
packaging the primary standard messages in the secondary message queue to obtain a plurality of message work packages;
delivering the message work package to a preset thread pool by using a preset reorganization executor;
And extracting the message ID of the primary standard message in the message work packet by using the threads in the thread pool, and extracting the message with the same message ID in the database according to the message ID to obtain the secondary standard message.
In the embodiment of the invention, the secondary message queue can delay for a longer time, so that more complete messages are obtained, and the dispatching of the messages is more accurate.
According to the invention, the primary message queue is obtained by arranging the original messages, the primary message queue is packed in a grouping way to obtain the primary standard message, when the logical relation of the primary standard message is illegal, the primary standard message is arranged to obtain the secondary message queue, the secondary message queue is packed in a grouping way to obtain the secondary standard message, and the secondary standard message is logically scheduled. The invention carries out two-stage message queue through time sequential message, and simultaneously judges whether the logic relationship of the first-stage standard message is legal or not, and also considers the logic relationship between the messages, so that the message scheduling is more accurate. Therefore, the invention can solve the problem of disordered message scheduling sequence. .
Fig. 6 is a functional block diagram of an asynchronous message arrangement and scheduling apparatus according to an embodiment of the present invention.
The asynchronous message arrangement and scheduling apparatus 100 of the present invention may be installed in an electronic device. The asynchronous message arrangement and scheduling apparatus 100 may include a message storage module 101, a primary message arrangement module 102, a primary message query module 103, a secondary message arrangement module 104, and a secondary message query module 105, depending on the functions implemented. The module of the invention, which may also be referred to as a unit, refers to a series of computer program segments, which are stored in the memory of the electronic device, capable of being executed by the processor of the electronic device and of performing a fixed function.
In the present embodiment, the functions concerning the respective modules/units are as follows:
the message storage module 101 is configured to obtain an original message by using a preset distributed system, and store the original message in a preset database and a storage system.
In the embodiment of the invention, the original message can be a business operation flow message of business ends such as branch institutions, mobile ends, enterprise direct connection ends and the like in various fields. The original message comprises information such as a message serial number, a message ID, a message expiration time, a message content and the like.
Specifically, the message storage module 101 stores the original message in a preset database and storage system by:
Subscribing the original information of each service end in real time by using the distributed system;
Storing the subscribed original message into a preset database, and storing the message serial number, the message ID and the message expiration time in the original message into the storage system.
The distributed system may be a Halo framework, SPringBOOT framework, etc., and the embodiment of the present invention acquires the original message from the service end in real time by using the Halo framework, etc., and stores the original message in a database. The storage system may be a non-relational database such as Redis (Remote Dictionary Server, remote dictionary service) and SSDB. In the embodiment of the invention, the Redis is a key-value storage system, and supports more value types including string (character string), list (linked list), set (set), hash (hash type) and the like. The preset database may be a relational database, including Oracle, SQLServer, etc.
Further, the embodiment of the invention can acquire the original message in real time through the distributed system, thereby ensuring the timeliness and the accuracy of the original message.
The primary message queuing module 102 is configured to queue original messages in the storage system to form a primary message queue.
In the embodiment of the present invention, the coordinate calculation module 102 obtains the first-level message queue by:
Carrying out hash processing on the message serial number of the original message to obtain a hash value;
summarizing original messages with the same hash value to obtain task messages;
Sequencing the task messages according to the message deadline of the original message to obtain a task queue;
And executing message distribution on the task queue according to a preset message middleware, and performing decoupling, asynchronous message and flow cutting operation on the task queue to obtain the primary message queue.
In the embodiment of the invention, the original messages of the same message serial number are summarized into the task message, so that the problem of disordered sequence of the same service serial number caused by different sending time is effectively avoided.
The Hash processing is to convert a message serial number with any length into a Hash value with a fixed length, the Hash processing is compression processing, occupation of storage space can be reduced, and meanwhile, the Hash processing is a one-way algorithm and is not easy to crack, so that safety is higher. The preset message middleware can be the currently disclosed Kafka middleware, the Kafka is an open source stream processing platform, is written by Java and Scala, and is a high-throughput distributed message system which can process all information stream data in the task queue. The primary message queue refers to MQ (Message Queue) information, which adopts a data structure of first-in first-out, places data (original message) to be transmitted in a queue, and uses a queue mechanism to realize message transmission. The term "first-in first-out" means that the information arranged in front among the queue information is preprocessed. In the embodiment of the invention, the application decoupling means that when a message is correspondingly processed by a plurality of systems or modules, the coupling between the systems or modules can be reduced by the Kafka middleware. The asynchronous message refers to the asynchronous processing of computer multithreading, and does not block the threads of the current system or module to wait for processing to complete, but allows subsequent systems or modules to operate. The flow peak clipping refers to that under the condition of high concurrency, the Kafka middleware uses MQ information as a buffer, disperses the information in the MQ information, processes the information in a period of time, and provides peak service processing capacity.
Further, in the embodiment of the invention, the primary message queue is formed by arranging the original messages, so that the original messages are processed according to the queue, and the accuracy of asynchronous message processing is improved.
The primary message query module 103 is configured to package the primary message queue in a grouping manner, and search the database according to the packaged message queue to obtain a primary standard message.
Preferably, the primary message query module 103 performs packet packing on the primary message queue by:
Performing delay processing on task messages in the primary message queue by using a preset first delay time;
and packaging and grouping the delayed task messages to obtain a plurality of work packages.
The preset first delay time may be 10s, because the original message of the primary message queue is from an MQ cluster, the original message received through the MQ cluster is asynchronous, and may be delayed to cause the original message not to arrive according to the time sequence of production, so that obtaining the original message with disordered sequence is avoided, the original message of the last 10 seconds is not read, which is equivalent to delaying for 10s and then processing.
In detail, the primary message query module 103 obtains a primary standard message by:
delivering the work package to a preset thread pool by using a preset reorganization executor;
And extracting the message ID of the task message in the work packet by using the thread in the thread pool, extracting the message with the same message ID in the database according to the message ID, and taking the message as a primary standard message.
The preset reorganization executor may be a Batch job (Batch job), and the reorganization executor is configured to call a thread in a thread pool to process the work package.
Further, in the embodiment of the invention, the primary standard message is obtained by grouping and packaging the primary message queues and searching the database according to the packaged primary message queues, so that the accuracy of message processing is improved.
The secondary message arrangement module 104 is configured to determine whether the logical relationship of the primary standard message is legal, perform logical scheduling on the primary standard message when the logical relationship of the primary standard message is legal, and arrange the primary standard message when the logical relationship of the primary standard message is illegal, so as to obtain a secondary message queue.
In the embodiment of the invention, the logical relationship refers to a causal relationship, for example, the message A consumes according to the order of creation, grabbing and submitting, that is, the message A is necessarily created in advance and then grabbed by a teller, and the grabbing is performed and then the submitting is completed, and when the order is correct and complete, the logical relationship is legal.
And when the logic relation of the primary standard message is legal, carrying out logic scheduling on the primary standard message.
The logic scheduling refers to sequential consumption according to causal relation, so that scheduling of the message is completed.
And when the logic relation of the primary standard message is illegal, arranging the primary standard message to obtain a secondary message queue.
In detail, the secondary message queuing module 104 obtains a secondary message queue by:
Performing delay processing on the primary standard message according to a preset second delay time;
And arranging the primary standard messages according to the message deadline and the message serial number in the primary standard messages to obtain the secondary message queue.
Wherein the preset second delay time may be 30s. In the embodiment of the invention, the first-level standard message with illegal logic relationship (namely incorrect or incomplete causal relationship sequence) is subjected to delay processing, so that a more complete message is obtained.
The secondary message query module 105 is configured to package the secondary message queues in a grouping manner, search the database according to the packaged message queues, obtain secondary standard messages, and logically schedule the secondary standard messages.
In detail, the secondary message query module 105 obtains a secondary standard message by:
packaging the primary standard messages in the secondary message queue to obtain a plurality of message work packages;
delivering the message work package to a preset thread pool by using a preset reorganization executor;
And extracting the message ID of the primary standard message in the message work packet by using the threads in the thread pool, and extracting the message with the same message ID in the database according to the message ID to obtain the secondary standard message.
In the embodiment of the invention, the secondary message queue can delay for a longer time, so that more complete messages are obtained, and the dispatching of the messages is more accurate.
Fig. 7 is a schematic structural diagram of an electronic device for implementing an asynchronous message arrangement and scheduling method according to an embodiment of the present invention.
The electronic device 1 may comprise a processor 10, a memory 11 and a bus, and may further comprise a computer program, such as an asynchronous message arrangement and scheduling program 12, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may in other embodiments also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only for storing application software installed in the electronic device 1 and various types of data, such as codes of the asynchronous message arrangement and scheduling program 12, but also for temporarily storing data that has been output or is to be output.
The processor 10 may be comprised of integrated circuits in some embodiments, for example, a single packaged integrated circuit, or may be comprised of multiple integrated circuits packaged with the same or different functions, including one or more central processing units (Central Processing unit, CPU), microprocessors, digital processing chips, graphics processors, combinations of various control chips, and the like. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the entire electronic device using various interfaces and lines, and executes various functions of the electronic device 1 and processes data by running or executing programs or modules (e.g., asynchronous message arrangement and scheduling programs, etc.) stored in the memory 11, and calling data stored in the memory 11.
The bus may be a peripheral component interconnect standard (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus, or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The bus may be classified as an address bus, a data bus, a control bus, etc. The bus is arranged to enable a connection communication between the memory 11 and at least one processor 10 etc.
Fig. 7 shows only an electronic device with components, it being understood by a person skilled in the art that the structure shown in fig. 7 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than shown, or may combine certain components, or may be arranged in different components.
For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for supplying power to each component, and preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that functions of charge management, discharge management, power consumption management, and the like are implemented through the power management device. The power supply may also include one or more of any of a direct current or alternating current power supply, recharging device, power failure detection circuit, power converter or inverter, power status indicator, etc. The electronic device 1 may further include various sensors, bluetooth modules, wi-Fi modules, etc., which will not be described herein.
Further, the electronic device 1 may also comprise a network interface, optionally the network interface may comprise a wired interface and/or a wireless interface (e.g. WI-FI interface, bluetooth interface, etc.), typically used for establishing a communication connection between the electronic device 1 and other electronic devices.
The electronic device 1 may optionally further comprise a user interface, which may be a Display, an input unit, such as a Keyboard (Keyboard), or a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like. The display may also be referred to as a display screen or display unit, as appropriate, for displaying information processed in the electronic device 1 and for displaying a visual user interface.
It should be understood that the embodiments described are for illustrative purposes only and are not limited to this configuration in the scope of the patent application.
The asynchronous message arrangement and scheduling program 12 stored in the memory 11 of the electronic device 1 is a combination of instructions which, when executed in the processor 10, can implement:
Acquiring an original message by using a preset distributed system, and storing the original message to a preset database and a storage system;
Arranging original messages in the storage system to form a first-level message queue;
Grouping and packaging the primary message queues, and searching the database according to the packaged message queues to obtain primary standard messages;
Judging whether the logic relationship of the primary standard message is legal, when the logic relationship of the primary standard message is legal, carrying out logic scheduling on the primary standard message, and when the logic relationship of the primary standard message is illegal, arranging the primary standard message to obtain a secondary message queue;
And grouping and packaging the secondary message queues, searching the database according to the packaged message queues to obtain secondary standard messages, and carrying out logic scheduling on the secondary standard messages.
Specifically, the specific implementation method of the above instructions by the processor 10 may refer to descriptions of related steps in the corresponding embodiments of fig. 1 to 5, which are not repeated herein.
Further, the modules/units integrated in the electronic device 1 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as separate products. The computer readable storage medium may be volatile or nonvolatile. For example, the computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM).
The present invention also provides a computer readable storage medium storing a computer program which, when executed by a processor of an electronic device, can implement:
Acquiring an original message by using a preset distributed system, and storing the original message to a preset database and a storage system;
Arranging original messages in the storage system to form a first-level message queue;
Grouping and packaging the primary message queues, and searching the database according to the packaged message queues to obtain primary standard messages;
Judging whether the logic relationship of the primary standard message is legal, when the logic relationship of the primary standard message is legal, carrying out logic scheduling on the primary standard message, and when the logic relationship of the primary standard message is illegal, arranging the primary standard message to obtain a secondary message queue;
And grouping and packaging the secondary message queues, searching the database according to the packaged message queues to obtain secondary standard messages, and carrying out logic scheduling on the secondary standard messages.
In the several embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be other manners of division when actually implemented.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.
The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the system claims can also be implemented by means of software or hardware by means of one unit or means. The terms second, etc. are used to denote a name, but not any particular order.
Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.
Claims (8)
1. An asynchronous message arrangement and scheduling method, the method comprising:
Acquiring an original message by using a preset distributed system, storing the original message into a preset database, and storing a message serial number, a message ID and a message deadline in the original message into a storage system, wherein the preset database is a relational database, and the storage system is a non-relational database;
Arranging original messages in the storage system to form a first-level message queue, wherein the first-level message queue comprises: carrying out hash processing on the message serial numbers of the original messages to obtain hash values, summarizing the original messages with the same hash values to obtain task messages, sequencing the task messages according to the message deadlines of the original messages to obtain task queues, carrying out message distribution on the task queues according to preset message middleware, and applying decoupling, asynchronous message and flow cutting operation to obtain the primary message queues;
The primary message queue is packed in a grouping mode, the preset database is searched according to the packed message queue, and primary standard messages are obtained, wherein the primary message queue is packed in a grouping mode, and the method comprises the following steps: performing delay processing on task messages in the primary message queue by using a preset first delay time, and packaging and grouping the delayed task messages to obtain a plurality of work packages;
Judging whether the logic relationship of the primary standard message is legal, when the logic relationship of the primary standard message is legal, carrying out logic scheduling on the primary standard message, and when the logic relationship of the primary standard message is illegal, arranging the primary standard message to obtain a secondary message queue;
And grouping and packaging the secondary message queues, searching the preset database according to the packaged message queues to obtain secondary standard messages, and carrying out logic scheduling on the secondary standard messages.
2. The method for arranging and scheduling asynchronous messages according to claim 1, wherein said obtaining an original message using a predetermined distributed system and storing said original message in a predetermined database comprises:
Subscribing the original information of each service end in real time by using the distributed system;
And storing the subscribed original message into a preset database.
3. The method for arranging and scheduling asynchronous messages according to claim 1, wherein said searching said predetermined database according to the packed message queue to obtain a primary standard message comprises:
delivering the work package to a preset thread pool by using a preset reorganization executor;
And extracting the message ID of the task message in the work packet by using the thread in the thread pool, extracting the message with the same message ID in the preset database according to the message ID, and taking the message as a primary standard message.
4. An asynchronous message queuing and scheduling method according to any one of claims 1 to 3 wherein said queuing said primary standard message to obtain a secondary message queue comprises:
Performing delay processing on the primary standard message according to a preset second delay time;
And arranging the primary standard messages according to the message deadline and the message serial number in the primary standard messages to obtain the secondary message queue.
5. The method for arranging and scheduling asynchronous messages according to any one of claims 1 to 3, wherein said grouping and packaging the secondary message queues, and searching the preset database according to the packaged message queues, to obtain secondary standard messages, comprises:
packaging the primary standard messages in the secondary message queue to obtain a plurality of message work packages;
delivering the message work package to a preset thread pool by using a preset reorganization executor;
And extracting the message ID of the primary standard message in the message work packet by using the thread in the thread pool, and extracting the message with the same message ID in the preset database according to the message ID to obtain the secondary standard message.
6. An asynchronous message arrangement and scheduling apparatus, the apparatus comprising:
The message storage module is used for acquiring an original message by using a preset distributed system, storing the original message into a preset database, and storing a message serial number, a message ID and a message deadline in the original message into a storage system, wherein the preset database is a relational database, and the storage system is a non-relational database;
A primary message arrangement module, configured to arrange original messages in the storage system to form a primary message queue, including: carrying out hash processing on the message serial numbers of the original messages to obtain hash values, summarizing the original messages with the same hash values to obtain task messages, sequencing the task messages according to the message deadlines of the original messages to obtain task queues, carrying out message distribution on the task queues according to preset message middleware, and applying decoupling, asynchronous message and flow cutting operation to obtain the primary message queues;
The primary message query module is configured to package the primary message queue in a grouping manner, and find the preset database according to the packaged message queue to obtain a primary standard message, where the package of the primary message queue in a grouping manner includes: performing delay processing on task messages in the primary message queue by using a preset first delay time, and packaging and grouping the delayed task messages to obtain a plurality of work packages;
the second-level message arrangement module is used for judging whether the logic relationship of the first-level standard message is legal, carrying out logic scheduling on the first-level standard message when the logic relationship of the first-level standard message is legal, and arranging the first-level standard message when the logic relationship of the first-level standard message is illegal to obtain a second-level message queue;
and the secondary message query module is used for grouping and packaging the secondary message queues, searching the preset database according to the packaged message queues to obtain secondary standard messages, and carrying out logic scheduling on the secondary standard messages.
7. An electronic device, the electronic device comprising:
at least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the asynchronous message arrangement and scheduling method of any one of claims 1 to 5.
8. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the asynchronous message arrangement and scheduling method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011587499.2A CN112631806B (en) | 2020-12-28 | 2020-12-28 | Asynchronous message arrangement and scheduling method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011587499.2A CN112631806B (en) | 2020-12-28 | 2020-12-28 | Asynchronous message arrangement and scheduling method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112631806A CN112631806A (en) | 2021-04-09 |
| CN112631806B true CN112631806B (en) | 2024-04-23 |
Family
ID=75285775
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011587499.2A Active CN112631806B (en) | 2020-12-28 | 2020-12-28 | Asynchronous message arrangement and scheduling method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112631806B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113163009A (en) * | 2021-04-20 | 2021-07-23 | 平安消费金融有限公司 | Data transmission method, device, electronic equipment and storage medium |
| CN113687928B (en) * | 2021-08-03 | 2024-03-22 | 广州华多网络科技有限公司 | Message scheduling control method and corresponding device, equipment and medium thereof |
| CN113704258B (en) * | 2021-08-20 | 2023-12-26 | 辽宁振兴银行股份有限公司 | Method for storing transaction messages based on MongoDB database |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107391271A (en) * | 2017-05-17 | 2017-11-24 | 阿里巴巴集团控股有限公司 | A kind of delayed tasks triggering method and device based on Message Queuing system |
| CN107656825A (en) * | 2017-09-01 | 2018-02-02 | 上海艾融软件股份有限公司 | Message treatment method, apparatus and system |
| CN111045839A (en) * | 2019-12-04 | 2020-04-21 | 中国建设银行股份有限公司 | Sequence calling method and device based on two-phase transaction message in distributed environment |
| CN112084486A (en) * | 2020-09-08 | 2020-12-15 | 中国平安财产保险股份有限公司 | User information verification method and device, electronic equipment and storage medium |
-
2020
- 2020-12-28 CN CN202011587499.2A patent/CN112631806B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107391271A (en) * | 2017-05-17 | 2017-11-24 | 阿里巴巴集团控股有限公司 | A kind of delayed tasks triggering method and device based on Message Queuing system |
| CN107656825A (en) * | 2017-09-01 | 2018-02-02 | 上海艾融软件股份有限公司 | Message treatment method, apparatus and system |
| CN111045839A (en) * | 2019-12-04 | 2020-04-21 | 中国建设银行股份有限公司 | Sequence calling method and device based on two-phase transaction message in distributed environment |
| CN112084486A (en) * | 2020-09-08 | 2020-12-15 | 中国平安财产保险股份有限公司 | User information verification method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112631806A (en) | 2021-04-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112631806B (en) | Asynchronous message arrangement and scheduling method and device, electronic equipment and storage medium | |
| CN111651471B (en) | Target data query method and device, electronic equipment and storage medium | |
| US20230020324A1 (en) | Task Processing Method and Device, and Electronic Device | |
| CN112015815B (en) | Data synchronization method, device and computer readable storage medium | |
| CN110928905B (en) | Data processing method and device | |
| CN113163009A (en) | Data transmission method, device, electronic equipment and storage medium | |
| CN112256783A (en) | Data export method and device, electronic equipment and storage medium | |
| CN116755637B (en) | Transaction data storage method, device, equipment and medium | |
| CN111371848A (en) | Request processing method, device, equipment and storage medium | |
| CN111522707A (en) | Big data platform scheduling early warning method and device and computer readable storage medium | |
| CN113010542A (en) | Service data processing method and device, computer equipment and storage medium | |
| CN114697316B (en) | Batch downloading method, device and equipment of data and computer readable medium | |
| CN115269709B (en) | Inquiry and playback method, device, equipment and medium based on market data | |
| WO2023272853A1 (en) | Ai-based sql engine calling method and apparatus, and device and medium | |
| CN113918305B (en) | Node scheduling method, node scheduling device, electronic equipment and readable storage medium | |
| CN109829005A (en) | A kind of big data processing method and processing device | |
| CN110825342A (en) | Memory scheduling device and system, method and apparatus for processing information | |
| CN112328656B (en) | Service query method, device, equipment and storage medium based on middle platform architecture | |
| CN116263717A (en) | Order service processing method and device based on event | |
| CN113778700A (en) | Message processing method, system, medium and computer system | |
| CN111210288A (en) | Tax control server-based invoicing batch invoicing job optimized scheduling method and system | |
| CN115964307B (en) | Automatic test method, device, equipment and medium for transaction data | |
| CN115174698B (en) | Market data decoding method, device, equipment and medium based on table entry index | |
| CN112527842B (en) | System flow pushing method and device, electronic equipment and computer readable storage medium | |
| CN112364062A (en) | Data arrival detection method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |