CN110609841A - Method, device, readable medium and electronic equipment for asynchronously processing messages - Google Patents

Method, device, readable medium and electronic equipment for asynchronously processing messages Download PDF

Info

Publication number
CN110609841A
CN110609841A CN201910880883.2A CN201910880883A CN110609841A CN 110609841 A CN110609841 A CN 110609841A CN 201910880883 A CN201910880883 A CN 201910880883A CN 110609841 A CN110609841 A CN 110609841A
Authority
CN
China
Prior art keywords
time
target
message
nosql
main key
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910880883.2A
Other languages
Chinese (zh)
Other versions
CN110609841B (en
Inventor
李耕勇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mgjia (beijing) Technology Co Ltd
Original Assignee
Mgjia (beijing) Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mgjia (beijing) Technology Co Ltd filed Critical Mgjia (beijing) Technology Co Ltd
Priority to CN201910880883.2A priority Critical patent/CN110609841B/en
Publication of CN110609841A publication Critical patent/CN110609841A/en
Application granted granted Critical
Publication of CN110609841B publication Critical patent/CN110609841B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2282Tablespace storage structures; Management thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing

Abstract

The disclosure relates to a method, a device, a readable medium and an electronic device for asynchronously processing messages. According to the technical scheme, the NoSQL database is adopted, the writing and reading speed of NoSQL data is high compared with that of the MySQL database, and the problem that the NoSQL database cannot be suitable due to no partition key is avoided. Meanwhile, the data reading performance is assisted and optimized by establishing a separate target query set instead of adopting a secondary index established on the NoSQL service table, and because the target query set only comprises a main key and insertion time of the NoSQL service table, redundant data is limited. And when the query is carried out, the main key set with the target query set inserted with time meeting the preset conditions is sequentially queried, so that the problem of no stateness during the query of the NoSQL database is solved, and the query efficiency is improved.

Description

Method, device, readable medium and electronic equipment for asynchronously processing messages
Technical Field
The present disclosure relates to the field of asynchronous processing technologies, and in particular, to a method, an apparatus, a readable medium, and an electronic device for asynchronously processing a message.
Background
In a high concurrency environment, the stress of the system can be relieved by processing messages asynchronously. In the related art, generally, a partition key (partition key) based MySQL database partitioning scheme and a secondary index based NoSQL database data redundancy scheme are adopted to implement asynchronous processing. The MySQL database is a relational database, and the NoSQL database is a non-relational database.
The MySQL database and table partitioning scheme mainly has the following defects: the MySQL database has poor performance and slow data writing and reading speed, the improvement is limited even if the log disk refreshing optimization is set, and meanwhile, more server nodes are required for processing, so that resources are wasted. On the other hand, all queries need to be based on the partition key and are used for confirming the library and the table where the data are stored, and the query conditions do not include the partition key, so that the database and the table where the data are located are unknown, and the query conditions cannot be applied.
The data redundancy scheme of the NoSQL database adopts a NoSQL database, and the NoSQL database has higher data writing and reading speed compared with the MySQL database. The disadvantage is that the introduction of the secondary index greatly compromises the reading performance of the NoSQL database. The query of the NoSQL database has statelessness and weak query function, and some redundant data storage designs are needed for facilitating the query.
Disclosure of Invention
The present disclosure is directed to a method, an apparatus, a readable medium, and an electronic device for asynchronously processing messages based on a NoSQL database, so as to solve the above problems.
In order to achieve the above object, an embodiment of the present disclosure provides a method for asynchronously processing a message, where the method includes:
acquiring a main key set with the insertion time meeting preset conditions in a target query set, wherein the main key and the insertion time of a message are stored in the target query set;
inquiring an NoSQL service table according to each main key in the main key set to obtain a target message, wherein the NoSQL service table stores the main key, a message body, a processing state and processing times of the message, and the target message is a message which has the processing times of not reaching the preset times and the processing state of failure in the message corresponding to each main key in the main key set;
if the target message is obtained, storing a primary key of the target message to the target query set, storing the storage time as the insertion time of the target message to the target query set, and submitting the target message to a task thread pool for reprocessing.
Optionally, the target query set is a NoSQL index table, and obtaining a primary key set in the target query set, where an insertion time of the primary key set meets a preset condition, includes:
acquiring time offset of a target database, wherein the target database is a redis database or a mongodb database;
acquiring a main key set of which the insertion time is within a preset time range and the query quantity is within a preset number in the NoSQL index table, wherein the starting time of the preset time range is the time offset of the target database, and the ending time is the current time minus the processing timeout time;
and updating the time offset of the target database to the latest insertion time in the primary key set.
Optionally, before the step of obtaining the primary key set with an insertion time satisfying a preset condition in the target query set, the method further includes: locks that compete for global;
the step of obtaining the primary key set with the insertion time meeting the preset conditions in the target query set comprises the following steps: after the lock is obtained, acquiring a main key set of which the target query set insertion time meets a preset condition;
after the step of inserting the primary key set with time meeting the preset condition in the target query set, the method further includes: the lock is released.
Optionally, the method further comprises: and deleting the primary key with the insertion time before the time offset of the target database in the NoSQL index table and the insertion time corresponding to the primary key.
Optionally, the target query set is a delay queue, and the delay queue further stores a delay duration of the message;
the acquiring of the primary key set with the insertion time meeting the preset conditions in the target query set comprises: and acquiring a main key set of which the insertion time is equal to the current time minus the delay time length in the delay queue.
Optionally, the method further comprises:
and if the target message is obtained, adding a primary key and retry time of the message into a retry table, wherein the retry time is the time for submitting the target message to a task thread pool.
The embodiment of the present disclosure further provides an apparatus for asynchronously processing a message, including:
the system comprises a main key set acquisition module, a main key set acquisition module and a main key set acquisition module, wherein the main key set acquisition module is used for acquiring a main key set of which the insertion time meets a preset condition in a target query set, and the main key and the insertion time of a message are stored in the target query set;
the query module is used for querying a NoSQL service table according to each main key in the main key set to obtain a target message, wherein the NoSQL service table stores the main key, a message body, a processing state and the processing times of the message, and the target message is a message of which the processing times are not up to the preset times and the processing state is failed in the message corresponding to each main key in the main key set; if the target message is obtained, storing a primary key of the target message to the target query set, storing the storage time as the insertion time of the target message to the target query set, and submitting the target message to a task thread pool for reprocessing.
Optionally, the target query set is a NoSQL index table, and the primary key set obtaining module includes:
the offset acquisition submodule is used for acquiring the time offset of the target database;
a first primary key set obtaining sub-module, configured to obtain a primary key set, in the NoSQL index table, where an insertion time is within a preset time range and a query amount is within a preset number, the target database is a redis database or a mongodb database, an initial time of the preset time range is a time offset of the target database, and an end time is a current time minus a processing timeout time;
and the offset updating submodule is used for updating the time offset of the target database to the latest insertion time in the main key set.
Embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the above-described method.
An embodiment of the present disclosure further provides an electronic device, including:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to implement the steps of the above method.
Through the technical scheme, the NoSQL database is adopted, the writing and reading speed of NoSQL data is high compared with that of the MySQL database, and the problem that the NoSQL database cannot be suitable due to no partition key is avoided. Meanwhile, the data reading performance is assisted and optimized by establishing a separate target query set instead of adopting a secondary index established on the NoSQL service table, and because the target query set only comprises a main key and insertion time of the NoSQL service table, redundant data is limited. And when the query is carried out, the main key set with the insertion time meeting the preset condition in the target query set is sequentially queried, so that the problem of no stateness in the NoSQL database query is solved, and the query efficiency is improved.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure without limiting the disclosure. In the drawings:
fig. 1 is a flowchart of a method for asynchronously processing a message according to an embodiment of the present disclosure.
FIG. 2 is a flowchart illustrating the sub-steps included in step S11 shown in FIG. 1, according to one embodiment.
Fig. 3 is a block diagram illustrating an apparatus for asynchronously processing messages according to an embodiment of the disclosure.
FIG. 4 is a block diagram illustrating an electronic device in accordance with an example embodiment.
Detailed Description
The following detailed description of specific embodiments of the present disclosure is provided in connection with the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present disclosure, are given by way of illustration and explanation only, not limitation.
Referring to fig. 1, a flowchart of a method for asynchronously processing a message according to an embodiment of the present disclosure is shown. As shown in fig. 1, the method for asynchronously processing messages comprises the following steps:
step S11, acquiring a primary key set with the insertion time meeting the preset conditions in a target query set, wherein the primary key and the insertion time of the message are stored in the target query set.
And storing the main key and the insertion time of the message into the target query set when the message is received. In order to avoid or reduce repeated inquiry, meaningless inquiry caused by incomplete execution of tasks during inquiry, program crash caused by excessive memory overflow of the inquiry quantity and the like, the main key set of each inquiry is limited by whether the insertion time meets the preset condition or not.
Step S13, querying a NoSQL service table according to each primary key in the primary key set to obtain a target message, where the NoSQL service table stores the primary key, the message body, the processing status, and the processing frequency of the message, and the target message is a message corresponding to each primary key in the primary key set, where the processing frequency is not up to a preset frequency and the processing status is a failed message.
When a message is received, storing a main key and insertion time of the message into the target query set, storing the main key and a message body of the message into a NoSQL service table of a non-relational database, and submitting the message to a task thread pool. And the task threads in the task thread pool process the messages and increase the processing times in the NoSQL business table for each message, and update the processing state of the messages in the NoSQL business table according to the processing result of the messages, wherein the processing state comprises success or failure. The NoSQL service table stores the main key, the message body, the processing status, and the processing times of the message. And querying the NoSQL service table according to each main key in the main key set, namely querying the processing state and the processing times of the message corresponding to each main key, wherein the processing times are not up to the preset times, and the message with the processing state failed is the target message.
Step S15, if the target message is obtained, storing the primary key of the target message in the target query set, storing the storage time as the insertion time of the target message in the target query set, and submitting the target message to a task thread pool for reprocessing.
If the target message is not obtained, the NoSQL service table is inquired according to each primary key in the primary key set, and the inquired message is processed successfully or the processing state is failed but the processing times reach the preset times, the processing is not carried out. If the target message is obtained, storing the primary key of the target message to the target query set, storing the storage time as the insertion time of the target message to the target query set, and submitting the target message to a task thread pool for reprocessing, thereby realizing reprocessing of the message of which the processing times does not reach the preset times and the processing state is failure, and re-querying the processing state.
Through the technical scheme, the NoSQL database is adopted, the writing and reading speed of NoSQL data is high compared with that of the MySQL database, and the problem that the NoSQL database cannot be suitable due to no partition key is avoided. Meanwhile, the data reading performance is assisted and optimized by establishing a separate target query set instead of adopting a secondary index established on the NoSQL service table, and because the target query set only comprises a main key and insertion time of the NoSQL service table, redundant data is limited. And when the query is carried out, the main key set with the insertion time meeting the preset condition in the target query set is sequentially queried, so that the problem of no stateness in the NoSQL database query is solved, and the query efficiency is improved.
Optionally, in an embodiment, the target query set is a NoSQL index table, as shown in fig. 2, step S11 includes:
and a substep S111 of obtaining a time offset of a target database, wherein the target database is a redis database or a mongodb database.
The redis database and the mongodb database have good performance and high access speed and can store data persistently. Initially, the time offset of the target database is earlier than the earliest insertion time in the target query set.
And a substep S113, obtaining a primary key set with an insertion time within a preset time range and a query quantity within a preset number in the NoSQL index table, where an initial time of the preset time range is a time offset of the target database, and an end time is a current time minus a processing timeout time.
And the processing overtime time is the overtime time for processing the message by the task thread in the task thread pool. The messages corresponding to the primary key with the insertion time within the preset time range are all messages of which the processing is finished, and the processing state is success or failure. Obviously, when the processing state of the message is failure, the message may be processing failure, or processing timeout may be regarded as failure. Therefore, by setting the current time minus the processing timeout time as the termination time, it is possible to avoid meaningless query caused by a task not being completely executed at the time of query. In a high concurrency situation, many messages may be received at the same time, and thus the number of primary keys (messages) within a preset time range may be many. In order to avoid program crash caused by excessive memory overflow of the query quantity, only the primary key sets within the preset number are queried during query. That is, when the number of the primary keys in the primary key set with the insertion time within the preset time range is less than or equal to the preset number, acquiring the primary key set with the insertion time within the preset time range in the NoSQL index table; and when the number of the primary keys in the primary key set with the insertion time within a preset time range is larger than the preset number, acquiring the primary key set from the time offset of the insertion time in the target database in the NoSQL index table to a target time, wherein the number of the primary keys in the primary key set is equal to the preset number, and the target time is earlier than the current time minus processing timeout time.
And a substep S115, updating the time offset of the target database to the latest insertion time in the primary key set.
As seen from sub-step S113, the time offset of the target database is the starting time for querying the NoSQL index table. Then, through sub-step S115, the latest insertion time can be used as the starting time for next query of the NoSQL index table, so as to avoid or reduce repeated queries.
By the technical scheme, when the NoSQL index table is queried, the main key sets with the insertion time within the preset time range and the query quantity within the preset number are sequentially queried, so that the problem of statelessness during NoSQL database query is solved, and the query efficiency is improved. During query, only the primary key sets within the preset number are queried, and program collapse caused by excessive memory overflow of query quantity is avoided. And the termination time of the preset time range is the current time minus the processing overtime, and the current time minus the processing overtime is set as the termination time, so that meaningless query caused by the fact that the task is not completely executed during query can be avoided. And after one query is finished, updating the time offset of the target database to the latest insertion time in the main key set, and avoiding or reducing repeated queries.
In addition, since the message whose processing times do not reach the preset times and whose processing state is failed is not only submitted to the task thread pool for reprocessing, but also its primary key is added to the NoSQL index table again, and the storage time is used as the insertion time of the primary key, the NoSQL index table can also be used for retrying query. That is, the retried message can be queried by querying the repeated primary key in the NoSQL index table, so as to facilitate the retried query, for example, counting the number of retried tasks (messages), and obtain the retried time (i.e., the time inserted in the NoSQL index table).
Optionally, before step S11, the method further comprises: contending for a global lock. Step S11 is to obtain the primary key set whose insertion time in the target query set satisfies the preset condition after obtaining the lock. After step S11, the method further includes: the lock is released.
In practical implementation, if there are multiple threads executing the above asynchronous message processing method, multiple threads may compete for a global lock, obtain a primary key set whose insertion time in the target query set meets a preset condition, release the lock, and then execute step S13. After the thread that acquires the lock releases the lock, other threads continue to contend for the lock, acquire the primary key set with the insertion time meeting the preset condition in the target query set by the other threads that acquire the lock, release the lock, and then execute step S13 again by the other threads that acquire the lock. Therefore, according to the above technical solution, when the thread executes step S13, other threads can continue to execute step S11, that is, start the next query, so that parallel processing is realized, and time is saved.
Optionally, the method further comprises: and deleting the primary key with the insertion time before the time offset of the target database in the NoSQL index table and the insertion time corresponding to the primary key.
As can be seen from the above, the primary key inserted into the NoSQL index table before the time offset of the target database is already queried, and after the primary key is deleted, the NoSQL index table can be simplified, thereby saving the memory.
Optionally, the target query set is a delay queue, where the delay queue further stores a delay duration of the message, and step S11 includes: and acquiring a main key set of which the insertion time is equal to the current time minus the delay time length in the delay queue.
Through the technical scheme, after the message is received, the main key of the message is added into the delay queue, the delay time of the message is set, the delay time of the message is determined according to the insertion time and the delay time of the message, and then the main key set acquired in the step S11 is the main key set after the delay time (namely the current time is equal to the delay time) is reached in the delay queue. Therefore, according to the technical scheme, the main key of the message is added into the delay queue and the delay time is set, and the main key set with the current time after the delay time is inquired when the inquiry is carried out, so that the problem of statelessness during the NoSQL database inquiry is solved, and the inquiry efficiency is improved. In addition, in the scheme, because the secondary index is not established on the service table, the problem that the reading performance of the NoSQL database is affected by the secondary index is avoided.
Optionally, the method further comprises: and if the target message is obtained, adding a primary key and retry time of the message into a retry table, wherein the retry time is the time for submitting the target message to a task thread pool.
And if the target message is a message with processing times not reaching the preset times and processing state being failure in the message corresponding to each primary key in the primary key set, adding the primary key and retry time of the message into a retry table for the message with processing times not reaching the preset times and processing state being failure by the scheme, so as to facilitate retry inquiry, for example, counting the number of retry tasks (messages).
Optionally, the NoSQL database is a Cassandra or HBase or LevelDB database. The NoSQL service list is a Cassandra service list, an HBase service list or a LevelDB service list, and the corresponding NoSQL index list is a Cassandra index list, an HBase index list or a LevelDB index list.
Based on the inventive concept, the embodiment of the present disclosure further provides a device for asynchronously processing a message. Fig. 3 is a block diagram illustrating an apparatus for asynchronously processing messages according to an embodiment of the disclosure. As shown in fig. 3, the apparatus for asynchronously processing messages includes: a primary key set acquisition module 11 and a query module 13.
A primary key set obtaining module 11, configured to obtain a primary key set in which insertion time in a target query set meets a preset condition, where the target query set stores primary keys and insertion time of messages;
the query module 13 is configured to query an NoSQL service table according to each primary key in the primary key set to obtain a target message, where the NoSQL service table stores a primary key, a message body, a processing state, and a processing frequency of a message, and the target message is a message in which the processing frequency does not reach a preset frequency and the processing state fails, in a message corresponding to each primary key in the primary key set; if the target message is obtained, storing a primary key of the target message to the target query set, storing the storage time as the insertion time of the target message to the target query set, and submitting the target message to a task thread pool for reprocessing.
Through the technical scheme, the NoSQL database is adopted, the writing and reading speed of NoSQL data is high compared with that of the MySQL database, and the problem that the NoSQL database cannot be suitable due to no partition key is avoided. Meanwhile, the data reading performance is assisted and optimized by establishing a separate target query set instead of adopting a secondary index established on the NoSQL service table, and because the target query set only comprises a main key and insertion time of the NoSQL service table, redundant data is limited. And when the query is carried out, the main key set with the insertion time meeting the preset condition in the target query set is sequentially queried, so that the problem of no stateness in the NoSQL database query is solved, and the query efficiency is improved.
Optionally, the target query set is a NoSQL index table, and the primary key set obtaining module 11 includes: the device comprises an offset acquisition submodule, a first main key set acquisition submodule and an offset updating submodule.
And the offset acquisition submodule is used for acquiring the time offset of the target database.
The first primary key set obtaining sub-module is configured to obtain a primary key set, where an insertion time of the primary key set is within a preset time range and a query amount of the primary key set is within a preset number, in the NoSQL index table, the target database is a redis database or a mongodb database, and an initial time of the preset time range is a time offset of the target database and an end time of the preset time range is a time obtained by subtracting a processing timeout time from a current time.
And the offset updating submodule is used for updating the time offset of the target database to the latest insertion time in the main key set.
By the technical scheme, when the NoSQL index table is queried, the main key sets with the insertion time within the preset time range and the query quantity within the preset number are sequentially queried, so that the problem of statelessness during NoSQL database query is solved, and the query efficiency is improved. During query, only the primary key sets within the preset number are queried, and program collapse caused by excessive memory overflow of query quantity is avoided. And the termination time of the preset time range is the current time minus the processing overtime, and the current time minus the processing overtime is set as the termination time, so that meaningless query caused by the fact that the task is not completely executed during query can be avoided. And after one query is finished, updating the time offset of the target database to the latest insertion time in the main key set, and avoiding or reducing repeated queries.
In addition, since the message whose processing times do not reach the preset times and whose processing state is failed is not only submitted to the task thread pool for reprocessing, but also its primary key is added to the NoSQL index table again, and the storage time is used as the insertion time of the primary key, the NoSQL index table can also be used for retrying query. That is, the retried message can be queried by querying the repeated primary key in the NoSQL index table, so as to facilitate the retried query, for example, counting the number of retried tasks (messages), and obtain the retried time (i.e., the time inserted in the NoSQL index table).
Optionally, in practical implementation, the primary key set obtaining module 11 and the query module 13 are function modules (programs) included in the same thread, the device for asynchronously processing messages is applied to multiple threads, and the device for asynchronously processing messages further includes a lock contention module and a lock release module. Each thread then includes: the system comprises a lock competition module, a primary key set acquisition module 11, a lock release module and a query module 13.
And the lock competition module is used for competing global locks.
The lock releasing module is used for releasing the lock.
And the threads compete for global locking, a primary key set with the insertion time meeting preset conditions in a target query set is obtained by the threads of the lock, the lock is released, and then the threads query a NoSQL service table according to each primary key in the primary key set to obtain target messages. After the thread obtaining the lock releases the lock, other threads continue to compete for the lock, the other threads obtaining the lock obtain a main key set with the target query set insertion time meeting preset conditions, the lock is released, and then the other threads obtaining the lock query a NoSQL service table according to each main key in the main key set to obtain a target message. Therefore, according to the technical scheme, when the thread queries the NoSQL service table according to each main key in the main key set to obtain the target message, other threads can continue to obtain the main key set with the target query set and the insertion time meeting the preset condition, namely, the next query is started, so that the parallel processing is realized, and the time is saved.
Optionally, the apparatus for asynchronously processing a message further comprises a deletion module.
And the deleting module is used for deleting a main key with the insertion time before the time offset of the target database in the NoSQL index table and the insertion time corresponding to the main key.
The primary key in the NoSQL index table inserted with a time before the time offset of the target database is already queried. Therefore, by the technical scheme, the primary key with the time before the time offset of the target database is inserted into the NoSQL index table is deleted, so that the NoSQL index table can be simplified, and the memory is saved.
Optionally, the target query set is a delay queue, the delay queue further stores a delay duration of the message, and the primary key set obtaining module 11 is specifically configured to: and acquiring a main key set of which the insertion time is equal to the current time minus the delay time length in the delay queue.
Through the technical scheme, after receiving the message, the main key of the message is added into the delay queue, the delay time of the message is set, the delay time of the message is determined according to the insertion time and the delay time of the message, and the main key set acquired by the main key set acquisition module 11 is the main key set after the delay time is reached (namely the current time is equal to the delay time) in the delay queue. Therefore, according to the technical scheme, the main key of the message is added into the delay queue and the delay time is set, and the main key set with the current time after the delay time is inquired when the inquiry is carried out, so that the problem of statelessness during the NoSQL database inquiry is solved, and the inquiry efficiency is improved. In addition, in the scheme, because the secondary index is not established on the service table, the problem that the reading performance of the NoSQL database is affected by the secondary index is avoided.
Optionally, the apparatus for asynchronously processing messages further comprises a retry module.
And the retry module is used for adding a primary key and retry time of the message into a retry table if the target message is obtained, wherein the retry time is the time for submitting the target message to a task thread pool.
And if the target message is a message with processing times not reaching the preset times and processing state being failure in the message corresponding to each primary key in the primary key set, adding the primary key and retry time of the message into a retry table for the message with processing times not reaching the preset times and processing state being failure by the scheme, so as to facilitate retry inquiry, for example, counting the number of retry tasks (messages).
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
Embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method for asynchronously processing messages.
The embodiment of the disclosure also provides an electronic device. Fig. 4 is a block diagram illustrating an electronic device 700 according to an example embodiment. As shown in fig. 4, the electronic device 700 may include: a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input/output (I/O) interface 704, and a communication component 705.
The processor 701 is configured to control the overall operation of the electronic device 700, so as to complete all or part of the steps in the above-mentioned method for asynchronously processing messages. The memory 702 is used to store various types of data to support operation at the electronic device 700, such as instructions for any application or method operating on the electronic device 700 and application-related data, such as contact data, transmitted and received messages, pictures, audio, video, and the like. The Memory 702 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk, or optical disk. The multimedia components 703 may include screen and audio components. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may further be stored in the memory 702 or transmitted through the communication component 705. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 704 provides an interface between the processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IOT, eMTC, or other 5G, or the like, or a combination of one or more of them, which is not limited herein. The corresponding communication component 705 may thus include: Wi-Fi module, Bluetooth module, NFC module, etc.
In an exemplary embodiment, the electronic Device 700 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above-described method for asynchronously Processing messages.
In another exemplary embodiment, a computer readable storage medium comprising program instructions which, when executed by a processor, implement the steps of the above-described method of asynchronously processing messages is also provided. For example, the computer readable storage medium may be the memory 702 described above that includes program instructions executable by the processor 701 of the electronic device 700 to perform the method of asynchronously processing messages described above.
In another exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by a programmable apparatus, the computer program having code portions for performing the above-mentioned method of asynchronously processing messages when executed by the programmable apparatus.
The preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings, however, the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solution of the present disclosure within the technical idea of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.
It should be noted that, in the foregoing embodiments, various features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various combinations that are possible in the present disclosure are not described again.
In addition, any combination of various embodiments of the present disclosure may be made, and the same should be considered as the disclosure of the present disclosure, as long as it does not depart from the spirit of the present disclosure.

Claims (10)

1. A method of asynchronously processing messages, the method comprising:
acquiring a main key set with the insertion time meeting preset conditions in a target query set, wherein the main key and the insertion time of a message are stored in the target query set;
inquiring an NoSQL service table according to each main key in the main key set to obtain a target message, wherein the NoSQL service table stores the main key, a message body, a processing state and processing times of the message, and the target message is a message which has the processing times of not reaching the preset times and the processing state of failure in the message corresponding to each main key in the main key set;
if the target message is obtained, storing a primary key of the target message to the target query set, storing the storage time as the insertion time of the target message to the target query set, and submitting the target message to a task thread pool for reprocessing.
2. The method of claim 1, wherein the target query set is a NoSQL index table, and obtaining the primary key set with an insertion time satisfying a preset condition in the target query set comprises:
acquiring time offset of a target database, wherein the target database is a redis database or a mongodb database;
acquiring a main key set of which the insertion time is within a preset time range and the query quantity is within a preset number in the NoSQL index table, wherein the starting time of the preset time range is the time offset of the target database, and the ending time is the current time minus the processing timeout time;
and updating the time offset of the target database to the latest insertion time in the primary key set.
3. The method according to claim 2, wherein before the step of obtaining the set of primary keys whose insertion time satisfies a preset condition, the method further comprises: locks that compete for global;
the step of obtaining the primary key set with the insertion time meeting the preset conditions in the target query set comprises the following steps: after the lock is obtained, acquiring a main key set of which the target query set insertion time meets a preset condition;
after the step of inserting the primary key set with time meeting the preset condition in the target query set, the method further includes: the lock is released.
4. A method according to claim 2 or 3, characterized in that the method further comprises: and deleting the primary key with the insertion time before the time offset of the target database in the NoSQL index table and the insertion time corresponding to the primary key.
5. The method of asynchronous processing of messages as recited in claim 1, wherein the set of target queries is a delay queue, the delay queue further storing a delay duration for the message;
the acquiring of the primary key set with the insertion time meeting the preset conditions in the target query set comprises: and acquiring a main key set of which the insertion time is equal to the current time minus the delay time length in the delay queue.
6. The method of claims 1, 2, 3, 5, further comprising:
and if the target message is obtained, adding a primary key and retry time of the message into a retry table, wherein the retry time is the time for submitting the target message to a task thread pool.
7. An apparatus for asynchronously processing messages, comprising:
the system comprises a main key set acquisition module, a main key set acquisition module and a main key set acquisition module, wherein the main key set acquisition module is used for acquiring a main key set of which the insertion time meets a preset condition in a target query set, and the main key and the insertion time of a message are stored in the target query set;
the query module is used for querying a NoSQL service table according to each main key in the main key set to obtain a target message, wherein the NoSQL service table stores the main key, a message body, a processing state and the processing times of the message, and the target message is a message of which the processing times are not up to the preset times and the processing state is failed in the message corresponding to each main key in the main key set; if the target message is obtained, storing a primary key of the target message to the target query set, storing the storage time as the insertion time of the target message to the target query set, and submitting the target message to a task thread pool for reprocessing.
8. The apparatus of claim 6, wherein the target set of queries is a NoSQL index table, and wherein the primary key set obtaining module comprises:
the offset acquisition submodule is used for acquiring the time offset of the target database;
a first primary key set obtaining sub-module, configured to obtain a primary key set, in the NoSQL index table, where an insertion time is within a preset time range and a query amount is within a preset number, the target database is a redis database or a mongodb database, an initial time of the preset time range is a time offset of the target database, and an end time is a current time minus a processing timeout time;
and the offset updating submodule is used for updating the time offset of the target database to the latest insertion time in the main key set.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.
10. An electronic device, comprising:
a memory having a computer program stored thereon;
a processor for executing the computer program in the memory to carry out the steps of the method of any one of claims 1 to 6.
CN201910880883.2A 2019-09-18 2019-09-18 Method, device, readable medium and electronic equipment for asynchronously processing messages Active CN110609841B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910880883.2A CN110609841B (en) 2019-09-18 2019-09-18 Method, device, readable medium and electronic equipment for asynchronously processing messages

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910880883.2A CN110609841B (en) 2019-09-18 2019-09-18 Method, device, readable medium and electronic equipment for asynchronously processing messages

Publications (2)

Publication Number Publication Date
CN110609841A true CN110609841A (en) 2019-12-24
CN110609841B CN110609841B (en) 2020-09-15

Family

ID=68891425

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910880883.2A Active CN110609841B (en) 2019-09-18 2019-09-18 Method, device, readable medium and electronic equipment for asynchronously processing messages

Country Status (1)

Country Link
CN (1) CN110609841B (en)

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101056275A (en) * 2006-04-10 2007-10-17 腾讯科技(深圳)有限公司 A setting method for group message receiving mode in the instant communication system
CN101056421A (en) * 2007-06-15 2007-10-17 中兴通讯股份有限公司 A method for optimizing Push notice message
CN101131747A (en) * 2006-08-22 2008-02-27 国际商业机器公司 Method, device and system for catching and/or analyzing Web page events at client terminal
CN100562184C (en) * 2006-05-22 2009-11-18 余克艰 The step synchronization match method of asynchronous search and device
US7814142B2 (en) * 2003-08-27 2010-10-12 International Business Machines Corporation User interface service for a services oriented architecture in a data integration platform
CN102541529A (en) * 2010-12-30 2012-07-04 北京亿阳信通软件研究院有限公司 Query page generating device and method
CN103177027A (en) * 2011-12-23 2013-06-26 北京新媒传信科技有限公司 Method and system for obtaining dynamic feed index
US8522022B2 (en) * 2009-10-30 2013-08-27 Cleversafe, Inc. Distributed storage network employing multiple encoding layers in data routing
CN104794130A (en) * 2014-01-20 2015-07-22 华为技术有限公司 Inter-table correlation query method and device
CN108024275A (en) * 2017-12-07 2018-05-11 锐捷网络股份有限公司 Data transmission method, device, electronic equipment and computer-readable recording medium based on WDS
CN109614402A (en) * 2018-12-11 2019-04-12 北京京东金融科技控股有限公司 Multidimensional data query method and device

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7814142B2 (en) * 2003-08-27 2010-10-12 International Business Machines Corporation User interface service for a services oriented architecture in a data integration platform
CN101056275A (en) * 2006-04-10 2007-10-17 腾讯科技(深圳)有限公司 A setting method for group message receiving mode in the instant communication system
CN100562184C (en) * 2006-05-22 2009-11-18 余克艰 The step synchronization match method of asynchronous search and device
CN101131747A (en) * 2006-08-22 2008-02-27 国际商业机器公司 Method, device and system for catching and/or analyzing Web page events at client terminal
CN101056421A (en) * 2007-06-15 2007-10-17 中兴通讯股份有限公司 A method for optimizing Push notice message
US8522022B2 (en) * 2009-10-30 2013-08-27 Cleversafe, Inc. Distributed storage network employing multiple encoding layers in data routing
CN102541529A (en) * 2010-12-30 2012-07-04 北京亿阳信通软件研究院有限公司 Query page generating device and method
CN103177027A (en) * 2011-12-23 2013-06-26 北京新媒传信科技有限公司 Method and system for obtaining dynamic feed index
CN104794130A (en) * 2014-01-20 2015-07-22 华为技术有限公司 Inter-table correlation query method and device
CN108024275A (en) * 2017-12-07 2018-05-11 锐捷网络股份有限公司 Data transmission method, device, electronic equipment and computer-readable recording medium based on WDS
CN109614402A (en) * 2018-12-11 2019-04-12 北京京东金融科技控股有限公司 Multidimensional data query method and device

Also Published As

Publication number Publication date
CN110609841B (en) 2020-09-15

Similar Documents

Publication Publication Date Title
CN107688500B (en) Distributed task processing method, device, system and equipment
WO2021180025A1 (en) Message processing method and apparatus, electronic device and medium
EP3467668A1 (en) Method for processing database transaction, client and server
CN109788027B (en) Data synchronization method, device, server and computer storage medium
CN109086382B (en) Data synchronization method, device, equipment and storage medium
US11269829B2 (en) Row level locking for columnar data
US11308060B2 (en) Method, apparatus, device and storage medium for managing index
EP3825865A2 (en) Method and apparatus for processing data
CN111444147A (en) Data page creating method and device, terminal equipment and storage medium
CN115480894A (en) Method, device and equipment for realizing distributed delay task
CN110609841B (en) Method, device, readable medium and electronic equipment for asynchronously processing messages
CN111124355B (en) Information processing method and device, readable storage medium and electronic equipment
US11269825B2 (en) Privilege retention for database migration
CN112000321A (en) Method and device for realizing undo and/or redo of three-dimensional detection software
CN109101191B (en) Data storage method, data storage device, storage medium and electronic equipment
CN111596864A (en) Method, device, server and storage medium for data delayed deletion
CN111367948A (en) Data processing method and device, electronic equipment and computer readable storage medium
CN106354830B (en) Method and device for data synchronization between database cluster nodes
CN113986942A (en) Message queue management method and device based on man-machine conversation
EP4044043A1 (en) Storage process running method and apparatus, database system, and storage medium
CN110896391B (en) Message processing method and device
US10452424B2 (en) Unique transaction identifier based transaction processing
CN110727666A (en) Cache assembly, method, equipment and storage medium for industrial internet platform
US10482077B2 (en) System and method for asynchronous update of a search index
CN112000482B (en) Memory management method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant