CN113778321A - Message processing method and device - Google Patents
Message processing method and device Download PDFInfo
- Publication number
- CN113778321A CN113778321A CN202110054031.5A CN202110054031A CN113778321A CN 113778321 A CN113778321 A CN 113778321A CN 202110054031 A CN202110054031 A CN 202110054031A CN 113778321 A CN113778321 A CN 113778321A
- Authority
- CN
- China
- Prior art keywords
- message
- processing
- display page
- refresh
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims description 20
- 238000012545 processing Methods 0.000 claims abstract description 119
- 238000000034 method Methods 0.000 claims abstract description 49
- 230000003111 delayed effect Effects 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 238000010586 diagram Methods 0.000 description 22
- 238000005457 optimization Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and a device for processing messages, and relates to the technical field of computers. One embodiment of the method comprises: receiving a message sent from a server; checking the message for one time to confirm whether the message is in a state of waiting for processing or being processed; when the message is not in a state of waiting for processing or processing, merging the message into a message queue based on the arrival sequence of the message, and processing the messages in the message queue in sequence; and displaying the message of completing the processing on the display page. The implementation method reduces the system overhead of high-concurrency message processing, improves the efficiency of high-concurrency message processing, and simultaneously reduces the problem of crash of instant messaging in the high-concurrency message processing.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for processing a message, and in particular, to a method and an apparatus for processing a high-concurrency message that can be used for instant messaging.
Background
Message handling, particularly high-concurrency message handling, is common in the field of instant messaging. In an application environment with high concurrent messages, a large number of messages are received from a server side in a short time, and if the messages cannot be displayed on a display page (UI) timely and quickly, the messages cannot be replied in time, so that accumulation of messages to be processed, delay of message processing and the like can be caused.
Conventionally, for an application scenario of high-concurrency messages, a method of message processing as shown in fig. 10 is generally adopted. Receiving messages from a server, and enabling the messages to enter a queue according to an arrival sequence and be processed in sequence; and after the cancellation processing is finished, storing the message into a database and displaying the message on a display page.
However, the inventors have found that the above conventional message processing method has at least the following problems:
(1) the message is repeatedly sent, so that the message is repeatedly processed;
(2) due to the characteristics of instant messaging, the system overhead of writing into the database cannot be obviously reduced when the database transaction is started;
(3) when reading the message, extra disk reading and writing are caused under the condition of index miss;
(4) in a high concurrency message scene, the UI refreshing times in a short time are too many, so that the page is blocked.
Disclosure of Invention
In view of this, embodiments of the present invention provide a message processing method and a message processing apparatus, which can reduce the overhead of high concurrent message processing, improve the efficiency of high concurrent message processing, and reduce the problem of crash of instant messaging in high concurrent message processing.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a message processing method, including: receiving a message sent from a server; checking the message for one time to confirm whether the message is in a state of waiting for processing or in the process; when the message is not in a state of waiting for processing or processing, merging the message into a message queue based on the arrival sequence of the message, and processing the message in the message queue according to the sequence; and displaying the message of completing the processing on the display page.
Preferably, according to the message processing method in an aspect of the embodiment of the present invention, before the message that is completed with the processing is displayed on the display page, the message that is completed with the processing is stored, and the storing includes successively storing the message that is completed with the processing in the memory cache and the database.
Preferably, according to the method for processing a message in an aspect of the embodiment of the present invention, before processing the message in the message queue, a second duplicate checking is performed to determine whether the message is already stored in the memory cache; when the message is not stored in the memory cache, carrying out three-time duplicate checking to confirm whether the message is stored in the database; and processing the message when the message is not stored in the database.
Preferably, according to the method for message processing in one aspect of the embodiments of the present invention, the database is SQLite, and the write mode of SQLite is set to the WAL mode.
Preferably, according to the message processing method in an aspect of the embodiments of the present invention, before the message that is completed to be processed is displayed on the display page, the target message is extracted, the target message is queried in the memory cache, and when the target message is queried, the target message is displayed on the display page; and when the target message is not found in the memory cache, further inquiring in the database, and displaying the inquired target message on the display page.
Preferably, according to the method for processing a message in one aspect of the embodiment of the present invention, the message that has been processed is sent to a display queue of the display page, and a refresh operation of the display page is performed according to a time difference between a time point when the message that has been processed reaches the display queue and a last refresh and a predetermined refresh threshold.
Preferably, according to the message processing method in an aspect of the embodiments of the present invention, the performing a display page refresh operation according to the time difference and a predetermined refresh threshold includes: when the time difference is larger than or equal to the refresh threshold, directly performing the refresh operation of the display page; and when the time difference is smaller than the refresh threshold and the sum of the time difference and the preset delay time is larger than the refresh threshold, performing the display page refresh operation at a time point when the refresh threshold is passed from the last refresh.
Preferably, according to the message processing method in an aspect of the embodiments of the present invention, the performing a display page refresh operation according to the time difference and a predetermined refresh threshold includes: and when the time difference is smaller than the refresh threshold and the sum of the time difference and the preset delay time is smaller than the refresh threshold, generating a delay refresh instruction and performing the display page refresh operation at a time point when the preset delay time passes from a time point when the processed message reaches the display queue, wherein when the display page refresh operation is performed, other generated delay refresh instructions are cancelled.
Preferably, according to the method for processing a message in an aspect of the embodiment of the present invention, when the user slides the display page, the display page refresh operation is not performed.
Preferably, according to the method for processing a message in an aspect of the embodiment of the present invention, a flag is added to the message that is not in the waiting state or the processing state through the one-time re-check confirmation, for retrieval in a subsequent operation.
According to another aspect of the embodiments of the present invention, there is provided a message processing apparatus, including: the receiving module is used for receiving the message sent from the server; a duplicate checking module for checking the duplicate of the message once and confirming whether the message is in a state of waiting for processing or being processed; the queue module merges the messages into a message queue based on the sequence of the messages reaching the receiving module and processes the messages in the message queue according to the sequence when the messages are not in the state of waiting for processing or processing; and the display module is used for displaying the processed message on a display page.
According to still another aspect of an embodiment of the present invention, there is provided an electronic device for message processing, including: one or more processors; a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement a method of message processing according to an aspect of an embodiment of the present invention.
According to yet another aspect of embodiments of the present invention, there is provided a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing a method of message processing according to one aspect of embodiments of the present invention.
The invention provides an effective method for processing high-concurrency messages through four aspects of message duplicate checking, writing performance optimization, query performance optimization and message UI refreshing condition optimization. The message processing method constructed by the invention can solve the technical problems encountered by the message processing with high concurrency performance in most scenes except for normal operation in common low concurrency scenes, and constructs a stable and efficient message processing method and a message processing device which can be used for instant messaging.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
fig. 1 is a schematic diagram of a main flow of a method of message processing according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the main modules of an apparatus for message processing according to an embodiment of the present invention;
fig. 3 is a diagram mainly illustrating one-time duplicate checking in a method of message processing according to an embodiment of the present invention;
fig. 4 is a diagram mainly illustrating a flow including one-time duplicate checking, two-time duplicate checking, and three-time duplicate checking in the method of message processing according to the embodiment of the present invention;
FIG. 5 is a diagram illustrating the logic of database add-drop-and-delete-check in a method of message processing according to an embodiment of the present invention;
FIG. 6 is a diagram illustrating display page delayed refresh logic in a method of message processing according to an embodiment of the present invention;
fig. 7 is a diagram showing one specific example of the overall flow of a method of message processing according to an embodiment of the present invention;
FIG. 8 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
FIG. 9 is a block diagram of a computer system suitable for use with a terminal device or server that implements an embodiment of the invention;
fig. 10 is a schematic diagram of a conventional method of message processing.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Fig. 1 is a schematic diagram of a main flow of a method of message processing according to an embodiment of the present invention, and fig. 3 is a schematic diagram mainly showing one-time duplication checking in the method of message processing according to an embodiment of the present invention.
In the traditional message processing method, after a certain message is processed and stored in a database, a client side sends a receipt to a server side, and the server side does not repeatedly send the message to the client side after receiving the receipt. However, in a real scenario, when the client already receives the message, but the message processing is too slow or the sent receipt is not received by the server due to network packet loss, the server may not receive the message receipt in time. In this case, since the server cannot distinguish the specific reason for not receiving the receipt, it generally determines that the client has not received the message, and then sends the message to the client again. While the repeated processing of messages is a waste of resources. Meanwhile, because the message queue to be processed is a serial queue, messages can be queued in the queue according to the time receiving sequence, if the server side can not receive the receipt of the client side for a long time and repeatedly sends the messages, a large amount of repeated messages can be generated in the message queue under extreme conditions, the pressure of the message queue can be increased, and the system is crashed due to the sudden increase of the memory.
To solve the above problem, as shown in fig. 1 and 3, a method for message processing according to an embodiment of the present invention includes: receiving a message sent from a server; checking the message for one time to confirm whether the message is in a state of waiting for processing or in the process; when the messages are not in a state of waiting for processing or being processed, merging the messages into a message queue based on the arrival sequence of the messages, and processing the messages in the message queue in sequence; and displaying the message of completing the processing on the display page. The above processing refers to extracting fields of the message, such as a sender of the message, a sending time of the message, content of the message, and the like.
In the preferred embodiment of the present invention, the processed message is stored, the message is stored in the memory cache and the database in sequence, the message is removed from the message queue, and then the message is displayed on the display page. The processed message is firstly stored in the memory cache and is stored in the database in a timed/quantitative batch mode. And updating and replacing the processed information stored in the memory cache according to the capacity of the memory cache. And after the message is stored in the database, sending a receipt to the server to inform the server that the message is received. The server does not repeatedly send the message after receiving the receipt.
Fig. 4 is a diagram mainly illustrating a flow including first, second, and third duplication checking in a message processing method according to an embodiment of the present invention. As shown in fig. 4, before the message in the message queue is processed, a second duplicate checking is performed to determine whether the message is already stored in the memory cache; when the message is not stored in the memory cache, performing three times of duplicate checking to confirm whether the message is already stored in the database (for example, querying through an sql (structured query language)); and processing the message when the message is not stored in the database.
When a message is checked for duplication once, if the message is confirmed to be in a waiting state or a processing state in the message queue, the message is judged to be a repeated message and is directly discarded. A tag may be added to a message that passes a duplication check for duplication checking in subsequent operations and/or retrieval upon reading. When the message is checked for duplication for the second time, if a message is confirmed to be stored in the memory cache, the message is also judged to be a duplicate message and is directly discarded. Similarly, if a message is determined to be stored in the database after three duplicate checks of the message, the message is also determined to be a duplicate message and discarded directly.
That is, if a message has been determined to be a duplicate message at the time of a duplicate check, the message may be discarded directly, and the message may not be entered into the message queue for subsequent processing. When the message passes the first time of duplicate checking (namely, the message is confirmed not to be in a state of waiting for processing or being processed in the message queue), the second time of duplicate checking is carried out, if the message is already determined to be a repeated message at the second time of duplicate checking, the message can be directly discarded, and the message is not subjected to the third time of duplicate checking and subsequent processing. Therefore, if the existence of the message is confirmed in the memory cache, the query of the database can be reduced once, and the I/O time of the database is saved. Only when a message passes through three times of duplicate checking, two times of duplicate checking and three times of duplicate checking, namely the message is not in a state of waiting for processing or processing in a message queue and is not in a state of processing completion and is stored in a memory cache and/or a database, the message is processed, and fields such as a sender, sending time, content and the like of the message are extracted.
After the message is processed, the message needs to be stored in a database, and the writing operation of the database belongs to input/output (I/O) operation on a data file, so that the file is frequently opened and closed due to frequent writing of data, and the performance is very lost. One conventional optimization method is to open a database transaction. The database transaction is a batch warehousing mode, data are firstly cached in a memory by starting the database transaction, all the data cached in the memory are updated to a database file once when the database transaction is submitted, at the moment, the database file only needs to be opened and closed once, and the problem of low performance caused by long-term occupation of file I/O is also avoided. But for instant messaging systems, in most cases (i.e. the scenario of ordinary low-concurrency messages), messages do not arrive in bulk, but are sent one by one. In addition, due to the message arrival sequence, the final situation becomes more complicated due to the network and other conditions. In this case, since the database transaction is started, the processed messages need to be buffered in the memory, and the messages accumulated in the memory need to be stored in the memory in batch processing at regular time or quantitatively, which requires additional logic implementation, and the complexity of the code is increased, thereby increasing the maintenance cost.
To solve the above problem, according to a preferred embodiment of the present invention, the database is a SQLite (a lightweight database), and a Write mode of the SQLite is set to a WAL (Write-through Log) mode.
Compared to the default rollback (rollback) mode, the WAL mode has the following advantages: in most cases, writing speed is faster using the WAL mode; the WAL mode further improves the concurrency of the database, because the read does not block the write, and the write does not block the read, and the read and the write can be executed concurrently; using the WAL mode, disk I/O operations are more orderly; using WAL mode reduces the number of fsync () operations (to store all modified file data in function-synchronous memory) and is therefore not susceptible to system call interrupts (system calls) of fsync () on the system.
Specifically, the WAL mode may be turned on using the following code:
PRAGMA journal_mode＝WAL。
during reading in the WAL mode, the SQLite searches the-WAL file (write-through file) first, finds the last write point, remembers the write point, and ignores the write points after that (this ensures the execution of concurrent reading and writing and concurrent reading). Subsequently, it determines whether the page where the data to be read exists in the-wal file, and if so, reads the data in the-wal file, and if not, directly reads the data in the database file. To avoid scanning the entire-WAL file for pages per read operation, SQLite provides a WAL-index file to assist in the page lookup. The WAL-index file uses a technology of sharing memory among processes, and the shared memory is a file which is at the end of shm and is in the same directory as the database file. The file is more particularly characterized in that a mapping relation exists between a memory and the file, the file can be read and written like the memory after the address of the file is obtained, and the general file can be read and written only by calling read and write functions. The WAL-index can help the reading operation to quickly locate the page in the-WAL file, and the reading performance is greatly improved. Under the condition that reading does not influence writing, the writing performance of the database can be further greatly improved.
FIG. 5 is a diagram illustrating the logic of a database add-drop-and-delete-check, according to an embodiment of the present invention.
After the processed message is stored in the database, when the processed message is displayed on a display page, the message needs to be searched and extracted from the database, that is, data in the database is read. If the database is queried by a message label (ID) each time, all messages in the database need to be traversed for matching. The speed of the query will be slower and slower with an increasing number of messages in the database. One conventional optimization method is to add indexes, which are provided by common relational databases, and the reading performance of the databases can be improved by adding the indexes appropriately. However, although the reading performance of the database can be improved by adding the index, the disk I/O is still required when the cache of the database misses, and the disk I/O is time-consuming.
To solve the above problem, according to a preferred embodiment of the present invention, a target message (message to be displayed) is extracted before a message whose processing is completed is displayed on a display page. At this time, the target message is queried in the memory Cache (the memory Cache layer in fig. 5), and when the target message is queried, the target message is returned for displaying on the display page, and at this time, one database reading operation can be reduced. The memory cache may use a simple key-value structure, with a key as a primary key (PrimaryKey, or other unique key, which should often be considered as a query condition). When the target message is not hit in the memory cache, further query is performed in the database (DB layer in fig. 5), and the queried target message is updated to the memory cache first and then returned for display on the display page.
According to the preferred embodiment of the invention, the Where condition can also be directly used to query the database and return the relevant data of the message to be displayed on the display page. Similarly, when writing and updating data, as described above, the data may be written/updated to the memory cache layer first, and then written/updated to the database.
FIG. 6 is a diagram illustrating display page delayed refresh logic in a method of message processing according to an embodiment of the present invention.
After the message is processed, the message needs to be displayed on the UI next step. If each message is processed, the page is refreshed immediately, so that the problem of page jamming is caused by excessive refreshing times of the UI in a short time under the common low-concurrency scene but under the high-concurrency scene.
To solve the above problem, as shown in fig. 6, according to the preferred embodiment of the present invention, the processed message is sent to the display queue of the display page, and the display page is refreshed according to the time difference between the time point when the processed message reaches the display queue and the last refresh and the predetermined refresh threshold.
Specifically, the performing of the display page refresh operation according to the time difference and the predetermined refresh threshold includes: when the time difference is larger than or equal to the refresh threshold, directly performing the refresh operation of the display page; and when the time difference is smaller than the refresh threshold and the sum of the time difference and the preset delay time is larger than the refresh threshold, performing the display page refresh operation at a time point when the refresh threshold is passed from the last refresh.
For example, the refresh threshold may be set to 2000ms and the predetermined delay period to 100 ms. When the message A arrives, the time difference from the last refreshing is equal to 2000ms, and the display page refreshing operation is directly carried out. When the message A arrives, the time difference from the last refreshing is equal to 2500ms, and the display page refreshing operation is also directly carried out.
If the time difference from the last refresh is less than the refresh threshold (e.g., 1950ms) and the sum of the time difference from the last refresh and the predetermined delay time is greater than the refresh threshold (i.e., 1950ms +100ms > 2000ms), a display page refresh operation is performed at a point in time when the refresh threshold has passed since the last refresh (2000 ms from the last refresh).
Specifically, the performing of the display page refresh operation according to the time difference and the predetermined refresh threshold includes: and when the time difference is smaller than the refresh threshold and the sum of the time difference and the preset delay time is smaller than the refresh threshold, generating a delay refresh instruction and performing the display page refresh operation at a time point when the preset delay time passes from a time point when the processed message reaches the display queue, wherein when the display page refresh operation is performed, other generated delay refresh instructions are cancelled.
For example, if the time difference between the arrival time of the message a and the last refresh is less than 2000ms (e.g. 1600ms) and the sum of the time difference between the arrival time of the message a and the last refresh and the predetermined delay time is less than the refresh threshold (i.e. 1600ms +100ms < 2000ms), a delayed refresh command is generated and a display page refresh operation is performed at a time point when the predetermined delay time elapses from the arrival time of the message a to the display queue (i.e. 1600ms +100ms is 1700ms apart from the last refresh). If there is a message B at the same time, and the time difference between the arrival time of the message B at the display queue and the last refresh is 1650ms, a delayed refresh command performed in 1750ms is correspondingly generated. And when the 1700ms delayed refresh command generated by the message A is carried out and the display page refresh operation is carried out, the 1750ms delayed refresh command generated by the message B is cancelled, namely the display page refresh operation is not carried out at 1750 ms.
According to the preferred embodiment of the invention, when the user slides the display page (such as the conversation list/conversation page message list), the display page is not refreshed, and the display page is refreshed when the sliding of the display page is stopped, so that the sliding fluency of the display page can be ensured.
Fig. 7 is a diagram showing one specific example of the overall flow of the method of message processing according to the embodiment of the present invention. As shown in fig. 7, a message transmitted from a server is received. Then, the message is checked for duplication to determine whether the message is in a state of waiting for processing or in a state of processing. When the message is confirmed to be in a waiting state or a processing state in the message queue, the message is judged to be a repeated message and is directly discarded; incorporating the messages into a message queue based on an order in which the messages arrive when the messages are not pending or in progress. Then, the second duplicate checking is carried out to confirm whether the message is stored in the memory cache. When the message is confirmed to be stored in the memory cache, the message is also judged to be a repeated message and is directly discarded; when the message is not stored in the memory cache, three times of duplicate checking are performed to confirm whether the message is already stored in the database (for example, the message is queried through sql). When a message is confirmed to be stored in the database, the message is also judged to be a repeated message and is directly discarded; and processing the message when the message is not stored in the database. The processed messages are firstly buffered in a memory cache, and the processed messages stored in the memory cache are stored in a database in a batch mode at regular time/quantity. At this time, the message queue informs the UI queue that a new message exists, and the display page refreshing operation is carried out according to the time difference between the time point when the processed message reaches the display queue and the last refreshing time and a preset refreshing threshold value.
Fig. 2 is a schematic diagram of main blocks of an apparatus for message processing according to an embodiment of the present invention. As shown in fig. 2, the message processing apparatus 200 includes:
a receiving module 201, configured to receive a message sent from a server;
a duplicate checking module 202, configured to perform a duplicate checking on the message to determine whether the message is in a state of waiting for processing or being processed;
a queue module 203, which merges the messages into a message queue based on the order of the messages arriving at the receiving module 201 and processes the messages in the message queue according to the order, when the messages are not waiting for processing or in the processing state; and
and a display module 204, configured to display the processed message on a display page.
The queue module 203 adds a flag to a message that is not in a pending or processing state as confirmed by a single check for retrieval in subsequent operations.
The message processing apparatus 200 may store the processed message by using a storage module (not shown) before displaying the processed message on the display page, and store the processed message in the memory cache and the database sequentially. The database is SQLite, and the write mode of SQLite is set to WAL mode.
In addition, before processing, the queue module 203 performs a second duplicate checking to determine whether the message is already stored in the memory cache; when the message is not stored in the memory cache, carrying out three times of duplicate checking to confirm whether the message is stored in the database; and the queue module 203 processes the message when the message is not stored in the database.
Before the display module 204 displays the processed message on the display page, an extraction module (not shown) extracts the target message, queries the target message in the memory cache, and displays the target message on the display page when the target message is queried. And when the target message is not found in the memory cache, the extraction module further queries in the database and displays the queried target message on a display page.
The display module 204 receives the target message extracted by the extraction module and merges the target message into the display queue, and performs a display page refresh operation according to a time difference between a time point when the target message (the processed message) reaches the display queue and the last refresh and a predetermined refresh threshold. When the time difference is greater than or equal to the refresh threshold, the display module 204 directly performs a display page refresh operation. And when the time difference is smaller than the refresh threshold and the sum of the time difference and the predetermined delay time is greater than the refresh threshold, the display module 204 performs the display page refresh operation at a time point when the refresh threshold has passed since the last refresh. When the time difference is less than the refresh threshold and the sum of the time difference and the predetermined delay time is less than the refresh threshold, the display module 204 generates a delayed refresh command and performs a display page refresh operation at a time point when the predetermined delay time elapses from a time point when the target message (the processed message) reaches the display queue. The display module 204 cancels other delayed refresh commands that have been generated when performing a display page refresh operation. When the user slides the display page, the display module 204 does not perform the display page refresh operation.
Fig. 8 shows an exemplary system architecture 800 of a message processing apparatus or method to which embodiments of the invention may be applied.
As shown in fig. 8, the system architecture 800 may include end devices (e.g., clients that receive messages) 801, 802, 803, a network 804, and a server (e.g., a server that sends messages) 805. The network 804 serves to provide a medium for communication links between the terminal devices 801, 802, 803 and the server 805. Network 804 may include various types of connections, such as wire, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 801, 802, 803 to interact with a server 805 over a network 804 to receive or send messages or the like. The terminal devices 801, 802, 803 may have installed thereon various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, and the like.
The terminal devices 801, 802, 803 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 805 may be a server that provides various services, such as a background management server that supports shopping websites browsed by users using the terminal devices 801, 802, 803. The background management server can send the received message to the terminal device.
It should be noted that the method for message processing provided by the embodiment of the present invention is generally executed by terminal devices (e.g., clients receiving messages) 801, 802, and 803, and accordingly, the apparatus for message processing is generally disposed in the terminal devices 801, 802, and 803.
It should be understood that the number of terminal devices, networks, and servers in fig. 8 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 9, shown is a block diagram of a computer system 900 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 9, the computer system 900 includes a Central Processing Unit (CPU)901 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data necessary for the operation of the system 900 are also stored. The CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.
The following components are connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The above-described functions defined in the system of the present invention are executed when the computer program is executed by a Central Processing Unit (CPU) 901.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a receiving module, a one-time duplication checking module, a queue module, and a display module. The names of these modules do not in some cases constitute a limitation on the module itself, for example, a receiving module may also be described as a "module that receives messages from a server".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving a message sent from a server; checking the message for one time to confirm whether the message is in a state of waiting for processing or processing; when the message is not in a state of waiting for processing or processing, merging the message into a message queue based on the arrival sequence of the message, and processing the messages in the message queue in sequence; and displaying the message of completing the processing on the display page.
The invention provides an effective method for processing high-concurrency messages through four aspects of message duplicate checking, writing performance optimization, query performance optimization and message UI refreshing condition optimization. The message processing method constructed by the invention can solve the technical problems encountered by the message processing with high concurrency performance in most scenes except for normal operation in common low concurrency scenes, and constructs a stable and efficient message processing method and a message processing device which can be used for instant messaging.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (13)
1. A method of message processing, comprising:
receiving a message sent by a server;
checking the message for one time to confirm whether the message is in a state of waiting for processing or in the process;
when the message is not in a state of waiting for processing or processing, merging the message into a message queue based on the arrival sequence of the message, and processing the message in the message queue according to the sequence; and
and displaying the message of completing the processing on the display page.
2. The method according to claim 1, wherein the message of which the processing is completed is stored before being displayed on the display page, and wherein
And the storage comprises the step of successively storing the processed messages in a memory cache and a database.
3. The message processing method according to claim 2, wherein before processing the message in the message queue, a second duplicate checking is performed to confirm whether the message is already stored in the memory cache;
when the message is not stored in the memory cache, carrying out three-time duplicate checking to confirm whether the message is stored in the database; and is
Processing the message when the message is not stored in the database.
4. The message processing method according to claim 2, wherein the database is SQLite, and a writing mode of SQLite is set to the WAL mode.
5. The method of claim 2, wherein before the message of which the processing is completed is displayed on the display page, a target message is extracted,
inquiring the target message in the memory cache, and displaying the target message on the display page when the target message is inquired;
and when the target message is not found in the memory cache, further inquiring in the database, and displaying the inquired target message on the display page.
6. The message processing method according to claim 1, wherein the message with completed processing is sent to a display queue of the display page, and a display page refresh operation is performed according to a time difference between a time point when the message with completed processing reaches the display queue and a last refresh and a predetermined refresh threshold.
7. The method of claim 6, wherein performing a display page refresh operation based on the time difference and a predetermined refresh threshold comprises:
when the time difference is larger than or equal to the refresh threshold, directly performing the refresh operation of the display page; and is
And when the time difference is smaller than the refresh threshold and the sum of the time difference and the preset delay time is larger than the refresh threshold, performing the display page refresh operation at the time point when the refresh threshold passes from the last refresh.
8. The method of claim 6, wherein performing a display page refresh operation based on the time difference and a predetermined refresh threshold comprises:
generating a delayed refresh command and performing the display page refresh operation at a point of time when a predetermined delay time period has elapsed from a point of time when the processed message reaches the display queue when the time difference is smaller than the refresh threshold and a sum of the time difference and a predetermined delay time period is smaller than the refresh threshold,
and when the display page refreshing operation is carried out, canceling other generated delayed refreshing instructions.
9. The message processing method according to any one of claims 6 to 8, wherein the display page refresh operation is not performed when the user slides the display page.
10. The message processing method according to claim 1, wherein a flag is added to the message which is not in the waiting state or in the processing state and is confirmed by the one-time re-check for retrieval in the subsequent operation.
11. An apparatus for message processing, comprising:
the receiving module is used for receiving the message sent from the server;
a duplicate checking module for checking the duplicate of the message once and confirming whether the message is in a state of waiting for processing or being processed;
the queue module merges the messages into a message queue based on the sequence of the messages reaching the receiving module and processes the messages in the message queue according to the sequence when the messages are not in the state of waiting for processing or processing; and
and the display module is used for displaying the processed message on a display page.
12. An electronic device for message processing, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-10.
13. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110054031.5A CN113778321A (en) | 2021-01-15 | 2021-01-15 | Message processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110054031.5A CN113778321A (en) | 2021-01-15 | 2021-01-15 | Message processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113778321A true CN113778321A (en) | 2021-12-10 |
Family
ID=78835438
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110054031.5A Pending CN113778321A (en) | 2021-01-15 | 2021-01-15 | Message processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113778321A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105227662A (en) * | 2015-09-30 | 2016-01-06 | 努比亚技术有限公司 | Message treatment method, device and system |
| WO2016177081A1 (en) * | 2015-08-07 | 2016-11-10 | 中兴通讯股份有限公司 | Notification message processing method and device |
| WO2018036478A1 (en) * | 2016-08-24 | 2018-03-01 | 腾讯科技（深圳）有限公司 | Method, apparatus and system for displaying interaction message, and storage medium |
| WO2019061647A1 (en) * | 2017-09-26 | 2019-04-04 | 平安科技（深圳）有限公司 | Queue message processing method and device, terminal device and medium |
| KR20200045639A (en) * | 2018-10-23 | 2020-05-06 | 삼성에스디에스 주식회사 | Apparatus and method for managing queue |
| CN111212390A (en) * | 2019-12-23 | 2020-05-29 | 北京健康之家科技有限公司 | Message queue processing method, device and equipment |
| CN111381987A (en) * | 2020-03-13 | 2020-07-07 | 北京金山云网络技术有限公司 | Message processing method and device, electronic equipment and medium |
-
2021
- 2021-01-15 CN CN202110054031.5A patent/CN113778321A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2016177081A1 (en) * | 2015-08-07 | 2016-11-10 | 中兴通讯股份有限公司 | Notification message processing method and device |
| CN105227662A (en) * | 2015-09-30 | 2016-01-06 | 努比亚技术有限公司 | Message treatment method, device and system |
| WO2018036478A1 (en) * | 2016-08-24 | 2018-03-01 | 腾讯科技（深圳）有限公司 | Method, apparatus and system for displaying interaction message, and storage medium |
| WO2019061647A1 (en) * | 2017-09-26 | 2019-04-04 | 平安科技（深圳）有限公司 | Queue message processing method and device, terminal device and medium |
| KR20200045639A (en) * | 2018-10-23 | 2020-05-06 | 삼성에스디에스 주식회사 | Apparatus and method for managing queue |
| CN111212390A (en) * | 2019-12-23 | 2020-05-29 | 北京健康之家科技有限公司 | Message queue processing method, device and equipment |
| CN111381987A (en) * | 2020-03-13 | 2020-07-07 | 北京金山云网络技术有限公司 | Message processing method and device, electronic equipment and medium |
Non-Patent Citations (1)
| Title |
|---|
| 邓波: "防止消息队列阻塞的办法", 华南金融电脑, no. 06, 10 June 2003 (2003-06-10) * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10552372B2 (en) | Systems, methods, and computer-readable media for a fast snapshot of application data in storage | |
| US9507821B2 (en) | Mail indexing and searching using hierarchical caches | |
| US9811577B2 (en) | Asynchronous data replication using an external buffer table | |
| US8280917B1 (en) | Batching content management operations to facilitate efficient database interactions | |
| CN108984553B (en) | Caching method and device | |
| CN111198751A (en) | Service processing method and device | |
| CN110764796A (en) | Method and device for updating cache | |
| CN115455058A (en) | Cache data processing method and device, computer equipment and storage medium | |
| CN112748866A (en) | Method and device for processing incremental index data | |
| CN113010535B (en) | Cache data updating method, device, equipment and storage medium | |
| WO2020192663A1 (en) | Data management method and related device | |
| CN115658171A (en) | Method and system for solving dynamic refreshing of java distributed application configuration in lightweight mode | |
| CN113778321A (en) | Message processing method and device | |
| CN115469810A (en) | Data acquisition method, device, equipment and storage medium | |
| CN115617859A (en) | Data query method and device based on knowledge graph cluster | |
| CN107423375B (en) | Application program searching method and device | |
| CN111625500B (en) | File snapshot method and device, electronic equipment and storage medium | |
| CN110879818B (en) | Method, device, medium and electronic equipment for acquiring data | |
| CN112699116A (en) | Data processing method and system | |
| CN111209308B (en) | Method and device for optimizing distributed cache | |
| CN113760860A (en) | Data reading method and device | |
| CN115599712A (en) | Data caching method based on application system, device, equipment and medium thereof | |
| CN113868278A (en) | Data processing method, device and equipment | |
| CN116955364A (en) | Block chain-based data storage method, equipment and storage medium | |
| CN115587091A (en) | Data storage method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |