WO2019001256A1 - High-concurrency data processing method and apparatus, and computer readable storage medium - Google Patents

Abstract

The present application provides a high-concurrency data processing method, which is applied to a data processing apparatus. The high-concurrency data processing method comprises the following steps: receiving an access request sent by a client; storing the received access request into a message queue in a message manner; and processing messages in the message queue by invoking a preset quantity of threads, reading related data from a database, and feeding back the processing result to the client. When a server receives a great quantity of data access requests at the same time, the concurrent pressure of the server can be effectively reduced, and the availability and the throughput of an application system are ensured. The present application also provides a high-concurrency data processing apparatus and a computer readable storage medium.

Description

High concurrent data processing method, device and computer readable storage medium

This application claims priority to Chinese Patent Application No. 201710490374.X, entitled "High Concurrency Data Processing Method, Apparatus, and Computer Readable Storage Medium", which is filed on June 25, 2017, the entire disclosure of which is incorporated herein by reference. The content is incorporated herein by reference.

Technical field

The present application relates to data processing technologies, and in particular, to a high concurrent data processing method, apparatus, and computer readable storage medium.

Background technique

With the gradual wide-ranging and in-depth informationization, there are more and more cases of multi-user concurrent operation data, for example, online ticket purchase, shopping, etc., especially the Spring Festival ticket grab or time-limited snap-up business, when the sales time is reached, the server If the received user requests are too large, it will cause server thread congestion, database abnormal wait events, etc., the user request will not get a response, and even the entire system crashes. Therefore, there is a need for a highly concurrent data processing method, apparatus, and computer readable storage medium that can reduce the concurrency pressure of a server, thereby ensuring the availability and throughput of the application system.

Summary of the invention

The present application provides a high concurrent data processing method, apparatus, and computer readable storage medium, the main purpose of which is to reduce the concurrent pressure of the server, thereby ensuring the availability and throughput of the application system.

To achieve the above objective, the present application provides a high concurrent data processing method, which is applied to a data processing apparatus, and the method includes:

Receiving step: receiving an access request sent by the client;

Queue management step: storing the received access request as a message to the message queue; and

The first processing step: calling a preset number of threads to process the message in the message queue, reading the relevant data from the database, and feeding back the processing result to the client.

The present application also provides a high concurrent data processing method, which is applied to a data processing apparatus, and the method includes:

Receiving step: receiving an access request sent by the client;

Queue management step: storing the received access request in a message manner to the message queue;

Request merge step: merge the same message in the message queue with the same access request to form multiple message groups;

The second processing step: calling a preset number of threads to process the plurality of message groups, reading related data from the database, and feeding back the processing result to the client.

In addition, the present application provides a high concurrent data processing apparatus, the apparatus comprising a memory, a processor, and a high concurrent data processing system stored on the memory and operable on the processor, the high concurrent data processing system The method includes: a receiving module, a queue management module, and a first processing module.

When the processor executes the high concurrent data processing system, the following steps can be implemented:

Receiving step: receiving an access request sent by the client;

Queue management step: storing the received access request as a message to the message queue; and

The first processing step: calling a preset number of threads to process the message in the message queue, reading the relevant data from the database, and feeding back the processing result to the client.

The present application also provides a high concurrent data processing apparatus, the apparatus comprising a memory, a processor, and a high concurrent data processing system stored on the memory and operable on the processor, the high concurrent data processing system comprising : a receiving module, a queue management module, a request combining module, and a second processing module.

When the processor executes the high concurrent data processing system, the following steps can be implemented:

Receiving step: receiving an access request sent by the client;

Queue management step: storing the received access request in a message manner to the message queue;

Request merge step: merge the same message in the message queue with the same access request to form multiple message groups;

The second processing step: calling a preset number of threads to process the plurality of message groups, reading related data from the database, and feeding back the processing result to the client.

Moreover, the present application also provides a computer readable storage medium storing a high concurrent data processing system, the high concurrent data processing system being executable by at least one processor to implement the following steps:

Receiving step: receiving an access request sent by the client;

Queue management step: storing the received access request as a message to the message queue; and

The first processing step: invoking a preset number of threads to process the message in the message queue, reading the relevant data from the database, and feeding back the processing result to the client.

The application further provides a computer readable storage medium storing a high concurrent data processing system, the high concurrent data processing system being executable by at least one processor to implement the following steps:

Receiving step: receiving an access request sent by the client;

Queue management step: storing the received access request in a message manner to the message queue;

Request merge step: merge the same message in the message queue with the same access request to form multiple message groups;

The second processing step: calling a preset number of threads to process the plurality of message groups, reading related data from the database, and feeding back the processing result to the client.

Compared with the prior art, the high concurrent data processing method, apparatus and computer readable storage medium provided by the present application use an asynchronous queue to manage client data access requests, merge the same data access requests, and call a preset number. The thread processes the message in the message queue, so that when the server receives a large amount of data access requests at the same time, the concurrent pressure of the server can be effectively reduced, the time for the end user to participate in the interface interaction is saved, and the computing resources of the server are fully utilized. Reduce the interaction between users under high load to ensure the availability and throughput of the application system.

DRAWINGS

1 is a schematic diagram of a preferred embodiment of a high concurrent data processing apparatus of the present application.

2 is a block diagram of a preferred embodiment of a high concurrent data processing system of the present application.

3 is a block diagram of another preferred embodiment of the high concurrent data processing system of the present application.

4 is a flow chart of a preferred embodiment of a high concurrent data processing method of the present application.

FIG. 5 is a flowchart of another preferred embodiment of a high concurrent data processing method of the present application.

Reference mark:

1	High concurrent data processing device
2	The internet
3	Client
4	database

11	Memory
12	processor
13	Communication bus
10	High concurrent data processing system
110	Receiving module
120	Queue management module
130	First processing module
140	Request merge module
150	Second processing module

The implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings.

Detailed ways

It is understood that the specific embodiments described herein are merely illustrative of the application and are not intended to be limiting.

As shown in FIG. 1, it is an exemplary diagram of a preferred embodiment of the high concurrent data processing apparatus 1 of the present application.

The high concurrent data processing apparatus 1 includes a memory 11, a processor 12, a communication bus 13, and a high concurrent data processing system 10 stored on the memory 11 and operable on the processor 12, the apparatus 1 The network 2 is connected to one or more clients 3 and is communicatively coupled to the database 4 via a communication bus 13.

The network 2 can be a network of a local area network, a wide area network, a metropolitan area network, etc., and can be a wired network or a wireless network.

The client 3 can be a desktop computer, a notebook, a tablet, a mobile phone, or other terminal device that can communicate with the device 1 via the network 2.

The processor 12 can be a Central Processing Unit (CPU), microprocessor or other data processing chip for running the high concurrent data processing system 10 stored in the memory 11, such as executing high concurrent data processing programs and the like.

The memory 11 stores the program code of the high concurrent data processing system 10, the data of the client that issues the data request, such as customer identity information, account information, and data request information. The memory 11 connected to the high concurrent data processing device 1 may be an internal storage unit of the terminal, such as a hard disk or a memory of the terminal, or may be a memory device independent of the aforementioned processor 12 connected through a network.

The memory 11 may include at least one type of storage medium including a flash memory, a hard disk, a multimedia card, a card type memory (eg, SD or DX memory, etc.), a random access memory (RAM), a static random access memory. (SRAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, and the like. The processor 12 reads and executes the program code of the high concurrent data processing system 10 from the memory 11 to provide the following functions of the high concurrent data processing system 10.

Figure 1 shows only the high concurrent data processing device 1 with components 11-13 and a high concurrent data processing system, but it should be understood that not all illustrated components may be implemented, alternative implementations may be more or less s component.

2 is a block diagram of a preferred embodiment of the high concurrent data processing system 10 of the present application.

In the embodiment, the high concurrent data processing system 10 includes a receiving module 110, a queue management module 120, and a first processing module 130.

The receiving module 110 is configured to receive an access request sent by the client 3. For example, when a customer participates in an online online rushing activity of a merchant through the client 3, the receiving module 110 receives n snapping requests issued by the client.

The queue management module 120 is configured to store the received access request in a message manner to the message queue. For example, the queue management module 120 stores the n snap requests in chronological order and stores them in a message queue.

The first processing module 130 calls a preset number of threads to process the message in the message queue, reads the related data from the database 4, and feeds back the processing result to the client 3. For example, the first processing module 130 invokes a preset number of threads, processes the previous snap request in the message queue, reads the information of the snapped product from the database 4, and feeds the processing result back to the client. 3.

3 is a block diagram of another preferred embodiment of the high concurrent data processing system 10 of the present application.

In this embodiment, the high concurrent data processing system 10 includes:

The receiving module 110 is configured to receive an access request sent by the client 3. For example, when the customer participates in the online panning activity of a merchant through the client 3, the receiving module 110 receives n snap requests issued by the client.

The queue management module 120 is configured to store the received access request in a message manner to the message queue. For example, the queue management module 120 stores the n snap requests in chronological order and stores them in a message queue.

The request merge module 140 is configured to merge the same message in the message queue to form a plurality of message groups. For example, among the n snapping requests, there are 20 snapping requests to snap up product A, there are 10 snapping products B, and other snapping requests without the same request each constitute a message group, then the request combining module 140 snaps up the 20 pieces. The snap request of product A is merged into message group C, and the snap requests of the 10 snapped products B are merged into message group D. If the time of the first snap request in the message group C is earlier than the first snap request in any other message group, the message of the message group C will be processed in preference to the messages of other message groups.

The second processing module 150 is configured to invoke a preset number of threads to process the plurality of message groups, read related data from the database 4, and feed back the processing result to the client 3. For example, if the message group C is arranged at the head end of the message queue, the second processing module 150 calls a preset number of threads, extracts the information of the message group C for processing, and reads the snapped A from the database 4. The information of the product, and the processing result is fed back to all the clients 3 that issued the snap request.

As shown in FIG. 4, it is a flowchart of a preferred embodiment of the high concurrent data processing method of the present application.

In this embodiment, the high concurrent data processing method includes the following steps:

Step S111: Receive a data access request sent by the client 3. For example, when a customer purchases a ticket through an application on the client 3 during the Spring Festival, the G6034 (Shenzhen-Yueyang) is taken as an example. When the ticketing time of the train is reached, the receiving module 110 instantaneously receives n ticket grab requests. When the client 3 makes a request, the client 3 interface displays "in queue".

Step S112: Store the received access request in a message manner to the message queue. For example, the queue management module 120 stores the n ticket rush requests in a chronological order in the form of a message to the message queue, wherein the information of each rushed train ticket represents a message in the message queue.

Step S113: Calling a preset number of threads to process the message in the message queue, reading the related data from the database 4, and feeding back the processing result to the client 3. For example, the first processing module 130 invokes a preset number of threads, processes the previous ticket rush request in the message queue, reads the remaining ticket information from the database 4, and feeds the processing result back to the client 3 .

As shown in FIG. 5, it is a flow chart of another preferred embodiment of the high concurrent data processing method of the present application.

In this embodiment, the high concurrent data processing method includes the following steps:

Step S111: Receive a data access request sent by the client 3. For example, when a customer purchases a ticket on an APP through the client 3 during the Spring Festival, the G6034 (Shenzhen-Yueyang) is taken as an example. When the ticketing time of the train is reached, the receiving module 110 instantaneously receives n ticket grab requests. When the client 3 makes a request, the client 3 interface displays "in queue".

Step S112: Store the received access request in a message manner to the message queue. For example, the queue management module 120 stores the n snap requests in chronological order and stores them in a message queue. For example, the queue management module 120 stores the n ticket rush requests in a chronological order in the form of a message to the message queue, wherein the information of each rushed train ticket represents a message in the message queue.

Step S114: Combine the messages with the same access request in the message queue to form a plurality of message groups. For example, among the n ticket rushing requests, 20 robbed requests to snap up the second-class seat of Shenzhen to Yueyang, and 10 rushed to buy the first-class seat of Shenzhen to Zhuzhou. Then, the request merging module 140 snapped the 20 pieces to Shenzhen. The snap request to the second-class seat of Yueyang is merged into message group E, and the snap-up requests of 10 snapped products B are merged into message group F, and the other non-identical grab requests respectively form a message group, if the message group E is the first The initiation time of the ticket rushing request is earlier than the initiation time of the first rush ticket request in any other message group, and the message in the message group E will be processed in preference to the messages of other message groups.

Step S115: Calling a preset number of threads to process the plurality of message groups, reading related data from the database 4, and feeding back the processing result to the client 3. For example, if the message group E is arranged at the head end of the message queue, the second processing module 150 calls a preset number of threads, extracts the information of the message group E for processing, and reads the remaining ticket information from the database 4. The processing result is fed back to all clients 3 that issued the snap request.

Moreover, the application also includes a computer readable storage medium storing a high concurrent data processing system 10.

In the present embodiment, the high concurrent data processing system 10 is executed by at least one processor 12 to implement the following steps:

Step S111: Receive an access request sent by the client 3. For example, when a customer participates in an online online rushing activity of a merchant through the client 3, the receiving module 110 receives n snapping requests issued by the client.

Step S112: Store the received access request in a message manner to the message queue. For example, the queue management module 120 stores the n snap requests in chronological order and stores them in a message queue.

Step S113: Calling a preset number of threads to process the message, reading related data from the database 4, and feeding back the processing result to the client 3. For example, the first processing module 140 extracts and analyzes the key information of the first ticket rush request in the message queue, reads the remaining ticket information from the database 4, and feeds back the processing result to the client 3.

In another preferred embodiment, the high concurrent data processing system 10 is executed by at least one processor 12 to implement the following steps:

Step S111: Receive an access request sent by the client 3. For example, when a customer participates in an online online rushing activity of a merchant through the client 3, the receiving module 110 receives n snapping requests issued by the client.

Step S112: Store the received access request in a message manner to the message queue. For example, the queue management module 120 stores the n snap requests in chronological order and stores them in a message queue.

Step S114: Combine the messages with the same access request in the message queue to form a plurality of message groups. For example, among the n snapping requests, there are 20 snapping requests to snap up the product A, and 10 snapping the product B. Then, the merging module 140 merges the snapping requests of the 20 snapped products A into the message group C, the 10 The snap-up request for snapping up product B is merged into message group D. If the time of the first snap request in the message group C is earlier than the first snap request in the message group D, the message in the message group C will be processed in preference to the message of the message group D.

Step S115: Calling a preset number of threads to process the plurality of message groups, reading related data from the database 4, and feeding back the processing result to the client 3. For example, if the message group C is arranged at the head end of the message queue, the second processing module 150 extracts the key information of the message group C for processing, and reads the information of the snapped A product from the database 4, and feedback. To all clients 3 that issued the snap request.

It is to be understood that the term "comprises", "comprising", or any other variants thereof, is intended to encompass a non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes those elements. It also includes other elements that are not explicitly listed, or elements that are inherent to such a process, method, article, or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device that comprises the element.

The "one embodiment" and "another embodiment" mentioned in the embodiment may be the same embodiment or different embodiments.

It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to be limiting, although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technology of the present application can be applied. Modifications or equivalents are made without departing from the spirit and scope of the technical solutions of the present application.

Claims (20)

1. A high concurrent data processing method is applied to a data processing device, the method comprising:

   Receiving step: receiving an access request sent by the client;

   Queue management step: storing the received access request as a message to the message queue; and

   The first processing step: calling a preset number of threads to process the message in the message queue, reading the relevant data from the database, and feeding back the processing result to the client.
2. The high concurrent data processing method according to claim 1, wherein the messages in the message queue are stored in chronological order.
3. The high concurrent data processing method according to claim 1, wherein said first processing step processes the messages in the message queue in a first in first out order.
4. The high concurrent data processing method according to claim 2, wherein said first processing step processes the messages in the message queue in a first in first out order.
5. A high concurrent data processing method is applied to a data processing device, the method comprising:

   Receiving step: receiving an access request sent by the client;

   Queue management step: storing the received access request in a message manner to the message queue;

   Request merge step: merge the same message in the message queue with the same access request to form multiple message groups;

   The second processing step: calling a preset number of threads to process the plurality of message groups, reading related data from the database, and feeding back the processing result to the client.
6. The high concurrent data processing method according to claim 5, wherein the messages in the message queue are stored in chronological order.
7. The high concurrent data processing method according to claim 5, wherein said second processing step processes the messages in the message queue in a first in first out order.
8. The high concurrent data processing method according to claim 6, wherein said second processing step processes the messages in the message queue in a first in first out order.
9. A high concurrent data processing apparatus, comprising: a memory, a processor, and a high concurrent data processing system stored on the memory and operative on the processor, the processor executing the When a concurrent data processing system is high, the following steps can be implemented:

   Receiving step: receiving an access request sent by the client;

   Queue management step: storing the received access request as a message to the message queue; and

   The first processing step: calling a preset number of threads to process the message in the message queue, reading the relevant data from the database, and feeding back the processing result to the client.
10. The high concurrent data processing apparatus according to claim 9, wherein the messages in the message queue are stored in chronological order.
11. The high concurrent data processing apparatus according to claim 9, wherein said first processing step processes the messages in the message queue in a first in first out order.
12. The high concurrent data processing apparatus according to claim 10, wherein said first processing step processes the messages in the message queue in a first in first out order.
13. A high concurrent data processing apparatus, comprising: a memory, a processor, and a high concurrent data processing system stored on the memory and operative on the processor, the processor executing the When a concurrent data processing system is high, the following steps can be implemented:

    Receiving step: receiving an access request sent by the client;

    Queue management step: storing the received access request in a message manner to the message queue;

    Request merge step: merge the same message in the message queue with the same access request to form multiple message groups;

    The second processing step: calling a preset number of threads to process the plurality of message groups, reading related data from the database, and feeding back the processing result to the client.
14. The high concurrent data processing apparatus according to claim 13, wherein the messages in the message queue are stored in chronological order.
15. The high concurrent data processing apparatus according to claim 13, wherein said second processing step processes the messages in the message queue in a first in first out order.
16. The high concurrent data processing apparatus of claim 14 wherein said second processing step processes the messages in the message queue in a first in first out order.
17. A computer readable storage medium, characterized in that the computer readable storage medium stores a high concurrent data processing system, the high concurrent data processing system being executable by at least one processor to implement the following steps:

    Receiving step: receiving an access request sent by the client;

    Queue management step: storing the received access request as a message to the message queue; and

    The first processing step: calling a preset number of threads to process the message in the message queue, reading the relevant data from the database, and feeding back the processing result to the client.
18. The computer readable storage medium according to claim 17, wherein the messages in the message queue are stored in chronological order, and the first processing step processes the messages in the message queue to comply with the first in first out order of.
19. A computer readable storage medium, characterized in that the computer readable storage medium stores a high concurrent data processing system, the high concurrent data processing system being executable by at least one processor to implement the following steps:

    Receiving step: receiving an access request sent by the client;

    Queue management step: storing the received access request in a message manner to the message queue;

    Request merge step: merge the same message in the message queue with the same access request to form multiple message groups;

    The second processing step: calling a preset number of threads to process the plurality of message groups, reading related data from the database, and feeding back the processing result to the client.
20. The computer readable storage medium of claim 19, wherein the messages in the message queue are stored in chronological order, and the second processing step processes the messages in the message queue to follow a first in first out order of.

Images