CN111464582A - High-concurrency data system, data processing method, storage medium and computer equipment - Google Patents
High-concurrency data system, data processing method, storage medium and computer equipment Download PDFInfo
- Publication number
- CN111464582A CN111464582A CN201910059280.6A CN201910059280A CN111464582A CN 111464582 A CN111464582 A CN 111464582A CN 201910059280 A CN201910059280 A CN 201910059280A CN 111464582 A CN111464582 A CN 111464582A
- Authority
- CN
- China
- Prior art keywords
- service
- service data
- data
- layer
- storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 27
- 238000012545 processing Methods 0.000 claims abstract description 52
- 238000000034 method Methods 0.000 claims abstract description 29
- 230000015654 memory Effects 0.000 claims description 39
- 239000003999 initiator Substances 0.000 claims description 17
- 230000008569 process Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 239000010410 layer Substances 0.000 description 166
- 238000010586 diagram Methods 0.000 description 6
- 238000005034 decoration Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000026676 system process Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000011229 interlayer Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/60—Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
- H04L67/63—Routing a service request depending on the request content or context
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a high-concurrency data system, a data processing method, a data processing device, a storage medium and computer equipment, wherein the method comprises the following steps: acquiring a first access frequency of a plurality of service data in a high concurrency system; acquiring a first preset number of service data from the plurality of service data according to the first access frequency of each service data; storing the first preset amount of service data to a service layer of the high concurrency system, wherein the service layer is used for processing service requests; and storing the service data except the first preset amount of service data in the plurality of service data into a data layer of the high concurrency system for storing the service data. The method can relieve the server pressure caused by high concurrent access of the client to the server, and improve the efficiency of high concurrent data access of the system.
Description
Technical Field
The invention relates to the technical field of high-concurrency systems, in particular to a high-concurrency data system, a data processing method, a data processing device, a storage medium and computer equipment.
Background
With the development of internet services, the number of users on an internet platform is gradually huge, and the traffic volume, access volume and data traffic volume of a user system are also rapidly increased, so that a server not only can support concurrent access of a large number of users, but also can efficiently process transaction demands submitted by the users with low delay.
Because the size of the data packet of the HTTP protocol is much larger than that of the data packet of the TCP/IP, after the client sends the transmission request, the response of the server can be obtained only after the transmission of the large message data is completed, and the response time cannot be guaranteed, so that the concurrent server based on the HTTP protocol has a greater difficulty in processing the concurrent service.
Disclosure of Invention
The invention provides a high-concurrency data system, a data processing method, a data processing device, a storage medium and computer equipment, which are used for relieving server pressure caused by high-concurrency access of a client to a server and improving the efficiency of high-concurrency data access of the system.
The present invention provides the following scheme:
a data processing method based on a high concurrency system comprises the following steps: acquiring a first access frequency of a plurality of service data in a high concurrency system; acquiring a first preset number of service data from the plurality of service data according to the first access frequency of each service data; storing the first preset amount of service data to a service layer of the high concurrency system, wherein the service layer is used for processing service requests; and storing the service data except the first preset amount of service data in the plurality of service data into a data layer of the high concurrency system for storing the service data.
In an embodiment, the method further comprises: confirming that the target service processing corresponding to the target service data in the plurality of service data is finished; acquiring data information after the target service processing is finished; and modifying the target service data in the service layer or the target service data in the data layer according to the data information.
In an embodiment, the acquiring a first preset number of service data according to a first access frequency of each service data includes: acquiring a first preset number of service data of which the first access frequency reaches a preset value according to the first access frequency of each service data; or acquiring a first preset number of service data with the first access frequency ranked at the top according to the first access frequency of each service data.
In an embodiment, after storing the plurality of service data, the method further includes: acquiring a second access frequency of the plurality of service data within a time period which is preset time length away from the current time point; acquiring a second preset number of service data from the plurality of service data according to the second access frequency of each service data; removing the service data which are not in the second preset number of service data from the service layer in the first preset number of service data, and storing the removed service data to the data layer; and removing the service data which do not belong to the first preset number of service data from the second preset number of service data from the data layer, and storing the service data in the service layer.
In an embodiment, after storing, in the data layer of the high concurrency system, service data other than the first preset number of service data in the plurality of service data, the storing further includes: when confirming that a service data acquisition request is received, judging whether service data corresponding to the service data acquisition request exists in the service layer; if yes, reading the corresponding service data from the service layer, and feeding the service data back to the initiator of the service data acquisition request; if not, reading the corresponding service data from the data layer, and feeding the service data back to the initiator of the service data acquisition request.
In an embodiment, after storing the first preset amount of service data to the service layer of the high concurrency system for processing the service request, the method further includes: when the service data acquisition request is confirmed to be received, processing the service data acquisition request through a single-process multithreading mode; acquiring a thread for processing the service data acquisition request according to the multithreading mode of the single process, judging whether service data corresponding to the service data acquisition request exists in the service layer or not through the thread, reading the corresponding service data from the service layer, and feeding the service data back to an initiator of the service data acquisition request; or, reading the corresponding service data from the data layer, and feeding back the service data to the initiator of the service data acquisition request.
A high concurrency data system comprises an interface layer, a service layer and a data layer; the interface layer is used for receiving concurrent service requests; the service layer is used for receiving the concurrent service request forwarded by the interface layer, and reading service data corresponding to the concurrent service request from the service layer, or reading the service data corresponding to the concurrent service request from the data layer; the service layer is further configured to store a first preset number of service data, which is obtained according to a first access frequency of each service data, in the plurality of service data of the high concurrency data system; the data layer is used for storing the service data except the first preset number of service data in the plurality of service data; and the service layer is also used for feeding back the read service data corresponding to the concurrent service request to the initiator of the concurrent service request through the interface layer.
A high concurrency system based data processing apparatus comprising: the first acquisition module is used for acquiring a first access frequency of a plurality of service data in the high-concurrency system; the second acquisition module is used for acquiring a first preset number of service data according to the first access frequency of each service data; the first storage module is used for storing the first preset amount of service data to a service layer of the high concurrency system, wherein the service layer is used for processing service requests; and the second storage module is used for storing the service data except the first preset number of service data in the plurality of service data into a data layer of the high concurrency system, wherein the data layer is used for storing the service data.
A storage medium having a computer program stored thereon; the computer program is adapted to be loaded by a processor and to perform the data processing method based on a high concurrency system according to any of the above embodiments.
A computer device, comprising: one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more application programs being configured to perform the high concurrency system based data processing method according to any of the embodiments described above.
In the data processing method based on the high concurrency system provided in the above embodiment, the server system obtains the first preset number of service data from the plurality of service data according to the first access frequency of each service data in the system, and specifically, obtains the preset number of service data with the access frequency ranked at the top. Further, the first preset amount of service data is stored in the service layer, and other service data is stored in the data layer. Therefore, when the client initiates a service requirement, the client requests to acquire the server system data. When the system service layer processes the client request, if the required service data is stored in the service layer, the system can directly read the relevant data from the service layer and feed the relevant data back to the client. And reading the relevant database from the data layer only when the service data is not stored in the service layer. According to the data processing method based on the high-concurrency system, the system service data are respectively stored in the service layer and the data layer according to the use frequency, so that the server pressure caused by high-concurrency access of the client to the server is relieved, and the efficiency of high-concurrency data access of the system is improved.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the present invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
FIG. 1 is a system architecture diagram of an embodiment of a highly concurrent data system according to the present invention;
FIG. 2 is a topology diagram in one embodiment of the high concurrency data system of FIG. 1;
FIG. 3 is a flowchart of a method in an embodiment of a data processing method based on a high concurrency system according to the present invention;
FIG. 4 is a partial flowchart of a data processing method according to another embodiment of the present invention;
FIG. 5 is a partial flowchart of a data processing method according to another embodiment of the present invention;
fig. 6 is a flowchart of user login in an embodiment of a data processing method based on a high concurrency system according to the present invention;
FIG. 7 is a block diagram of an embodiment of a data processing apparatus based on a high concurrency system according to the present invention;
fig. 8 is a schematic structural diagram of an embodiment of a computer device provided in the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are illustrative only and should not be construed as limiting the invention.
It will be understood by those skilled in the art that, unless otherwise specified, the singular forms "a", "an", "the" and "the" may include the plural forms as well, and the "first" and "second" used herein are only used to distinguish one technical feature from another and are not intended to limit the order, number, etc. of the technical features. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The invention provides a high concurrency data system. The system can be a user system and is used for processing services such as user requirements and the like. Of course, other data processing data systems are possible. In one embodiment, as shown in fig. 1, the system architecture of the high-concurrency data system mainly includes a data layer, a service layer and an interface layer.
And the interface layer is used for receiving the concurrent service request. Specifically, the interface layer provides a user-oriented HTTP interface for receiving a concurrent service request initiated by a user and processing a part of the service. The processed services include receiving user requests, displaying data submitted back, providing access to applications for clients, and the like.
The service layer is used for receiving the concurrent service request forwarded by the interface layer and reading service data corresponding to the concurrent service request from the service layer or reading the service data corresponding to the concurrent service request from the data layer; the service layer is further configured to store a first preset number of service data, which is obtained according to the first access frequency of each service data, in the plurality of service data of the high concurrency data system. And the service layer is also used for feeding back the read service data corresponding to the concurrent service request to the initiator of the concurrent service request through the interface layer. Specifically, after receiving a concurrent service request initiated by a user, the interface layer forwards the request to the service layer. The service layer is responsible for accelerating the processing of service requests of the interface layer and operating the data layer, wherein the processing comprises caching of processing data, communication of data, storage of data and the like.
The data layer is used for storing the service data except the service data stored by the service layer in the high concurrency data system, and particularly comprises an operation layer for the database, which is mainly used for realizing the operations of inquiring, adding, deleting, modifying and the like of user data in a data table and providing data services for the service layer or an interface layer, wherein the data layer comprises a plurality of database units, and a core database adopts MongoDB which has a better large concurrency coping strategy compared with the traditional database MySQ L.
The services illustrated in fig. 1 include access services of products such as Minik, Minishow, and third-party products, and user data basic services such as user registration, login, and binding, and login access services of a third-party platform include a WeChat service, a Payment treasure, and a payment verification service of a payment platform such as a WeChat service, a payment treasure.
To better illustrate the above-described high concurrency data system, the following explanation is made in conjunction with a topological diagram of the system. As shown in fig. 2, the high concurrency data system is a user system.
The service layer of the user system is a user data processing module used for processing user data. User data is distributed by a central server group (ac), where each server is permanently connected to a database MongoDB of the data layer of the user system. Each server within the ac is responsible for handling the respective login platform, as well as user services. The system comprises a logout interface, a data acquisition interface, a registration interface, a verification interface/Token mechanism, a data updating interface and a verification interface, wherein the corresponding servers belong to a secondary server (as for short). The secondary server corresponds to an interface layer of a user system and is used for solving the user requirements of the B end (client) and supporting the user behaviors of the B end. The outermost business server is an external product or a third-party platform (ap for short), and the business server solves the requirements of common users and comprises a public server, a payment premium server and the like.
Wherein, a TCP connection mode is adopted between the service layer and the interface layer (as-ac) to bear the data pressure of the intranet communication of the server. An external product or a third-party platform and an interface interlayer (ap-as) adopt an HTTP request mode, bear basic data operations of a common user, and comprise a series of behaviors of registering, logging in, acquiring user information and the like.
Based on the high concurrency data system, the invention also provides a data processing method based on the high concurrency system. In one embodiment, as shown in fig. 3, the data processing method based on the high concurrency system includes the following steps:
s100, acquiring a first access frequency of a plurality of service data in a high concurrency system.
In this embodiment, the high concurrency system obtains the first access frequency of the plurality of service data in the system. The first access frequency may be an access frequency of the user accessing each service data by the current time. The service data includes data corresponding to all services in the system. Specifically, the service data may be service data corresponding to all users. Such as user registration data, user access data, user login data, etc.
S200, acquiring a first preset amount of service data from the plurality of service data according to the first access frequency of each service data.
In this embodiment, the first access frequencies corresponding to the plurality of service data are different. The system may screen a first preset number of service data from the plurality of service data according to the first access frequency of each service data. The first preset amount may be determined according to a capacity of the service layer to store data.
In one embodiment, step S200 includes: acquiring a first preset number of service data of which the first access frequency reaches a preset value according to the first access frequency of each service data; or acquiring a first preset number of service data with the first access frequency ranked at the top according to the first access frequency of each service data.
In this embodiment, the system sets a preset value for a first access frequency of traffic data. And when the first access frequency of the service data reaches a preset value, listing the service data into the service data with the first preset quantity. Or, the system may first obtain the first access frequency of each service data, rank the service data according to the first access frequency of each service data, and obtain a first preset number of service data with the first access frequency ranked higher.
S300, storing the first preset amount of service data to a service layer of the high concurrency system for processing service requests.
In this embodiment, the service layer of the high concurrency system is configured to receive the concurrent service request forwarded by the interface layer, read service data corresponding to the concurrent service request from the service layer or the data layer according to the concurrent service request, and further forward the read service data to the request initiator through the interface layer. The service layer is provided with a storage space and can be used for storing various types of service data. The system may store the obtained first preset amount of service data to a service layer of the high concurrency system for processing the service request. When the service layer receives a service request initiated by a user, if service data corresponding to the service request is stored in the service layer, the service data can be directly read from the service layer and fed back to the user side.
S400, storing the service data, except for the first preset amount of service data, in the plurality of service data into a data layer of the high concurrency system for storing the service data.
In this embodiment, the data layer of the high concurrency system is used for storing the service data except the first preset amount of service data in the plurality of service data of the system. The data layer may store most of the different types of traffic data in the system. The service layer can read corresponding service data from the data layer through the related calling interface.
In an embodiment, after step S300, the method further includes: when confirming that a service data acquisition request is received, judging whether service data corresponding to the service data acquisition request exists in the service layer; if yes, reading the corresponding service data from the service layer, and feeding the service data back to the initiator of the service data acquisition request; if not, reading the corresponding service data from the data layer, and feeding the service data back to the initiator of the service data acquisition request.
In this embodiment, when the high concurrency system receives a service data acquisition request for acquiring service data, which is initiated by a client user, it is determined whether the service data corresponding to the request is stored in a service layer. If yes, corresponding service data are directly called from the service layer, and the service data are fed back to the client. Therefore, the problem of overlong request response caused by accessing the data layer through the service layer can be avoided, and the server stress caused by simultaneously accessing the data layer by a plurality of concurrent requests can also be avoided.
In an embodiment, after step S300, the method further includes: when the service data acquisition request is confirmed to be received, processing the service data acquisition request through a single-process multithreading mode; acquiring a thread for processing the service data acquisition request according to the multithreading mode of the single process, judging whether service data corresponding to the service data acquisition request exists in the service layer or not through the thread, reading the corresponding service data from the service layer, and feeding the service data back to an initiator of the service data acquisition request; or, reading the corresponding service data from the data layer, and feeding back the service data to the initiator of the service data acquisition request.
In this embodiment, the system processes the service data acquisition request initiated by the client in a single-process multithreading mode. Specifically, a plurality of concurrent service requests of the system are processed through a single-process multithreading mode, the task waiting time is reasonably distributed, meanwhile, the shared memory is repeatedly utilized, and the multiprocess expenditure is reduced. By reasonably setting the service logic and reasonably utilizing the memory, the loss of the server can be reduced when high concurrent transactions are processed.
In the data processing method based on the high concurrency system provided in the above embodiment, the server system obtains the first preset number of service data from the plurality of service data according to the first access frequency of each service data in the system, and specifically, obtains the preset number of service data with the access frequency ranked at the top. Further, the first preset amount of service data is stored in the service layer, and other service data is stored in the data layer. Therefore, when the client initiates a service requirement, the client requests to acquire the server system data. When the system service layer processes the client request, if the required service data is stored in the service layer, the system can directly read the relevant data from the service layer and feed the relevant data back to the client. And reading the relevant database from the data layer only when the service data is not stored in the service layer. According to the data processing method based on the high-concurrency system, the system service data are respectively stored in the service layer and the data layer according to the use frequency, so that the server pressure caused by high-concurrency access of the client to the server is relieved, and the efficiency of high-concurrency data access of the system is improved.
In an embodiment, as shown in fig. 4, the data processing method based on the high concurrency system further includes:
and S500, confirming that the target service processing corresponding to the target service data in the plurality of service data is finished.
S600, acquiring data information after the target service processing is finished.
S700, modifying the target service data in the service layer or the target service data in the data layer according to the data information.
In this embodiment, when the system receives a service request from a user, the system performs target service processing according to target service data corresponding to the service request. And after the target service processing is confirmed to be finished, modifying the target service data in the service layer or the data layer according to the data information after the processing is finished. In a specific embodiment, the system reads corresponding service data from the service layer or the data layer according to the service request, and feeds back the service data to the user side. When the user terminal modifies the service data or the system modifies the service data according to the service request, the system returns the data processing result to the data layer or the service layer through the service layer or the interface layer to modify the storage information of the data.
In an embodiment, as shown in fig. 5, after storing the plurality of service data, the data processing method based on the high concurrency system further includes:
s410, acquiring a second access frequency of the plurality of service data within a time period which is preset time length away from the current time point.
S420, obtaining a second preset number of service data from the plurality of service data according to the second access frequency of each service data.
And S430, removing the service data which is not included in the second preset number of service data from the service layer in the first preset number of service data, and storing the removed service data in the data layer.
S440, removing, from the data layer, the service data that is not included in the service data of the first preset amount in the service data of the second preset amount, and storing the service data in the service layer.
In this embodiment, the high concurrency system continuously receives service requests initiated by the client user. However, as the service requirements of users change, the access frequency of service data in the system also changes. The service data stored by the service layer in the system also changes. In this embodiment, the system acquires a second access frequency of the plurality of service data within a time period of a preset duration from the current time point each time, and determines a second preset number of service data according to the second access frequency. The second preset number may be the same as or different from the first preset number. The first preset number and the second preset number are determined by the capacity of the service layer for storing service data. Further, the system adjusts the service data stored in the service layer and the data layer. And storing the service data of the second preset quantity by adopting the service layer, and storing the service data except the service data of the second preset quantity by adopting the data layer.
In order to further detail the data processing method based on the high concurrency system, the following description is made in conjunction with a specific implementation scenario:
in this implementation scenario, the high-concurrency system is a user system, and the service request initiates a registration or login request for the user. As shown in fig. 6, when a user initiates a registration or login request, the steps performed by the user system are:
1. the user initiates a registration or login request, and the interface layer receives login authentication information input by the user and sends the login authentication information to the user data processing center (service layer).
2. And when the user data processing center (service layer) receives the login authentication information, authenticating the user and judging whether the user exists.
3. When judging that the user exists, executing the following step 4; and when judging that the user does not exist, creating a new user account and executing the following step 5.
4. And further judging whether the user data is updated, if so, verifying the user account and finishing data updating, and if not, directly verifying and logging in the user account.
5. The user data processing center (service layer) returns response data to the interface layer, and the user is allowed to access and log in. Meanwhile, the user data processing center (service layer) synchronizes the updated user data to the data layer, and modifies the data storage information.
When a user initiates a login request, task requests triggered by a plurality of users are added into a request thread in a system process queue, different service requests are distributed to corresponding service threads in a service layer or an interface layer, and a processing result is returned to the interface layer after the service layer or the interface layer executes corresponding service processing. Meanwhile, the service layer or the interface layer returns the data processing result to the data layer to modify the data storage information.
There are two important factors in performing the above steps:
1. and storing the user data which needs to be used for a long time in the memory of the service layer. When the business needs to use the data, the memory can be directly accessed to access the data. And if the required data is not stored in the business layer, accessing the database of the data layer to inquire the required data. The data are respectively stored in the service layer and the data layer according to the use frequency, so that the pressure caused by high concurrency of a user can be relieved.
2. And classifying the user data, wherein each user has a unique corresponding UID, and each UID is a plurality of IDs generated by logging in through different channels. The data structure of the user data is subject to the red and black trees, so that the quick query is facilitated.
Furthermore, concurrent service requests are processed through a single-process multi-thread mode, the task waiting time is reasonably distributed, meanwhile, the shared memory is repeatedly utilized, and the multi-process overhead is reduced. By reasonably setting the service logic and reasonably utilizing the memory, the loss of the server can be reduced when high concurrent transactions are processed. Through testing, under the condition of i5 and 4-core cpu, the data packet bearing is already broken through 10w/s, and the data transmission reaches 12 m/s. The user has been >1w/s per second concurrency. The test record takes the registration login as a test case. Under the pressure, the server runs well and is not fully loaded, which proves that the server can provide stable support and guarantee for the high concurrent service requirements of users.
The invention also provides a data processing device based on the high concurrency system. In one embodiment, as shown in fig. 7, the data processing apparatus based on the high concurrency system includes a first obtaining module 10, a second obtaining module 20, a first storing module 30 and a second storing module 40.
The first obtaining module 10 is configured to obtain a first access frequency of multiple service data in a high concurrency system. In this embodiment, the high concurrency system obtains the first access frequency of the plurality of service data in the system. The first access frequency may be an access frequency of the user accessing each service data by the current time. The service data includes data corresponding to all services in the system. Specifically, the service data may be service data corresponding to all users. Such as user registration data, user access data, user login data, etc.
The second obtaining module 20 is configured to obtain a first preset number of service data according to the first access frequency of each service data. In this embodiment, the first access frequencies corresponding to the plurality of service data are different. The system may screen a first preset number of service data from the plurality of service data according to the first access frequency of each service data. The first preset amount may be determined according to a capacity of the service layer to store data.
The first storage module 30 is configured to store the first preset amount of service data to a service layer of the high concurrency system, where the service layer is used for processing service requests. In this embodiment, the service layer of the high concurrency system is configured to receive the concurrent service request forwarded by the interface layer, read service data corresponding to the concurrent service request from the service layer or the data layer according to the concurrent service request, and further forward the read service data to the request initiator through the interface layer. The service layer is provided with a storage space and can be used for storing various types of service data. The system may store the obtained first preset amount of service data to a service layer of the high concurrency system for processing the service request. When the service layer receives a service request initiated by a user, if service data corresponding to the service request is stored in the service layer, the service data can be directly read from the service layer and fed back to the user side.
The second storage module 40 is configured to store, in the multiple service data, service data except for the first preset number of service data into a data layer of the high concurrency system, where the data layer is used for storing service data. In this embodiment, the data layer of the high concurrency system is used for storing the service data except the first preset amount of service data in the plurality of service data of the system. The data layer may store most of the different types of traffic data in the system. The service layer can read corresponding service data from the data layer through the related calling interface.
In other embodiments, each module in the data processing apparatus based on a high concurrency system provided by the present invention is further used for executing the operation executed corresponding to each step in the data processing method based on a high concurrency system described in the present invention, and a detailed description thereof is omitted here.
The invention also provides a storage medium. The storage medium having stored thereon a computer program; when being executed by a processor, the computer program realizes the data processing method based on the high concurrency system in any one of the above embodiments. The storage medium may be a memory. For example, internal memory or external memory, or both. The memory may comprise read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, or random access memory. The external memory may include a hard disk, a floppy disk, a ZIP disk, a usb-disk, a magnetic tape, etc. The storage media disclosed herein include, but are not limited to, these types of memories. The disclosed memory is by way of example only and not by way of limitation.
The invention also provides computer equipment. A computer device comprising: one or more processors; a memory; one or more applications. Wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the high concurrency system based data processing method of any of the above embodiments.
Fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present invention. The computer device in this embodiment may be a server, a personal computer, and a network device. As shown in fig. 8, the apparatus includes a processor 803, a memory 805, an input unit 807, and a display unit 809. Those skilled in the art will appreciate that the device configuration means shown in fig. 8 do not constitute a limitation of all devices and may include more or less components than those shown, or some components in combination. The memory 805 may be used to store the application program 801 and various functional modules, and the processor 803 executes the application program 801 stored in the memory 805, thereby performing various functional applications of the device and data processing. The memory may be internal or external memory, or include both internal and external memory. The memory may comprise read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, or random access memory. The external memory may include a hard disk, a floppy disk, a ZIP disk, a usb-disk, a magnetic tape, etc. The disclosed memory includes, but is not limited to, these types of memory. The disclosed memory is by way of example only and not by way of limitation.
The input unit 807 is used to receive input of signals and keywords input by a user. The input unit 807 may include a touch panel and other input devices. The touch panel can collect touch operations of a user on or near the touch panel (for example, operations of the user on or near the touch panel by using any suitable object or accessory such as a finger, a stylus and the like) and drive the corresponding connecting device according to a preset program; other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., play control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like. The display unit 809 may be used to display information input by the user or information provided to the user and various menus of the computer apparatus. The display unit 809 may take the form of a liquid crystal display, an organic light emitting diode, or the like. The processor 803 is a control center of the computer device, connects various parts of the entire computer using various interfaces and lines, and performs various functions and processes data by operating or executing software programs and/or modules stored in the memory 805 and calling data stored in the memory.
In one embodiment, the device includes one or more processors 803, and one or more memories 805, one or more applications 801. Wherein the one or more application programs 801 are stored in the memory 805 and configured to be executed by the one or more processors 803, the one or more application programs 801 configured to perform the high concurrency system based data processing method described in the above embodiments.
In addition, functional units in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or two or more units are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium.
Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer readable storage medium, and the storage medium may include a memory, a magnetic disk, an optical disk, or the like.
The foregoing is only a partial embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
It should be understood that each functional unit in the embodiments of the present invention may be integrated into one processing module, each unit may exist alone physically, or two or more units may be integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.
The foregoing is only a partial embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A data processing method based on a high concurrency system is characterized by comprising the following steps:
acquiring a first access frequency of a plurality of service data in a high concurrency system;
acquiring a first preset number of service data from the plurality of service data according to the first access frequency of each service data;
storing the first preset amount of service data to a service layer of the high concurrency system, wherein the service layer is used for processing service requests;
and storing the service data except the first preset amount of service data in the plurality of service data into a data layer of the high concurrency system for storing the service data.
2. The method of claim 1, further comprising:
confirming that the target service processing corresponding to the target service data in the plurality of service data is finished;
acquiring data information after the target service processing is finished;
and modifying the target service data in the service layer or the target service data in the data layer according to the data information.
3. The method of claim 1, wherein the obtaining a first preset number of service data according to the first access frequency of each service data comprises:
acquiring a first preset number of service data of which the first access frequency reaches a preset value according to the first access frequency of each service data; or the like, or, alternatively,
and acquiring a first preset number of service data with the first access frequency ranked at the top according to the first access frequency of each service data.
4. The method of claim 1, wherein after storing the plurality of service data, further comprising:
acquiring a second access frequency of the plurality of service data within a time period which is preset time length away from the current time point;
acquiring a second preset number of service data from the plurality of service data according to the second access frequency of each service data;
removing the service data which are not in the second preset number of service data from the service layer in the first preset number of service data, and storing the removed service data to the data layer;
and removing the service data which do not belong to the first preset number of service data from the second preset number of service data from the data layer, and storing the service data in the service layer.
5. The method according to claim 1, wherein after storing the service data of the plurality of service data except the first preset amount of service data into a data layer of the high concurrency system for storing service data, the method further comprises:
when confirming that a service data acquisition request is received, judging whether service data corresponding to the service data acquisition request exists in the service layer;
if yes, reading the corresponding service data from the service layer, and feeding the service data back to the initiator of the service data acquisition request;
if not, reading the corresponding service data from the data layer, and feeding the service data back to the initiator of the service data acquisition request.
6. The method of claim 1, wherein after storing the first preset amount of service data in a service layer of the high concurrency system for processing service requests, further comprising:
when the service data acquisition request is confirmed to be received, processing the service data acquisition request through a single-process multithreading mode;
acquiring a thread for processing the service data acquisition request according to the multithreading mode of the single process, judging whether service data corresponding to the service data acquisition request exists in the service layer or not through the thread, reading the corresponding service data from the service layer, and feeding the service data back to an initiator of the service data acquisition request; or, reading the corresponding service data from the data layer, and feeding back the service data to the initiator of the service data acquisition request.
7. A high concurrency data system is characterized by comprising an interface layer, a service layer and a data layer;
the interface layer is used for receiving concurrent service requests;
the service layer is used for receiving the concurrent service request forwarded by the interface layer, and reading service data corresponding to the concurrent service request from the service layer, or reading the service data corresponding to the concurrent service request from the data layer; the service layer is further configured to store a first preset number of service data, which is obtained according to a first access frequency of each service data, in the plurality of service data of the high concurrency data system;
the data layer is used for storing the service data except the first preset number of service data in the plurality of service data;
and the service layer is also used for feeding back the read service data corresponding to the concurrent service request to the initiator of the concurrent service request through the interface layer.
8. A data processing apparatus based on a high concurrency system, comprising:
the first acquisition module is used for acquiring a first access frequency of a plurality of service data in the high-concurrency system;
the second acquisition module is used for acquiring a first preset number of service data according to the first access frequency of each service data;
the first storage module is used for storing the first preset amount of service data to a service layer of the high concurrency system, wherein the service layer is used for processing service requests;
and the second storage module is used for storing the service data except the first preset number of service data in the plurality of service data into a data layer of the high concurrency system, wherein the data layer is used for storing the service data.
9. A storage medium, characterized in that a computer program is stored thereon; the computer program is adapted to be loaded by a processor and to perform the method for high concurrency system based data processing according to any of the preceding claims 1 to 6.
10. A computer device, comprising:
one or more processors;
a memory;
one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, the one or more application programs configured to perform the high concurrency system based data processing method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910059280.6A CN111464582A (en) | 2019-01-22 | 2019-01-22 | High-concurrency data system, data processing method, storage medium and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910059280.6A CN111464582A (en) | 2019-01-22 | 2019-01-22 | High-concurrency data system, data processing method, storage medium and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111464582A true CN111464582A (en) | 2020-07-28 |
Family
ID=71685633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910059280.6A Pending CN111464582A (en) | 2019-01-22 | 2019-01-22 | High-concurrency data system, data processing method, storage medium and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111464582A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1508703A (en) * | 2002-12-18 | 2004-06-30 | 华为技术有限公司 | Data bank accessing method and client machine/server system for realizing same |
| US20040172459A1 (en) * | 2003-02-27 | 2004-09-02 | Schwalm Brian E. | Multi-tier business layer architecture for information systems |
| CN106649145A (en) * | 2016-12-15 | 2017-05-10 | Tcl集团股份有限公司 | Self-adaptive cache strategy updating method and system |
| CN109086030A (en) * | 2018-06-26 | 2018-12-25 | 中国电子科技集团公司第二十九研究所 | A kind of the Java Web three-tier architecture and optimization method of optimization |
-
2019
- 2019-01-22 CN CN201910059280.6A patent/CN111464582A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1508703A (en) * | 2002-12-18 | 2004-06-30 | 华为技术有限公司 | Data bank accessing method and client machine/server system for realizing same |
| US20040172459A1 (en) * | 2003-02-27 | 2004-09-02 | Schwalm Brian E. | Multi-tier business layer architecture for information systems |
| CN106649145A (en) * | 2016-12-15 | 2017-05-10 | Tcl集团股份有限公司 | Self-adaptive cache strategy updating method and system |
| CN109086030A (en) * | 2018-06-26 | 2018-12-25 | 中国电子科技集团公司第二十九研究所 | A kind of the Java Web three-tier architecture and optimization method of optimization |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10785322B2 (en) | Server side data cache system | |
| US8281382B1 (en) | Dynamic throttling systems and services | |
| US9239868B2 (en) | Virtual session management and reestablishment | |
| US8176037B2 (en) | System and method for SQL query load balancing | |
| CN107943841A (en) | Stream data processing method, system and computer-readable recording medium | |
| US20130097130A1 (en) | Method and system for resolving data inconsistency | |
| JP5999574B2 (en) | Database management system and computer system | |
| US11086841B1 (en) | Streams on shared database objects | |
| WO2016169237A1 (en) | Data processing method and device | |
| US9483493B2 (en) | Method and system for accessing a distributed file system | |
| US11405328B2 (en) | Providing on-demand production of graph-based relationships in a cloud computing environment | |
| CN107368324A (en) | A kind of component upgrade methods, devices and systems | |
| US8280910B2 (en) | Database server capable of relocating data distributed among plural processors and retrieving data method | |
| CN110704130A (en) | Data processing method and device | |
| US9405786B2 (en) | System and method for database flow management | |
| US11394748B2 (en) | Authentication method for anonymous account and server | |
| US9934275B2 (en) | Query union and split | |
| CN111464582A (en) | High-concurrency data system, data processing method, storage medium and computer equipment | |
| US9280384B2 (en) | Method, server and system for processing task data | |
| US10572531B1 (en) | Predictive session-based search engine | |
| CN112764897B (en) | Task request processing method, device and system and computer readable storage medium | |
| CN112100208A (en) | Operation request forwarding method and device | |
| US20190303595A1 (en) | Method and apparatus for managing database and non-transitory storage medium | |
| CN105511813A (en) | Method, device and system for accessing server disk | |
| US10313438B1 (en) | Partitioned key-value store with one-sided communications for secondary global key lookup by range-knowledgeable clients |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200728 |