CN112818021A - Data request processing method and device, computer equipment and storage medium - Google Patents
Data request processing method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN112818021A CN112818021A CN202110200896.8A CN202110200896A CN112818021A CN 112818021 A CN112818021 A CN 112818021A CN 202110200896 A CN202110200896 A CN 202110200896A CN 112818021 A CN112818021 A CN 112818021A
- Authority
- CN
- China
- Prior art keywords
- task processing
- instruction
- database
- task
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title abstract description 14
- 238000000034 method Methods 0.000 claims abstract description 48
- 238000004590 computer program Methods 0.000 claims description 13
- 230000009471 action Effects 0.000 claims description 7
- 230000001960 triggered effect Effects 0.000 claims description 7
- 238000012163 sequencing technique Methods 0.000 claims description 6
- 230000000977 initiatory effect Effects 0.000 claims description 5
- 230000000875 corresponding effect Effects 0.000 description 107
- 230000008569 process Effects 0.000 description 21
- 238000004364 calculation method Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 6
- 210000004899 c-terminal region Anatomy 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 230000002596 correlated effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data request processing method, a data request processing device, computer equipment and a storage medium. The method comprises the following steps: acquiring a task processing list; according to the task processing list, sequentially executing corresponding task processing instructions in the current period, and caching the task processing instructions and corresponding task processing results into a cache space in a correlation manner; after all the task processing instructions in the current period are executed, writing the task processing instructions and task processing results which are in the cache space and are in the associated cache into a database; receiving a result query instruction initiated by a client, wherein the result query instruction carries target request information corresponding to a task processing instruction previously sent by the current client; and according to the target request information, obtaining a target task processing result corresponding to the target request information from the database and feeding the target task processing result back to the client. By adopting the method, the problem of over-burden of the database when a plurality of timing tasks are processed can be effectively relieved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data request processing method and apparatus, a computer device, and a storage medium.
Background
With the development of computer technology, database reporting technology is becoming an indispensable part of the current computer technology field. In the database reporting environment, the timing task executed by the database is different from the timing task executed by the server (also called as a terminal or a client). And when the server executes the tasks, the timing task of each terminal is respectively sent to the server. If multiple terminals send timed tasks to the server during the same time period, multiple terminals will be connected to the same database during the current time period. Meanwhile, the timing tasks initiated by each terminal are independent from each other, so that the database can be simultaneously operated by a plurality of timing tasks, the database needs to be frequently operated repeatedly, and the data service load is large, so that the performance of the database is unstable, and the data crash is easy to occur.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data request processing method, device, computer device and storage medium capable of effectively improving the stability of a database.
A method of data request processing, the method comprising:
acquiring a task processing list, wherein the task processing list comprises a plurality of task processing instructions;
according to the task processing list, sequentially executing corresponding task processing instructions in the current period, accessing a database to obtain corresponding task processing results when each task processing instruction is executed, and performing associated cache on the task processing instructions and the corresponding task processing results to a cache space;
after all the task processing instructions in the current period are executed, writing the task processing instructions and task processing results which are in cache space and are in associated cache into the database;
receiving a result query instruction initiated by a client, wherein the result query instruction carries target request information corresponding to a task processing instruction previously sent by the current client;
and according to the target request information, obtaining a target task processing result corresponding to the target request information from a database and feeding the target task processing result back to the client.
In one embodiment, the method further comprises the following steps: the task processing list further includes storage cycle information and a storage instruction, where the storage cycle information is used to indicate that the execution of the storage instruction is triggered temporarily at a corresponding cycle time, so as to implement the step of writing the task processing instruction and the task processing result, which are cached in association in the cache space, into the database after the task processing instruction in the current cycle is completely executed.
In one embodiment, the method further comprises:
determining at least one execution time corresponding to the storage period information according to the storage period information;
and correspondingly adding a storage instruction into the task processing list according to the execution time.
In one embodiment, the obtaining the task processing list includes:
acquiring the plurality of task processing instructions; each task processing instruction comprises identity information and request content corresponding to a client initiating the current task processing instruction;
acquiring a first weight corresponding to each task processing instruction according to the identity information;
acquiring a second weight corresponding to each task processing instruction according to the request content;
for each task processing instruction, calculating a comprehensive weight corresponding to each task processing instruction based on a first weight and a second weight corresponding to the current task processing instruction;
and sequencing the plurality of task processing instructions according to the comprehensive weight corresponding to the task processing instruction to obtain a task processing list.
In one embodiment, the database includes a primary database and a backup database, and the method further includes:
acquiring a database backup instruction; and the database backup instruction is used for correspondingly backing up the task processing instruction written into the main database and the corresponding task processing result into the backup database.
In one embodiment, the store instruction is a set of SQL statements used to complete a store action.
A data request processing apparatus, the apparatus comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a task processing list which comprises a plurality of task processing instructions;
the execution module is used for sequentially executing corresponding task processing instructions in the current period according to the task processing list, accessing a database to obtain corresponding task processing results when each task processing instruction is executed, and performing associated cache on the task processing instructions and the corresponding task processing results to a cache space;
the storage module is used for writing the task processing instruction and the task processing result which are cached in a cache space in a relevant manner into the database after the task processing instruction in the current period is completely executed;
the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a result query instruction initiated by a client, and the result query instruction carries target request information corresponding to a task processing instruction previously sent by the current client;
and the feedback module is used for acquiring a target task processing result corresponding to the target request information from a database according to the target request information and feeding the target task processing result back to the client.
In one embodiment, the task processing list further includes storage cycle information and a storage instruction, where the storage cycle information is used to indicate that the execution of the storage instruction is triggered temporarily at a corresponding cycle time, so as to implement the step of writing the task processing instruction and the task processing result, which are cached in association in the cache space, into the database after the task processing instruction in the current cycle is completely executed.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring a task processing list, wherein the task processing list comprises a plurality of task processing instructions;
according to the task processing list, sequentially executing corresponding task processing instructions in the current period, accessing a database to obtain corresponding task processing results when each task processing instruction is executed, and performing associated cache on the task processing instructions and the corresponding task processing results to a cache space;
after all the task processing instructions in the current period are executed, writing the task processing instructions and task processing results which are in cache space and are in associated cache into the database;
receiving a result query instruction initiated by a client, wherein the result query instruction carries target request information corresponding to a task processing instruction previously sent by the current client;
and according to the target request information, obtaining a target task processing result corresponding to the target request information from a database and feeding the target task processing result back to the client.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring a task processing list, wherein the task processing list comprises a plurality of task processing instructions;
according to the task processing list, sequentially executing corresponding task processing instructions in the current period, accessing a database to obtain corresponding task processing results when each task processing instruction is executed, and performing associated cache on the task processing instructions and the corresponding task processing results to a cache space;
after all the task processing instructions in the current period are executed, writing the task processing instructions and task processing results which are in cache space and are in associated cache into the database;
receiving a result query instruction initiated by a client, wherein the result query instruction carries target request information corresponding to a task processing instruction previously sent by the current client;
and according to the target request information, obtaining a target task processing result corresponding to the target request information from a database and feeding the target task processing result back to the client.
The data request processing method, the data request processing device, the computer equipment and the storage medium form a task processing list by counting and summarizing the task processing instructions received by the database. And then, the database processes each task processing instruction according to the sequence in the task processing list, completes the corresponding task and caches the task. For each task period, after all tasks corresponding to the task processing instructions in each task period are executed, writing task results obtained by executing the current task period into a database. The client queries the task results stored in the database according to the task processing instruction initiated by the client, and feeds back the obtained corresponding target request information to the client. In this way, the execution end of the task processing instruction is transferred from the server end to the database, so that the database cannot be repeatedly operated no matter how many server ends are connected; meanwhile, the data are summarized at regular time, the summarized result is stored in the summary table, complex calculation is not needed to be carried out again when the report data are obtained, the burden of the database can be greatly reduced, and the stability of the database can be effectively improved when the data processing problem under a complex system is processed.
Drawings
FIG. 1 is a diagram of an application environment of a data request processing method in one embodiment;
FIG. 2 is a flow diagram that illustrates a method for processing data requests in accordance with one embodiment;
FIG. 3 is a flow diagram that illustrates the processing steps of a data request in one embodiment;
FIG. 4 is a block diagram of a data request processing device in one embodiment;
FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The data request processing method provided by the application can be applied to the application environment shown in fig. 1. Wherein the terminal 102 may communicate with the server 104 over a network. Specifically, the number of the terminals 102 may be multiple, and the server 104 generates a task processing list after acquiring the task processing instruction sent by each terminal 102. Then, the server 104 starts processing each task processing instruction in sequence according to the task processing list. After each task processing instruction is executed, the server 104 caches results obtained by executing the task processing instruction in a period according to the sequence of "caching first and then storing", and writes each task processing result corresponding to the task processing instruction in the current task period into a database position specified by the server 104 after the task processing instructions in the current task period are executed and corresponding result caches are obtained. Meanwhile, after the terminal 102 sends the task processing instruction, the terminal does not wait for the server 104 to finish the thread after completing the corresponding task processing instruction, but after a period of time, the terminal initiates an inquiry instruction to the server 104, and obtains the corresponding task processing result through the inquiry instruction, and after receiving the inquiry instruction of the terminal 102, the server 104 obtains the corresponding task processing result from the database according to the inquiry instruction and feeds the corresponding task processing result back to the terminal 102 requesting for inquiry. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the server 104 may be implemented by an independent server or a server cluster formed by a plurality of servers.
Before describing the data request processing method in the present application, the following explanations will be made to some terms involved in the embodiments of the present application:
and a task processing instruction: and the instruction is initiated by the terminal with the data processing requirement and requests the server terminal to perform specific data processing on the database.
Task processing list: the instruction list comprises one or more task processing instructions, wherein a plurality of task processing instructions can be sourced from one or more terminals.
And (5) result query instruction: and the query instruction sent by the terminal within a period of time after the task processing instruction is initiated is used for searching the corresponding task processing result from the database space in which the task processing result is stored.
In one embodiment, as shown in fig. 2, a data request processing method is provided, which is described by taking the application of the method to the server in fig. 1 as an example, and includes the following steps:
step S202, a task processing list is obtained, where the task processing list includes a plurality of task processing instructions.
Specifically, for a plurality of received task processing instructions, the server will process the received task processing instructions first to form an ordered task processing list. The plurality of task processing instructions in the task processing list may originate from a plurality of terminals, or may be a plurality of different task processing instructions initiated by one terminal.
Step S204, according to the task processing list, the corresponding task processing instructions in the current period are sequentially executed, and when each task processing instruction is executed, the database is accessed to obtain the corresponding task processing result, and the task processing instructions and the corresponding task processing result are associated and cached in the cache space.
Specifically, in order to improve the processing efficiency, the server executes the corresponding tasks one by one according to the sequence of the task processing instructions in the task processing list. The server accesses the database to obtain corresponding data content and processes the data content to obtain corresponding task processing results, and after the server completes each task processing instruction, each task processing instruction and the corresponding task processing result are cached in the cache space in a correlated mode for subsequent operation. In this embodiment, the cache space may exist depending on the server or the database, and is only used for temporarily storing a result obtained after the server executes the task processing instruction. Further, before the target task processing result is written into the database, the terminal cannot obtain the target task processing result through the task query instruction.
Step S206, after all the task processing instructions in the current period are executed, writing the task processing instructions and the task processing results in the cache space into the database.
Specifically, the server may process a plurality of task processing instructions by cycles in the process of executing the task processing instructions according to the task processing list. For a task period, the server executes the task processing instructions according to the existing task processing list and caches corresponding results, and stores each task processing result obtained in the task period and the corresponding task processing instruction in a database in a correlation manner after each task processing instruction in the task period is executed. Further, the server may also receive a task processing instruction from the terminal during the process of executing the task processing instruction, add the task processing instruction to the task processing list to obtain an updated task processing list, and when a next period starts, the server may start executing the task processing instruction of the next period based on the updated task processing list.
The task period may be set according to a requirement, and may be a task period according to a time when the server executes the task processing instruction, for example, every 24 hours, or may be a task period according to a number of the task processing instructions executed by the server, for example, every 500 task processing instructions executed by the server is a task period.
Step S208, receiving a result query instruction initiated by the client, wherein the result query instruction carries target request information corresponding to a task processing instruction previously sent by the current client.
Specifically, after the server executes the task corresponding to the task processing instruction, the server writes the task processing results in the cache into the database after the current task cycle is finished or when the next task cycle is started, where each task processing result is associated with the corresponding task processing instruction. After the client initiates the task processing instruction, the client does not wait for the server to return a corresponding result in real time, but initiates a query instruction to the server after a period of time, and queries whether the task processing instruction sent earlier by the client has a corresponding task processing result in the database. The terminal inquires target request information contained in the result inquiry instruction, the target request information comprises a terminal identifier and a task identifier corresponding to the initiated task processing instruction, the information requested by the result inquiry instruction can be determined as a task processing result corresponding to which instruction initiated by which client, and the server can determine the corresponding task processing result according to the target request information.
And step S210, according to the target request information, obtaining a target task processing result corresponding to the target request information from the database and feeding the target task processing result back to the client.
Specifically, after receiving the result query instruction, the server acquires the target request information therein, determines the processing state of the corresponding task processing instruction, and determines whether to feed back the corresponding task processing result according to the processing state. For example, in the foregoing embodiment, the server receives the first task processing instruction from the a terminal, and then receives the first result query instruction from the a terminal. The server determines that the first task processing result corresponding to the first task processing instruction initiated by the terminal A is inquired by the first result inquiry instruction according to the first result inquiry instruction. If the first task processing instruction is processed completely, the obtained first task processing result is written into the database, and the server can obtain the corresponding first task processing result from the database and feed the first task processing result back to the terminal A. Further, the first result query instruction may be issued after the first task processing instruction issues a length of time for one task cycle.
In the data request processing method, the data request processing method performs statistics and summarization on the task processing instructions received by the database to form a task processing list. And then, the database processes each task processing instruction according to the sequence in the task processing list, completes the corresponding task and caches the task. For each task period, after all tasks corresponding to the task processing instructions in each task period are executed, writing task results obtained by executing the current task period into a database. The client queries the task results stored in the database according to the task processing instruction initiated by the client, and feeds back the obtained corresponding target request information to the client. In this way, the execution end of the task processing instruction is transferred from the server end to the database, so that the database cannot be repeatedly operated no matter how many server ends are connected; meanwhile, the data are summarized at regular time, the summarized result is stored in the summary table, complex calculation is not needed to be carried out again when the report data are obtained, the burden of the database can be greatly reduced, and the stability of the database can be effectively improved when the data processing problem under a complex system is processed.
In an embodiment, the task processing list further includes storage cycle information and a storage instruction, where the storage cycle information is used to indicate that the execution of the storage instruction is triggered temporarily at a corresponding cycle time, so as to implement a step of writing the task processing instruction and the task processing result, which are cached in association in the cache space, into the database after the task processing instruction in the current cycle is completely executed.
Specifically, the task processing list also has a task for instructing the server to execute the storage instruction at an appropriate time, and when the corresponding cycle time indicated by the storage cycle information is reached temporarily during the execution of each task processing instruction in the task processing list by the server, the server is triggered to execute the storage instruction, and each task processing result located in the cache space between the current cycle time and the previous cycle time is written into the database in a corresponding relationship with the task processing instruction.
In the above embodiment, by storing the cycle information and the storage instruction, the server can ensure the ordered execution of the task processing instruction on the one hand, and can write the obtained task processing result into the database in time for query on the other hand.
In one embodiment, the method further comprises: determining at least one execution time corresponding to the storage period information according to the storage period information; and correspondingly adding the storage instruction into the task processing list according to the execution time.
Specifically, in the above embodiment, the storage period information is stored, the storage instruction is added to the task processing list, the server may select any time as a start time when performing processing on each task processing in the task processing list, a subsequent execution time corresponding to the storage period information may be determined based on the start time and the storage period information, and according to the execution time, the server may temporarily execute the storage instruction at the execution time, and store the task processing result in the cache in the database.
For example, if the storage period information is "every 24 hours as a task period", the storage period information is the first execution time after 24 hours from the start time, and if the storage period information is "every 500 task processing instructions are executed as a task period", the server is the first execution time after 500 task processing instructions are executed from the start time, and the calculation manners of the execution times are different according to the difference of the storage period information.
In the above embodiment, by setting the storage period information, the server can regularly and effectively store the task processing result in the cache in the database in time.
In one embodiment, obtaining a task processing list comprises: acquiring a plurality of task processing instructions; each task processing instruction comprises identity information and request content corresponding to a client initiating the current task processing instruction; acquiring a first weight corresponding to each task processing instruction according to the identity information; acquiring a second weight corresponding to each task processing instruction according to the request content; for each task processing instruction, calculating a comprehensive weight corresponding to each task processing instruction based on a first weight and a second weight corresponding to the current task processing instruction; and sequencing the plurality of task processing instructions according to the comprehensive weight corresponding to the task processing instruction to obtain a task processing list.
Specifically, after receiving a plurality of task processing instructions, the server may perform further sorting processing on each instruction to obtain a task processing list with the best efficiency. Because each task processing instruction comprises the identity information and the request content corresponding to the client initiating the current task processing instruction, for each task processing instruction, the identity information and the request content have corresponding weight proportion. For example, in a time period, the server receives an a task processing instruction from the terminal a and a B task processing instruction from the terminal B, and by identifying the two task processing instructions, first weights corresponding to the terminal a and the terminal B, and second weights corresponding to the request content of the a task processing instruction and the request content of the B task processing instruction, respectively, may be determined, and based on the first weight of the terminal a and the second weight of the request content of the a task processing instruction, a weight ordering a 'of the a task processing instruction in the task processing list may be determined, and similarly, a weight ordering B' of the B task processing instruction in the task processing list may be determined, and according to a 'and B', the order of the a task processing instruction and the B task processing instruction in the task processing list may be determined.
Further, it is preferable for the server to perform deduplication processing on the task processing list in order to avoid repetitive operations on the database. For example, in a period of time, the server receives the a task processing instruction from the a terminal and the C task processing instruction from the C terminal, and by identifying the two task processing instructions, the server finds that the task processing actions, the processed data objects, and the like to be performed by the two task processing instructions are completely consistent, so that in the task processing list, the a task processing instruction and the C task processing instruction can be marked as an a 'task processing instruction, and the a' task processing instruction is marked to indicate that the task instructions are respectively sent out by the a terminal and the C terminal separately for subsequent query. Or, directly marking the task processing instruction a, indicating that the C terminal also sends the same task processing instruction, and directing the C terminal to a task processing result corresponding to the task processing instruction a according to a result query request of the task processing instruction C. In the embodiment, the efficiency of the server for processing the task processing instructions of different request sources and request contents can be effectively improved by sequencing the task processing list.
In one embodiment, the database includes a primary database and a backup database, and the method further includes: acquiring a database backup instruction; the database backup instruction is used for correspondingly backing up the task processing instruction written into the main database and the corresponding task processing result into the backup database.
Specifically, the database in this embodiment is divided into a master database and a backup database, and master-slave backup is performed at regular time to ensure the security of the database. When the main database is used for executing and completing each task processing instruction according to the task processing list, the backup database synchronizes the content (the task processing instruction and the corresponding task processing result) written into the main database into the backup database at regular time according to the database backup instruction.
In the above embodiment, the database backup instruction is independent of each instruction in the task processing list, so that the server can still ensure the security of the database when executing the task processing instruction.
In one embodiment, the store instruction is a set of SQL statements used to complete the store action.
Specifically, in the present embodiment, the storage action is implemented by a set of SQL statements for performing a specific function (storage).
In the embodiment, the SQL statement is used for storage, so that the database has great flexibility and strong functions in the storage process.
Fig. 3 is a schematic flow chart illustrating data request processing steps in a specific embodiment. In this embodiment, to implement the timing summary using the database event plan, the start event plan is first configured (i.e., a task plan for processing the data request by using the task processing list is set for the server); then establishing an event plan (namely, constructing a task processing list), and periodically calling a storage process; then storing the process calculation data and writing the calculation result into a summary table (namely writing the task processing instruction and the task processing result in the cache into a database through a storage instruction); and finally, when the client needs to acquire report data, the server reads the data from the summary table and returns the data to the client (namely, each task processing result is stored in the database in the form of the summary table, and the server can acquire a corresponding target task processing result from the summary table of the database and feed the target task processing result back to the client through a result query instruction). According to the method, the timing task is executed on the database instead of the server, so that the problems of repeated connection, high database load and the like caused by simultaneous connection of a plurality of servers with the same database are solved, and the user request can be quickly responded through timely calculation and data summarization in advance.
Specifically, the embodiment of the data request processing shown in fig. 3 includes the following steps:
step 1: the configuration file enables the event plan, and in addition, if the database opens the master-slave backup, the summary table needs to be set for not being synchronized. The event plan is a new characteristic introduced into a new version of the database, and a specific task is executed in a specified time unit, so that the data timing operation does not need to depend on an external program any more. Master-slave backup, or master-slave synchronization, refers to a data disaster recovery measure for a large-scale website, and includes a master database and a slave database, when the master database fails, the master database is switched to the slave database, and data of the two databases needs to be continuously synchronized. In this embodiment, no matter whether the event plan is executed by using the master database or the backup database, in the execution process, since the task processing result in the current task period is not written into the database for executing the event plan but is only backed up, the event plan itself does not need to synchronize the summary table, but only needs to synchronize the task processing result written into the database, and thus the summary table in this embodiment is preferably not synchronized.
For the above step 1, the corresponding pseudo code format is exemplified as follows:
# open event plan
event_scheduler＝1
No synchronization is performed with the # designation table (named gcam _ report _ Alipay table exactly)
replicate-ignore-table＝gcam_report_alipay
# assignment Table not synchronized (fuzzy assignment table with names beginning with gcam _ report _ beginning)
replicate-wild-ignore-table＝gcam_report_*
Step 2: and establishing an event plan and calling the storage process regularly. In this embodiment, the storage process refers to a group of SQL statement sets for completing a specific function; the executed task content is the processing result of each task in the cache and is written into the database. And calling the storage process regularly, namely executing the storage action according to the storage period information and the storage instruction. Taking the storage period as 24 hours as an example, for the step 2, the corresponding pseudo code form is as follows:
-half-call procedure _ gcam _ report _ alipay 12 points per day
DROP EVENT IF EXISTS`event_gcam_report_alipay`；
DELIMITER；；
CREATE EVENT`event_gcam_report_alipay`ON SCHEDULE EVERY 1DAY STARTS'2000-01-01 12:30:00'ON COMPLETION PRESERVE ENABLE DO call procedure_gcam_report_alipay()
；；
DELIMITER；
And step 3: and storing the process calculation data, and writing the calculation result into a summary table. The summary table is a data table specially used for storing summary results, after a terminal (or a server) initiates a data request, the terminal does not wait for the processing result of the server in real time, but requests the server to obtain the task processing result corresponding to the previously initiated data request after a period of time, and when the report data is obtained, the server only needs to access the summary table, and the corresponding task processing result is taken out from the summary table and fed back to the terminal, so that complex calculation is not needed again, and the burden of the server is greatly reduced.
For the above step 3, the corresponding pseudo code format is exemplified as follows:
calculating yesterday data and writing into a summary sheet
And 4, step 4: when the client needs to acquire report data, the server reads the data from the summary table and returns the data to the client;
SQL for obtaining report data for all months in 2016
select DATE _ FORMAT ('% Y year% c month') as reportDate, SUM (IFNULL (total,0)) as total
from gcam_report_alipay
where DATE_FORMAT(reportDate,'％Y')＝'2016'
group by DATE _ FORMAT (reportDate, '% Y year% m month')
SQL for obtaining report data for all the days of month 6 in 2016
select DATE _ FORMAT ('% c month% e day') as reportDate, SUM (IFNULL (total,0)) as total
from gcam_report_alipay
where DATE_FORMAT(reportDate,'％Y-％m')＝'2016-06'
group by DATE _ FORMAT (reportDate, '% m month% d day')
Further, as can be seen from fig. 3, the process of requesting to obtain report data by the client is independent from the process of executing the database timing plan by the server, that is, the above embodiment executes the timing task at the database instead of executing the timing task at the server, thereby solving the problems of repeated connection, high database load and the like caused by simultaneous connection of multiple servers to the same database, and simultaneously, the user request can be quickly responded by calculating and summarizing data at a predetermined timing in advance, thereby solving the problem of efficient database report data processing under a complex system.
It should be understood that although the various steps in the flow charts of fig. 2-3 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-3 may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed in turn or alternately with other steps or at least some of the other steps.
In one embodiment, as shown in fig. 4, there is provided a data request processing apparatus including: an obtaining module 402, an executing module 404, a storing module 406, a receiving module 408, and a feedback module 410, wherein:
an obtaining module 402, configured to obtain a task processing list, where the task processing list includes a plurality of task processing instructions.
The execution module 404 is configured to sequentially execute the task processing instructions corresponding to the current period according to the task processing list, access the database to obtain corresponding task processing results when each task processing instruction is executed, and cache the task processing instructions and the corresponding task processing results in the cache space in an associated manner.
The storage module 406 is configured to write the task processing instruction and the task processing result that are cached in association in the cache space into the database after all the task processing instructions in the current cycle are executed.
The receiving module 408 is configured to receive a result query instruction initiated by the client, where the result query instruction carries target request information corresponding to a task processing instruction previously sent by the current client.
And a feedback module 410, configured to obtain a target task processing result corresponding to the target request information from the database according to the target request information, and feed the target task processing result back to the client.
The data request processing device is used for forming a task processing list by counting and summarizing the task processing instructions received by the database. And then, the database processes each task processing instruction according to the sequence in the task processing list, completes the corresponding task and caches the task. For each task period, after all tasks corresponding to the task processing instructions in each task period are executed, writing task results obtained by executing the current task period into a database. The client queries the task results stored in the database according to the task processing instruction initiated by the client, and feeds back the obtained corresponding target request information to the client. In this way, the execution end of the task processing instruction is transferred from the server end to the database, so that the database cannot be repeatedly operated no matter how many server ends are connected; meanwhile, the data are summarized at regular time, the summarized result is stored in the summary table, complex calculation is not needed to be carried out again when the report data are obtained, the burden of the database can be greatly reduced, and the stability of the database can be effectively improved when the data processing problem under a complex system is processed.
In an embodiment, the task processing list further includes storage cycle information and a storage instruction, where the storage cycle information is used to indicate that the execution of the storage instruction is triggered temporarily at a corresponding cycle time, so as to implement a step of writing the task processing instruction and the task processing result, which are cached in association in the cache space, into the database after the task processing instruction in the current cycle is completely executed.
In the above embodiment, by storing the cycle information and the storage instruction, the server can ensure the ordered execution of the task processing instruction on the one hand, and can write the obtained task processing result into the database in time for query on the other hand.
In one embodiment, the obtaining module is further configured to determine, according to the storage cycle information, at least one execution time corresponding to the storage cycle information; and correspondingly adding the storage instruction into the task processing list according to the execution time.
In the above embodiment, by setting the storage period information, the server can regularly and effectively store the task processing result in the cache in the database in time.
In one embodiment, the obtaining module is further configured to obtain a plurality of task processing instructions; each task processing instruction comprises identity information and request content corresponding to a client initiating the current task processing instruction; acquiring a first weight corresponding to each task processing instruction according to the identity information; acquiring a second weight corresponding to each task processing instruction according to the request content; for each task processing instruction, calculating a comprehensive weight corresponding to each task processing instruction based on a first weight and a second weight corresponding to the current task processing instruction; and sequencing the plurality of task processing instructions according to the comprehensive weight corresponding to the task processing instruction to obtain a task processing list.
In the embodiment, the efficiency of the server for processing the task processing instructions of different request sources and request contents can be effectively improved by sequencing the task processing list.
In one embodiment, the database includes a primary database and a backup database, and the apparatus further includes: the backup module is used for acquiring a database backup instruction; the database backup instruction is used for correspondingly backing up the task processing instruction written into the main database and the corresponding task processing result into the backup database.
In the above embodiment, the database backup instruction is independent of each instruction in the task processing list, so that the server can still ensure the security of the database when executing the task processing instruction.
In one embodiment, the store instruction is a set of SQL statements used to complete the store action.
In the embodiment, the SQL statement is used for storage, so that the database has great flexibility and strong functions in the storage process.
For specific limitations of the data request processing device, reference may be made to the above limitations of the data request processing method, which is not described herein again. The respective modules in the data request processing apparatus may be wholly or partially implemented by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing the data content of the data request processing and the task processing result obtained through the processing. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data request processing method.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for processing data requests, the method comprising:
acquiring a task processing list, wherein the task processing list comprises a plurality of task processing instructions;
according to the task processing list, sequentially executing corresponding task processing instructions in the current period, accessing a database to obtain corresponding task processing results when each task processing instruction is executed, and performing associated cache on the task processing instructions and the corresponding task processing results to a cache space;
after all the task processing instructions in the current period are executed, writing the task processing instructions and task processing results which are in cache space and are in associated cache into the database;
receiving a result query instruction initiated by a client, wherein the result query instruction carries target request information corresponding to a task processing instruction previously sent by the current client;
and according to the target request information, obtaining a target task processing result corresponding to the target request information from a database and feeding the target task processing result back to the client.
2. The method according to claim 1, wherein the task processing list further includes storage cycle information and a storage instruction, and the storage cycle information is used to indicate that, temporarily at a corresponding cycle time, execution of the storage instruction is triggered, so as to implement the step of writing the task processing instruction and the task processing result buffered in association in a buffer space into the database after all execution of the task processing instruction in the current cycle is completed.
3. The method of claim 2, further comprising:
determining at least one execution time corresponding to the storage period information according to the storage period information;
and correspondingly adding a storage instruction into the task processing list according to the execution time.
4. The method of claim 1, wherein obtaining the task processing list comprises:
acquiring the plurality of task processing instructions; each task processing instruction comprises identity information and request content corresponding to a client initiating the current task processing instruction;
acquiring a first weight corresponding to each task processing instruction according to the identity information;
acquiring a second weight corresponding to each task processing instruction according to the request content;
for each task processing instruction, calculating a comprehensive weight corresponding to each task processing instruction based on a first weight and a second weight corresponding to the current task processing instruction;
and sequencing the plurality of task processing instructions according to the comprehensive weight corresponding to the task processing instruction to obtain a task processing list.
5. The method of claim 1, wherein the database comprises a primary database and a backup database, the method further comprising:
acquiring a database backup instruction; and the database backup instruction is used for correspondingly backing up the task processing instruction written into the main database and the corresponding task processing result into the backup database.
6. The method according to claim 2 or 3, wherein the storage instruction is a set of SQL statements for completing a storage action.
7. A data request processing apparatus, characterized in that the apparatus comprises:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a task processing list which comprises a plurality of task processing instructions;
the execution module is used for sequentially executing corresponding task processing instructions in the current period according to the task processing list, accessing a database to obtain corresponding task processing results when each task processing instruction is executed, and performing associated cache on the task processing instructions and the corresponding task processing results to a cache space;
the storage module is used for writing the task processing instruction and the task processing result which are cached in a cache space in a relevant manner into the database after the task processing instruction in the current period is completely executed;
the system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a result query instruction initiated by a client, and the result query instruction carries target request information corresponding to a task processing instruction previously sent by the current client;
and the feedback module is used for acquiring a target task processing result corresponding to the target request information from a database according to the target request information and feeding the target task processing result back to the client.
8. The apparatus according to any one of claims 7, wherein the task processing list further includes storage cycle information and a storage instruction, and the storage cycle information is used to indicate that, at a corresponding cycle time, execution of the storage instruction is triggered temporarily, so as to implement the step of writing the task processing instruction and the task processing result buffered in the buffer space into the database after all execution of the task processing instruction in the current cycle is completed.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110200896.8A CN112818021B (en) | 2021-02-23 | 2021-02-23 | Data request processing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110200896.8A CN112818021B (en) | 2021-02-23 | 2021-02-23 | Data request processing method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112818021A true CN112818021A (en) | 2021-05-18 |
| CN112818021B CN112818021B (en) | 2024-02-02 |
Family
ID=75865065
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110200896.8A Active CN112818021B (en) | 2021-02-23 | 2021-02-23 | Data request processing method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112818021B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656155A (en) * | 2021-07-21 | 2021-11-16 | 广州三七互娱科技有限公司 | Task processing method and device, computer equipment and storage medium |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120246390A1 (en) * | 2011-03-24 | 2012-09-27 | Kabushiki Kaisha Toshiba | Information processing apparatus, program product, and data writing method |
| US20130081100A1 (en) * | 2011-09-23 | 2013-03-28 | Oracle International Corporation | System and method of real-time change propagation and activation using a distributed object cache |
| CN106874316A (en) * | 2015-12-14 | 2017-06-20 | 广州爱九游信息技术有限公司 | A kind of methods of exhibiting of user's combined data, device and server |
| CN109033315A (en) * | 2018-07-18 | 2018-12-18 | 张小剑 | Data query method, client, server and computer-readable medium |
| CN109145020A (en) * | 2018-07-23 | 2019-01-04 | 程之琴 | Information query method, from server, client and computer readable storage medium |
| CN109597818A (en) * | 2018-11-28 | 2019-04-09 | 优刻得科技股份有限公司 | Data-updating method, device, storage medium and equipment |
| CN109816198A (en) * | 2018-12-12 | 2019-05-28 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium of data processing |
| CN109962947A (en) * | 2017-12-22 | 2019-07-02 | 阿里巴巴集团控股有限公司 | Method for allocating tasks and device in a kind of peer-to-peer network |
| CN110163474A (en) * | 2019-04-12 | 2019-08-23 | 平安普惠企业管理有限公司 | A kind of method and apparatus of task distribution |
| CN110347692A (en) * | 2019-06-03 | 2019-10-18 | 阿里巴巴集团控股有限公司 | Inventory management method and device based on caching |
| CN110351342A (en) * | 2019-06-20 | 2019-10-18 | 平安科技（深圳）有限公司 | Service order processing method, device, computer equipment and storage medium |
| CN110597858A (en) * | 2019-08-30 | 2019-12-20 | 深圳壹账通智能科技有限公司 | Task data processing method and device, computer equipment and storage medium |
| CN110738554A (en) * | 2019-10-28 | 2020-01-31 | 拉扎斯网络科技（上海）有限公司 | Task processing method and device, electronic equipment and computer readable storage medium |
| CN110851465A (en) * | 2019-11-15 | 2020-02-28 | 腾讯科技（深圳）有限公司 | Data query method and system |
| CN110866112A (en) * | 2018-08-14 | 2020-03-06 | 阿里巴巴集团控股有限公司 | Response sequence determination method, server and terminal equipment |
-
2021
- 2021-02-23 CN CN202110200896.8A patent/CN112818021B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120246390A1 (en) * | 2011-03-24 | 2012-09-27 | Kabushiki Kaisha Toshiba | Information processing apparatus, program product, and data writing method |
| US20130081100A1 (en) * | 2011-09-23 | 2013-03-28 | Oracle International Corporation | System and method of real-time change propagation and activation using a distributed object cache |
| CN106874316A (en) * | 2015-12-14 | 2017-06-20 | 广州爱九游信息技术有限公司 | A kind of methods of exhibiting of user's combined data, device and server |
| CN109962947A (en) * | 2017-12-22 | 2019-07-02 | 阿里巴巴集团控股有限公司 | Method for allocating tasks and device in a kind of peer-to-peer network |
| CN109033315A (en) * | 2018-07-18 | 2018-12-18 | 张小剑 | Data query method, client, server and computer-readable medium |
| CN109145020A (en) * | 2018-07-23 | 2019-01-04 | 程之琴 | Information query method, from server, client and computer readable storage medium |
| CN110866112A (en) * | 2018-08-14 | 2020-03-06 | 阿里巴巴集团控股有限公司 | Response sequence determination method, server and terminal equipment |
| CN109597818A (en) * | 2018-11-28 | 2019-04-09 | 优刻得科技股份有限公司 | Data-updating method, device, storage medium and equipment |
| CN109816198A (en) * | 2018-12-12 | 2019-05-28 | 平安科技（深圳）有限公司 | Method, apparatus, computer equipment and the storage medium of data processing |
| CN110163474A (en) * | 2019-04-12 | 2019-08-23 | 平安普惠企业管理有限公司 | A kind of method and apparatus of task distribution |
| CN110347692A (en) * | 2019-06-03 | 2019-10-18 | 阿里巴巴集团控股有限公司 | Inventory management method and device based on caching |
| CN110351342A (en) * | 2019-06-20 | 2019-10-18 | 平安科技（深圳）有限公司 | Service order processing method, device, computer equipment and storage medium |
| CN110597858A (en) * | 2019-08-30 | 2019-12-20 | 深圳壹账通智能科技有限公司 | Task data processing method and device, computer equipment and storage medium |
| CN110738554A (en) * | 2019-10-28 | 2020-01-31 | 拉扎斯网络科技（上海）有限公司 | Task processing method and device, electronic equipment and computer readable storage medium |
| CN110851465A (en) * | 2019-11-15 | 2020-02-28 | 腾讯科技（深圳）有限公司 | Data query method and system |
Non-Patent Citations (2)
| Title |
|---|
| 杨霄翼;: "基于Python的"地理处理"并行方案", 地理信息世界, no. 06, pages 125 - 129 * |
| 薛安荣, 鲍培德: "基于数据库的协同CAD系统设计", 江苏理工大学学报, no. 03, pages 72 - 75 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113656155A (en) * | 2021-07-21 | 2021-11-16 | 广州三七互娱科技有限公司 | Task processing method and device, computer equipment and storage medium |
| CN113656155B (en) * | 2021-07-21 | 2024-04-12 | 广州三七互娱科技有限公司 | Task processing method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112818021B (en) | 2024-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8694647B2 (en) | Read-only operations processing in a paxos replication system | |
| US9563426B1 (en) | Partitioned key-value store with atomic memory operations | |
| CN106610876A (en) | Method and device for recovering data snapshot | |
| CN111797121A (en) | Strong consistency query method, device and system for read-write separation architecture service system | |
| CN110427386B (en) | Data processing method, device and computer storage medium | |
| CN112162846B (en) | Transaction processing method, device and computer readable storage medium | |
| CN107798135B (en) | Paging query method and device and electronic equipment | |
| CN110795171B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN113094430B (en) | Data processing method, device, equipment and storage medium | |
| CN113111038B (en) | File storage method, device, server and storage medium | |
| US11687533B2 (en) | Centralized storage for search servers | |
| CN110457401B (en) | Data storage method and device, computer equipment and storage medium | |
| CN110866011A (en) | Data table synchronization method and device, computer equipment and storage medium | |
| CN112866339B (en) | Data transmission method and device, computer equipment and storage medium | |
| CN112818021B (en) | Data request processing method, device, computer equipment and storage medium | |
| US11625503B2 (en) | Data integrity procedure | |
| CN110008284A (en) | Method for synchronizing data of database and equipment based on data page preloading and rollback | |
| CN113590613A (en) | Data table partitioning method and device, computer equipment and storage medium | |
| CN110955461B (en) | Processing method, device, system, server and storage medium for computing task | |
| CN110287187B (en) | Method, system and storage medium for synchronously updating data tables in multiple servers | |
| CN115544172A (en) | Method and system for synchronizing data among clusters of one master and multiple slaves in real time | |
| CN113672640A (en) | Data query method and device, computer equipment and storage medium | |
| CN110162542B (en) | Data page turning method and device based on cassandra, computer equipment and storage medium | |
| CN114416689A (en) | Data migration method and device, computer equipment and storage medium | |
| US8560540B2 (en) | Information search system, information search method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |