CN112632363A - Method, device and equipment for processing batch query requests and readable storage medium - Google Patents
Method, device and equipment for processing batch query requests and readable storage medium Download PDFInfo
- Publication number
- CN112632363A CN112632363A CN202011514804.5A CN202011514804A CN112632363A CN 112632363 A CN112632363 A CN 112632363A CN 202011514804 A CN202011514804 A CN 202011514804A CN 112632363 A CN112632363 A CN 112632363A
- Authority
- CN
- China
- Prior art keywords
- request
- query
- batch
- processed
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 97
- 238000000034 method Methods 0.000 title claims abstract description 47
- 230000008030 elimination Effects 0.000 claims abstract description 8
- 238000003379 elimination reaction Methods 0.000 claims abstract description 8
- 230000007246 mechanism Effects 0.000 claims description 14
- 230000008569 process Effects 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 3
- 238000003672 processing method Methods 0.000 abstract description 7
- 238000005516 engineering process Methods 0.000 abstract description 6
- 238000011835 investigation Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 239000002699 waste material Substances 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000011143 downstream manufacturing Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000007639 printing Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9538—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of financial science and technology, and discloses a processing method, a device, equipment and a computer readable storage medium for batch query requests, wherein the method comprises the following steps: receiving batch query requests, configuring the priority of each query request based on preset dimensionality, and generating a batch to-be-looked-up table based on the priority of the query requests; starting a timing task, and acquiring a query request to be processed from a batch of tables to be looked up according to priority; the method comprises the steps that after the duplication elimination processing is carried out on threads processing batch query requests, corresponding request files are downloaded based on the identity identification of the query requests to be processed, and request contents are read; and querying through the request content of the query request to be processed by the preset thread to obtain a query result. Therefore, the priority of each query request in the batch of query requests is set based on the preset dimensionality, then the requests to be processed are obtained based on the priority, and query is sequentially carried out to obtain query results, so that the flexibility of processing batch query is improved, and the maximization of resource utilization is realized.
Description
Technical Field
The present invention relates to the field of financial technology (Fintech), and in particular, to a method, an apparatus, a device, and a computer-readable storage medium for processing a batch query request.
Background
With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to financial technology (Fintech), but higher requirements are also put forward on the technologies due to the requirements of the financial industry on safety and real-time performance.
At present, the processing mode of the batch query request is generally to convert the batch query request into a single query, cannot configure the priority of each query request in the batch query request, and only can sequentially query based on the time sequence, so that the flexibility of the batch query is low, and the utilization rate of resources is not high enough
Disclosure of Invention
The invention provides a processing method, a processing device, processing equipment and a computer readable storage medium for batch query requests, aiming at improving the flexibility of processing batch queries and realizing the maximization of resource utilization.
In order to achieve the above object, the present invention provides a method for processing batch query requests, including:
receiving batch query requests, configuring the priority of each query request based on preset dimensionality, and generating a batch to-be-searched list based on the priority of the query requests;
starting a timing task, and acquiring a query request to be processed from the batch of tables to be looked up according to the priority;
after the duplication elimination processing is carried out on the thread processing the batch query requests, downloading corresponding request files based on the identity identification of the query request to be processed, and reading the request content of each request file;
and querying through the request content of the query request to be processed by a preset thread to obtain a query result.
Optionally, the obtaining the query request to be processed from the batch pending form according to the priority includes:
acquiring a single query request to be processed from the batch of query lists according to the sequence of the priorities from high to low;
the obtaining of the single query request to be processed from the batch of the query lists according to the sequence of the priorities from high to low further comprises:
transferring a single query request to be processed into an online queuing table, and determining the queue depth of the online queuing table and/or the processing success rate of the query request to be processed;
and if the queue depth is greater than the depth threshold and/or the processing success rate is less than the success rate threshold, suspending the processing of the corresponding batch query request and receiving the next batch query request.
Optionally, the obtaining the query request to be processed from the batch of pending queries according to the priority further includes:
and after carrying out deduplication processing on the thread processing the batch query requests, downloading corresponding request files based on the identity of the query requests to be processed, and reading the request content of each request file.
Optionally, the reading the request content of each request file includes:
monitoring whether the application stops or not in the process of reading the request content;
if the application is monitored to stop, backing up the read request content and/or storing the interrupt position read when the application is stopped, and accessing the next batch of request files.
Optionally, the reading the request content of each request file includes:
determining the number of files of the request file;
if the number of the files is larger than a first preset number, reading the request content of the request file in an iterator mode;
and if the number of the files is less than the second preset number, accessing the next request file until the number of the accessed request files reaches a third preset number, and reading the request content of each request file.
Optionally, the querying through the content of the request file by a preset thread to obtain a query result includes:
and submitting the read content of the request file to a batch thread pool, respectively carrying out online inquiry on each inquiry request to be processed by an asynchronous thread in the batch thread pool, and returning an inquiry result to a calling party.
The reading is queried through the request content of the query request to be processed through a preset thread to obtain a query result, and the method also comprises the following steps:
and checking the parameters of the request content, and accessing the query request to be processed to the preset thread based on a preset flow and/or quantity control mechanism after the checking is successful.
Optionally, the accessing the to-be-processed query request to the preset thread based on a preset flow and/or quantity control mechanism includes:
determining the size of a data packet of the query request to be processed and the size of a buffer idle area of each preset thread;
determining a target preset thread accessed to the query request to be processed based on the size of the data packet and the size of the buffer idle area, and accessing the query request to be processed to the target preset thread; and/or
Determining the number of accessed requests of the accessed to-be-processed query requests in each preset thread;
and determining a target preset thread based on the accessed quantity and the processing capacity of the corresponding thread, wherein the query request to be processed is accessed to the target preset thread.
In addition, to achieve the above object, the present invention further provides an apparatus for processing batch query requests, including:
the receiving module is used for receiving the batch query requests, configuring the priority of each query request based on the preset dimensionality and generating a batch to-be-looked-up table based on the priority of the query requests;
the reading module is used for downloading corresponding request files based on the identity of the query request to be processed after the duplication elimination processing is carried out on the threads for processing the batch query requests, and reading the request content of each request file;
the acquisition module is used for starting a timing task and acquiring the query requests to be processed from the batch of tables to be looked up according to the priority;
a query module for querying through the request content of the query request to be processed by a preset thread to obtain a query result
In addition, in order to achieve the above object, the present invention further provides a device for processing a batch query request, where the device for processing a batch query request includes a processor, a memory, and a handler of a batch query request stored in the memory, and when the handler of a batch query request is executed by the processor, the steps of the method for processing a batch query request as described above are implemented.
In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, on which a batch query request handler is stored, and when the batch query request handler is executed by a processor, the steps of the batch query request handling method are implemented.
Compared with the prior art, the invention provides a processing method, a device, equipment and a computer-readable storage medium for batch query requests, wherein the processing method, the device, the equipment and the computer-readable storage medium are used for receiving the batch query requests, configuring the priority of each query request based on the preset dimensionality and generating a batch to-be-checked list based on the priority of the query requests; starting a timing task, and acquiring a query request to be processed from the batch of tables to be looked up according to the priority; after the duplication elimination processing is carried out on the thread processing the batch query requests, downloading corresponding request files based on the identity identification of the query request to be processed, and reading the request content of each request file; and querying through the request content of the query request to be processed by a preset thread to obtain a query result. Therefore, the priority of each query request in the batch of query requests is set based on the preset dimensionality, then the requests to be processed are obtained based on the priority, and query is sequentially carried out to obtain query results, so that the flexibility of processing batch query is improved, and the maximization of resource utilization is realized.
Drawings
FIG. 1 is a diagram illustrating a hardware architecture of a device for processing a batch query request according to embodiments of the present invention;
FIG. 2 is a flowchart illustrating a method for processing a batch query request according to a first embodiment of the present invention;
FIG. 3 is a diagram illustrating a first exemplary embodiment of a method for processing a batch query request according to the present invention;
FIG. 4 is a flowchart illustrating a method for processing a batch query request according to a second embodiment of the present invention;
FIG. 5 is a functional block diagram of a processing apparatus for batch query requests according to a first embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The processing device of the batch query request mainly related to the embodiment of the invention is a network connection device capable of realizing network connection, and the processing device of the batch query request can be a server, a cloud platform and the like.
Referring to fig. 1, fig. 1 is a hardware configuration diagram of a processing device for batch query requests according to embodiments of the present invention. In this embodiment of the present invention, the Processing device for batch query requests may include a processor 1001 (e.g., a Central Processing Unit, CPU), a communication bus 1002, an input port 1003, an output port 1004, and a memory 1005. The communication bus 1002 is used for realizing connection communication among the components; the input port 1003 is used for data input; the output port 1004 is used for data output, the memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as a magnetic disk memory, and the memory 1005 may optionally be a storage device independent of the processor 1001. Those skilled in the art will appreciate that the hardware configuration depicted in FIG. 1 is not intended to be limiting of the present invention, and may include more or less components than those shown, or some components in combination, or a different arrangement of components.
With continued reference to FIG. 1, the memory 1005 of FIG. 1, which is one type of readable computer-readable storage medium, may include an operating system, a network communication module, an application module, and a handler of batch query requests. In fig. 1, the network communication module is mainly used for connecting to a server and performing data communication with the server; the processor 1001 may call a processing program of the batch query request stored in the memory 1005 and execute the processing method of the batch query request provided by the embodiment of the present invention.
The embodiment of the invention provides a method for processing batch query requests.
Referring to fig. 2, fig. 2 is a flowchart illustrating a method for processing a batch query request according to a first embodiment of the present invention.
In this embodiment, the method for processing the batch query request is applied to a device for processing the batch query request, and the method includes:
step S101, receiving batch query requests, configuring the priority of each query request based on preset dimensionality, and generating a batch to-be-looked-up table based on the priority of the query requests;
in this embodiment, the batch query request includes a large number of query requests. In this embodiment, a uniform parameter input format file is predefined, and a batch query request conforming to the parameter input format file is accessed. It will be appreciated that the formats of the batch query requests uploaded by different callers may not be consistent. Therefore, after receiving the batch query request uploaded by the caller, the batch query file needs to be analyzed to obtain the relevant parameters, and then the relevant parameters are recombined into the batch query request corresponding to the parameter input format file based on the parameter input format file to obtain the batch query request with the format conforming to the parameter input format, so that the access, subsequent analysis and conversion of the batch query request can be facilitated.
In this embodiment, the priority of the query request is configured based on the preset dimension. The preset dimension comprises at least one of a service line, a product and a subsystem. For example, the priority of the product 1 is set as the lowest priority, the priority of the product 2 is set as the third priority, the priority of the product 3 is set as the second priority, and the priority of the product requested to be increased in priority is set as the highest priority. It is to be understood that the priority setting may also be performed based on a combination of a plurality of dimensions in the preset dimension.
In this embodiment, the basic principle of setting the priority is as follows: the minimum query resource is guaranteed, and meanwhile, when a certain product is idle in query, the corresponding resource is occupied by other products.
For example, if the total number of resources is 10, product 1 allocates 1 resource, product 2 allocates 2 resources, product 3 allocates 3 resources, and when there is no request at the highest priority (4 resources), 4 file-processed resources are all allocated to product 3; if 6 files exist in the queue of the product 3, the product 3 occupies 6 resources, and the remaining 1 resource can be allocated to the product 2 first; if there are 2 files in the queue for product 3, product 3 occupies 2, the remaining 5 resources are allocated to product 2, and so on.
Referring to fig. 3, fig. 3 is a schematic view of a first embodiment of a method for processing a batch query request according to the present invention. After accessing the batch query requests, performing storage request distribution on the accessed batch query requests, configuring priorities based on product dimensions, acquiring query requests of all priorities, and forming a batch priority queue. In addition, it is considered that when the number of query requests is too large, the service of raising the priority request can be set based on the service scene or the customer requirement, that is, the query request with the lowest priority, the query request with the second priority or the query request with the third priority is raised to the highest priority.
Furthermore, the configuration of the priority is dynamically loaded, and manual real-time adjustment is supported, so that the configuration of the priority is adjusted in time when the query is blocked, and the stability of a business side system is ensured.
After the priority of each query request in the batch of query requests is configured, acquiring a single query request to be processed from the batch of query requests according to the sequence of the priorities from high to low; it will be appreciated that for a plurality of individual query requests having the same priority, the query requests may be sorted in time order, user name order, or the like, or randomly determined so as to obtain a query request arranged in a certain order.
Further, transferring a single query request to be processed into an online queuing table, and determining the queue depth of the online queuing table and/or the processing success rate of the query request to be processed; the queue depth refers to the number of query requests to be processed in the online queuing table, and is determined by the processing capacity of the processing device for batch query requests. The success rate refers to the ratio of the number of the query requests to be processed which can be normally processed by the system to the total number of the query requests to be processed.
And if the queue depth is greater than the depth threshold and/or the processing success rate is less than the success rate threshold, suspending the processing of the corresponding batch query request and receiving the next batch query request. It is understood that when the depth threshold is exceeded or the success rate is lower than the success rate threshold in the online queuing table when the batch query request is converted into a single query request, the batch query request of the batch is directly suspended and the batch query request of the next batch is accessed, so as to prevent the query queue from being blocked by query requests with too many requests or to increase the downstream processing pressure, which causes the downstream system to crash.
Specifically, the batch query requests are batch credit investigation query requests, and the success rate is a ratio of the number of credit investigation query requests inquiring to a credit investigation source to the total number of credit investigation query requests in the batch credit investigation query requests. In practical application, the conditions of all information sources in the flowing water are counted regularly according to categories and query channels, when the success rate is lower than a success rate threshold value within 5 minutes, the corresponding information source route is closed, the flow is switched to an available backup information source route, if no backup information source route exists, related batch query is also suspended, and query requests in batch query requests are continuously queried after the information source route is recovered.
After the credit investigation source route is closed, the application starts a detection task, and according to a preset detection strategy: and if the total query requests per minute are more than 10, the query success rate exceeds 90%, and the query success rate lasts for more than 5 minutes, 50% of the traffic of the credit investigation source route is recovered, and if the query success rate lasts for more than 10 minutes, 100% of the traffic of the credit investigation source route is recovered, and the batch of query requests in the batch are also recovered because the success rate exceeds the success rate threshold.
Step S102, starting a timing task, and acquiring a query request to be processed from the batch of tables to be looked up according to the priority;
and marking the state of the query request accessed into the batch of the table to be looked up as to be processed. And sequentially processing and acquiring the query requests to be processed according to the priority of each query request to be processed and the sequence from the highest priority to the lowest priority by a preset timing task, and marking the state of the acquired query requests to be processed as processing.
Step S103: after the duplication elimination processing is carried out on the thread processing the batch query requests, downloading corresponding request files based on the identity identification of the query request to be processed, and reading the request content of each request file;
for a distributed system, in order to ensure a normal query request processing flow, batch query requests of a batch are downloaded by only one thread at the same time, so that duplicate removal processing is required to avoid resource waste caused by access of multiple threads to batch processing requests of the batch. And downloading corresponding request files according to the identity of the query request to be processed, reading the request content of each request file, and downloading the request content to the local.
And after downloading the request content to the local, selecting batch files, and acquiring files of various priorities according to priority batches. With continued reference to fig. 3, the lowest priority corresponds to 10% of the requested files, the third priority corresponds to 20% of the requested files, the second priority corresponds to 30% of the requested files, and the highest priority corresponds to 40% of the requested files, so that the files are sequentially queried by the corresponding threads after being acquired.
Specifically, determining the number of files of the request file;
if the number of the files is larger than a first preset number, reading the request content of the request file in an iterator mode; and if the number of the files is less than the second preset number, accessing the next request file until the number of the accessed request files reaches a third preset number, and reading the request content of each request file. The first preset number, the second preset number and the third preset number are single batch processing stroke number threshold values and are specifically set according to actual conditions. The threshold value of the number of single batch processing strokes is set to prevent the overflow of the memory when the files are read easily caused by the overlarge request files (the number of the request files is larger than a first preset number); an iterator mode (reading non-full files) is adopted during reading, when the number of the requested files is larger than the first preset number, batch query requests of 1 batch are processed each time, and when the number of the requested files is smaller than the second preset number, batch query requests of multiple batches can be processed because the third preset number is not reached, so that breakpoint continuous operation based on the processing amount is realized, the load of a system is not exceeded, and high-efficiency processing of the query requests can be realized.
Further, it is also necessary to set a corresponding breakpoint for the application stop condition, specifically, it is monitored whether the application is stopped during the process of reading the request content; if the application is monitored to stop, backing up the read request content and/or storing the interrupt position read when the application is stopped, and accessing the next batch of request files.
Specifically, in the process of reading the request file, if the application stops, the interruption position of the request file is saved; when the application is restarted, accessing and processing the next batch of batch query requests, or jumping to the recorded interruption position of the last request file, so as to filter the processed request file (if the request file is not interrupted, the application starts to process from the first line of the file).
When the application stops, the application needs to be stopped elegantly, namely, information such as an interruption position in a request file is backed up first, and the application information is ensured not to be lost due to the stop of the application; if the application is interrupted accidentally (for example, power failure), after the application is restarted, the 'interrupted position' of the log printing can be intercepted manually, the relevant records are updated, and the processing of the corresponding batch query requests can be recovered.
And step S104, inquiring through the request content of the to-be-processed inquiry request through a preset thread to obtain an inquiry result.
Specifically, the read content of the request file is submitted to a batch thread pool, and the asynchronous threads in the batch thread pool perform online query on each query request to be processed respectively, and return a query result to a calling party.
The batch thread pool comprises at least one asynchronous thread, so that multithread processing can be realized based on the batch thread pool, a single request is stored in an online flow water meter and enters a queuing table based on priority, a plurality of asynchronous threads simultaneously and respectively carry out online query on each query request to be processed, corresponding query results are respectively obtained, a report is generated based on the query results, and then the report comprising the query results is returned to a corresponding caller.
According to the scheme, the method comprises the steps of receiving batch query requests, configuring the priority of each query request based on the preset dimension, and generating a batch to-be-searched list based on the priority of the query requests; starting a timing task, and acquiring a query request to be processed from the batch of tables to be looked up according to the priority; and querying through the request content of the query request to be processed by a preset thread to obtain a query result. Therefore, the priority of each query request in the batch of query requests is set based on the preset dimensionality, then the requests to be processed are obtained based on the priority, and query is sequentially carried out to obtain query results, so that the flexibility of processing batch query is improved, and the maximization of resource utilization is realized.
Based on the first embodiment shown in fig. 2, a second embodiment of the present invention provides a method for processing a batch of query requests. Specifically, referring to fig. 4, fig. 4 is a flowchart illustrating a processing method of a batch query request according to a second embodiment of the present invention. As shown in fig. 4, the step S104 further includes:
step S104-1: and checking the parameters of the request content, and accessing the query request to be processed to the preset thread based on a preset flow and/or quantity control mechanism after the checking is successful.
In this embodiment, parameter verification is implemented based on the annotation expression, for example:
@SerializedName(“onlylocalquery”)
@ Length (max ═ 1, message ═ 1 maximum Length of onlylocalquery)
@ pattern (regexp ═ 0-1 {1} ", message ═ onlylocalquery value 0 or 1")
Protected String onlyLocalQuery＝“0”
In addition, in order to prevent the data source system from being abnormal due to excessive query requests accessed by each thread, and the request is overtime, the flow control needs to be performed on the accessed query requests.
In this embodiment, the control mechanism includes a flow control mechanism and a numerical control mechanism.
The step of accessing the query request to be processed to the preset thread based on a flow control mechanism comprises the following steps:
determining the size of a data packet of the query request to be processed and the size of a buffer idle area of each preset thread; and determining a target preset thread accessed to the query request to be processed based on the size of the data packet and the size of the buffer idle area, and accessing the query request to be processed to the target preset thread.
It can be understood that, when communicating with the client, the sending rate and the receiving rate of the query request are not necessarily equal, if the sending rate is too fast, the batch query request cannot be processed, and at this time, only the data that cannot be processed cannot be stored in the buffer of each preset thread (the out-of-order data packet is also stored in the buffer). If the cache region is full of the continuous access query request, the corresponding thread may drop the received data packet, and a large amount of packet loss may greatly waste network resources, so that the access traffic needs to be controlled.
The method comprises the steps of firstly obtaining the size of a data packet of the query request to be processed, obtaining the size of a cache idle area of each preset thread, determining the preset threads with the size of the cache idle area larger than that of the data packet as candidate preset threads, and then selecting a target preset thread based on a first target preset thread determination rule. The first target preset thread determining rule can be randomly allocated, allocated according to names, sequentially allocated according to time sequence, and allocated with preset threads with larger buffer idle area preferentially. Therefore, the to-be-processed query requests accessed in each target preset thread can be ensured not to exceed the traffic load.
The step of accessing the query request to be processed to the preset thread based on a quantity control mechanism comprises the following steps:
determining the number of accessed requests of the accessed to-be-processed query requests in each preset thread; and determining a target preset thread based on the accessed quantity and the processing capacity of the corresponding thread, wherein the query request to be processed is accessed to the target preset thread. Specifically, candidate preset threads are determined based on the number of accessed requests and the maximum processing capacity in each thread, and then a target preset thread is determined based on a second target preset thread. The first target preset thread determining rule can be randomly allocated, allocated according to names, sequentially allocated according to time sequence, and allocated with preset threads with larger buffer idle area preferentially. Therefore, the to-be-processed query requests accessed in each target preset thread can be ensured not to exceed the traffic load.
It will be appreciated that the flow control mechanism and the quantity control mechanism may be used simultaneously or separately.
In addition, the to-be-processed inquiry request may be accessed to the preset thread based on a concurrency control mechanism, that is, whether to access the to-be-inquired request is determined by controlling the number of the concurrently executed preset threads, and strict limitation may be made on TPS (transaction per second) to control the flow rate.
According to the scheme, the parameters of the request content are verified, and the query request to be processed is accessed to the preset thread based on the preset flow and/or quantity control mechanism after the verification is successful, so that query blockage can be prevented, and normal operation of each service system is guaranteed.
In addition, the embodiment also provides a device for processing the batch query requests. Referring to fig. 5, fig. 5 is a functional module diagram of a processing apparatus for batch query requests according to a first embodiment of the present invention.
In this embodiment, the processing apparatus of the batch query request is a virtual apparatus, and is stored in the memory 1005 of the processing device of the batch query request shown in fig. 1, so as to implement all functions of the processing program of the batch query request: the query system is used for receiving batch query requests, configuring the priority of each query request based on preset dimensionality and generating a batch to-be-searched list based on the priority of the query requests; the system is used for starting a timing task and acquiring a query request to be processed from the batch of tables to be looked up according to the priority; and the query is carried out through the request content of the query request to be processed through a preset thread to obtain a query result.
Specifically, the processing device for the batch query request includes:
the receiving module 10 is configured to receive the batch query requests, configure the priority of each query request based on a preset dimension, and generate a batch to-be-looked up table based on the priority of the query requests;
an obtaining module 20, configured to start a timing task, and obtain a to-be-processed query request from the batch of to-be-looked-up tables according to the priority;
a reading module 30, configured to download corresponding request files based on the identity of the query request to be processed after performing deduplication processing on the threads that process the batch query requests, and read request content of each request file;
and the query module 40 is configured to perform query through the request content of the query request to be processed by using a preset thread to obtain a query result.
Further, the receiving module is further configured to:
acquiring a single query request to be processed from the batch of query lists according to the sequence of the priorities from high to low;
transferring a single query request to be processed into an online queuing table, and determining the queue depth of the online queuing table and/or the processing success rate of the query request to be processed;
and if the queue depth is greater than the depth threshold and/or the processing success rate is less than the success rate threshold, suspending the processing of the corresponding batch query request and receiving the next batch query request.
Further, the obtaining module is further configured to:
and after carrying out deduplication processing on the thread processing the batch query requests, downloading corresponding request files based on the identity of the query requests to be processed, and reading the request content of each request file.
Further, the obtaining module is further configured to:
monitoring whether the application stops or not in the process of reading the request content;
if the application is monitored to stop, backing up the read request content and/or storing the interrupt position read when the application is stopped, and accessing the next batch of request files.
Further, the obtaining module is further configured to:
determining the number of files of the request file;
if the number of the files is larger than a first preset number, reading the request content of the request file in an iterator mode;
and if the number of the files is less than the second preset number, accessing the next request file until the number of the accessed request files reaches a third preset number, and reading the request content of each request file.
Further, the query module is further configured to:
and submitting the read content of the request file to a batch thread pool, respectively carrying out online inquiry on each inquiry request to be processed by an asynchronous thread in the batch thread pool, and returning an inquiry result to a calling party.
Further, the reading module is further configured to:
and submitting the read content of the request file to a batch thread pool, respectively carrying out online inquiry on each inquiry request to be processed by an asynchronous thread in the batch thread pool, and returning an inquiry result to a calling party.
Further, the reading module is further configured to:
determining the size of a data packet of the query request to be processed and the size of a buffer idle area of each preset thread;
determining a target preset thread accessed to the query request to be processed based on the size of the data packet and the size of the buffer idle area, and accessing the query request to be processed to the target preset thread; and/or
Determining the number of accessed requests of the accessed to-be-processed query requests in each preset thread;
and determining a target preset thread based on the accessed quantity and the processing capacity of the corresponding thread, wherein the query request to be processed is accessed to the target preset thread.
In addition, an embodiment of the present invention further provides a computer-readable storage medium, where a batch query request handler is stored on the computer-readable storage medium, and when the batch query request handler is executed by a processor, the steps of the method for processing the batch query request are implemented.
Compared with the prior art, the processing method, the processing device, the processing equipment and the computer-readable storage medium for the batch query requests provided by the invention receive the batch query requests, configure the priority of each query request based on the preset dimensionality, and generate a batch to-be-checked list based on the priority of the query requests; starting a timing task, and acquiring a query request to be processed from the batch of tables to be looked up according to the priority; after the duplication elimination processing is carried out on the thread processing the batch query requests, downloading corresponding request files based on the identity identification of the query request to be processed, and reading the request content of each request file; and querying through the request content of the query request to be processed by a preset thread to obtain a query result. Therefore, the priority of each query request in the batch of query requests is set based on the preset dimensionality, then the requests to be processed are obtained based on the priority, and query is sequentially carried out to obtain query results, so that the flexibility of processing batch query is improved, and the maximization of resource utilization is realized.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM/RAM, magnetic disk, and optical disk) as described above, and includes several instructions for enabling a terminal device to execute the method according to the embodiments of the present invention.
The above description is only for the preferred embodiment of the present invention and is not intended to limit the scope of the present invention, and all equivalent structures or flow transformations made by the present specification and drawings, or applied directly or indirectly to other related arts, are included in the scope of the present invention.
Claims (10)
1. A method for processing a batch of query requests, the method comprising:
receiving batch query requests, configuring the priority of each query request based on preset dimensionality, and generating a batch to-be-searched list based on the priority of the query requests;
starting a timing task, and acquiring a query request to be processed from the batch of tables to be looked up according to the priority;
after the duplication elimination processing is carried out on the thread processing the batch query requests, downloading corresponding request files based on the identity identification of the query request to be processed, and reading the request content of each request file;
and querying through the request content of the query request to be processed by a preset thread to obtain a query result.
2. The method of claim 1, wherein obtaining pending query requests from the batched pending forms according to the priority comprises:
acquiring a single query request to be processed from the batch of query lists according to the sequence of the priorities from high to low;
the obtaining of the single query request to be processed from the batch of the query lists according to the sequence of the priorities from high to low further comprises:
transferring a single query request to be processed into an online queuing table, and determining the queue depth of the online queuing table and/or the processing success rate of the query request to be processed;
and if the queue depth is greater than the depth threshold and/or the processing success rate is less than the success rate threshold, suspending the processing of the corresponding batch query request and receiving the next batch query request.
3. The method according to claim 1, wherein said reading the requested content of each of said requested files comprises:
monitoring whether the application stops or not in the process of reading the request content;
if the application is monitored to stop, backing up the read request content and/or storing the interrupt position read when the application is stopped, and accessing the next batch of request files.
4. The method according to claim 1, wherein said reading the requested content of each of said requested files comprises:
determining the number of files of the request file;
if the number of the files is larger than a first preset number, reading the request content of the request file in an iterator mode;
and if the number of the files is less than the second preset number, accessing the next request file until the number of the accessed request files reaches a third preset number, and reading the request content of each request file.
5. The method according to claim 1, wherein the obtaining a query result by querying through the content of the request file through a preset thread comprises:
and submitting the read content of the request file to a batch thread pool, respectively carrying out online inquiry on each inquiry request to be processed by an asynchronous thread in the batch thread pool, and returning an inquiry result to a calling party.
6. The method according to claim 1, wherein the reading is queried through a predetermined thread by request content of the query request to be processed, and a query result is obtained, and before the reading, the method further comprises:
and checking the parameters of the request content, and accessing the query request to be processed to the preset thread based on a preset flow and/or quantity control mechanism after the checking is successful.
7. The method according to claim 6, wherein the accessing the pending query request to the predetermined thread based on a predetermined flow and/or quantity control mechanism comprises:
determining the size of a data packet of the query request to be processed and the size of a buffer idle area of each preset thread;
determining a target preset thread accessed to the query request to be processed based on the size of the data packet and the size of the buffer idle area, and accessing the query request to be processed to the target preset thread; and/or
Determining the number of accessed requests of the accessed to-be-processed query requests in each preset thread;
and determining a target preset thread based on the accessed quantity and the processing capacity of the corresponding thread, wherein the query request to be processed is accessed to the target preset thread.
8. An apparatus for processing a batch of query requests, the apparatus comprising:
the receiving module is used for receiving the batch query requests, configuring the priority of each query request based on the preset dimensionality and generating a batch to-be-looked-up table based on the priority of the query requests;
the acquisition module is used for starting a timing task and acquiring the query requests to be processed from the batch of tables to be looked up according to the priority;
the reading module is used for downloading corresponding request files based on the identity of the query request to be processed after the duplication elimination processing is carried out on the threads for processing the batch query requests, and reading the request content of each request file;
and the query module is used for querying through the request content of the query request to be processed through a preset thread to obtain a query result.
9. A device for processing a batch query request, the device comprising a processor, a memory and a handler of the batch query request stored in the memory, wherein when the handler of the batch query request is executed by the processor, the steps of the method for processing a batch query request according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, on which a handler of a batch query request is stored, the handler of the batch query request implementing the steps of the method for processing a batch query request according to any one of claims 1 to 7 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011514804.5A CN112632363B (en) | 2020-12-18 | Method, device and equipment for processing batch query requests and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011514804.5A CN112632363B (en) | 2020-12-18 | Method, device and equipment for processing batch query requests and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112632363A true CN112632363A (en) | 2021-04-09 |
| CN112632363B CN112632363B (en) | 2024-11-08 |
Family
ID=
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112800318A (en) * | 2021-04-14 | 2021-05-14 | 统信软件技术有限公司 | Mail searching method and computing device |
| CN114860814A (en) * | 2022-07-05 | 2022-08-05 | 天津金城银行股份有限公司 | Credit investigation processing system, method, credit investigation system, device and medium |
| CN114912995A (en) * | 2022-03-03 | 2022-08-16 | 平安消费金融有限公司 | Credit report query method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100083264A1 (en) * | 2008-09-30 | 2010-04-01 | Abhay Mehta | Processing Batch Database Workload While Avoiding Overload |
| CN106126354A (en) * | 2016-06-21 | 2016-11-16 | 中国建设银行股份有限公司 | A kind of asynchronous batch processing method and system |
| CN109840144A (en) * | 2018-12-28 | 2019-06-04 | 航天信息股份有限公司 | A kind of information service dispatching method and system for the request of across mechanism bulk service |
| CN110489447A (en) * | 2019-07-16 | 2019-11-22 | 招联消费金融有限公司 | Data query method, apparatus, computer equipment and storage medium |
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100083264A1 (en) * | 2008-09-30 | 2010-04-01 | Abhay Mehta | Processing Batch Database Workload While Avoiding Overload |
| CN106126354A (en) * | 2016-06-21 | 2016-11-16 | 中国建设银行股份有限公司 | A kind of asynchronous batch processing method and system |
| CN109840144A (en) * | 2018-12-28 | 2019-06-04 | 航天信息股份有限公司 | A kind of information service dispatching method and system for the request of across mechanism bulk service |
| CN110489447A (en) * | 2019-07-16 | 2019-11-22 | 招联消费金融有限公司 | Data query method, apparatus, computer equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112800318A (en) * | 2021-04-14 | 2021-05-14 | 统信软件技术有限公司 | Mail searching method and computing device |
| CN112800318B (en) * | 2021-04-14 | 2021-07-16 | 统信软件技术有限公司 | Mail searching method and computing device |
| CN114912995A (en) * | 2022-03-03 | 2022-08-16 | 平安消费金融有限公司 | Credit report query method and device, computer equipment and storage medium |
| CN114860814A (en) * | 2022-07-05 | 2022-08-05 | 天津金城银行股份有限公司 | Credit investigation processing system, method, credit investigation system, device and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107832126B (en) | Thread adjusting method and terminal thereof | |
| CN108683604B (en) | Concurrent access control method, terminal device, and medium | |
| CN110489447B (en) | Data query method and device, computer equipment and storage medium | |
| US5797005A (en) | Shared queue structure for data integrity | |
| CN112256417B (en) | Data request processing method and device and computer readable storage medium | |
| CN110597858A (en) | Task data processing method and device, computer equipment and storage medium | |
| US9552386B2 (en) | Reducing lock occurrences in server/database systems | |
| US9477618B2 (en) | Information processing device, information processing system, storage medium storing program for controlling information processing device, and method for controlling information processing device | |
| CN111901249A (en) | Service current limiting method, device, equipment and storage medium | |
| US20190332328A1 (en) | Storage Controller and IO Request Processing Method | |
| CN111586140A (en) | Data interaction method and server | |
| CN113296929A (en) | Resource matching method, device and system based on cloud computing | |
| CN109522273B (en) | Method and device for realizing data writing | |
| WO2019227623A1 (en) | Method and apparatus for processing data of call platforms, device, and storage medium | |
| US11444998B2 (en) | Bit rate reduction processing method for data file, and server | |
| CN115658292A (en) | Resource scheduling method, device, computer equipment and storage medium | |
| CN108388409B (en) | Print request processing method, apparatus, computer device and storage medium | |
| CN116301568A (en) | Data access method, device and equipment | |
| CN107239543A (en) | Application icon processing method and processing device | |
| CN112632363A (en) | Method, device and equipment for processing batch query requests and readable storage medium | |
| CN110750350B (en) | Large resource scheduling method, system, device and readable storage medium | |
| CN112632363B (en) | Method, device and equipment for processing batch query requests and readable storage medium | |
| WO2017167070A1 (en) | Method for copying clustered data, and method and device for determining priority | |
| US9659041B2 (en) | Model for capturing audit trail data with reduced probability of loss of critical data | |
| CN110955502B (en) | Task scheduling method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |