CN118012902A - Database query method and device, electronic equipment and storage medium - Google Patents
Database query method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN118012902A CN118012902A CN202410235685.1A CN202410235685A CN118012902A CN 118012902 A CN118012902 A CN 118012902A CN 202410235685 A CN202410235685 A CN 202410235685A CN 118012902 A CN118012902 A CN 118012902A
- Authority
- CN
- China
- Prior art keywords
- target
- information
- memory
- query
- request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 238000013507 mapping Methods 0.000 claims abstract description 27
- 230000000875 corresponding effect Effects 0.000 claims description 109
- 238000004364 calculation method Methods 0.000 claims description 33
- 238000004590 computer program Methods 0.000 claims description 28
- 230000008859 change Effects 0.000 claims description 16
- 230000002596 correlated effect Effects 0.000 claims description 3
- 238000004891 communication Methods 0.000 description 8
- 238000012545 processing Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000005070 sampling Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 230000002085 persistent effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a database query method, a database query device, electronic equipment and a storage medium, and relates to the technical field of big data. The method comprises the following steps: acquiring a target query request, and acquiring target memory occupation information according to target identification information corresponding to the target query request, wherein the target memory occupation information represents the mapping relation between a single data object and occupied memory in a query result; acquiring the number of data objects corresponding to the query result of the target query request, and determining estimated memory occupation information of the target query request according to the number of data objects and the target memory occupation information; and if the estimated memory occupation information meets a preset interception condition, intercepting the target query request. The technical scheme provided by the invention can avoid fullgc problems during database query and improve the smoothness and stability of application program operation.
Description
Technical Field
The present invention relates to the field of big data, and in particular, to a database query method, a database query device, an electronic device, and a storage medium.
Background
The flow of an application database query is typically: loading a JDBC (Java DataBase Connectivity, java database connection) driver; establishing database connection; executing the query and returning a result; packaging the business objects through a frame layer; closing the database connection. If the single query results return a large amount, fullgc (Full Garbage Collection overall system garbage collection) is very easy to cause, resulting in slow application response, memory overflow, and even application unavailability.
Disclosure of Invention
The invention provides a database query method, a device, electronic equipment and a storage medium, and by the method provided by the embodiment of the invention, the problem that fullgc is easy to cause when a large result is returned during database query can be solved, and the smoothness and stability of application running are improved.
In a first aspect, an embodiment of the present invention provides a database query method, including:
Acquiring a target query request, and acquiring target memory occupation information according to target identification information corresponding to the target query request, wherein the target memory occupation information represents the mapping relation between a single data object and occupied memory in a query result;
acquiring the number of data objects corresponding to the query result of the target query request, and determining estimated memory occupation information of the target query request according to the number of data objects and the target memory occupation information;
And if the estimated memory occupation information meets a preset interception condition, intercepting the target query request.
In a second aspect, an embodiment of the present invention provides a database query apparatus, including:
the information acquisition module is used for acquiring a target query request and acquiring target memory occupation information according to target identification information corresponding to the target query request, wherein the target memory occupation information represents the mapping relation between a single data object and occupied memory in a query result;
the memory occupation estimating module is used for acquiring the number of data objects corresponding to the query result of the target query request and determining estimated memory occupation information of the target query request according to the number of data objects and the target memory occupation information;
And the request interception module is used for intercepting the target query request if the estimated memory occupation information meets a preset interception condition.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
The database query method is characterized by comprising a memory, a processor and a computer program stored on the memory and capable of being run by the processor, wherein the processor executes the computer program to realize the database query method according to any one of the embodiments of the invention.
In a fourth aspect, an embodiment of the present invention provides a computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a database query method according to any of the embodiments of the present invention.
In a fifth aspect, embodiments of the present invention further provide a computer program product comprising a computer program which, when executed by a processor, implements a database query method according to any of the embodiments of the present invention.
The embodiment of the invention provides a database query method, a device, electronic equipment and a storage medium, which can be used for matching corresponding target memory occupation information according to target identification information corresponding to a target query request to obtain the mapping relation between a single data object and occupation content, further determining estimated memory occupation information of the target query request according to the number of data objects corresponding to the query result of the target query request and the mapping relation between the single data object and occupation content, accurately estimating the memory occupied by the query result of the target query request, intercepting the target query request of which the memory occupation information meets preset interception conditions, and realizing interception of the query request possibly causing fullgc from the source.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a database query method provided according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for determining a mapping relationship between a single data object and occupied memory according to an embodiment of the present invention;
FIG. 3 is a flow chart of a database query method provided in accordance with an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a database query device according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical scheme of the application obtains, stores, uses, processes and the like the data, which all meet the relevant regulations of national laws and regulations.
Currently, in order to prevent fullgc from being caused by a large-result query, the main solutions in the industry mainly include the following solutions:
The method is simple and violent, and due to the development of services, some special service scenes, such as full data export and other service scenes, the maximum result set is defined as how many cannot be accurately quantized, or fullgc hidden danger can be caused.
With the use of streaming query, one database connection can only serve one returned result at a time, and if the returned result is not closed, the use of other database connections is affected, the database connection time is long, and network congestion may be caused.
When a query statement is executed, a cursor is opened at the client and the server, and a memory space is respectively applied as a buffer area for storing data objects corresponding to query results, and the number of data stored in the whole buffer area is determined by the fetch size. The cursor inquiry has higher program literacy requirement on users, has limited application range and is not easy to popularize and implement.
In order to solve the related problems in the related art, a method for avoiding fullgc problems caused by a large result is provided.
Fig. 1 is a schematic diagram of a database query method according to an embodiment of the present invention, where the method may be applied to a database query scenario, and particularly when the number of data objects of a database query result is large, the problem of memory overflow, memory leakage, and the like is easily caused by fullgc.
Step 110, acquiring a target query request, and acquiring target memory occupation information according to target identification information corresponding to the target query request, wherein the target memory occupation information represents a mapping relation between a single data object and occupied memory in a query result.
The target query request may be a database search request, including a database query statement, a database query field, etc., through which a corresponding query result may be queried from the database, where the query result may include at least one data object, that is, a target query request may return a plurality of data objects; the target identification information may be used to uniquely identify the target query request. Because the candidate memory occupancy information determined based on the historical query request is stored in association with the identification information of the historical query request, the target memory occupancy information can be determined according to the target identification information, wherein the target memory occupancy information characterizes the mapping relation between a single data object and occupied memory in the query result. Further, the mapping relationship between a single data object and occupied memory can be understood as the actual memory occupied by the single data object.
The target query request is obtained, the target identification information corresponding to the target query request is obtained, and the corresponding target memory occupation information can be matched through the target identification information, so that the mapping relation between a single data object and occupied memory in the query result corresponding to the target query request is obtained.
Exemplary, the target identification information of the target query request is obtained, the target identification information and the candidate identification information are further matched, and the target memory occupancy information is determined from the candidate memory occupancy information according to the candidate identification information matched with the target identification information.
Step 120, obtaining the number of data objects corresponding to the query result of the target query request, and determining estimated memory occupation information of the target query request according to the number of data objects and the target memory occupation information.
The estimated memory occupation information is information of the actual memory size occupied by the query result of the target query request to be stored or processed.
Illustratively, an object number query statement is determined from the target query request, wherein the object number query statement is used to request from the database a number of data objects included in a query result of the target query request. And inquiring the database according to the object number inquiry statement to obtain the number of data objects corresponding to the inquiry result of the target inquiry request. The estimated memory occupation information is used as an estimated value, and the multiplication result of the number of data objects and the target memory occupation information is calculated to obtain the estimated memory occupation information.
And 130, intercepting the target query request if the estimated memory occupation information meets a preset interception condition.
The preset interception conditions are used for representing conditions for intercepting query requests of the application program to the database.
Specifically, if the memory occupation size represented by the estimated memory occupation information corresponding to the target query request meets the preset interception condition, the memory occupied by all the data objects returned by the database corresponding to the target query request is excessive and may cause fullgc, so that the target query request is intercepted in order to avoid the situations that the response of the application program is slow, the memory overflows and even the application program is unavailable due to the excessive amount of returned data. If the memory occupancy size represented by the estimated memory occupancy information corresponding to the target query request does not meet the preset interception condition, the target query request does not need to be intercepted.
Optionally, the invention further provides a method for determining the mapping relation between the single data object and the occupied memory. Fig. 2 is a flowchart of a method for determining a mapping relationship between a single data object and occupied memory according to an embodiment of the present invention. As shown in fig. 2, the method includes:
Step 210, request information and result information of the historical query request are periodically acquired.
The historical query request may be understood as a database query request in a historical time window, where the historical time window may be a time interval set according to an actual requirement, and the embodiment of the present invention is not limited specifically. And if the sampling event is triggered, acquiring a database query request of the application program in the historical time window. Wherein the sampled event represents an event of the acquisition history query request. The sampling event may be triggered periodically, or in response to a manual request, or the number of clicks of the application may be detected to satisfy a set number of thresholds, etc. The disclosed embodiments are not particularly limited to triggering conditions for sampling events.
The request information includes history identification information of a history query request, a query sentence, and the like, wherein the query sentence includes a plurality of fields. The result information comprises the number of data objects corresponding to the query result returned by the historical query request, the actual memory space occupied after the data objects are converted into service objects under the persistent layer framework, and the like.
Illustratively, the application, upon obtaining the query request, determines whether the query request contains a query statement of the persistence tier framework. If so, acquiring a unique identifier of a precompiled query statement corresponding to the query request, the query statement, the number of data objects of a query result corresponding to the query statement, and the actual memory space occupied after the data objects are converted into service objects under a persistent layer framework, packaging the acquired information into objects, and asynchronously sending the objects to a target service. The target service is an acquisition service for executing the operation of determining the mapping relation between the single data object and the occupied memory, which is provided by the embodiment of the invention, so that the influence on the performance of the application program caused by determining the mapping relation between the single data object and the occupied memory can be reduced. Optionally, the target service and the application are configured in the same server or in the same server cluster.
Step 220, determining the target byte number corresponding to the historical query result according to the request information.
The historical query result may include a plurality of historical data objects, and the target byte number is used for representing the maximum byte number of the historical query result at the database attribute definition level. The target number of bytes may be determined based on the number of bytes occupied by the field type of each field included in the historical query statement. For example, a historical query statement contains a number of fields, the field type of each field characterizes the number of bytes that the field occupies at the database attribute definition level.
Exemplary, a history query statement in the request information is obtained, and a target byte number corresponding to the history query result is determined according to the byte number corresponding to the field type of the field in the history query statement.
In particular, it is possible toAnd determining the target byte number corresponding to the historical query result, wherein n is the total number of fields in the query request, and byte i is the byte number of the database program statement corresponding to the ith field.
If the program statement of the history query request is select a, B, and C, where a and B are int types, and C is long, further, in the database attribute definition layer, the int type occupies 4 bytes, and the long type occupies 8 bytes, so that the maximum byte number of the data object in the history query result can be estimated to be 16 bytes, that is, the target byte number ms=4+4+8=16. It should be noted that, the content corresponding to a certain field may be empty, for example, the field a is the age, and a certain piece of data object may be empty in the column of the age, so the maximum number of bytes corresponding to the historical query result is 16 bytes, and the target number of bytes corresponding to the historical query result is determined to be 16 bytes, so as to ensure the accuracy of the estimated memory usage information.
In some embodiments, a calculation task in a task queue is obtained, where the calculation task is used to calculate a target byte number corresponding to a history query result of an incremental query request or a change query request;
and determining the target byte number corresponding to the historical query result according to the request information corresponding to the increment query request or the change query request.
The query statement is typically fixed for the application, but when a database version iteration occurs, an incremental query statement or a change query statement may occur. Wherein the change query term indicates a query term having a content change. The target service needs to identify the increment query statement or the change query statement so as to calculate the target byte number corresponding to the history query result of the increment query request or the change query request.
Optionally, before the acquiring the computing task in the task queue, the method further includes:
Matching the identification information of the historical query request with candidate identification information;
And generating a calculation task according to the request information of the historical query request, and adding the calculation task into a task queue for the historical query request which is not matched with the candidate identification information.
For example, after acquiring the request information and the result information of the history query request, the target service acquires the identification information of the history query request in the request information, matches the acquired identification information with the candidate identification information, and if there is no matched candidate identification information, determines the history query request as an incremental query request. And generating a calculation task according to the query statement corresponding to the increment query request, and adding the calculation task into a task queue.
Further, the method further comprises the following steps:
judging whether query sentences in request information of the historical query requests are changed or not according to the historical query requests matched with the candidate identification information;
and if the query statement is changed, generating a calculation task according to the request information of the historical query request, and adding the calculation task into a task queue.
For example, if there is matching candidate identification information and a change occurs in the query statement, the history query request is determined as a change query request. And generating a calculation task according to the query statement corresponding to the change query request, and adding the calculation task into a task queue.
It should be noted that, the history query request of the unmatched candidate identification information is the candidate identification information that needs to be added, i.e. the incremental query request. The history inquiry request of the matched candidate identification information is changed due to version iteration or update of the inquiry statement in the request information, and the history inquiry request is the change inquiry request.
Step 230, determining the number of memory occupied bytes corresponding to the single data object according to the historical memory occupied information corresponding to the result information and the number of the historical data objects.
For each historical query request corresponding to the computing task, determining historical memory occupation information and the number of historical data objects according to result information corresponding to the historical query request;
and determining the average memory occupied byte number of the single data object according to the historical memory occupied information and the number of the historical data objects, and taking the average memory occupied byte number of the single data object as the memory occupied byte number of the single data object corresponding to the calculation task.
In particular, it is possible toAnd determining the number of memory occupied bytes of a single data object, wherein B i is the historical memory occupied information of the ith historical query request, C i is the number of data objects in the query result of the ith historical query request, and n is the number of query requests identical to the identification information of the historical query requests. The historical memory occupation information can be divided by the number of the historical data objects, and the quotient is the average memory occupation byte number of the single data object, so that the average memory occupation byte number can be used as the memory occupation byte number of the single data object corresponding to the calculation task.
Step 240, determining candidate memory occupation information according to the memory occupied byte number and the target byte number corresponding to the single data object, and storing the candidate memory occupation information and the candidate identification information in the request information in a correlated manner.
The current memory occupation information is calculated according to the memory occupation byte number and the target byte number corresponding to the calculation task. And determining candidate memory occupancy information according to the current memory occupancy information and the historical memory occupancy information corresponding to the computing task. And storing the candidate memory occupation information and the candidate identification information in the request information in a correlated way.
In particular, it is possible toDetermining the actual occupied memory of a single piece of data, wherein/>For the memory occupied byte number of the ith data object, MC is the target byte number of the ith data object, m is the cycle iteration number, and the cycle can be the sampling cycle of the query request.
Further, the maximum byte number of the single data object at the database attribute definition level can be determined through step 220, the size of the memory space actually occupied by the single data object is determined through step 230, and further, the division operation result of the size of the memory space actually occupied by the single data object and the maximum byte number of the data object at the database attribute definition level is calculated, so as to obtain the memory space occupied by the single byte. And calculating the average value of the memory space occupied by the single byte corresponding to the sampling periods through step 240, and determining candidate memory occupation information according to the average value and the maximum byte number of the single data object in the database attribute definition layer. And taking the candidate identification information in the request information of the historical query request as a key, taking the candidate memory occupation information as a value, and adopting a key value pair mode to store the candidate memory occupation information and the candidate identification information in the request information in a correlation mode.
The embodiment of the invention provides a database query method, which can be used for matching corresponding target identification information according to a target query request to corresponding target memory occupation information to obtain a mapping relation between a single data object and occupation contents, and further determining estimated memory occupation information of the target query request according to the number of data objects corresponding to the query result of the target query request and the mapping relation between the single data object and the occupation contents.
Fig. 3 is a flowchart of another database query method according to an embodiment of the present invention, where the steps of the database query method based on the foregoing embodiments are further defined. As shown in fig. 3, includes:
step 310, a target query request is obtained.
Step 320, obtaining target identification information corresponding to the target query request.
Step 330, determining target memory occupancy information corresponding to the target identification information according to the target identification information and a corresponding relation between the preset candidate identification information and the candidate memory occupancy information.
Specifically, target identification information in request information of a target query request is obtained, further, each piece of candidate identification information corresponds to one piece of candidate memory occupation information, the target identification information is matched with the candidate identification information, and the target memory information is determined from the candidate memory occupation information according to the candidate identification information matched with the target identification information.
Step 340, executing the object number query statement corresponding to the target query request to obtain the number of data objects corresponding to the query result of the target query request.
Step 350, determining a mapping relation between a single data object and occupied memory according to the target memory occupancy information, and determining estimated memory occupancy information of the target query request by combining the mapping relation and the number of the data objects.
Wherein, the number of the data objects returned by the target query request can be determined by the object number query statement.
Further, since the target memory information is the actual memory occupation of a single data object, the estimated memory occupation information of the target query request can be determined by the product of the target memory information and the number of data objects.
For example, if the memory occupation represented by the target memory information is 5000kb, at the same time, it may be determined by the object number query statement that the query result of the target query request has 100 pieces of object data, so that the query result of the target query request occupies 100×5000kb of real memory, and further, it may be determined whether to intercept the target query information by comparing 500000 bytes with a preset interception condition.
Step 360, determining whether the estimated memory usage information meets a preset interception condition, if yes, executing step 370, otherwise executing step 390.
Optionally, the preset interception condition is determined based on an available memory and an available memory usage threshold, wherein the available memory usage threshold is determined based on the number of services and the concurrency of the services.
The available memory is the available memory of the computer equipment to which the application program belongs, and the available memory usage threshold value is used for representing the current available memory usage proportion.
If the query result of the target query request occupies more than the product of the available memory and the available memory usage threshold, then processing or saving the query result of the target query request may cause fullgc problems, so that the target query request needs to be intercepted.
And 370, intercepting the target query request.
And 380, generating interception information according to the target query request, the identification information, the attribute information and the execution information of the target query request, and executing interception prompt operation according to the interception information.
Specifically, after intercepting the target query request, the interception information such as the request time, the request path, the request parameter, the unique identifier of the precompiled query statement, the query parameter, the number of returned records, the occurrence times, the operation user, the operation time and the like can be recorded, and the interception result, the interception information and the like are sent to the staff to be processed by the staff.
Step 390, query the database according to the target query request.
The embodiment of the invention provides a database query method, which is characterized in that target identification information in request information of a target query request is obtained, the target identification information and candidate identification information are matched, target memory information is determined from candidate memory occupation information according to the candidate identification information matched with the target identification information, further estimated memory occupation information can be determined according to the target memory information and the number of data objects, when the memory usage space represented by the estimated memory occupation information is larger than a preset interception condition, the target query request is intercepted, request information which possibly causes fullgc problems is intercepted according to the estimated memory occupation information of the target query request, fullgc problems of a database are avoided, and the smoothness and stability of application running are improved. In addition, by generating interception information based on the identification information, the attribute information and the execution information of the target query request and executing interception prompt operation according to the interception information, the problem of fullgc is conveniently and accurately positioned, the problem of fullgc is avoided to be positioned by adopting a manual query mode, the efficiency of discovering and solving the problem of fullgc is improved, and the robustness of an application program is ensured.
Fig. 4 is a schematic structural diagram of a database query device according to an embodiment of the present invention, including:
The information obtaining module 410 is configured to obtain a target query request, and obtain target memory occupancy information according to target identification information corresponding to the target query request, where the target memory occupancy information characterizes a mapping relationship between a single data object and occupied memory in a query result;
The memory occupation estimating module 420 is configured to obtain the number of data objects corresponding to the query result of the target query request, and determine estimated memory occupation information of the target query request according to the number of data objects and the target memory occupation information;
The request interception module 430 is configured to intercept the target query request if the estimated memory usage information meets a preset interception condition.
Optionally, the preset interception condition is determined based on an available memory and an available memory usage threshold, wherein the available memory usage threshold is determined based on the number of services and the concurrency of the services.
The device of the embodiment of the invention can execute a database query method, which comprises the following steps: acquiring a target query request, and acquiring target memory occupation information according to target identification information corresponding to the target query request, wherein the target memory occupation information characterizes the mapping relation between a single data object and occupied memory in a query result; acquiring the number of data objects corresponding to the query result of the target query request, and determining estimated memory occupation information of the target query request according to the number of data objects and the target memory occupation information; and if the estimated memory occupation information meets the preset interception condition, intercepting the target query request. Specifically, the target memory occupation information can be obtained by determining the corresponding target identification information according to the target query request, and further the estimated memory occupation information of the target query request can be determined, in this way, the estimated memory occupation information of the search request can be accurately determined, and further the search request which possibly causes fullgc is intercepted.
Optionally, the device of the embodiment of the present invention further includes an information management module, including:
An acquisition unit configured to periodically acquire request information and result information of a history inquiry request;
A target byte number determining unit, configured to determine a target byte number corresponding to a historical query result according to the request information;
The memory occupied byte number determining unit is used for determining the memory occupied byte number corresponding to a single data object according to the historical memory occupied information corresponding to the result information and the number of the historical data objects;
An association unit for determining candidate memory occupation information according to the memory occupation byte number and the target byte number corresponding to the single data object, and associating and storing the candidate memory occupation information with candidate identification information in the request information
Optionally, the target byte number determining unit includes:
A calculation task obtaining subunit, configured to obtain a calculation task in a task queue, where the calculation task is configured to calculate a target byte number corresponding to a history query result of an incremental query request or a change query request;
a determining subunit, configured to determine a target byte number corresponding to the historical query result according to the request information corresponding to the incremental query request or the change query request
Optionally, the target byte number determining unit further includes:
the matching subunit is used for matching the identification information of the historical query request with the candidate identification information;
And the generation subunit is used for generating a calculation task according to the request information of the history query request which is not matched with the candidate identification information, and adding the calculation task into a task queue.
Optionally, the target byte number determining unit further includes:
A judging subunit, configured to judge, for a history query request that matches the candidate identifier information, whether a query statement in request information of the history query request is changed;
and the change subunit is used for generating a calculation task according to the request information of the historical query request and adding the calculation task into a task queue if the query statement is changed.
Optionally, the target byte number determining unit includes a matching subunit, configured to obtain a history query statement in the request information, and determine, according to the number of bytes corresponding to a field type of a field in the history query statement, a target byte number corresponding to the history query result
Optionally, the memory occupied byte number determining unit further includes:
the first determining unit is used for determining historical memory occupation information and the number of historical data objects according to result information corresponding to the historical query requests for each historical query request corresponding to the computing task.
And the second determining unit is used for determining the average memory occupied byte number of the single data object according to the historical memory occupied information and the number of the historical data objects, and the average memory occupied byte number is used as the memory occupied byte number of the single data object corresponding to the calculation task.
Optionally, the association unit further includes:
the first calculation unit is used for calculating current memory occupation information according to the memory occupation byte number and the target byte number corresponding to the calculation task;
and the second calculation unit is used for determining candidate memory occupation information according to the current memory occupation information and the historical memory occupation information corresponding to the calculation task.
Optionally, the information obtaining module 410 includes:
The target identification information determining unit is used for acquiring target identification information corresponding to the target query request;
The corresponding unit is used for determining target memory occupation information corresponding to the target identification information according to the target identification information and the corresponding relation between the preset candidate identification information and the candidate memory occupation information.
Optionally, the memory footprint estimation module 420 includes:
The execution unit is used for executing the object number query statement corresponding to the target query request to obtain the number of data objects corresponding to the query result of the target query request;
And the mapping unit is used for determining the mapping relation between a single data object and the occupied memory according to the target memory occupied information, and determining the estimated memory occupied information of the target query request by combining the mapping relation and the number of the data objects.
Optionally, the request interception module 430 further includes: the interception information unit is used for generating interception information according to the target query request, the identification information, the attribute information and the execution information of the target query request, and executing interception prompt operation according to the interception information.
The database query device provided by the embodiment of the invention can execute the database query method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 5 shows a schematic diagram of the structure of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 5, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as database query methods.
In some embodiments, the database query method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more of the steps of the database query method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the database query method in any other suitable manner (e.g., by means of firmware).
Various implementations of the systems and techniques described here above can be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Embodiments of the present application also provide a computer program product comprising a computer program which, when executed by a processor, implements a database query method as provided by any of the embodiments of the present application.
Computer program product in the implementation, the computer program code for carrying out operations of the present invention may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (16)
1. A database query method, comprising:
Acquiring a target query request, and acquiring target memory occupation information according to target identification information corresponding to the target query request, wherein the target memory occupation information represents the mapping relation between a single data object and occupied memory in a query result;
acquiring the number of data objects corresponding to the query result of the target query request, and determining estimated memory occupation information of the target query request according to the number of data objects and the target memory occupation information;
And if the estimated memory occupation information meets a preset interception condition, intercepting the target query request.
2. The method as recited in claim 1, further comprising:
periodically acquiring request information and result information of a historical query request;
Determining the number of target bytes corresponding to the historical query result according to the request information;
determining the number of memory occupied bytes corresponding to a single data object according to the historical memory occupied information corresponding to the result information and the number of historical data objects;
And determining candidate memory occupation information according to the memory occupation byte number corresponding to the single data object and the target byte number, and storing the candidate memory occupation information and the candidate identification information in the request information in a correlated manner.
3. The method according to claim 2, wherein determining the target number of bytes corresponding to the historical query result according to the request information includes:
Acquiring a calculation task in a task queue, wherein the calculation task is used for calculating a target byte number corresponding to a historical query result of an incremental query request or a change query request;
and determining the target byte number corresponding to the historical query result according to the request information corresponding to the increment query request or the change query request.
4. The method of claim 3, further comprising, prior to the acquiring the computing task in the task queue:
Matching the identification information of the historical query request with candidate identification information;
And generating a calculation task according to the request information of the historical query request, and adding the calculation task into a task queue for the historical query request which is not matched with the candidate identification information.
5. The method as recited in claim 4, further comprising:
judging whether query sentences in request information of the historical query requests are changed or not according to the historical query requests matched with the candidate identification information;
and if the query statement is changed, generating a calculation task according to the request information of the historical query request, and adding the calculation task into a task queue.
6. The method according to claim 2, wherein determining the target number of bytes corresponding to the historical query result according to the request information includes:
And acquiring a historical query statement in the request information, and determining a target byte number corresponding to the historical query result according to the byte number corresponding to the field type of the field in the historical query statement.
7. The method according to claim 2 or 6, wherein determining the number of memory occupied bytes corresponding to a single data object according to the historical memory occupied information corresponding to the result information and the number of historical data objects includes:
for each historical query request corresponding to the computing task, determining historical memory occupation information and the number of historical data objects according to result information corresponding to the historical query request;
and determining the average memory occupied byte number of the single data object according to the historical memory occupied information and the number of the historical data objects, and taking the average memory occupied byte number of the single data object as the memory occupied byte number of the single data object corresponding to the calculation task.
8. The method of claim 7, wherein determining candidate memory usage information based on the number of memory usage bytes and the number of target bytes corresponding to the single data object comprises:
calculating current memory occupation information according to the memory occupation byte number and the target byte number corresponding to the calculation task;
and determining candidate memory occupancy information according to the current memory occupancy information and the historical memory occupancy information corresponding to the computing task.
9. The method of claim 1, wherein the obtaining the target memory occupancy information according to the target identification information corresponding to the target query request comprises:
Acquiring target identification information corresponding to the target query request;
and determining target memory occupancy information corresponding to the target identification information according to the target identification information and the corresponding relation between the preset candidate identification information and the candidate memory occupancy information.
10. The method according to claim 1, wherein the obtaining the number of data objects corresponding to the query result of the target query request, and determining the estimated memory usage information of the target query request according to the number of data objects and the target memory usage information, includes:
Executing the object number query statement corresponding to the target query request to obtain the number of data objects corresponding to the query result of the target query request;
and determining the mapping relation between a single data object and occupied memory according to the target memory occupancy information, and determining the estimated memory occupancy information of the target query request by combining the mapping relation and the number of the data objects.
11. The method of claim 1, wherein the preset intercept condition is determined based on available memory and an available memory usage threshold, wherein the available memory usage threshold is determined based on a traffic volume and traffic concurrency.
12. The method of claim 1, further comprising, after intercepting the target query request:
generating interception information according to the target query request, the identification information, the attribute information and the execution information of the target query request, and executing interception prompt operation according to the interception information.
13. A database query apparatus, comprising:
the information acquisition module is used for acquiring a target query request and acquiring target memory occupation information according to target identification information corresponding to the target query request, wherein the target memory occupation information represents the mapping relation between a single data object and occupied memory in a query result;
the memory occupation estimating module is used for acquiring the number of data objects corresponding to the query result of the target query request and determining estimated memory occupation information of the target query request according to the number of data objects and the target memory occupation information;
And the request interception module is used for intercepting the target query request if the estimated memory occupation information meets a preset interception condition.
14. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable by the processor, wherein the processor implements the database query method of any of claims 1-12 when the computer program is executed by the processor.
15. A computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a database querying method as claimed in any one of claims 1 to 12.
16. A computer program product comprising a computer program which, when executed by a processor, implements the database query method of any of claims 1-12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410235685.1A CN118012902A (en) | 2024-03-01 | 2024-03-01 | Database query method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410235685.1A CN118012902A (en) | 2024-03-01 | 2024-03-01 | Database query method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118012902A true CN118012902A (en) | 2024-05-10 |
Family
ID=90956081
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410235685.1A Pending CN118012902A (en) | 2024-03-01 | 2024-03-01 | Database query method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118012902A (en) |
-
2024
- 2024-03-01 CN CN202410235685.1A patent/CN118012902A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11645179B2 (en) | Method and apparatus of monitoring interface performance of distributed application, device and storage medium | |
| CN112818013B (en) | Time sequence database query optimization method, device, equipment and storage medium | |
| US20220358178A1 (en) | Data query method, electronic device, and storage medium | |
| CN116225769B (en) | Method, device, equipment and medium for determining root cause of system fault | |
| CN116578646A (en) | Time sequence data synchronization method, device, equipment and storage medium | |
| CN118012902A (en) | Database query method and device, electronic equipment and storage medium | |
| CN117271840B (en) | Data query method and device of graph database and electronic equipment | |
| CN117539840B (en) | Log acquisition method, device, equipment and medium | |
| CN114416414B (en) | Fault information positioning method, device, equipment and storage medium | |
| US20230130399A1 (en) | Wakeup Indicator Monitoring Method, Apparatus and Electronic Device | |
| US20220107949A1 (en) | Method of optimizing search system | |
| CN117215589A (en) | Cloud primary state evaluation method, device, equipment and storage medium | |
| CN116302371A (en) | Transaction visibility judging method, device, equipment and storage medium | |
| CN115203246A (en) | Linked list query method and device, electronic equipment and storage medium | |
| CN114610490A (en) | Method, device, equipment and storage medium for preventing memory overflow during data synchronization | |
| CN117271518A (en) | Method, device, equipment and storage medium for generating multi-transaction fact table | |
| CN116150031A (en) | Program performance test early warning method, device, equipment and storage medium | |
| CN117171165A (en) | Database falling method and device, electronic equipment and storage medium | |
| CN115983222A (en) | EasyExcel-based file data reading method, device, equipment and medium | |
| CN116644111A (en) | Index determination method and device based on Internet of things data and electronic equipment | |
| CN117668363A (en) | Recommendation method, device, equipment and medium | |
| CN116522158A (en) | Method and device for predicting cold and hot states of data, electronic equipment and storage medium | |
| CN116955394A (en) | Data query method, device, equipment and medium | |
| CN117827840A (en) | Index creation method and device, electronic equipment and storage medium | |
| CN117608944A (en) | Method and device for calculating weight migration volume ratio, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |