CN108170860A - Data query method, apparatus, electronic equipment and computer readable storage medium - Google Patents
Data query method, apparatus, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN108170860A CN108170860A CN201810060918.3A CN201810060918A CN108170860A CN 108170860 A CN108170860 A CN 108170860A CN 201810060918 A CN201810060918 A CN 201810060918A CN 108170860 A CN108170860 A CN 108170860A
- Authority
- CN
- China
- Prior art keywords
- query
- data
- data query
- request
- complexity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012795 verification Methods 0.000 claims description 39
- 230000008569 process Effects 0.000 claims description 18
- 230000000903 blocking effect Effects 0.000 abstract 1
- 230000007423 decrease Effects 0.000 abstract 1
- 238000010586 diagram Methods 0.000 description 15
- 238000005192 partition Methods 0.000 description 8
- 230000001360 synchronised effect Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 238000004590 computer program Methods 0.000 description 6
- 230000003993 interaction Effects 0.000 description 6
- 230000002776 aggregation Effects 0.000 description 4
- 238000004220 aggregation Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000013475 authorization Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the present disclosure discloses a kind of data query method, apparatus, electronic equipment and computer readable storage medium.The method includes：Receive data inquiry request；Determine the complexity of the corresponding inquiry operation of the data inquiry request；When the complexity of the inquiry operation is no more than predetermined threshold value, the data inquiry request is submitted into query engine.Disclosure embodiment, it can be reviewed before data inquiry request reaches query engine, when the concurrency that query engine blocking will likely be caused or query engine performance is caused to decline to a great extent operates, the data inquiry request is prevented to be submitted to query engine, greatly improves the stability and concurrency of query engine.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data query method and apparatus, an electronic device, and a computer-readable storage medium.
Background
In the query scenario of big data, the back-end service is more MPP (maximum power point) distributed service, while the big data emphasizes the online analysis Processing OLAP of the re-analyzed scenario, and the business scenario is more online Transaction Processing (OLTP).
The complexity of the analyzed query is generally high, and when resources (user SQL or other query APIs) needed by the (engine) are not symmetrical, it is found that the engine may be blocked by part of the query scenarios, or the concurrence may be seriously reduced.
Disclosure of Invention
The embodiment of the disclosure provides a data query method and device, electronic equipment and a computer-readable storage medium.
In a first aspect, a data query method is provided in the embodiments of the present disclosure.
Specifically, the data query method includes:
receiving a data query request;
determining the complexity of the query operation corresponding to the data query request;
and when the complexity of the query operation does not exceed a preset threshold value, submitting the data query request to a query engine.
With reference to the first aspect, in a first implementation manner of the first aspect, before determining the complexity of the query operation corresponding to the data query request, the method further includes:
and verifying the authority of the requester of the data query request.
With reference to the first implementation manner of the first aspect, verifying the authority of the requester of the data query request includes at least one of:
verifying the validity of the requester according to the identity verification information carried in the data query request;
verifying and submitting the query resource surplus of the requester according to the identity verification information carried in the data query request;
and verifying and submitting the operation authority of the requester on the query engine according to the identity verification information carried by the data query request.
With reference to the first aspect, in a second implementation manner of the first aspect, the determining complexity of a query operation corresponding to the data query request includes:
determining a query step of the query operation according to the data query request;
determining the operation score of the query step according to a preset rule;
determining the complexity according to the operation score.
With reference to the first aspect, the first implementation manner of the first aspect, or the second implementation manner of the first aspect, in a third implementation manner of the first aspect, the present disclosure further includes:
and when the complexity of the query operation exceeds the preset threshold value, returning query failure information to a requester submitting the data query request.
With reference to the first aspect, the first implementation manner of the first aspect, or the second implementation manner of the first aspect, in a fourth implementation manner of the first aspect, after submitting the data query request to a query engine, the present disclosure further includes:
receiving a query result returned by the query engine;
determining whether the query result satisfies a resource quota of a requestor submitting the data query request;
and when the query result does not meet the resource quota, returning query failure information to the requester.
With reference to the first aspect, the first implementation manner of the first aspect, or the second implementation manner of the first aspect, in a fifth implementation manner of the first aspect, the present disclosure further includes:
in the query process, acquiring the execution condition of the query engine on the query operation;
and determining whether the execution condition meets a preset condition.
With reference to the fifth implementation manner of the first aspect, in a sixth implementation manner of the first aspect, the present disclosure further includes:
and when the execution condition does not meet a preset condition, sending a request for terminating the query operation to the query engine.
In a second aspect, an embodiment of the present disclosure provides a data query apparatus, including:
a first receiving module configured to receive a data query request;
a first determining module configured to determine a complexity of a query operation corresponding to the data query request;
a submitting module configured to submit the data query request to a query engine when the complexity of the query operation does not exceed a preset threshold.
With reference to the second aspect, in a first implementation manner of the second aspect, the present disclosure further includes, before the first determining module:
a verification module configured to verify a permission of a requestor of the data query request.
With reference to the second aspect, in a first implementation manner, the verification module includes at least one of:
the first verification sub-module is configured to verify the validity of the requester according to the identity verification information carried in the data query request;
the second verification submodule is configured to verify the query resource surplus of the requester according to the identity verification information carried in the data query request;
and the third verification sub-module is configured to verify the operation authority of the requester for the query engine according to the identity verification information carried by the data query request.
With reference to the second aspect, in a second implementation manner of the second aspect, the first determining module includes:
a first determining sub-module configured to determine a query step of the query operation according to the data query request;
a second determining submodule configured to determine an operation score of the querying step according to a preset rule;
a third determination submodule configured to determine the complexity from the operation score.
With reference to the second aspect, the first implementation manner of the second aspect, or the second implementation manner of the second aspect, in a third implementation manner of the second aspect, the present disclosure further includes:
the first returning module is configured to return query failure information to a requester submitting the data query request when the complexity of the query operation exceeds the preset threshold.
With reference to the second aspect, the first implementation manner of the second aspect, or the second implementation manner of the second aspect, in a fourth implementation manner of the second aspect, after the submitting module, the method further includes:
the second receiving module is configured to receive the query result returned by the query engine;
a second determination module configured to determine whether the query result satisfies a resource quota of a requestor submitting the data query request;
and the second returning module is configured to return query failure information to the requester when the query result does not meet the resource quota.
With reference to the second aspect, the first implementation manner of the second aspect, or the second implementation manner of the second aspect, in a fifth implementation manner of the second aspect, the present disclosure further includes:
the acquisition module is configured to acquire the execution condition of the query engine on the query operation in a query process;
a third determination module configured to determine whether the execution status satisfies a preset condition.
With reference to the fifth implementation manner of the second aspect, in a sixth implementation manner of the second aspect, the present disclosure further includes:
a sending module configured to send a request for terminating the query operation to the query engine when the execution status does not satisfy a preset condition.
The functions can be realized by hardware, and the functions can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above-described functions.
In one possible design, the data query device includes a memory and a processor, the memory is used for storing one or more computer instructions for supporting the data query device to execute the data query method in the first aspect, and the processor is configured to execute the computer instructions stored in the memory. The data query apparatus may further include a communication interface for the data query apparatus to communicate with other devices or a communication network.
In a third aspect, an embodiment of the present disclosure provides an electronic device, including a memory and a processor; wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of the first aspect.
In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions for a data query device, which contains computer instructions for executing the data query method in the first aspect.
The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:
the data query access device is arranged between the data query request party and the query engine party and used for calculating the complexity of query operation corresponding to the received data query request and determining whether to provide the data query request to the query engine according to the complexity. Through the embodiment of the disclosure, the data query request can be audited before reaching the query engine, and the data query request is prevented from being submitted to the query engine when the concurrent operation possibly causes the query engine to be blocked or causes the performance of the query engine to be greatly reduced, so that the stability and the concurrency of the query engine are greatly improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
Other features, objects, and advantages of the present disclosure will become more apparent from the following detailed description of non-limiting embodiments when taken in conjunction with the accompanying drawings. In the drawings:
FIG. 1 illustrates a flow diagram of a data query method according to an embodiment of the present disclosure;
FIG. 2 shows a flow chart of step S102 according to the embodiment shown in FIG. 1;
FIG. 3 illustrates a flow diagram for detecting query results returned by a query engine in accordance with an embodiment of the present disclosure;
FIG. 4 illustrates a flow diagram for monitoring query engine execution during a query process according to an embodiment of the present disclosure;
FIG. 5 illustrates a block diagram of a data query, according to an embodiment of the present disclosure;
FIG. 6 illustrates a block diagram of the first determination module 502 according to the embodiment shown in FIG. 5;
FIG. 7 is a block diagram illustrating a portion of the structure of the query result detection portion returned by the query engine according to one embodiment of the present disclosure;
FIG. 8 is a block diagram of a portion for monitoring query engine execution during a query process according to an embodiment of the present disclosure;
fig. 9 is a schematic structural diagram of an electronic device suitable for implementing a data query method according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. Also, for the sake of clarity, parts not relevant to the description of the exemplary embodiments are omitted in the drawings.
In the present disclosure, it is to be understood that terms such as "including" or "having," etc., are intended to indicate the presence of the disclosed features, numbers, steps, behaviors, components, parts, or combinations thereof, and are not intended to preclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or combinations thereof may be present or added.
It should be further noted that the embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 shows a flow diagram of a data query method according to an embodiment of the present disclosure. As shown in fig. 1, the data query method includes the following steps S101 to S103:
in step S101, a data query request is received;
in step S102, determining a complexity of a query operation corresponding to the data query request;
in step S103, when the complexity of the query operation does not exceed a preset threshold, the data query request is submitted to a query engine.
In the prior art, a user or a service initiates a data query request, and then the data query request is directly submitted to a query engine in a mode of assembling SQL (structured query language) or API (application program interface) calling; after receiving the data query request, the query engine feeds back the result to the user or the service after a series of queries. However, this way of directly connecting the query engines is inconvenient for authority verification; furthermore, unlimited submission of data query requests to the query engine can result in significant compromises in the stability and service concurrency of the query engine.
In this embodiment, a set of query admission devices is provided between the requester of the data query request and the query engine, so that the data query request is checked before being submitted to the query engine, thereby avoiding submitting the data query request causing the query engine to be blocked or having the concurrency affected to the query engine, improving the security of the query engine, and ensuring higher stability and concurrency of the query engine.
The complexity of the data query request can be analyzed through the query steps of the data query request, and then the operation difficulty involved in each query step, occupied engine resources and the like are scored, and finally the complexity of the data query request is obtained. For example, for the SQL query statement of the database, whether operations such as aggregation, scanning, Union, Join, and the like are included may be analyzed, and the complexity of the SQL query statement is determined by determining scores corresponding to different operations according to the analysis result; the scores corresponding to different operations can be comprehensively scored according to previous experiences or query engines on the running time, occupied resources and the like of the operations, and are preset.
In this embodiment, when the complexity of the query operation does not exceed the preset threshold, the data query request is provided to the query engine, and the query engine executes the corresponding query operation. The preset threshold may be determined according to the performance of the query engine, for example, the complexity of the query operation that occupies 85% of the performance resource of the query engine may be set as the preset threshold, so as to avoid the performance of the query engine from being seriously degraded due to the excessively complex query operation.
In an optional implementation manner of this embodiment, before the step S102, that is, before the step of determining the complexity of the query operation corresponding to the data query request, the method further includes the following steps:
and verifying the authority of the requester of the data query request.
In this alternative implementation, the admission device also verifies the authority of the requestor of the data query request. The query request passing through the access device can be interacted by a remote interface calling mode and the like, which is not the traditional interaction mode of JDBC and ODBC, so that the identity authentication information can be carried when the interaction is carried, namely, the data query request is submitted. The access device can verify the authority of the requesting party according to the identity verification information carried in the data query request, and different requesting parties can have different authorities. For example, an authorized requestor may invoke an interface of the admission apparatus to submit a data query request to the query engine, while an unauthorized requestor may not submit a data query request to the query engine. The data query request may carry a token, a user, a password, or other identification information; the service address of the access device can be carried; a waiting mode (synchronous or asynchronous) for the query result, other additional information, etc.
In an optional implementation manner of this embodiment, the step of verifying the authority of the requester of the data query request includes at least one of the following steps:
verifying the validity of the requester according to the identity verification information carried in the data query request;
verifying and submitting the query resource surplus of the requester according to the identity verification information carried in the data query request;
and verifying and submitting the operation authority of the requester on the query engine according to the identity verification information carried by the data query request.
In this optional implementation manner, the authentication information of the data query request may include a user name, a password, and the like of the requestor, and the admission apparatus may determine whether the requestor is a valid user according to the information of the user name/the password, and the like. The admission device can also determine the remaining quantity of the query resources of the requester according to the user name of the requester. The query resource remaining amount is used to identify whether the query resource that the requester can obtain has a remaining amount, such as the daily query times of the user. The admission device may also determine, according to the identity information such as the user name of the requester, the operation authority that the requester can operate the query engine, for example, by analyzing SQL statements or submitted data query requests, the specific library table and corresponding fields to be accessed are involved, and check whether the authorization information of the requester has the authority to check the corresponding database table and corresponding fields. It is understood that, at the time of verification, only one of the three ways may be verified, or multiple ways may be verified simultaneously. The validity of the requester and the verification mode of the remaining amount of the query resources do not need to be intervened by a query engine, so that the verification can be carried out before the complexity of the query operation is calculated, and if the verification fails, failure information can be directly returned to the requester; if the verification is passed, the complexity of the query operation is calculated; the mode of verifying the user operation authority is determined by the query engine, so that the mode can be determined after the complexity of the query operation is calculated, the interaction with the query engine can be reduced, and the time and the resources can be saved.
In an optional implementation manner of this embodiment, as shown in fig. 2, the step S102, that is, the step of determining the complexity of the query operation corresponding to the data query request, further includes the following steps S201 to S203:
in step S201, determining a query step of the query operation according to the data query request;
in step S202, determining an operation score of the querying step according to a preset rule;
in step S203, the complexity is determined according to the operation score.
In the optional implementation manner, query operation for one data query request may need to be completed through multiple query steps, and operations related to each step need to spend certain resources of a query engine, including query time, occupied memory, the number of data partitions that need to be scanned, the number of data to be scanned in each data partition, and the like, different operations may be scored in advance according to resources required by different operations, and then, when calculating complexity actually, the query steps are scored according to preset operation scores, and finally, the complexity of the data query request is determined through the scores of the query steps.
The following takes the data query request of the SQL statement as an example. For a data query request, a query plan is formed in a query engine, and the query plan is an analysis of a plurality of query steps of a query process, such as operations of aggregation, scanning, Union, Join and the like. An exemplary SQL data query request statement is shown in Table 1:
TABLE 1
The query plan formed may be analyzed as shown in Table 2:
TABLE 2
As can be seen from the above table, the SQL statements in the data query request collectively relate to a plurality of operation items as shown in the above table, and the complexity of the data query request can be finally obtained according to predefined operation item scores, data scanning partition numbers, data partition scanning numbers, and the like.
In an optional implementation manner of this embodiment, the data query method further includes the following steps:
and when the complexity of the query operation exceeds the preset threshold value, returning query failure information to a requester submitting the data query request.
In this optional implementation, if it is determined that the complexity of the query operation corresponding to the data query request exceeds the preset threshold by analyzing the data query request, which may affect the stability and concurrency of the query engine, the data query request of the requester may be directly rejected, and the query failure information is output.
In an optional implementation manner of this embodiment, as shown in fig. 3, after the step S103 of submitting the data query request to the query engine, the method further includes the following steps S301 to S303:
in step S301, receiving a query result returned by the query engine;
in step S302, determining whether the query result satisfies a resource quota of a requester submitting the data query request;
in step S303, when the query result does not satisfy the resource quota, query failure information is returned to the requester.
In this optional implementation manner, after the data query request conforming to the admission rule of the admission apparatus is submitted to the query engine, the query engine returns the query result to the admission apparatus after the query operation is completed. After receiving the query result, the admission device may further determine whether the query result satisfies the resource quota of the requester according to the identity information of the requester, for example, check whether the current query result satisfies admission restrictions such as a single maximum record number, a maximum daily return record number, and the like, and return the query result to the requester if the current query result satisfies the admission restrictions, or return the query result to the requester if the current query result satisfies the admission restrictions, otherwise, return the query failure information.
When the admission device returns the query result to the request party, the admission device can return the query result to the request party in a structured packaging mode or a streaming data mode, and can return the result in a synchronous return mode or an asynchronous callback mode. Specifically, asynchronous return or synchronous return is adopted, when a requester makes a data query request, a required return mode can be carried in the request, if the request is asynchronous return, an asynchronous callback address also needs to be carried in the data query request, and when an access device returns a query result to the requester, the asynchronous callback address is called to return. The synchronous mode can always block the request from inquiring to feedback results.
In an optional implementation manner of this embodiment, as shown in fig. 4, the method further includes the following steps 401 to S402:
in step S401, in the query process, acquiring an execution status of the query engine for the query operation;
in step S402, it is determined whether the execution condition satisfies a preset condition.
In this optional implementation manner, the admission apparatus may further obtain, after submitting the data query request to the query engine, an execution status of the query engine for the data query request in an operation process of the query engine, where the execution status includes, but is not limited to, a maximum query duration, a data amount scanned by the query, a memory occupied by the query, and the like. The admission device can periodically check the execution condition of the query engine for the data query request and determine that the resources used by the query engine in the process of executing the query operation exceed the expectation. In this way, the stability and concurrency of the query engine can be monitored during the query process.
In an optional implementation manner of this embodiment, the method further includes the following steps:
and when the execution condition does not meet a preset condition, sending a request for terminating the query operation to the query engine.
In this alternative implementation, during the query process, the admission apparatus periodically detects the execution status of the query engine, and if it is determined that the execution status does not meet the expectation, may terminate the current query operation and return the information of the query failure to the requesting party. In this way, factors that may cause the stability and concurrency of the query engine to be reduced can be timely discovered, and the current query operation is terminated so as not to cause serious consequences.
The following are embodiments of the disclosed apparatus that may be used to perform embodiments of the disclosed methods.
Fig. 5 shows a block diagram of a data query apparatus according to an embodiment of the present disclosure, which may be implemented as part of or all of an electronic device by software, hardware, or a combination of the two. As shown in fig. 5, the data query apparatus includes a first receiving module 501, a first determining module 502, and a submitting module 503:
a first receiving module 501 configured to receive a data query request;
a first determining module 502 configured to determine a complexity of a query operation corresponding to the data query request;
a submitting module 503 configured to submit the data query request to a query engine when the complexity of the query operation does not exceed a preset threshold.
In the prior art, a user or a service initiates a data query request, and then the data query request is directly submitted to a query engine in a mode of assembling SQL (structured query language) or API (application program interface) calling; after receiving the data query request, the query engine feeds back the result to the user or the service after a series of queries. However, this way of directly connecting the query engines is inconvenient for authority verification; furthermore, unlimited submission of data query requests to the query engine can result in significant compromises in the stability and service concurrency of the query engine.
In this embodiment, a set of query admission devices is provided between the requester of the data query request and the query engine, so that the data query request is checked before being submitted to the query engine, thereby avoiding submitting the data query request causing the query engine to be blocked or having the concurrency affected to the query engine, improving the security of the query engine, and ensuring higher stability and concurrency of the query engine. The data querying means may be the admission means or may be part of the admission means.
The complexity of the data query request can be analyzed through the query steps of the data query request, and then the operation difficulty involved in each query step, occupied engine resources and the like are scored, and finally the complexity of the data query request is obtained. For example, for the SQL query statement of the database, whether operations such as aggregation, scanning, Union, Join, and the like are included may be analyzed, and the complexity of the SQL query statement is determined by determining scores corresponding to different operations according to the analysis result; the scores corresponding to different operations can be comprehensively scored according to previous experiences or query engines on the running time, occupied resources and the like of the operations, and are preset.
In this embodiment, when the complexity of the query operation does not exceed the preset threshold, the data query request is provided to the query engine, and the query engine executes the corresponding query operation. The preset threshold may be determined according to the performance of the query engine, for example, the complexity of the query operation that occupies 85% of the performance resource of the query engine may be set as the preset threshold, so as to avoid the performance of the query engine from being seriously degraded due to the excessively complex query operation.
In an optional implementation manner of this embodiment, before the first determining module 502, the method further includes:
a verification module configured to verify a permission of a requestor of the data query request.
In this alternative implementation, the admission device also verifies the authority of the requestor of the data query request. The query request passing through the access device can be interacted by a remote interface calling mode and the like, which is not the traditional interaction mode of JDBC and ODBC, so that the identity authentication information can be carried when the interaction is carried, namely, the data query request is submitted. The access device can verify the authority of the requesting party according to the identity verification information carried in the data query request, and different requesting parties can have different authorities. For example, an authorized requestor may invoke an interface of the admission apparatus to submit a data query request to the query engine, while an unauthorized requestor may not submit a data query request to the query engine. The data query request may carry a token, a user, a password, or other identification information; the service address of the access device can be carried; a waiting mode (synchronous or asynchronous) for the query result, other additional information, etc.
In an optional implementation manner of this embodiment, the verification module includes at least one of:
the first verification sub-module is configured to verify the validity of the requester according to the identity verification information carried in the data query request;
the second verification submodule is configured to verify the query resource surplus of the requester according to the identity verification information carried in the data query request;
and the third verification sub-module is configured to verify the operation authority of the requester for the query engine according to the identity verification information carried by the data query request.
In this optional implementation manner, the authentication information of the data query request may include a user name, a password, and the like of the requestor, and the admission apparatus may determine whether the requestor is a valid user according to the information of the user name/the password, and the like. The admission device can also determine the remaining quantity of the query resources of the requester according to the user name of the requester. The query resource remaining amount is used to identify whether the query resource that the requester can obtain has a remaining amount, such as the daily query times of the user. The admission device may also determine, according to the identity information such as the user name of the requester, the operation authority that the requester can operate the query engine, for example, by analyzing SQL statements or submitted data query requests, the specific library table and corresponding fields to be accessed are involved, and check whether the authorization information of the requester has the authority to check the corresponding database table and corresponding fields. It is understood that, at the time of verification, only one of the three ways may be verified, or multiple ways may be verified simultaneously. The validity of the requester and the verification mode of the remaining amount of the query resources do not need to be intervened by a query engine, so that the verification can be carried out before the complexity of the query operation is calculated, and if the verification fails, failure information can be directly returned to the requester; if the verification is passed, the complexity of the query operation is calculated; the mode of verifying the user operation authority is determined by the query engine, so that the mode can be determined after the complexity of the query operation is calculated, the interaction with the query engine can be reduced, and the time and the resources can be saved.
In an optional implementation manner of this embodiment, as shown in fig. 6, the first determining module 502 includes:
a first determining sub-module 601 configured to determine a query step of the query operation according to the data query request;
a second determining sub-module 602 configured to determine an operation score of the querying step according to a preset rule;
a third determining submodule 603 configured to determine the complexity from the operation score.
In the optional implementation manner, query operation for one data query request may need to be completed through multiple query steps, and operations related to each step need to spend certain resources of a query engine, including query time, occupied memory, the number of data partitions that need to be scanned, the number of data to be scanned in each data partition, and the like, different operations may be scored in advance according to resources required by different operations, and then, when calculating complexity actually, the query steps are scored according to preset operation scores, and finally, the complexity of the data query request is determined through the scores of the query steps.
The following takes the data query request of the SQL statement as an example. For a data query request, a query plan is formed in a query engine, and the query plan is an analysis of a plurality of query steps of a query process, such as operations of aggregation, scanning, Union, Join and the like. An exemplary SQL data query request statement is shown in Table 1:
TABLE 1
The query plan formed may be analyzed as shown in Table 2:
TABLE 2
As can be seen from the above table, the SQL statements in the data query request collectively relate to a plurality of operation items as shown in the above table, and the complexity of the data query request can be finally obtained according to predefined operation item scores, data scanning partition numbers, data partition scanning numbers, and the like.
In an optional implementation manner of this embodiment, the data query apparatus further includes:
the first returning module is configured to return query failure information to a requester submitting the data query request when the complexity of the query operation exceeds the preset threshold.
In this optional implementation, if it is determined that the complexity of the query operation corresponding to the data query request exceeds the preset threshold by analyzing the data query request, which may affect the stability and concurrency of the query engine, the data query request of the requester may be directly rejected, and the query failure information is output.
In an optional implementation manner of this embodiment, as shown in fig. 7, after the submitting module 503, the method further includes:
a second receiving module 701 configured to receive a query result returned by the query engine;
a second determination module 702 configured to determine whether the query result satisfies a resource quota of a requestor submitting the data query request;
a second returning module 703 configured to return query failure information to the requestor when the query result does not satisfy the resource quota.
In this optional implementation manner, after the data query request conforming to the admission rule of the admission apparatus is submitted to the query engine, the query engine returns the query result to the admission apparatus after the query operation is completed. After receiving the query result, the admission device may further determine whether the query result satisfies the resource quota of the requester according to the identity information of the requester, for example, check whether the current query result satisfies admission restrictions such as a single maximum record number, a maximum daily return record number, and the like, and return the query result to the requester if the current query result satisfies the admission restrictions, or return the query result to the requester if the current query result satisfies the admission restrictions, otherwise, return the query failure information.
When the admission device returns the query result to the request party, the admission device can return the query result to the request party in a structured packaging mode or a streaming data mode, and can return the result in a synchronous return mode or an asynchronous callback mode. Specifically, asynchronous return or synchronous return is adopted, when a requester makes a data query request, a required return mode can be carried in the request, if the request is asynchronous return, an asynchronous callback address also needs to be carried in the data query request, and when an access device returns a query result to the requester, the asynchronous callback address is called to return. The synchronous mode can always block the request from inquiring to feedback results.
In an optional implementation manner of this embodiment, as shown in fig. 8, the data query apparatus further includes:
an obtaining module 801 configured to obtain, in a query process, an execution status of the query engine for the query operation;
a third determining module 802 configured to determine whether the execution status satisfies a preset condition.
In this optional implementation manner, the admission apparatus may further obtain, after submitting the data query request to the query engine, an execution status of the query engine for the data query request in an operation process of the query engine, where the execution status includes, but is not limited to, a maximum query duration, a data amount scanned by the query, a memory occupied by the query, and the like. The admission device can periodically check the execution condition of the query engine for the data query request and determine that the resources used by the query engine in the process of executing the query operation exceed the expectation. In this way, the stability and concurrency of the query engine can be monitored during the query process.
In an optional implementation manner of this embodiment, the data query apparatus further includes:
a sending module configured to send a request for terminating the query operation to the query engine when the execution status does not satisfy a preset condition.
In this alternative implementation, during the query process, the admission apparatus periodically detects the execution status of the query engine, and if it is determined that the execution status does not meet the expectation, may terminate the current query operation and return the information of the query failure to the requesting party. In this way, factors that may cause the stability and concurrency of the query engine to be reduced can be timely discovered, and the current query operation is terminated so as not to cause serious consequences.
Fig. 9 is a schematic structural diagram of an electronic device suitable for implementing a data query method according to an embodiment of the present disclosure.
As shown in fig. 9, the electronic apparatus 900 includes a Central Processing Unit (CPU)901 that can execute various processes in the embodiment shown in fig. 1 described above in accordance with a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. In the RAM903, various programs and data necessary for the operation of the electronic apparatus 900 are also stored. The CPU901, ROM902, and RAM903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.
The following components are connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary.
In particular, according to embodiments of the present disclosure, the method described above with reference to fig. 1 may be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a medium readable thereby, the computer program comprising program code for performing the method of fig. 1. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 909, and/or installed from the removable medium 911.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, a program segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units or modules described in the embodiments of the present disclosure may be implemented by software or hardware. The units or modules described may also be provided in a processor, and the names of the units or modules do not in some cases constitute a limitation of the units or modules themselves.
As another aspect, the present disclosure also provides a computer-readable storage medium, which may be the computer-readable storage medium included in the apparatus in the above-described embodiment; or it may be a separate computer readable storage medium not incorporated into the device. The computer readable storage medium stores one or more programs for use by one or more processors in performing the methods described in the present disclosure.
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is possible without departing from the inventive concept. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.
Claims (10)
1. A method for querying data, comprising:
receiving a data query request;
determining the complexity of the query operation corresponding to the data query request;
and when the complexity of the query operation does not exceed a preset threshold value, submitting the data query request to a query engine.
2. The data query method of claim 1, before determining the complexity of the query operation corresponding to the data query request, further comprising:
and verifying the authority of the requester of the data query request.
3. The data query method of claim 2, wherein verifying the authority of the requestor of the data query request comprises at least one of:
verifying the validity of the requester according to the identity verification information carried in the data query request;
verifying and submitting the query resource surplus of the requester according to the identity verification information carried in the data query request;
and verifying and submitting the operation authority of the requester on the query engine according to the identity verification information carried by the data query request.
4. The data query method of claim 1, wherein determining the complexity of the query operation corresponding to the data query request comprises:
determining a query step of the query operation according to the data query request;
determining the operation score of the query step according to a preset rule;
determining the complexity according to the operation score.
5. The data query method of claim 1, further comprising:
and when the complexity of the query operation exceeds the preset threshold value, returning query failure information to a requester submitting the data query request.
6. The data query method of claim 1, after submitting the data query request to a query engine, further comprising:
receiving a query result returned by the query engine;
determining whether the query result satisfies a resource quota of a requestor submitting the data query request;
and when the query result does not meet the resource quota, returning query failure information to the requester.
7. The data query method of claim 1, further comprising:
in the query process, acquiring the execution condition of the query engine on the query operation;
and determining whether the execution condition meets a preset condition.
8. A data query apparatus, comprising:
a first receiving module configured to receive a data query request;
a first determining module configured to determine a complexity of a query operation corresponding to the data query request;
a submitting module configured to submit the data query request to a query engine when the complexity of the query operation does not exceed a preset threshold.
9. An electronic device comprising a memory and a processor; wherein,
the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any of claims 1-7.
10. A computer-readable storage medium having stored thereon computer instructions, characterized in that the computer instructions, when executed by a processor, carry out the method steps of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810060918.3A CN108170860A (en) | 2018-01-22 | 2018-01-22 | Data query method, apparatus, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810060918.3A CN108170860A (en) | 2018-01-22 | 2018-01-22 | Data query method, apparatus, electronic equipment and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108170860A true CN108170860A (en) | 2018-06-15 |
Family
ID=62515267
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810060918.3A Pending CN108170860A (en) | 2018-01-22 | 2018-01-22 | Data query method, apparatus, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108170860A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109145015A (en) * | 2018-08-02 | 2019-01-04 | 北京百度网讯科技有限公司 | Data query methods, devices and systems based on structured query language |
| CN109960709A (en) * | 2019-04-18 | 2019-07-02 | 上海达梦数据库有限公司 | A kind of processing method of database-driven, device, equipment and storage medium |
| CN110321388A (en) * | 2019-02-26 | 2019-10-11 | 南威软件股份有限公司 | A kind of quicksort querying method and system based on Greenplum |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103530568A (en) * | 2012-07-02 | 2014-01-22 | 阿里巴巴集团控股有限公司 | Authority control method, device and system |
| CN104050297A (en) * | 2014-07-03 | 2014-09-17 | 中国工商银行股份有限公司 | Inquiry transaction distribution method and device |
| CN105786992A (en) * | 2016-02-17 | 2016-07-20 | 中国建设银行股份有限公司 | Data query method and device used for online transaction |
| CN106649503A (en) * | 2016-10-11 | 2017-05-10 | 北京集奥聚合科技有限公司 | Query method and system based on sql |
| US20170323118A1 (en) * | 2016-05-05 | 2017-11-09 | The Johns Hopkins University | Apparatus and Method for Private Information Retrieval |
-
2018
- 2018-01-22 CN CN201810060918.3A patent/CN108170860A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103530568A (en) * | 2012-07-02 | 2014-01-22 | 阿里巴巴集团控股有限公司 | Authority control method, device and system |
| CN104050297A (en) * | 2014-07-03 | 2014-09-17 | 中国工商银行股份有限公司 | Inquiry transaction distribution method and device |
| CN105786992A (en) * | 2016-02-17 | 2016-07-20 | 中国建设银行股份有限公司 | Data query method and device used for online transaction |
| US20170323118A1 (en) * | 2016-05-05 | 2017-11-09 | The Johns Hopkins University | Apparatus and Method for Private Information Retrieval |
| CN106649503A (en) * | 2016-10-11 | 2017-05-10 | 北京集奥聚合科技有限公司 | Query method and system based on sql |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109145015A (en) * | 2018-08-02 | 2019-01-04 | 北京百度网讯科技有限公司 | Data query methods, devices and systems based on structured query language |
| CN109145015B (en) * | 2018-08-02 | 2022-12-23 | 北京百度网讯科技有限公司 | Data query method, device and system based on structured query language |
| CN110321388A (en) * | 2019-02-26 | 2019-10-11 | 南威软件股份有限公司 | A kind of quicksort querying method and system based on Greenplum |
| CN109960709A (en) * | 2019-04-18 | 2019-07-02 | 上海达梦数据库有限公司 | A kind of processing method of database-driven, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20160307264A1 (en) | Method and device for processing service request | |
| US10621651B2 (en) | Automatic recharge system and method, and server | |
| US8051422B2 (en) | Resource assignment method for query partioning based on processing cost of each partition | |
| CN108170860A (en) | Data query method, apparatus, electronic equipment and computer readable storage medium | |
| US20130185645A1 (en) | Determining repeat website users via browser uniqueness tracking | |
| CN111371684B (en) | Routing processing method and device and double-activity data center system | |
| WO2020000716A1 (en) | Big data analysis system, server, data processing method, program and storage medium | |
| CN110659418A (en) | Content searching method and device, storage medium and computing equipment | |
| CN112817995A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN116975893A (en) | Access request processing method and device, storage medium and computer equipment | |
| CN114090113B (en) | Method, device, equipment and storage medium for dynamically loading data source processing plug-in | |
| US20240205020A1 (en) | Data Storage Method, Apparatus, and System, Storage Medium, and Program Product | |
| US9037609B1 (en) | Method for determining access of queries | |
| CN111831453B (en) | Information processing method, device, electronic equipment and medium | |
| CN110866757B (en) | Overdraft prevention method and device for electronic account | |
| CN116467085A (en) | Task processing method, system, electronic device and storage medium | |
| CN113761503B (en) | Interface call processing method and device | |
| CN110851483B (en) | Method, apparatus, electronic device, and medium for screening objects | |
| CN113240266A (en) | Risk management method and device | |
| CN113051008A (en) | Interface request processing method and device, computer equipment and storage medium | |
| US20050286415A1 (en) | System and method for lightweight deadlock detection | |
| CN113076331B (en) | Method, device, equipment, storage medium and program product for processing middle-stage data | |
| CN117041356B (en) | Index distribution method, index calculation method, apparatus, electronic device and system | |
| CN109101440B (en) | Method for processing trace data concurrent request based on JVM (Java virtual machine) cache | |
| CN115935420A (en) | Data processing method, device, server and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180615 |