CN112905592A - Data query method, system and server - Google Patents
Data query method, system and server Download PDFInfo
- Publication number
- CN112905592A CN112905592A CN202110171197.5A CN202110171197A CN112905592A CN 112905592 A CN112905592 A CN 112905592A CN 202110171197 A CN202110171197 A CN 202110171197A CN 112905592 A CN112905592 A CN 112905592A
- Authority
- CN
- China
- Prior art keywords
- query
- execution
- data
- authority
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 64
- 230000036541 health Effects 0.000 claims description 23
- 238000012216 screening Methods 0.000 claims description 21
- 230000015654 memory Effects 0.000 claims description 11
- 238000012544 monitoring process Methods 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 10
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 239000000284 extract Substances 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 10
- 238000012423 maintenance Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 6
- 238000007726 management method Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 3
- 238000012550 audit Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 238000013499 data model Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000000750 progressive effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the specification relates to the technical field of big data, and discloses a data query method, a system and a server, wherein the method is applied to a flexible query statement generation component of a data query system, and an interface of the flexible query statement generation component is configured at the starting position of a query execution component; the system is also provided with a data authority table, and the data authority table is provided with data inquiry authority information of a requester; the method comprises the following steps: acquiring an original query statement and a requester identifier in a data query request; analyzing the original query statement to obtain a main table identifier of the target data main table; extracting target inquiry authority information of the requester identifier under the main table identifier from the data authority table; and associating the target query authority information with the original query statement to obtain a flexible query statement with authority control, so that the query execution component executes data query operation based on the flexible query statement, and the code construction of data authority control is simpler and more convenient.
Description
Technical Field
The present disclosure relates to the field of big data technologies, and in particular, to a data query method, system and server.
Background
With the increasing demand for data table query flexibility, data authority control is more and more diversified. At present, data authority control of flexible query generally needs to be coded based on different data tables respectively, and correspondingly, each additional data table with authority control needs to be developed with a matched authority statement independently, so that the development cost is high, and the code maintenance amount is large.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
An object of the embodiments of the present disclosure is to provide a data query method, system and server, which can improve accuracy and efficiency of data query.
The specification provides a data query method, a system and a server, which are realized by the following modes:
a data query method is applied to a flexible query statement generation component of a data query system, wherein an interface of the flexible query statement generation component is configured at the starting position of a query execution component; the data inquiry system is also provided with a data authority table, and the data authority table is provided with inquiry authority information of different data of a requester; correspondingly, the method comprises the following steps: acquiring an original query statement and a requester identifier in a data query request; analyzing the original query statement to obtain a main table identifier of a target data main table queried by the data query request; extracting target query authority information of the requester identifier under the main table identifier from the data authority table; and associating the target query authority information with the original query statement to obtain a flexible query statement with authority control so that the query execution component executes data query operation based on the flexible query statement.
In other embodiments of the method provided in this specification, the query permission information includes a permission field, an intermediate table, and a filtering type, where the filtering type is an intersection; the associating the target query permission information with the original query statement includes: storing the extracted target inquiry authority information into a temporary authority table of the requester identifier; taking out the authority field with the screening type of taking intersection and a middle table from the temporary authority table, and constructing an intersection authority list; and associating the authority field and the intermediate table in the intersection authority list to the original query statement by using an intersection authority association statement.
In other embodiments of the methods provided herein, the type of screening is a union set; associating the target query permission information with the original query statement, further comprising: taking the authority field and the intermediate table with the screening type of the union set from the temporary authority table, and constructing a union set authority list; and associating the authority field and the intermediate table in the union authority list to the original query statement by utilizing a union authority association statement.
In other embodiments of the method provided in this specification, the interface of the flexible query statement generation component is configured at the start position of the query execution component in the following manner: acquiring configuration information of the flexible query statement generation component; under the condition that the permission control switch is determined to be turned on based on the configuration information, acquiring a class name of an execution class of the query execution component; acquiring the byte code class of the execution class from a class pool, and locking the execution method of the byte code class so as to keep the byte code class unchanged in the flexible query statement generation processing; inserting a call statement of an interface of the flexible query statement generation component in a first line of an execution method of the query execution component.
In other embodiments of the method provided in this specification, the query execution component is further configured with an override monitoring post-proxy tangent point, so as to record override information corresponding to the data query request when it is determined that the requestor identifier does not have a query right based on a data query result; the override information includes a query identifier, a flexible query statement, a requestor identifier, and a time to trigger execution of the method.
In other embodiments of the method provided in this specification, the query execution component includes a general control node and a plurality of execution nodes; the general control node is also provided with an execution node table, and the execution node table comprises execution nodes capable of executing data query currently; the general control node receives the flexible query statement, and extracts the execution node with the lowest load occupation in the execution node table under the condition that the execution node capable of executing data query is determined to exist based on the execution node table; sending the flexible query statement to the extracted execution node; and the extracted execution node executes data query operation based on the flexible query statement and feeds back an execution result to the master control node.
In other embodiments of the method provided in this specification, in a case that the general control node determines that the execution node performs an exception based on the execution result, the execution node performing the exception is removed from the execution node table; and the master control node carries out health check on the removed execution node at preset time intervals, and adds the execution node to the execution node table under the condition that the check is passed.
In other embodiments of the method provided in this specification, the performing health check on the removed execution node at preset time intervals by the general control node includes: the general control node initiates a network connection request to the removed execution node, and determines that the health check of the removed execution node does not pass under the condition that the feedback of the removed execution node is not received within a specified time; and under the condition that the master control node receives the feedback of the removed execution node within the appointed time, acquiring the load information of the removed execution node, associating the removed execution node with the corresponding load information, and adding the associated load information into the execution node table.
In another aspect, embodiments of the present specification provide a data query system, where the system includes at least a flexible query statement generation component and a query execution component, and an interface of the flexible query statement generation component is configured at a start position of the query execution component; the data inquiry system is also provided with a data authority table, and the data authority table is provided with inquiry authority information of different data of a requester; the flexible query statement generation component is used for acquiring an original query statement and a requester identifier in a data query request; analyzing the original query statement to obtain a main table identifier of a target data main table queried by the data query request; extracting target query authority information of the requester identifier under the main table identifier from the data authority table; associating the target query authority information with the original query statement to obtain a flexible query statement with authority control; the query execution component is configured to perform a data query operation based on the flexible query statement.
In other embodiments of the apparatus provided in this specification, the query execution component includes at least a general control node and a plurality of execution nodes; the general control node is also provided with an execution node table, and the execution node table comprises execution nodes capable of executing data query currently; the general control node is used for receiving the flexible query statement and extracting the execution node with the lowest load occupation in the execution node table under the condition that the execution node capable of executing data query is determined to exist based on the execution node table; sending the flexible query statement to the extracted execution node; and the extracted execution node is used for executing data query operation based on the flexible query statement and feeding back an execution result to the master control node.
In another aspect, embodiments of the present specification provide a server including at least one processor and a memory for storing processor-executable instructions, which when executed by the processor implement steps of a method including any one or more of the above embodiments.
In the data query method, the system, and the server provided in one or more embodiments of the present specification, a flexible query statement generating component is pre-constructed, and an interface of the flexible query statement generating component is configured at a start position of a query execution component. And pre-configuring a data authority table in the data query system, wherein the data authority table can be configured with query authority information of different data of a requester. After generating the original query statement based on the data query request, the flexible query statement generation component configured at the start position of the query execution component will be triggered first. The flexible query statement generation component may obtain an original query statement, and analyze the original query statement to obtain a main table identifier of a data main table queried by the data query request. Target inquiry authority information of a requester under the identification of the main table can be extracted from the data authority table. Then, the target query permission information may be associated with the original query statement to obtain a flexible query statement with data query permission control, so that the query execution component performs a data query operation based on the flexible query statement. The method has the advantages that the code construction of the data authority control is simpler and more convenient, the maintenance and the management are convenient, and the problem that the flexible data query development and the maintenance with the authority control are complicated is solved.
Drawings
In order to more clearly illustrate the embodiments of the present specification or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only some embodiments described in the present specification, and for those skilled in the art, other drawings can be obtained according to the drawings without any creative effort. In the drawings:
FIG. 1 is a flow chart illustrating a configuration of a data permission table in one embodiment provided herein;
FIG. 2 is a loading diagram of a flexible query statement generation component in one embodiment provided by the subject specification;
FIG. 3 is a flow diagram illustrating the construction of a flexible query statement in one embodiment provided by the present specification;
FIG. 4 is a flow diagram illustrating the construction of a flexible query statement in another embodiment provided by the present specification;
FIG. 5 is a schematic illustration of an override monitoring process in one embodiment provided herein;
FIG. 6 is a flow diagram illustrating an execution cluster executing a flexible query in one embodiment provided by the present specification;
FIG. 7 is a schematic diagram illustrating a health check process of an executing node in one embodiment provided herein;
fig. 8 is a schematic block diagram of a data query system provided in this specification.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in one or more embodiments of the present specification will be clearly and completely described below with reference to the drawings in one or more embodiments of the present specification, and it is obvious that the described embodiments are only a part of the embodiments of the specification, and not all embodiments. All other embodiments obtained by a person skilled in the art based on one or more embodiments of the present specification without making any creative effort shall fall within the protection scope of the embodiments of the present specification.
In one scenario example provided by the embodiments of the present specification, the data query method may apply to a data query system. The data query system may include one or more servers connected to a database. The server may be configured to generate a data query statement based on the data query request, access the database based on the data query statement, screen out data requested by the data query request from the database, and feed back a query result to a requester corresponding to the data query request. The requesting party may refer to a service person or a user, or to a terminal of the service person or the user, or to a service system having a data query requirement, and the like. Of course, the data query system may also include a database that needs to be accessed.
In this embodiment of the present specification, a flexible query statement generation component may be pre-constructed, and an interface of the flexible query statement generation component may be configured at a start position of a query execution component. And a data authority table is pre-configured in the data inquiry system, and inquiry authority information of different data of a requester can be configured in the data authority table. After the server generates the original query statement based on the data query request, the flexible query statement generation component configured at the start position of the query execution component is triggered first.
Correspondingly, the flexible query statement generation component can acquire an original query statement and analyze the original query statement to obtain a main table identifier of a main table of data queried by the data query request. And also can obtain a requester identifier in the data query request, and extract the target query authority information of the requester identifier under the main table identifier from the data authority table. Then, the target query permission information may be associated with the original query statement to obtain a flexible query statement with data query permission control, so that the query execution component performs a data query operation based on the flexible query statement. The method has the advantages that the code construction of the data authority control is simpler and more convenient, the maintenance and the management are convenient, and the problem that the flexible data query development and the maintenance with the authority control are complicated is solved.
Fig. 1 is a flow chart of a configuration of a data authority table. In some embodiments, as shown in fig. 1, the data permission table may be configured in the following manner of steps 101-103.
101: a developer configures related authority fields and intermediate tables for a data main table in a data authority table, wherein the authority fields of a data main table _ A are column _ a, and the intermediate tables are table _ b and table _ c; correspondingly, the authority fields of the intermediate tables of table _ b and table _ c are [ column _ a, column _ b ], [ column _ a, column _ c ] ], respectively. The filter type may also be configured in the data permission table. The screening type may include, for example, intersection and union. Assuming that the relationship between column _ b and the screening authority is intersection, and the relationship between column _ c and the screening authority is union, the relationship structure can be expressed as:
{table_A:
{ field [ column _ a ],
the middle table [ alpha ], [ beta ]
{ table _ b: { field: [ column _ a, column _ b ], type: intersection } },
{ table _ c: { field: [ column _ a, column _ c ], type: union }
]
}
}。
For example, the combined fund needs to filter the authority of the requester according to the area identifier and the combination number, but the requester in a certain area only has the authority to view the combined fund data of the area and the head office, but some area requesters also have the authority to view the combined fund data corresponding to each area under a certain type of combination number. At this time, the intersection can be taken according to the area identifier and the main table identifier of the main table of the combined fund data, so that the authority information of the area where the intersection is located, the authority field of the combined fund data of the head office, the middle table and the like can be screened out, and then the authorized combination number is added, so that all the authority information of the requester on the combined fund data can be obtained.
102: and loading the authority control plug-in corresponding to the flexible query statement generating assembly into a data query program of the server. The authority control plug-in corresponding to the flexible query statement generating assembly can be constructed in advance and stored in a database. The right management and control plug-in can load corresponding programs, configuration files and the like for the flexible query statement generating assembly.
103: and opening a switch of the authority control plug-in the configuration file.
In some embodiments, the interface of the flexible query statement generation component may be configured at the start of the query execution component in the following manner. When the server is started, the server may obtain the configuration information of the flexible query statement generation component, and obtain the class name of the execution class of the query execution component when it is determined that the authority control switch is turned on based on the configuration information. The server may further obtain the bytecode class of the execution class in the class pool, and lock the execution method of the bytecode class, so that the bytecode class remains unchanged in the flexible query statement generation processing. Then, a call statement of an interface of the flexible query statement generation component may be inserted in a first line of an execution method of the query execution component. According to the embodiment, the flexible data authority query function is pluged in a mode of dynamically changing byte codes and loaded into the system when the system is started, so that the loading of the component is simpler and more convenient, meanwhile, the component and the system can have better component adaptability, and the accuracy and the efficiency of flexible query processing flow execution are improved.
FIG. 2 is an example flow of flexible query statement generation component loading. As shown in FIG. 2, the server may perform the following steps 201 and 208 to effect the loading of the components.
201: and reading the configuration file.
202: the entitlement management plug-in switch in the configuration file is checked.
203: the database driver is read.
204: the switch is closed and the entitlement control process is skipped.
205: the class name of the SQL (Structured Query Language) execution class is obtained according to the database driver.
206: and acquiring the byte code class of the corresponding SQL execution class in the class pool.
207: the SQL execution method for locking the byte code class ensures invariability in the modification process.
208: the first line on the SQL execution method inserts the calling code that assembles the interface of the flexible query statement generation component.
The requestor may initiate a data query request that may include at least the identity of the master table it requested and the identity of the requestor. Or, the IP address of the terminal device where the requester is located may also be included. The primary table identification may be identification information of a primary table of a certain type of data stored in a database. The master table of the above-described portfolio data may correspond to a master table identifier in the database. The main table may also have an intermediate table and various fields corresponding thereto to distinguish and identify data within the main table from different perspectives. The storage mode of the data in the database may refer to an actual application scenario, which is not described herein. The requestor identification may be information identifying the requestor. Such as an Identity (Identity) of the requestor, a terminal ID, or a system Identity of the service system.
After receiving the data query request, the server may first generate an original query statement based on the query request. The original query statement refers to a query statement which is generated by the server to access the database and screen out the data required to be queried by the data query request from the database based on the configuration relationship between the server and the accessed database before authority control. After generating the original query statement, the server may further send the query execution component to access the database based on the original query statement and perform a data query operation without permission control. In this embodiment of the present specification, in the case of adding authority control, after generating an original query statement, the server will first trigger the flexible query statement generation component configured at the start position of the query execution component, so as to construct a flexible query statement with authority control. FIG. 3 is a flow diagram of one embodiment of a data query method provided by the present specification that may be applied to the flexible query statement generation component. As shown in fig. 3, the method may include the following steps.
S20: and acquiring an original query statement and a requester identifier in the data query request.
Such as a server, may send the original query statement and the requestor identification to the query execution component after generating the original query statement. After receiving the information, the query execution component triggers the flexible query statement generation component, and sends the original query statement and the requester identifier as input parameters to the flexible query statement generation component through an interface of the flexible query statement generation component. Correspondingly, the flexible query statement generation component obtains the original query statement and the requester identifier in the data query request.
S22: and analyzing the original query statement to obtain a main table identifier of a target data main table queried by the data query request.
The flexible query statement generation component can analyze the original query statement to obtain a main table identifier of a target data main table queried by the data query request. The original query statement may be parsed using AST (abstract syntax tree), for example, to obtain a tree-structured statement parsing result. And acquiring a main table identifier of the target data main table from the statement analysis result. Of course, other types of syntax parsing tools may be used to parse the original query statement, which is not limited herein.
S24: and extracting target query authority information of the requester identifier under the main table identifier from the data authority table.
The flexible query statement generation component can query the data permission table according to the requester identifier and the main table identifier, and extract the target query permission information of the requester identifier under the main table identifier. The target query authority information may include, for example, an authority field of the requestor identifier under the master table identifier, an intermediate table, a filtering type, and the like.
If the record is not obtained when the authority table is queried, the query can be considered to be free from adding authority control, the step of flexible query statement construction is skipped, and the query execution component directly accesses the database based on the original query statement. Or the requestor can also be considered to have no query authority for the target data main table, an unauthorized statement is associated in the original query statement, and correspondingly, the statement obtained after the query execution component executes association is returned to null. The above is merely a preferable example, and the specific situation may be configured according to an actual application scenario.
If the record is obtained when the authority table is queried, a statement for creating the temporary authority table can be generated, and the temporary authority table is created based on the statement. The full amount of the queried records may be added to the temporary permission table. Step S26 may then be performed for flexible query statement construction.
S26: and associating the target query authority information with the original query statement to obtain a flexible query statement with authority control so that the query execution component executes data query operation based on the flexible query statement.
The flexible query statement generation component can extract the authority field, the intermediate table and the like from the temporary authority table to generate the screening condition statement during data screening. Then, the screening conditional statement and the original query statement can be associated to obtain a flexible query statement with authority control. If the temporary authority table can be traversed in sequence, the extracted authority field and the intermediate table generate a screening conditional statement, and the screening conditional statement is associated with the original query statement to obtain a flexible query statement with authority control. If the authority field and the intermediate table which are not associated exist, the authority field and the intermediate table can be further extracted to generate the screening conditional statement, the generated screening conditional statement is associated to the generated flexible query statement, and the flexible query statement is updated. And ending the traversal until the temporary authority table does not have the authority field and the intermediate table which are not associated, and outputting the final flexible query statement.
The query execution component can receive the output information of the flexible query statement generation component, execute data query operation based on the flexible query statement, and query the data under the authority corresponding to the requester from a database. The server may feed back the queried data to the device where the requesting party is located.
In some embodiments, if the filtering type is to take an intersection, the permission field and the intermediate table whose filtering type is to take an intersection may be further taken from the temporary permission table to construct an intersection permission list. And associating the authority field and the intermediate table in the intersection authority list to the original query statement by using an intersection authority association statement. The data query statements corresponding to the intersection and the union are different, so that the association statements corresponding to the intersection and the union can be respectively and correspondingly generated according to different screening types, and then association operation is performed based on the corresponding association statements. Correspondingly, the permission field and the intermediate table can be extracted in a mode of traversing the intersection permission list, and the permission field and the intermediate table are sequentially associated to the original query statement by using the generated intersection permission association statement.
If the screening type is the union set, the flexible query statement generation component can take the authority field and the intermediate table of which the screening type is the union set from the temporary authority table to construct a union set authority list. And associating the authority field and the intermediate table in the union authority list to the original query statement by using a union authority association statement. Correspondingly, the permission field and the intermediate table can be extracted in a mode of traversing the union permission list, and the permission field and the intermediate table are sequentially associated to the original query statement by using the generated union permission association statement.
FIG. 4 is a schematic diagram of the construction of a flexible query statement. As shown in FIG. 4, in one scenario example, the flexible query statement generation component may utilize step 401 and 417 to construct the flexible query statement.
401: an interface is invoked that assembles the flexible query statement generation component, and the incoming parameters are the original query SQL (original query statement).
402: and analyzing the original query SQL by using the AST syntax tree to obtain an SQL analysis result of the tree structure.
403: and acquiring a target data main table of flexible query from the SQL analysis result.
404: and inquiring the authority table according to the current requester identifier and the main table identifier to obtain the associated authority field, the intermediate table and the screening type.
405: and if the record is not acquired when the authority table is inquired, the flexible inquiry is considered to be free from adding authority control.
406: SQL assembly is skipped.
407: an SQL statement is generated that creates a temporary permission table.
408: and adding the extracted full-amount authority into the temporary authority table.
409: and taking out the authority fields and the intermediate table with the screening types of intersection according to the acquired authority information, generating an intersection authority list, and judging whether the list is empty or not.
410: and generating an intersection connection SQL statement.
411: and traversing the intersection permission list, and judging whether the permission field of the unassembled intersection type exists at present.
412: and splicing the currently traversed intersection permission field and the related intermediate table into query SQL.
413: and taking out the authority field and the intermediate table with the screening type of union set according to the acquired authority information, generating a union set authority list, and judging whether the list is empty or not.
414: a union join SQL statement is generated.
415: and traversing the union authority list and judging whether the authority field of the unassembled union type exists at present.
416: the union permission field and the associated intermediate table currently traversed are spliced into the SQL generated in step 412.
417: the assembly of the flexible query SQL (flexible query statement) is completed.
In other embodiments, the query execution component may be further configured with an override monitoring post-proxy tangent point. After the query execution component executes the data query operation, whether the requester identifier has the query right can be determined based on the data query result. If the data query result is null, it may be determined that the requestor identification does not have query permission. And under the condition that the requester identification does not have the query authority, an override recording component can be called to record the override information corresponding to the data query request. The override information may include a query identifier, a flexible query statement, a requester identifier, an IP address of a device where the requester is located, a time for triggering the execution method, and the like.
By monitoring the unauthorized, when the final flexible query is executed, the related unauthorized information for auditing is obtained and inserted into an audit log table. The unauthorized information of the audit log table can be analyzed regularly, for example, the unauthorized frequency of a requester and the unauthorized data table frequency are extracted, the unauthorized high-risk requester or the unauthorized high-risk data table is evaluated so as to control the requester or the data table, and the security of data query can be further improved.
Fig. 5 shows an override monitoring flow chart. As shown in fig. 5, the server may perform the unauthorized monitoring using steps 501 to 507.
501: the SQL execution class is read.
502: adding a post-proxy tangent point at the execution method of the SQL execution class.
503: waiting for the flexible query to trigger the SQL execution method.
504: and acquiring a currently queried data master table through the right management and control plug-in.
505: and inquiring whether the requester has the inquiry authority of the data master table or not according to the current requester and the name of the data master table.
506: if the requester has no inquiry authority, the SQL ID, the SQL statement and the current requester ID of the SQL are obtained, the time of the execution method is triggered, and the time is recorded in an audit log table.
507: the requestor is not involved in the override and does not do processing.
In other embodiments, the query execution component may further include a general control node and a plurality of execution nodes. The general control node may be further configured with an execution node table, and the execution node table may include an execution node that can currently execute a data query. The general control node can receive the flexible query statement, extract the execution node with the lowest load occupation in the execution node table under the condition that the execution node capable of executing data query is determined to exist based on the execution node table, and send the flexible query statement to the extracted execution node. Correspondingly, the extracted execution node can execute data query operation based on the flexible query statement, and feed back an execution result to the general control node.
In other embodiments, the general control node may remove the execution node with the execution exception from the execution node table when determining that the execution node performs the exception based on the execution result, perform health check on the removed execution node at preset time intervals by the general control node, and further add the execution node to the execution node table when the check passes.
The SQL is received and the query result is returned through the master control node, and each execution node is monitored and scheduled, so that the flexible query efficiency and the problem positioning accuracy are improved, and the overall operation and maintenance cost is reduced.
In other embodiments, the general control node may perform health check in the following manner. The general control node can initiate a network connection request to the removed execution node, and determine that the health check of the removed execution node fails in the case that the feedback of the removed execution node is not received within a specified time. The general control node may further obtain the load information of the removed execution node under the condition that the feedback of the removed execution node is received within a specified time, and add the removed execution node and the corresponding load information into the execution node table after associating them. Therefore, the scheduling of the execution nodes can be realized more simply and effectively. Of course, the above scheduling and health check method is a preferred example, and in the specific implementation, the node scheduling and health check may be performed by referring to other factors.
In the traditional scene, the query functions are distributed in each interface of each system, and due to the fact that the height of the dependent programs is inconsistent, once query failure occurs, error points are difficult to locate, and the operation and maintenance cost is high. Moreover, HTTP forwarding is performed through a single forwarding tool, the load condition of the system cannot be obtained, and slow flexible queries may be accumulated on a single node, so that the hardware utilization efficiency is low. The flexible query with query authority control has more complicated and changeable query logic, and if the existing query execution architecture is adopted, the query load is heavier, so that the operation and maintenance success is higher. In this embodiment, flexible query execution is rearranged, and the function of executing SQL is extracted and deployed as an SQL execution cluster through operations such as node health check and node scheduling. The SQL master control node is used for receiving the SQL and returning the query result, monitoring and scheduling each execution node, improving the efficiency of flexible query and the accuracy of problem location, and further reducing the overall operation and maintenance cost.
Each assembled flexible query SQL is submitted to a master control node firstly, the master control node queries the load condition of an execution node, and the node with the lowest load is selected from the load condition to carry out flexible query SQL distribution. If a problem occurs in the SQL execution process, recording an error log and the node equipment information into an SQL execution log table. Before distribution, the general control node can also perform health check on the SQL execution node, and if the health check of the node fails, the SQL general control node performs flow isolation on the SQL general control node and removes the SQL general control node from the execution node table. Or, when a problem occurs in the execution process and the execution fails, the general control node may also remove the execution node with failed execution from the execution node table based on the execution result. The general control node can try the health check again on the removed SQL execution node every preset time interval, and if the check is passed, the execution node can be added into the execution node table. By clustering the query by adopting node scheduling, the problems of difficult flexible data query and positioning, low hardware use efficiency and the like can be effectively solved, and the efficiency and the accuracy of flexible query are greatly improved.
FIG. 6 shows an exemplary flow diagram for an SQL execution cluster executing a flexible query. As shown in fig. 6, the cluster may perform a data query using steps 601 to 610.
601: and the master control node receives the flexible query SQL.
602: and judging whether the normal node list has available execution nodes.
603: and if the normal node list is empty, returning a message without available nodes.
604: and traversing the normal node list, and selecting the SQL execution node with the lowest charge of the CPU and the memory.
605: distributing the flexible query SQL to the execution nodes.
606: the executing node executes the SQL.
607: and checking whether the execution node executes the SQL abnormally.
608: if the execution is not abnormal, the SQL execution result is returned.
609: and recording the SQL ID, the SQL statement and the requester ID, and the time for executing the SQL to the SQL execution log table.
610: and returning an execution failure message.
FIG. 7 is a diagram illustrating an example of a health check flow for an executing node. As shown in FIG. 7, the general control node may perform a health check on the executing nodes using steps 701 through 707.
701: and the master control node requests the health check interface of the execution node.
702: it is determined whether the request can establish a network connection.
703: if the network connection cannot be established, the master control node considers that the execution node is inaccessible, the node is removed from the normal node list, the removed node is classified into the abnormal node list, the nodes in the abnormal node list are subjected to health check once every 1 hour, and if the health check is passed, the nodes are added into the execution node list again.
704: and judging whether the data connection of the execution node is normal or not.
705: if the data connection of the execution node is abnormal, the health check is returned to fail, and the general control node removes the execution node from the normal node list.
706: and the execution node acquires the running load information of the CPU and the memory of the execution node.
707: returning the load information of the node, and considering that the health examination is passed by the master control node;
708: the master control node adds the execution node into the execution node table.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. For details, reference may be made to the description of the related embodiments of the related processing, and details are not repeated herein.
The foregoing description has been directed to specific embodiments of this disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
Based on the scheme provided by the above embodiment, the present specification further provides a data query system. As shown in fig. 8, the system may include at least a flexible query statement generation component and a query execution component, and an interface of the flexible query statement generation component is configured at a start position of the query execution component. The data inquiry system can also be configured with a data authority table, and the data authority table is configured with inquiry authority information of different data of a requester.
The system can also include an original query statement generation component for generating an original query statement based on a data query request of a requestor. Correspondingly, the flexible query statement generation component may be configured to obtain an original query statement and a requestor identifier in a data query request; analyzing the original query statement to obtain a main table identifier of a target data main table queried by the data query request; extracting target query authority information of the requester identifier under the main table identifier from the data authority table; and associating the target query authority information with the original query statement to obtain a flexible query statement with authority control.
The query execution component may be operative to perform a data query operation based on the flexible query statement.
In other embodiments, the query execution component may include at least a general control node and a plurality of execution nodes; and the general control node is also provided with an execution node table, and the execution node table comprises execution nodes capable of executing data query currently.
The general control node can be used for receiving the flexible query statement and extracting the execution node with the lowest load occupation in the execution node table under the condition that the execution node capable of executing data query is determined to exist based on the execution node table; and sending the flexible query statement to the extracted execution node.
The extracted execution node can be used for executing data query operation based on the flexible query statement and feeding back an execution result to the general control node.
The system may also include an override monitoring component that may be invoked by adding a post-proxy node to the execution method of the query execution component. The override monitoring component may be operative to determine whether the requestor is overridden based on a query result of the query execution component. The specific override processing mode may be implemented by referring to the above embodiments, and is not described herein.
It should be noted that the above-described system may also include other embodiments according to the description of the method embodiment. The specific implementation manner may refer to the description of the related method embodiment, and is not described in detail herein.
The present specification also provides a server that may be implemented in a single data query system or in multiple computer data processing systems. The system may comprise a server cluster, a system (including a distributed system), software (applications), actual operating devices, logical gate devices, quantum computers, etc. using one or more of the methods described herein or one or more of the example devices, in conjunction with terminal devices implementing the necessary hardware. In some embodiments, the server may include at least one processor and a memory for storing processor-executable instructions that, when executed by the processor, perform steps comprising the method of any one or more of the embodiments described above.
The memory may include physical means for storing information, typically by digitizing the information for storage on a medium using electrical, magnetic or optical means. The storage medium may include: devices that store information using electrical energy, such as various types of memory, e.g., RAM, ROM, etc.; devices that store information using magnetic energy, such as hard disks, floppy disks, tapes, core memories, bubble memories, and usb disks; devices that store information optically, such as CDs or DVDs. Of course, there are other ways of storing media that can be read, such as quantum memory, graphene memory, and so forth.
It should be noted that the above-mentioned device may also include other implementation manners according to the description of the method or apparatus embodiment, and specific implementation manners may refer to the description of the related method embodiment, which is not described in detail herein.
It should be noted that the embodiments of the present disclosure are not limited to the cases where the data model/template is necessarily compliant with the standard data model/template or the description of the embodiments of the present disclosure. Certain industry standards, or implementations modified slightly from those described using custom modes or examples, may also achieve the same, equivalent, or similar, or other, contemplated implementations of the above-described examples. The embodiments using these modified or transformed data acquisition, storage, judgment, processing, etc. may still fall within the scope of the alternative embodiments of the present description.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment. In the description of the specification, reference to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the specification. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
The above description is only an example of the present specification, and is not intended to limit the present specification. Various modifications and alterations to this description will become apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification should be included in the scope of the claims of the present specification.
Claims (11)
1. A data query method is applied to a flexible query statement generation component of a data query system, and an interface of the flexible query statement generation component is configured at the starting position of a query execution component; the data inquiry system is also provided with a data authority table, and the data authority table is provided with inquiry authority information of different data of a requester; correspondingly, the method comprises the following steps:
acquiring an original query statement and a requester identifier in a data query request;
analyzing the original query statement to obtain a main table identifier of a target data main table queried by the data query request;
extracting target query authority information of the requester identifier under the main table identifier from the data authority table;
and associating the target query authority information with the original query statement to obtain a flexible query statement with authority control so that the query execution component executes data query operation based on the flexible query statement.
2. The method of claim 1, wherein the query permission information comprises a permission field, an intermediate table, and a filter type, wherein the filter type is an intersection; the associating the target query permission information with the original query statement includes:
storing the extracted target inquiry authority information into a temporary authority table of the requester identifier;
taking out the authority field with the screening type of taking intersection and a middle table from the temporary authority table, and constructing an intersection authority list;
and associating the authority field and the intermediate table in the intersection authority list to the original query statement by using an intersection authority association statement.
3. The method of claim 2, wherein the type of filtering is a union set; associating the target query permission information with the original query statement, further comprising:
taking the authority field and the intermediate table with the screening type of the union set from the temporary authority table, and constructing a union set authority list;
and associating the authority field and the intermediate table in the union authority list to the original query statement by utilizing a union authority association statement.
4. The method of claim 1, wherein the interface of the flexible query statement generation component is configured at the start of the query execution component in the following manner:
acquiring configuration information of the flexible query statement generation component;
under the condition that the permission control switch is determined to be turned on based on the configuration information, acquiring a class name of an execution class of the query execution component;
acquiring the byte code class of the execution class from a class pool, and locking the execution method of the byte code class so as to keep the byte code class unchanged in the flexible query statement generation processing;
inserting a call statement of an interface of the flexible query statement generation component in a first line of an execution method of the query execution component.
5. The method according to claim 1, wherein an override monitoring post-proxy tangent point is further configured in the query execution component to record override information corresponding to the data query request when it is determined that the requestor identifier has no query authority based on a data query result; the override information includes a query identifier, a flexible query statement, a requestor identifier, and a time to trigger execution of the method.
6. The method of claim 1, wherein the query execution component comprises a general control node and a plurality of execution nodes; the general control node is also provided with an execution node table, and the execution node table comprises execution nodes capable of executing data query currently;
the general control node receives the flexible query statement, and extracts the execution node with the lowest load occupation in the execution node table under the condition that the execution node capable of executing data query is determined to exist based on the execution node table; sending the flexible query statement to the extracted execution node;
and the extracted execution node executes data query operation based on the flexible query statement and feeds back an execution result to the master control node.
7. The method according to claim 6, wherein the general control node removes the execution node with abnormal execution from the execution node table in case of determining that the execution node performs abnormal based on the execution result;
and the master control node carries out health check on the removed execution node at preset time intervals, and adds the execution node to the execution node table under the condition that the check is passed.
8. The method of claim 7, wherein the general control node performs health check on the removed executing node at preset time intervals, comprising:
the general control node initiates a network connection request to the removed execution node, and determines that the health check of the removed execution node does not pass under the condition that the feedback of the removed execution node is not received within a specified time;
and under the condition that the master control node receives the feedback of the removed execution node within the appointed time, acquiring the load information of the removed execution node, associating the removed execution node with the corresponding load information, and adding the associated load information into the execution node table.
9. A data query system is characterized in that the system at least comprises a flexible query statement generation component and a query execution component, wherein an interface of the flexible query statement generation component is configured at the starting position of the query execution component; the data inquiry system is also provided with a data authority table, and the data authority table is provided with inquiry authority information of different data of a requester;
the flexible query statement generation component is used for acquiring an original query statement and a requester identifier in a data query request; analyzing the original query statement to obtain a main table identifier of a target data main table queried by the data query request; extracting target query authority information of the requester identifier under the main table identifier from the data authority table; associating the target query authority information with the original query statement to obtain a flexible query statement with authority control;
the query execution component is configured to perform a data query operation based on the flexible query statement.
10. The system of claim 9, wherein the query execution component comprises at least a headmaster node and a plurality of execution nodes; the general control node is also provided with an execution node table, and the execution node table comprises execution nodes capable of executing data query currently;
the general control node is used for receiving the flexible query statement and extracting the execution node with the lowest load occupation in the execution node table under the condition that the execution node capable of executing data query is determined to exist based on the execution node table; sending the flexible query statement to the extracted execution node;
and the extracted execution node is used for executing data query operation based on the flexible query statement and feeding back an execution result to the master control node.
11. A server, characterized in that the server comprises at least one processor and a memory for storing processor-executable instructions, which when executed by the processor implement steps comprising the method of any of the preceding claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110171197.5A CN112905592A (en) | 2021-02-08 | 2021-02-08 | Data query method, system and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110171197.5A CN112905592A (en) | 2021-02-08 | 2021-02-08 | Data query method, system and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112905592A true CN112905592A (en) | 2021-06-04 |
Family
ID=76123988
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110171197.5A Pending CN112905592A (en) | 2021-02-08 | 2021-02-08 | Data query method, system and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112905592A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282591A (en) * | 2021-06-29 | 2021-08-20 | 深圳平安智汇企业信息管理有限公司 | Authority filtering method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109947796A (en) * | 2019-04-12 | 2019-06-28 | 北京工业大学 | A kind of caching method of distributed data base system inquiry intermediate result set |
| CN111414407A (en) * | 2020-02-13 | 2020-07-14 | 中国平安人寿保险股份有限公司 | Data query method and device of database, computer equipment and storage medium |
| CN111506611A (en) * | 2020-04-21 | 2020-08-07 | 北京同邦卓益科技有限公司 | Data query method, device, equipment and storage medium |
| CN111552678A (en) * | 2020-03-30 | 2020-08-18 | 平安医疗健康管理股份有限公司 | Data permission configuration method and device and computer equipment |
-
2021
- 2021-02-08 CN CN202110171197.5A patent/CN112905592A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109947796A (en) * | 2019-04-12 | 2019-06-28 | 北京工业大学 | A kind of caching method of distributed data base system inquiry intermediate result set |
| CN111414407A (en) * | 2020-02-13 | 2020-07-14 | 中国平安人寿保险股份有限公司 | Data query method and device of database, computer equipment and storage medium |
| CN111552678A (en) * | 2020-03-30 | 2020-08-18 | 平安医疗健康管理股份有限公司 | Data permission configuration method and device and computer equipment |
| CN111506611A (en) * | 2020-04-21 | 2020-08-07 | 北京同邦卓益科技有限公司 | Data query method, device, equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113282591A (en) * | 2021-06-29 | 2021-08-20 | 深圳平安智汇企业信息管理有限公司 | Authority filtering method and device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110290212B (en) | Service call recording method, device, computer equipment and storage medium | |
| WO2020238066A1 (en) | Software pressure test method and apparatus, and storage medium and server | |
| US8255409B2 (en) | Systems and methods for generating a change log for files in a managed network | |
| US20160196307A1 (en) | Statistical identification of instances during reconciliation process | |
| US9356966B2 (en) | System and method to provide management of test data at various lifecycle stages | |
| US8214809B2 (en) | Grid-enabled ANT compatible with both stand-alone and grid-based computing systems | |
| CN114595201A (en) | Method, equipment and storage medium for inquiring acquisition record of interface access log | |
| CN111967017A (en) | Method and device for generating dependency relationship, terminal equipment and storage medium | |
| CN112395157A (en) | Audit log obtaining method and device, computer equipment and storage medium | |
| CN111078551A (en) | Full link testing method, device and system and computer readable storage medium | |
| CN112363907A (en) | Test method and device for Dubbo interface, electronic device and storage medium | |
| CN112905592A (en) | Data query method, system and server | |
| CN111611276A (en) | Data query method, device and storage medium | |
| Rana et al. | Resource Discovery for Dynamic Clusters in Computational Grids. | |
| CN112306973A (en) | Picture importing system and method | |
| EP1710698A2 (en) | Generic software requirements analyser | |
| CN111435327B (en) | Log record processing method, device and system | |
| CN111309986A (en) | Big data acquisition and sharing system | |
| CN115858466A (en) | Operation log generation method, device, equipment and medium | |
| CN112749197B (en) | Data fragment refreshing method, device, equipment and storage medium | |
| CN111984473B (en) | Memory snapshot data acquisition method and related device | |
| CN114722118A (en) | Data synchronization method and system, computer readable storage medium | |
| CN115221125A (en) | File processing method and device, electronic equipment and readable storage medium | |
| CN116881880B (en) | Space-time data management system and space-time data service resource cooperative scheduling method | |
| CN114115853B (en) | Method and system for dynamically controlling API (application program interface) access |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |