Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
For the purpose of facilitating understanding of the embodiments of the present application, the following description will be made in terms of specific embodiments with reference to the accompanying drawings, which are not intended to limit the embodiments of the present application.
The data source information verification method and device provided by the embodiment of the application are suitable for a scene of verifying a database accessed by a database access operation in some specific scenes in an internet system, wherein the internet system comprises a client and a server, the client can send a database access request to the server, and the database access request comprises: writing a database request and reading a database request; after receiving the database access request, the server is connected to the corresponding database by calling the database access interface, and then various database access operations (including database reading operation and database writing operation) can be performed.
The server side of the application can comprise a plurality of databases, the databases can be deployed in different places, and the databases can be divided into two types: the number of the write data bases can be one, and the number of the read data bases can be multiple. The data in the read database is copied from the write database, so that the real-time performance of the write database is highest, and when the data in the write database changes, the changed data in the write database can be synchronized to the multiple read databases in sequence. In the default state, the database accessed by the read database operation is the read database, and the database accessed by the write database operation is the write database. It will be appreciated that because there may be multiple read databases, the database accessed by the current read database operation may be determined according to a preset routing policy.
It should be noted that the specific scenarios mentioned above may include, but are not limited to, the following three scenarios:
scene one: in the database-writing mandatory scenario, that is, when the server receives a database-writing request and parses the database-writing request into one or more database access operations, assuming that the one or more database access operations include a database-reading operation, it is desirable that the database-reading operation reads data from the database-writing operation in order to ensure real-time data, and at this time, the database accessed by the database-reading operation may be checked, that is, the database accessed by the database-reading operation is checked as the database-writing.
Scene two: in the consistent reading scenario, according to the above description, when data in a write database changes, the changed data in the write database may be sequentially synchronized to multiple read databases, and in the data synchronization process, when a server receives a read database request and parses the database request into multiple read database operations, assuming that the multiple read database operations respectively access different read databases, because of a time difference in synchronization of the multiple read databases, a part of the read database operations accesses a synchronized read database, and a part of the read database operations accesses an unsynchronized read database, that is, any one of the database access operations accesses an unsynchronized database, the reading result is erroneous. Therefore, the database accessed by the plurality of database access operations can be checked, namely, the database accessed by the plurality of database access operations is checked to be consistent, in this case, the read result error is caused only when the read database is not synchronized, and therefore, the probability of the read result error is reduced to one fifth of the worst case.
For example, in the case that multiple databases are deployed in different places, that is, assuming that a write database is deployed at a place a and multiple read databases are deployed at a place B, when data in the write database of the place a changes, time for synchronizing the multiple read databases to the place B may be different, and when the data is not completely synchronized, the place B triggers a read database query operation, assuming that the query operation requires querying the 5 times of databases, if each query is routed to a different read database, data of any one of the 5 read databases is old, which results in a query result being incorrect, and if it can be guaranteed that the 5 times of databases are all routed to the same read database, only if the read databases are not synchronized, which results in a query result being incorrect.
Scene three: in the development process of the internet system, data migration (namely, data migration from one database to another database) is inevitably encountered, and particularly when the data is migrated to the heterogeneous database, in order to ensure that the internet system can normally operate after the data migration, the database accessed by the operation of the read database can be checked, that is, whether the database accessed by the operation of the read database is the migrated database is checked, and the correctness of the access result can be verified at the same time; and if the internet system cannot operate correctly after the data migration, the data can be migrated back to the previous database, and whether the database accessed by the database reading operation is the database before the migration is further verified.
In summary, in the above three scenarios, it is necessary to check the database accessed by the database access operation. Because the connection information of the database is usually stored in the data source, the corresponding database can be searched and connected according to the data source information. Thus, the verification of the database accessed by the database access operation may be translated into a verification of the data source information.
It should be noted that, the data source information of the present application is also referred to as logical data source information, and the logical data source information corresponds to an actual physical database, that is, one physical database can be uniquely determined according to any logical data source information.
The data source information verification method provided by the present application may be executed by a data source information verification apparatus shown in fig. 1, where in fig. 1, the verification apparatus is also referred to as a data blood margin verification apparatus, and may include: a database operation interceptor 11, a rule parser 12, and a path verifier 13. The database operation interceptor 11 is configured to intercept a database access operation on the database access interface; the rule parser 12 is configured to parse annotation information of the database access interface to obtain a predefined checking algorithm (also referred to as a routing rule or a routing policy); here, the annotation information of the database access interface may be written in advance by a tester, and the written annotation information may be added to the test script corresponding to the database access interface, so that when the database access interface is called, the rule parser 12 parses the annotation information, and here, since the annotation information does not participate in debugging, when the annotation information is added to the test script, the test script does not need to be debugged again; the path verifier 13 is configured to verify, according to a predefined verification algorithm obtained by parsing by the rule parser 12, data source information corresponding to the database access operation intercepted by the database operation interceptor 11.
In this application, the number of the path verifiers 13 may be multiple, and the number may be consistent with the number of the data of the predefined verification algorithm, for example, when the predefined verification algorithms obtained by the analysis include three, the number of the path verifiers 13 may be three, and the predefined verification algorithms correspond to the path verifiers 13 one to one. In one example, the three path checkers 13 may be: the system comprises a first path checker, a second path checker and a third path checker, wherein the first path checker is used for checking whether a plurality of data source information are consistent; the second path checker is used for checking whether one or more data source information is consistent with preset data source information; the third path checker is used for checking whether the one or more data source information is consistent with the data source information of the predefined type.
It can be understood that the data source information verifying devices shown in fig. 1 correspond to the database access interfaces one to one, that is, one data source information verifying device is used to verify data source information corresponding to the database access operation of one database access interface, and when data source information corresponding to the database access operation of multiple database access interfaces needs to be verified, the data source information verifying device can complete verification through multiple data source information verifying devices, where the verifying process of each verifying device is similar. The data source information corresponding to the database access operation of the database access interface is verified.
Fig. 2 is a flowchart of a data source information verification method according to an embodiment of the present application, where an execution subject of the method may be the data source information verification apparatus shown in fig. 1, and in fig. 1, the method may specifically include:
step 210, when a call request for a database access interface is received, reading annotation information of the database access interface.
Referring to fig. 3, which is a schematic diagram of a database access interface provided in the present application, in fig. 3, a test script is used to test the database access interface; the test driver is used for simulating a client to send a calling request to the database access interface; the database access interface is distributed at a service interface (service) layer, a component (component) layer, and a Data Access (DAO) layer. It should be noted that, the database access operation obtained after the resolution of the call request starts to be shunted at the data access layer.
In fig. 3, when a test script of a database access interface is run to test the database access interface, a test driver starts a corresponding thread, the started thread sends a call request to the database access interface, and a data source information verification apparatus reads annotation information of the database access interface from the test script corresponding to the database access interface when detecting that the database access interface is called.
That is, the verification process of the data source information of the present application is performed in combination with the test process of the database access interface, that is, the verification method of the data source information of the present application can be used as an extension of the test of the database access interface, and the verification result information obtained finally in the present application can be returned together with the test result information of the database access interface.
Step 220, analyzing the annotation information to obtain a predefined verification algorithm.
Here, the predefined checking algorithm includes, but is not limited to, checking whether the plurality of data source information are consistent; and/or, checking whether one or more data source information is consistent with preset data source information or predefined types of data source information.
Specifically, after reading the annotation information of the database access interface, the data source information verification apparatus starts the rule parser 12, so that the rule parser 12 parses the annotation information of the database access interface. In one example, the annotation information for the database access interface may be: @ DbPathRule (checker ═ consistchecker. class, rule ═ default "), after parsing the annotation information, the obtained predefined checking algorithm may be: and checking whether the information of the plurality of data sources is consistent, namely calling the first path checker. In addition, the application can further set the predefined verification algorithm. In particular, the above-mentioned predefined check algorithm may be further set by a variable "rule". For example, when the parsed predefined check algorithm is to check whether the information of the multiple data sources is consistent, and assuming that the variable "rule" is assigned "oracle", the predefined check algorithm may be further refined as follows: and checking whether the plurality of data source information are consistent and whether a database connected with the consistent data source information is an oracle database. Of course, the variable "rule" may be assigned to other types of databases, such as "oceanbase" and "mysql". It is understood that when the variable "rule" is assigned to "default", the predefined check algorithm is not further configured.
In another example, the annotation information for the database access interface may be: @ DbPathRule (checker ═ obchecker. class, rule ═ default "), after parsing the annotation information, the obtained predefined checking algorithm may be: and checking whether the one or more data source information is the data source information of the oceanbase database, namely checking whether the one or more data source information is consistent with the data source information of the predefined type, so that a second path checker can be called. Of course, a database of predefined types of data source information connections can be further set by assigning a value to the variable "rule".
In yet another example, the annotation information for the database access interface may be: @ DbPathRule (checker ═ writedbchecker. class, rule ═ default "), after parsing the annotation information, the obtained predefined checking algorithm may be: and checking whether the one or more data source information is the data source information of the write database, namely checking whether the one or more data source information is consistent with the preset data source information, so that a third path checker can be called.
Of course, in practical applications, the predefined checking algorithm may not be limited to the above three algorithms, and may further include transaction status detection write bank enforcement, white list prioritization, and the like. Note information corresponding to each predefined verification algorithm may be written in conjunction with the above examples. In addition, more than two predefined verification algorithms can be contained in the annotation information, when more than two predefined verification algorithms are obtained after the annotation information of the database access interface is analyzed, the predefined verification algorithm with higher priority can be selected from the more than two predefined verification algorithms according to the preset priority, and then the verifier corresponding to the predefined verification algorithm with higher priority is called for verification.
It should be further noted that the above example only shows a writing method of annotation information, and in practical applications, the annotation information may also be written in other forms as long as the annotation information can be distinguished from the code in the test script, which is subject to debugging, and the rule parser 12 can parse out the corresponding predefined verification algorithm.
Step 230, intercepting one or more database access operations of the database access interface, and acquiring one or more data source information corresponding to the database access operations.
In particular, one or more database access operations of the database access interface shown in FIG. 3 may be intercepted at the data access layer of the database access interface.
For example, if the test driver calls the database access interface to read the user information of the user, that is, when the call request in step 210 is used to read the user information of the user, and if the user information includes login information and identity information, and the two kinds of information of the user are respectively recorded in two different tables, the implementation code of the database access interface may parse the call request into two database reading operations, and then the database operation interceptor 11 intercepts the two database access operations on the data access layer of the database access interface; and then, acquiring corresponding data source information from the operation statements corresponding to the two database access operations. Preferably, the data source information may be a database name. Here, it is in the prior art that the operation statement corresponding to the database access operation includes data source information, and this application is not repeated herein.
Of course, in practical applications, step 230 may be executed first, and then step 220 is executed, or both steps may be executed simultaneously, which is not limited in this application.
And 240, verifying the data source information according to the predefined verification algorithm, and obtaining verification result information.
Specifically, the path verifier 13 corresponding to the predefined verification algorithm is called to verify the data source information and obtain verification result information.
When the predefined checking algorithm is to check whether the information of the multiple data sources is consistent, step 240 may specifically be:
calling a first path checker to compare the one or more data source information, and if the comparison is consistent, obtaining the verification result information as information indicating successful verification; otherwise, the obtained verification result information is information indicating that verification is unsuccessful.
As in the foregoing example, the first path verifier is invoked to compare two pieces of data source information corresponding to two read database operations, and it is assumed that the two tables belong to two read databases respectively, and the two pieces of data source information can be "oracle 1" and "oracle 2", respectively, because the two pieces of data source information are inconsistent, the obtained verification result information is information indicating that the verification is unsuccessful. If the two tables belong to one read database respectively, if the data source information corresponding to the two read database operations is "oracle 1", the obtained verification result information is information indicating that the verification is successful because the data source information is consistent; further, if the variable "rule" is also assigned in the annotation information corresponding to the database access interface in the foregoing example, if the assignment is "oracle", when the data source information is consistent, it may also be checked whether the database connected to the data source information is an oracle database, if so, the obtained verification result information is information indicating that the verification is successful, otherwise, the obtained verification result information is still information indicating that the verification is unsuccessful.
When the predefined verification algorithm is to verify whether one or more pieces of data source information are consistent with preset data source information, step 240 may further specifically be:
calling a second path checker to compare the one or more pieces of data source information with preset data source information, and if any piece of data source information is inconsistent with the preset data information, obtaining verification result information which is information indicating that verification is unsuccessful; otherwise, the obtained verification result information is information indicating successful verification.
As in the foregoing scenario one, a database to which a read database operation is expected to access is a write database, that is, a second path verifier may be invoked to check whether data source information corresponding to one or more read database operations is data source information of the write database, for example, when the data source information of the write database is "WriteDbOracle", the acquired data source information may be compared with "WriteDbOracle", and if the data source information is not consistent, the acquired verification result information is information indicating that the verification is unsuccessful; otherwise, the obtained verification result information is information indicating successful verification. Of course, the linked write database may be further set by assigning a value to the variable "rule".
When the predefined verification algorithm is to verify whether the multiple pieces of data source information are consistent with the predefined types of data source information, step 240 may further specifically be:
calling a third path checker to compare the one or more pieces of data source information with the data source information of the predefined type, wherein if any piece of data source information is inconsistent with the data source information of the predefined type, the obtained verification result information is information indicating that verification is unsuccessful; otherwise, the obtained verification result information is information indicating successful verification.
As in the foregoing scenario three, when data is migrated from the oracle-type database to the oceanbase-type database, a third path verifier may be invoked to verify whether data source information corresponding to one or more read database operations is data source information of the oceanbase-type database, for example, when the data source information of the oceanbase-type database is "ObOracle", the obtained data source information may be compared with the "ObOracle", and if not, the obtained verification result information is information indicating that the verification is unsuccessful; otherwise, the obtained verification result information is information indicating successful verification. Of course, a database of predefined types of data source information connections can be further set by assigning a value to the variable "rule". When the oceanbase type database has a problem and needs to cut data back to the oracle type database, if the data source information corresponding to one or more read database operations needs to be checked whether is the data source information of the oracle type database, only the preset type of data source information needs to be modified.
Therefore, after data migration, only part of information in the information needs to be modified in the test script, the modified test script of the database access interface does not need to be debugged again, and the problem that in the prior art, when the data source information of the oracle type database is switched to the data source information of the oceanbase type database, the test script needs to be modified and debugged again, and therefore human resources are consumed is solved.
And step 250, outputting the verification result information.
The path verifier 13 may output verification result information indicating whether the verification is successful or not, and a developer may modify data source information corresponding to the database access operation according to the verification result information, so that the database access operation can access an expected database.
In summary, in the manner of combining the data source information verification method and the test method of the database access interface, only annotation information needs to be newly added to the test script of the database access interface, and the annotation information is analyzed by the data source information verification device to obtain a predefined verification algorithm, and then the data source information corresponding to the database access operation of the database access interface is verified according to the predetermined verification algorithm, and since the annotation information is not compiled, the test script after the annotation information is added does not need to be debugged again, so that the problems that in the prior art, when the data source information is verified, the verification logic needs to be newly added to the test script, and the workload is large when the test script after the newly added verification logic is debugged again can be avoided.
The data source information verification device can be embedded into a test script of an existing database access interface in a plug-in mode, and verification of data source information corresponding to database access operation on the database access interface is automatically completed by compiling annotation information for the database access interface on the basis of testing of the existing database access interface. Through the tangent plane thought of the interceptor, the verification of the data source information is increased at low cost.
Corresponding to the above method for verifying data source information, an embodiment of the present application further provides a device for verifying data source information, as shown in fig. 4, where the device includes: reading unit 401, parsing unit 402, intercepting unit 403, checking unit 404, and output unit 405.
The reading unit 401 is configured to, when a call request for a database access interface is received, read annotation information of the database access interface.
An analyzing unit 402, configured to analyze the annotation information read by the reading unit 401, so as to obtain a predefined checking algorithm.
Wherein the predefined verification algorithm comprises:
checking whether the information of the plurality of data sources is consistent; and/or the presence of a gas in the gas,
and checking whether the one or more data source information is consistent with preset data source information or predefined types of data source information.
The intercepting unit 403 is configured to intercept one or more database access operations of the database access interface, and obtain one or more data source information corresponding to the database access operations.
The intercepting unit 403 is specifically configured to:
intercepting, at a data access layer of the database access interface, one or more database access operations of the database access interface.
A checking unit 404, configured to check the data source information according to the predefined checking algorithm obtained by the analyzing unit 402, and obtain checking result information.
And an output unit 405, configured to output the verification result information obtained by the verification unit 404.
Optionally, the verification unit 404 may be specifically configured to:
comparing the one or more data source information, and if the comparison is consistent, obtaining the verification result information as information indicating successful verification; otherwise, the obtained verification result information is information indicating that verification is unsuccessful.
Optionally, the checking unit 404 may be further specifically configured to:
comparing the one or more pieces of data source information with preset data source information, and if any piece of data source information is inconsistent with the preset data information, obtaining verification result information which is information indicating that verification is unsuccessful; otherwise, the obtained verification result information is information representing successful verification; or,
comparing the one or more pieces of data source information with data source information of a predefined type, and if any piece of data source information is inconsistent with the data source information of the predefined type, obtaining verification result information which is information indicating that verification is unsuccessful; otherwise, the obtained verification result information is information indicating successful verification. The functions of the functional modules of the device in the embodiment of the present application may be implemented through the steps in the method embodiment described above, and therefore, the specific working process of the device provided in the present application is not repeated herein.
In the verification device for data source information provided by the application, when receiving a call request for a database access interface, a reading unit 401 reads annotation information of the database access interface; the analyzing unit 402 analyzes the annotation information to obtain a predefined checking algorithm; the intercepting unit 403 intercepts one or more database access operations of the database access interface, and obtains one or more data source information corresponding to the database access operations; the checking unit 404 checks the data source information according to the predefined checking algorithm, and obtains checking result information; the output unit 405 outputs the obtained verification result information. Therefore, automatic verification of data source information can be achieved, and therefore labor cost and time cost are greatly saved.
Those of skill would further appreciate that the various illustrative objects and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied in hardware, a software module executed by a processor, or a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The above-mentioned embodiments, objects, technical solutions and advantages of the present application are described in further detail, it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present application, and are not intended to limit the scope of the present application, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present application should be included in the scope of the present application.