CN110162544B - Heterogeneous data source data acquisition method and device - Google Patents
Heterogeneous data source data acquisition method and device Download PDFInfo
- Publication number
- CN110162544B CN110162544B CN201910465165.9A CN201910465165A CN110162544B CN 110162544 B CN110162544 B CN 110162544B CN 201910465165 A CN201910465165 A CN 201910465165A CN 110162544 B CN110162544 B CN 110162544B
- Authority
- CN
- China
- Prior art keywords
- query
- data
- information
- data table
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24558—Binary matching operations
- G06F16/2456—Join operations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a heterogeneous data source data acquisition method and a device, wherein the method comprises the following steps: receiving a cross-database data query request sent by a user; determining the associated query information of each data table across databases according to the data query request across databases; analyzing the associated query information, determining data table information in each related database, and generating corresponding query statements for the related data tables to obtain query results of the corresponding data tables; and according to the associated query information, associating the query results of each data table for secondary query to obtain the query results corresponding to the cross-database data query request. According to the invention, the data query across databases can be realized by the user, so that the user does not need to pay attention to the problems of different query languages, different data storage types and the like of different databases to obtain the query result.
Description
Technical Field
The invention relates to the field of databases, in particular to a heterogeneous data source data acquisition method and device.
Background
With the rapid development of services, the amount of services increases. When a business is executed, a large amount of data is generated, so that the amount of data stored in the database is increasingly huge. Sometimes, the data generated by different service scenarios are stored in different databases according to the different service scenarios. This allows the storage of business data in heterogeneous data sources. Heterogeneous data sources such as Explorer, Mysql, ODPS, HBase, etc. For a user needing to query data, the query may involve querying data from a plurality of different heterogeneous data sources, and different databases in the heterogeneous data sources have different query languages, different types of data stored in the databases, and the like, so that the user is difficult to query.
Disclosure of Invention
In view of the above, the present invention is proposed to provide a heterogeneous data source data acquisition method and apparatus that overcomes or at least partially solves the above problems.
According to an aspect of the present invention, a data obtaining method for a heterogeneous data source is provided, where the heterogeneous data source includes a plurality of databases of different types, and the method includes:
receiving a cross-database data query request sent by a user;
determining the associated query information of each data table across databases according to the data query request across databases;
analyzing the associated query information, determining data table information in each related database, and generating corresponding query statements for the related data tables to obtain query results of the corresponding data tables; and according to the associated query information, associating the query results of each data table for secondary query to obtain the query results corresponding to the cross-database data query request.
Optionally, after receiving a cross-database data query request sent by a user, the method further includes:
acquiring user inquiry authority information according to the user login information to determine inquiry limiting conditions corresponding to the user;
analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement for the related data table to obtain a query result of the corresponding data table, further comprising:
and analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement containing query limiting conditions for the related data table to obtain a query result of the corresponding data table.
Optionally, analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement for the related data table to obtain a query result of the corresponding data table further includes:
analyzing the associated query information, and determining data table information in each related database; the data table information comprises table names, fields and/or query conditions obtained through analysis;
generating a corresponding query statement according to the data table information;
and executing the query statement to obtain a query result of the corresponding data table.
Optionally, after obtaining the query result of the corresponding data table, the method further includes:
and converting the field format contained in the query result of the acquired data table into a preset specified format.
Optionally, the associating the query results of each data table to perform secondary query, and obtaining the query result corresponding to the cross-database data query request further includes:
analyzing the association query information to obtain association relation information of each data table across the database;
and generating query sentences which are associated with the query results of the data tables according to the association relation information so as to perform secondary query, thereby obtaining the query results corresponding to the cross-database data query request.
Optionally, after obtaining a query result corresponding to the cross-database data query request, the method further includes:
performing format processing on a query result corresponding to the cross-database data query request according to a result display format specified in the cross-database data query request;
and displaying the query result corresponding to the cross-database data query request after format processing.
Optionally, the method further comprises:
storing a query result corresponding to the cross-database data query request into a data file for a user to download; and the header information of the data file is set according to the cross-database data query request.
Optionally, the method further comprises:
and uploading the data file to a cloud server.
Optionally, the method further comprises:
when the following abnormal conditions occur in monitoring, recording abnormal information in a log, and performing alarm processing; wherein the abnormal condition comprises: determining the abnormity generated by the associated query information, analyzing the associated query information, acquiring the abnormity generated by the query result of the corresponding data table, performing secondary query on the query result associated with each data table, and/or storing the query result in a data file.
According to another aspect of the present invention, there is provided an apparatus for obtaining data from heterogeneous data sources, where the heterogeneous data sources include a plurality of databases of different types, the apparatus including:
the receiving module is suitable for receiving a cross-database data query request sent by a user;
the association module is suitable for determining association query information of each data table across the database according to the cross-database data query request;
the query module is suitable for analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement for the related data table to acquire a query result of the corresponding data table; and according to the associated query information, associating the query results of each data table for secondary query to obtain the query results corresponding to the cross-database data query request.
Optionally, the apparatus further comprises:
the authority module is suitable for acquiring user inquiry authority information according to the user login information so as to determine inquiry limiting conditions corresponding to the user;
the query module is further adapted to: and analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement containing query limiting conditions for the related data table to obtain a query result of the corresponding data table.
Optionally, the query module is further adapted to:
analyzing the associated query information, and determining data table information in each related database; the data table information comprises table names, fields and/or query conditions obtained through analysis;
generating a corresponding query statement according to the data table information;
and executing the query statement to obtain a query result of the corresponding data table.
Optionally, the apparatus further comprises:
and the conversion module is suitable for converting the field format contained in the acquired query result of the data table into a preset specified format.
Optionally, the query module is further adapted to:
analyzing the association query information to obtain association relation information of each data table across the database;
and generating query sentences which are associated with the query results of the data tables according to the association relation information so as to perform secondary query, thereby obtaining the query results corresponding to the cross-database data query request.
Optionally, the apparatus further comprises:
the display module is suitable for performing format processing on a query result corresponding to the cross-database data query request according to a result display format specified in the cross-database data query request; and displaying the query result corresponding to the cross-database data query request after format processing.
Optionally, the apparatus further comprises:
the downloading module is suitable for storing a query result corresponding to the cross-database data query request into a data file for a user to download; wherein, the header information of the data file is set according to the cross-database data query request
Optionally, the apparatus further comprises:
and the uploading module is suitable for uploading the data file to the cloud server.
Optionally, the apparatus further comprises:
the abnormal module is suitable for recording the abnormal information in the log and carrying out alarm processing when the following abnormal conditions occur; wherein the abnormal condition comprises: determining the abnormity generated by the associated query information, analyzing the associated query information, acquiring the abnormity generated by the query result of the corresponding data table, performing secondary query on the query result associated with each data table, and/or storing the query result in a data file.
According to still another aspect of the present invention, there is provided an electronic apparatus including: the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the heterogeneous data source data acquisition method.
According to still another aspect of the present invention, a computer storage medium is provided, where at least one executable instruction is stored in the storage medium, and the executable instruction causes a processor to perform operations corresponding to the above-mentioned heterogeneous data source data obtaining method.
According to the method and the device for acquiring the data of the heterogeneous data source, a cross-database data query request sent by a user is received; determining the associated query information of each data table across databases according to the data query request across databases; analyzing the associated query information, determining data table information in each related database, and generating corresponding query statements for the related data tables to obtain query results of the corresponding data tables; and according to the associated query information, associating the query results of each data table for secondary query to obtain the query results corresponding to the cross-database data query request. According to the invention, the data query across databases can be realized by the user, so that the user does not need to pay attention to the problems of different query languages, different data storage types and the like of different databases to obtain the query result.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 illustrates a flow diagram of a method for data acquisition from a heterogeneous data source according to one embodiment of the present invention;
FIG. 2 illustrates a flow diagram of a method for data acquisition from a disparate data source in accordance with another embodiment of the present invention;
FIG. 3 shows a functional block diagram of a heterogeneous data source data acquisition device according to one embodiment of the present invention;
fig. 4 shows a schematic structural diagram of an electronic device according to an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
FIG. 1 shows a flow diagram of a heterogeneous data source data acquisition method according to one embodiment of the invention. As shown in fig. 1, the method for obtaining data of a heterogeneous data source specifically includes the following steps:
step S101, receiving a cross-database data query request sent by a user.
When a user needs to perform cross-database query on a plurality of different types of databases, namely query on heterogeneous data sources, a cross-database data query request sent by the user is received. The query request comprises user-defined query conditions and information of a result to be queried.
In this embodiment, the heterogeneous data source includes a plurality of different types of databases, such as different types of databases like Explorer, Mysql, ODPS, and HBase. The above database types are examples, and the database in the heterogeneous data source is determined according to the implementation situation. The different types of databases have the problems of different database query statement formats, different types of data stored in the databases and the like. The cross-database data query request sent by the user needs to carry out related query on a plurality of different types of databases. According to the embodiment, the user does not need to pay attention to the problems, and the query result of the cross-database data query request can be directly obtained.
Step S102, determining the correlation query information of each data table across databases according to the cross-database data query request.
And analyzing the query conditions and the information of the result to be queried according to the received cross-database data query request, and analyzing a plurality of databases and data tables of the databases related to the query request. And if the XX query condition exists in the cross-database data query request, acquiring XX result information to be queried. And analyzing the database and the data table where the fields related in the query conditions are located, and analyzing the database and the data table where the fields related in the information of the result to be queried are located. Meanwhile, the database and the data table related to the query condition and the information of the result to be queried need to be analyzed to obtain the association relationship between the data tables. Specifically, the query association relationship between data tables across databases and the query association relationship between data tables in a single database need to be determined according to the business association relationship among the data tables in multiple databases. The service incidence relation existing in the data tables among the databases can be obtained through pre-established table relation data, and the table relation data is related to actual execution of the service.
Besides the databases and data tables involved in the query condition and the information of the result to be queried, other databases and data tables are sometimes involved. If the database and the data table related to the query condition are the A data table in the AA database, the database and the data table related to the information of the result to be queried are the B data table in the BB database, and the database and the data table do not have a direct association relationship, the A data table in the AA database and the B data table in the BB database need to be connected through other intermediate databases and data tables. Other databases and data tables need to find other intermediate databases and data tables which are respectively associated with the other databases and the data tables and other association relations between the other databases and the data tables by analyzing the service association relations among the fields in the A data table in the AA database, the fields in the B data table in the BB database and the data tables among the plurality of databases. The database and the data table may be one or more. Namely, the cross-database data query request is analyzed to obtain the query request, the related multiple databases and the data tables of the multiple databases. If there is no direct association relationship between the obtained data tables of the multiple databases, the data tables of the intermediate database, and the fields, query conditions, etc. required by the data tables of the intermediate database need to be determined according to the business association relationship existing between the data tables of the multiple databases.
Through the analysis of the cross-database data query request, the associated query information of each data table of the cross-database is obtained. Namely, the associated query information of each data table across the databases comprises a plurality of databases, the data tables of the databases, fields to be queried of each data table and query association relations among the data tables across the databases.
Step S103, analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement for the related data table to obtain a query result of the corresponding data table.
And analyzing the associated query information, and splitting the data table information in each related database. The data table information includes table names, fields, query conditions and the like obtained through analysis. And generating a query statement of the data table according to the data table information. If the data table information includes an explicit query condition and the field that needs to be obtained is, for example, the value of the field a1 in the a data table is 9, the fields b1 and c1 need to be obtained, and the corresponding query statement for generating the a data table queries the fields b1 and c1 from the a data table, and the query condition is A a 1-9. If the data table information does not contain an explicit query condition, the generated query statement acquires all the information of the data table, and the query statement may have no query condition or set the query condition to 1 or the like. The query field in the query statement is generated according to the data table information, or all fields of the data table can be queried when the field is not specified.
And generating a corresponding query statement for the data table according to the data table information, executing the query statement in the database to which the data table belongs, and acquiring a query result of the corresponding data table. When generating the query statement, the query statement is generated according to the query statement rule used by the database type to which the query statement belongs, and the formats of the query statement to be generated by the data tables of different database types are also different. And after generating a query statement according to the type of the database, executing the query statement to obtain a query result of the data table. The obtained query result of the data table can be stored in a memory or a designated address in advance, so that the subsequent use is facilitated. And generating query sentences for the data tables in the related databases, and acquiring and storing the query results of the data tables.
Further, the query result of the obtained data table has different data types due to different database types. After the query results of the data table are obtained, in order to facilitate subsequent use of the query results, the field formats contained in the query results need to be converted, and all the field formats are converted into preset specified formats. As converted by:
VARCHAR(“VARCHAR”，“VARCHAR”，String.class)；
LONGVARCHAR(“LONGVARCHAR”，“LONGVARCHAR”，String.class)；
CHAR(“CHAR”，“CHAR”，String.class)；
for example, during the conversion, data in character forms of CHAR, VARCHAR, LONGVARCHAR, TEXT, etc. are converted into uniform character string types, big byte data are converted into array types, data related to numerical values, such as INT, inter, tintint, etc., are converted into uniform numerical values, data types BIT, boolean, etc., are converted into uniform boolean, and data types, such as DATE, TIMESTAMP, etc., are converted into uniform DATE types, etc.
And step S104, associating the query results of each data table for secondary query according to the associated query information to obtain the query results corresponding to the cross-database data query request.
After the query result of each data table is obtained, each data table needs to be associated according to the associated query information. Specifically, the association query information needs to be analyzed to obtain association relationship information of each data table across the database, and then the query conditions of the interconnection of each data table are obtained. And according to the association relation information, taking the query result of each data table as a query sub-table, and generating a query statement associating the query result of each data table. And the query statement associates each query sub-table according to the query conditions, executes the query statement, namely performs secondary query, so as to obtain a query result corresponding to the cross-database data query request.
Furthermore, after the query result corresponding to the cross-database data query request is obtained, the query result needs to be displayed according to the cross-database data query request sent by the user, so that the user can conveniently check the query result. During display, format processing can be performed on the query result corresponding to the cross-database data query request according to a result display format specified in the cross-database data query request. If the data with the floating point type data type in the query result is 0.38, and the specified result display format is the percentage format, the format processing needs to be performed to obtain 38%. And then displaying the query result corresponding to the cross-database data query request after format processing to the user, so as to meet the requirements of the user.
According to the heterogeneous data source data acquisition method provided by the invention, a cross-database data query request sent by a user is received; determining the associated query information of each data table across databases according to the data query request across databases; analyzing the associated query information, determining data table information in each related database, and generating corresponding query statements for the related data tables to obtain query results of the corresponding data tables; and according to the associated query information, associating the query results of each data table for secondary query to obtain the query results corresponding to the cross-database data query request. According to the invention, the data query across databases can be realized by the user, so that the user does not need to pay attention to the problems of different query languages, different data storage types and the like of different databases to obtain the query result.
Fig. 2 shows a flowchart of a data acquisition method of a heterogeneous data source according to another embodiment of the present invention. As shown in fig. 2, the method for obtaining data of a heterogeneous data source specifically includes the following steps:
step S201, receiving a cross-database data query request sent by a user.
Step S202, determining the associated query information of each data table across the database according to the data query request across the database.
The above steps refer to the description of steps S101-S102 in the embodiment of fig. 1, and are not repeated herein.
Step S203, obtaining the user inquiry authority information according to the user login information to determine the inquiry limiting condition corresponding to the user.
When a user needs to perform data query across the database, the query authority problem of the user needs to be considered. The inquiry authority guarantees data safety, and the problems of data leakage and the like caused by the fact that a user inquires data outside the inquiry authority are avoided. The user inquiry authority information can ensure that the user only inquires data in the user inquiry authority when inquiring.
The inquiry authority information of the user is determined according to the user login information, such as the user id and the like. And determining the query limit condition corresponding to the user according to the query authority information of the user. If the cross-database data query request sent by the user relates to user information query, determining that the query limiting condition corresponding to the user is to query only user information of a user id, or to query only the user id and user information of subordinate users of the user id, and the like; when a cross-database data query request sent by a user relates to transaction information query, determining a query limiting condition corresponding to the user as only querying transaction information created by the user id, or only querying transaction information created by the user id and subordinate users of the user id, and the like. For example, the query limit condition corresponding to the specific query authority information is set according to the implementation situation.
Step S204, analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement containing query limiting conditions for the related data table to obtain a query result of the corresponding data table.
And analyzing the associated query information, splitting the data table information in each related database, and generating query sentences of each data table.
When the user query limitation condition relates to a certain data table and a query statement is generated, the user query limitation condition needs to be added into the generated query statement to avoid the queried data from exceeding the query authority of the user. And executing the generated query statement to obtain a query result of the data table corresponding to the query authority of the user.
And S205, associating the query results of each data table for secondary query according to the associated query information to obtain the query results corresponding to the cross-database data query request.
This step is described with reference to step S104 in the embodiment of fig. 1, and is not described herein again.
Further, when the query statement is generated, the query limit condition of the user may not be included, and when the query result associated with each data table is subjected to secondary query, the query limit condition of the user is included in the query condition of the secondary query, so that the finally obtained query result is guaranteed to be in line with the query authority of the user.
Step S206, storing the query result corresponding to the cross-database data query request into a data file for downloading by a user.
And writing the data contained in the query result into a data file for storage, such as an Excel file. When writing, data can be written item by item according to the query result. The specific write file can use the existing write file technology, and is not explained herein.
To facilitate the user's viewing, the header information in the data file may be set according to the cross-database data query request. For example, the name of each field of the result to be queried in the query request is used as the header information of the data file.
Further, the query result is stored in the data file, or the data file is generated according to the user requirement after the query result is displayed, and the data file is directly stored to the address specified by the user; the data file can be uploaded to the cloud server at first, and then the data file is stored to the user-specified address when the user needs to download the data file, so that the data file can be conveniently downloaded from the cloud server when the user downloads the data file for multiple times, and the operation of writing the data contained in the query result into the data file every time is reduced.
Further, in the execution process, the embodiment further includes an exception monitoring process. When the following abnormal conditions occur during monitoring, the abnormal information can be recorded in a log, and alarm processing is performed. The abnormal conditions include, for example, when the associated query information is determined, an abnormality that occurs when the associated query information cannot be created, an abnormality such as analysis that occurs when the associated query information is analyzed, an abnormality that occurs when a query result of a corresponding data table is obtained, an abnormality that occurs when an abnormality is obtained, an abnormality that occurs when a query result associated with each data table is subjected to secondary query, various abnormalities that occur when a query result is stored in a data file, and an abnormality such as writing and storing that occurs when a query result is stored in a data file. When the abnormity occurs, the generated abnormal information is recorded in a log file, and the corresponding alarm processing is carried out. The alarm processing includes e.g. mail alarm, short message alarm, etc. so as to timely and accurately process the abnormal condition.
According to the data acquisition method of the heterogeneous data source, the data is queried according to the query authority of the user during query, and data leakage is avoided. Meanwhile, when the query result is obtained, the query result is written into the data file, so that the user can conveniently download the data of the heterogeneous data source.
Fig. 3 shows a functional block diagram of a heterogeneous data source data acquisition device according to an embodiment of the present invention. As shown in fig. 3, the heterogeneous data source data acquiring apparatus includes the following modules:
the receiving module 310 is adapted to: and receiving a cross-database data query request sent by a user.
The association module 320 is adapted to: and determining the associated query information of each data table across the database according to the data query request across the database.
The query module 330 is adapted to: analyzing the associated query information, determining data table information in each related database, and generating corresponding query statements for the related data tables to obtain query results of the corresponding data tables; and according to the associated query information, associating the query results of each data table for secondary query to obtain the query results corresponding to the cross-database data query request.
Optionally, the apparatus further comprises: a rights module 340.
The rights module is adapted to: and acquiring user inquiry authority information according to the user login information so as to determine inquiry limiting conditions corresponding to the user.
The query module 330 is further adapted to: and analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement containing query limiting conditions for the related data table to obtain a query result of the corresponding data table.
Optionally, the query module 330 is further adapted to:
analyzing the associated query information, and determining data table information in each related database; the data table information comprises table names, fields and/or query conditions obtained through analysis; generating a corresponding query statement according to the data table information; and executing the query statement to obtain a query result of the corresponding data table.
Optionally, the apparatus further comprises: a conversion module 350.
The conversion module 350 is adapted to: and converting the field format contained in the query result of the acquired data table into a preset specified format.
Optionally, the query module 330 is further adapted to: analyzing the association query information to obtain association relation information of each data table across the database; and generating query sentences which are associated with the query results of the data tables according to the association relation information so as to perform secondary query, thereby obtaining the query results corresponding to the cross-database data query request.
Optionally, the apparatus further comprises: the module 360 is shown.
The display module 360 is adapted to: performing format processing on a query result corresponding to the cross-database data query request according to a result display format specified in the cross-database data query request; and displaying the query result corresponding to the cross-database data query request after format processing.
Optionally, the apparatus further comprises: a download module 370.
The download module 370 is adapted to: storing a query result corresponding to the cross-database data query request into a data file for a user to download; wherein, the header information of the data file is set according to the cross-database data query request
Optionally, the apparatus further comprises: an upload module 380.
The upload module 380 is adapted to: and uploading the data file to a cloud server.
Optionally, the apparatus further comprises: an exception module 390.
The exception module 390 is adapted to: when the following abnormal conditions occur in the monitoring, recording the abnormal information in a log, and performing alarm processing; wherein the abnormal condition comprises: determining the abnormity generated by the associated query information, analyzing the associated query information, acquiring the abnormity generated by the query result of the corresponding data table, performing secondary query on the query result associated with each data table, and/or storing the query result in a data file.
The descriptions of the modules refer to the corresponding descriptions in the method embodiments, and are not repeated herein.
The application also provides a non-volatile computer storage medium, where at least one executable instruction is stored in the computer storage medium, and the computer executable instruction can execute the data acquisition method of the heterogeneous data source in any method embodiment.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the electronic device.
As shown in fig. 4, the electronic device may include: a processor (processor)402, a Communications Interface 404, a memory 406, and a Communications bus 408.
Wherein:
the processor 402, communication interface 404, and memory 406 communicate with each other via a communication bus 408.
A communication interface 404 for communicating with network elements of other devices, such as clients or other servers.
The processor 402 is configured to execute the program 410, and may specifically perform relevant steps in the above-described data acquisition method for heterogeneous data sources.
In particular, program 410 may include program code comprising computer operating instructions.
The processor 402 may be a central processing unit CPU or an application Specific Integrated circuit asic or one or more Integrated circuits configured to implement embodiments of the present invention. The electronic device comprises one or more processors, which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 406 for storing a program 410. Memory 406 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 410 may be specifically configured to enable the processor 402 to execute the data obtaining method of the heterogeneous data source in any of the method embodiments. For specific implementation of each step in the program 410, reference may be made to corresponding steps and corresponding descriptions in units in the above-mentioned data acquisition embodiment of the heterogeneous data source, which are not described herein again. It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described devices and modules may refer to the corresponding process descriptions in the foregoing method embodiments, and are not described herein again.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose systems may also be used with the teachings herein. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: rather, the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in a heterogeneous data source data acquisition device according to embodiments of the present invention. The present invention may also be embodied as apparatus or device programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
Claims (20)
1. A data acquisition method for a heterogeneous data source is provided, wherein the heterogeneous data source comprises a plurality of databases of different types, and the method comprises the following steps:
receiving a cross-database data query request sent by a user; wherein the cross-database data query request carries out correlation query on a plurality of different types of databases; the query request comprises query conditions defined by a user and result information to be queried;
analyzing the query conditions and the information of the result to be queried according to the cross-database data query request to obtain a plurality of databases and/or data tables of the plurality of databases; determining the association query information of each data table across databases according to the business association relationship among the data tables among the databases;
analyzing the associated query information, determining data table information in each related database, and generating corresponding query statements for the related data tables to obtain query results of the corresponding data tables; and associating the query results of each data table for secondary query according to the associated query information to obtain the query result corresponding to the cross-database data query request.
2. The method of claim 1, wherein after receiving a cross-database data query request sent by a user, the method further comprises:
acquiring user inquiry authority information according to the user login information to determine inquiry limiting conditions corresponding to the user;
the analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement for the related data table to obtain a query result of the corresponding data table further includes:
and analyzing the associated query information, determining data table information in each related database, and generating a corresponding query statement containing the query limiting condition for the related data table to obtain a query result of the corresponding data table.
3. The method of claim 1, wherein the parsing the associated query information, determining data table information in each database involved, and generating a corresponding query statement for the data table involved to obtain a query result of the corresponding data table further comprises:
analyzing the associated query information, and determining data table information in each related database; the data table information comprises table names, fields and/or query conditions obtained through analysis;
generating a corresponding query statement according to the data table information;
and executing the query statement to obtain a query result of the corresponding data table.
4. The method of any of claims 1-3, wherein after the obtaining of the query result for the corresponding data table, the method further comprises:
and converting the field format contained in the query result of the acquired data table into a preset specified format.
5. The method according to claim 1, wherein the performing a secondary query on the query results associated with each data table according to the associated query information to obtain the query result corresponding to the cross-database data query request further comprises:
analyzing the association query information to obtain association relation information of each data table across databases;
and generating query statements related to the query results of each data table according to the association relation information so as to perform secondary query, thereby obtaining the query results corresponding to the cross-database data query request.
6. The method of any of claims 1-3, wherein after the obtaining of the query result corresponding to the cross-database data query request, the method further comprises:
performing format processing on a query result corresponding to the cross-database data query request according to a result display format specified in the cross-database data query request;
and displaying the query result corresponding to the cross-database data query request after format processing.
7. The method according to any one of claims 1-3, wherein the method further comprises:
storing a query result corresponding to the cross-database data query request into a data file for a user to download; and setting the header information of the data file according to the cross-database data query request.
8. The method of claim 7, wherein the method further comprises:
and uploading the data file to a cloud server.
9. The method according to any one of claims 1-3, wherein the method further comprises:
when the following abnormal conditions occur in monitoring, recording abnormal information in a log, and performing alarm processing; wherein the abnormal condition comprises: determining the abnormity generated by the associated query information, analyzing the associated query information, acquiring the abnormity generated by the query result of the corresponding data table, performing secondary query on the query result associated with each data table, and/or storing the query result in a data file.
10. A data acquisition device for heterogeneous data sources, wherein the heterogeneous data sources comprise a plurality of databases of different types, the device comprises:
the receiving module is suitable for receiving a cross-database data query request sent by a user; wherein the cross-database data query request carries out correlation query on a plurality of different types of databases; the query request comprises query conditions defined by a user and result information to be queried;
the association module is suitable for analyzing the query conditions and the information of the result to be queried according to the cross-database data query request to obtain a plurality of databases and/or data tables of the plurality of databases; determining the association query information of each data table across databases according to the business association relationship among the data tables among the databases;
the query module is suitable for analyzing the associated query information, determining the data table information in each related database, and generating a corresponding query statement for the related data table to obtain a query result of the corresponding data table; and associating the query results of each data table for secondary query according to the associated query information to obtain the query result corresponding to the cross-database data query request.
11. The apparatus of claim 10, wherein the apparatus further comprises:
the authority module is suitable for acquiring user inquiry authority information according to the user login information so as to determine inquiry limiting conditions corresponding to the user;
the query module is further adapted to: and analyzing the associated query information, determining data table information in each related database, and generating a corresponding query statement containing the query limiting condition for the related data table to obtain a query result of the corresponding data table.
12. The apparatus of claim 10, wherein the query module is further adapted to:
analyzing the associated query information, and determining data table information in each related database; the data table information comprises table names, fields and/or query conditions obtained through analysis;
generating a corresponding query statement according to the data table information;
and executing the query statement to obtain a query result of the corresponding data table.
13. The apparatus of any of claims 10-12, wherein the apparatus further comprises:
and the conversion module is suitable for converting the field format contained in the acquired query result of the data table into a preset specified format.
14. The apparatus of claim 10, wherein the query module is further adapted to:
analyzing the association query information to obtain association relation information of each data table across databases;
and generating query statements for associating the query results of each data table according to the association relation information so as to perform secondary query to obtain the query results corresponding to the cross-database data query request.
15. The apparatus of any of claims 10-12, wherein the apparatus further comprises:
the display module is suitable for carrying out format processing on the query result corresponding to the cross-database data query request according to a result display format specified in the cross-database data query request; and displaying the query result corresponding to the cross-database data query request after format processing.
16. The apparatus of any of claims 10-12, wherein the apparatus further comprises:
the downloading module is suitable for storing the query result corresponding to the cross-database data query request into a data file for a user to download; and setting the header information of the data file according to the cross-database data query request.
17. The apparatus of claim 16, wherein the apparatus further comprises:
and the uploading module is suitable for uploading the data file to a cloud server.
18. The apparatus of any of claims 10-12, wherein the apparatus further comprises:
the abnormal module is suitable for recording the abnormal information in the log and carrying out alarm processing when the following abnormal conditions occur; wherein the abnormal condition comprises: determining the abnormity generated by the associated query information, analyzing the associated query information, acquiring the abnormity generated by the query result of the corresponding data table, performing secondary query on the query result associated with each data table, and/or storing the query result in a data file.
19. An electronic device, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the heterogeneous data source data acquisition method of any one of claims 1-9.
20. A computer storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the heterogeneous data source data acquisition method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910465165.9A CN110162544B (en) | 2019-05-30 | 2019-05-30 | Heterogeneous data source data acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910465165.9A CN110162544B (en) | 2019-05-30 | 2019-05-30 | Heterogeneous data source data acquisition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110162544A CN110162544A (en) | 2019-08-23 |
| CN110162544B true CN110162544B (en) | 2022-05-27 |
Family
ID=67630634
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910465165.9A Active CN110162544B (en) | 2019-05-30 | 2019-05-30 | Heterogeneous data source data acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110162544B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111104426B8 (en) * | 2019-11-22 | 2024-04-23 | 北京傲速科技有限公司 | Data query method and system |
| CN110955662A (en) * | 2019-11-29 | 2020-04-03 | 车智互联（北京）科技有限公司 | Method, computing device and storage medium for maintaining data table association relation |
| CN111177213B (en) * | 2019-12-16 | 2024-04-19 | 北京淇瑀信息科技有限公司 | Privacy cluster self-service query platform, method and electronic equipment |
| WO2021129498A1 (en) * | 2019-12-24 | 2021-07-01 | 阿里巴巴集团控股有限公司 | Data processing method and apparatus for distributed query system |
| CN111259036B (en) * | 2020-01-10 | 2022-10-11 | 苏州达家迎信息技术有限公司 | Cross-library and cross-table query method, device, server and storage medium |
| CN111259038B (en) * | 2020-01-16 | 2023-05-30 | 北京思特奇信息技术股份有限公司 | Database query and data export method, system, medium and device |
| CN111581231A (en) * | 2020-04-20 | 2020-08-25 | 北京明略软件系统有限公司 | Query method and device based on heterogeneous database |
| CN113946594B (en) * | 2021-12-22 | 2022-07-12 | 昆仑智汇数据科技(北京)有限公司 | Integrated query method, device and equipment for industrial multi-source heterogeneous data |
| CN114490241B (en) * | 2021-12-25 | 2023-09-15 | 苏州浪潮智能科技有限公司 | Chip monitoring method, system, storage medium and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104573115A (en) * | 2015-02-04 | 2015-04-29 | 新余兴邦信息产业有限公司 | Method and system for achieving integration interface supporting operation of multi-type databases |
| CN105224613A (en) * | 2015-09-17 | 2016-01-06 | 西安未来国际信息股份有限公司 | Based on integrating heterogeneous data source system and the integration method thereof of the federal technology of data |
| CN106372177A (en) * | 2016-08-30 | 2017-02-01 | 东华大学 | Query expansion method supporting correlated query and fuzzy grouping of mixed data type |
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
| CN107491510A (en) * | 2017-08-03 | 2017-12-19 | 国网江苏省电力公司信息通信分公司 | One kind mixing heterogeneous data source unified query system and distributed enquiring method |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120036146A1 (en) * | 2010-10-26 | 2012-02-09 | ParElastic Corporation | Apparatus for elastic database processing with heterogeneous data |
| CN104239320B (en) * | 2013-06-14 | 2017-09-19 | 深圳中兴网信科技有限公司 | A kind of data merging method and system |
| US10139997B2 (en) * | 2014-10-05 | 2018-11-27 | Splunk Inc. | Statistics time chart interface cell mode drill down |
| US10191946B2 (en) * | 2015-03-11 | 2019-01-29 | International Business Machines Corporation | Answering natural language table queries through semantic table representation |
| CN108804460A (en) * | 2017-05-03 | 2018-11-13 | 北京润乾信息系统技术有限公司 | A kind of query language based on SQL |
| CN108509637A (en) * | 2018-04-10 | 2018-09-07 | 口碑(上海)信息技术有限公司 | Tables of data relation query method and device |
-
2019
- 2019-05-30 CN CN201910465165.9A patent/CN110162544B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104573115A (en) * | 2015-02-04 | 2015-04-29 | 新余兴邦信息产业有限公司 | Method and system for achieving integration interface supporting operation of multi-type databases |
| CN105224613A (en) * | 2015-09-17 | 2016-01-06 | 西安未来国际信息股份有限公司 | Based on integrating heterogeneous data source system and the integration method thereof of the federal technology of data |
| CN106372177A (en) * | 2016-08-30 | 2017-02-01 | 东华大学 | Query expansion method supporting correlated query and fuzzy grouping of mixed data type |
| CN107066499A (en) * | 2016-12-30 | 2017-08-18 | 江苏瑞中数据股份有限公司 | The data query method of multi-source data management and visualization system is stored towards isomery |
| CN107491510A (en) * | 2017-08-03 | 2017-12-19 | 国网江苏省电力公司信息通信分公司 | One kind mixing heterogeneous data source unified query system and distributed enquiring method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110162544A (en) | 2019-08-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110162544B (en) | Heterogeneous data source data acquisition method and device | |
| CN108519967B (en) | Chart visualization method and device, terminal and storage medium | |
| WO2021189954A1 (en) | Log data processing method and apparatus, computer device, and storage medium | |
| CN106682097B (en) | Method and device for processing log data | |
| CN111078140B (en) | Nuclear power station file uploading management method and device, terminal equipment and medium | |
| US20140114822A1 (en) | Method and system for creating tax configuration templates | |
| CN109284323B (en) | Management method and device for detection data | |
| CN110737689B (en) | Data standard compliance detection method, device, system and storage medium | |
| CN110673839B (en) | Distributed tool configuration construction generation method and system | |
| CN111177113B (en) | Data migration method, device, computer equipment and storage medium | |
| CN111522728A (en) | Method for generating automatic test case, electronic device and readable storage medium | |
| CN108460068B (en) | Method, device, storage medium and terminal for importing and exporting report | |
| US9087137B2 (en) | Detection of custom parameters in a request URL | |
| CN112860730A (en) | SQL statement processing method and device, electronic equipment and readable storage medium | |
| CN110955674A (en) | Asynchronous export method and component based on java service | |
| CN112667733A (en) | Data warehouse data importing method and system | |
| CN116450890A (en) | Graph data processing method, device and system, electronic equipment and storage medium | |
| CN113934733A (en) | Problem positioning method, device, system, storage medium and electronic equipment | |
| CN110188083B (en) | Interface information mining method and device | |
| CN112883088B (en) | Data processing method, device, equipment and storage medium | |
| CN117493309A (en) | Standard model generation method, device, equipment and storage medium | |
| CN112187509A (en) | Multi-architecture cloud platform execution log management method, system, terminal and storage medium | |
| CN111752916A (en) | Data acquisition method and device, computer readable storage medium and electronic equipment | |
| CN112491943A (en) | Data request method, device, storage medium and electronic equipment | |
| CN111045983B (en) | Nuclear power station electronic file management method, device, terminal equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |