CN114490729A

CN114490729A - Data acquisition method, device and equipment based on multiple databases

Info

Publication number: CN114490729A
Application number: CN202210109110.6A
Authority: CN
Inventors: 赵飞
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-01-28
Filing date: 2022-01-28
Publication date: 2022-05-13

Abstract

The disclosure provides a data acquisition method, a data acquisition device and data acquisition equipment based on multiple databases, and relates to the field of data processing, in particular to the field of data security. The specific implementation scheme is as follows: acquiring a data acquisition request, wherein the data acquisition request is used for indicating a database to be queried; acquiring data requested by a data acquisition request from a database to be queried according to configuration information corresponding to the database to be queried, wherein the configuration information is used for indicating a data sampling mode when the data is acquired from the database; the retrieved data is stored, wherein the retrieved data is used to test the code program.

Description

Data acquisition method, device and equipment based on multiple databases

Technical Field

The present disclosure relates to the field of data security in the field of data processing, and in particular, to a method, an apparatus, and a device for acquiring data based on multiple databases.

Background

As computer technology has developed, developers need to develop different software platforms, such as platforms, applications, applets, etc., based on different requirements. Before the software platform is put into use, the software platform needs to be tested according to the test data, namely, a code program of the software platform needs to be tested according to the test data.

In the prior art, test data meeting the requirements of software platforms need to be provided according to the requirements of different software platforms; and then testing the software platform according to the test data. Moreover, for different functions of the same software platform, different functions of the software platform may also need to be tested according to different test data of the different functions. In the above process, the developer is often required to provide the test data by himself.

However, in the above manner, if the developer is replaced, the new developer does not know the software platform enough, and the written test data does not meet the test requirement of the software platform. A large amount of time is consumed, and the development efficiency of the software platform is reduced; and the written test data can be inaccurate, the test on the software platform is inaccurate, and further the software platform can not be normally put into use.

Disclosure of Invention

The disclosure provides a data acquisition method, a device and equipment based on multiple databases.

According to a first aspect of the present disclosure, a method for acquiring data based on multiple databases is provided, including:

acquiring a data acquisition request, wherein the data acquisition request is used for indicating a database to be queried;

acquiring data requested by the data acquisition request from the database to be inquired according to configuration information corresponding to the database to be inquired, wherein the configuration information is used for indicating a data sampling mode when the data is acquired from the database;

the retrieved data is stored, wherein the retrieved data is used to test the code program.

According to a second aspect of the present disclosure, there is provided a data acquisition apparatus based on multiple databases, comprising:

the device comprises a first acquisition unit, a second acquisition unit and a query unit, wherein the first acquisition unit is used for acquiring a data acquisition request, and the data acquisition request is used for indicating a database to be queried;

a second obtaining unit, configured to obtain, from the database to be queried, data requested by the data obtaining request according to configuration information corresponding to the database to be queried, where the configuration information is used to indicate a data sampling manner when obtaining the data from the database;

and the storage unit is used for storing the acquired data, wherein the acquired data is used for testing the code program.

According to a third aspect of the present disclosure, there is provided an electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of the first aspect.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of the first aspect.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising: a computer program, stored in a readable storage medium, from which at least one processor of an electronic device can read the computer program, execution of the computer program by the at least one processor causing the electronic device to perform the method of the first aspect.

According to the technology disclosed by the invention, the development efficiency of the software platform is improved, the accuracy of the software platform test is improved, and the software platform can be ensured to be normally put into use.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure;

FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure;

FIG. 3 is a schematic illustration according to a third embodiment of the present disclosure;

FIG. 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

FIG. 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

fig. 6 is a block diagram of an electronic device for implementing a method for data acquisition based on multiple databases according to an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of embodiments of the present disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

The invention provides a data acquisition method, a data acquisition device and data acquisition equipment based on multiple databases, which are applied to the field of data security in the field of data processing so as to improve the development efficiency of a software platform, improve the accuracy of software platform testing and further ensure the purpose that the software platform can be normally put into use.

Fig. 1 is a schematic diagram according to a first embodiment of the present disclosure, and as shown in fig. 1, fig. 1 illustrates a method for acquiring data based on multiple databases, the method including:

s101, a data acquisition request is acquired, wherein the data acquisition request is used for indicating a database to be inquired.

In this embodiment, the acquired data acquisition request may be performed by a current data acquisition platform, and specifically, the data acquisition request may be one or multiple data acquisition requests. Wherein the data acquisition request is different for different query-with-query databases. Further, the database to be queried may be mySQL or PGSQL database in a relational database management system, where mySQL is a relational database management system that stores data in different tables, rather than putting all data in a large warehouse, which increases access speed and improves query flexibility.

Further, PGSQL provides multi-version concurrency control, supporting almost all SQL components. In particular, sub-queries, transactions, and user-defined types and functions in the SQL construct are included, and a very broad range of development languages are available, including C, C + +, Java, perl, tcl, and python. In one example, the number of databases to be queried may be one or more.

S102, acquiring data requested by the data acquisition request from the database to be inquired according to configuration information corresponding to the database to be inquired, wherein the configuration information is used for indicating a data sampling mode when the data is acquired from the database.

In one example, the configuration information is information of a data sampling mode configured on a current data acquisition platform, where the configuration information corresponding to different databases to be queried is different. And the identification of the configuration information of different databases to be queried is different. For example, the identifier corresponding to the configuration information of the database a to be queried is the configuration information a, and the identifier corresponding to the configuration information of the database B to be queried is the configuration information B. When it is determined to query the database a to be queried, the configuration information a of the database a to be queried may be invoked.

In one example, the configuration information includes a plurality of data sampling modes, and each data sampling mode is different in the number and the form of data sampled in the database to be queried.

And S103, storing the acquired data, wherein the acquired data is used for testing the code program.

In one example, after data sampled from a database to be queried is stored, the stored data is tested for a code program, and then a bug in the code program is detected, so that the detected code program can run normally.

The disclosure provides a data acquisition method based on multiple databases, comprising: acquiring a data acquisition request, wherein the data acquisition request is used for indicating a database to be queried; acquiring data requested by a data acquisition request from a database to be queried according to configuration information corresponding to the database to be queried, wherein the configuration information is used for indicating a data sampling mode when the data is acquired from the database; the retrieved data is stored, wherein the retrieved data is used to test the code program. By adopting the technical scheme, the development efficiency of the software platform can be improved, the accuracy of the software platform test is improved, and the software platform can be ensured to be normally put into use.

Fig. 2 is a schematic diagram according to a second embodiment of the present disclosure, and as shown in fig. 2, fig. 2 illustrates a method for acquiring data based on multiple databases, the method including:

s201, acquiring a data acquisition request, wherein the data acquisition request is used for indicating a database to be queried.

For example, this step may refer to step S101 described above, and is not described again.

S202, acquiring data requested by the data acquisition request from the database to be inquired according to configuration information corresponding to the database to be inquired, wherein the configuration information is used for indicating a data sampling mode when the data is acquired from the database.

In one example, the configuration information includes: sampling frequency, and/or data derivation; wherein the sampling frequency is indicative of a frequency at which data is sampled from the database; data derivation means is used to indicate the way data is derived from the database.

In one example, the sampling frequency may be one time of sampling data from the database, and may be a plurality of times of sampling data from the database at preset intervals. For example, the sampling frequency may be that data is sampled from the database every 3 hours, or may be that data is sampled from the database every two days.

Specifically, the sampling frequency may be adjusted according to the use condition of the database to be queried by the user and the characteristics of the database to be queried.

In one example, the data derivation means a specific form of deriving data from a database, and may be preset.

In this embodiment, the configuration information may include three types: the first configuration information includes only sampling frequency; the second configuration information only comprises a data export mode; the third configuration information includes two types of sampling frequency and data derivation mode. The method has the advantages that different configuration information is configured according to different requirements of the user on the database to be queried, and further more user requirements can be met.

In an optional example, the configuration information includes a sampling frequency; acquiring data requested by a data acquisition request from a database to be queried according to configuration information corresponding to the database to be queried, wherein the data acquisition request comprises the following steps:

and acquiring the data requested by the data acquisition request from the database to be queried at intervals according to the sampling frequency in the configuration information.

In this embodiment, data information of a sampling frequency in the configuration information is obtained, and data sampling is performed on a database to be queried corresponding to the configuration information according to the sampling frequency. For example, the configuration information a corresponds to the database a to be queried, and if the sampling frequency in the configuration information a is sampling only once, the data of the database a to be queried may be sampled only once. The advantage of this arrangement is that data can be acquired more intelligently according to the preset sampling frequency without manually deriving the data from the database.

In an optional example, the configuration information includes a data export manner; acquiring data requested by a data acquisition request from a database to be queried according to configuration information corresponding to the database to be queried, wherein the data acquisition request comprises the following steps:

and if the data export mode in the configuration information indicates the structural relationship of the export data, acquiring the data requested by the data acquisition request and the structural relationship between the data and other data from the database to be queried.

In an example, the data in the database to be queried is obtained according to a data export manner in the configuration information, and if the data export manner is a structural relationship of the export data, the structural relationship of the data found from the database to be queried may be a linear structural relationship, a tree structural relationship, or a graph structural relationship. The benefits of this arrangement are: in terms of data safety or efficiency, only the minimum data volume used for completing the task can be provided, and the whole table and the whole library are not exported, so that the risk of exposing data is reduced.

In one example, if it is determined that the data export manner in the configuration information indicates a data table to which the export data belongs, acquiring the data requested by the data acquisition request and the data table to which the data belongs from the database to be queried; the data table includes data and other data having an association relationship with the data.

In this embodiment, if the data export manner is to export a data table to which the data belongs, the data is queried in the database to be queried according to the data in the data acquisition request, and after the data table to which the data belongs is queried, other data having an association relationship with the data is also exported. The benefits of this arrangement are: and can provide more comprehensive data information under the condition of user requirement.

In an optional example, obtaining, from a database to be queried, data requested by a data obtaining request and a data table to which the data belongs includes:

determining other fields having an association relation with the fields of the data representation requested by the data acquisition request;

acquiring data requested by a data acquisition request from a database to be queried; and determining data under other fields from the database to be queried to obtain a data table to which the data belongs.

In this embodiment, a field is first obtained in a data obtaining request, data under the field is obtained, and then a data table to which the data belongs is determined according to the data, and the data table is exported. For example, if the field is a, and the data table to which the data in the database a to be queried belongs in the field a is Y, the data table Y is exported. Further, if the other field associated with the field a is a field b, determining that the data table to which the data of the field b in the database a to be queried belongs is a data table X, and exporting the data table X. In this embodiment, the data table may be exported from the database in a table manner, and in the process of exporting the data table, the data table needs to be checked to avoid repeated exporting of the data table, and in the process of exporting the data table, the latest N data tables may be exported, or the data table may be exported in full quantity. The export can also be carried out according to the SQL mode, the export of a single data table can be limited, the export of a limited number of data tables can also be carried out, and the AS script file is not allowed to appear in the export process. The advantage of this arrangement is that it can ensure that the user's data information is more fully captured, but without exposing a large amount of data information.

In this embodiment, the depth of the data table obtained from the database may also be preset, where the depth of the data table may be set as a preset value, or may also be set as a maximum N layers.

Further, if the data sampled from the database contains a sensitive field or sensitive content, the sensitive field or sensitive content may be left as it is.

In an alternative example, determining other fields having an association relationship with a field of the data representation requested by the data acquisition request includes:

determining other fields having incidence relation with the fields of the data representation requested by the data acquisition request according to the keywords represented by the data requested by the data acquisition request;

or determining other fields having association relation with the fields of the data representation requested by the data acquisition request according to the mapping relation between the preset fields.

In one example, after a keyword of data in the data acquisition request is acquired, a field is determined according to the keyword, and then other fields having an association relationship with the field are determined according to the field. Further, according to the fields in the data acquisition request, determining other fields having a mapping relation or an association relation with the fields in the mapping relation among the preset fields. The advantage of this is that the data in the data acquisition request can be more comprehensively exported.

S203, compressing the acquired data and storing the compressed data to the local; or storing the acquired data into a repository corresponding to the database to be queried, wherein the databases are in one-to-one correspondence with the repositories. Wherein the acquired data is used to test the code program.

In this embodiment, the obtained data is stored, and a specific storage manner may be to compress the obtained data and then store the compressed data in a sampling data platform; the acquired data may also be stored in a storage library, for example, the acquired data is compressed into a zip form and then stored in a sampling data platform; after the data is acquired from the database a, the acquired data may be stored in the repository corresponding to the database a, where the name of the repository corresponding to the database a is the same as the name of the database a. Further, the prefix of the database name of the database to be queried is readonly _ and the prefix of the repository name is sample _. The advantage of this arrangement is that the result of the code program for data test in the repository is the result of the data in the database to be queried corresponding to the repository.

In this embodiment, the manner of storing the acquired data in the repository includes: REPLACE INTO, INSERT IGNORE, and INSERT INTO. Wherein REPLACE INTO and INSERT IGNORE perform special processing in the database of PGSQL.

The present disclosure provides a data acquisition method based on multiple databases, including: acquiring a data acquisition request, wherein the data acquisition request is used for indicating a database to be queried; acquiring data requested by a data acquisition request from a database to be queried according to configuration information corresponding to the database to be queried, wherein the configuration information is used for indicating a data sampling mode when the data is acquired from the database; compressing the acquired data, and storing the compressed data to the local; or storing the acquired data into a repository corresponding to the database to be queried, wherein the databases are in one-to-one correspondence with the repositories. Wherein the acquired data is used to test the code program. By adopting the technical scheme, the data use scene can be traced and the data use scene is simple and convenient to use, the data is reasonably applied to the development or troubleshooting process, and the engineering efficiency in the development process is greatly and effectively improved.

Fig. 3 is a schematic diagram according to a third embodiment of the present disclosure, and as shown in fig. 3, fig. 3 illustrates a method for acquiring data based on multiple databases, the method including:

s301, a data acquisition request is acquired, wherein the data acquisition request is used for indicating a database to be inquired.

In one example, a get data get request includes: responding to trigger information of a user on an operation interface, wherein the trigger information is used for indicating a database to be queried and requested data, and generating a data acquisition request; the trigger information is used for indicating that the sampling frequency in the configuration information is one time.

In this embodiment, the data acquisition request may be triggered by a user on an operation interface of the sample data platform, and trigger information is generated after the user triggers, where the content of the trigger information may include a database to be queried and requested data, and further, a sampling frequency in configuration information in the trigger information triggered by the user on the operation interface is once. The advantage of this arrangement is that when the frequency of use of the requested data is low, multiple fetches are not required, and memory space in the repository can be saved.

In one example, wherein obtaining the data acquisition request comprises:

responding to a preset data acquisition task, wherein the data acquisition task is used for indicating a database to be queried and requested data, and generating a data acquisition request;

the data acquisition task is used for indicating that the sampling frequency in the configuration information is multiple times.

In this embodiment, if the sampling frequency in the configuration information is multiple times, in this case, it is not necessary for the user to trigger and generate a trigger message on the operation interface, but a data acquisition request is generated by the data acquisition task. The data acquisition task comprises the name of a database to be inquired and a task list of the current data acquisition task, wherein the task list comprises a data task state, data task details, the starting time of the data task execution, the ending time of the data task execution, the number of data tables derived from the data task, and whether the type derived from the data task is structural relationship data or a data table. The advantage of setting up like this is, under the higher condition of data frequency of use of data acquisition request, can record the sampling process of the data requested, and then can guarantee the security of data, guarantee the traceability of data sampling simultaneously.

Further, if the acquired data is compressed and then the compressed data is stored in the sampling data platform, the data acquisition task further includes a list for downloading the compressed data in the sampling data platform. In this embodiment, the data task state includes an unexecuted state, an executed state, an executing state, and a termination state. The data task details are divided according to a data table, and specifically include: the number of successful entries in the data table, the number of failed entries in the data table, the number of skipped entries in the data table, the execution status of the data table, the start ID in the data table, and the end ID in the data table.

The data acquisition request comprises a user identifier; the method further comprises the following steps:

determining the user authority corresponding to the user identifier in the data acquisition request according to the corresponding relation between the preset user identifier and the user authority; wherein the user right characterizes a database accessible by the user;

and if the database to be queried indicated by the data acquisition request is determined to exist in the database represented by the user right corresponding to the user identifier in the data acquisition request, executing a step of acquiring the data requested by the data acquisition request from the database to be queried according to the configuration information corresponding to the database to be queried.

In one example, the data acquisition request may include different user identities, wherein the user identities may be managed by a sample data platform, wherein the user identities include a name of a user and a password of the user, and the sample data platform is logged in by the name of the user, and the different user names correspond to a login time, a login location, a login IP, and a login status of the user. The sampling data platform supports the user to modify the password of the user or register and add a new user identification. The method has the advantages that the safety of the database to be queried can be guaranteed, and data are prevented from being exported under the unauthorized condition.

In this embodiment, different user identifiers correspond to different user permissions, and databases to be queried accessed by different user permissions are different. After determining the user authority corresponding to the user identifier in the data acquisition request, checking whether a database to be queried, which is requested to be queried in the data acquisition request, is within the user authority, and if so, acquiring data in the database to be queried according to configuration information of the database to be queried.

S302, according to a preset database connection protocol, connection between databases represented by user rights corresponding to user identifications in the data acquisition request is established.

In this embodiment, according to a preset database in the sampling data platform, a protocol connection is performed on the preset database, and a connection is established between the database corresponding to the user right and the sampling data platform. For example, if the databases of the user permissions corresponding to the user identifiers in the data acquisition request are the database a and the database B, the database a and the database B are connected with the sampling data platform according to a preset database connection protocol. The method has the advantage that the database to be queried can be accessed after the connection between the sampling data platform and the database to be queried is established.

And S303, acquiring the data requested by the data acquisition request from the database to be inquired according to the configuration information corresponding to the database to be inquired, wherein the configuration information is used for indicating a data sampling mode when the data is acquired from the database.

For example, this step may refer to step S202, which is not described again.

And S304, storing the acquired data, wherein the acquired data is used for testing the code program.

For example, this step may refer to step S203, which is not described again.

The disclosure provides a data acquisition method based on multiple databases, comprising: acquiring a data acquisition request, wherein the data acquisition request is used for indicating a database to be queried; acquiring data requested by a data acquisition request from a database to be queried according to configuration information corresponding to the database to be queried, wherein the configuration information is used for indicating a data sampling mode when the data is acquired from the database; compressing the acquired data, and storing the compressed data to the local; or storing the acquired data into a repository corresponding to the database to be queried, wherein the databases are in one-to-one correspondence with the repositories. Wherein the acquired data is used to test the code program. By adopting the technical scheme, the development efficiency can be ensured under the condition of replacing developers.

Fig. 4 is a schematic diagram of a fourth embodiment according to the present disclosure, and as shown in fig. 4, fig. 4 shows a data acquisition apparatus based on multiple databases, the apparatus 400 includes:

a first obtaining unit 401, configured to obtain a data obtaining request, where the data obtaining request is used to indicate a database to be queried.

A second obtaining unit 402, configured to obtain, from the database to be queried, data requested by the data obtaining request according to configuration information corresponding to the database to be queried, where the configuration information is used to indicate a data sampling manner when obtaining the data from the database.

A storage unit 403, configured to store the acquired data, where the acquired data is used for testing the code program.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the above-described apparatus may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.

Fig. 5 is a schematic diagram of a fifth embodiment according to the present disclosure, and as shown in fig. 5, fig. 5 shows a data acquisition apparatus based on multiple databases, the apparatus 500 includes:

a first obtaining unit 501, configured to obtain a data obtaining request, where the data obtaining request is used to indicate a database to be queried.

A second obtaining unit 502, configured to obtain, from the database to be queried, data requested by the data obtaining request according to configuration information corresponding to the database to be queried, where the configuration information is used to indicate a data sampling manner when obtaining the data from the database.

The storage unit 503 is configured to store the acquired data, where the acquired data is used for testing the code program.

In one example, the configuration information includes a sampling frequency; the second obtaining unit 502 includes:

the first obtaining module 5021 is configured to obtain the data requested by the data obtaining request from the database to be queried at intervals according to the sampling frequency in the configuration information.

In one example, the configuration information includes a data derivation manner; the second obtaining unit 502 includes:

a second obtaining module 5022, configured to obtain, from the database to be queried, the data requested by the data obtaining request and the structural relationship between the data and other data if it is determined that the data deriving manner in the configuration information indicates the structural relationship of the derived data.

In one example, the second obtaining unit 502 includes:

a third obtaining module 5023, configured to obtain, from the database to be queried, the data requested by the data obtaining request and the data table to which the data belongs if it is determined that the data deriving manner in the configuration information indicates the data table to which the derived data belongs; the data table includes data and other data having an association relationship with the data.

In one example, the third acquisition module 5023 comprises:

a determining submodule 50231, configured to determine other fields having association relationships with fields of the data representation requested by the data obtaining request;

the acquiring sub-module 50232 is used for acquiring data requested by the data acquiring request from a database to be queried; and determining data under other fields from the database to be queried to obtain a data table to which the data belongs.

In one example, the determining submodule 50231 comprises:

In one example, wherein storing the acquired data comprises:

compressing the acquired data, and storing the compressed data to the local;

or storing the acquired data into a repository corresponding to the database to be queried, wherein the databases are in one-to-one correspondence with the repositories.

In one example, the first obtaining unit 501 includes:

the first response module 5011 is configured to generate a data acquisition request in response to trigger information of a user on an operation interface, where the trigger information is used to indicate a database to be queried and requested data;

the trigger information is used for indicating that the sampling frequency in the configuration information is one time.

In one example, the first obtaining unit 501 includes:

the second response module 5012 is configured to respond to a preset data acquisition task, where the data acquisition task is used to indicate a database to be queried and requested data, and generate a data acquisition request;

In one example, the data acquisition request includes a user identification; the first acquisition unit 501 further includes:

the first determining module 5013 is configured to determine, according to a corresponding relationship between a preset user identifier and a user right, the user right corresponding to the user identifier in the data acquisition request; wherein the user right characterizes a database accessible by the user;

the first obtaining module 5014 is configured to, if it is determined that the database to be queried indicated by the data obtaining request exists in the database represented by the user right corresponding to the user identifier in the data obtaining request, perform a step of obtaining the data requested by the data obtaining request from the database to be queried according to the configuration information corresponding to the database to be queried.

In one example, the apparatus further comprises:

the establishing unit 504 is configured to establish, according to a preset database connection protocol, a connection between databases represented by user permissions corresponding to user identifiers in the data obtaining request.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and other processing of the personal information of the related user are all in accordance with the regulations of related laws and regulations and do not violate the good customs of the public order.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

According to an embodiment of the present disclosure, the present disclosure also provides a computer program product comprising: a computer program, stored in a readable storage medium, from which at least one processor of the electronic device can read the computer program, the at least one processor executing the computer program causing the electronic device to perform the solution provided by any of the embodiments described above.

FIG. 6 illustrates a schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 6, the apparatus 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)602 or a computer program loaded from a storage unit 608 into a Random Access Memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The calculation unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.

A number of components in the device 600 are connected to the I/O interface 605, including: an input unit 606 such as a keyboard, a mouse, or the like; an output unit 607 such as various types of displays, speakers, and the like; a storage unit 608, such as a magnetic disk, optical disk, or the like; and a communication unit 609 such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 601 performs the various methods and processes described above, such as a multi-database based data acquisition method. For example, in some embodiments, the multi-database based data acquisition method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 602 and/or the communication unit 609. When loaded into RAM 603 and executed by the computing unit 601, a computer program may perform one or more steps of the multi-database based data acquisition method described above. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the multi-database based data acquisition method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A data acquisition method based on multiple databases comprises the following steps:

acquiring data requested by the data acquisition request from the database to be queried according to configuration information corresponding to the database to be queried, wherein the configuration information is used for indicating a data sampling mode when the data is acquired from the database;

2. The method of claim 1, wherein the configuration information comprises: sampling frequency, and/or data derivation; wherein the sampling frequency is indicative of a frequency at which data is sampled from a database; the data derivation means is used to indicate a manner in which data is derived from a database.

3. The method of claim 2, wherein the configuration information includes a sampling frequency; acquiring the data requested by the data acquisition request from the database to be queried according to the configuration information corresponding to the database to be queried, wherein the method comprises the following steps:

and acquiring the data requested by the data acquisition request from the database to be inquired at intervals according to the sampling frequency in the configuration information.

4. The method of claim 2, wherein the configuration information includes a data derivation manner; acquiring the data requested by the data acquisition request from the database to be queried according to the configuration information corresponding to the database to be queried, wherein the method comprises the following steps:

and if the data export mode in the configuration information indicates the structure relationship of the export data, acquiring the data requested by the data acquisition request and the structure relationship between the data and other data from the database to be queried.

5. The method of claim 4, further comprising:

if the data export mode in the configuration information indicates the data table to which the export data belongs, acquiring the data requested by the data acquisition request and the data table to which the data belongs from the database to be inquired; the data table includes data and other data having an association relationship with the data.

6. The method of claim 5, wherein the obtaining the data requested by the data obtaining request and the data table to which the data belongs from the database to be queried comprises:

acquiring data requested by the data acquisition request from the database to be queried; and determining the data under the other fields from the database to be queried to obtain a data table to which the data belongs.

7. The method of claim 6, wherein determining other fields having an associative relationship with a field of the data representation requested by the data acquisition request comprises:

or determining other fields having association relation with the field of the data representation requested by the data acquisition request according to the mapping relation between preset fields.

8. The method of any of claims 1-7, wherein the storing the acquired data comprises:

compressing the acquired data, and storing the compressed data to the local;

9. The method of any of claims 1-8, wherein obtaining a data acquisition request comprises:

responding to trigger information of a user on an operation interface, wherein the trigger information is used for indicating the database to be queried and the requested data, and generating the data acquisition request;

wherein the trigger information is used for indicating that the sampling frequency in the configuration information is one time.

10. The method of any of claims 1-8, wherein obtaining a data acquisition request comprises:

responding to a preset data acquisition task, wherein the data acquisition task is used for indicating the database to be queried and the requested data, and generating the data acquisition request;

11. The method of any of claims 1-10, wherein the data acquisition request includes a user identification; the method further comprises the following steps:

determining the user authority corresponding to the user identifier in the data acquisition request according to the corresponding relation between the preset user identifier and the user authority; wherein the user permissions characterize a database accessible by a user;

12. The method according to claim 11, before acquiring the data requested by the data acquisition request from the database to be queried according to the configuration information corresponding to the database to be queried, further comprising:

and establishing connection between databases represented by user rights corresponding to the user identifications in the data acquisition request according to a preset database connection protocol.

13. A multi-database based data acquisition apparatus comprising:

14. The apparatus of claim 13, wherein the configuration information comprises: sampling frequency, and/or data derivation; wherein the sampling frequency is indicative of a frequency at which data is sampled from a database; the data derivation means is used to indicate a manner in which data is derived from the database.

15. The apparatus of claim 14, wherein the configuration information comprises a sampling frequency; the second acquisition unit includes:

and the first acquisition module is used for acquiring the data requested by the data acquisition request from the database to be inquired at intervals according to the sampling frequency in the configuration information.

16. The apparatus of claim 14, wherein the configuration information includes a data derivation manner; the second acquisition unit includes:

and a second obtaining module, configured to obtain, from the database to be queried, the data requested by the data obtaining request and a structural relationship between the data and other data if it is determined that the data derivation manner in the configuration information indicates a structural relationship for deriving data.

17. The apparatus of claim 16, the second acquisition unit, comprising:

a third obtaining module, configured to obtain, from the database to be queried, data requested by the data obtaining request and a data table to which the data belongs if it is determined that a data export manner in the configuration information indicates a data table to which export data belongs; the data table includes data and other data having an association relationship with the data.

18. The apparatus of claim 17, wherein the third obtaining means comprises:

the determining submodule is used for determining other fields which have incidence relation with the fields of the data representation requested by the data acquisition request;

the obtaining sub-module is used for obtaining the data requested by the data obtaining request from the database to be queried; and determining the data under the other fields from the database to be queried to obtain a data table to which the data belongs.

19. The apparatus of claim 18, wherein the determination submodule comprises:

20. The apparatus of any of claims 13-19, wherein the storing the acquired data comprises:

compressing the acquired data, and storing the compressed data to the local;

21. The apparatus according to any one of claims 13-20, wherein the first obtaining unit comprises:

the first response module is used for responding to trigger information of a user on an operation interface, wherein the trigger information is used for indicating the database to be queried and the requested data and generating the data acquisition request;

22. The apparatus according to any one of claims 13-20, wherein the first obtaining unit comprises:

the second response module is used for responding to a preset data acquisition task, wherein the data acquisition task is used for indicating the database to be queried and the requested data and generating the data acquisition request;

23. The apparatus according to any one of claims 13-22, wherein the data acquisition request includes a user identifier; the first acquisition unit further includes:

the first determining module is used for determining the user authority corresponding to the user identifier in the data acquisition request according to the corresponding relation between the preset user identifier and the user authority; wherein the user permissions characterize a database accessible by a user;

a first obtaining module, configured to, if it is determined that the database to be queried indicated by the data obtaining request exists in the database represented by the user right corresponding to the user identifier in the data obtaining request, perform a step of obtaining, according to configuration information corresponding to the database to be queried, data requested by the data obtaining request from the database to be queried.

24. The apparatus of claim 23, the apparatus further comprising:

and the establishing unit is used for establishing the connection between the databases represented by the user authority corresponding to the user identification in the data acquisition request according to a preset database connection protocol.

25. An electronic device, comprising:

at least one processor; and

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

26. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-12.

27. A computer program product comprising a computer program which, when executed by a processor, carries out the steps of the method of any one of claims 1 to 12.