CN107688598B - Source table structure analysis method, application server and computer readable storage medium - Google Patents
Source table structure analysis method, application server and computer readable storage medium Download PDFInfo
- Publication number
- CN107688598B CN107688598B CN201710490452.6A CN201710490452A CN107688598B CN 107688598 B CN107688598 B CN 107688598B CN 201710490452 A CN201710490452 A CN 201710490452A CN 107688598 B CN107688598 B CN 107688598B
- Authority
- CN
- China
- Prior art keywords
- source
- data
- table structure
- analysis
- source table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Transfer Between Computers (AREA)
- Stored Programmes (AREA)
Abstract
The invention discloses a source table structure analysis method, which comprises the following steps: establishing a first connection between a data platform and at least one data source and establishing a second connection between the data platform and the application server through a data transmission tool in the data platform; running a preset analysis script file in the application server, and controlling the data transmission tool to execute a source table structure analysis task in the analysis script file through the second connection; and receiving the source table structure of each data source obtained by the data transmission tool through the analysis of the first connection from each data source. The invention can simplify the architecture design of the application server and each data source and simultaneously improve the analysis efficiency.
Description
Technical Field
The invention relates to the technical field of computer information, in particular to a source table structure analysis method, an application server and a computer readable storage medium.
Background
In the data synchronization work of multiple data sources, if the automatic analysis of the source table structure is to be implemented, the existing architecture is that a server for analysis work connects two ends of a data source and a target platform (such as an internal data platform). In an internal data platform (such as a Hadoop platform), when a plurality of data sources need to be counted to the internal data platform, all the data sources need to be connected to the internal data platform in an architecture, and all the data sources need to be connected to an analysis server at the same time, so that the architecture is complex and the efficiency is low. Therefore, the structure for analyzing the structure of the multi-data-source table in the prior art is not reasonable enough, and needs to be improved urgently.
Disclosure of Invention
In view of this, the present invention provides a method for analyzing a source table structure, an application server and a computer-readable storage medium, which can simplify the architectural design of the application server and each data source and improve the analysis efficiency.
First, to achieve the above object, the present invention provides an application server, where the application server includes a memory, a processor, and a source table structure parsing system stored in the memory and operable on the processor, and when executed by the processor, the source table structure parsing system implements the following steps:
establishing a first connection between a data platform and at least one data source and establishing a second connection between the data platform and the application server through a data transmission tool in the data platform;
running a preset analysis script file in the application server, and controlling the data transmission tool to execute a source table structure analysis task in the analysis script file through the second connection; and
and receiving the source table structure of each data source obtained by the data transmission tool through the analysis of the first connection from each data source.
Preferably, the parsing the script file includes: the method includes the steps of parsing a first parsing script file of a first type data source, parsing a second parsing script file of a second type data source, and parsing a third parsing script file of a third type data source.
Preferably, the controlling the data transmission tool to execute the source table structure parsing task in the parsing script file includes:
and when a source table structure analysis task in the analysis script file starts to be executed, controlling the data transmission tool to analyze the source table structure of each data source through the first connection, and returning the source table structure of each data source obtained through analysis to the application server through the second connection.
Preferably, the controlling the data transmission tool to execute the source table structure parsing task in the parsing script file includes:
if the data source is a first type data source, controlling the data transmission tool to execute a source table structure analysis task in the first analysis script file, and returning a source table structure of the first type data source obtained through analysis to the application server through the second connection;
if the data source is a second type data source, controlling the data transmission tool to execute a source table structure analysis task in the second analysis script file, and returning a source table structure of the second type data source obtained through analysis to the application server through the second connection; and
and if the data source is a third type data source, controlling the data transmission tool to execute a source table structure analysis task in the third analysis script file, and returning the source table structure of the third type data source obtained by analysis to the application server through the second connection.
Preferably, the specific step of parsing the source table structure includes:
according to the type of each data source, the data transmission tool generates a script command corresponding to each data source, and sends the script command to a source database for execution, wherein the execution result is the source table structure information corresponding to each data source;
controlling the data transmission tool to extract the execution result to the data platform;
the data platform controls the data transmission tool to push the extracted source table structure information to the application server;
and the application server processes and stores the source table structure information.
In addition, in order to achieve the above object, the present invention further provides a method for parsing a source table structure, where the method is applied to an application server, and the method includes:
establishing a first connection between a data platform and at least one data source and establishing a second connection between the data platform and the application server through a data transmission tool in the data platform;
running a preset analysis script file in the application server, and controlling the data transmission tool to execute a source table structure analysis task in the analysis script file through the second connection; and
and receiving the source table structure of each data source obtained by the data transmission tool through the analysis of the first connection from each data source.
Preferably, the controlling the data transmission tool to execute the source table structure parsing task in the parsing script file includes:
and when a source table structure analysis task in the analysis script file starts to be executed, controlling the data transmission tool to analyze the source table structure of each data source through the first connection, and returning the source table structure of each data source obtained through analysis to the application server through the second connection.
Preferably, the controlling the data transmission tool to execute the source table structure parsing task in the parsing script file includes:
if the data source is a first type data source, controlling the data transmission tool to execute a source table structure analysis task in the first analysis script file, and returning a source table structure of the first type data source obtained through analysis to the application server through the second connection;
if the data source is a second type data source, controlling the data transmission tool to execute a source table structure analysis task in the second analysis script file, and returning a source table structure of the second type data source obtained through analysis to the application server through the second connection; and
and if the data source is a third type data source, controlling the data transmission tool to execute a source table structure analysis task in the third analysis script file, and returning the source table structure of the third type data source obtained by analysis to the application server through the second connection.
Preferably, the specific step of parsing the source table structure includes:
according to the type of each data source, the data transmission tool generates a script command corresponding to each data source, and sends the script command to a source database for execution, wherein the execution result is the source table structure information corresponding to each data source;
controlling the data transmission tool to extract the execution result to the data platform;
the data platform controls the data transmission tool to push the extracted source table structure information to the application server;
and the application server processes and stores the source table structure information.
Further, to achieve the above object, the present invention also provides a computer readable storage medium storing a source table structure parsing system, which is executable by at least one processor, so as to cause the at least one processor to execute the steps of the source table structure parsing method as described above.
Compared with the prior art, the application server, the source table structure analysis method and the computer readable storage medium provided by the invention have the advantages that the first connection between the data platform and each data source is established in advance, the second connection between the data platform and the application server is established, the source table structure of each data source is obtained by analyzing from each data source through the first connection, and the source table structure of each data source obtained by analyzing is returned to the application server through the second connection, so that the architectural design of the application server and each data source is simplified, and the analysis efficiency is improved.
Drawings
FIG. 1 is a schematic diagram of an alternative application environment for various embodiments of the present invention;
FIG. 2 is a schematic diagram of an alternative hardware architecture of the application server of FIG. 1;
FIG. 3 is a functional block diagram of an embodiment of a source table structure parsing system according to the invention;
fig. 4 is a schematic flow chart of an embodiment of a source table structure parsing method according to the present invention.
Reference numerals:
| |
2 |
| Network | 3 |
| Data platform | 4 |
| |
5 |
| |
21 |
| Processor with a memory having a plurality of |
22 |
| |
23 |
| Source table |
20 |
| |
201 |
| |
202 |
| |
203 |
| Procedure step | S41-S43 |
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the description relating to "first", "second", etc. in the present invention is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
It is further noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Fig. 1 is a schematic diagram of an alternative application environment according to various embodiments of the present invention.
In this embodiment, the present invention may be applied in an application environment including, but not limited to, an application server 2, a network 3, a data platform 4, and a data source 5. The application server 2 may be a rack server, a blade server, a tower server, or a rack server, and the application server 2 may be an independent server or a server cluster composed of a plurality of servers. The network 3 may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), Wi-Fi, and the like.
The application server 2 is in communication connection with a data platform 4 through the network 3, and the data platform 4 is in communication connection with each data source 5 through the network 3. A first connection (or referred to as a first type connection) is established between the data platform 4 and each data source 5, so that the data platform 4 can perform data transmission and interaction with each data source 5 through the first connection. A second connection (or referred to as a second type connection) is established between the data platform 4 and the application server 2, so that the data platform 4 can perform data transmission and interaction with the application server 2 through the second connection.
Fig. 2 is a schematic diagram of an alternative hardware architecture of the application server 2 in fig. 1. In this embodiment, the application server 2 may include, but is not limited to, a memory 21, a processor 22, and a network interface 23, which may be communicatively connected to each other through a system bus. It is noted that fig. 2 only shows the application server 2 with components 21-23, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
The memory 21 includes at least one type of readable storage medium, which includes a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 21 may be an internal storage unit of the application server 2, such as a hard disk or a memory of the application server 2. In other embodiments, the memory 21 may also be an external storage device of the application server 2, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the application server 2. Of course, the memory 21 may also comprise both an internal storage unit of the application server 2 and an external storage device thereof. In this embodiment, the memory 21 is generally used for storing an operating system installed in the application server 2 and various types of application software, such as program codes of the source table structure parsing system 20. Further, the memory 21 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 22 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 22 is generally configured to control the overall operation of the application server 2, such as performing control and processing related to data interaction or communication with the application server 2. In this embodiment, the processor 22 is configured to run the program codes stored in the memory 21 or process data, for example, run the source table structure parsing system 20.
The network interface 23 may comprise a wireless network interface or a wired network interface, and the network interface 23 is generally used for establishing a communication connection between the application server 2 and other electronic devices. In this embodiment, the network interface 23 is mainly used to connect the application server 2 and the data platform 4 through the network 3, and establish a data transmission channel and a communication connection between the application server 2 and the data platform 4.
The application environment and the hardware structure and function of the related devices of the various embodiments of the present invention have been described in detail so far. Hereinafter, various embodiments of the present invention will be proposed based on the above-described application environment and related devices.
First, the present invention provides a source table structure analysis system 20.
Fig. 3 is a functional block diagram of a source table structure analysis system 20 according to an embodiment of the present invention. In this embodiment, the source table structure parsing system 20 may be divided into one or more modules, and the one or more modules are stored in the memory 21 and executed by one or more processors (in this embodiment, the processor 22) to complete the present invention. For example, in fig. 3, the source table structure parsing system 20 may be divided into a connection module 201, a parsing module 202, and a receiving module 203. The functional module referred to in the present invention refers to a series of computer program instruction segments capable of performing specific functions, and is more suitable than a program for describing the execution process of the source table structure parsing system 20 in the application server 2. The functions of the function modules 201 and 203 will be described in detail below.
The connection module 201 is configured to establish a first connection (or referred to as a first type connection) between a data platform 4 and at least one data source 5 through a data transmission tool in the data platform 4, and establish a second connection (or referred to as a second type connection) between the data platform 4 and an application server 2 (e.g., an analysis server).
Preferably, in this embodiment, the data platform 4 is a Hadoop data platform, and the data transmission tool is a Sqoop tool set in Hadoop. It will be appreciated by those skilled in the art that in other embodiments, the data platform and data transfer means may be of other types.
The parsing module 202 is configured to run a parsing script file preset in the application server 2, and control the data transmission tool to execute a source table structure parsing task in the parsing script file through the second connection.
Preferably, in this embodiment, each data source is provided with a corresponding parsing script file, for example, the parsing script files include, but are not limited to, a first parsing script file for parsing a first type data source (e.g., MS SQL Server data source), a second parsing script file for parsing a second type data source (e.g., Oracle data source), a third parsing script file for parsing a third type data source (e.g., MySQL data source), and the like. It should be understood by those skilled in the art that in other embodiments, other types of parse script files may be added, depending on the number of data source types.
Specifically, if the data source is a first type data source, controlling the data transmission tool to execute a source table structure parsing task in the first parsing script file; if the data source is a second type data source, controlling the data transmission tool to execute a source table structure analysis task in the second analysis script file; and if the data source is a third type data source, controlling the data transmission tool to execute a source table structure analysis task in the third analysis script file.
The receiving module 203 is configured to receive a source table structure of each data source 5 obtained by the data transmission tool through parsing from each data source 5 through the first connection. In this embodiment, the source table structure of the data source includes, but is not limited to, a data type and included field information of each data source.
Preferably, in this embodiment, the controlling the data transmission tool to execute the source table structure parsing task in the parsing script file includes: and when a source table structure analysis task in the analysis script file starts to be executed, controlling the data transmission tool to analyze the source table structure of each data source 5 through the first connection, and returning the source table structure of each data source 5 obtained through analysis to the application server 2 through the second connection.
Specifically, if the data source 5 is a first type data source, the data transmission tool is controlled to execute a source table structure parsing task in the first parsing script file, and the source table structure of the first type data source obtained through parsing is returned to the application server 2 through the second connection.
And if the data source 5 is a second type data source, controlling the data transmission tool to execute a source table structure analysis task in the second analysis script file, and returning the source table structure of the second type data source obtained through analysis to the application server 2 through the second connection.
And if the data source 5 is a third type data source, controlling the data transmission tool to execute a source table structure analysis task in the third analysis script file, and returning a source table structure of the third type data source obtained through analysis to the application server 2 through the second connection.
Preferably, in this embodiment, the concrete step of parsing the source table structure includes:
A. according to the type of each data source 5, the data transmission tool generates a script command corresponding to each data source, and sends the script command to a source database for execution, wherein the execution result is the source table structure information corresponding to each data source.
Specifically, if the data source 5 is a first type data source, the data transmission tool generates a first script command corresponding to the first type data source, and sends the first script command to a database corresponding to the first type data source for execution, where an execution result is source table structure information corresponding to the first type data source.
If the data source 5 is a second type data source, the data transmission tool generates a second script command corresponding to the second type data source, and sends the second script command to a database corresponding to the second type data source for execution, and the execution result is the source table structure information corresponding to the second type data source.
If the data source 5 is a third type data source, the data transmission tool generates a third script command corresponding to the third type data source, and sends the third script command to a database corresponding to the third type data source for execution, and the execution result is the source table structure information corresponding to the third type data source.
B. And controlling the data transmission tool to extract the execution result to the data platform 4.
C. And the data platform 4 controls the data transmission tool to push the extracted source table structure information to the application server 2.
D. The application server 2 processes (such as data cleaning, normalization, and other operations) and stores the source table structure information, and the source table structure information can be used by a subsequent program.
In the invention, the application server 2 is connected with the data platform 4 only through the second connection, and the connection between the application server 2 and each data source 5 is not required to be established, so the structure is simple and the efficiency of analyzing the structure of the source table can be improved.
Preferably, in other embodiments, the source table structure parsing system 20 when executed by the processor 22 further implements the steps of:
and controlling the data transmission tool to execute the source table structure analysis tasks in each analysis script file in parallel through a parallel scheduling mechanism, analyzing to obtain the source table structure of each data source, and returning the source table structure of each data source 5 obtained through analysis to the application server 2 through the second connection.
Through the functional module 201 and 203, the source table structure analysis system 20 provided by the present invention analyzes and obtains the source table structure of each data source from each data source 5 by using the first connection through pre-establishing the first connection between the data platform 4 and each data source 5 and establishing the second connection between the data platform 4 and the application server 2, and returns the source table structure of each data source 5 obtained by analysis to the application server 2 through the second connection, thereby simplifying the architectural design of the application server 2 and each data source 5 and simultaneously improving the analysis efficiency.
In addition, the invention also provides a source table structure analysis method.
Fig. 4 is a schematic flow chart showing an embodiment of the source table structure analysis method according to the present invention. In this embodiment, the execution order of the steps in the flowchart shown in fig. 4 may be changed and some steps may be omitted according to different requirements.
Step S41, a first connection (or referred to as a first type connection) between the data platform 4 and at least one data source 5 is established through a data transmission tool in the data platform 4, and a second connection (or referred to as a second type connection) between the data platform 4 and an application server 2 (e.g., a resolution server) is established.
Preferably, in this embodiment, the data platform 4 is a Hadoop data platform, and the data transmission tool is a Sqoop tool set in Hadoop. It will be appreciated by those skilled in the art that in other embodiments, the data platform and data transfer means may be of other types.
Step S42, running an analysis script file preset in the application server 2, and controlling the data transmission tool to execute a source table structure analysis task in the analysis script file through the second connection.
Preferably, in this embodiment, each data source is provided with a corresponding parsing script file, for example, the parsing script files include, but are not limited to, a first parsing script file for parsing a first type data source (e.g., MS SQL Server data source), a second parsing script file for parsing a second type data source (e.g., Oracle data source), a third parsing script file for parsing a third type data source (e.g., MySQL data source), and the like. It should be understood by those skilled in the art that in other embodiments, other types of parse script files may be added, depending on the number of data source types.
Specifically, if the data source is a first type data source, controlling the data transmission tool to execute a source table structure parsing task in the first parsing script file; if the data source is a second type data source, controlling the data transmission tool to execute a source table structure analysis task in the second analysis script file; and if the data source is a third type data source, controlling the data transmission tool to execute a source table structure analysis task in the third analysis script file.
Step S43, receiving the source table structure of each data source 5 parsed from each data source 5 by the data transmission tool through the first connection. In this embodiment, the source table structure of the data source includes, but is not limited to, a data type and included field information of each data source.
Preferably, in this embodiment, the controlling the data transmission tool to execute the source table structure parsing task in the parsing script file includes: and when a source table structure analysis task in the analysis script file starts to be executed, controlling the data transmission tool to analyze the source table structure of each data source 5 through the first connection, and returning the source table structure of each data source 5 obtained through analysis to the application server 2 through the second connection.
Specifically, if the data source 5 is a first type data source, the data transmission tool is controlled to execute a source table structure parsing task in the first parsing script file, and the source table structure of the first type data source obtained through parsing is returned to the application server 2 through the second connection.
And if the data source 5 is a second type data source, controlling the data transmission tool to execute a source table structure analysis task in the second analysis script file, and returning the source table structure of the second type data source obtained through analysis to the application server 2 through the second connection.
And if the data source 5 is a third type data source, controlling the data transmission tool to execute a source table structure analysis task in the third analysis script file, and returning a source table structure of the third type data source obtained through analysis to the application server 2 through the second connection.
Preferably, in this embodiment, the concrete step of parsing the source table structure includes:
A. according to the type of each data source 5, the data transmission tool generates a script command corresponding to each data source, and sends the script command to a source database for execution, wherein the execution result is the source table structure information corresponding to each data source.
Specifically, if the data source 5 is a first type data source, the data transmission tool generates a first script command corresponding to the first type data source, and sends the first script command to a database corresponding to the first type data source for execution, where an execution result is source table structure information corresponding to the first type data source.
If the data source 5 is a second type data source, the data transmission tool generates a second script command corresponding to the second type data source, and sends the second script command to a database corresponding to the second type data source for execution, and the execution result is the source table structure information corresponding to the second type data source.
If the data source 5 is a third type data source, the data transmission tool generates a third script command corresponding to the third type data source, and sends the third script command to a database corresponding to the third type data source for execution, and the execution result is the source table structure information corresponding to the third type data source.
B. And controlling the data transmission tool to extract the execution result to the data platform 4.
C. And the data platform 4 controls the data transmission tool to push the extracted source table structure information to the application server 2.
D. The application server 2 processes (such as data cleaning, normalization, and other operations) and stores the source table structure information, and the source table structure information can be used by a subsequent program.
In the invention, the application server 2 is connected with the data platform 4 only through the second connection, and the connection between the application server 2 and each data source 5 is not required to be established, so the structure is simple and the efficiency of analyzing the structure of the source table can be improved.
Preferably, in other embodiments, the method further comprises the steps of:
and controlling the data transmission tool to execute the source table structure analysis tasks in each analysis script file in parallel through a parallel scheduling mechanism, analyzing to obtain the source table structure of each data source, and returning the source table structure of each data source 5 obtained through analysis to the application server 2 through the second connection.
Through the steps S41-S43, the method for analyzing the source table structure provided by the present invention analyzes and obtains the source table structure of each data source from each data source 5 by using the first connection through establishing the first connection between the data platform 4 and each data source 5 in advance and establishing the second connection between the data platform 4 and the application server 2, and returns the source table structure of each data source 5 obtained by analysis to the application server 2 through the second connection, thereby simplifying the architectural design of the application server 2 and each data source 5, and improving the analysis efficiency.
Further, to achieve the above object, the present invention also provides a computer readable storage medium (such as ROM/RAM, magnetic disk, optical disk) storing a source table structure parsing system 20, where the source table structure parsing system 20 is executable by at least one processor 22 to make the at least one processor 22 execute the steps of the source table structure parsing method as described above.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, and are not to be construed as limiting the scope of the invention. The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments. Additionally, while a logical order is shown in the flow diagrams, in some cases, the steps shown or described may be performed in an order different than here.
Those skilled in the art can implement the invention in various modifications, such as features from one embodiment can be used in another embodiment to yield yet a further embodiment, without departing from the scope and spirit of the invention. All the equivalent structures or equivalent processes performed by using the contents of the specification and the drawings of the invention, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (8)
1. An application server, comprising a memory, a processor, and a source table structure parsing system stored on the memory and operable on the processor, wherein the source table structure parsing system when executed by the processor implements the steps of:
establishing a first connection between a data platform and at least one data source and establishing a second connection between the data platform and the application server through a data transmission tool in the data platform;
running a preset analysis script file in the application server, and controlling the data transmission tool to execute a source table structure analysis task in the analysis script file through the second connection; and
receiving a source table structure of each data source obtained by the data transmission tool through parsing from each data source through the first connection, wherein the source table structure of each data source includes, but is not limited to, a data type and included field information of each data source;
the controlling the data transmission tool to execute the source table structure parsing task in the parsing script file comprises:
and when a source table structure analysis task in the analysis script file starts to be executed, controlling the data transmission tool to analyze the source table structure of each data source through the first connection, and returning the source table structure of each data source obtained through analysis to the application server through the second connection.
2. The application server of claim 1, wherein parsing the script file comprises: the method includes the steps of parsing a first parsing script file of a first type data source, parsing a second parsing script file of a second type data source, and parsing a third parsing script file of a third type data source.
3. The application server of claim 1, wherein the controlling the data transport facility to perform a source table structure parsing task in the parsing script file comprises:
if the data source is a first type data source, controlling the data transmission tool to execute a source table structure analysis task in a first analysis script file, and returning a source table structure of the first type data source obtained through analysis to the application server through the second connection;
if the data source is a second type data source, controlling the data transmission tool to execute a source table structure analysis task in a second analysis script file, and returning a source table structure of the second type data source obtained through analysis to the application server through the second connection; and
and if the data source is a third type data source, controlling the data transmission tool to execute a source table structure analysis task in a third analysis script file, and returning the source table structure of the third type data source obtained by analysis to the application server through the second connection.
4. The application server of claim 1, wherein the specific step of parsing the source table structure comprises:
according to the type of each data source, the data transmission tool generates a script command corresponding to each data source, and sends the script command to a source database for execution, wherein the execution result is the source table structure information corresponding to each data source;
controlling the data transmission tool to extract the execution result to the data platform;
the data platform controls the data transmission tool to push the extracted source table structure information to the application server;
and the application server processes and stores the source table structure information.
5. A source table structure parsing method applied to an application server is characterized by comprising the following steps:
establishing a first connection between a data platform and at least one data source and establishing a second connection between the data platform and the application server through a data transmission tool in the data platform;
running a preset analysis script file in the application server, and controlling the data transmission tool to execute a source table structure analysis task in the analysis script file through the second connection; and
receiving a source table structure of each data source obtained by the data transmission tool through parsing from each data source through the first connection, wherein the source table structure of each data source includes, but is not limited to, a data type and included field information of each data source;
the controlling the data transmission tool to execute the source table structure parsing task in the parsing script file comprises:
and when a source table structure analysis task in the analysis script file starts to be executed, controlling the data transmission tool to analyze the source table structure of each data source through the first connection, and returning the source table structure of each data source obtained through analysis to the application server through the second connection.
6. The method of claim 5, wherein the controlling the data transport tool to perform a source table structure parsing task in the parsing script file comprises:
if the data source is a first type data source, controlling the data transmission tool to execute a source table structure analysis task in a first analysis script file, and returning a source table structure of the first type data source obtained through analysis to the application server through the second connection;
if the data source is a second type data source, controlling the data transmission tool to execute a source table structure analysis task in a second analysis script file, and returning a source table structure of the second type data source obtained through analysis to the application server through the second connection; and
and if the data source is a third type data source, controlling the data transmission tool to execute a source table structure analysis task in a third analysis script file, and returning the source table structure of the third type data source obtained by analysis to the application server through the second connection.
7. The method of claim 5, wherein the step of parsing the source table structure comprises:
according to the type of each data source, the data transmission tool generates a script command corresponding to each data source, and sends the script command to a source database for execution, wherein the execution result is the source table structure information corresponding to each data source;
controlling the data transmission tool to extract the execution result to the data platform;
the data platform controls the data transmission tool to push the extracted source table structure information to the application server;
and the application server processes and stores the source table structure information.
8. A computer-readable storage medium storing a source table structure parsing system executable by at least one processor to cause the at least one processor to perform the steps of the source table structure parsing method of any of claims 5-7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710490452.6A CN107688598B (en) | 2017-06-25 | 2017-06-25 | Source table structure analysis method, application server and computer readable storage medium |
| PCT/CN2017/100050 WO2019000628A1 (en) | 2017-06-25 | 2017-08-31 | Source table structure parsing method and system, application server and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710490452.6A CN107688598B (en) | 2017-06-25 | 2017-06-25 | Source table structure analysis method, application server and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107688598A CN107688598A (en) | 2018-02-13 |
| CN107688598B true CN107688598B (en) | 2021-02-09 |
Family
ID=61152696
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710490452.6A Active CN107688598B (en) | 2017-06-25 | 2017-06-25 | Source table structure analysis method, application server and computer readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN107688598B (en) |
| WO (1) | WO2019000628A1 (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915341A (en) * | 2014-03-10 | 2015-09-16 | 中国科学院沈阳自动化研究所 | Visual multi-database ETL integration method and system |
| CN105205105A (en) * | 2015-08-27 | 2015-12-30 | 浪潮集团有限公司 | Data ETL (Extract Transform Load) system based on storm and treatment method based on storm |
-
2017
- 2017-06-25 CN CN201710490452.6A patent/CN107688598B/en active Active
- 2017-08-31 WO PCT/CN2017/100050 patent/WO2019000628A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104915341A (en) * | 2014-03-10 | 2015-09-16 | 中国科学院沈阳自动化研究所 | Visual multi-database ETL integration method and system |
| CN105205105A (en) * | 2015-08-27 | 2015-12-30 | 浪潮集团有限公司 | Data ETL (Extract Transform Load) system based on storm and treatment method based on storm |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107688598A (en) | 2018-02-13 |
| WO2019000628A1 (en) | 2019-01-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107689982B (en) | Multi-data source data synchronization method, application server and computer readable storage medium | |
| CN107844634B (en) | Modeling method of multivariate general model platform, electronic equipment and computer readable storage medium | |
| JP6695984B2 (en) | Multitask scheduling method, system, application server, and computer-readable storage medium | |
| CN108427705B (en) | Electronic device, distributed system log query method and storage medium | |
| CN107844851B (en) | Survey grid optimization method, electronic device, and computer-readable storage medium | |
| CN108241529B (en) | Salary calculation method, application server and computer readable storage medium | |
| CN110704521A (en) | Interface data access method and system | |
| CN110928802A (en) | Test method, device, equipment and storage medium based on automatic generation of case | |
| CN110599354B (en) | Online checking method, online checking system, computer device and computer readable storage medium | |
| CN110175157B (en) | Query method and query device for column storage file | |
| WO2019075970A1 (en) | Line wrap recognition method for table information, electronic device, and computer-readable storage medium | |
| CN109308305B (en) | Monitoring data query method and device and computer equipment | |
| CN108170551B (en) | Crawler system based front-end and back-end error processing method, server and storage medium | |
| WO2019075968A1 (en) | Cross-page recognition method for form information, electronic device, and computer-readable storage medium | |
| CN108255735B (en) | Associated environment testing method, electronic device and computer readable storage medium | |
| CN108446110B (en) | Lua script generation method, Lua script generation device, Lua script generation terminal and computer readable medium | |
| CN111352851A (en) | Program testing method and device, computer equipment and storage medium | |
| CN115455925A (en) | Contract generation method and device, electronic equipment and storage medium | |
| CN111552712A (en) | Report data extraction method and device and computer equipment | |
| CN111723077A (en) | Data dictionary maintenance method and device and computer equipment | |
| CN110647564A (en) | Hive table establishing method, electronic device and computer readable storage medium | |
| CN108053459B (en) | Signature file synthesis method, application server and computer readable storage medium | |
| CN107688598B (en) | Source table structure analysis method, application server and computer readable storage medium | |
| CN112416648A (en) | Data verification method and device | |
| CN110442843B (en) | Character replacement method, system, computer device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |