CN110661851A - Data exchange method and device - Google Patents
Data exchange method and device Download PDFInfo
- Publication number
- CN110661851A CN110661851A CN201910833291.5A CN201910833291A CN110661851A CN 110661851 A CN110661851 A CN 110661851A CN 201910833291 A CN201910833291 A CN 201910833291A CN 110661851 A CN110661851 A CN 110661851A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- sub
- key client
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 239000000284 extract Substances 0.000 claims description 25
- 238000012546 transfer Methods 0.000 claims description 11
- 238000012423 maintenance Methods 0.000 abstract description 5
- 238000000605 extraction Methods 0.000 description 33
- 230000006870 function Effects 0.000 description 22
- 238000010586 diagram Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 11
- 238000012545 processing Methods 0.000 description 11
- 238000004590 computer program Methods 0.000 description 10
- 238000006243 chemical reaction Methods 0.000 description 8
- 238000013075 data extraction Methods 0.000 description 8
- 238000012544 monitoring process Methods 0.000 description 6
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
Abstract
The application provides a data exchange method and a device, wherein the method comprises the following steps: sending a first type of parameters to a first key client, and creating a first data source aiming at a first sub-database by the first key client according to the first type of parameters; sending the second type of parameters to a second key client, and creating a second data source aiming at a second sub-database by the second key client according to the second type of parameters; sending a first control message to a first key client, extracting data from a first sub-database by the first key client according to a first data source, and storing the extracted data in a central database; and sending a second control message to a second key client, extracting data from the central database by the second key client, and storing the extracted data into the second sub-database according to the second data source. Through the technical scheme, the user operation is simplified, the maintenance and management of the client side are facilitated, the workload of the user is reduced, and the error rate caused by the user operation is reduced.
Description
Technical Field
The present application relates to the field of communications technologies, and in particular, to a data exchange method and apparatus.
Background
The keyboard is an open-source ETL (Extract Transform Load) tool, can run on Windows, Linux, Unix and other operating systems, adopts a visual programming mode for data processing, is convenient and easy to operate, is efficient and stable in data extraction, and is a necessary tool for establishing a data warehouse.
As shown in fig. 1, in order to deploy a networking schematic diagram of a key client, a key client 1 is installed on a device 1, and a key client 2 is installed on a device 2. In order to exchange the data of the sub-database 1 to the sub-database 3, the corresponding processing flow is as follows: the user logs in the button client 1 and performs related configuration for the button client 1. Then, the user logs in to the key client 2 and performs related configuration for the key client 2. Then, the user logs in the key client 1 again, triggers the key client 1 to extract the data of the sub-database 1, and stores the data in the central database. After all the data are successfully stored in the central database, the user logs in the key client 2 again, the key client 2 is triggered to extract the data of the central database, and the data are stored in the sub-database 3 until all the data are successfully stored in the sub-database 3.
In the above manner, the user can operate and manage only by logging in the key client, for example, in the data exchange process, the user needs to log in each key client frequently, the operation is very complicated, the workload of the user is increased, the operation difficulty of the user is increased, and the error rate of the user operation is increased.
Disclosure of Invention
The application provides a data exchange method, which is applied to a server side in a data exchange system, wherein the data exchange system further comprises at least two button clients, the button clients operate in a specified mode, and under the specified mode, the button clients have a hypertext transfer protocol (HTTP) service function and operate a task matched with a request by monitoring the request of the server side, and the method comprises the following steps:
sending a first type of parameters to a first key client in at least two key clients so that the first key client creates a first data source aiming at a first sub-database according to the first type of parameters; sending a second type of parameters to a second key client in the at least two key clients so that the second key client creates a second data source for a second sub-database according to the second type of parameters;
sending a first control message to a first key client so that the first key client extracts data from the first sub-database according to the first data source and stores the extracted data in a central database;
and sending a second control message to the second key client so that the second key client extracts data from the central database and stores the extracted data in the second sub-database according to the second data source.
After the sending the first control message to the first key client and before the sending the second control message to the second key client, the method further includes:
receiving statistical information sent by the first button client, wherein the statistical information comprises an extraction state; if the extraction state is not finished, forbidding sending a second control message to a second button client; and if the extraction state is finished, executing a process of sending a second control message to the second button client.
The first type of parameter comprises one or any combination of the following: uniform resource locator URL information, authentication information, port information, schema information and data table information corresponding to the first sub-database;
the second type of parameter comprises one or any combination of the following: URL information, authentication information, port information, schema information and data table information corresponding to the second sub-database;
the statistical information further comprises one or any combination of the following: extraction result, log information, history information, execution time, currently extracted data amount, and currently extracted data rate.
Before sending the first type parameters to the first key client of the at least two key clients, the method further includes: and receiving a user request message through a WEB page, wherein the user request message carries the address information of the first button client and the address information of the second button client.
After receiving the statistical information sent by the first button client, the method further includes:
and displaying the statistical information through the WEB page.
The application provides a data exchange device, is applied to the server in the data exchange system, the data exchange system still includes two at least button customer ends, the operation of button customer end is in appointed mode, and under appointed mode, the button customer end has hypertext transfer protocol HTTP service function, and through monitoring the request of server end comes the operation with the task that the request matches, the device includes:
the first sending module is used for sending a first type of parameters to a first key client in the at least two key clients so that the first key client can create a first data source aiming at a first sub-database according to the first type of parameters;
a second sending module, configured to send a second type of parameter to a second key client of the at least two key clients, so that the second key client creates a second data source for a second sub-database according to the second type of parameter;
a third sending module, configured to send a first control message to the first key client, so that the first key client extracts data from the first sub-database according to the first data source, and stores the extracted data in a central database;
and the fourth sending module is used for sending a second control message to the second key client so as to enable the second key client to extract data from the central database and store the extracted data into the second sub-database according to the second data source.
The device further comprises:
the receiving module is used for receiving statistical information sent by a first key client after the third sending module sends a first control message to the first key client and before the fourth sending module sends a second control message to a second key client, wherein the statistical information comprises an extraction state;
the fourth sending module is specifically configured to prohibit sending of the second control message to the second key client if the extraction state is incomplete in a process of sending the second control message to the second key client; and if the extraction state is finished, sending a second control message to the second button client.
The first type of parameter comprises one or any combination of the following: uniform resource locator URL information, authentication information, port information, schema information and data table information corresponding to the first sub-database;
the second type of parameter comprises one or any combination of the following: URL information, authentication information, port information, schema information and data table information corresponding to the second sub-database;
the statistical information further comprises one or any combination of the following: extraction result, log information, history information, execution time, currently extracted data amount, and currently extracted data rate.
The receiving module is further configured to receive a user request message through a WEB page before the first sending module sends the first type of parameters to a first key client of the at least two key clients, where the user request message carries address information of the first key client and address information of the second key client.
The device further comprises: and the display module is used for displaying the statistical information through the WEB page after the receiving module receives the statistical information sent by the first button client.
Based on the technical scheme, in the embodiment of the application, the server side sends the first type of parameters to the first key client side so that the first key client side creates a first data source according to the first type of parameters, and sends the first control message to the first key client side so that the first key client side extracts data from the first sub-database according to the first data source and stores the extracted data in the central database; and the server side sends the second type parameters to the second key client side so that the second key client side creates a second data source according to the second type parameters, sends a second control message to the second key client side so that the second key client side extracts data from the central database, and stores the extracted data into a second sub-database according to the second data source. Based on the mode, the server side can manage the plurality of clients without logging in the clients, the functions of the plurality of clients are controlled remotely, user operation is simplified, maintenance and management of the clients are facilitated, workload of the users is reduced, and error rate caused by user operation is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings of the embodiments of the present application.
FIG. 1 is a schematic diagram of a network deployment of a button client;
FIG. 2 is a flow chart of a method of data exchange in one embodiment of the present application;
FIG. 3 is a schematic diagram of an application scenario in an embodiment of the present application;
FIG. 4 is a diagram of a hardware configuration of a server in one embodiment of the present application;
fig. 5 is a block diagram of a data exchange device according to an embodiment of the present application.
Detailed Description
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein is meant to encompass any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. Depending on the context, moreover, the word "if" as used may be interpreted as "at … …" or "when … …" or "in response to a determination".
The embodiment of the application provides a data exchange method, which can be applied to a data exchange system, wherein the data exchange system can comprise a server and at least two button clients, the button clients operate in a specified mode, and in the specified mode, the button clients have an HTTP (Hyper Text Transfer Protocol) service function and operate a task matched with a request by monitoring the request of the server. The at least two key clients include a first key client and a second key client, the number of the first key client and the second key client may be one or multiple, and for convenience of description, a first key client and a second key client are taken as examples in the following.
To sum up, the data exchange system may include a server, a first key client, a second key client, a first sub-database, a second sub-database, and a central database, and the method is used to exchange data of the first sub-database to the second sub-database. The first key client is used for managing a first sub-database, the second key client is used for managing a second sub-database, and the server is used for managing the first key client and the second key client. In the application scenario, referring to fig. 2, which is a flowchart of the data exchange method, the method may be applied to a server, and the method may include the following steps:
step 201, sending a first type parameter to a first key client, so that the first key client creates a first data source for a first sub-database according to the first type parameter; and sending the second type of parameters to the second key client so that the second key client creates a second data source aiming at the second sub-database according to the second type of parameters.
In one example, the first type of parameter may include, but is not limited to, one or any combination of the following: the first sub-database corresponds to URL (Uniform Resource Locator) information, authentication information (e.g., user name, password), port information, schema information, and data table information (e.g., table name, column name of the data table). The second type of parameter includes, but is not limited to, one or any combination of the following: and URL information, authentication information, port information, schema information and data table information corresponding to the second sub-database.
In an example, the first data source stores information used when the first key client establishes a connection with the first sub-database, such as URL information, authentication information, port information, schema information, and data table information corresponding to the first sub-database. Based on the URL information corresponding to the first sub-database, the first key client can know which sub-database the connection should be established with. Based on the authentication information and the port information corresponding to the first sub-database, the first key client can access the first sub-database and establish connection with the first sub-database. Based on the schema information and the data table information corresponding to the first sub-database, the first button client may know from which data table of the first sub-database the data is extracted. For example, assuming that the schema information is schema1, and the data table information is data table 1 and column 20 of data table 1, since one or more schemas may be created under the first sub-database and each schema may include one or more data tables, the first key client may extract schema1 from all schemas in the first sub-database and extract data table 1 from all data tables in schema1, and the first key client needs to extract all data in column 20 of data table 1.
In an example, the second data source stores information used when the second key client establishes a connection with the second sub-database, such as URL information, authentication information, port information, schema information, and data table information corresponding to the second sub-database. Based on the URL information corresponding to the second sub-database, the second key client can know which sub-database the connection should be established with. Based on the authentication information and the port information corresponding to the second sub-database, the second key client can access the second sub-database and establish connection with the second sub-database. Based on the schema information and the data table information corresponding to the second sub-database, the second button client may know which schema of the second sub-database the data table is established under. For example, assuming that the schema information is schema2 and the data table information is data table 2, the second key client may establish one data table 2 under the schema2 of the second sub-database, and may store the data extracted from the central database into the data table 2.
Step 202, sending a first control message to the first key client, so that the first key client extracts data from the first sub-database according to the first data source, and stores the extracted data in the central database.
In one example, the first control message is used to instruct the start of data extraction, and the first key client knows that the data extraction can be started after receiving the first control message. The first key client may establish a connection with the first sub-database based on the first data source, and extract data from the specified data table of the first sub-database based on the first data source, such as extracting all data in the 20 th column of the data table 1. Furthermore, after the first key client extracts the data, the extracted data may be stored in a central database.
And step 203, sending a second control message to the second key client, so that the second key client extracts data from the central database and stores the extracted data into a second sub-database according to the second data source.
In one example, the second control message is used to instruct the start of data extraction, and the second key client knows that the data extraction can be started after receiving the second control message. The second key client may establish a connection with the second sub-database based on the second data source, extract data from the central database, and store the extracted data in a designated data table of the second sub-database based on the second data source, such as establishing a data table 2 under the schema2 of the second sub-database, and store the extracted data in the data table 2.
In one example, the first control message and the second control message may also carry data table information, such as data table 3, where data table 3 is a data table of the central database. The first key client may store the extracted data to table 3 of the central database and the second key client may extract the data from table 3 of the central database.
In an example, after sending the first control message to the first key client, before sending the second control message to the second key client, the server may further receive statistical information sent by the first key client (for example, the server may periodically send a query message to the first key client, and the first key client returns current statistical information of the first key client to the server each time the query message is received), where the statistical information may include, but is not limited to, an extraction state, and the extraction state is incomplete or completed.
In one example, if the extraction status is incomplete, it indicates that the first key client has not extracted all the data to be extracted, and therefore the server prohibits sending the second control message to the second key client. If the extraction state is complete, it indicates that the first key client has already extracted all the data to be extracted, so the server may send a second control message to the second key client. In another example, the server may also display the extraction status to the user, and the user may decide whether to send the second control message to the second key client based on the extraction status; if the server receives a command that a second control message needs to be sent to the second button client, the server sends the second control message to the second button client; and if the server receives a command that the second control message does not need to be sent to the second key client, the server prohibits sending the second control message to the second key client.
In an example, before sending the first type of parameters to the first key client, the server may further receive a user request message through a WEB page, where the user request message may carry address information (e.g., an IP address) of the first key client and address information of the second key client. Based on the address information, the server can send the first type of parameters and the first control message to the first button client based on the address information of the first button client; the server may send the second type parameter and the second control message to the second key client based on the address information of the second key client.
In an example, after receiving the statistical information sent by the first button client, the server may also display the statistical information through a WEB page, so that the user views the statistical information. Further, the statistical information may include, but is not limited to, one or any combination of the following in addition to the extraction status: extraction results (such as extraction success or extraction failure), log information, history information, execution time, the number of currently extracted data, the number of data extracted in error, the currently extracted data rate, and the like.
In one example, the first and second key clients are key type clients, and the key is an open-source ETL tool, and can run on windows, Linux, Unix and other operating systems, and a visual programming mode is adopted for data processing, so that the operation is convenient and easy, and the data extraction is efficient and stable.
In a conventional manner, a button client runs in a shoon mode, which is an Integrated Development Environment (IDE), and a user needs to log in to the button client to perform functions such as creating/editing a job or conversion, executing/debugging a job or conversion, and the like.
In the embodiment of the present application, the button client operates in a designated mode (e.g., Carte mode). For the button client operating in the Carte mode, the user can trigger the button client to execute the functions of creating/editing operation or conversion, executing/debugging operation or conversion and the like through the server without logging in the button client. The client has an HTTP service function, and runs a task by monitoring a request of the server, based on which, the server may send an HTTP type request message (such as a message carrying a first type of parameter, a message carrying a second type of parameter, a first control message, a second control message, etc.) to the client. The key client may run a related task after listening to the HTTP-type request message, for example, after listening to the HTTP-type first control message, execute a task "extract data from the first sub-database according to the first data source, store the extracted data in the central database".
Based on the technical scheme, in the embodiment of the application, the server side sends the first type of parameters to the first key client side so that the first key client side creates a first data source according to the first type of parameters, and sends the first control message to the first key client side so that the first key client side extracts data from the first sub-database according to the first data source and stores the extracted data in the central database; and the server side sends the second type parameters to the second key client side so that the second key client side creates a second data source according to the second type parameters, sends a second control message to the second key client side so that the second key client side extracts data from the central database, and stores the extracted data into a second sub-database according to the second data source. Based on the mode, the server side can manage the plurality of clients without logging in the clients, the functions of the plurality of clients are controlled remotely, user operation is simplified, maintenance and management of the clients are facilitated, workload of the users is reduced, and error rate caused by user operation is reduced.
The following describes the technical solution of the embodiment of the present application with reference to the application scenario shown in fig. 3. The key client 1 is installed on the device 1, and the key client 2 is installed on the device 2. Assuming that data of the sub-database 1 needs to be exchanged to the sub-database 3, the sub-database 1 is a first sub-database, the sub-database 3 is a second sub-database, the key client 1 is a first key client, and the key client 2 is a second key client.
As shown in fig. 3, the server is used as a control center for the key client 1 and the key client 2, and has a WEB page function, and a user can manage and control the server through the WEB page, and then trigger the server to manage and control the key client 1 and the key client 2. The service end may provide a WEB processing framework (e.g., a WEB processing framework based on Spring and Hibernate, etc.), so as to process the WEB page request. Moreover, the server can also control the key client 1 and the key client 2 to complete operations such as data source creation, data conversion and job by sending HTTP type request messages to the key client 1 and the key client 2. Moreover, the server can also display statistical information such as extraction state, extraction result, log information, history information, execution time, currently extracted data quantity, extraction error data quantity, currently extracted data rate and the like through a WEB page.
As shown in fig. 3, the key client 1 and the key client 2 run in the Carte mode, that is, run a background lightweight service (such as an HTTP service), and for the key client running in the Carte mode, the user can trigger the key client to execute creating/editing jobs or conversion, execute/debug jobs or conversion, and other functions through the server without logging in to the key client. Moreover, the button client has the function of HTTP service, and can run tasks by monitoring the request of the server. The server side sends HTTP type request information to the button client side to realize management and control of the button client side. Furthermore, the key client can run related tasks after listening to the HTTP type request message.
In order to make the key clients 1 and 2 operate in the Carte mode, a Jetty RESTful (Representational State Transfer) API (Application Programming Interface) request service may be added to the key clients 1 and 2. For example, a ConnectionResource class is added to process requests related to the data source, including operations of creating, updating, and deleting the data source. For other functions, the function can also be realized by adding a ConnectionResource class, which is not described again. Finally, the key client 1 and the key client 2 can be enabled to process related functions such as data source, conversion and job.
In the application scenario, the data exchange method provided in the embodiment of the present application may include the following steps:
step 1, a user opens a WEB interface of a server and operates various configuration items.
And 2, the server receives a user request message through a WEB page, wherein the user request message can carry the IP address 1 of the button client 1 and the IP address 2 of the button client 2.
In the process that the user operates various configuration items, the user can input the IP address 1 of the button client 1 and the IP address 2 of the button client 2, so that the service end can receive a user request message carrying the IP address 1 and the IP address 2 through a WEB page and store the IP address 1 and the IP address 2.
And 3, the server side determines that the data in the sub-database 1 needs to be exchanged to the sub-database 3. For example, data in data table 1 of sub-database 1 needs to be swapped into data table 2 of sub-database 3.
And 4, based on the IP address 1 of the client 1, the server sends a request message carrying first-class parameters to the client 1, wherein the first-class parameters can include URL information, authentication information (such as a user name and a password), port information, schema information and data table information (such as a data table 1) of the sub-database 1.
And 5, after receiving the first type parameters, the button client 1 creates a first data source aiming at the sub-database 1 according to the first type parameters. The first data source stores information used when the button client 1 establishes a connection with the sub-database 1, such as URL information, authentication information, port information, schema information (e.g., schema1), and data table information (e.g., data table 1) of the sub-database 1.
Based on the URL information corresponding to the sub-database 1, the key client 1 can know to establish a connection with the sub-database 1. Based on the authentication information and the port information corresponding to the sub-database 1, the key client 1 can access the sub-database 1 and establish a connection with the sub-database 1. Based on the schema1 and the data table 1 corresponding to the sub-database 1, the key client 1 can know that data is extracted from the data table 1 of the schema1 of the sub-database 1.
And 6, based on the IP address 2 of the key client 2, the server sends a request message carrying second type parameters to the key client 2, wherein the second type parameters can include URL information, authentication information (such as a user name and a password), port information, schema information and data table information (such as a data table 2) of the sub-database 3.
And 7, after receiving the second type of parameters, the button client 2 creates a second data source for the sub-database 3 according to the second type of parameters. The second data source stores information used when the button client 2 establishes a connection with the sub-database 3, such as URL information, authentication information, port information, schema information (e.g., schema2), and data table information (e.g., data table 2) of the sub-database 3.
Based on the URL information corresponding to the sub-database 3, the key client 2 can know to establish a connection with the sub-database 3. Based on the authentication information and the port information corresponding to the sub-database 3, the key client 2 can access the sub-database 3 and establish a connection with the sub-database 3. Based on the schema2 and the data table 2 corresponding to the sub-database 3, the key client 2 can know that a data table 2 is established under the schema2 of the sub-database 3, and store the data extracted from the central database (the extraction process is described later) into the data table 2.
And step 8, the server determines to start to exchange the data in the sub-database 1 into the sub-database 3.
And 9, the server sends a first control message to the button client 1, wherein the first control message carries the data table 3.
And step 10, after receiving the first control message, the button client 1 extracts data from the data table 1 of the sub-database 1 according to the first data source, and stores the extracted data in the data table 3 of the central database.
And step 11, in the data extraction process, the server periodically sends a query message to the button client 1, and the button client 1 returns the current statistical information to the server each time the query message is received. Wherein, the statistical information may include but is not limited to one or any combination of the following: extraction status (e.g., incomplete or completed), extraction results, log information, history information, execution time, number of data currently extracted, number of data extracted in error, data rate currently extracted, etc.
And step 12, after receiving the statistical information, the server displays the statistical information through a WEB page.
After the statistical information is displayed through the WEB page, a user can conveniently check the statistical information and know the state of the whole data exchange so as to judge how to perform the subsequent data exchange.
And step 13, the server judges whether the current extraction state is finished. If yes, step 14 is executed, if no, the statistical information of the key client 1 is continuously acquired, and then step 13 is executed.
In one example, after the server displays the extraction status through the WEB page, if a command for sending the second control message to the key client 2 is received, step 14 may be executed.
And step 14, the server sends a second control message to the button client 2, wherein the second control message carries the data table 3.
The server side sends a first control message to the key client 1 based on the IP address 1 of the key client 1, and sends a second control message to the key client 2 based on the IP address 2 of the key client 2.
And step 15, after receiving the second control message, the key client 2 extracts data from the data table 3 of the central database according to the second data source, and stores the extracted data in the data table 2 of the sub-database 3.
And step 16, in the data extraction process, the server periodically sends a query message to the button client 2, and the button client 2 returns the current statistical information to the server each time the query message is received. Wherein, the statistical information may include but is not limited to one or any combination of the following: extraction status (e.g., incomplete or completed), extraction results, log information, history information, execution time, number of data currently extracted, number of data extracted in error, data rate currently extracted, etc.
And step 17, after receiving the statistical information, the server displays the statistical information through a WEB page.
After the statistical information is displayed through the WEB page, a user can conveniently check the statistical information and know the state of the whole data exchange so as to judge how to perform the subsequent data exchange.
And step 18, the server judges whether the current extraction state is finished. If yes, the flow is ended, if not, the statistical information of the key client 2 is continuously acquired, and then step 18 is executed.
Based on the technical scheme, in the embodiment of the application, the multiple button clients can be managed by the server side without logging in the button clients by a user, the functions of the multiple button clients are controlled remotely, the user operation is simplified, the maintenance and management of the button clients are facilitated, the workload of the user is reduced, and the error rate caused by the user operation is reduced. Moreover, the information of each button client can be displayed on a WEB page of the server, and a user can inquire the running state and the result of each button client, so that the maintenance operation of managers is greatly simplified, and the real-time dynamic expansion of the button clients is supported. Moreover, the terminal device can log in the WEB interface of the server, and configure and manage all the button clients on the WEB interface.
Based on the same application concept as the method, the embodiment of the application also provides a data exchange device, and the data exchange device can be applied to a server. The data exchange device may be implemented by software, or may be implemented by hardware, or a combination of hardware and software. Taking a software implementation as an example, a device in a logical sense is formed by reading corresponding computer program instructions in a nonvolatile memory through a processor of a service end where the device is located. From a hardware aspect, as shown in fig. 4, for a hardware structure diagram of a service end where a data exchange device provided by the present application is located, in addition to the processor and the nonvolatile memory shown in fig. 4, the service end may further include other hardware, such as a forwarding chip, a network interface, and a memory, which are responsible for processing a packet; in terms of hardware structure, the server may also be a distributed device, and may include multiple interface cards, so as to perform an extension of message processing on a hardware level.
As shown in fig. 5, for the structure diagram of the data exchange device provided in the present application, the data exchange device is applied to a server in a data exchange system, the data exchange system further includes at least two key clients, the key clients operate in a specified mode, and in the specified mode, the key clients have a hypertext transfer protocol HTTP service function and operate a task matching with a request by monitoring the request of the server, the device includes:
a first sending module 11, configured to send a first type of parameter to a first key client of the at least two key clients, so that the first key client creates a first data source for a first sub-database according to the first type of parameter;
a second sending module 12, configured to send a second type of parameter to a second key client of the at least two key clients, so that the second key client creates a second data source for a second sub-database according to the second type of parameter;
a third sending module 13, configured to send a first control message to the first key client, so that the first key client extracts data from the first sub-database according to the first data source, and stores the extracted data in a central database;
a fourth sending module 14, configured to send a second control message to the second key client, so that the second key client extracts data from the central database, and stores the extracted data in the second sub-database according to the second data source.
In one example, the data exchange device further comprises (not represented in the figure): a receiving module; the client side is used for receiving statistical information sent by a first key client side after the third sending module sends a first control message to the first key client side and before the fourth sending module sends a second control message to a second key client side, wherein the statistical information comprises an extraction state; the fourth sending module 14 is specifically configured to prohibit sending the second control message to the second key client if the extraction state is incomplete in a process of sending the second control message to the second key client; and if the extraction state is finished, sending a second control message to the second button client.
In one example, the first type parameter includes one or any combination of the following: uniform resource locator URL information, authentication information, port information, schema information and data table information corresponding to the first sub-database; the second type of parameter comprises one or any combination of the following: URL information, authentication information, port information, schema information and data table information corresponding to the second sub-database; the statistical information further comprises one or any combination of the following: extraction result, log information, history information, execution time, currently extracted data amount, and currently extracted data rate.
In an example, the receiving module is further configured to receive a user request message through a WEB page before the first sending module sends the first type of parameter to a first key client of the at least two key clients, where the user request message carries address information of the first key client and address information of the second key client.
In one example, the data exchange device further comprises (not represented in the figure): a display module;
and the display module is used for displaying the statistical information through the WEB page after the receiving module receives the statistical information sent by the first button client.
The systems, devices, modules or units illustrated in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, which may take the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email messaging device, game console, tablet computer, wearable device, or a combination of any of these devices.
For convenience of description, the above devices are described as being divided into various units by function, and are described separately. Of course, the functionality of the units may be implemented in one or more software and/or hardware when implementing the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Furthermore, these computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (which may include, but is not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A data exchange method is applied to a server side in a data exchange system, and is characterized in that the data exchange system also comprises at least two button clients, and the method comprises the following steps:
receiving a user request through a WEB page; the user request comprises at least two address information;
responding to the user request, and determining a target button client according to the at least two pieces of address information; the target key client operates in a mode supporting a hypertext transfer protocol (HTTP) service function;
and sending the HTTP request to the target key client so that a first key client in the target key client transfers the data extracted from the first sub-database to a second sub-database through a second key client.
2. The data exchange method of claim 1, wherein the HTTP request comprises: a first control message, a second control message;
sending an HTTP request to the target key client, comprising:
sending a first control message to the first key client so that the first key client extracts data from the first sub-database and stores the extracted data in a central database;
and sending a second control message to the second key client so that the second key client extracts data from the central database and stores the extracted data in the second sub-database.
3. The data exchange method of claim 2, wherein the HTTP request further comprises: a first type parameter and a second type parameter;
sending the HTTP request to the target key client, further comprising:
sending a first type parameter to the first key client, so that the first key client creates a first data source aiming at a first sub-database according to the first type parameter; the first data source is used for the first button client to extract data from the first sub-database;
sending a second type of parameters to the second key client, so that the second key client creates a second data source for a second sub-database according to the second type of parameters; and the second data source is used for the second key client to store the extracted data into the second sub-database.
4. The method of claim 3, wherein the first type of parameter comprises one or any combination of the following: uniform resource locator URL information, authentication information, port information, schema information and data table information corresponding to the first sub-database;
the second type of parameter comprises one or any combination of the following: and URL information, authentication information, port information, schema information and data table information corresponding to the second sub-database.
5. The method of claim 1, wherein the button client requests the service to switch to the HTTP service function mode through a representational state transfer interface.
6. A data exchange device is applied to a server side in a data exchange system, and is characterized in that the data exchange system further comprises at least two button clients, and the device comprises:
the receiving module is used for receiving a user request through a WEB page; the user request comprises at least two address information;
the determining module is used for responding to the user request and determining a target button client according to the at least two pieces of address information; the target key client operates in a mode supporting a hypertext transfer protocol (HTTP) service function;
and the sending module is used for sending the HTTP request to the target key client so that a first key client in the target key client transfers the data extracted from the first sub-database to a second sub-database through a second key client.
7. The data exchange apparatus of claim 6, wherein the HTTP request comprises: a first control message, a second control message;
the sending module comprises:
the first sending unit is used for sending a first control message to the first key client so that the first key client extracts data from the first sub-database and stores the extracted data in a central database;
and the second sending unit is used for sending a second control message to the second key client so that the second key client extracts data from the central database and stores the extracted data in the second sub-database.
8. The data exchange apparatus of claim 7, wherein the HTTP request further comprises: a first type parameter and a second type parameter;
the sending module further comprises:
a third sending unit, configured to send a first type of parameter to the first key client, so that the first key client creates a first data source for the first sub-database according to the first type of parameter; the first data source is used for the first button client to extract data from the first sub-database;
a fourth sending unit, configured to send a second type of parameter to the second key client, so that the second key client creates a second data source for the second sub-database according to the second type of parameter; and the second data source is used for the second key client to store the extracted data into the second sub-database.
9. The apparatus of claim 8, wherein the first type of parameter comprises one or any combination of the following: uniform resource locator URL information, authentication information, port information, schema information and data table information corresponding to the first sub-database;
the second type of parameter comprises one or any combination of the following: and URL information, authentication information, port information, schema information and data table information corresponding to the second sub-database.
10. The apparatus of claim 6, wherein the button client requests the service to switch to the HTTP service function mode through a representational state transfer interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910833291.5A CN110661851A (en) | 2017-03-02 | 2017-03-02 | Data exchange method and device |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710121280.5A CN106888264B (en) | 2017-03-02 | 2017-03-02 | A kind of method for interchanging data and device |
| CN201910833291.5A CN110661851A (en) | 2017-03-02 | 2017-03-02 | Data exchange method and device |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710121280.5A Division CN106888264B (en) | 2017-03-02 | 2017-03-02 | A kind of method for interchanging data and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110661851A true CN110661851A (en) | 2020-01-07 |
Family
ID=59180308
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910833291.5A Pending CN110661851A (en) | 2017-03-02 | 2017-03-02 | Data exchange method and device |
| CN201710121280.5A Active CN106888264B (en) | 2017-03-02 | 2017-03-02 | A kind of method for interchanging data and device |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710121280.5A Active CN106888264B (en) | 2017-03-02 | 2017-03-02 | A kind of method for interchanging data and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN110661851A (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102422032B1 (en) * | 2017-08-16 | 2022-07-19 | 에스케이하이닉스 주식회사 | Memory system and operating method of memory system |
| CN107563219B (en) * | 2017-08-28 | 2020-04-24 | 深圳市华傲数据技术有限公司 | Database management method and device |
| CN110888636A (en) * | 2019-12-03 | 2020-03-17 | 中电工业互联网有限公司 | ETL Web application system architecture method based on button |
| CN114143043A (en) * | 2021-11-10 | 2022-03-04 | 杭州数梦工场科技有限公司 | Data secure transmission method and device and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN103455526A (en) * | 2012-06-05 | 2013-12-18 | 杭州勒卡斯广告策划有限公司 | ETL (extract-transform-load) data processing method, device and system |
| CN103902593A (en) * | 2012-12-27 | 2014-07-02 | 中国移动通信集团河南有限公司 | Data transfer method and device |
| CN104933202A (en) * | 2015-07-15 | 2015-09-23 | 北京京东尚科信息技术有限公司 | Method and device for data extraction, transformation and ETL loading |
| US20160085764A1 (en) * | 2014-09-22 | 2016-03-24 | Oracle Financial Services Software Limited | Simplifying invocation of import procedures to transfer data from data sources to data targets |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105701094B (en) * | 2014-11-24 | 2019-03-19 | 北京航管科技有限公司 | A kind of ETL collecting method and device |
| CN105069142B (en) * | 2015-08-18 | 2019-04-05 | 山大地纬软件股份有限公司 | Data increment extraction conversion and dissemination system and method |
| CN105389131A (en) * | 2015-11-23 | 2016-03-09 | 江苏瑞中数据股份有限公司 | Data acquisition method for process industry production system |
| CN106126753B (en) * | 2016-08-23 | 2019-03-05 | 易联众信息技术股份有限公司 | The method of increment extraction based on big data |
-
2017
- 2017-03-02 CN CN201910833291.5A patent/CN110661851A/en active Pending
- 2017-03-02 CN CN201710121280.5A patent/CN106888264B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102999537A (en) * | 2011-09-19 | 2013-03-27 | 阿里巴巴集团控股有限公司 | System and method for data migration |
| CN103455526A (en) * | 2012-06-05 | 2013-12-18 | 杭州勒卡斯广告策划有限公司 | ETL (extract-transform-load) data processing method, device and system |
| CN103902593A (en) * | 2012-12-27 | 2014-07-02 | 中国移动通信集团河南有限公司 | Data transfer method and device |
| US20160085764A1 (en) * | 2014-09-22 | 2016-03-24 | Oracle Financial Services Software Limited | Simplifying invocation of import procedures to transfer data from data sources to data targets |
| CN104933202A (en) * | 2015-07-15 | 2015-09-23 | 北京京东尚科信息技术有限公司 | Method and device for data extraction, transformation and ETL loading |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106888264B (en) | 2019-09-17 |
| CN106888264A (en) | 2017-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110971655B (en) | Offline client playback and synchronization | |
| US20180365254A1 (en) | Method and apparatus for processing information flow data | |
| CN111930521A (en) | Method and device for deploying application, electronic equipment and readable storage medium | |
| CN108259533B (en) | Data transmission method and device | |
| WO2021203979A1 (en) | Operation and maintenance processing method and apparatus, and computer device | |
| CN110661851A (en) | Data exchange method and device | |
| US8326913B2 (en) | Method and system for service contract discovery | |
| CN109245908A (en) | A kind of method and apparatus of principal and subordinate's cluster switching | |
| CN113127732A (en) | Method and device for acquiring service data, computer equipment and storage medium | |
| CN109828830B (en) | Method and apparatus for managing containers | |
| US10033737B2 (en) | System and method for cross-cloud identity matching | |
| CN113626002A (en) | Service execution method and device | |
| CN109391658B (en) | Account data synchronization method and equipment, storage medium and terminal thereof | |
| US9577967B2 (en) | Method and system for managing an informational site using a social networking application | |
| CN112069137A (en) | Method and device for generating information, electronic equipment and computer readable storage medium | |
| CN109921920A (en) | A kind of failure information processing method and relevant apparatus | |
| CN104219280A (en) | Intelligent application data transmission channel | |
| CN109213955B (en) | Data processing method and related equipment | |
| CN109040331B (en) | Electronic business card processing method and device, computing equipment and storage medium | |
| US10713278B2 (en) | Flexible configuration of offline synchronization scope | |
| CN112231408B (en) | Data integration method and device, electronic equipment and storage medium | |
| CN115567596A (en) | Cloud service resource deployment method, device, equipment and storage medium | |
| CN102904742B (en) | To method of operation and the system of executable node | |
| US10397072B2 (en) | Systems, devices, and methods for incident management interactions | |
| CN103067365A (en) | Set top box, client-side, system and method for virtual desktop access |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200107 |