CN112749219A

CN112749219A - Data extraction method, data extraction device, electronic equipment, storage medium and program product

Info

Publication number: CN112749219A
Application number: CN202110004077.6A
Authority: CN
Inventors: 不公告发明人
Original assignee: Lakala Payment Co ltd
Current assignee: Lakala Payment Co ltd
Priority date: 2021-01-04
Filing date: 2021-01-04
Publication date: 2021-05-04

Abstract

The embodiment of the disclosure discloses a data extraction method, a data extraction device, an electronic device, a storage medium and a program product, wherein the method comprises the following steps: receiving a data extraction request from a client; the data extraction request comprises an identifier and configuration parameters of the target data extraction template; configuring the target data extraction template by using the configuration parameters; running an execution script of the target data extraction template so as to acquire data from one or more production systems by using the configured target data extraction template; storing the data obtained from the production system to a big data cluster system. According to the technical scheme, the data extraction efficiency of the big data cluster system is improved, the script development time in the data extraction process is saved, and the operation and maintenance flow of the big data cluster system is simplified.

Description

Data extraction method, data extraction device, electronic equipment, storage medium and program product

Technical Field

The embodiment of the disclosure relates to the technical field of computers, in particular to a data extraction method, a data extraction device, electronic equipment, a storage medium and a program product.

Background

The large data cluster platform needs to centralize data of a core production system, but in some large enterprises, the production systems have more than one and may be hundreds, and data tables generated by the production systems can be ten thousand levels, so that the process of centralizing the production systems is subject to adjustment. One common method is to correspond one table in the production system to one data extraction script in the big data cluster platform, and when a new data table is added to the production system, a background manager of the big data cluster platform needs to develop the data extraction script once, so that the development efficiency is extremely low, and the operation and maintenance work is extremely complicated. Therefore, how to improve the process of extracting data from a production system by a large data cluster platform and improve the data extraction efficiency is one of the technical problems that needs to be solved currently in the field.

Disclosure of Invention

The embodiment of the disclosure provides a data extraction method, a data extraction device, electronic equipment, a storage medium and a program product.

In a first aspect, an embodiment of the present disclosure provides a data extraction method, including:

receiving a data extraction request from a client; the data extraction request comprises an identifier and configuration parameters of the target data extraction template;

configuring the target data extraction template by using the configuration parameters;

running an execution script of the target data extraction template so as to acquire data from one or more production systems by using the configured target data extraction template;

storing the data obtained from the production system to a big data cluster system.

Further, the target data extraction template comprises a full data extraction template and an incremental data extraction template.

Further, the method further comprises:

and receiving the established full-volume data extraction template and the incremental data extraction template from the client.

Further, the method further comprises:

receiving the execution script running the full data extraction template and the incremental data extraction template from a client.

Further, the receiving a data extraction request from a client includes:

receiving an identification of a full data extraction template and configuration parameters of corresponding parameter options in the full data extraction template from a client; or,

and receiving the identification of the incremental data extraction template and the configuration parameters of the corresponding parameter options in the incremental data extraction template from the client.

Further, the receiving a data extraction request from a client includes:

receiving a template configuration request from a client;

determining a current data extraction stage; the data extraction stage comprises a full data extraction stage and an incremental data extraction stage;

when the current data extraction stage is a full data extraction stage, returning an instruction for configuring the full data extraction template to the client;

when the current data extraction stage is an incremental data extraction stage, returning an indication for configuring the incremental data extraction template to the client;

the configuration parameters are received from a client.

Further, configuring the target data extraction template by using the configuration parameters includes:

and acquiring the configured target data extraction template in a mode of filling the configuration parameters into corresponding parameter options in the target data extraction template.

Further, the method further comprises:

acquiring an execution script from a client and storing the execution script; the execution script is used for executing the full data extraction template and the incremental data extraction template.

Further, running an execution script of the target data extraction template to acquire data from a production system by using the configured target data extraction template includes:

acquiring the corresponding execution script from a storage device according to the identifier of the target data extraction template;

and running the execution script.

sending a request for acquiring the execution script to a client;

and receiving the execution script from the client and running the execution script.

Further, storing the data obtained from the production system to a big data cluster system, comprising:

receiving returned data from one or more of the production systems;

and merging the data according to a preset rule and storing the merged data to a data warehouse of the big data cluster system.

In a second aspect, an embodiment of the present disclosure provides a data extraction method, including:

detecting the configuration operation of a user on a data extraction template;

responding to the detected configuration operation, displaying a configuration interface, and acquiring the identification and configuration parameters of the target data extraction template from the configuration interface;

sending a data extraction request to a background management server so that the background management server can acquire data from one or more production systems based on the target data extraction template and the configuration parameters and store the acquired data in a big data cluster system; the data extraction request comprises an identification of the target data extraction template and configuration parameters.

Further, the method further comprises:

responding to a request of a user for creating a target data extraction template, and displaying a creation page for creating the target data extraction template;

receiving the content of a target data extraction template from the creation page;

and generating a target data extraction template according to the content, wherein the target data extraction template comprises a full data extraction template or an incremental data extraction template.

Further, in response to the detected configuration operation, presenting a configuration interface, and acquiring an identifier and configuration parameters of the target data extraction template from the configuration interface, includes:

in response to the detected configuration operation, displaying options of a full data extraction template and an incremental data extraction template on a configuration interface;

receiving the selection of the user on the option, and determining the identification of the target data extraction template according to the selection;

displaying an input interface of a corresponding parameter option in the target data extraction template on a configuration interface;

receiving the configuration parameters from the input interface.

in response to the detected configuration operation, sending a template configuration request to the background management server;

receiving an indication of configuring a target full-size data extraction template from the backend management server; the target full-volume data extraction template is a full-volume data extraction template or an incremental data extraction template;

receiving the configuration parameters from the input interface;

and sending the configuration parameters to the background management server.

Further, the method further comprises:

responding to a request of a user for creating an execution script of the target data extraction template, and displaying a script writing page for creating the execution script;

obtaining script statements from the script compiling page and generating an execution script of the target data extraction template;

and storing the execution script and/or sending the execution script to a background management server.

In a third aspect, an embodiment of the present disclosure provides a data extraction method, including:

the method comprises the steps that a client detects configuration operation of a user on a data extraction template, responds to the detected configuration operation, displays a configuration interface, and obtains identification and configuration parameters of a target data extraction template from the configuration interface;

the client sends a data extraction request to the background management server; the data extraction request comprises an identifier and configuration parameters of the target data extraction template;

the background management server receives a data extraction request from a client and configures the target data extraction template by using the configuration parameters;

the background management server runs an execution script of the target data extraction template so as to acquire data from one or more production systems by using the configured target data extraction template;

and the background management server stores the data acquired from the production system to a big data cluster system.

Further, the method further comprises:

the client responds to a request of a user for creating a target data extraction template, shows a creation page for creating the target data extraction template, receives the content of the target data extraction template from the creation page, and generates the target data extraction template according to the content, wherein the target data extraction template comprises a full data extraction template or an incremental data extraction template;

and the client sends the created target data extraction template to the background management server.

the client-side responds to the detected configuration operation and shows options of a full data extraction template and an incremental data extraction template on a configuration interface;

the client receives the selection of the user on the option and determines the identification of the target data extraction template according to the selection;

the client displays an input interface of a corresponding parameter option in the target data extraction template on a configuration interface;

the client receives the configuration parameters from the input interface.

the client side responds to the detected configuration operation and sends a template configuration request to the background management server;

the client receives an indication of configuring a target full-data extraction template from the background management server; the target full-volume data extraction template is a full-volume data extraction template or an incremental data extraction template;

the client receives the configuration parameters from the input interface;

and the client sends the configuration parameters to the background management server.

Further, the method further comprises:

the client responds to a request of a user for creating an execution script of the target data extraction template, and shows a script writing page for creating the execution script;

the client acquires script statements from the script compiling page and generates an execution script of the target data extraction template;

and the client stores the execution script and/or sends the execution script to a background management server.

Further, the background management server receives a data extraction request from a client, and the data extraction request comprises:

the background management server receives an identifier of a full data extraction template and configuration parameters of corresponding parameter options in the full data extraction template from a client; or,

and the background management server receives the identification of the incremental data extraction template and the configuration parameters of the corresponding parameter options in the incremental data extraction template from a client.

the background management server receives a template configuration request from a client;

the background management server determines the current data extraction stage; the data extraction stage comprises a full data extraction stage and an incremental data extraction stage;

when the current data extraction stage is a full data extraction stage, the background management server returns an indication for configuring the full data extraction template to the client;

when the current data extraction stage is an incremental data extraction stage, the background management server returns an indication for configuring the incremental data extraction template to the client;

the background management server receives the configuration parameters from a client.

and the background management server acquires the configured target data extraction template in a mode of filling the configuration parameters into corresponding parameter options in the target data extraction template.

Further, the method further comprises:

the background management server acquires an execution script from a client and stores the execution script; the execution script is used for executing the full data extraction template and the incremental data extraction template.

Further, the background management server runs an execution script of the target data extraction template so as to obtain data from a production system by using the configured target data extraction template, and the method includes:

the background management server acquires the corresponding execution script from the storage device according to the identification of the target data extraction template;

and the background management server runs the execution script.

the background management server sends a request for acquiring the execution script to a client;

and the background management server receives the execution script from the client and runs the execution script.

Further, the background management server stores the data acquired from the production system to a big data cluster system, including:

the background management server receives returned data from one or more production systems;

and the background management server combines the data according to a preset rule and stores the data to a data warehouse of the big data cluster system.

In a fourth aspect, an embodiment of the present disclosure provides a data extraction apparatus, including:

a receiving module configured to receive a data extraction request from a client; the data extraction request comprises an identifier and configuration parameters of the target data extraction template;

a configuration module configured to configure the target data extraction template with the configuration parameters;

the running module is configured to run the execution script of the target data extraction template so as to acquire data from one or more production systems by using the configured target data extraction template;

a storage module configured to store the data obtained from the production system to a big data cluster system.

In a fifth aspect, an embodiment of the present disclosure provides a data extraction apparatus, including:

the detection module is configured to detect the configuration operation of a user on the data extraction template;

the response module is configured to respond to the detected configuration operation, display a configuration interface, and acquire the identification and the configuration parameters of the target data extraction template from the configuration interface;

a sending module configured to send a data extraction request to a background management server; the data extraction request comprises identification of the target data extraction template and configuration parameters, so that the background management server can acquire data from one or more production systems based on the target data extraction template and the configuration parameters and store the acquired data in a big data cluster system.

The functions can be realized by hardware, and the functions can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above-described functions.

In one possible design, the apparatus includes a memory configured to store one or more computer instructions that enable the apparatus to perform the corresponding method, and a processor configured to execute the computer instructions stored in the memory. The apparatus may also include a communication interface for the apparatus to communicate with other devices or a communication network.

In a sixth aspect, an embodiment of the present disclosure provides a data extraction system, including: the system comprises a client and a background management server;

the client detects the configuration operation of a user on the data extraction template, responds to the detected configuration operation, displays a configuration interface, and acquires the identification and the configuration parameters of the target data extraction template from the configuration interface;

In a seventh aspect, an embodiment of the present disclosure provides an electronic device, including a memory and a processor, where the memory is used to store one or more computer instructions that support any of the above apparatuses to perform the corresponding methods described above, and the processor is configured to execute the computer instructions stored in the memory. Any of the above may also include a communication interface for communicating with other devices or a communication network.

In an eighth aspect, the present disclosure provides a computer-readable storage medium for storing computer instructions for use by any one of the above apparatuses, which includes computer instructions for performing any one of the above methods.

In a ninth aspect, the disclosed embodiments provide a computer program product comprising computer instructions for implementing the steps of the method of any one of the above aspects when executed by a processor.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects:

according to the technical scheme, in the process that the big data cluster system extracts data from a plurality of production systems, a target data extraction template and an execution script are pre-established, when data extraction is needed, relevant personnel configure the target data extraction template from a client side, and send configuration parameters of the target data extraction template to a background management server, so that the background management server configures the target data extraction template through the configuration parameters, and extracts needed data from one or more production systems according to the configured target data extraction template by running the corresponding execution script. Through the mode, the data extraction task is abstracted to the mode of the target data extraction template, so that the data extraction script is not required to be re-developed aiming at a newly increased data table in the production system, and only the target data extraction template is configured, and the execution script can be a universal execution script and can be used for executing any target data extraction template, the data extraction efficiency of the big data cluster system is improved, the script development time in the data extraction process is saved, and the operation and maintenance flow of the big data cluster system is simplified.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of embodiments of the disclosure.

Drawings

Other features, objects, and advantages of embodiments of the disclosure will become more apparent from the following detailed description of non-limiting embodiments when taken in conjunction with the accompanying drawings. In the drawings:

FIG. 1 illustrates a flow diagram of a data extraction method according to an embodiment of the present disclosure;

FIG. 2 shows a flow diagram of a data extraction method according to another embodiment of the present disclosure;

FIG. 3 shows a flow diagram of a data extraction method according to another embodiment of the present disclosure;

FIG. 4 is a schematic diagram illustrating an application scenario of a data extraction method according to an embodiment of the present disclosure;

FIG. 5 illustrates an overall flow diagram of a data extraction method according to an embodiment of the present disclosure;

FIG. 6 shows a block diagram of a data extraction device according to an embodiment of the present disclosure;

FIG. 7 shows a block diagram of a data extraction system according to another embodiment of the present disclosure;

FIG. 8 shows a block diagram of a data extraction system according to another embodiment of the present disclosure;

FIG. 9 is a schematic block diagram of a computer system suitable for implementing a data extraction method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, exemplary embodiments of the disclosed embodiments will be described in detail with reference to the accompanying drawings so that they can be easily implemented by those skilled in the art. Also, for the sake of clarity, parts not relevant to the description of the exemplary embodiments are omitted in the drawings.

In the disclosed embodiments, it is to be understood that terms such as "including" or "having," etc., are intended to indicate the presence of the disclosed features, numbers, steps, behaviors, components, parts, or combinations thereof, and are not intended to preclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or combinations thereof may be present or added.

It should be further noted that the embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Fig. 1 shows a flowchart of a data extraction method according to an embodiment of the present disclosure, which includes the following steps S101-S104, as shown in fig. 1:

in step S101, a data extraction request is received from a client; the data extraction request comprises an identifier and configuration parameters of the target data extraction template;

in step S102, configuring the target data extraction template by using the configuration parameters;

in step S103, running an execution script of the target data extraction template to obtain data from one or more production systems by using the configured target data extraction template;

in step S104, the data obtained from the production system is stored in a big data cluster system.

As mentioned above, the large data cluster platform needs to centralize the data of the core production systems, but in some large enterprises, there are more than one production system and may be hundreds, and the data tables generated by the production systems may be ten thousand levels, so that the adjustment process is encountered in the data centralization process of the production systems. One common method is to correspond one table in the production system to one data extraction script in the big data cluster platform, and when a new data table is added to the production system, a background manager of the big data cluster platform needs to develop the data extraction script once, so that the development efficiency is extremely low, and the operation and maintenance work is extremely complicated.

In view of the above problem, in this embodiment, a data extraction method is provided, in a process of extracting data from a plurality of production systems by a big data cluster system, by establishing a target data extraction template and an execution script in advance, when data extraction is required, a relevant person configures the target data extraction template from a client, and sends configuration parameters of the target data extraction template to a background management server, so that the background management server configures the target data extraction template through the configuration parameters, and runs a corresponding execution script, so as to extract required data from one or more production systems according to the configured target data extraction template. Through the mode, the data extraction task is abstracted to the mode of the target data extraction template, so that the data extraction script is not required to be re-developed aiming at a newly increased data table in the production system, and only the target data extraction template is configured, and the execution script can be a universal execution script and can be used for executing any target data extraction template, the data extraction efficiency of the big data cluster system is improved, the script development time in the data extraction process is saved, and the operation and maintenance flow of the big data cluster system is simplified.

In an embodiment of the present disclosure, the data extraction method may be suitable for running on a background management server in a big data cluster system.

In one embodiment of the present disclosure, a relevant person, such as a data operation maintenance person, may extract the required data from the production system through a client request. The relevant personnel may create a data extraction template in advance, and the data extraction template may be created by abstracting the data extraction task, for example, the data extraction template may include various database query statements, but specific data objects in the database query statements may be formed in the form of configuration parameters.

In an embodiment of the present disclosure, the relevant personnel may create a plurality of data extraction templates in advance, for example, a full data abstraction template for full data in the production system, and an incremental data extraction template for incremental data in the production system. The full data extraction template is used to extract all relevant data from the production system, and the incremental data extraction template is used to extract newly added data from the relevant data from the production system.

In an embodiment of the present disclosure, the relevant person may select, by the client, a data extraction template required for currently performing data extraction, for example, and configure corresponding configuration parameters for the selected data extraction template, and initiate, by the client, a data extraction request to the background management server, where the data extraction request may include an identifier and configuration parameters of the selected target data extraction template.

In an embodiment of the present disclosure, the target data extraction template may be one of pre-established data extraction templates, and the identifier of the target data extraction template may be a name, an ID, and the like defined when the data extraction template is pre-established, so as to distinguish different data extraction templates.

In an embodiment of the present disclosure, the configuration parameter corresponding to the target data extraction template may be, for example, a database connection string, a database object set name (schema name), a table name, a partition field, and the like for extracting data, one database object set may include one or more database objects, and the database objects may be a database table, a database view, a data storage process, an index table, and the like. It is understood that the configuration parameters corresponding to the incremental data extraction template may also include a time range, i.e., a time range for extracting the incremental data, and the production system may return the newly added data within the time range.

In an embodiment of the present disclosure, after receiving the configuration parameters configured by the relevant personnel, the backend management server may configure the target data extraction template by using the configuration parameters, that is, assign data indicated by the configuration parameters to corresponding positions in the data extraction template, for example, query conditions of a database query statement, and the like.

In an embodiment of the present disclosure, the relevant person may also set up an execution script in advance for the data extraction template, where the execution script writes an execution rule of the data extraction template through a script statement. The relevant personnel can create the execution script through the client and send the execution script to the background management server.

In an embodiment of the present disclosure, the same execution script may be established for one or more data extraction templates, and since the data extraction templates are abstract data extraction tasks, the execution scripts of the data extraction templates may be the same.

In an embodiment of the present disclosure, after configuring the target data extraction template by using the configuration parameters, the background management server may run an execution script corresponding to the target data extraction template, and then call, by an execution statement in the execution script, content, such as a data query statement, in the target data extraction template, so as to obtain corresponding data from the production system.

In an embodiment of the present disclosure, the data extraction template may extract data for one or more production systems, for example, the configuration parameters may configure the data newly added in the current day to be obtained from a plurality of production systems. The background management server can respectively obtain corresponding data from a plurality of production systems by running the execution script, and store the obtained data in the big data cluster system so as to improve the use of the data for a front-end business department.

In an embodiment of the present disclosure, the big data cluster system may include a plurality of cluster storage nodes, where the plurality of cluster storage nodes constitute physical storage nodes of a data warehouse in the big data cluster system, and data acquired from the production system is merged and then stored in the data warehouse. The data warehouse may improve data access services to front-end business departments.

In an embodiment of the present disclosure, the method may include the steps of:

In the embodiment, related personnel can establish a full data extraction template and an incremental data extraction template on a client and send the full data extraction template and the incremental data extraction template to the background management server for storage.

In this embodiment, the relevant person may also create an execution script corresponding to the full-data extraction template and the incremental data extraction template on the client, and send the execution script together with the data extraction request to the background management server. After the background management server receives the data extraction request, the execution script can be directly run.

In an embodiment of the present disclosure, the step S101 of receiving a data extraction request from a client may further include the following steps:

In this embodiment, relevant personnel can configure configuration parameters at the client according to the full-data extraction template or the incremental data extraction template, and then send the configuration parameters and identifiers of the data extraction template, such as the full-data extraction template identifier or the incremental data extraction template identifier, to the background management server, and the background management server can acquire the stored data extraction template by using the identifiers, and then configure and complete an executable instance according to the received configuration parameters, and then run the instance of the target data extraction template by running the execution script.

receiving a template configuration request from a client;

the configuration parameters are received from a client.

In this embodiment, when a relevant person configures a target data extraction template through a client, the client may send a template configuration request to a backend management server, and the backend management server may determine whether full data extraction or incremental data extraction should be performed currently, specifically, according to a pre-established rule, for example, when data extraction is performed for the first time or data extraction is performed for a new service, it may be determined that the current data extraction stage is in a full data extraction stage, and it is determined that the current data extraction stage is in an incremental data extraction stage in a subsequent data extraction process. The background management server may instruct the client to configure a corresponding data extraction template for the current data extraction phase. For example, in the full data extraction stage, the configuration of the full data extraction template is instructed, and in the incremental data extraction stage, the configuration of the incremental data extraction template is instructed.

In an embodiment of the present disclosure, the step S102, namely the step of configuring the target data extraction template by using the configuration parameter, may further include the following steps: and acquiring the configured target data extraction template in a mode of filling the configuration parameters into corresponding parameter options in the target data extraction template.

In an embodiment of the present disclosure, the method may further include the steps of: acquiring an execution script from a client and storing the execution script; the execution script is used for executing the full data extraction template and the incremental data extraction template.

In an embodiment of the present disclosure, the step S103 of executing the execution script of the target data extraction template to obtain data from the production system by using the configured target data extraction template may further include the following steps:

and running the execution script.

In this embodiment, the relevant person may also create different execution scripts in advance for different target data extraction templates, and store the execution scripts in the storage device. After receiving the data extraction request, the background management server may obtain a corresponding execution script from the storage device according to the identifier of the target data extraction template, and execute the execution script.

sending a request for acquiring the execution script to a client;

In this embodiment, after the target data extraction template is configured by using the configuration parameters, the background management server may request the client to retrieve an execution script for running the target data extraction template. The client can output a notification for providing the execution script to the user, or display a page for writing the execution script, and the user can directly upload the written execution script, or write the execution script online. The client returns the written execution script to the background management server, and the background management server runs the execution script.

In an embodiment of the present disclosure, the step S104 of storing the data acquired from the production system to a big data cluster system may further include the steps of:

receiving returned data from one or more of the production systems;

In this embodiment, after receiving a data query statement or the like in a target data extraction template initiated by executing a script, the production system may return corresponding data to the background management server, and the background management server merges the returned data according to a preset rule and stores the merged data in a data warehouse of the big data cluster system. In some embodiments, a big data cluster system may employ Hadoop.

Fig. 2 shows a flowchart of an extraction method according to another embodiment of the present disclosure, which, as shown in fig. 2, includes the following steps S201 to S203:

in step S201, detecting a configuration operation of a user on a data extraction template;

in step S202, in response to the detected configuration operation, a configuration interface is displayed, and an identifier and configuration parameters of a target data extraction template are acquired from the configuration interface;

in step S203, a data extraction request is sent to the background management server; the data extraction request comprises identification of the target data extraction template and configuration parameters, so that the background management server can acquire data from one or more production systems based on the target data extraction template and the configuration parameters and store the acquired data in a big data cluster system.

In an embodiment of the present disclosure, the data extraction method may be suitable for running on an operation and maintenance management client in a big data cluster system.

In an embodiment of the present disclosure, a relevant person may perform a configuration operation of a data extraction template through a client, and after the client detects the configuration operation of the data extraction template by the relevant person, a configuration interface of a target data extraction template is displayed on a display screen of the client. The configuration interface may provide options for one or more data extraction templates that may be configured by the relevant person, for example, configuration options for a full data extraction template and an incremental data extraction template may be displayed, and the relevant person may select one of the templates for configuration. The client can give a unique identifier to the established data extraction template in advance, relevant personnel select one data extraction template for configuration, and after corresponding configuration parameters are input from the configuration interface, the client can send the identifier of the data extraction template selected by the relevant personnel and the configuration parameters to the background management server so as to request the background management server to extract data.

In an embodiment of the present disclosure, the method may further include the steps of:

In this embodiment, the relevant personnel may create a target data extraction template, such as a full data extraction template or an incremental data extraction template, through the client. After detecting the template creation request, the client displays a creation page for creating the target data extraction template on a display interface, relevant personnel can input template content in the creation page, and the client can create a corresponding target data extraction template according to the received content and store the data extraction template locally and/or send the data extraction template to the background management server.

In an embodiment of the present disclosure, step S202, which is to display a configuration interface in response to the detected configuration operation, and obtain the identifier of the target data extraction template and the configuration parameters from the configuration interface, further includes the following steps:

receiving the configuration parameters from the input interface.

In this embodiment, for the configuration operation of the target data extraction template by the relevant person, options of the data extraction template that can be configured may be displayed on the configuration interface, and the relevant person may select one of the options as the target data extraction template for configuration, for example, the extraction options of the full-volume data extraction template and the incremental data extraction template may be displayed. After the relevant person selects one of the target data extraction templates, the client can determine the identifier of the target data extraction template according to the selection of the relevant person, and then the configuration page of the target extraction template is displayed to the relevant person, the configuration page can comprise input interfaces of parameter options needing to be configured in the target data extraction template, and the relevant person can input corresponding configuration parameters through the input interfaces.

receiving the configuration parameters from the input interface;

and sending the configuration parameters to the background management server.

In this embodiment, after detecting that a relevant person performs a parameter configuration operation on a target data extraction template, a client may send a template configuration request to a backend management server, and receive an indication of a target full-data extraction template that can be configured from the backend management server, for example, the backend management server may indicate to configure the full-data extraction template, or may indicate to configure an incremental data extraction template, which is specifically determined according to a data extraction stage currently located by the backend management server. When the current data extraction stage is a full data extraction stage, an instruction for configuring the full data extraction template may be received, and when the current data extraction stage is an incremental data extraction stage, an instruction for configuring the incremental data extraction template may be received.

In this embodiment, the relevant person may also create an execution script for any one or more of the data extraction templates. After receiving a request for creating an execution script from a relevant person, the client can display a script compiling page on a display interface, the relevant person can compile the script on the script compiling page, perform running test and the like, finally submit the compiled execution script to the client for storage by the client, and the client can also send the execution script to a background management server so that the background management server can directly run the execution script in the data extraction process.

Technical terms and technical features related to the technical terms and technical features shown in fig. 2 and related embodiments are the same as or similar to those of the technical terms and technical features shown in fig. 1 and related embodiments, and for the explanation and description of the technical terms and technical features related to the technical terms and technical features shown in fig. 2 and related embodiments, reference may be made to the above explanation of the explanation of fig. 1 and related embodiments, and no further description is provided here.

Fig. 3 shows a flowchart of an extraction method according to another embodiment of the present disclosure, which, as shown in fig. 3, includes the following steps S301-S305:

in step S301, the client detects a configuration operation of a user on a data extraction template, and in response to the detected configuration operation, displays a configuration interface, and obtains an identifier and configuration parameters of a target data extraction template from the configuration interface;

in step S302, the client sends a data extraction request to the background management server; the data extraction request comprises an identifier and configuration parameters of the target data extraction template;

in step S303, the background management server receives a data extraction request from a client, and configures the target data extraction template by using the configuration parameters;

in step S304, the background management server runs an execution script of the target data extraction template, so as to obtain data from one or more production systems by using the configured target data extraction template;

in step S305, the background management server stores the data acquired from the production system to a big data cluster system.

In an embodiment of the present disclosure, the data extraction method may be suitable for operating in a big data cluster system.

In an embodiment of the present disclosure, in step 301, in response to the detected configuration operation, the steps of presenting a configuration interface, and acquiring an identifier of a target data extraction template and configuration parameters from the configuration interface further include the following steps:

the client receives the configuration parameters from the input interface.

the client receives the configuration parameters from the input interface;

In an embodiment of the present disclosure, in step 303, the step of receiving, by the backend management server, a data extraction request from a client may further include the following steps:

In an embodiment of the present disclosure, in step 303, the step of configuring the target data extraction template by using the configuration parameter may further include the following steps:

In an embodiment of the present disclosure, in step 304, the step of the background management server running an execution script of the target data extraction template so as to obtain data from a production system by using the configured target data extraction template may further include the following steps:

and the background management server runs the execution script.

In an embodiment of the present disclosure, in step 305, the step of the background management server storing the data acquired from the production system to a big data cluster system may further include the following steps:

Technical terms and technical features related to the technical terms and technical features shown in fig. 3 and related embodiments are the same as or similar to those of the technical terms and technical features shown in fig. 1 and 2 and related embodiments, and for the explanation and description of the technical terms and technical features related to the technical terms and technical features shown in fig. 3 and related embodiments, the above explanation of the technical terms and technical features shown in fig. 1 and 2 and related embodiments can be referred to, and will not be repeated herein.

Fig. 4 is a schematic diagram illustrating an application scenario of a data extraction method according to an embodiment of the present disclosure. Fig. 5 illustrates an overall flowchart of a data extraction method according to an embodiment of the present disclosure. As shown in fig. 4 and 5, the big data cluster system may include a plurality of clients, which may be used by a plurality of operation and maintenance managers, and the background management server may be a virtual machine, which may be composed of a plurality of physical machines. Related personnel can configure the data extraction template through the client and submit the configured data extraction template to the background management server. The related personnel can also write an execution script of the execution data extraction template through the client and provide the execution script to the background management server. When data is extracted, related personnel can configure the data extraction template through the client and submit configuration parameters to the background management server. In the full data extraction stage, relevant personnel configure configuration parameters to be configured in the full data extraction template, and the client provides the identification of the full data extraction template and the configuration parameters received from the relevant personnel to the background management server. After receiving the configuration parameters, the background management server configures the configuration parameters into the full-data extraction template, then runs an execution script, and in the process of running the execution script, the script calls contents such as data query statements and the like in the full-data extraction template to further send a data acquisition request to the production system, after receiving the request, the production system returns corresponding data to the background management server, and the background management server can store the data into a data warehouse in the big data cluster system.

In the incremental data extraction stage, relevant personnel configure configuration parameters to be configured in the incremental data extraction template, and the client provides the identification of the incremental data extraction template and the configuration parameters received from the relevant personnel to the background management server. After receiving the configuration parameters, the background management server configures the configuration parameters into the incremental data extraction template, then runs an execution script, in the process of running the execution script, the script calls contents such as data query statements and the like in the incremental data extraction template to further send a data acquisition request to the production system, after receiving the request, the production system returns newly added corresponding data in a time range to the background management server, and the background management server can merge the incremental data into corresponding data in a data warehouse.

The following are embodiments of the disclosed apparatus that may be used to perform embodiments of the disclosed methods.

Fig. 6 shows a block diagram of a data extraction device according to an embodiment of the present disclosure, which may be implemented as part or all of an electronic device by software, hardware, or a combination of both. As shown in fig. 6, the data extraction device includes:

a receiving module 601 configured to receive a data extraction request from a client; the data extraction request comprises an identifier and configuration parameters of the target data extraction template;

a configuration module 602 configured to configure the target data extraction template with the configuration parameters;

a running module 603 configured to run an execution script of the target data extraction template so as to obtain data from one or more production systems by using the configured target data extraction template;

a storage module 604 configured to store the data obtained from the production system to a big data cluster system.

In an embodiment of the present disclosure, the data extraction device may be adapted to run on a background management server in a big data cluster system.

Fig. 7 shows a block diagram of a data extraction apparatus according to another embodiment of the present disclosure, which may be implemented as part or all of an electronic device by software, hardware, or a combination of both. As shown in fig. 7, the data extraction device includes:

a detection module 701 configured to detect a configuration operation of a user on a data extraction template;

a response module 702 configured to, in response to the detected configuration operation, present a configuration interface and obtain an identification of a target data extraction template and configuration parameters from the configuration interface;

a sending module 703 configured to send a data extraction request to the background management server; the data extraction request comprises identification of the target data extraction template and configuration parameters, so that the background management server can acquire data from one or more production systems based on the target data extraction template and the configuration parameters and store the acquired data in a big data cluster system.

In an embodiment of the present disclosure, the data processing apparatus may be adapted to run on an operation and maintenance management client in a big data cluster system.

Fig. 8 shows a block diagram of a data extraction system according to an embodiment of the present disclosure, which may be implemented as part or all of an electronic device by software, hardware, or a combination of both. As shown in fig. 8, the data extraction system includes: a client 801 and a background management server 802;

the client 801 detects a configuration operation of a user on a data extraction template, responds to the detected configuration operation, displays a configuration interface, and acquires an identifier and configuration parameters of a target data extraction template from the configuration interface;

the client 801 sends a data extraction request to the background management server 802; the data extraction request comprises an identifier and configuration parameters of the target data extraction template;

the background management server 802 receives a data extraction request from the client 801 and configures the target data extraction template by using the configuration parameters;

the background management server 802 runs an execution script of the target data extraction template so as to obtain data from one or more production systems by using the configured target data extraction template;

the background management server 802 stores the data obtained from the production system to a big data cluster system.

In an embodiment of the present disclosure, the data extraction system may be adapted to extract full or incremental data from the production system by abstracting the data extraction tasks into the form of data extraction templates.

The technical features related to the above device embodiments and the corresponding explanations and descriptions thereof are the same as, corresponding to or similar to the technical features related to the above method embodiments and the corresponding explanations and descriptions thereof, and for the technical features related to the above device embodiments and the corresponding explanations and descriptions thereof, reference may be made to the technical features related to the above method embodiments and the corresponding explanations and descriptions thereof, and details of the disclosure are not repeated herein.

The embodiment of the present disclosure also discloses an electronic device, which includes a memory and a processor; wherein,

the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to perform any of the method steps described above.

As shown in fig. 9, the computer system 900 includes a processing unit 901 which can execute various processes in the above-described embodiments according to a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. In the RAM903, various programs and data necessary for the operation of the computer system 900 are also stored. The processing unit 901, the ROM902, and the RAM903 are connected to each other via a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.

The following components are connected to the I/O interface 905: an input portion 906 including a keyboard, a mouse, and the like; an output section 907 including components such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 908 including a hard disk and the like; and a communication section 909 including a network interface card such as a LAN card, a modem, or the like. The communication section 909 performs communication processing via a network such as the internet. The drive 910 is also connected to the I/O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 910 as necessary, so that a computer program read out therefrom is mounted into the storage section 908 as necessary. The processing unit 901 may be implemented as a CPU, a GPU, a TPU, an FPGA, an NPU, or other processing units.

In particular, the above described methods may be implemented as computer software programs, according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a medium readable thereby, the computer program comprising program code for performing the data transmission method. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section 909, and/or installed from the removable medium 911.

A computer program product is also disclosed in embodiments of the present disclosure, the computer program product comprising computer programs/instructions which, when executed by a processor, implement any of the above method steps.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, a program segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units or modules described in the embodiments of the present disclosure may be implemented by software or hardware. The units or modules described may also be provided in a processor, and the names of the units or modules do not in some cases constitute a limitation of the units or modules themselves.

As another aspect, the disclosed embodiment also provides a computer-readable storage medium, which may be the computer-readable storage medium included in the apparatus in the foregoing embodiment; or it may be a separate computer readable storage medium not incorporated into the device. The computer readable storage medium stores one or more programs for use by one or more processors in performing the methods described in the embodiments of the present disclosure.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept. For example, the above features and (but not limited to) the features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.

Claims

1. A method of data extraction, comprising:

2. The method of claim 1, wherein the target data extraction templates include full data extraction templates and incremental data extraction templates.

3. A method of data extraction, comprising:

detecting the configuration operation of a user on a data extraction template;

4. A method of data extraction, comprising:

5. A data extraction apparatus comprising:

6. A data extraction apparatus comprising:

7. A data extraction system, comprising: the system comprises a client and a background management server;

8. An electronic device comprising a memory and a processor; wherein,

the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the steps of the method of any one of claims 1-4.

9. A computer readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the steps of the method of any one of claims 1-4.

10. A computer program product comprising computer programs/instructions which, when executed by a processor, carry out the steps of the method of any one of claims 1 to 4.