CN113392343A - Data extraction method, device, medium and computer program product - Google Patents
Data extraction method, device, medium and computer program product Download PDFInfo
- Publication number
- CN113392343A CN113392343A CN202110940853.3A CN202110940853A CN113392343A CN 113392343 A CN113392343 A CN 113392343A CN 202110940853 A CN202110940853 A CN 202110940853A CN 113392343 A CN113392343 A CN 113392343A
- Authority
- CN
- China
- Prior art keywords
- data extraction
- message information
- data
- message
- extraction result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9538—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/957—Browsing optimisation, e.g. caching or content distillation
- G06F16/9574—Browsing optimisation, e.g. caching or content distillation of access to content, e.g. by caching
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The application discloses a data extraction method, equipment, a medium and a computer program product, wherein the data extraction method comprises the following steps: inputting configuration parameters corresponding to a data extraction task in a user interface, packaging the configuration parameters into message information, sending the message information to a message queue for a server to read the message information in the message queue, performing data processing on the message information to obtain a data extraction result, sending the data extraction result and an identification tag corresponding to the message information to the message queue, reading the data extraction result cached in the message queue based on the identification tag, and feeding the data extraction result back to the user interface. The application solves the technical problem of low usability of data extraction.
Description
Technical Field
The present application relates to the field of big data technologies, and in particular, to a data extraction method, device, medium, and computer program product.
Background
With the continuous development of big data technology, at present, data stored in a database in the big data field can be transmitted through a Sqoop tool (open source tool), where the Sqoop is an open source tool used for data transmission between a Hadoop (distributed system infrastructure) and a traditional database, and at present, in the process of data transmission through the Sqoop tool, the Sqoop tool needs to be operated through manually logging in a Linux system (operating system) remotely, however, when the Sqoop tool is operated based on an operating command of Linux, a user needs to master a series of input instructions of Linux to normally operate the Sqoop tool, and when the Sqoop tool is operated to complete data extraction and transmission, a large number of Sqoop instruction scripts also need to be mastered, thereby resulting in low usability of data extraction.
Disclosure of Invention
The present application mainly aims to provide a data extraction method, device, medium and computer program product, and aims to solve the technical problem of low usability of data extraction in the prior art.
In order to achieve the above object, the present application provides a data extraction method, where the data extraction method is applied to a client of a framework system, and the data extraction method includes:
inputting configuration parameters corresponding to the data extraction task in a user interface;
packaging the configuration parameters into message information, sending the message information to a message queue for a server to read the message information of the message queue, performing data processing on the message information to obtain a data extraction result, and sending the data extraction result and an identification label corresponding to the message information to the message queue;
and reading the data extraction result cached in the message queue based on the identification tag, and feeding back the data extraction result to the user interface.
Optionally, the step of inputting the configuration parameters corresponding to the data extraction task in the user interface includes:
inputting configuration parameters corresponding to data source information;
and selecting the table names of the big data source and the target source and configuring configuration parameters corresponding to the extraction frequency of the data extraction task.
Optionally, the step of encapsulating the configuration parameters into message information, sending the message information to a message queue, so that a server reads the message information of the message queue, performs data processing on the message information, obtains a data extraction result, and sends the data extraction result and an identification tag corresponding to the message information to the message queue includes:
carrying out system reinforcement on the configuration parameters to obtain reinforced configuration parameters;
and encapsulating the enhanced configuration parameters into message information, and sending the message information to the message queue.
Optionally, the step of reading the data extraction result cached in the message queue based on the identification tag and feeding back the data extraction result to the user interface includes:
when a data extraction result cached in the message queue is monitored, reading the data extraction result;
and feeding back the data extraction result to a user interface corresponding to the identification label based on the identification label.
Optionally, after the step of reading the data extraction result cached in the message queue based on the identification tag and feeding back the data extraction result to the user interface, the data extraction method further includes:
and informing a target user to view the data extraction result in the user interface for the target user.
In order to achieve the above object, the present application provides a data extraction method, where the data extraction method is applied to a server of a framework system, and the data extraction method further includes:
reading message information sent by a client when monitoring that the message information is cached in a message queue;
performing data processing on the message information to obtain a data extraction result;
and sending the data extraction result and the identification tag corresponding to the message information to the message queue so that the client receives the data extraction result in the message queue based on the identification tag.
Optionally, the step of performing data processing on the message information to obtain a data extraction result includes:
performing data analysis on the message information to obtain configuration parameters corresponding to the message information;
encapsulating the configuration parameters into data extraction parameters;
and calling a preset extraction method to perform data processing on the data extraction parameters to obtain the data extraction result.
The present application further provides a data extraction device, the data extraction device is a virtual device, and the data extraction device is applied to the client of the framework system, the data extraction device includes:
the parameter configuration module is used for inputting configuration parameters corresponding to the data extraction task in the user interface;
the first sending module is used for packaging the configuration parameters into message information, sending the message information to a message queue for a server to read the message information of the message queue, performing data processing on the message information to obtain a data extraction result, and sending the data extraction result and an identification label corresponding to the message information to the message queue;
and the feedback module is used for reading the data extraction result cached in the message queue based on the identification tag and feeding back the data extraction result to the user interface.
The present application further provides a data extraction device, the data extraction device is a virtual device, and the data extraction device is applied to the server of the frame system, the data extraction device includes:
the reading module is used for reading the message information sent by the client when the message information cached in the message queue is monitored;
the data processing module is used for carrying out data processing on the message information to obtain a data extraction result;
and the second sending module is used for sending the data extraction result and the identification tag corresponding to the message information to the message queue so that the client can receive the data extraction result in the message queue based on the identification tag.
The present application further provides a data extraction device, the data extraction device is an entity device, the data extraction device includes: the data extraction program is executed by the processor to realize the steps of the data extraction method.
The present application also provides a medium, which is a readable storage medium, on which a data extraction program is stored, and the data extraction program is executed by a processor to implement the steps of the data extraction method as described above.
The present application also provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of the data extraction method as described above.
Compared with the technical means of carrying out data transmission by manually logging in a Linux system and operating a Sqoop tool in a remote manner in the prior art, the method for extracting data comprises the steps of firstly inputting configuration parameters corresponding to a data extraction task in a user interface, further realizing the purpose of dynamically configuring the data extraction task based on the user interface, further packaging the configuration parameters into message information, sending the message information to a message queue so that a server can read the message information of the message queue, carrying out data processing on the message information to obtain a data extraction result, sending the data extraction result and an identification label corresponding to the message information to the message queue, realizing asynchronous communication between the client and the server based on the message queue, furthermore, based on the identification tag, reading a data extraction result cached in the message queue, and feeding the data extraction result back to the user interface, so that data extraction is performed by asynchronously scheduling a Sqoop tool through the client based on a configuration parameter corresponding to the user interface, and the data extraction result is also asynchronously transmitted to the message queue, that is, a user can automatically extract data only by inputting a corresponding parameter in the user interface without mastering complex instructions such as Linux and Sqoop and the like, the difficulty of using a large data assembly by the user is reduced, the technical defect that in the prior art, the Sqoop tool can be normally used only by needing the user to master a series of input instructions of Linux, and a large number of Sqoop instruction scripts are also needed to master when the Sqoop tool is operated to complete data extraction and transmission, so that the usability of data extraction is low is overcome, thereby improving the usability of data extraction.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious to those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a schematic flow chart of a first embodiment of a data extraction method according to the present application;
FIG. 2 is a schematic flow chart of a data extraction method according to a second embodiment of the present application;
fig. 3 is a schematic flow chart illustrating communication between a client and a server in the data extraction method of the present application;
fig. 4 is a schematic structural diagram of a data extraction device in a hardware operating environment related to a data extraction method in an embodiment of the present application.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In a first embodiment of the data extraction method, the data extraction method is applied to a client of a framework system, and referring to fig. 1, the data extraction method includes:
step S10, inputting configuration parameters corresponding to the data extraction task in the user interface;
in this embodiment, it should be noted that the data extraction method is applied to a framework system, where the framework system is a system constructed based on a Springboot framework or an SSM (Spring + SpringMVC + mybalancing) framework, and the framework system includes a client and a server, where the server is deployed on a server where Sqoop is located to perform data extraction operation, and the configuration parameters are task parameters for performing data extraction, and include parameters such as data source information, a database table name, and an extraction frequency.
Inputting configuration parameters corresponding to a data extraction task in a user interface, specifically, inputting data source information in a front-end user interface of a client to select a big data source, wherein the big data source comprises big data sources such as an HDFS (distributed file system), an HBase (distributed storage system) and a Hive (a set of data warehouse analysis system constructed based on Hadoop), and further, the extraction frequency of the data extraction task is configured;
the step of inputting the configuration parameters corresponding to the data extraction task in the user interface comprises the following steps:
step S11, inputting configuration parameters corresponding to data source information in the user interface;
in this embodiment, it should be noted that the data source information includes information such as a database type, a URL (Uniform Resource Locator) address, a port number, a user name, a database name, and a database password.
And inputting configuration parameters corresponding to the data source information in the user interface, specifically, inputting the required data source information in a preset input box in the user interface.
And step S12, selecting the names of the large data source and the target source table and configuring configuration parameters corresponding to the extraction frequency of the data extraction task.
In this embodiment, it should be noted that the big data sources include big data sources such as HDFS, HBase, and Hive.
Selecting a big data source and a target source table name and configuring configuration parameters corresponding to the extraction frequency of the data extraction task, specifically, inputting a target library and a target source table name corresponding to the big data source to be imported into the user interface, and further configuring the extraction frequency of the data extraction task so as to periodically perform the data extraction task based on the extraction frequency.
Step S20, packaging the configuration parameters into message information, sending the message information to a message queue for a server to read the message information in the message queue, performing data processing on the message information to obtain a data extraction result, and sending the data extraction result and an identification label corresponding to the message information to the message queue;
in this embodiment, it should be noted that the message information is the message information for performing an HTTP request based on the configuration parameter, where the message information of an HTTP request is composed of 3 parts, namely, a request line (request line), a request header (header) and request data, and the message queue is a message queue of a topic mode (subscription mode) and includes a RabbitMQ message queue, an ActiveMQ message queue, a rocktmq message queue, and the like, where the topic mode allows multiple clients to receive the same message, that is, a message may be transmitted to multiple clients, for example, a server issues 10 messages, and there are two clients a and B that establish a persistent subscription or two clients a and B that are in an active state when the server issues a message, and then two clients a and B each receive 10 messages.
Packaging the configuration parameters into message information, sending the message information to a message queue for a server to read the message information in the message queue, performing data processing on the message information to obtain a data extraction result, sending the data extraction result and an identification tag corresponding to the message information to the message queue, specifically, receiving the configuration parameters input by the user interface, further creating connection between the client and the message queue, further, packaging the configuration parameters into the message information, wherein the message information comprises the identification tag corresponding to the message information, the configuration parameters and other information, and then sending the message information to a corresponding queue for the server to read the message information cached in the message queue, and further based on the message information, and calling a preset data extraction method to extract data, obtaining a data extraction result, and sending the data extraction result and the identification tag corresponding to the message information to the message queue through the server.
Step S21, carrying out system reinforcement on the configuration parameters to obtain reinforced configuration parameters;
in this embodiment, it should be noted that the enhanced configuration parameters are parameters obtained by supplementing incomplete configuration parameters.
Performing system reinforcement on the configuration parameters to obtain enhanced configuration parameters, specifically, when the configuration parameters corresponding to the data extraction task are completed in the user interface and then the configuration parameters are obtained through a back-end architecture corresponding to a client, supplementing the configuration parameters according to a past configuration parameter record when the configuration parameters input through the user interface are not complete, and then obtaining the enhanced configuration parameters, performing data extraction based on the enhanced configuration parameters, for example, taking an id value specified in an input box of the user interface as a form parameter of a function, receiving an input value of the input box of the user interface by using a RequestParam annotation or a back-end HttpServletRequest and the like based on the id value, and then performing system reinforcement on the configuration parameters to obtain the enhanced configuration parameters.
Step S22, encapsulating the enhanced configuration parameters into message information, and sending the message information to the message queue.
In this embodiment, the enhanced configuration parameters are encapsulated into message information, and the message information is sent to the message queue, specifically, the enhanced configuration parameters are added to a topic corresponding to the message queue by the client, further, the message information is sent to the topic queue corresponding to the message queue, and after the sending is completed, a task submission response is sent to the user interface through a background of the client, for example, one HTTP request message is composed of 3 parts, namely, a request line (request line), a request header (header), and request data, and therefore, when sending an HTTP request command, message information, such as a corresponding request header, data corresponding to the configuration parameters, and the like, is also sent.
Step S30, based on the identification tag, reading the data extraction result cached in the message queue, and feeding back the data extraction result to the user interface.
In this embodiment, based on the identification tag, the data extraction result cached in the message queue is read, and the data extraction result is fed back to the user interface, specifically, the message queue is monitored in real time by the client, and when the identification tag corresponding to the message information is monitored, the data extraction result is fed back to the user interface, so that the user views the data extraction result in the user interface.
Wherein the step of reading the data extraction result cached in the message queue based on the identification tag and feeding back the data extraction result to the user interface comprises:
step S31, when a data extraction result is cached in the message queue, reading the data extraction result;
in this embodiment, when it is monitored that the data extraction result is cached in the message queue, the data extraction result is read, specifically, after the message information is sent to the message queue, the message queue is monitored in real time by the client, and when it is monitored that the data extraction result is cached in the message queue, the data extraction result is read.
And step S32, feeding back the data extraction result to a user interface corresponding to the identification label based on the identification label.
In this embodiment, the data extraction result is fed back to the user interface corresponding to the identification tag based on the identification tag, and specifically, the data extraction result is fed back to the user interface corresponding to the identification tag based on the identification tag corresponding to the message information, so that a user can view the extraction result on the user interface.
After the step of reading the data extraction result cached in the message queue based on the identification tag and feeding back the data extraction result to the user interface, the data extraction method further includes:
step A10, notifying a target user for the target user to view the data extraction result in the user interface.
In this embodiment, a target user is notified to view the data extraction result in the user interface, specifically, a mailbox or a short message is sent by the client to notify the target user so that the target user can view the data extraction result in the user interface.
Additionally, based on the message queue, the client only needs to send the message information to the message queue, and the server only needs to read the message information from the message queue, so that asynchronous communication between the client and the server and decoupling between the client and the server are realized, and further based on the message queue, a plurality of clients and a plurality of servers can be deployed, so that a high-availability and telescopic system architecture is achieved, robustness and quick response of the client are improved, and user experience is better.
Compared with the technical means of carrying out data transmission by manually logging in a Linux system remotely and operating a Sqoop tool in the prior art, the embodiment of the application firstly inputs the configuration parameters corresponding to the data extraction task in the user interface, further realizes the purpose of dynamically configuring the data extraction task based on the user interface, further encapsulates the configuration parameters into message information, sends the message information to the message queue so that the server side can read the message information of the message queue, carries out data processing on the message information to obtain a data extraction result, and sends the data extraction result and the identification tag corresponding to the message information to the message queue, thereby realizing asynchronous communication between the client side and the server side based on the message queue, furthermore, based on the identification tag, reading a data extraction result cached in the message queue, and feeding the data extraction result back to the user interface, so that data extraction is performed by asynchronously scheduling a Sqoop tool through the client based on a configuration parameter corresponding to the user interface, and the data extraction result is also asynchronously transmitted to the message queue, that is, a user can automatically extract data only by inputting a corresponding parameter in the user interface without mastering complex instructions such as Linux and Sqoop and the like, the difficulty of using a large data assembly by the user is reduced, the technical defect that in the prior art, the Sqoop tool can be normally used only by needing the user to master a series of input instructions of Linux, and a large number of Sqoop instruction scripts are also needed to master when the Sqoop tool is operated to complete data extraction and transmission, so that the usability of data extraction is low is overcome, thereby improving the usability of data extraction.
Further, referring to fig. 2, in another embodiment of the present application, the data extraction method is applied to a server of a framework system, and the data extraction method includes:
step B10, when monitoring that the message information sent by the client is cached in the message queue, reading the message information;
in this embodiment, when it is monitored that the message information sent by the client is cached in the message queue, the message information is read, specifically, the message queue is monitored in real time by the server, and when it is monitored that the message information sent by the client is cached in the message queue, the message information is read.
Step B20, processing the message information to obtain data extraction result;
in this embodiment, it should be noted that the data processing is a processing manner of performing data extraction according to configuration parameters, and includes full extraction and incremental extraction, where the full extraction is data migration or data replication, that is, a manner of extracting both a table and a view in a data source from a database, and the incremental extraction is a manner of extracting newly added or modified data in the database according to a last data extraction record.
And performing data processing on the message information to obtain a data extraction result, specifically, analyzing the message information, further encapsulating the analyzed data into data extraction parameters, and further calling a preset data extraction method based on the data extraction parameters to obtain the data extraction result.
Wherein, the step of performing data processing on the message information to obtain a data extraction result comprises:
step B21, carrying out data analysis on the message information to obtain configuration parameters corresponding to the message information;
in this embodiment, the data analysis is performed on the message information to obtain the configuration parameters corresponding to the message information, and specifically, if it is monitored that the message information is cached in the message queue, the message information is read by the server, and then the message information is analyzed to obtain the configuration parameters corresponding to the message information.
Step B22, packaging the configuration parameters into data extraction parameters;
step B23, calling a preset extraction method to perform data processing on the data extraction parameters to obtain the data extraction result;
in this embodiment, it should be noted that the preset extraction method is a method for extracting data according to configuration parameters.
Calling a preset extraction method to perform data processing on the data extraction parameters to obtain the data extraction result, specifically, after analyzing the message information, executing the preset extraction method based on the configuration parameters corresponding to the message information to obtain the data extraction result, for example, directly calling a Sqoop object under an apache Sqoop package to execute a runSqoop () method to perform a data extraction task.
Step B30, sending the data extraction result and the identification tag corresponding to the message information to the message queue, so that the client receives the data extraction result in the message queue based on the identification tag.
In this embodiment, it should be noted that the identification tag corresponding to the message information is a unique identity of the message information.
And sending the data extraction result and the identification tag corresponding to the message information to the message queue so that the client receives the data extraction result in the message queue based on the identification tag, and specifically sending the data extraction result and the identification tag corresponding to the message information to the message queue after the data extraction task is completed by the server so that the client responds the data extraction result to the user interface based on the identification tag corresponding to the message information.
The embodiment of the application provides a data extraction method, that is, when monitoring that message information sent by a client is cached in a message queue, reading the message information, further performing data processing on the message information to obtain a data extraction result, further sending the data extraction result and an identification tag corresponding to the message information to the message queue, so that the client receives the data extraction result in the message queue based on the identification tag, thereby realizing that the server and the client can perform asynchronous communication based on the message queue, further, according to the configuration parameters, the server automatically performs a data extraction task, that is, a user does not need to master complex instructions such as Linux and Sqoop, and only needs to input corresponding parameters in a user interface to automatically extract data, the technical defect that in the prior art, a user needs to master a series of input instructions of Linux to normally use the Sqoop tool, and a large number of Sqoop instruction scripts need to be mastered when the Sqoop tool is operated to complete data extraction and transmission, so that the usability of data extraction is low is overcome, and the usability of data extraction is improved.
Further, referring to fig. 3, fig. 3 is a schematic diagram of a flow of communication between a client and a server in the data extraction method of the present application, where a user parameter is the configuration parameter, an enhanced message is the enhanced message information, a rabbitMQ is the message queue, a topic queue is the message queue in the topic mode (subscription mode), read information is the read message information, runSqoop is executed to perform data extraction, the preset extraction method is executed to perform data processing, a task execution result is the data extraction result, and a received message ID is an identification tag corresponding to the message information.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a data extraction device in a hardware operating environment related to a data extraction method in an embodiment of the present application.
As shown in fig. 4, the data extracting apparatus may include: a processor 1001, such as a CPU, a memory 1005, and a communication bus 1002. The communication bus 1002 is used for realizing connection communication between the processor 1001 and the memory 1005. The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a memory device separate from the processor 1001 described above.
Optionally, the data extraction device may further include a rectangular user interface, a network interface, a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so forth. The rectangular user interface may comprise a Display screen (Display), an input sub-module such as a Keyboard (Keyboard), and the optional rectangular user interface may also comprise a standard wired interface, a wireless interface. The network interface may optionally include a standard wired interface, a wireless interface (e.g., WIFI interface).
Those skilled in the art will appreciate that the data extraction device configuration shown in fig. 4 does not constitute a limitation of the data extraction device and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 4, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, and a data extraction program. The operating system is a program that manages and controls the hardware and software resources of the data extraction device, supporting the operation of the data extraction program as well as other software and/or programs. The network communication module is used for communication among the components in the memory 1005 and with other hardware and software in the data extraction method system.
In the data extraction device shown in fig. 4, the processor 1001 is configured to execute a data extraction program stored in the memory 1005, and implement the steps of the data extraction method described in any one of the above.
The specific implementation of the data extraction device of the present application is substantially the same as that of each embodiment of the data extraction method, and is not described herein again.
The present application further provides a data extraction device, the data extraction device is applied to the client, the data extraction device includes:
the parameter configuration module is used for inputting configuration parameters corresponding to the data extraction task in the user interface;
the first sending module is used for packaging the configuration parameters into message information, sending the message information to a message queue for a server to read the message information of the message queue, performing data processing on the message information to obtain a data extraction result, and sending the data extraction result and an identification label corresponding to the message information to the message queue;
and the feedback module is used for reading the data extraction result cached in the message queue based on the identification tag and feeding back the data extraction result to the user interface.
Optionally, the parameter configuration module is further configured to:
inputting configuration parameters corresponding to data source information;
and selecting the table names of the big data source and the target source and configuring configuration parameters corresponding to the extraction frequency of the data extraction task.
Optionally, the first sending module is further configured to:
carrying out system reinforcement on the configuration parameters to obtain reinforced configuration parameters;
and encapsulating the enhanced configuration parameters into message information, and sending the message information to the message queue.
Optionally, the feedback module is further configured to:
when a data extraction result cached in the message queue is monitored, reading the data extraction result;
and feeding back the data extraction result to a user interface corresponding to the identification label based on the identification label.
Optionally, the data extraction device is further configured to:
and informing a target user to view the data extraction result in the user interface for the target user.
The specific implementation of the data extraction device of the present application is substantially the same as that of the embodiments of the data extraction method, and is not described herein again.
The present application further provides a data extraction device, the data extraction device is applied to the server, the data extraction device includes:
the reading module is used for reading the message information sent by the client when the message information cached in the message queue is monitored;
the data processing module is used for carrying out data processing on the message information to obtain a data extraction result;
and the second sending module is used for sending the data extraction result and the identification tag corresponding to the message information to the message queue so that the client can receive the data extraction result in the message queue based on the identification tag.
Optionally, the data processing module is further configured to:
performing data analysis on the message information to obtain configuration parameters corresponding to the message information;
encapsulating the configuration parameters into data extraction parameters;
and calling a preset extraction method to perform data processing on the data extraction parameters to obtain the data extraction result.
The specific implementation of the data extraction device of the present application is substantially the same as that of the embodiments of the data extraction method, and is not described herein again.
The present application provides a medium, which is a readable storage medium, and the readable storage medium stores one or more programs, and the one or more programs are further executable by one or more processors for implementing the steps of the data extraction method described in any one of the above.
The specific implementation of the readable storage medium of the present application is substantially the same as that of the embodiments of the data extraction method, and is not described herein again.
The present application provides a computer program product, and the computer program product includes one or more computer programs, which can also be executed by one or more processors for implementing the steps of the data extraction method described in any one of the above.
The specific implementation of the computer program product of the present application is substantially the same as that of the embodiments of the data extraction method, and is not described herein again.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings, or which are directly or indirectly applied to other related technical fields, are included in the scope of the present application.
Claims (10)
1. A data extraction method is applied to a client of a framework system, and comprises the following steps:
inputting configuration parameters corresponding to the data extraction task in a user interface;
packaging the configuration parameters into message information, sending the message information to a message queue for a server to read the message information in the message queue, performing data processing on the message information to obtain a data extraction result, and sending the data extraction result and an identification label corresponding to the message information to the message queue;
and reading the data extraction result cached in the message queue based on the identification tag, and feeding back the data extraction result to the user interface.
2. The data extraction method of claim 1, wherein the step of inputting configuration parameters corresponding to the data extraction task in the user interface comprises:
inputting configuration parameters corresponding to data source information;
and selecting the table names of the big data source and the target source and configuring configuration parameters corresponding to the extraction frequency of the data extraction task.
3. The data extraction method according to claim 1, wherein the step of encapsulating the configuration parameters into message information, sending the message information to a message queue for a server to read the message information of the message queue, performing data processing on the message information to obtain a data extraction result, and sending the data extraction result and an identification tag corresponding to the message information to the message queue comprises:
carrying out system reinforcement on the configuration parameters to obtain reinforced configuration parameters;
and encapsulating the enhanced configuration parameters into message information, and sending the message information to the message queue.
4. The data extraction method of claim 1, wherein the step of reading the data extraction result buffered in the message queue based on the identification tag and feeding back the data extraction result to the user interface comprises:
when a data extraction result cached in the message queue is monitored, reading the data extraction result;
and feeding back the data extraction result to a user interface corresponding to the identification label based on the identification label.
5. The data extraction method as claimed in claim 1, wherein after the step of reading the data extraction result buffered in the message queue based on the identification tag and feeding back the data extraction result to the user interface, the data extraction method further comprises:
and informing a target user to view the data extraction result in the user interface for the target user.
6. A data extraction method is applied to a server side of a framework system, and is characterized in that the data extraction method comprises the following steps:
reading message information sent by a client when monitoring that the message information is cached in a message queue;
performing data processing on the message information to obtain a data extraction result;
and sending the data extraction result and the identification tag corresponding to the message information to the message queue so that the client receives the data extraction result in the message queue based on the identification tag.
7. The data extraction method according to claim 6, wherein the step of performing data processing on the message information to obtain a data extraction result comprises:
performing data analysis on the message information to obtain configuration parameters corresponding to the message information;
encapsulating the configuration parameters into data extraction parameters;
and calling a preset extraction method to perform data processing on the data extraction parameters to obtain the data extraction result.
8. A data extraction device characterized by comprising: a memory, a processor, and a data extraction program stored on the memory,
the data extraction program is executed by the processor to implement the steps of the data extraction method according to any one of claims 1 to 5 or 6 to 7.
9. A medium which is a readable storage medium, characterized in that the readable storage medium has stored thereon a data extraction program, the data extraction program being executed by a processor to implement the steps of the data extraction method according to any one of claims 1 to 5 or 6 to 7.
10. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the data extraction method according to any one of claims 1 to 5 or 6 to 7 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110940853.3A CN113392343A (en) | 2021-08-17 | 2021-08-17 | Data extraction method, device, medium and computer program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110940853.3A CN113392343A (en) | 2021-08-17 | 2021-08-17 | Data extraction method, device, medium and computer program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113392343A true CN113392343A (en) | 2021-09-14 |
Family
ID=77622634
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110940853.3A Pending CN113392343A (en) | 2021-08-17 | 2021-08-17 | Data extraction method, device, medium and computer program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113392343A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115037708A (en) * | 2022-08-10 | 2022-09-09 | 深圳星云智联科技有限公司 | Message processing method, system, device and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106250429A (en) * | 2016-07-26 | 2016-12-21 | 浪潮软件股份有限公司 | A kind of data pick-up method based on sqoop |
| CN110661849A (en) * | 2019-08-30 | 2020-01-07 | 中国人民财产保险股份有限公司 | Request processing method and device, electronic equipment and storage medium |
| CN111985446A (en) * | 2020-09-02 | 2020-11-24 | 深圳壹账通智能科技有限公司 | Face recognition method and system |
-
2021
- 2021-08-17 CN CN202110940853.3A patent/CN113392343A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106250429A (en) * | 2016-07-26 | 2016-12-21 | 浪潮软件股份有限公司 | A kind of data pick-up method based on sqoop |
| CN110661849A (en) * | 2019-08-30 | 2020-01-07 | 中国人民财产保险股份有限公司 | Request processing method and device, electronic equipment and storage medium |
| CN111985446A (en) * | 2020-09-02 | 2020-11-24 | 深圳壹账通智能科技有限公司 | Face recognition method and system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115037708A (en) * | 2022-08-10 | 2022-09-09 | 深圳星云智联科技有限公司 | Message processing method, system, device and computer readable storage medium |
| CN115037708B (en) * | 2022-08-10 | 2022-11-18 | 深圳星云智联科技有限公司 | Message processing method, system, device and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111083225B (en) | Data processing method and device in Internet of things platform and Internet of things platform | |
| US11477298B2 (en) | Offline client replay and sync | |
| CN104636421B (en) | Use the industry monitoring of cloud computing | |
| US9811400B2 (en) | End-to-end application tracking framework | |
| CN112261118B (en) | Multimedia data anomaly detection method, terminal and server | |
| CN110058987B (en) | Method, apparatus, and computer readable medium for tracking a computing system | |
| EP3128416B1 (en) | Sdn application integration, management and control method, system and device | |
| CN110224896B (en) | Network performance data acquisition method and device and storage medium | |
| CN109815107B (en) | Method and device for automatic testing | |
| CN107766509B (en) | Method and device for static backup of webpage | |
| US20210081263A1 (en) | System for offline object based storage and mocking of rest responses | |
| CN110232091B (en) | Method, system and apparatus for synchronizing data | |
| CN109947774B (en) | On-demand real-time sensor data distribution system | |
| US20150082286A1 (en) | Real-time code instrumentation | |
| CN115114044A (en) | Message pushing method, device, equipment and medium | |
| CN113392343A (en) | Data extraction method, device, medium and computer program product | |
| US9614900B1 (en) | Multi-process architecture for a split browser | |
| CN113342503A (en) | Real-time progress feedback method, device, equipment and storage medium | |
| CN112417016A (en) | Data exchange method, system, equipment and storage medium | |
| CN112511631A (en) | Control system and method of intelligent device | |
| CN113378346A (en) | Method and device for model simulation | |
| CN111767176A (en) | Method and device for remotely controlling terminal equipment | |
| CN107347024B (en) | Method, equipment and system for storing operation log | |
| US9525754B1 (en) | Task-based approach to execution, monitoring and execution control of actions | |
| US20190332416A1 (en) | Natively monitoring software services |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210914 |