CN114706578A - Data processing method, device, equipment and medium - Google Patents
Data processing method, device, equipment and medium Download PDFInfo
- Publication number
- CN114706578A CN114706578A CN202210313998.5A CN202210313998A CN114706578A CN 114706578 A CN114706578 A CN 114706578A CN 202210313998 A CN202210313998 A CN 202210313998A CN 114706578 A CN114706578 A CN 114706578A
- Authority
- CN
- China
- Prior art keywords
- task
- data
- processing
- mode
- instruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 172
- 238000013500 data storage Methods 0.000 claims abstract description 36
- 238000000034 method Methods 0.000 claims abstract description 30
- 238000004590 computer program Methods 0.000 claims description 16
- 238000013480 data collection Methods 0.000 claims description 6
- 230000004044 response Effects 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 2
- 238000013461 design Methods 0.000 abstract description 5
- 230000002452 interceptive effect Effects 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/38—Creation or generation of source code for implementing user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a data processing device, data processing equipment and a data processing medium, and belongs to the technical field of data processing. The method comprises the following steps: responding to task configuration operation on a task configuration interface, and generating a task configuration instruction according to the task configuration operation; generating a task processing mode according to the task configuration instruction; the task processing mode comprises a data acquisition mode, a data processing mode and a data storage mode; and processing the target task by adopting the task processing mode. By the technical scheme, configurable design is provided for batch processing of data offline data, a friendly interactive interface is provided for developers, and the efficiency of processing mass data is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method, apparatus, device, and medium.
Background
With the rapid development of the internet, a large amount of data can be processed in an enterprise every day, and designing a batch processing framework to process the large amount of data in the enterprise is a necessary consideration for modern enterprises. The batch processing refers to the collection, processing and processing of data without manual intervention according to a designed algorithm and flow in the process of processing the data.
The extraction of multiple data sources, the processing logic of data processing, the application and the data characteristics of data are primarily considered when designing a batch processing framework, and the processing result is stored in a corresponding database. During the data storage process, it is particularly important to perform visual statistical analysis on the data.
Disclosure of Invention
The invention provides a data processing method, a data processing device, data processing equipment and a data processing medium, which are used for improving the efficiency of processing mass data.
According to an aspect of the present invention, there is provided a data processing method, the method including:
responding to task configuration operation on a task configuration interface, and generating a task configuration instruction according to the task configuration operation;
generating a task processing mode according to the task configuration instruction; the task processing mode comprises a data acquisition mode, a data processing mode and a data storage mode;
and processing the target task by adopting the task processing mode.
According to another aspect of the present invention, there is provided a data processing apparatus comprising:
the task configuration instruction generation module responds to task configuration operation on a task configuration interface and generates a task configuration instruction according to the task configuration operation;
the task processing mode generating module is used for generating a task processing mode according to the task configuration instruction; the task processing mode comprises a data acquisition mode, a data processing mode and a data storage mode;
and the target task processing module is used for processing the target task by adopting the task processing mode.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the data processing method according to any of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement a data processing method according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the technical scheme of the embodiment of the invention, a task configuration instruction is generated according to task configuration operation by responding to the task configuration operation on a task configuration interface, and then a task processing mode is generated according to the task configuration instruction; the task processing mode comprises a data acquisition mode, a data processing mode and a data storage mode, and further the target task is processed by adopting the task processing mode. According to the technical scheme, the task configuration interface is introduced, configurable design is provided for batch processing of data offline data, a friendly interactive interface is provided for developers, and the efficiency of processing mass data is improved.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1A is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 1B is a diagram illustrating a task configuration interface according to an embodiment of the present invention;
FIG. 1C is a schematic diagram of a data collection sub-interface according to an embodiment of the present invention;
fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention;
FIG. 3 is a flowchart of a data processing method according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device implementing the data processing method according to the embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "target," "original," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example one
Fig. 1A is a flowchart of a data processing method according to an embodiment of the present invention. The present embodiment is applicable to a case how to visualize processing manners involved in a data batch processing process, and the method may be executed by a data processing apparatus, where the apparatus may be implemented in a form of hardware and/or software, and may be integrated in an electronic device bearing a data processing function, for example, a data processing platform in a server. As shown in fig. 1A, the data processing method of the present embodiment may include:
and S110, responding to the task configuration operation on the task configuration interface, and generating a task configuration instruction according to the task configuration operation.
In this embodiment, the task configuration interface is an interface for configuring a data processing task. The task configuration operation is the configuration operation of a data user on a data processing task in a task configuration interface. The task configuration command indicates the quality of a processing method for generating a task.
Specifically, the developer can log in the task configuration interface by using the account and the password, and perform task configuration operation on the task configuration interface. As shown in fig. 1B, for example, a developer may set a task name and a task flow, for example, perform data processing after acquiring yesterday device information, and analyze an ACC operating state, so as to determine whether to clear history data according to the ACC operating state, and obtain a data result. Correspondingly, the background server responds to task configuration operation of a developer on the task configuration interface and generates a corresponding task configuration instruction according to the task configuration operation.
For example, a developer may click a data collection button on a task configuration interface, enter a data collection sub-interface, and perform a data collection configuration operation on the data collection sub-interface, and as shown in fig. 1C, may set a data source name, a data source type, a task priority, a number of failed retries, a failed retries interval, a structured query (sql) statement, and the like. Correspondingly, the server background responds to the data acquisition configuration operation on the data acquisition sub-interface of the task configuration interface and generates a data acquisition instruction in the task configuration instruction according to the data acquisition configuration operation.
For example, the developer may click a data processing button on the task configuration interface, enter a data processing sub-interface, and perform a data processing configuration operation on the data processing sub-interface, for example, select some data calculation algorithms (find difference, etc.). Correspondingly, the server background responds to the data processing configuration operation on the data processing sub-interface of the task configuration interface and generates the data processing instruction in the task configuration instruction according to the data processing configuration operation.
For example, the developer may also click a data storage button on the task configuration interface, enter the data storage sub-interface, and perform a data storage configuration operation in the data storage sub-interface, such as configuring a data source, a data storage source, a storage table name, and the like. Correspondingly, responding to the storage configuration operation of the data storage sub-interface of the task configuration interface, and generating the data storage instruction in the task configuration instruction according to the storage configuration operation.
And S120, generating a task processing mode according to the task configuration instruction.
In this embodiment, the task processing mode includes a data acquisition mode, a data processing mode, and a data storage mode; the data acquisition mode is used for acquiring data; the data processing mode is a processing mode such as calculation for data; the data storage method is how to store the processed data.
Specifically, the data acquisition mode can be generated according to the data acquisition command based on the conversion relation between the command and the data processing mode; generating a data processing mode according to the data processing instruction; and generating a data storage mode according to the data storage instruction.
And S130, processing the target task by adopting a task processing mode.
In this embodiment, the target task is a task that needs to perform data processing, and the optional target task may include at least one subtask.
Specifically, a task processing mode is adopted to process the target task.
It should be noted that, the sub-tasks in the target task may also be paused or started.
According to the technical scheme of the embodiment of the invention, a task configuration instruction is generated according to task configuration operation by responding to the task configuration operation on a task configuration interface, and then a task processing mode is generated according to the task configuration instruction; the task processing mode comprises a data acquisition mode, a data processing mode and a data storage mode, and further the target task is processed by adopting the task processing mode. According to the technical scheme, the task configuration interface is introduced, configurable design is provided for batch processing of data offline data, a friendly interactive interface is provided for developers, and the efficiency of processing mass data is improved.
Example two
Fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention. On the basis of the above embodiment, the present embodiment further optimizes "processing a target task by using a task processing manner", and provides an optional implementation. As shown in fig. 2, the data processing method of the present embodiment may include:
s210, responding to task configuration operation on the task configuration interface, and generating a task configuration instruction according to the task configuration operation.
And S220, generating a task processing mode according to the task configuration instruction.
And S230, extracting the original data corresponding to the target task according to the data acquisition mode to obtain extracted data.
In this embodiment, the data acquisition manner includes a data source type, a data source name, and a first structured query statement, where the data source may include different types of data sources such as hive, mysql, hbase, clickhouse, es, and the like; the first structured query statement is an sql statement, which can be set by a person skilled in the art according to actual needs.
Illustratively, according to the type and the name of a data source, original data corresponding to a target task is obtained; and extracting the original data according to the first structured query statement to obtain extracted data. For example, according to the type and name of the data source, the original data corresponding to the target task is obtained from the corresponding database, and then the original data is queried according to the first structured query statement, and the queried result is used as the extracted data. A specific example is to look up yesterday and today's device information from all device information.
For example, if a data extraction failure is identified, original data corresponding to the target task may be re-extracted according to at least one of the number of failed retries and the failed retries interval.
And S240, processing the extracted data according to the data processing mode to obtain target data.
Specifically, the extracted data may be processed according to a data processing rule in the data processing mode to obtain the target data. A specific example is to query yesterday and today's device information, compare yesterday's device information with today's device information, and use the result of the comparison as target data.
And S250, storing the target data according to the data storage mode.
In this embodiment, the data storage manner may include a data source, a storage type, a storage format, and the like.
Specifically, the target data can be stored according to a storage rule specified in the data storage mode, so that subsequent related personnel can check the target data.
According to the technical scheme of the embodiment of the invention, the task configuration instruction is generated according to the task configuration operation in response to the task configuration operation on the task configuration interface, then the task processing mode is generated according to the task configuration instruction, further the original data corresponding to the target task is extracted according to the data acquisition mode to obtain the extracted data, the extracted data is processed according to the data processing mode to obtain the target data, and the target data is stored according to the data storage mode. According to the technical scheme, the data acquisition mode is visualized, a user can conveniently check the data acquisition process, and therefore the data processing efficiency is improved.
EXAMPLE III
FIG. 3 is a flowchart of a data processing method according to a third embodiment of the present invention; on the basis of the above embodiments, the present embodiment further optimizes "processing a target task by using a task processing method", and provides an optional implementation scheme. As shown in fig. 3, the data processing method of the present embodiment may include:
and S310, responding to the task configuration operation on the task configuration interface, and generating a task configuration instruction according to the task configuration operation.
And S320, generating a task processing mode according to the task configuration instruction.
And S330, processing the target task by adopting a task processing mode when the task execution time is reached.
In this embodiment, an offline timing operation may also be performed on the target task, and if it is recognized that the developer or the user performs an offline task timing configuration operation on the target task, the target task is processed by using a task processing manner when the task execution time is reached.
According to the technical scheme of the embodiment, the task configuration instruction is generated according to the task configuration operation in response to the task configuration operation on the task configuration interface, then the task processing mode is generated according to the task configuration instruction, and the target task is processed by adopting the task processing mode when the task execution time is reached. According to the technical scheme, the target task is set in an off-line timing mode, data processing can be flexibly performed, and the data processing efficiency is improved.
On the basis of the foregoing embodiment, as an optional mode of the present invention, a task processing mode may be adopted, and the target task may be processed by sequentially adopting the task processing modes according to the task priorities.
Specifically, when there are at least two subtasks in the target task, the subtasks in the target task are processed by sequentially adopting a task processing mode according to the configured priority of each subtask.
Example four
Fig. 4 is a schematic structural diagram of a data processing apparatus according to a fourth embodiment of the present invention. The embodiment is applicable to the case of how to visualize the processing modes involved in the data batch processing process, and the apparatus may be implemented in the form of hardware and/or software, and may be integrated in an electronic device bearing a data processing function, for example, a server. As shown in fig. 4, the data processing apparatus of the present embodiment may include:
the task configuration instruction generating module 410 is used for responding to task configuration operation on the task configuration interface and generating a task configuration instruction according to the task configuration operation;
a task processing mode generating module 420, configured to generate a task processing mode according to the task configuration instruction; the task processing mode comprises a data acquisition mode, a data processing mode and a data storage mode;
and the target task processing module 430 is configured to process the target task in a task processing manner.
According to the technical scheme of the embodiment of the invention, a task configuration instruction is generated according to task configuration operation by responding to the task configuration operation on a task configuration interface, and then a task processing mode is generated according to the task configuration instruction; the task processing mode comprises a data acquisition mode, a data processing mode and a data storage mode, and further the target task is processed by adopting the task processing mode. According to the technical scheme, the task configuration interface is introduced, the configurable design is provided for batch processing of the data offline data, a friendly interactive interface is provided for developers, and the efficiency of processing mass data is improved.
Optionally, the task configuration instruction generating module 410 is specifically configured to:
responding to data acquisition configuration operation on a data acquisition sub-interface of the task configuration interface, and generating a data acquisition instruction in the task configuration instruction according to the data acquisition configuration operation;
responding to data processing configuration operation on a data processing sub-interface of the task configuration interface, and generating a data processing instruction in the task configuration instruction according to the data processing configuration operation;
and responding to the storage configuration operation of the data storage sub-interface of the task configuration interface, and generating a data storage instruction in the task configuration instruction according to the storage configuration operation.
Optionally, the task processing manner generating module 420 is specifically configured to:
generating a data acquisition mode according to the data acquisition instruction;
generating a data processing mode according to the data processing instruction;
and generating a data storage mode according to the data storage instruction.
Optionally, the target task processing module 430 includes:
the extracted data determining unit is used for extracting the original data corresponding to the target task according to the data acquisition mode to obtain extracted data;
the target data determining unit is used for processing the extracted data according to the data processing mode to obtain target data;
and the target data storage unit is used for storing the target data according to the data storage mode.
Optionally, the data acquisition mode includes a data source type, a data source name, and a first structured query statement; correspondingly, the extracted data determining unit is specifically configured to:
acquiring original data corresponding to the target task according to the type and the name of the data source;
and extracting the original data according to the first structured query statement to obtain extracted data.
Optionally, the target task processing module 430 is specifically configured to:
and when the task execution time is reached, processing the target task by adopting a task processing mode.
Optionally, the target task processing module 430 is further specifically configured to:
and processing the subtasks in the target task by sequentially adopting a task processing mode according to the task priority.
The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE five
Fig. 5 is a schematic structural diagram of an electronic device implementing the data processing method according to the embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 5, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM)12, a Random Access Memory (RAM)13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM)12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as a data processing method.
In some embodiments, the data processing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data processing method by any other suitable means (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A data processing method, comprising:
responding to task configuration operation on a task configuration interface, and generating a task configuration instruction according to the task configuration operation;
generating a task processing mode according to the task configuration instruction; the task processing mode comprises a data acquisition mode, a data processing mode and a data storage mode;
and processing the target task by adopting the task processing mode.
2. The method of claim 1, wherein generating task configuration instructions in accordance with task configuration operations in response to the task configuration operations at the task configuration interface comprises:
responding to data acquisition configuration operation on a data acquisition sub-interface of the task configuration interface, and generating a data acquisition instruction in a task configuration instruction according to the data acquisition configuration operation;
responding to data processing configuration operation on a data processing sub-interface of the task configuration interface, and generating a data processing instruction in the task configuration instruction according to the data processing configuration operation;
and responding to the storage configuration operation of the data storage sub-interface of the task configuration interface, and generating a data storage instruction in the task configuration instruction according to the storage configuration operation.
3. The method according to claim 2, wherein the generating a task processing manner according to the task configuration instruction comprises:
generating a data acquisition mode according to the data acquisition instruction;
generating a data processing mode according to the data processing instruction;
and generating a data storage mode according to the data storage instruction.
4. The method according to claim 1, wherein the processing the target task by using the task processing manner includes:
extracting the original data corresponding to the target task according to the data acquisition mode to obtain extracted data;
processing the extracted data according to the data processing mode to obtain target data;
and storing the target data according to the data storage mode.
5. The method of claim 4, wherein the data collection modality comprises a data source type, a data source name, and a first structured query statement;
correspondingly, the extracting the original data corresponding to the target task according to the data acquisition mode to obtain extracted data includes:
acquiring original data corresponding to the target task according to the data source type and the data source name;
and extracting the original data according to the first structured query statement to obtain extracted data.
6. The method according to claim 1, wherein the processing the target task by using the task processing manner includes:
and when the task execution time is reached, processing the target task by adopting the task processing mode.
7. The method according to claim 1, wherein the processing the target task by using the task processing manner includes:
and processing the subtasks in the target task by sequentially adopting the task processing mode according to the task priority.
8. A data processing apparatus, comprising:
the task configuration instruction generation module responds to task configuration operation on a task configuration interface and generates a task configuration instruction according to the task configuration operation;
the task processing mode generating module is used for generating a task processing mode according to the task configuration instruction; the task processing mode comprises a data acquisition mode, a data processing mode and a data storage mode;
and the target task processing module is used for processing the target task by adopting the task processing mode.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data processing method of any one of claims 1-7.
10. A computer-readable storage medium, characterized in that it stores computer instructions for causing a processor to implement the data processing method of any of claims 1-7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210313998.5A CN114706578A (en) | 2022-03-28 | 2022-03-28 | Data processing method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210313998.5A CN114706578A (en) | 2022-03-28 | 2022-03-28 | Data processing method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114706578A true CN114706578A (en) | 2022-07-05 |
Family
ID=82170674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210313998.5A Pending CN114706578A (en) | 2022-03-28 | 2022-03-28 | Data processing method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114706578A (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101533417A (en) * | 2009-04-28 | 2009-09-16 | 阿里巴巴集团控股有限公司 | A method and system for realizing ETL scheduling |
| CN109918437A (en) * | 2019-03-08 | 2019-06-21 | 北京中油瑞飞信息技术有限责任公司 | Distributed data processing method, apparatus and data assets management system |
-
2022
- 2022-03-28 CN CN202210313998.5A patent/CN114706578A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101533417A (en) * | 2009-04-28 | 2009-09-16 | 阿里巴巴集团控股有限公司 | A method and system for realizing ETL scheduling |
| CN109918437A (en) * | 2019-03-08 | 2019-06-21 | 北京中油瑞飞信息技术有限责任公司 | Distributed data processing method, apparatus and data assets management system |
Non-Patent Citations (1)
| Title |
|---|
| 张阳: "大数据可视化统计分析通用平台的设计与实现", 《中国优秀硕士学位论文全文数据库》, no. 2020, 15 January 2020 (2020-01-15), pages 138 - 385 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115146000A (en) | Database data synchronization method and device, electronic equipment and storage medium | |
| CN112925811B (en) | Method, apparatus, device, storage medium and program product for data processing | |
| CN115801589B (en) | Event topological relation determination method, device, equipment and storage medium | |
| CN115048352B (en) | Log field extraction method, device, equipment and storage medium | |
| CN116545905A (en) | Service health detection method and device, electronic equipment and storage medium | |
| CN115934550A (en) | Test method, test device, electronic equipment and storage medium | |
| CN115454971A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN114706578A (en) | Data processing method, device, equipment and medium | |
| CN114691781A (en) | Data synchronization method, system, device, equipment and medium | |
| CN113656239A (en) | Monitoring method and device for middleware and computer program product | |
| CN116431698B (en) | Data extraction method, device, equipment and storage medium | |
| CN116821217A (en) | Data distribution conversion method, device, equipment and storage medium | |
| CN115757304A (en) | Log storage method, device and system, electronic equipment and storage medium | |
| CN114756398A (en) | Data processing method, device, equipment and medium | |
| CN115455060A (en) | Data processing method, device, equipment and medium | |
| CN118132536A (en) | Data migration method, device, equipment and storage medium | |
| CN114416881A (en) | Real-time synchronization method, device, equipment and medium for multi-source data | |
| CN117632120A (en) | Processing system, method, equipment and storage medium for report data | |
| CN117081939A (en) | Traffic data processing method, device, equipment and storage medium | |
| CN115730000A (en) | Medical data integration method, device, equipment and medium based on data lake | |
| CN114996243A (en) | Database operation method and device, electronic equipment and storage medium | |
| CN115567624A (en) | Message processing method and device, electronic equipment and medium | |
| CN116401269A (en) | Data query method and device, electronic equipment and storage medium | |
| CN115794255A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN115983222A (en) | EasyExcel-based file data reading method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |