CN114816583A - Flink-based data automatic processing method and device and electronic equipment - Google Patents
Flink-based data automatic processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN114816583A CN114816583A CN202210608800.6A CN202210608800A CN114816583A CN 114816583 A CN114816583 A CN 114816583A CN 202210608800 A CN202210608800 A CN 202210608800A CN 114816583 A CN114816583 A CN 114816583A
- Authority
- CN
- China
- Prior art keywords
- module
- task
- flink
- web
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title description 5
- 238000012545 processing Methods 0.000 claims abstract description 112
- 238000000034 method Methods 0.000 claims abstract description 58
- 230000003993 interaction Effects 0.000 claims abstract description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 230000000694 effects Effects 0.000 abstract description 5
- 238000004891 communication Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 238000004140 cleaning Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G06F9/4451—User profiles; Roaming
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method and a device for automatically processing data based on Flink and electronic equipment, and relates to the technical field of data processing, wherein the method is applied to a webpage loading terminal, and the webpage loading terminal comprises the following steps: the system comprises a web module, a database module and a job processing module; the web module is used for carrying out data interaction with the database module; the web module is also used for sending a job instruction to the job processing module; the method comprises the following steps: responding to an execution instruction aiming at the web module, and calling flink task configuration information corresponding to the execution instruction in the database module; executing the instruction includes a data processing task; generating a task configuration file based on the flink task configuration information; and starting the flight job corresponding to the task configuration file by using the job processing module so as to complete the data processing task. The technical problem that other jobs are blocked in operation after the flight task is interrupted is solved, and the technical effect of optimizing data processing is achieved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a method and a device for automatically processing data based on Flink and electronic equipment.
Background
The current data automation processing scheme based on the data stream Flink is generally as follows: the method comprises the steps that information of a data processing task configured on a web page by a user is stored through a database module, when the user clicks on the web page to execute the task, the web module reads information data such as data source connection information, data processing logic, target library connection information and the like stored in the database module, then the information data is sent to an information queue module, and finally an operation processing module which monitors the information queue module all the time reads the information to perform operation processing, so that automatic processing of the data is completed finally.
In practical application, data processing tasks to be completed are continuously added, and a "job processing module" in a traditional mode is likely to cause the current module to interrupt operation due to the exception of a certain task job or other reasons, so that normal processing of other task jobs is blocked. That is to say, there is a technical problem that after the flight task is interrupted, the rest of the jobs are blocked in operation in the existing data processing technology.
Disclosure of Invention
Accordingly, the present invention is directed to a method, an apparatus and an electronic device for alleviating the above-mentioned problems in the prior art.
In order to achieve the above purpose, the embodiment of the present invention adopts the following technical solutions:
in a first aspect, an embodiment of the present invention provides a method for automatically processing data based on Flink, where the method is applied to a web page loading terminal, where the web page loading terminal includes: the system comprises a web module, a database module and a job processing module; the web module is used for carrying out data interaction with the database module; the web module is also used for sending a job instruction to the job processing module; the method comprises the following steps: responding to an execution instruction aiming at the web module, and calling flink task configuration information corresponding to the execution instruction in the database module; the execution instruction comprises a data processing task; generating a task configuration file based on the flink task configuration information; and starting the flight operation corresponding to the task configuration file by using the operation processing module so as to complete the data processing task.
In some possible embodiments, the method further comprises: constructing a data processing task on the preloaded webpage by utilizing the web module; and sending the configuration information of the data processing task to the database module and storing the configuration information.
In some possible embodiments, the executing the instruction includes: data input, data cleansing and data output.
In some possible embodiments, the flink task configuration information includes: source component information and sink component information.
In some possible embodiments, the method further comprises: and sending the task configuration file to a flink server through a web interface.
In some possible embodiments, the web interface is generated by the job processing module requesting the web module.
In a second aspect, an embodiment of the present invention provides an apparatus for automated data processing based on Flink, including:
the webpage loading terminal is used for responding to the execution instruction aiming at the web module and calling the Flink task configuration information corresponding to the execution instruction in the database module; the execution instruction comprises a data processing task; the webpage loading terminal is also used for generating a task configuration file based on the Flink task configuration information; the web page loading terminal is further configured to start a Flink job corresponding to the task configuration file by using the job processing module, so as to complete the data processing task.
In some possible embodiments, the web page loading terminal is further configured to: constructing a data processing task on the preloaded webpage by utilizing the web module; and sending the configuration information of the data processing task to the database module and storing the configuration information.
In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, where the memory stores a computer program operable on the processor, and the processor implements the steps of the method according to any one of the first aspect when executing the computer program.
In a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing machine executable instructions that, when invoked and executed by a processor, cause the processor to perform the method of any of the first aspects.
The invention provides a method, a device and electronic equipment for automatically processing data based on Flink, relating to the technical field of data processing, wherein the method is applied to a webpage loading terminal, and the webpage loading terminal comprises the following steps: the system comprises a web module, a database module and a job processing module; the web module is used for carrying out data interaction with the database module; the web module is also used for sending a job instruction to the job processing module; the method comprises the following steps: responding to an execution instruction aiming at the web module, and calling flink task configuration information corresponding to the execution instruction in the database module; executing the instruction includes a data processing task; generating a task configuration file based on the flink task configuration information; and starting the flight job corresponding to the task configuration file by using the job processing module so as to complete the data processing task. The technical problem that other jobs are blocked in operation after the flight task is interrupted is solved, and the technical effect of optimizing data processing is achieved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic structural diagram of a conventional data automated processing system according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a web page loading terminal according to an embodiment of the present invention;
fig. 3 is a schematic flowchart of a method for automated data processing based on Flink according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Some embodiments of the invention are described in detail below with reference to the accompanying drawings. The embodiments described below and the features of the embodiments can be combined with each other without conflict.
Referring to fig. 1, a schematic structural diagram of a conventional data automation processing system, a conventional data automation processing scheme based on a data stream Flink generally includes: the information of the data processing task configured on the web page by the user is stored through the "database module" 110, when the task is clicked and executed on the web page, the "web module" 120 reads the information data such as the "data source connection information", "data processing logic", and "connection information of the target library" stored in the "database module" 110, then sends the information data to the "message queue module" 130, and finally, the "job processing module" 140 which monitors the "message queue module" all the time reads the information to perform job processing, and finally, the automatic processing of the data is completed. In practical applications, data processing tasks to be completed are continuously added, and the "job processing module" 140 in the conventional mode is likely to cause the current module to interrupt operation due to an exception of a certain task job or other reasons, thereby blocking normal processing of other task jobs. That is to say, there is a technical problem that after the flight task is interrupted, the rest of the jobs are blocked in operation in the existing data processing technology.
Based on the above, the embodiment of the invention provides a method and a device for automatically processing data based on Flink and electronic equipment.
To facilitate understanding of the embodiment, first, a detailed description is given to a method for automated data processing based on Flink disclosed in the embodiment of the present invention, where the method is applied to a web page loading terminal, and referring to a schematic structural diagram of a web page loading terminal shown in fig. 2, the web page loading terminal includes: web module 210, database module 220, and job processing module 230; the web module 210 is used for data interaction with the database module 220; the web module 210 is also used to send job instructions to the job processing module 230.
Referring to fig. 3, a flow chart of a method for automated Flink-based data processing is shown, which may be performed by an electronic device. The automatic data processing method based on the Flink mainly comprises the following steps S101 to S103:
s101: responding to an execution instruction aiming at the web module, calling Flink task configuration information corresponding to the execution instruction in the database module; executing the instruction includes a data processing task;
s102: generating a task configuration file based on the Flink task configuration information;
s103: and starting the Flink job corresponding to the task configuration file by using the job processing module so as to complete the data processing task.
Wherein, the executing instruction may include: data input, data cleansing and data output.
The Flink task configuration information may include: source component information and sink component information. As a specific example, in the embodiment of the present application, the information of the source component and the sink component of the Flink task are both written into the yaml configuration file, the configuration file is specified when the Flink task is started, and then "reading-processing-warehousing" of data can be completed according to the configuration items of the source and the sink.
In one embodiment, the method further comprises:
(1) constructing a data processing task on the preloaded webpage by using a web module;
(2) and sending the configuration information of the data processing task to the database module and storing the configuration information.
In one embodiment, the foregoing may further include: and sending the task configuration file to a Flink server through a web interface. Wherein the web interface is generated by a job processing module requesting a web module. The embodiment of the application provides a data automatic processing method based on Flink, which generates task configuration files required by the Flink operation according to the configuration information of the tasks in a database module and provides web interface services of file contents, so that the data processing tasks correspond to the Flink operation one by one, and the problem of operation blockage of other operations after the whole Flink task is interrupted due to abnormal operation of a certain operation in the prior art is solved. In addition, the webpage loading terminal structure provided by the embodiment of the application cancels a message queue module between a web module and a job processing module in the traditional architecture, simplifies the architecture of the existing method and solves the architectural design of the defects of the prior art.
It should be noted that the one-to-one correspondence between the data processing task and the Flink job means: when a configured task is started on a page, a background correspondingly starts a Flink job to realize the data processing function of the task. The specific process is as follows: starting a task by a web page; the back-end code calls an interface of an uploading jar packet carried by the Flink (uploads a Flink program packet to the Flink service); calling a job running interface of the flight (dependent on a jar packet uploaded in the last step, appointing a yaml configuration file and starting flight operation) by using the back-end code; and (5) running the flight job to complete the task corresponding to the web page.
As a specific example, an embodiment of the present application provides a method for automatically processing data based on Flink, where the method is applied to a web page loading terminal, and the web page loading terminal includes: the system comprises a web module, a database module and a job processing module; the web module is used for carrying out data interaction with the database module; the web module is also used for sending a job instruction to the job processing module. The method comprises the following data processing procedures:
(1) a user pulls three types of components of 'input', 'cleaning' and 'output' in a dragging mode on a web page in a 'web module', selects information of 'data source', 'cleaning rule', 'target library' and the like, and selects and submits the information to construct a complete data processing task;
(2) after the designed components submit and form tasks, the configuration information of the current task is stored in a database module;
(3) a user clicks and executes a certain task on a web page in a 'web module' to start processing jobs of the corresponding task;
(4) the back-end code of the web module pulls the configuration information of the corresponding task from the database module and generates a file with a suffix of the ". yaml" (for finishing data processing based on the information after reading the Flink task);
(5) the back end code of the 'web module' generates a yaml file, and then starts a Flink task of the 'job processing module' by calling a Flink Rest API (a configuration file path of a 'yaml' suffix is assigned to the current Flink task in the calling parameter);
(6) after being called and started, the flight task of the 'job processing module' requests an 'interface for acquiring file content' provided by the 'web module' to acquire the content of the '. yaml' file (stored in the web server) generated in the step (4) and writes the content of the '. yaml' file (stored in the flight server) specified for the current flight task in the step (5);
(7) after the Flink task of the 'job processing module' writes the new task configuration information (the content of the file 'yaml' in the step (4)) into the configuration file (the file 'yaml' in the step (5)) specified by the current task, the connection and data pulling of the 'data source' in the step (1) can be completed according to the information, the execution of the 'cleaning rule' of the data is realized, and finally the connection and data warehousing of the 'target library' is realized;
(8) the 'web module' monitors the running state of the Flink task through the Flink Rest API at regular time and feeds back the running state to the web page.
It should be noted that: the configuration file of the Flink task can only load the file on the Flink server, and the server where the Flink is located and the web server are not the same machine, so the configuration file of the web server needs to be read and generated to the Flink server through the web interface. Wherein, the web interface is an interface for acquiring the file content and is provided by the web module in the step (6); the configuration file is the ". yaml" file in step (4).
The embodiment of the invention also provides a data automatic processing device based on Flink, which comprises: the webpage loading terminal is used for responding to the execution instruction aiming at the web module and calling the Flink task configuration information corresponding to the execution instruction in the database module; executing the instruction includes a data processing task;
the webpage loading terminal is also used for generating a task configuration file based on the Flink task configuration information; the webpage loading terminal is also used for starting the flight operation corresponding to the task configuration file by using the operation processing module so as to complete the data processing task.
In an embodiment, the web page loading terminal may be further configured to: constructing a data processing task on the preloaded webpage by using a web module; and sending the configuration information of the data processing task to the database module and storing the configuration information.
The invention provides a method, a device and electronic equipment for automatically processing data based on Flink, wherein the method is applied to a webpage loading terminal, and the webpage loading terminal comprises the following steps: the system comprises a web module, a database module and a job processing module; the web module is used for carrying out data interaction with the database module; the web module is also used for sending a job instruction to the job processing module; the method comprises the following steps: responding to an execution instruction aiming at the web module, calling Flink task configuration information corresponding to the execution instruction in the database module; executing the instruction includes a data processing task; generating a task configuration file based on the Flink task configuration information; the method solves the technical problem that after the flight task is interrupted, the other tasks are blocked in operation, and achieves the technical effect of optimizing data processing.
The Flink-based data automation processing device provided by the embodiment of the application can be specific hardware on a device or software or firmware installed on the device. The device provided by the embodiment of the present application has the same implementation principle and technical effect as the foregoing method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the foregoing method embodiments where no part of the device embodiments is mentioned. It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the foregoing systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again. The Flink-based data automated processing device provided by the embodiment of the application has the same technical characteristics as the Flink-based data automated processing method provided by the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
The embodiment of the application further provides an electronic device, and specifically, the electronic device comprises a processor and a storage device; the storage means has stored thereon a computer program which, when executed by the processor, performs the method of any of the above described embodiments.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application, where the electronic device 400 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43, wherein the processor 40, the communication interface 43 and the memory 41 are connected through the bus 42; the processor 40 is arranged to execute executable modules, such as computer programs, stored in the memory 41.
The Memory 41 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, etc. may be used.
The bus 42 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 4, but that does not indicate only one bus or one type of bus.
The memory 41 is used for storing a program, the processor 40 executes the program after receiving an execution instruction, and the method executed by the apparatus defined by the flow process disclosed in any of the foregoing embodiments of the present invention may be applied to the processor 40, or implemented by the processor 40.
The processor 40 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 40. The Processor 40 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in a memory 41, and the processor 40 reads the information in the memory 41 and completes the steps of the method in combination with the hardware thereof.
Corresponding to the method, the embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores machine executable instructions, and when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to execute the steps of the method.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, an electronic device, or a network device) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It should be noted that: like reference numbers and letters indicate like items in the figures, and thus once an item is defined in a figure, it need not be further defined or explained in subsequent figures, and moreover, the terms "first," "second," "third," etc. are used merely to distinguish one description from another and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for automatically processing data based on Flink is characterized in that the method is applied to a webpage loading terminal, and the webpage loading terminal comprises the following steps: the system comprises a web module, a database module and a job processing module; the web module is used for carrying out data interaction with the database module; the web module is also used for sending a job instruction to the job processing module;
the method comprises the following steps:
responding to an execution instruction aiming at the web module, and calling Flink task configuration information corresponding to the execution instruction in the database module; the execution instruction comprises a data processing task;
generating a task configuration file based on the Flink task configuration information;
and starting the Flink job corresponding to the task configuration file by using the job processing module so as to complete the data processing task.
2. The method of claim 1, further comprising:
constructing a data processing task on the preloaded webpage by using the web module;
and sending the configuration information of the data processing task to the database module and storing the configuration information.
3. The method of claim 1, wherein executing the instruction comprises: data input, data cleansing and data output.
4. The method according to claim 1, wherein the Flink task configuration information comprises: source component information and sink component information.
5. The method of claim 1, further comprising:
and sending the task configuration file to a Flink server through a web interface.
6. The method of claim 5, wherein the web interface is generated by the job processing module requesting generation from the web module.
7. A Flink-based data automation processing device, comprising:
the webpage loading terminal is used for responding to an execution instruction aiming at the web module and calling the Flink task configuration information corresponding to the execution instruction in the database module; the execution instruction comprises a data processing task;
the webpage loading terminal is also used for generating a task configuration file based on the Flink task configuration information;
and the webpage loading terminal is also used for starting the flight operation corresponding to the task configuration file by using an operation processing module so as to complete the data processing task.
8. The apparatus of claim 7, wherein the web page loading terminal is further configured to: constructing a data processing task on the preloaded webpage by utilizing the web module; and sending the configuration information of the data processing task to the database module and storing the configuration information.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program operable on the processor, and wherein the processor implements the steps of the method of any of claims 1 to 6 when executing the computer program.
10. A computer readable storage medium having stored thereon machine executable instructions which, when invoked and executed by a processor, cause the processor to execute the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210608800.6A CN114816583B (en) | 2022-05-31 | 2022-05-31 | Flink-based data automatic processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210608800.6A CN114816583B (en) | 2022-05-31 | 2022-05-31 | Flink-based data automatic processing method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114816583A true CN114816583A (en) | 2022-07-29 |
| CN114816583B CN114816583B (en) | 2024-03-19 |
Family
ID=82518842
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210608800.6A Active CN114816583B (en) | 2022-05-31 | 2022-05-31 | Flink-based data automatic processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114816583B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115756586A (en) * | 2022-11-25 | 2023-03-07 | 中电金信软件有限公司 | Method and device for executing Flink job, computer equipment and storage medium |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190034221A1 (en) * | 2017-07-30 | 2019-01-31 | International Business Machines Corporation | Persisting distributed data sets into eventually consistent storage systems |
| US20190114251A1 (en) * | 2017-10-18 | 2019-04-18 | Proov Systems Ltd. | Software proof-of-concept platform, including simulation of production behavior and/or data |
| CN111061715A (en) * | 2019-12-16 | 2020-04-24 | 北京邮电大学 | Web and Kafka-based distributed data integration system and method |
| CN111930700A (en) * | 2020-07-13 | 2020-11-13 | 车智互联（北京）科技有限公司 | Distributed log processing method, server, system and computing equipment |
| CN112130976A (en) * | 2020-09-21 | 2020-12-25 | 厦门南讯股份有限公司 | REST-based multi-engine big data task management method |
| KR102201651B1 (en) * | 2020-02-04 | 2021-01-11 | 강원대학교산학협력단 | Probability-based data stream partitioning method considering task locality and downstream status |
| CN112286905A (en) * | 2020-10-15 | 2021-01-29 | 北京沃东天骏信息技术有限公司 | Data migration method and device, storage medium and electronic equipment |
| CN112328458A (en) * | 2020-11-27 | 2021-02-05 | 杭州安恒信息技术股份有限公司 | Data processing method and device based on flink data engine |
| CN112558995A (en) * | 2020-12-24 | 2021-03-26 | 恩亿科(北京)数据科技有限公司 | Flink integration method and system based on TBDS Hadoop |
| CN112765166A (en) * | 2021-01-06 | 2021-05-07 | 深圳市欢太科技有限公司 | Data processing method, device and computer readable storage medium |
| US11010191B1 (en) * | 2020-07-02 | 2021-05-18 | Ryan L. Hornbeck | Platform-independent interface for generating virtualized multi-service hardware systems and infrastructure |
| CN112835924A (en) * | 2021-02-04 | 2021-05-25 | 北京高途云集教育科技有限公司 | Real-time computing task processing method, device, equipment and storage medium |
| CN113010512A (en) * | 2021-02-24 | 2021-06-22 | 上海中通吉网络技术有限公司 | Real-time data processing method, platform and equipment based on Flink |
| US20210334385A1 (en) * | 2020-04-27 | 2021-10-28 | Oracle International Corporation | Greybox fuzzing for web applications |
| CN113656157A (en) * | 2021-08-10 | 2021-11-16 | 北京锐安科技有限公司 | Distributed task scheduling method and device, storage medium and electronic equipment |
| US20220004369A1 (en) * | 2020-07-01 | 2022-01-06 | Johnson Controls Tyco IP Holdings LLP | Rule builder and simulator tool for creating complex event processing rules |
| US11275726B1 (en) * | 2020-12-06 | 2022-03-15 | Kamu Data Inc. | Distributed data processing method with complete provenance and reproducibility |
-
2022
- 2022-05-31 CN CN202210608800.6A patent/CN114816583B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190034221A1 (en) * | 2017-07-30 | 2019-01-31 | International Business Machines Corporation | Persisting distributed data sets into eventually consistent storage systems |
| US20190114251A1 (en) * | 2017-10-18 | 2019-04-18 | Proov Systems Ltd. | Software proof-of-concept platform, including simulation of production behavior and/or data |
| CN111061715A (en) * | 2019-12-16 | 2020-04-24 | 北京邮电大学 | Web and Kafka-based distributed data integration system and method |
| KR102201651B1 (en) * | 2020-02-04 | 2021-01-11 | 강원대학교산학협력단 | Probability-based data stream partitioning method considering task locality and downstream status |
| US20210334385A1 (en) * | 2020-04-27 | 2021-10-28 | Oracle International Corporation | Greybox fuzzing for web applications |
| US20220004369A1 (en) * | 2020-07-01 | 2022-01-06 | Johnson Controls Tyco IP Holdings LLP | Rule builder and simulator tool for creating complex event processing rules |
| US11010191B1 (en) * | 2020-07-02 | 2021-05-18 | Ryan L. Hornbeck | Platform-independent interface for generating virtualized multi-service hardware systems and infrastructure |
| CN111930700A (en) * | 2020-07-13 | 2020-11-13 | 车智互联（北京）科技有限公司 | Distributed log processing method, server, system and computing equipment |
| CN112130976A (en) * | 2020-09-21 | 2020-12-25 | 厦门南讯股份有限公司 | REST-based multi-engine big data task management method |
| CN112286905A (en) * | 2020-10-15 | 2021-01-29 | 北京沃东天骏信息技术有限公司 | Data migration method and device, storage medium and electronic equipment |
| CN112328458A (en) * | 2020-11-27 | 2021-02-05 | 杭州安恒信息技术股份有限公司 | Data processing method and device based on flink data engine |
| US11275726B1 (en) * | 2020-12-06 | 2022-03-15 | Kamu Data Inc. | Distributed data processing method with complete provenance and reproducibility |
| CN112558995A (en) * | 2020-12-24 | 2021-03-26 | 恩亿科(北京)数据科技有限公司 | Flink integration method and system based on TBDS Hadoop |
| CN112765166A (en) * | 2021-01-06 | 2021-05-07 | 深圳市欢太科技有限公司 | Data processing method, device and computer readable storage medium |
| CN112835924A (en) * | 2021-02-04 | 2021-05-25 | 北京高途云集教育科技有限公司 | Real-time computing task processing method, device, equipment and storage medium |
| CN113010512A (en) * | 2021-02-24 | 2021-06-22 | 上海中通吉网络技术有限公司 | Real-time data processing method, platform and equipment based on Flink |
| CN113656157A (en) * | 2021-08-10 | 2021-11-16 | 北京锐安科技有限公司 | Distributed task scheduling method and device, storage medium and electronic equipment |
Non-Patent Citations (22)
| Title |
|---|
| ALESSIO PAGLIARI 等: "NAMB: A Quick and Flexible Stream Processing Application Prototype Generator", 2020 20TH IEEE/ACM INTERNATIONAL SYMPOSIUM ON CLUSTER, CLOUD AND INTERNET COMPUTING (CCGRID), pages 61 - 70 * |
| G. PAUL DAVIDSON等: "Technical Review Of Apache Flink For Big Data", INTERNATIONAL JOURNAL OF AQUATIC SCIENCE, vol. 12, no. 2, pages 3340 - 3346 * |
| GISELLE VAN DONGEN: "A Performance Analysis of Fault Recovery in Stream Processing Frameworks", IEEE ACCESS, vol. 9, 28 June 2021 (2021-06-28), pages 93745, XP011864503, DOI: 10.1109/ACCESS.2021.3093208 * |
| KARTIK KHARE: "Here‘s How Apache Flink Stores Your State data", Retrieved from the Internet <URL:https://towardsdatascience.com/heres-how-flink-stores-your-state-7b37fbb60e1a> * |
| M. HASEEB JAVED: "Characterization of Big Data Stream Processing Pipeline: A Case Study using Flink and Kafka", THE FOURTH IEEE/ACM INTERNATIONAL CONFERENCE ON BIG DATA COMPUTING, pages 1 - 10 * |
| MICHAEL ARMBRUST 等: "Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark", SIGMOD \'18： PROCEEDINGS OF THE 2018 INTERNATIONAL CONFERENCE ON MANAGEMENT OF DATA, 18 May 2018 (2018-05-18), pages 601 - 013 * |
| OSCAR CEBALLOS: "SPARQL2Flink: Evaluation of SPARQL Queries on Apache Flink", APPLIED SCIENCE, vol. 11, no. 15, pages 1 - 24 * |
| PARIS CARBONE等: "State management in Apache Flink: consistent stateful distributed stream processing", PROCEEDINGS OF THE VLDB ENDOWMENT, vol. 10, no. 12, pages 1718 * |
| SONIA BERGAMASCHI等: "BigBench Workload Executed by using Apache Flink", PROCEDIA MANUFACTURING, vol. 11, 31 December 2017 (2017-12-31), pages 695 - 702 * |
| SONIA BERGAMASCHI等: "BigBench Workload Executed by using Apache Flink", PROCEDIA MANUFACTURING, vol. 11, pages 695 - 702 * |
| TILMANN RABL EMAIL LOGO: "Apache Flink in current research", INFORMATION TECHNOLOGY, vol. 58, no. 4, 24 June 2016 (2016-06-24), pages 157 - 165 * |
| 吉喆: "基于Elasticsearch的实时大数据统计分析平台的研究与设计", CNKI优秀硕士学位论文全文库, no. 1, 15 January 2020 (2020-01-15), pages 1 - 63 * |
| 周超: "面向云服务的日志处理系统关键技术研发", CNKI优秀硕士学位论文, no. 5, pages 1 - 100 * |
| 张译天等: "大数据流式计算框架Heron环境下的流分类任务调度策略", 计算机应用, vol. 39, no. 4, pages 1106 - 1116 * |
| 数据与智能: "配置 Flink 流式应用 (九)", pages 2, Retrieved from the Internet <URL:https://xie.infoq.cn/article/4363b42e639cb705ec8970147> * |
| 李程: "面向大数据云平台的资源管理系统", CNKI优秀硕士学位论文全文库, no. 5, 15 May 2019 (2019-05-15), pages 1 - 67 * |
| 杜小勇 等: "大数据与OLAP系统", 大数据, vol. 1, no. 1, 31 May 2015 (2015-05-31), pages 55 - 67 * |
| 胡程，叶枫: "一种高效的Flink与MongoDB连接中间件的研究与实现", 计算机工程与应用, no. 23, pages 64 - 69 * |
| 蒲勇霖等: "基于Storm平台的数据恢复节能策略", 计算机研究与发展, vol. 58, no. 3, pages 479 - 496 * |
| 谢缙: "基于Flink的电商实时计算平台的设计与实现", CNKI优秀硕士学位论文, no. 3, pages 1 - 104 * |
| 魏碧晴: "基于流计算Flink框架的资源调度方法研究", CNKI优秀硕士学位论文, no. 1, pages 1 - 63 * |
| 鲁亮: "大数据流式计算框架Storm的任务迁移策略", 计算机研究与发展, vol. 55, no. 1, 31 January 2018 (2018-01-31), pages 71 - 92 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115756586A (en) * | 2022-11-25 | 2023-03-07 | 中电金信软件有限公司 | Method and device for executing Flink job, computer equipment and storage medium |
| CN115756586B (en) * | 2022-11-25 | 2024-01-19 | 中电金信软件有限公司 | Method and device for executing Flink job, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114816583B (en) | 2024-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108055343B (en) | Data synchronization method and device for computer room | |
| CN108255620B (en) | Service logic processing method, device, service server and system | |
| CN110119306B (en) | Method, device and equipment for balancing automatic scheduling of jobs and storage medium | |
| CN110753084B (en) | Uplink data reading method, cache server and computer readable storage medium | |
| US11237761B2 (en) | Management of multiple physical function nonvolatile memory devices | |
| CN111694677A (en) | Message queue management method, device, terminal and computer-readable storage medium | |
| CN114816583A (en) | Flink-based data automatic processing method and device and electronic equipment | |
| CN116089040A (en) | Service flow scheduling method and device, electronic equipment and storage medium | |
| CN114490078A (en) | Dynamic capacity reduction and expansion method, device and equipment for micro-service | |
| CN110347546B (en) | Dynamic adjustment method, device, medium and electronic equipment for monitoring task | |
| CN111831408A (en) | Asynchronous task processing method and device, electronic equipment and medium | |
| CN113760522A (en) | Task processing method and device | |
| CN112817687A (en) | Data synchronization method and device | |
| CN114880194B (en) | Service abnormity monitoring method and device, electronic equipment and computer storage medium | |
| CN114090268B (en) | Container management method and container management system | |
| CN110019045B (en) | Log floor method and device | |
| CN114625502A (en) | Word-throwing task processing method and device, storage medium and electronic equipment | |
| CN110058866B (en) | Cluster component installation method and device | |
| WO2019169696A1 (en) | Platform client data backflow method, electronic apparatus, device, and storage medium | |
| CN110868477A (en) | Task scheduling method, device and system | |
| CN115039079A (en) | Managing provenance information for a data processing pipeline | |
| CN111694628A (en) | Page display method based on application program thread and related equipment | |
| CN111858498A (en) | Storage type conversion method, system, device and equipment | |
| CN113127221A (en) | Method, device, equipment and storage medium for limiting message consumption rate | |
| CN110673931A (en) | Distributed calculation method for document synthesis, document synthesis system and control device thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |