CN115237435A - Method for deploying PyFlink task to horn cluster - Google Patents
Method for deploying PyFlink task to horn cluster Download PDFInfo
- Publication number
- CN115237435A CN115237435A CN202210951622.7A CN202210951622A CN115237435A CN 115237435 A CN115237435 A CN 115237435A CN 202210951622 A CN202210951622 A CN 202210951622A CN 115237435 A CN115237435 A CN 115237435A
- Authority
- CN
- China
- Prior art keywords
- pyflink
- task
- python
- cluster
- files
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 70
- 238000004590 computer program Methods 0.000 claims description 8
- 230000006835 compression Effects 0.000 claims description 5
- 238000007906 compression Methods 0.000 claims description 5
- 230000001419 dependent effect Effects 0.000 claims description 4
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 238000012546 transfer Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/76—Adapting program code to run in a different environment; Porting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/36—Software reuse
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
The application discloses a method for deploying PyFlink tasks to a horn cluster, which relates to the technical field of big data calculation and comprises the following steps: downloading all resource files of PyFlink tasks uploaded by a front end to a back end, and acquiring Python related information; constructing a PackagedProgramm parameter of the PyFlink task according to the resource file and the Python related information, and calling a deployJobCluster method of a yarnClusterDescriptor to upload all related files of the PyFlink task to the HDFS; starting a Python process, generating JobGraph according to the logic of the PyFlink task, and submitting the JobGraph to a yann cluster through a yannCluster Descriptor. According to the method, the uploaded resources and the dependencies are directly reused when the PyFlink task is submitted, and the related dependencies of the PyFlink task and the PyFlink environment do not need to be installed in the client in advance, so that the PyFlink task can be run in different PyFlink environments.
Description
The application relates to the technical field of big data computing, in particular to a method for deploying PyFlink tasks to a yann cluster.
Background
In the prior art, submission of a Flink task written by Python mainly depends on a command line mode, the mode requires that a user installs a Python environment and related dependencies of Python in advance at a client, and the Python environment dependency needs to be manually uploaded to a server every time, so that the Python environment dependency cannot be reused, the process is complicated, and resources such as Python program files, jar package dependencies and Python dependencies of Python cannot be effectively managed.
Disclosure of Invention
The method for deploying the PyFlink task to the horn cluster aims to effectively manage Python program files, jar package dependencies, python dependencies and other resources related to the PyFlink task.
In order to achieve the purpose, the following technical scheme is adopted in the application:
the method for deploying the PyFlink task to the horn cluster comprises the following steps:
downloading all resource files of PyFlink tasks uploaded by a front end to a back end, and acquiring Python related information;
constructing a PackagedProgramm parameter of the PyFlink task according to the resource file and the Python related information, and calling a deployJobCluster method of a yarnClusterDescriptor to upload all related files of the PyFlink task to the HDFS;
starting a Python process, generating a JobGraph according to the logic of the PyFlink task, and submitting the JobGraph to a yann cluster through the yannClusterDescriptor.
Preferably, the downloading all resource files of the PyFlink task uploaded by the front end to the back end includes:
receiving a resource file of a PyFlink task uploaded by a front end, wherein the resource file comprises a Python file, a PyFlink environment compression package and a third party dependent jar package;
and storing the resource files into different storage media according to resource types, wherein the storage media comprise HDFS and SFTP, and downloading all the resource files to a back end.
Preferably, after downloading all resource files of the PyFlink task uploaded by the front end to the back end, the method further includes:
and decompressing the PyFlink environment compression packet in the resource file, and packaging the decompressed directory path into PyFlinkInfo.
Preferably, the obtaining Python related information includes:
and searching a Flink-Python. Jar package under a back-end Flink Lib directory, setting a jar package path into PyFlinkInfo, and acquiring path information of a PyFlink environment downloaded to the back end, path information of a PyFlink environment running on a yann cluster and path information of resource files stored in the HDFS and the SFTP.
Preferably, the PackagedProgram parameter includes a Python file, a Python join, a path of a backend PyFlink environment, and a PyFlink environment path of a yarm cluster, where the Python join belongs to the resource file.
Preferably, the related files include all packages in the flink lib, jar packages depended on by PyFlink tasks, log configuration files, hdfs configuration files, and yarn configuration files.
Preferably, the starting of the Python process, the generation of the JobGraph according to the logic of the PyFlink task, and the submission of the JobGraph to the yann cluster through the yanncrusterdescriptor include:
calling a Flink Pythondriver to start a Python process, wherein the Python process is used for communicating with a Java JVM process;
and generating JobGraph according to the logic of the PyFlink task and the Java JVM process, and submitting the JobGraph to a yann cluster through the yannClusterDescriptor.
Preferably, the method further comprises:
and after the PyFlink task is submitted, recursively deleting all files downloaded in the task submitting process.
An electronic device comprising a memory and a processor, the memory for storing one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method of deploying PyFlink tasks to a yann cluster as in any of the above.
A computer readable storage medium storing a computer program which when executed by a computer implements a method of deploying PyFlink tasks to a yann cluster as claimed in any one of the above.
The invention has the following beneficial effects:
compared with the Python ecological integration method based on the Flink computing framework in the prior art, the Python ecological threshold using the Python ecological integration method not only is reduced, but also the user can directly edit the Python task on the platform, the uploaded resources and the dependence can be directly multiplexed when submitting the Python task, the Python environment and the dependence of the Python task do not need to be installed at the client in advance, the Python task can be operated in different Python environments, and various resources of the Python task can be effectively managed.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a flowchart of a method for deploying PyFlink tasks to a horn cluster according to the present application;
FIG. 2 is an exemplary diagram of PyFlink task resources provided at the front end of the present application;
FIG. 3 is an exemplary diagram of a front end user configuration relative path in accordance with the subject application.
Detailed Description
The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the accompanying drawings, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms "first," "second," and the like in the claims and in the description of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order, it being understood that the terms so used are interchangeable under appropriate circumstances and are merely used to describe a distinguishing manner between similar elements in the embodiments of the present application and that the terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive inclusion such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Examples
As shown in fig. 1, a method for deploying PyFlink tasks to a yarn cluster includes the following steps:
s110, downloading all resource files of PyFlink tasks uploaded by a front end to a back end, and acquiring Python related information;
s120, constructing a PackagedProgramm parameter of the PyFlink task according to the resource file and the Python related information, and calling a deployJobCluster method of a yarnClusterDescriptor to upload all related files of the PyFlink task to the HDFS;
s130, starting a Python process, generating JobGraph according to the logic of the PyFlink task, and submitting the JobGraph to a yann cluster through the yarnClusterDescriptor.
The embodiment relates to interaction between a front end and a back end, where the front end can be simply understood as a client, and is mainly used for providing a resource file of a PyFlink task and transmitting the resource file to the back end, i.e., a server, for management, and related programming languages are HTML and JavaScript, as shown in fig. 2, the main contents of the related programming languages include:
[001] task name and task type
The task name is the task name of the current PyFlink task, such as Flink1, so that the subsequent task operation and maintenance can be conveniently distinguished from the task, and the task type in the PyFlink task is fixed as PyFlink.
[002] Python file
The Python file is mainly logic of the current Pythink task, can be a Flink streaming task or a batch task, and the programming language is Python language, such as kafka _ stream. The Python file needs to be uploaded to project resources of the system in advance and finally stored in the SFTP.
[003] PyFlink environment
The PyFlink environment is primarily a Python environment that needs to be installed in Flink's Pythonomodule. Most developers use Windows or MacOS, but the backend server uses Linux, so that a Python environment capable of running Flunk is packaged in Linux by Docker, such as Linux _ venv _ final.
[004] PyFlink ginseng
Entries required by the PyFlink task, such as "a", "b".
[005] Third party dependencies
Because the Flink framework is written by Java, the Flink API of Python needs to communicate with the Flink JVM of Java, namely the actual function is realized by the Fink framework written by Java, and some functions of Flink are realized by the Connector plug-in, so that some Flink-dependent Jar packages need to be provided by users, for example, when the PyFlink task needs to read Kafka, the Flink-sql-Connector-Kafka _2.12-1.12.7.Jar needs to be used, and the users need to upload the Jar packages to the front-end page.
Further, receiving a resource file of a PyFlink task uploaded by a front end, wherein the resource file comprises a Python file, a PyFlink environment compression package and a third party dependent jar package;
and storing the resource files into different storage media according to resource types, wherein the storage media comprise HDFS and SFTP, and downloading all the resource files to a back end.
Illustratively, the backend receives all PyFlink task resources transmitted from the front end and stores them into different storage media according to specific contents: pyFlink environment files are large and stored in an HDFS (Hadoop Distributed File system), and other resource files such as Python files and third-party dependence files are small files relative to the PyFlink environment files and are stored in an SFTP (small File Transfer Protocol), wherein the HDFS is a Hadoop Distributed File system (Hadoop Distributed File system), stores oversized files in a streaming data access mode, is a File system stored across a plurality of computers in a management network and has high fault tolerance and high throughput, the SFTP is a safe File Transfer Protocol, is totally called SSH File Transfer Protocol, has the functions of File access, transmission and management, and can provide a safe network encryption method for transmitting files.
In addition, resources and dependence uploaded by the front end are stored in the storage medium, so that PyFlink task files required to be used when tasks are submitted can be directly reused with the uploaded files without re-uploading, and the method is simpler and more convenient.
When a task is submitted, all resources required by the current PyFlink task are to be submitted to the scheduling system for task priority ordering and calling of the PyFlink task submitting component, which is the prior art and is not described herein again.
Before the task is submitted, downloading the resource files transmitted by the front end to the local, decompressing PyFlink environment files, and packaging the decompressed directory paths into PyFlinkInfo to prepare for submitting the task.
And further searching a Flink-Python. Jar package under a back-end Flink Lib directory, setting a jar package path into PyFlinkInfo, and acquiring path information of a PyFlink environment downloaded to the back end, path information of a PyFlink environment running on a yann cluster and path information of resource files stored in the HDFS and the SFTP.
Then, acquiring related information of Python, wherein the related information mainly comprises three parts: finding a Flink-Python. Jar compressed packet which is directly uploaded to a rear end, wherein the jar packet contains a Flink Python module and needs to be submitted to a yard cluster, and the Flink Lib directory is placed in the packet at the rear end, wherein the Flink Lib directory is transferred to the rear end after the front end is configured by a user, and after finding, setting the path of the jar packet into PyFlinkInfo and directly taking out the PyFlinkInfo for use when submitting a task; the second part is a PyFlink environment path, which comprises python. Client. Executable and python. Executable, wherein the python. Client. Executable is a PyFlink environment path required by the back end, and is obtained by splicing two paths, namely a PyFlink environment position decompressed from a PyFlink environment file downloaded from the HDFS, and the second part is a relative path which is configured by a user and then transmitted to the back end, namely, the python. Client. Executable = PyFlink environment + user-configured relative path, the user-configured relative path is shown in fig. 3, and the python. Executable is a path for filing the PyFlink environment file on the HDFS; the third part is the path information of the resource file transmitted by the front end and also divided into two parts, namely the path information of PyFlink environment from HDFS, and the path information of other resource files from SFTP.
Further, calling a Flink Pythondriver to start a Python process, wherein the Python process is used for communicating with a Java JVM process;
and generating JobGraph according to the logic of the PyFlink task and the Java JVM process, and submitting the JobGraph to a yann cluster through the yannClusterDescriptor.
Illustratively, when a task is submitted, firstly constructing PackagedProgram parameter information of a PyFlink task, including a Python file, a PyFlink entry, a PyFlink environment path at the back end and a PyFlink environment path of a yard cluster, then calling a deployJobBcursor method of a yarnCluster Descriptor, uploading all resources of the current PyFlink task, namely all packets in a Flink lib, jar packets, log configuration files, HDFS configuration files and a yard configuration file depended by the PyFlink task to an HDFS, calling a Flink PyFlink driver class to start a Python process, communicating the process with a Flink JVM process, generating JobGraph according to logic of the PyFlink task and Java JVM communication, and finally submitting the JobGraph to the yard cluster through the YamFlink Descriptor.
After the task is submitted, all the downloaded folders of the resource of the task are deleted recursively, and the occupation of the disk space of the server by the garbage resource is avoided.
The method for compiling the Flink real-time computing task in the Python mode is based on a Flink computing framework, and is used for better managing Python files, jar package dependencies, python environments and other resources which are depended by the Flink task in the Python compiling process.
The present application also provides an electronic device comprising a memory and a processor, wherein the memory is used for storing one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-mentioned method for deploying PyFlink tasks to a yann cluster.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the electronic device described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.
The present application also provides a computer readable storage medium storing a computer program, which when executed by a computer, implements a method for deploying PyFlink tasks to a yann cluster as described above.
Illustratively, a computer program may be divided into one or more modules/units, one or more modules/units being stored in a memory and executed by a processor and performing I/O interface transfer of data by an input interface and an output interface to perform the present invention, and one or more modules/units may be a series of computer program instruction segments describing the execution of the computer program in a computer device.
The computer device may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The computer device may include, but is not limited to, a memory and a processor, and those skilled in the art will appreciate that the present embodiment is only an example of the computer device and does not constitute a limitation of the computer device, and may include more or less components, or combine certain components, or different components, for example, the computer device may further include an input device, a network access device, a bus, and the like.
The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The storage may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The memory may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc. provided on the computer device, and further, the memory may also include both an internal storage unit and an external storage device of the computer device, the memory is used for storing computer programs and other programs and data required by the computer device, and the memory may also be used for temporarily storing in the output device, and the aforementioned storage medium includes various Media capable of storing program codes, such as a usb disk, a removable hard disk, a read only memory ROM, a random access memory RAM, a disk, or an optical disk.
The above description is only an embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions within the technical scope of the present invention are intended to be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A method for deploying PyFlink tasks to a yann cluster, comprising the steps of:
downloading all resource files of PyFlink tasks uploaded by a front end to a back end, and acquiring Python related information;
constructing a PackagedProgramm parameter of the PyFlink task according to the resource file and the Python related information, and calling a deployJobCluster method of a yarnClusterDescriptor to upload all related files of the PyFlink task to the HDFS;
starting a Python process, generating a JobGraph according to the logic of the PyFlink task, and submitting the JobGraph to a yann cluster through the yannClusterDescriptor.
2. The method according to claim 1, wherein the downloading all resource files of the PyFlink task uploaded from the front end to the back end comprises:
receiving a resource file of a PyFlink task uploaded by a front end, wherein the resource file comprises a Python file, a PyFlink environment compression package and a third party dependent jar package;
and storing the resource files into different storage media according to resource types, wherein the storage media comprise HDFS and SFTP, and downloading all the resource files to a back end.
3. The method according to claim 1, wherein after downloading all resource files of the PyFlink task uploaded from the front end to the back end, the method further comprises:
and decompressing the PyFlink environment compression packet in the resource file, and packaging the decompressed directory path into PyFlinkInfo.
4. The method according to claim 2, wherein the obtaining PyFlink tasks comprises:
searching a Flink-Python. Jar package under a back-end Flink Lib directory, setting a jar package path to PyFlinkInfo, and acquiring path information of a PyFlink environment downloaded to the back end, path information of the PyFlink environment running on a yann cluster and path information of resource files stored in an HDFS and an SFTP.
5. The method of claim 1, wherein the PackagedProgram parameters include a Python file, a Python join parameter, a path of a backend PyFlink environment, and a PyFlink environment path of a yanm cluster, the Python join parameter being attributed to the resource file.
6. The method of claim 1, wherein the related files comprise all packages in a flinklib, jar packages on which the PyFlink task depends, a log configuration file, a hdfs configuration file, and a yarn configuration file.
7. The method for deploying PyFlink task to a yann cluster as claimed in claim 1, wherein the starting Python process and generating JobGraph according to the PyFlink task logic, submitting the JobGraph to the yann cluster through the YarnCluster Descriptor comprises:
calling a Flink Pythondriver to start a Python process, wherein the Python process is used for communicating with a Java JVM process;
and generating JobGraph according to the logic of the PyFlink task and the Java JVM process, and submitting the JobGraph to a yann cluster through the yannClusterDescriptor.
8. The method of claim 1, wherein the PyFlink task is deployed to a horn cluster, and wherein the Python process is initiated, the method further comprising:
and after the PyFlink task is submitted, recursively deleting all files downloaded in the task submitting process.
9. An electronic device, comprising a memory and a processor, wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executable by the processor to implement a method for deploying a PyFlink task to a yarn cluster as recited in any one of claims 1 to 8.
10. A computer readable storage medium storing a computer program, wherein the computer program when executed causes a computer to implement the method for deploying a PyFlink task to a yann cluster according to any one of claims 1 to 8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310027726.3A CN116382773A (en) | 2022-08-09 | 2022-08-09 | Method for deploying PyFlink task |
| CN202210951622.7A CN115237435B (en) | 2022-08-09 | 2022-08-09 | Method for deploying PyFlink task to horn cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210951622.7A CN115237435B (en) | 2022-08-09 | 2022-08-09 | Method for deploying PyFlink task to horn cluster |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310027726.3A Division CN116382773A (en) | 2022-08-09 | 2022-08-09 | Method for deploying PyFlink task |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115237435A true CN115237435A (en) | 2022-10-25 |
| CN115237435B CN115237435B (en) | 2023-02-14 |
Family
ID=83679681
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210951622.7A Active CN115237435B (en) | 2022-08-09 | 2022-08-09 | Method for deploying PyFlink task to horn cluster |
| CN202310027726.3A Pending CN116382773A (en) | 2022-08-09 | 2022-08-09 | Method for deploying PyFlink task |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310027726.3A Pending CN116382773A (en) | 2022-08-09 | 2022-08-09 | Method for deploying PyFlink task |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN115237435B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048533A (en) * | 2023-04-03 | 2023-05-02 | 浙江数新网络有限公司 | Implementation method and system for achieving dependency isolation in Flink task running |
| CN116841649A (en) * | 2023-08-28 | 2023-10-03 | 杭州玳数科技有限公司 | Method and device for hot restarting based on flink on horn |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180074852A1 (en) * | 2016-09-14 | 2018-03-15 | Salesforce.Com, Inc. | Compact Task Deployment for Stream Processing Systems |
| CN112882728A (en) * | 2021-03-25 | 2021-06-01 | 浪潮云信息技术股份公司 | Deployment method of big data platform real-time computing service Flink based on Yarn |
| CN113553098A (en) * | 2021-07-27 | 2021-10-26 | 未鲲(上海)科技服务有限公司 | Method and device for submitting Flink SQL (structured query language) operation and computer equipment |
| CN113642021A (en) * | 2021-08-20 | 2021-11-12 | 深信服科技股份有限公司 | Business code submitting method, processing method, device and electronic equipment |
| CN113961570A (en) * | 2021-12-22 | 2022-01-21 | 四川新网银行股份有限公司 | Real-time acquisition method applied to MYSQL BINLog change data |
| CN114489833A (en) * | 2021-12-31 | 2022-05-13 | 武汉达梦数据库股份有限公司 | Implementation method and device for submitting flash job to yarn cluster in application program |
-
2022
- 2022-08-09 CN CN202210951622.7A patent/CN115237435B/en active Active
- 2022-08-09 CN CN202310027726.3A patent/CN116382773A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180074852A1 (en) * | 2016-09-14 | 2018-03-15 | Salesforce.Com, Inc. | Compact Task Deployment for Stream Processing Systems |
| CN112882728A (en) * | 2021-03-25 | 2021-06-01 | 浪潮云信息技术股份公司 | Deployment method of big data platform real-time computing service Flink based on Yarn |
| CN113553098A (en) * | 2021-07-27 | 2021-10-26 | 未鲲(上海)科技服务有限公司 | Method and device for submitting Flink SQL (structured query language) operation and computer equipment |
| CN113642021A (en) * | 2021-08-20 | 2021-11-12 | 深信服科技股份有限公司 | Business code submitting method, processing method, device and electronic equipment |
| CN113961570A (en) * | 2021-12-22 | 2022-01-21 | 四川新网银行股份有限公司 | Real-time acquisition method applied to MYSQL BINLog change data |
| CN114489833A (en) * | 2021-12-31 | 2022-05-13 | 武汉达梦数据库股份有限公司 | Implementation method and device for submitting flash job to yarn cluster in application program |
Non-Patent Citations (1)
| Title |
|---|
| 蔡鲲鹏等: "基于Flink on YARN平台的应用研究", 《科技创新与应用》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048533A (en) * | 2023-04-03 | 2023-05-02 | 浙江数新网络有限公司 | Implementation method and system for achieving dependency isolation in Flink task running |
| CN116048533B (en) * | 2023-04-03 | 2023-07-25 | 浙江数新网络有限公司 | Implementation method and system for achieving dependency isolation in Flink task running |
| CN116841649A (en) * | 2023-08-28 | 2023-10-03 | 杭州玳数科技有限公司 | Method and device for hot restarting based on flink on horn |
| CN116841649B (en) * | 2023-08-28 | 2023-12-08 | 杭州玳数科技有限公司 | Method and device for hot restarting based on flink on horn |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116382773A (en) | 2023-07-04 |
| CN115237435B (en) | 2023-02-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115237435B (en) | Method for deploying PyFlink task to horn cluster | |
| CN108037961B (en) | Application program configuration method, device, server and storage medium | |
| US10114682B2 (en) | Method and system for operating a data center by reducing an amount of data to be processed | |
| US10185558B2 (en) | Language-independent program composition using containers | |
| CN106778351B (en) | Data desensitization method and device | |
| CN111177113B (en) | Data migration method, device, computer equipment and storage medium | |
| US20170185503A1 (en) | Method and system for recommending application parameter setting and system specification setting in distributed computation | |
| CN111858727A (en) | Multi-data-source data export system and method based on template configuration | |
| CN113687858A (en) | Configuration file checking method and device, electronic equipment and storage medium | |
| CN111459497A (en) | WebPack-based resource package compiling method, system, server and storage medium | |
| CN114461955A (en) | Method for automatically generating http interface based on web page configuration | |
| CN109725887B (en) | Data interaction method and device based on message research and development framework and terminal equipment | |
| CN109343970B (en) | Application program-based operation method and device, electronic equipment and computer medium | |
| CN113296987B (en) | Interface calling method and device for calling module, computer equipment and storage medium | |
| US9537931B2 (en) | Dynamic object oriented remote instantiation | |
| CN115268909A (en) | Method, system and terminal for establishing and running construction task at web front end | |
| US11861386B1 (en) | Application gateways in an on-demand network code execution system | |
| CN114189745A (en) | Set top box software switching management method and system and computer device | |
| CN113779122A (en) | Method and apparatus for exporting data | |
| CN116893860B (en) | Method for isolating data sources with different versions in Chunjun synchronous task | |
| CN114765606B (en) | Container mirror image transmission method, device, equipment and storage medium | |
| CN113505036B (en) | Application monitoring method, client and server | |
| US20120030273A1 (en) | Saving multiple data items using partial-order planning | |
| CN113076128B (en) | Method, device, electronic equipment and storage medium for robot configuration | |
| CN110392105B (en) | File transmission method, device and system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |