CN113031972A - Method for decoupling flink stream calculation dependent file and mirror image in cloud native scene - Google Patents
Method for decoupling flink stream calculation dependent file and mirror image in cloud native scene Download PDFInfo
- Publication number
- CN113031972A CN113031972A CN202011401301.7A CN202011401301A CN113031972A CN 113031972 A CN113031972 A CN 113031972A CN 202011401301 A CN202011401301 A CN 202011401301A CN 113031972 A CN113031972 A CN 113031972A
- Authority
- CN
- China
- Prior art keywords
- flink
- mirror image
- file
- decoupling
- file system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 230000001419 dependent effect Effects 0.000 title claims abstract description 30
- 241000289581 Macropus sp. Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
- G06F8/63—Image based installation; Cloning; Build to order
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/65—Updates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for decoupling a flink stream calculation dependent file and a mirror image in a cloud native scene, which comprises the following steps: 1) file system configuration and dependent file uploading; 2) constructing a task basic mirror image; 3) and realizing a flink-on-k8s plug-in and a flink custom mount. Compared with the prior art, the invention realizes the decoupling of the program dependent package and the docker mirror image, the mirror image can meet the requirements of the user after only being constructed once in the development iteration process, and the mirror image does not need to be updated by using the method no matter the user updates the task program package or updates the package on which the flink depends, thereby improving the efficiency of development iteration.
Description
Technical Field
The invention relates to a method for decoupling a stream computation dependent file and a mirror image based on a flash in a cloud native scene.
Background
At present, jar files and the like which are depended by flink on kubernets flow calculation tasks are implanted in advance when a docker mirror image is constructed, the mirror image needs to be reconstructed when the task dependent files are updated, and the burden of development iteration is increased.
Disclosure of Invention
In order to solve the problem that a flash dependency package and a user code package need to be rebuilt as mirror images when updated, the invention provides a method for decoupling a flash-based stream computation dependency file and a mirror image in a cloud native scene.
A method for decoupling a flink stream computation dependent file and a mirror image in a cloud native scene comprises the following steps:
1) file system configuration and dependent file uploading; adding a configuration target file system in an interface mode, selecting a created file system, and uploading a dependent file;
2) constructing a task basic mirror image; providing a basic environment for running the task;
3) and realizing a flink-on-k8s plug-in and a flink custom mount.
The step 1) comprises the following steps:
1.1) adding a configuration file system, clicking an adding component to select an adding target file system, and filling different configuration items according to different file systems;
1.2) clicking to store, and testing whether the added file system can be communicated;
1.3) selecting resource management of an interface, clicking uploading resources, and filling attribute information of a resource file, including a resource name, a resource type, a storage position and description.
The step 2) comprises the following steps:
2.1) constructing a mirror image based on openjdk:8u242-jdk, creating a flink user, and installing a necessary environment dependence package;
2.2) organizing a flink home directory structure, and creating directories conf, log, plugins and deps under the flink home; the lib and bin catalogues are mounted on a flink home in a mount mode;
2.3) set env FLINK _ CLASSPATH = $ FLINK _ CLASSPATH $ FLINK _ HOME/deps.
The step 3) comprises the following steps:
3.1) defining the format of creating volume and mount parameters;
3.2) two methods, parseVolumesWithPrefix and parseVolumeMountsWithPrefix, are implemented in KubernetesUtils class. The method has two entries respectively being prefix and flinkConfig, the value of prefix being kubernets. Or kubernets. Matching defined volumes in the flinkConfig through prefix by using parseVolumeWithPrefix, and matching defined mount in the flinkConfig through prefix by using parseVolumeMountWithPrefix;
3.3) calling parseVolumesWithPrefix in decotedElnteralResource method of FlinkMasterDeploymentDecocorator to add custom volumes for jobmanager pod, calling parseVolumeMountWithPrefix in createJobManagerContainer method to add custom moles;
3.4) call ParseVolumeWithPrefix in the decorateElnteralResource method of TaskManagerPodDecorar to add custom volumes for taskmanager pod, call ParseVolumeMountWithPrefix in the createTaskManagerContainer method to add custom moles;
3.5) the flink-on-k8s plug-in uploads the task dependent file from sftp to the target file system, organizes the custom volume parameter to add to the flinkConfig, and then submits the task to kubernets.
The invention has the beneficial effects that: compared with the prior art, the invention realizes the decoupling of the program dependent package and the docker mirror image, the mirror image can meet the requirements of the user after only being constructed once in the development iteration process, and the mirror image does not need to be updated by using the method no matter the user updates the task program package or updates the package on which the flink depends, thereby improving the efficiency of development iteration.
Drawings
A more complete understanding of the present invention, and the attendant advantages and features thereof, will be more readily understood by reference to the following detailed description when considered in conjunction with the accompanying drawings wherein:
FIG. 1 is a flow chart of step 1) in the present invention;
FIG. 2 is a flow chart of step 3) of the present invention;
FIG. 3 is a corresponding setup interface for adding a file system;
fig. 4 is a setting interface for uploading a dependent file.
Detailed Description
In order to make the contents of the present invention more comprehensible, the present invention is further described below with reference to the accompanying drawings. The invention is of course not limited to this particular embodiment, and general alternatives known to those skilled in the art are also covered by the scope of the invention.
Examples
The invention provides a method for decoupling a flink stream calculation dependent file and a mirror image in a cloud native scene, which comprises the following steps:
1) file system configuration and dependent file uploading; adding a configuration target file system in an interface mode, selecting a created file system, and uploading a dependent file;
2) constructing a task basic mirror image; providing a basic environment for running the task;
3) and realizing a flink-on-k8s plug-in and a flink custom mount.
As shown in fig. 1, the step 1) includes the following steps:
1.1) adding a configuration file system, clicking an adding component to select an adding target file system, and filling different configuration items according to different file systems;
1.2) clicking to store, and testing whether the added file system can be communicated;
1.3) selecting resource management of an interface, clicking uploading resources, and filling attribute information of a resource file, including a resource name, a resource type, a storage position and description.
The step 2) comprises the following steps:
2.1) constructing a mirror image based on openjdk:8u242-jdk, creating a flink user, and installing a necessary environment dependence package;
2.2) organizing a flink home directory structure, and creating directories conf, log, plugins and deps under the flink home; the lib and bin catalogues are mounted on a flink home in a mount mode;
2.3) set env FLINK _ CLASSPATH = $ FLINK _ CLASSPATH $ FLINK _ HOME/deps.
As shown in fig. 2, the step 3) includes the following steps:
3.1) defining the format of creating volume and mount parameters;
3.2) two methods, parseVolumesWithPrefix and parseVolumeMountsWithPrefix, are implemented in KubernetesUtils class. The method has two entries respectively being prefix and flinkConfig, the value of prefix being kubernets. Or kubernets. Matching defined volumes in the flinkConfig through prefix by using parseVolumeWithPrefix, and matching defined mount in the flinkConfig through prefix by using parseVolumeMountWithPrefix;
3.3) calling parseVolumesWithPrefix in decotedElnteralResource method of FlinkMasterDeploymentDecocorator to add custom volumes for jobmanager pod, calling parseVolumeMountWithPrefix in createJobManagerContainer method to add custom moles;
3.4) call ParseVolumeWithPrefix in the decorateElnteralResource method of TaskManagerPodDecorar to add custom volumes for taskmanager pod, call ParseVolumeMountWithPrefix in the createTaskManagerContainer method to add custom moles;
3.5) the flink-on-k8s plug-in uploads the task dependent file from sftp to the target file system, organizes the custom volume parameter to add to the flinkConfig, and then submits the task to kubernets.
Compared with the prior art, the invention realizes the decoupling of the program dependent package and the docker mirror image, the mirror image can meet the requirements of the user after only being constructed once in the development iteration process, and the mirror image does not need to be updated by using the method no matter the user updates the task program package or updates the package on which the flink depends, thereby improving the efficiency of development iteration.
The method is applied to a flow calculation development kit of a kangaroo cloud number stack. And adding a file system and uploading a dependent file through interface configuration. FIG. 3 is a corresponding setup interface for adding a file system; FIG. 4 is a setup interface for uploading a dependent file, which is first uploaded to sftp. When a stream computing task is created, an uploaded dependent file is selected, the dependent file in the sftp is uploaded to a configured target file system at a flink plug-in terminal, and then the dependent file mount in the target file system is uploaded to a flink pod.
Claims (4)
1. A method for decoupling a flink stream computation dependent file and a mirror image in a cloud native scene is characterized by comprising the following steps:
1) file system configuration and dependent file uploading; adding a configuration target file system in an interface mode, selecting a created file system, and uploading a dependent file;
2) constructing a task basic mirror image; providing a basic environment for running the task;
3) and realizing a flink-on-k8s plug-in and a flink custom mount.
2. The method for decoupling a flink stream computation dependent file from a mirror image in a cloud native scenario as claimed in claim 1, wherein said step 1) comprises the steps of:
1.1) adding a configuration file system, clicking an adding component to select an adding target file system, and filling different configuration items according to different file systems;
1.2) clicking to store, and testing whether the added file system can be communicated;
1.3) selecting resource management of an interface, clicking uploading resources, and filling attribute information of a resource file, including a resource name, a resource type, a storage position and description.
3. The method for decoupling a flink stream computation dependent file from a mirror image in a cloud native scenario as claimed in claim 2, wherein said step 2) comprises the steps of:
2.1) constructing a mirror image based on openjdk:8u242-jdk, creating a flink user, and installing a necessary environment dependence package;
2.2) organizing a flink home directory structure, and creating directories conf, log, plugins and deps under the flink home; the lib and bin catalogues are mounted on a flink home in a mount mode;
2.3) set env FLINK _ CLASSPATH $ FLINK _ HOME/deps.
4. The method for decoupling a flink stream computation dependent file from a mirror image in a cloud native scenario as claimed in claim 3, wherein said step 3) comprises the steps of:
3.1) defining the format of creating volume and mount parameters;
3.2) two methods of parseVolumesWithPrefix and parseVolumeMountWithPrefix are realized in KuberneteseUtils, the two references in each method are respectively prefix and flinkConfig, and the value of the prefix is kubernets. Or kubernets. Matching defined volumes in the flinkConfig through prefix by using parseVolumeWithPrefix, and matching defined mount in the flinkConfig through prefix by using parseVolumeMountWithPrefix;
3.3) calling parseVolumesWithPrefix in decotedElnteralResource method of FlinkMasterDeploymentDecocorator to add custom volumes for jobmanager pod, calling parseVolumeMountWithPrefix in createJobManagerContainer method to add custom moles;
3.4) call ParseVolumeWithPrefix in the decorateElnteralResource method of TaskManagerPodDecorar to add custom volumes for taskmanager pod, call ParseVolumeMountWithPrefix in the createTaskManagerContainer method to add custom moles;
3.5) the flink-on-k8s plug-in uploads the task dependent file from sftp to the target file system, organizes the custom volume parameter to add to the flinkConfig, and then submits the task to kubernets.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011401301.7A CN113031972A (en) | 2020-12-02 | 2020-12-02 | Method for decoupling flink stream calculation dependent file and mirror image in cloud native scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011401301.7A CN113031972A (en) | 2020-12-02 | 2020-12-02 | Method for decoupling flink stream calculation dependent file and mirror image in cloud native scene |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113031972A true CN113031972A (en) | 2021-06-25 |
Family
ID=76460461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011401301.7A Pending CN113031972A (en) | 2020-12-02 | 2020-12-02 | Method for decoupling flink stream calculation dependent file and mirror image in cloud native scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113031972A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048533A (en) * | 2023-04-03 | 2023-05-02 | 浙江数新网络有限公司 | Implementation method and system for achieving dependency isolation in Flink task running |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109358858A (en) * | 2018-09-19 | 2019-02-19 | 网易（杭州）网络有限公司 | Automatically dispose method, apparatus, medium and electronic equipment |
| CN109634657A (en) * | 2018-12-14 | 2019-04-16 | 武汉烽火信息集成技术有限公司 | A kind of micro services are packaged dispositions method and system |
| US20190130004A1 (en) * | 2017-10-27 | 2019-05-02 | Streamsimple, Inc. | Streaming Microservices for Stream Processing Applications |
| CN110032376A (en) * | 2018-12-13 | 2019-07-19 | 中国电子科技集团公司电子科学研究院 | Automated software dispositions method and device based on Docker |
| US20190273655A1 (en) * | 2018-03-01 | 2019-09-05 | Red Hat, Inc. | Configuration of computing devices via containers |
| US20190324786A1 (en) * | 2018-04-23 | 2019-10-24 | Hewlett Packard Enterprise Development Lp | Containerized application deployment |
| CN111026414A (en) * | 2019-12-12 | 2020-04-17 | 杭州安恒信息技术股份有限公司 | HDP platform deployment method based on kubernets |
| CN111399865A (en) * | 2020-04-21 | 2020-07-10 | 贵州新致普惠信息技术有限公司 | Method for automatically constructing target file based on container technology |
| CN111984373A (en) * | 2020-08-19 | 2020-11-24 | 上海翘腾科技有限公司 | Method and system for ensuring environment consistency in Kubernetes container environment |
-
2020
- 2020-12-02 CN CN202011401301.7A patent/CN113031972A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190130004A1 (en) * | 2017-10-27 | 2019-05-02 | Streamsimple, Inc. | Streaming Microservices for Stream Processing Applications |
| US20190273655A1 (en) * | 2018-03-01 | 2019-09-05 | Red Hat, Inc. | Configuration of computing devices via containers |
| US20190324786A1 (en) * | 2018-04-23 | 2019-10-24 | Hewlett Packard Enterprise Development Lp | Containerized application deployment |
| CN109358858A (en) * | 2018-09-19 | 2019-02-19 | 网易（杭州）网络有限公司 | Automatically dispose method, apparatus, medium and electronic equipment |
| CN110032376A (en) * | 2018-12-13 | 2019-07-19 | 中国电子科技集团公司电子科学研究院 | Automated software dispositions method and device based on Docker |
| CN109634657A (en) * | 2018-12-14 | 2019-04-16 | 武汉烽火信息集成技术有限公司 | A kind of micro services are packaged dispositions method and system |
| CN111026414A (en) * | 2019-12-12 | 2020-04-17 | 杭州安恒信息技术股份有限公司 | HDP platform deployment method based on kubernets |
| CN111399865A (en) * | 2020-04-21 | 2020-07-10 | 贵州新致普惠信息技术有限公司 | Method for automatically constructing target file based on container technology |
| CN111984373A (en) * | 2020-08-19 | 2020-11-24 | 上海翘腾科技有限公司 | Method and system for ensuring environment consistency in Kubernetes container environment |
Non-Patent Citations (1)
| Title |
|---|
| DSD2333: "企业部署实战-在k8s上搭建大数据计算引擎Flink", HTTPS://BLOG.CSDN.NET/M0_37868230/ARTICLE/DETAILS/109747046, 17 November 2020 (2020-11-17), pages 5 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116048533A (en) * | 2023-04-03 | 2023-05-02 | 浙江数新网络有限公司 | Implementation method and system for achieving dependency isolation in Flink task running |
| CN116048533B (en) * | 2023-04-03 | 2023-07-25 | 浙江数新网络有限公司 | Implementation method and system for achieving dependency isolation in Flink task running |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103036956B (en) | A kind of distributed configuration mass data filing system and implementation method | |
| CN109739815B (en) | File processing method, system, device, equipment and storage medium | |
| WO2015074427A1 (en) | Systems and methods for data migration | |
| US6912543B2 (en) | Object-oriented method and system for transferring a file system | |
| CN104331343A (en) | File backup method and system | |
| CN113672224A (en) | Method and device for generating small program page code and computer equipment | |
| CN113031972A (en) | Method for decoupling flink stream calculation dependent file and mirror image in cloud native scene | |
| Suen et al. | Efficient migration of virtual machines between public and private cloud | |
| CN112052011A (en) | Method and device for combining small programs, electronic equipment and medium | |
| CN105320577A (en) | Data backup and recovery method, system and device | |
| CN111176706A (en) | Terminal upgrading method and device, electronic equipment and storage medium | |
| CN111225032A (en) | Method, system, device and medium for separating application service and file service | |
| CN111984595B (en) | File processing method and device and storage medium | |
| CN116629015A (en) | Digital twin task processing method, device, equipment and storage medium | |
| CN112579877A (en) | Information source system control method, device, storage medium and equipment | |
| CN112464049B (en) | Method, device and equipment for downloading number detail list | |
| CN111857736B (en) | Cloud computing product generation method, device, equipment and storage medium | |
| CN115757287A (en) | Target file construction method and device and storage medium | |
| CN114327593A (en) | Method and system for generating aar by Unity packaging Android and electronic device | |
| CN110362546A (en) | Acquisition methods, device and the document conversion system of convert file | |
| CN111694596A (en) | Packing method and packing device for entry paths in webpack environment | |
| CN114826919B (en) | SDN-based load balancing software nanotube method, device, equipment and medium | |
| CN111198721A (en) | Application program running method and device | |
| CN113434598B (en) | Method, device and electronic device for realizing data double writing | |
| CN114490095B (en) | Request result determination method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 311121 room 102-1 / F, room 102-2 / F, building 6, Haizhi center, 2301 yuhangtang Road, Cangqian street, Yuhang District, Hangzhou, Zhejiang Province Applicant after: HANGZHOU DAISHU TECHNOLOGY Co.,Ltd. Address before: 310030 8F, building 2, Hangzhou Internet innovation and entrepreneurship Park, 176 Zixia street, Xihu District, Hangzhou City, Zhejiang Province Applicant before: HANGZHOU DAISHU TECHNOLOGY Co.,Ltd. |