CN114489833B - Implementation method and device for submitting flash job to yarn cluster in application program - Google Patents
Implementation method and device for submitting flash job to yarn cluster in application program Download PDFInfo
- Publication number
- CN114489833B CN114489833B CN202111674276.4A CN202111674276A CN114489833B CN 114489833 B CN114489833 B CN 114489833B CN 202111674276 A CN202111674276 A CN 202111674276A CN 114489833 B CN114489833 B CN 114489833B
- Authority
- CN
- China
- Prior art keywords
- cluster
- flink
- path
- hadoop
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of big data calculation, and discloses a method and a device for realizing submitting a flight job to a yann cluster in an application program, which solve the problem of deployment of a Hadoop configuration file and a third-party dependent jar package.
Description
[ technical field ] A method for producing a semiconductor device
The invention relates to the technical field of big data calculation, in particular to a method and a device for submitting a flash job to a yann cluster in an application program.
[ background of the invention ]
Flink is a high performance, high throughput, low latency stream processing framework. It is not just a streaming framework, but rather unifies batch processing (in Flink, batch processing is a special case of streaming). The Flink architecture also better solves the problem of tedious component accumulation of the traditional big data architecture, and batch flow can carry out batch processing or flow processing on the basis of not changing the original code, so that the Flink architecture becomes an increasingly popular big data processing framework.
Flink supports multiple deployment modes, such as Local, standalone, yarn, K8s and the like, and most enterprises generally select a deployment mode of flex-on-Yarn for convenience of management because Yarn is used as a resource manager by a big data platform.
flink runs on yarn and requires a ResourceManager connected to HDFS (Hadoop Distributed File System) File System and yarn, which must be able to find the Hadoop configuration File. The method for searching the Hadoop configuration file by the flink native code is to sequentially check whether environment variables such as YARN _ CONF _ DIR, HADOOP _ CONF _ DIR and HADOOP _ CONF _ PATH are set, and if so, reading configuration from a PATH pointed by the environment variables; otherwise, looking up the HADOOP _ HOME environment variable, and for the HADOOP 2.x, the searched configuration path is $ HADOOP _ HOME/etc/HADOOP; for HADOOP 1.x, the configuration path looked up is $ HADOOP _ HOME/conf. If none of the above environment variables is set, then the flink cannot be connected to the HDFS and the ResourceManager, and the flink cluster cannot be deployed on yarn, and cannot be submitted to run on the flink-on-yarn, and cannot be submitted to run on yarn. However, in a specific implementation environment, a user deploying an application program does not necessarily have the right to set the environment variables; even if the environment variable can be set, the configuration file in the environment variable belongs to public resources, and different application programs cannot be self-defined according to self requirements;
after finding the Hadoop configuration file, the flink will create a YarnClient object and a FileSysteme object. When a new flash's horn-session is created, the client will first check whether the resources (contacts and memory) to be requested are available, and then upload the jar packet and configuration related to the flash to hdfs, and submitting the flash job will occupy a large amount of network IO for uploading the jar packet and will cause the common resources to occupy space repeatedly;
in view of the above, overcoming the drawbacks of the prior art is an urgent problem in the art.
[ summary of the invention ]
The invention aims to provide an implementation method for submitting a flex job to a yarn cluster in an application program.
The invention further aims to solve the technical problem of providing an implementation device for submitting the flight job to the yann cluster in the application program.
The invention adopts the following technical scheme:
in a first aspect, a method for implementing submission of a flash job to a yann cluster in an application program includes:
the method comprises the steps that a Flink cluster is deployed on a yann cluster, a resident yannclient is constructed according to a Hadoop configuration file defined by an application program, and the Flink cluster interacts with the yann cluster through the yannclient;
when the permission is not available, directly uploading the flink system jar package and the third party dependent jar package to the HDFS file system, deploying the flink system jar package and the third party dependent jar package to local paths of all nodes of the Hadoop cluster, and when the flink cluster is deployed on the yann cluster, loading the flink system jar package and the third party dependent jar package in the local paths of all the nodes of the Hadoop cluster.
Preferably, the constructing a resident yarnclean in the path of the Hadoop configuration file customized according to the application program specifically includes:
the method comprises the steps of placing a Hadoop configuration file in a Hadoop configuration file path defined by an application program, constructing a resident YannClient according to the Hadoop configuration file, and enabling the flash cluster to interact with the yannCluster through the YannClient.
Preferably, a resident FileSystem is constructed according to the Hadoop cluster, and the flink cluster interacts with the HDFS through the FileSystem.
Preferably, the deploying one or more of the jar package of the flink system and the third-party dependent jar package to the local paths of all the nodes of the Hadoop cluster specifically comprises:
when one or more of a flink system jar package and a third party dependent jar package are directly uploaded to an HDFS file system without permission, original paths of the flink system jar package and the third party dependent jar package are traversed, one or more of the flink system jar package and the third party dependent jar package are deployed in local paths of all nodes of a Hadoop cluster, a full path name of the jar package is built under the condition that a file structure of the original path of the jar package is consistent with a file structure of the local paths of all nodes of the Hadoop cluster, a parent path is replaced by the local paths of all nodes of the Hadoop cluster, and the built full path name of the jar package is added into a classpath.
Preferably, the adding the constructed jar package full path name into the classpath specifically includes:
setting the CLASSPATH into a parameter value corresponding to YannConfigKeys, ENV _ FLINK _ CLASSPATH in an environment variable of the AM, starting the ApplicationMaster when a FLINK cluster is deployed on the yann cluster, and loading one or more of a FLINK system jar packet and a third party dependent jar packet from local paths of all nodes of the Hadoop cluster.
Preferably, when one or more of the flink system jar package and the third party dependent jar package are directly uploaded to the HDFS, the flink system jar package and the third party dependent jar package are uploaded to a first path of the HDFS before the flink cluster is deployed on the yann cluster.
Preferably, the yarn, provided, lib, dirs parameter is set to point to the first path of the HDFS file system, and when the flink cluster is deployed on the yarn cluster, the flink system jar package and the third party dependent jar package are loaded from the first path of the HDFS file system.
Preferably, after loading one or more of the flink system jar package and the third party dependent jar package in the full path homonymous path of all the nodes of the HDFS, the method further specifically includes:
the application client applies for a yann container to the ResourceMenager to start the applicationMaster, the application client registers the flink configuration file and the jar packet as the resource of the yann container, the nondemenager prepares the yann container by using the resource of the yann container, and the applicationMaster is started.
Preferably, the JobManager and the ApplicationMaster run on the same yann container, when the ApplicationMaster knows the address of the JobManager, the ApplicationMaster generates a new Flink configuration file, the Flink configuration file is used by the TaskManager, the ApplicationMaster starts the containers of the Flink TaskManager, the binders downloads the jar file and the Flink configuration file from the HDFS, and the Flink completes the deployment on yann and starts receiving tasks.
In a second aspect, an apparatus for implementing a method for submitting a flight job to a yann cluster in an application program includes at least one processor, and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the method of implementing the submission of a flight job to a yann cluster in an application.
The invention relates to the technical field of databases, and discloses a method for directly submitting a flight job to a yann cluster in an application program, which solves the problem that a Hadoop configuration file and a third party depend on deployment of a jar package. According to the method, by rewriting functions in part of flink system classes, the custom deployment of the Hadoop configuration file and the third-party dependence jar package is realized, the deployment flexibility, convenience and operation efficiency of the application programs are improved, the dependence degree on the environment is reduced, and possible mutual interference among different application programs is avoided.
[ description of the drawings ]
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 is a schematic flowchart of an implementation method for submitting a flash job to a yarn cluster in an application according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating an implementation method for submitting a flight job to a yarn cluster in an application according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating an implementation method for submitting a flink job to a yann cluster in an application according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating an implementation method for submitting a flight job to a yarn cluster in an application according to an embodiment of the present invention;
fig. 5 is a schematic diagram of an apparatus for implementing apparatus for submitting a flink job to a yann cluster in an application according to an embodiment of the present invention.
[ detailed description ] A
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
Example 1:
the embodiment 1 of the invention provides a method for submitting a flash job to a yarn cluster in an application program.
The method comprises the steps that a flink runs on the yann, a Hadoop configuration file needs to be found when the flink needs to be connected to an HDFS file system and a resource manager of the yann, when a user does not have permission to set environment variables so as to find the Hadoop configuration file, an application program self-defines a Hadoop configuration file path, the Hadoop configuration file is stored in the Hadoop configuration file path, a resident yannClient is constructed through the Hadoop configuration file, and the flink cluster interacts with the yann cluster through the yannClient;
the Hadoop configuration file path is an application program working directory, or other paths which can be found are defined through the application program working directory, and the Hadoop configuration file is stored in the path, and the path is designed by the person skilled in the art, and the protection scope of the invention is not limited.
When the flash system jar package and the third party dependent jar package are not directly uploaded to the HDFS, the flash system jar package and the third party dependent jar package are deployed in local paths of all nodes of the Hadoop cluster, and when the flash cluster is deployed on the yann cluster, the flash system jar package and the third party dependent jar package are loaded in the local paths of all nodes of the Hadoop cluster.
When deploying the flink cluster to the yarn cluster, one or more of a jar packet of the flink system and a third party dependent jar packet need to be uploaded to an HDFS system file, but when frequently deploying the flink cluster to the yarn cluster, a large number of network IOs are occupied, so that the jar packet is deployed to a full-path homonymous path of all nodes of the Hadoop, and therefore when the jar packet needs to be uploaded to the HDFS system file, uploading from a local file is not needed, the jar packet is already in the Hadoop node, and a path of the Hadoop node storing the jar packet is directly loaded.
flink runs on yarn, needs to be connected to the HDFS file system and yarn's ResourceMenager, and needs to build yarn client to make flink interact with yarn's ResourceMenager; a FileSystem is required to be constructed to enable the flink to interact with the HDFS file system; in order to construct the yarnclean and the FileSysteme, a Hadoop configuration file needs to be found;
as shown in fig. 1, the process is as follows:
in step 101, searching a Hadoop configuration file;
in step 102, whether the user has the authority to set the environment variable or not is judged, if yes, the step 102 is skipped, and if not, the step 103 is skipped;
in step 102, a user has the right to set environment variables, wherein the method for finding the Hadoop configuration file by the flink is to check whether to set environment variables such as YARN _ CONF _ DIR, HADOOP _ CONF _ PATH and the like, and when the environment variables such as YARN _ CONF _ DIR, HADOOP _ CONF _ PATH and the like are set, the Hadoop configuration is directly read from a PATH pointed by the environment variables, and the YarnClient and a FileSystems are constructed; when the environment variable is not set, checking the HADOOP _ HOME environment variable, and for the HADOOP 2.x, searching the configuration path to be $ HADOOP _ HOME/etc/HADOOP; for HADOOP 1.x, the searched configuration path is $ HADOOP _ HOME/conf; searching Hadoop configuration from the path, and jumping to step 104;
however, in a specific implementation scenario, a user deploying the application program does not necessarily have the right to set the environment variables; even if the environment variables can be set, the configuration files in the environment variables belong to public resources, and different application programs cannot be set by self-definition according to self needs, so the method has the following preferred methods:
in step 103, the Hadoop configuration file is placed in a Hadoop configuration file path defined by an application program, a resident YarnClient is constructed according to the Hadoop configuration file, and the flex cluster interacts with the yarncluster through the YarnClient.
In step 104, a resident FileSystems is constructed according to the Hadoop cluster, and the flex cluster interacts with the HDFS through the FileSystems.
The Hadoop configuration file is stored in a Hadoop configuration file path defined by an application program, wherein the Hadoop configuration file path defined by the application program is a known path of the application program and the flink, the application program and the flink can directly find the Hadoop configuration file stored in the Hadoop configuration file path according to the defined Hadoop configuration file path, the yarnClient and the FileSysteme are constructed through the Hadoop configuration file, the flink cluster interacts with the yarnCluster and the HDFS Cluster through the yarnClient and the FileSysteme, and specifically, an executeAsync method in a ContextEnvironment/StreanontextEnvironment class is rewritten to obtain the Hadoop configuration file, which is specifically as follows:
implementing the method getYarnclient () in the Environment factory class to construct a Yarnclient instance for accessing a Yarn cluster and an HDFS;
and initializing a Yannclient singleton according to the configuration file, wherein the Yannclient singleton is used for accessing the Yanncluster and the HDFS file system.
private YarnClient yarnClient＝null；
private YarnClient getYarnClient()
{
if(yarnClient＝＝null)
{
synchronized(EnvironmentGenerator.class)
{
if(yarnClient＝＝null)
{
YarnConfiguration yarnConf＝new YarnConfiguration()；
String[]hadoopResources＝new String[]{
new File(Constant.APP_HOME,“config/core-site.xml”).getPath(),
new File(Constant.APP_HOME,“config/hdfs-site.xml”).getPath(),
new File(Constant.APP_HOME,“config/yarn-site.xml”).getPath()}；
APP _ HOME: application root directory
for(String hadoopResource:hadoopResources)
{
try{
yarnConf.addResource(new FileInputStream(hadoopResource))；
}
catch(FileNotFoundException e){
throw new RuntimeException(e.getMassage())；
}
}
yarnClient＝YarnClient.createYarnClient()；
yarnClient.init(yarnConf)；
yarnClient.start()；
}
}
}
}
When a flink cluster is deployed on a yarn cluster, uploading jar packets related to the flink to an HDFS system file, wherein the jar packets related to the flink are stored in a local path of an application program, so that the jar packets related to the flink are uploaded to the HDFS system file when the flink cluster is deployed on the yarn cluster, and therefore a network IO is occupied, and two ways of deploying the flink cluster to the yarn cluster are described as follows:
a first kind of run-session (Start a long-running flag cluster on YARN), deploying a flag cluster on the yard cluster, receiving and running jobs, applying for a space only on the yard cluster, and only one flag cluster being deployed on the yard cluster, wherein the resources of the space are unchanged, and when a job is submitted, the space occupies the resources until the resources are full, and only when one job in the yard is executed, the resources are released to execute the next job; the method cannot simultaneously execute a large number of job tasks, and when a single flash cluster deployed in the yarn cluster fails, job characters cannot be executed, so that the flexibility and stability are poor;
a second kind of horn-per-job (Run a flight job on YARN), a plurality of flight clusters are deployed on the horn cluster, the submission of each job task corresponds to a single flight cluster, the job tasks apply for resources from the horn according to self conditions, are mutually independent and do not interfere with each other until the execution of the job is completed; the device method can simultaneously execute a large number of characters, and has more excellent stability and flexibility, so the yann-per-job method is more commonly used in practical situations; however, because the method of yarn-per-joba needs to frequently deploy the flink cluster on the yarn cluster, jar packets related to the flink need to be frequently uploaded to the HDFS system file, and a large number of network IO (input/output) can be occupied;
when the flink cluster is deployed on the yarn cluster, the method for uploading the flight related jar packets to the HDFS comprises the following three methods:
1. uploading all jar packages related to flink to a certain path of an HDFS system file in advance, and manually uploading the package to a user in advance, so that the user can directly know the determined path uploaded to the HDFS system file, and setting a yarn, parameter and parameter to point to the path in the HDFS system file according to the path of the HDFS system file, so that when a flink cluster is deployed to the yarn cluster, the user does not need to upload the jar packages related to flink to the HDFS system file, the stored jar packages related to flink are directly found in the path in the HDFS system file for loading and using, and the stored jar can realize the effect of uploading and using for the whole life at one time;
2. using a shaded mode to print an application program into a fat jar packet, recording a local path carried by the fat jar packet, uploading the fat jar packet to a system file of an HFDS from the path of the fat jar packet every time when a flink cluster is deployed to a yarn cluster for job submission, wherein a large amount of network IO is occupied when the uploading is performed every time, and a common resource occupies a space repeatedly;
3. the method is generally applied to a method that a jar package of a kernel-session is deployed on a kernel, and the method also has a process that the jar package needs to be uploaded to a system file of an HFDS, so that a large amount of network IO occupation is caused;
it can be seen from the above three methods for uploading the nail package related to flink to the HDFS that a large amount of network IO will be occupied as long as there is a frequent process for uploading the nail package from the local path to the system file of the HFDS, wherein the first method for uploading the nail package related to flink to the HDFS artificially uploads the nail package to the system file of the HFDS in advance, so that the nail package uploading required when the flink cluster is deployed on the yarn cluster is avoided, but in this way, the permission problem may be encountered in the implementation environment, that is, the deployer of the application may not have permission to directly upload the nail package to the HDFS system file;
as shown in fig. 2, the flow is as follows:
in step 201, deploying a flink cluster to a yarn cluster;
in step 202, judging whether a user has permission to directly upload a jar packet to an HDFS file system, and if yes, skipping to step 204; if no authority exists, skipping to 203;
in step 203, uploading the jar packet to a local path of the HDFS node in advance;
there are therefore the following preferred methods:
when there is no right to directly upload one or more of the flight system jar package and the third party dependent jar package to the HDFS file system, as shown in fig. 3, the flow is as follows:
in step 301, traversing original paths of jar packets of the flink system and third-party dependent jar packets;
in step 302, one or more of a jar packet of the flink system and a third-party dependent jar packet are deployed to local paths of all nodes of the Hadoop cluster;
in step 303, constructing a full path name of the jar package under the condition that the file structure of the original path of the jar package is consistent with the file structures of the local paths of all the nodes of the Hadoop cluster;
in step 304, replacing the parent path with the local paths of all nodes of the Hadoop cluster;
in step 305, adding the constructed jar package full path name into classpath;
when a deployer of an application program may not have the right to directly upload a jar package to an HDFS system file, the method does not directly upload the jar package to the HDFS system file first, but uploads the jar package to all HDFS nodes first, traverses the initial paths of the flash system jar package and a third party dependent jar package, identifies all jar packages needing to be uploaded, deploys the jar package to the paths of all the nodes of the HDFS, and can directly self-define the paths deployed to all the nodes of the HDFS because the deployment action is manually performed by a user, so that the path of the HDFS node deployed with the jar package can be grafted to a classpath, the following preferred method exists:
the path takes a designated path in an application program working path as a template, so that the HDFS node path with the jar package deployed can be grafted to the classpath, therefore, subfiles and directory structures in the HDFS node path are consistent with those of the designated path in the application program working path, wherein a parent path is an original path in which the jar package is locally stored, the parent path is replaced by the HDFS node path, and the constructed whole path name of the jar package is added to the classpath;
in step 306, the CLASSPATH is set to the parameter value corresponding to yarnconfigugkeys, env _ flip _ CLASSPATH in the environment variable of AM;
when the flink cluster is deployed on the yann cluster, the ApplicationMaster starts to load one or more of the flink system jar packet and the third party dependent jar packet from the local paths of all the nodes of the Hadoop cluster.
If the jar packet is not uploaded to the HDFS node, and the path of the HDFS node is not grafted to the classpath, when the flink cluster is deployed on the yann cluster, the ApplicationMaster is started, the jar packet is uploaded to an HDFS system file from the original path of the jar packet according to the original path of the jar packet needing to be uploaded recorded in the classpath, and occupation of network IO is generated; after uploading the jar packet to the HDFS node path and grafting the path with the classpath, when the flight cluster is deployed on the yann cluster, because the classpath is an HDFS node path for storing the jar packet instead of the original path of the jar packet, one or more of the jar packet of the flight system and a third party dependent jar packet are directly loaded from the HDFS node path, and under the condition that the HDFS system file is directly uploaded without permission, the step of uploading the file to the HDFS system from the original path of the jar packet is avoided, so that the network IO is not occupied.
The specific implementation manner is as follows:
package my.app.shaded.org.apache.flink.yarn；
public class YarnClusterDescriptor extends org.apache.flink.yarn.YarnClusterDescriptor
{
@Override
public ClusterClientProvider<ApplicationId> deployJobCluster(ClusterSpecification clusterSpecification,
JobGraph jobGraph,boolean detached)throws ClusterDeploymentException
{
return deployInternal(clusterSpecification,“Flink per-job cluster”,getYarnJobClusterEntrypoint(),
jobGraph,detached)；
}
private ClusterClientProvider<ApplicationId>deployInternal(...){
...
startAppMaster(...)；
...
}；
private ApplicationReport startAppMaster(Configuration configuration,String applicationName,
String yarnClusterEntrypoint,JobGraph jobGraph,YarnClient yarnClient,
YarnClientApplication yarnApplication,ClusterSpecification clusterSpecification)throws Exception
{
......
final List<Path>providedLibDirs＝ getRemoteSharedPaths(configuration)；
final YarnApplicationFileUploader fileUploader＝ YarnApplicationFileUploader.from(fs,
fs.getHomeDirectory(),providedLibDirs, appContext.getApplicationId(),getFileReplication())；
final List<String>systemClassPaths＝ fileUploader.registerProvidedLocalResources()；
if(providedLibDirs＝＝null||providedLibDirs.isEmpty())
{
List<String>localizedDependencies＝ getLocalizedDependencies(configuration)；
systemClassPaths.addAll(localizedDependencies)；
}
......
StringBuilder classPathBuilder＝new StringBuilder()；
systemClassPaths.forEach(classPath->
classPathBuilder.append(classpath).append(Utils.CLASS_PATH_SEPAR ATOR))；
......
appMasterEnv.put(YarnConfigKeys.ENV_FLINK_CLASSPATH, classPathBuilder.toString())；
......
}
private List<String>getLocalizedDependencies(Configuration configuration)throws Exception
{
List<String>localizedDependencies＝new ArrayList<String>()；
String appLocalLibDir＝ configuration.getString(YarnPerJobConfigOptions.APP_LOCAL_LIB_DIR)；
String hadoopLocalLibDir＝
configuration.getString(YarnPerJobConfigOptions.HADOOP_LOCAL_LIB _DIR)；
if(appLocalLibDir＝＝null||hadoopLocalLibDir＝＝null)
{
LOG.warn(“flink.on.yarn.app.local.lib.dir or flink.on.yarn.hadoop.local.lib.dir not found,
which may lead to NoClassDefFoundError”)；
return localizedDependencies；
}
final File appLocalFile＝new File(appLocalLibDir)；
if(appLocalFile.isDirectory())
{
final java.nio.file.Path path＝file.toPath()；
java.nio.file.Files.walkFileTree(path,new SimpleFileVisitor<java.nio.file.Path>(){
@Override
public FileVisitResult visitFile(java.nio.file.Path file, BasicFileAttributes attrs)
{
String relativeJarPath＝ (SystemUtil.CURRENT_OS＝＝SystemUtil.OS_WINDOWS)
？path.relativize(file).toString().replace(“\\”,“/”)
:path.relativize(file).toString()；
localizedDependencies.add(hadoopLocalLibDir+relativeJarPath)；
return FileVisitResult.CONTINUE；
}
})；
}
return localizedDependencies；
}
}
public class YarnPerJobConfigOptions
{
private YarnPerJobConfigOptions(){}
public static final ConfigOption<String>APP_LOCAL_LIB_DIR＝
ConfigOptions.key(“flink.on.yarn.app.local.lib.dir”)
.stringType()
.defaultValue(null)
.withDescription(“an app-local dir readable by app runed in flink-per-job mode.”
+“used to exclude local flink-job-dependent jars uploading to HDFS ”
+“to accelerate the flink job submission process coorperating with”
+“flink.on.yarn.hadoop.local.lib.dir”)；
public static final ConfigOption<String>HADOOP_LOCAL_LIB_DIR＝ ConfigOptions.key(“flink.on.yarn.hadoop.local.lib.dir”)
.stringType()
.defaultValue(null)
.withDescription(“an hadoop-local dir readable by yarn cluster.”
+“used to exclude local flink-job-dependent jars uploading to HDFS ”
+“to accelerate the flink job submission process coorperating with”
+“flink.on.yarn.app.local.lib.dir”)；
when a user has the right to directly upload a flash related jar package to an HDFS file system, the following preferred method exists:
when there is a right to directly upload one or more of a flight system jar package and a third party dependent jar package to the HDFS file system, as shown in fig. 4, the flow is as follows:
in step 401, before the flink cluster is deployed on the yann cluster, the flight system jar package and the third party dependent jar package are uploaded to a first path of the HDFS file system.
In step 402, setting a yarn, provided, lib, dirs parameter to point to a first path of the HDFS file system, when a flink cluster is deployed on the yarn cluster;
in step 403, the flink system jar package and the third party dependent jar package are loaded from the first path of the HDFS file system.
When a user has permission to directly upload a flink-related jar package to an HDFS (Hadoop distributed file system), when a flink cluster is deployed to a yann cluster, a fink-related jar package and related configuration are directly and manually uploaded to a first path of the HDFS, a yann, provided, lib, dir parameter is set according to the first path of the HDFS, when the flink cluster is deployed to the yann cluster, the flink-related jar package is directly loaded from the first path of the HDFS, the jar package does not need to be uploaded to the HDFS from the original path, and occupation of network IO (input output) caused by frequent uploading is avoided; the first path is set by the person skilled in the art at his discretion and should not limit the scope of protection of the present invention.
When a flink cluster is deployed on a yarn cluster, and after a flight related jar package is uploaded to an HDFS system file, the flink wants to normally receive an operation task after deployment on yarn, and the following preferred methods also exist:
the application program client applies a resource container to ResourceMenager for starting the ApplicationMaster, the application program client registers the flink configuration file and the jar package as the resource of the resource container, the nodemanager uses the resource of the resource container to prepare the resource container, and the ApplicationMaster is started.
JobManager and ApplicationMaster operate in the same horn container, when the ApplicationMaster knows the address of JobManager, the ApplicationMaster generates a new Flink configuration file, the Flink configuration file is used for the Flink TaskManager, the ApplicationMaster starts the associates of the Flink TaskManager, the associates downloads the jar file and the Flink configuration file from the HDFS, and the Flink completes the deployment on horn and starts to receive tasks.
When the Flink wants to normally receive the running task after deployment on yarn, the Flink task manager needs to be started, the client will apply a yarn container to resource manager to start the ApplicationMaster, and as the client has registered the configuration and jar files as the resource of the container, all nodemanagers will directly use the resources to prepare the container, wherein the prepared process is such as downloading files and the like; when the process is finished, starting the applicationMaster, operating the JobManager and the applicationMaster on the same container, and when the JobManager is successfully established, knowing the address of the JobManager by the applicationMaster; the ApplicationMaster generates a new Flink configuration file, the Flink configuration file can be used by a Flink TaskManager to be started, and the Flink configuration file also needs to be uploaded to an HDFS system file; the colleague, the container provides a web interface for the flink; the ports applied by the Yarn code are all temporary ports, and the purpose of the temporary ports is to enable a user to start a plurality of flight Yarn sessions in parallel; when the ApplicationMaster starts to apply for starting contacts of the Flink TaskManager, the contacts download the jar files and the modified configuration files related to the uploaded Flink from the HDFS system files, and after the steps are completed, the Flink completes deployment on yarn and starts to receive tasks.
Example 2:
embodiment 2 of the present invention provides an implementation method for submitting a flight job to a yann cluster in an application program, and this embodiment 2 shows an implementation process for solving the deployment problem of a Hadoop configuration file deployed by a flight on yann in this scheme in a more practical scenario than embodiment 1.
When the flink runs on yarn, a yarn client object needs to be constructed to access resource manager of yarn, a fileSystem object also needs to be constructed to access HDFS, and a Hadoop configuration file needs to be found to construct a yarn client object and a fileSystem object;
the flink native code checks whether the environment variables such as YARN _ CONF _ DIR, HADOOP _ CONF _ PATH and the like exist in sequence; finding that environment variables such as YARN _ CONF _ DIR, HADOOP _ CONF _ PATH and the like are not set, checking the HADOOP _ HOME environment variables, wherein for the HADOOP 2.x, the searched configuration PATH is $ HADOOP _ HOME/etc/HADOOP, finding that the setting is not set, and for the HADOOP 1.x, the searched configuration PATH is $ HADOOP _ HOME/CONF, finding that the setting is not set; at the moment, the Hadoop configuration file cannot be directly searched and deployed;
at this time, rewriting an acquired yankeent instance of an executeAsync method in a context environment/StreamConntextEnvironment class, and storing the Hadoop configuration file in an application program-defined Hadoop configuration file path, wherein the application program-defined Hadoop configuration file path is a known path of an application program and a flink, and the application program and the flink can directly find the Hadoop configuration file stored in the Hadoop configuration file path according to the defined Hadoop configuration file path; the specific implementation method comprises the following steps:
implementing the method getYarnclient () in the Environment factory class to construct a Yarnclient instance for accessing a Yarn cluster and an HDFS;
and initializing a Yannclient singleton according to the configuration file, wherein the Yannclient singleton is used for accessing the Yanncluster and the HDFS file system.
private YarnClient yarnClient＝null；
private YarnClient getYarnClient()
{
if(yarnClient＝＝null)
{
synchronized(EnvironmentGenerator.class)
{
if(yarnClient＝＝null)
{
YarnConfiguration yarnConf＝new YarnConfiguration()；
String[]hadoopResources＝new String[]{
new File(Constant.APP_HOME,“config/core-site.xml”).getPath(),
new File(Constant.APP_HOME,“config/hdfs-site.xml”).getPath(),
new File(Constant.APP_HOME,“config/yarn-site.xml”).getPath()}；
APP _ HOME: application root directory
for(String hadoopResource:hadoopResources)
{
try{
yarnConf.addResource(new FileInputStream(hadoopResource))；
}
catch(FileNotFoundException e){
throw new RuntimeException(e.getMassage())；
}
}
yarnClient＝YarnClient.createYarnClient()；
yarnClient.init(yarnConf)；
yarnClient.start()；
}
}
}
}
Example 3:
embodiment 3 of the present invention provides an implementation method for submitting a flash job to a yann cluster in an application program, and embodiment 3 shows an implementation flow for solving the problem that a large number of network IO are occupied when jar packets are uploaded to an HDFS in this scheme in a more practical scenario than embodiment 1.
When a job is required to be submitted from a flink to run on a yann, firstly checking whether a user has a right to directly upload a file to an HDFS system file, wherein the user does not have the right to directly upload the file to the HDFS system file, so that the original paths of a jar package of the flink system and a jar package depended by a third party are traversed firstly, the original paths existing in a folder of a D disc 110 are found, the user manually uploads the jar package of the flink system and the jar package depended by the third party from the folder of the D disc 110 to full-path homonymy paths of all nodes of the HDFS, the specified paths in an application program are taken as templates, subfiles and directory structures in the full-path homonymy paths of all nodes of the HDFS are kept consistent with subfiles and directory structures of the specified paths in the working path of the application program, the full-path homonymy paths of all nodes of the HDFS are replaced, and the full-path names of the constructed jar packages are added into a classpat; setting the CLASSPATH into a parameter value corresponding to YarnConfigKeys, ENV _ FLINK _ CLASSPATH in an environment variable of the AM, starting the ApplicationMaster when a FLINK cluster is deployed on the yann cluster, and loading one or more of a FLINK system jar packet and a third party dependent jar packet from a full path homonymous path of all nodes of the Hadoop.
Codes related to a jar uploading process and a classpath building process in the yarncrusterDescriptor class are rewritten, and the codes are as follows:
package my.app.shaded.org.apache.flink.yarn；
public class YarnClusterDescriptor extends org.apache.flink.yarn.YarnClusterDescriptor
{
@Override
public ClusterClientProvider<ApplicationId> deployJobCluster(ClusterSpecification clusterSpecification,
JobGraph jobGraph,boolean detached)throws ClusterDeploymentException
{
return deployInternal(clusterSpecification,“Flink per-job cluster”,getYarnJobClusterEntrypoint(),
jobGraph,detached)；
}
private ClusterClientProvider<ApplicationId>deployInternal(...){
...
startAppMaster(...)；
...
}; // native code, unmodified
private ApplicationReport startAppMaster(Configuration configuration,String applicationName,
String yarnClusterEntrypoint,JobGraph jobGraph,YarnClient yarnClient,
YarnClientApplication yarnApplication,ClusterSpecification clusterSpecification)throws Exception
{
......
final List<Path>providedLibDirs＝ getRemoteSharedPaths(configuration)；
final YarnApplicationFileUploader fileUploader＝ YarnApplicationFileUploader.from(fs,
fs.getHomeDirectory(),providedLibDirs, appContext.getApplicationId(),getFileReplication())；
final List<String>systemClassPaths＝ fileUploader.registerProvidedLocalResources()；
if(providedLibDirs＝＝null||providedLibDirs.isEmpty())
{
List<String>localizedDependencies＝ getLocalizedDependencies(configuration)；
systemClassPaths.addAll(localizedDependencies)；
}
......
StringBuilder classPathBuilder＝new StringBuilder()；
systemClassPaths.forEach(classPath->
classPathBuilder.append(classpath).append(Utils.CLASS_PATH_SEPAR ATOR))；
......
appMasterEnv.put(YarnConfigKeys.ENV_FLINK_CLASSPATH, classPathBuilder.toString())；
......
}
private List<String>getLocalizedDependencies(Configuration configuration)throws Exception
{
List<String>localizedDependencies＝new ArrayList<String>()；
String appLocalLibDir＝ configuration.getString(YarnPerJobConfigOptions.APP_LOCAL_LIB_DIR)；
String hadoopLocalLibDir＝
configuration.getString(YarnPerJobConfigOptions.HADOOP_LOCAL_LIB _DIR)；
if(appLocalLibDir＝＝null||hadoopLocalLibDir＝＝null)
{
LOG.warn(“flink.on.yarn.app.local.lib.dir or flink.on.yarn.hadoop.local.lib.dir not found,
which may lead to NoClassDefFoundError”)；
return localizedDependencies；
}
final File appLocalFile＝new File(appLocalLibDir)；
if(appLocalFile.isDirectory())
{
final java.nio.file.Path path＝file.toPath()；
java.nio.file.Files.walkFileTree(path,new SimpleFileVisitor<java.nio.file.Path>(){
@Override
public FileVisitResult visitFile(java.nio.file.Path file, BasicFileAttributes attrs)
{
String relativeJarPath＝
(SystemUtil.CURRENT_OS＝＝SystemUtil.OS_WINDOWS)
？path.relativize(file).toString().replace(“\\”,“/”)
:path.relativize(file).toString()；
localizedDependencies.add(hadoopLocalLibDir+relativeJarPath)；
return FileVisitResult.CONTINUE；
}
})；
}
return localizedDependencies；
}
}
public class YarnPerJobConfigOptions
{
private YarnPerJobConfigOptions(){}
public static final ConfigOption<String>APP_LOCAL_LIB_DIR＝
ConfigOptions.key(“flink.on.yarn.app.local.lib.dir”)
.stringType()
.defaultValue(null)
.withDescription(“an app-local dir readable by app runed in flink-per-job mode.”
+“used to exclude local flink-job-dependent jars uploading to HDFS
”
+“to accelerate the flink job submission process coorperating with”
+“flink.on.yarn.hadoop.local.lib.dir”)；
public static final ConfigOption<String>HADOOP_LOCAL_LIB_DIR＝
ConfigOptions.key(“flink.on.yarn.hadoop.local.lib.dir”)
.stringType()
.defaultValue(null)
.withDescription(“an hadoop-local dir readable by yarn cluster.”
+“used to exclude local flink-job-dependent jars uploading to HDFS
”
+“to accelerate the flink job submission process coorperating with”
+“flink.on.yarn.app.local.lib.dir”)；
Example 4:
fig. 5 is a schematic diagram of an apparatus for implementing submitting a flight job to a yarn cluster in an application according to an embodiment of the present invention. The implementation apparatus for submitting a flight job to a yarn cluster in an application program of the embodiment includes one or more processors 21 and a memory 22. In fig. 5, one processor 21 is taken as an example.
The processor 21 and the memory 22 may be connected by a bus or other means, and fig. 5 illustrates the connection by a bus as an example.
The memory 22, which is a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the implementation method of submitting a flash job to a yann cluster in an application program in embodiment 1. The processor 21 executes the flow decomposition method of the Flink framework running ETL by running non-volatile software programs and instructions stored in the memory 22.
The memory 22 may include high speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the memory 22 may optionally include memory located remotely from the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The program instructions/modules are stored in the memory 22, and when executed by the one or more processors 21, perform the implementation method for submitting a flight job to a yann cluster in an application program in the above embodiment 1, for example, perform the steps shown in fig. 1 to fig. 4 described above.
The above description is intended to be illustrative of the preferred embodiment of the present invention and should not be taken as limiting the invention, but rather, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.
Claims (6)
1. An implementation method for submitting a flight job to a yarn cluster in an application program, the method comprising:
the method comprises the steps that a Flink cluster is deployed on a yann cluster, a Hadoop configuration file is placed in a Hadoop configuration file path defined by an application program, a resident yannclient is constructed according to the Hadoop configuration file defined by the application program, and the Flink cluster interacts with the yann cluster through the yannclient; constructing a resident FileSystemaccording to the Hadoop cluster, and interacting the flink cluster with the HDFS through the FileSysteme;
when the file system has no right to directly upload a flash system jar package and a third party dependent jar package to the HDFS file system, traversing original paths of the flash system jar package and the third party dependent jar package, deploying one or more of the flash system jar package and the third party dependent jar package into local paths of all nodes of the Hadoop cluster, constructing a full path name of the jar package under the condition that file structures of the original paths of the flash system jar package and the third party dependent jar package are consistent with file structures of the local paths of all nodes of the Hadoop cluster, replacing a parent path with the local paths of all nodes of the Hadoop cluster, and adding the constructed full path name of the jar package into a classpath;
setting the CLASSPATH into a parameter value corresponding to YarnConfigKeys, ENV _ FLINK _ CLASSPATH in an environment variable of the AM, starting the ApplicationMaster when a FLINK cluster is deployed on the yann cluster, and loading a FLINK system jar packet and a third party dependent jar packet from local paths of all nodes of the Hadoop cluster;
when the flight cluster is deployed on the yarn cluster, loading the jar packages of the flight system and the third-party dependent jar packages in local paths of all nodes of the Hadoop cluster.
2. The method of claim 1, wherein when there is a right to directly upload the flink system jar package and the third party dependent jar package to the HDFS file system, the flink system jar package and the third party dependent jar package are uploaded to a first path of the HDFS file system before the flink cluster is deployed on the yarn cluster.
3. The method as claimed in claim 2, wherein the parameter "yann. Provided. Lib. Dirs" is set to point to the first path of the HDFS file system, and when a flink cluster is deployed on the yann cluster, the flink system jar package and the third party dependent jar package are loaded from the first path of the HDFS file system.
4. The method for implementing submission of a flink job to a yarn cluster in an application program according to claim 1, wherein the loading of the flight system jar package and the third party dependent jar package in the full path homonymous path of all nodes of the HDFS further comprises:
the application client applies for a yann container to the ResourceMenager to start the applicationMaster, the application client registers the flink configuration file and the jar packet as the resource of the yann container, the nondemenager prepares the yann container by using the resource of the yann container, and the applicationMaster is started.
5. The method as claimed in claim 4, wherein JobManager and ApplicationMaster run on the same yann container, when the ApplicationMaster knows the address of JobManager, the ApplicationMaster generates a new Flink configuration file, the Flink configuration file is used by the TaskManager, the ApplicationMaster starts the containers of the Flink TaskManager, the containers download the jar file and the Flink configuration file from the HDFS, and the Flink completes the deployment on yann and starts receiving tasks.
6. An implementation apparatus for submitting a flight job to a yarn cluster in an application program, wherein the implementation apparatus for submitting the flight job to the yarn cluster in the application program comprises at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor for performing the method of any of claims 1-5 for submitting a flight job to a yarn cluster in an application.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111674276.4A CN114489833B (en) | 2021-12-31 | 2021-12-31 | Implementation method and device for submitting flash job to yarn cluster in application program |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111674276.4A CN114489833B (en) | 2021-12-31 | 2021-12-31 | Implementation method and device for submitting flash job to yarn cluster in application program |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114489833A CN114489833A (en) | 2022-05-13 |
| CN114489833B true CN114489833B (en) | 2022-12-23 |
Family
ID=81508758
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111674276.4A Active CN114489833B (en) | 2021-12-31 | 2021-12-31 | Implementation method and device for submitting flash job to yarn cluster in application program |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114489833B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116382773A (en) * | 2022-08-09 | 2023-07-04 | 杭州玳数科技有限公司 | Method for deploying PyFlink task |
| CN116048533B (en) * | 2023-04-03 | 2023-07-25 | 浙江数新网络有限公司 | Implementation method and system for achieving dependency isolation in Flink task running |
| CN117519774A (en) * | 2023-11-01 | 2024-02-06 | 广州市玄武无线科技股份有限公司 | Packing method, device, equipment and storage medium of flink |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105893545A (en) * | 2016-04-01 | 2016-08-24 | 浪潮电子信息产业股份有限公司 | Efficient Hadoop cluster deployment method |
| CN109740373A (en) * | 2018-12-19 | 2019-05-10 | 福建新大陆软件工程有限公司 | A kind of Hadoop cluster management method, system and platform |
| US10628217B1 (en) * | 2017-09-27 | 2020-04-21 | Amazon Technologies, Inc. | Transformation specification format for multiple execution engines |
| CN112882728A (en) * | 2021-03-25 | 2021-06-01 | 浪潮云信息技术股份公司 | Deployment method of big data platform real-time computing service Flink based on Yarn |
-
2021
- 2021-12-31 CN CN202111674276.4A patent/CN114489833B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105893545A (en) * | 2016-04-01 | 2016-08-24 | 浪潮电子信息产业股份有限公司 | Efficient Hadoop cluster deployment method |
| US10628217B1 (en) * | 2017-09-27 | 2020-04-21 | Amazon Technologies, Inc. | Transformation specification format for multiple execution engines |
| CN109740373A (en) * | 2018-12-19 | 2019-05-10 | 福建新大陆软件工程有限公司 | A kind of Hadoop cluster management method, system and platform |
| CN112882728A (en) * | 2021-03-25 | 2021-06-01 | 浪潮云信息技术股份公司 | Deployment method of big data platform real-time computing service Flink based on Yarn |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114489833A (en) | 2022-05-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114489833B (en) | Implementation method and device for submitting flash job to yarn cluster in application program | |
| JP7275171B2 (en) | Operating System Customization in On-Demand Network Code Execution Systems | |
| EP3561669B1 (en) | Containerized application deployment | |
| US10942724B2 (en) | Release lifecycle management system for multi-node application | |
| CN106796508B (en) | Fast deployment on cloud platforms | |
| CN105657191B (en) | Application increment upgrading method and system based on Android system | |
| US20100205604A1 (en) | Systems and methods for efficiently running multiple instances of multiple applications | |
| CN108897547B (en) | Software automation deployment method and device | |
| CN105490860A (en) | Method, device and system for deploying application program operation environment | |
| US20040088397A1 (en) | System and method for management of software applications | |
| CN1781077A (en) | Selective loading of remote classes or resources for debugging a Java application in a Java micro device | |
| KR20050061376A (en) | Determining the maximal set of dependent software updates valid for installation | |
| CN112130871B (en) | Method and device for remotely deploying middleware, computer equipment and storage medium | |
| CN104699537A (en) | Program control method, activity module scheduling method and corresponding devices thereof | |
| CN111176818A (en) | Distributed prediction method, device, system, electronic equipment and storage medium | |
| US10203976B2 (en) | Virtual appliance management in a virtualized computing environment based on operational modes associated with virtual appliance | |
| CN116860746A (en) | Processing system for lightweight big data | |
| US20070061277A1 (en) | Method, system, and storage medium for providing dynamic deployment of grid services over a computer network | |
| CN109753302B (en) | Service method without service function based on hybrid cloud computing platform | |
| CN115480785A (en) | Container-based service deployment method and device and server | |
| CN115981789A (en) | Container-based Internet of things data batch processing method, processor and storage medium | |
| CN113076128B (en) | Method, device, electronic equipment and storage medium for robot configuration | |
| CN113342464A (en) | Application construction deployment method based on container service | |
| CN113076109B (en) | Cross-platform script language deployment method | |
| CN115129333B (en) | NPm resource package management method, device and system suitable for micro front-end framework |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |