CN112558995A - Flink integration method and system based on TBDS Hadoop - Google Patents
Flink integration method and system based on TBDS Hadoop Download PDFInfo
- Publication number
- CN112558995A CN112558995A CN202011549744.0A CN202011549744A CN112558995A CN 112558995 A CN112558995 A CN 112558995A CN 202011549744 A CN202011549744 A CN 202011549744A CN 112558995 A CN112558995 A CN 112558995A
- Authority
- CN
- China
- Prior art keywords
- tbds
- flink
- configuration
- authentication
- hadoop
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- DDCPKNYKNWXULB-RXMQYKEDSA-N (2r)-2-azaniumyl-3-[(2-methylpropan-2-yl)oxy]propanoate Chemical compound CC(C)(C)OC[C@@H]([NH3+])C([O-])=O DDCPKNYKNWXULB-RXMQYKEDSA-N 0.000 title claims abstract description 153
- 238000000034 method Methods 0.000 title claims abstract description 86
- 230000010354 integration Effects 0.000 title claims abstract description 19
- 238000012795 verification Methods 0.000 claims abstract description 19
- 238000009434 installation Methods 0.000 claims abstract description 9
- 230000008569 process Effects 0.000 claims description 8
- 238000012986 modification Methods 0.000 abstract description 8
- 230000004048 modification Effects 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 5
- 238000013475 authorization Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
- G06F8/61—Installation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Stored Programmes (AREA)
Abstract
The application relates to a FLink integration method and system based on TBDS Hadoop, wherein the method comprises the following steps: respectively carrying out configuration parameter increase and compatible modification on a configuration file and a verification part of the Flink Client; installing a Flink Client on a TBDS Hadoop Client, and configuring installation authentication configuration in the environment variable of the TBDS Hadoop Client; compiling a Flink-cached according to the current version of the TBDS Hadoop, generating a corresponding version jar packet, and storing the version jar packet into a Flink lib directory; configuring the authentication configuration of the TBDS Hadoop in a configuration file; and the Flink Client submits the Flink task, and the Flink task normally runs on the TBDS Hadoop according to the steps. By modifying the configuration file and the verification part of the Flink Client, the safety certification of the TBDS Hadoop cloud environment is increased on the premise of not influencing the existing safety certification, the problem that the Flink cannot normally run in the TBDS Hadoop cloud environment is solved, and extra machine resources required by deployment of a software product constructed based on the Flink are reduced.
Description
Technical Field
The application relates to the technical field of big data operation, in particular to a method and a system for integrating Flank based on TBDS Hadoop.
Background
Some customers in the customer environment purchase relevant Tencent cloud resources (TBDS Hadoop), and in order not to increase the cost of the customers additionally, and hopefully deploy products to the environment, the relevant technologies involved in the products need to be adapted to integrate with the third party Tencent cloud (TBDS Hadoop) environment. The Flink open source components used in the product cannot run in a TBDS cloud environment.
At present, no effective solution is provided for the problem that the related technology cannot operate.
Disclosure of Invention
The embodiment of the application provides a Flink integration method and system based on TBDS Hadoop, aiming at solving the technical problem that the exclusive property cannot be operated in the related technology, the code of a Flink security authentication module needs to be modified, and the TBDS Hadoop cloud environment security authentication module is added under the condition that the existing security authentication is not influenced.
In a first aspect, an embodiment of the present application provides a method for integrating a Flink based on TBDS Hadoop, including the following steps:
a step of Flink configuration, which is to respectively increase configuration parameters and compatibly modify the configuration file and the verification part of the Flink Client;
a software installation step, namely installing the Flink Client on a TBDS Hadoop Client, and configuring installation authentication configuration in the environment variable of the TBDS Hadoop Client;
compiling a Flink-cached according to the current version of the TBDS Hadoop to generate a corresponding version jar packet, and storing the version jar packet into a Flink lib directory;
an authentication configuration step, in which authentication configuration of TBDS Hadoop is configured in the configuration file;
and a task submitting step, wherein the Flink Client submits the Flink task according to the steps.
In some embodiments, the Flink configuring step specifically includes:
a configuration item parameter configuration step, namely adding the configuration parameters required by tbds security authentication in the configuration file, and adding corresponding configuration item variables in the document and security options;
and adding an isTBDS method and an isKerberos method in the HadoopUtils class for judging the authentication type.
In some embodiments, the configuring step specifically includes:
and adding three configuration parameters required by TBDS security authentication in the configuration file, adding a TBDS security configuration item constant in the Documentation class, wherein the TBDS security configuration item constant corresponds to the configuration parameters, adding a TBDS security configuration item variable required by the TBDS in the SecurityOptions class, and reading a variable value corresponding to the configuration parameters according to the TBDS security configuration item variable.
In some embodiments, the authentication configuring step specifically includes:
judging the authentication type, namely judging the authentication type according to the isTBDS method and the isKerberos method;
a TBDS security authentication step, namely entering normal authentication when the current authentication type is judged to be the TBDS, and adding the configuration parameters in the configuration file into a Hadoop configuration obtained by the Flank Client;
and a Kerberos security authentication step, when the authentication type is Kerberos, entering security authentication, and verifying a Kerberos verification code through a Hadoop Module authentication class and a YarnCluster Descriptor submission class.
In some embodiments, the TBDS security authentication step specifically includes the following steps:
and finally, judging the isTBDS method in a getHadoop configuration method in a Hadoop Utils class, and adding the configuration parameters into the current Hadoop configuration when the authentication type is TBDS.
In some embodiments, the Kerberos security authentication step comprises:
when the current authentication type is judged to be Kerberos, executing related authentication in an install method of a HadoopModule authentication type, and verifying a partial code of the Kerberos;
in the process of the Flink Client executing the Flink task submission, a DeltoInternal method and a startAppMaster method of a YarnClusterDescriptor submission type are adopted to verify a Kerberos partial code.
In a second aspect, an embodiment of the present application provides a Flink integration system based on TBDS Hadoop, where the method according to the first aspect is applied, and includes:
the Flink configuration module is used for respectively increasing configuration parameters and compatibly modifying the configuration file and the verification part of the Flink Client;
the software installation module is used for installing the Flink Client on a TBDS Hadoop Client and configuring installation authentication configuration in the environment variable of the TBDS Hadoop Client;
compiling a Flink-cached according to the current version of the TBDS Hadoop to generate a corresponding version jar packet, and storing the version jar packet into a Flink lib directory;
the authentication configuration module is used for configuring the authentication configuration of the TBDS Hadoop in the configuration file;
and the task submitting module is used for submitting the Flink task by the Flink Client according to the steps.
In some embodiments, the Flink configuration module specifically includes:
a configuration item parameter configuration unit, which adds the configuration parameters required by tbds security authentication in the configuration file, and adds the corresponding configuration item variables in the document and security options;
and the authentication type judging method adding unit is used for adding an isTBDS method and an isKerberos method in the HadoopUtils class and judging the authentication type.
In some embodiments, the configuration item parameter configuration unit specifically includes:
and adding three configuration parameters required by TBDS security authentication in the configuration file, adding a TBDS security configuration item constant in the Documentation class, wherein the TBDS security configuration item constant corresponds to the configuration parameters, adding a TBDS security configuration item variable required by the TBDS in the SecurityOptions class, and reading a variable value corresponding to the configuration parameters according to the TBDS security configuration item variable.
In some embodiments, the authentication configuration module specifically includes:
the authentication type judging unit judges the authentication type according to the isTBDS method and the isKerberos method;
the TBDS security authentication unit enters common authentication when judging that the current authentication type is the TBDS, and adds the configuration parameters in the configuration file into Hadoop configuration obtained by the Flank Client;
and the Kerberos security authentication unit enters security authentication when the authentication type is Kerberos, and verifies the Kerberos verification code through a Hadoop Module authentication class and a YarnCluster Descriptor submission class.
Compared with the related art, according to the method and the system for integrating the Flink based on the TBDS Hadoop, the configuration file and the verification part of the Flink Client are modified, the safety certification of the TBDS Hadoop cloud environment is increased on the premise that the existing safety certification is not influenced, the problem that the Flink cannot normally run in the TBDS Hadoop cloud environment is solved, the running of the Flink in the TBDS Hadoop cloud environment is realized, normal service is provided, and extra machine resources required by deployment of software products constructed based on the Flink are reduced.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a flow chart of a method for TBDS Hadoop based Flink integration according to an embodiment of the present application;
FIG. 2 is a flow chart of Flink configuration steps according to an embodiment of the present application;
FIG. 3 is a flowchart of authentication configuration steps according to an embodiment of the present application;
FIG. 4 is a flow chart of a method for TBDS Hadoop based Flink integration according to the preferred embodiment of the present application;
FIG. 5 is a block diagram of a TBDS Hadoop-based Flink integrated system according to an embodiment of the present application.
Description of the drawings:
flink configuration module 1; a software installation module 2; a step 3 of storing the version jar packet;
an authentication configuration module 4; a task submission module 5; a configuration item parameter configuration unit 11;
an authentication type determination method addition unit 12; an authentication type judgment unit 41;
TBDS security authentication unit 42; a Kerberos security authentication unit 43;
the Flink integration system 100 based on TBDSHadoop.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.
It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.
The embodiment provides a FLink integration method based on TBDS Hadoop. Fig. 1 is a flowchart of a method for integrating Flink based on TBDS Hadoop according to an embodiment of the present application, and as shown in fig. 1, the flowchart includes the following steps:
a step S1 of Flink configuration, in which configuration parameters are added and compatible modification is carried out on the configuration file and the verification part of the Flink Client respectively;
a software installation step S2, installing a Flink Client on a TBDS Hadoop Client, and configuring installation authentication configuration in the environment variable of the TBDS Hadoop Client;
a version jar packet storage step S3, compiling a Flink-cached according to the current version of the TBDS Hadoop, generating a corresponding version jar packet, and storing the version jar packet into a Flinklib directory;
an authentication configuration step S4, wherein the authentication configuration of the TBDS Hadoop is configured in the configuration file; authentication configuration parameters
And a task submitting step S5, namely a task submitting step, wherein the Flink Client submits the Flink task according to the steps.
In practical application, firstly, configuration parameters are added in a configuration file and are subjected to compatible modification, through the change modification, the Flink is compiled automatically, the Flink is installed in a TBDS Hadoop Client machine, and three installation authentication configurations of the TBDS are configured in environment variables in the TBDS Hadoop Client machine (namely, a TBDS Hadoop Client) and are used for performing interactive authentication between a command line and a Hadoop service.
Secondly, compiling a Flink-shaded engineering file is needed, generating jar packages depending on the current TBDS Hadoop version, and putting the jar packages into a Flink lib directory for interaction between the Flink and the TBDS Hadoop.
Thirdly, three authentication configurations of TBDS Hadoop are configured in a Flink-conf.yaml file.
And finally, after the flow is completed, the submission of the Flink task can be normally executed, and the integration of the Flink and the TBDS Hadoop is completed.
After submission, the FlinkAM provides an REST interface to receive the submission of the Flink task of the Flink Client, starts the JM and submits the Flink task, and runs the WebUI at the same time, so that the Flink task normally runs on the TBDS Hadoop.
Through the steps, the configuration file and the verification part of the Flink Client are modified, the safety certification of the TBDS Hadoop cloud environment is added on the premise of not influencing the existing safety certification, the problem that the Flink cannot normally run in the TBDS Hadoop cloud environment is solved, the running of the Flink in the TBDS Hadoop cloud environment is realized, normal service is provided, and extra machine resources required by deployment of a software product constructed based on the Flink are reduced.
Fig. 2 is a flowchart of a Flink configuration step according to an embodiment of the present application, and as shown in fig. 2, in some embodiments, the Flink configuration step S1 specifically includes:
a configuration item parameter configuration step S11, adding configuration parameters required by tbds security authentication in the configuration file, and adding corresponding configuration item variables in the document and security options;
the authentication type determination method adds step S12, adding an isTBDS method and an isKerberos method in the HadoopUtils class for determining the authentication type.
In practical application, the TBDS security configuration item constant corresponds to three configuration constants in the conf file, and the name of the configuration parameter in the configuration file is read.
The SecurityOptions class is a class that adds security authentication, and the addition of the authentication type of TBDS requires the TBDS type to be labeled in the TBDS security configuration item constant in the documentation class.
The Hadoop Utils class is an interactive help class used for Flink and hadoop and is used for judging the authentication type of the hadoop.
In some embodiments, the configuration item parameter configuring step S11 specifically includes:
three configuration parameters required by TBDS security authentication are added in the configuration file, TBDS security configuration item constants are added in the Documentation class, the TBDS security configuration item constants correspond to the configuration parameters, TBDS security configuration item variables required by TBDS are added in the SecurityOptions class, and variable values of the corresponding configuration parameters are read according to the TBDS security configuration item variables.
It should be noted that, the configuration parameters may be set to an authentication user name, an authentication user security period, and a security authentication authorization code.
Fig. 3 is a flowchart of an authentication configuration step according to an embodiment of the present application, and as shown in fig. 3, in some embodiments, the authentication configuration step S4 specifically includes:
an authentication type judgment step S41, judging the authentication type according to the isTBDS method and the isKerberos method;
a TBDS security authentication step S42, entering normal authentication when the current authentication type is judged to be TBDS, and adding configuration parameters in a configuration file into Hadoop configuration obtained by a Flank Client;
and a Kerberos security authentication step S43, when the authentication type is Kerberos, entering security authentication, and submitting a Kerberos verification code through a Hadoop Module authentication class and a YarnCluster Descriptor.
In some embodiments, the TBDS security authentication step S42 specifically includes the following steps:
and finally, judging an isTBDS method in the getHadoop configuration method in the Hadoop Utils class, and adding configuration parameters into the current Hadoop configuration when the authentication type is TBDS.
In practical application, when the authentication type is judged to be TBDS, reading the TBDS configuration parameters from the Flank configuration and the documentation file to obtain the corresponding configuration parameter values, and adding the configuration parameter values into the current Hadoop configuration.
In some of these embodiments, the Kerberos security authentication step S43 includes:
when the current authentication type is judged to be Kerberos, executing related authentication in an install method of a HadoopModule authentication type, and verifying a partial code of the Kerberos;
in the process of the Flink Client executing the Flink task submission, a DeltoInternal method and a startAppMaster method of a YarnClusterDescriptor submission type are adopted to verify a Kerberos partial code.
It should be noted that, in the present embodiment, only two authentication manners, i.e., TBDS authentication and Kerberos authentication, are mentioned alternatively, but the present invention is not limited thereto.
The HadoopModule class is a Flink security authentication execution class, related authentication is executed in the class install method, compatible modification is needed, the authentication type needs to be judged, security authentication needs to be carried out when Kerberos is judged, Kerberos authentication is executed by adding judgment of isKerberos, otherwise, the authentication is common authentication, for example, when TBDS Hadoop belongs to common authentication, extra authentication codes do not need to be added, and users obtaining authorization can be used for operation only by adding configuration parameters in the Hadoop configuration.
When the authentication mode is Kerberos, since Kerberos authentication is bound in the Flink submission task flow, the part needs to be modified in the authentication submission process, and Kerberos verification codes are verified in a deployInternal and startAppMaster method which submit a yarnClusterDescriptor-like.
The embodiments of the present application are described and illustrated below by means of preferred embodiments.
FIG. 4 is a flow chart of a method for TBDS Hadoop based Flink integration according to the preferred embodiment of the present application.
S401, first, the Flink verification section is changed:
s4011, add three configuration parameters required for TBDS authentication in the flink-conf.yaml configuration file, and add a TBDS security configuration item constant in the org.apache.flink.annotation.docs.documentation, and add which configuration item variable the TBDS requires in the org.apache.flink.configuration.security operations, for reading a corresponding configured variable value in the flink-conf.yaml file.
S4012, adding an isTBDS and an isKerberos method in org.apache.flink.runtime.util.Hadoop Utils class for judging the authentication type;
s40131, the judgment of isTBDS is finally added to the gethadoop configuration method in org.apache.flink.runtime.util.hadoop classes, and if the type is TBDS authentication type, TBDS configuration items (values of corresponding configuration items read from the document file) need to be added to the current Hadoop configuration from the flink configuration.
S401321, org, apache, flash, runtime, security, modules, hadoopmodule class is a Flink security authentication executing class, and executes related authentication in the class install method, where compatible modification is required, when the class is determined to be Kerberos authentication type, and security authentication is required, the class is determined to be isKerberos, and when security authentication is required, Kerbero authentication is executed by adding thereto a determination of isKerberos, otherwise, the class is normal authentication.
It should be noted that, when TBDSHadoop belongs to the common authentication, the user obtaining the authorization can complete the operation by adding the configuration item in the hadoop configuration without adding an additional authentication code.
S401322, if the current authentication type is Kerberos, a Kerberos verification code is moved in a deploy Internal and startAppMaster method of the submission type org.
S402, configuring and installing the Flink Client
Through the change and modification, compiling the flink by self, installing the flink in a TBDS Hadoop Client machine, and configuring three installation authentication configurations of TBDS (here, local command execution required by TBDS Hadoop authentication) in environment variables;
additionally compiling jar packets of the flink-shaded dependent TBDS Hadoop version and putting the jar packets into a flink lib directory;
configuring various configuration parameters for authentication of TBDS Hadoop in S401 in a flink-conf.yaml configuration file;
after the flow is completed, the submission of the Flink task can be normally executed, and the integration of the Flink and the TBDS Hadoop is completed.
The submission process is not improved in the present invention, which is the prior art and will not be described herein.
It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.
The embodiment also provides a Flink integration system based on TBDS Hadoop, which is used for implementing the above embodiments and preferred embodiments, and the description of the system is omitted. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. While the system described in the embodiments below is preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.
Fig. 5 is a block diagram of a TBDS Hadoop based Flink integrated system 100 according to an embodiment of the present application, and as shown in fig. 5, the system includes:
the Flink configuration module 1 is used for respectively increasing configuration parameters and compatibly modifying the configuration file and the verification part of the Flink Client;
the software installation module 2 is used for installing a Flink Client on a TBDS Hadoop Client and configuring installation authentication configuration in the environment variable of the TBDS Hadoop Client;
a version jar packet storage step 3, compiling a Flink-cached according to the current version of the TBDS Hadoop to generate a corresponding version jar packet, and storing the version jar packet into a Flinklib directory;
the authentication configuration module 4 is used for configuring the authentication configuration of the TBDS Hadoop in the configuration file;
and the task submitting module 5 is used for submitting the Flink task by the Flink Client according to the steps.
In some embodiments, the Flink configuration module 1 specifically includes:
a configuration item parameter configuration unit 11, which adds configuration parameters required by tbds security authentication in the configuration file, and adds corresponding configuration item variables in the document and security options classes;
the authentication type determining method adding unit 12 adds an isTBDS method and an isKerberos method to the HadoopUtils class to determine the authentication type.
In some embodiments, the configuration item parameter configuration unit 11 specifically includes:
three configuration parameters required by TBDS security authentication are added in the configuration file, TBDS security configuration item constants are added in the Documentation class, the TBDS security configuration item constants correspond to the configuration parameters, TBDS security configuration item variables required by TBDS are added in the SecurityOptions class, and variable values of the corresponding configuration parameters are read according to the TBDS security configuration item variables.
The configuration parameters are set as an authentication user name, an authentication user security period and a security authentication authorization code.
In some embodiments, the authentication configuration module 4 specifically includes:
an authentication type determination unit 41 that determines an authentication type according to the isTBDS method and the isKerberos method;
the TBDS security authentication unit 42 enters normal authentication when judging that the current authentication type is TBDS, and adds configuration parameters in the configuration file into Hadoop configuration obtained by the FlinkClient;
and the Kerberos security authentication unit 43 enters security authentication when the authentication type is Kerberos, and verifies the Kerberos verification code through a Hadoop Module authentication class and a YarnCluster Descriptor submission class.
In practical applications, the TBDS security authentication unit 42 finally adds the judgment of the isTBDS method to the gethadoop configuration method in the hadoopults class, and adds the configuration parameters to the current Hadoop configuration when the authentication type is TBDS.
The Kerberos security authentication unit 43 executes related authentication in the install method of the HadoopModule authentication class, and verifies a part of code of the Kerberos;
in the process of FlinkClient executing Flink task submission, a DeployInternal method and a startAppMaster method of a YarnClusterDescriptor submission type are adopted to verify a Kerberos partial code.
The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A Flink integration method based on TBDS Hadoop is characterized by comprising the following steps:
a step of Flink configuration, which is to respectively increase configuration parameters and compatibly modify the configuration file and the verification part of the Flink Client;
a software installation step, namely installing the Flink Client on a TBDS Hadoop Client, and configuring TBDS installation authentication configuration in the environment variable of the TBDS Hadoop Client;
compiling a Flink-cached according to the current version of the TBDS Hadoop to generate a corresponding version jar packet, and storing the version jar packet into a Flink lib directory;
an authentication configuration step, in which authentication configuration of TBDS Hadoop is configured in the configuration file;
and a task submitting step, wherein the Flink Client submits the Flink task according to the steps.
2. The method for integrating Flink based on TBDS Hadoop as claimed in claim 1, wherein the Flink configuring step specifically comprises:
a configuration item parameter configuration step, namely adding the configuration parameters required by tbds security authentication in the configuration file, and adding corresponding configuration item variables in the document and security options;
and adding an isTBDS method and an isKerberos method in the HadoopUtils class for judging the authentication type.
3. The TBDS Hadoop-based Flink integration method according to claim 2, wherein the configuration item parameter configuration step specifically comprises:
and adding three configuration parameters required by TBDS security authentication in the configuration file, adding a TBDS security configuration item constant in the Documentation class, wherein the TBDS security configuration item constant corresponds to the configuration parameters, adding a TBDS security configuration item variable required by the TBDS in the SecurityOptions class, and reading a variable value corresponding to the configuration parameters according to the TBDS security configuration item variable.
4. The TBDS Hadoop based Flink integration method of claim 2, wherein the authentication configuration step specifically comprises:
judging the authentication type, namely judging the authentication type according to the isTBDS method and the isKerberos method;
a TBDS security authentication step, namely entering normal authentication when the current authentication type is judged to be the TBDS, and adding the configuration parameters in the configuration file into a Hadoop configuration obtained by the Flank Client;
and a Kerberos security authentication step, when the authentication type is Kerberos, entering security authentication, and verifying a Kerberos verification code through a Hadoop Module authentication class and a YarnCluster Descriptor submission class.
5. The TBDS Hadoop-based Flink integration method as claimed in claim 4, wherein the TBDS security authentication step specifically comprises the following steps:
and finally, judging the isTBDS method in a getHadoop configuration method in a Hadoop Utils class, and adding the configuration parameters into the current Hadoop configuration when the authentication type is TBDS.
6. The TBDS Hadoop-based Flink integration method according to claim 4, wherein the Kerberos security authentication step comprises:
when the current authentication type is judged to be Kerberos, executing related authentication in an install method of a HadoopModule authentication type, and verifying a partial code of the Kerberos;
in the process of the Flink Client executing the Flink task submission, a DeltoInternal method and a startAppMaster method of a YarnClusterDescriptor submission type are adopted to verify a Kerberos partial code.
7. A Flink integration system based on TBDS Hadoop, applying the method of claims 1-6, comprising:
the Flink configuration module is used for respectively increasing configuration parameters and compatibly modifying the configuration file and the verification part of the Flink Client;
the software installation module is used for installing the Flink Client on a TBDS Hadoop Client and configuring installation authentication configuration in the environment variable of the TBDS Hadoop Client;
compiling a Flink-cached according to the current version of the TBDS Hadoop to generate a corresponding version jar packet, and storing the version jar packet into a Flink lib directory;
the authentication configuration module is used for configuring the authentication configuration of the TBDS Hadoop in the configuration file;
and the task submitting module is used for submitting the Flink task by the Flink Client according to the steps.
8. The TBDS Hadoop-based Flink integrated system according to claim 7, wherein the Flink configuration module specifically comprises:
a configuration item parameter configuration unit, which adds the configuration parameters required by tbds security authentication in the configuration file, and adds the corresponding configuration item variables in the document and security options;
and the authentication type judging method adding unit is used for adding an isTBDS method and an isKerberos method in the HadoopUtils class and judging the authentication type.
9. The TBDS Hadoop-based Flink integrated system according to claim 8, wherein the configuration item parameter configuration unit specifically comprises:
and adding three configuration parameters required by TBDS security authentication in the configuration file, adding a TBDS security configuration item constant in the Documentation class, wherein the TBDS security configuration item constant corresponds to the configuration parameters, adding a TBDS security configuration item variable required by the TBDS in the SecurityOptions class, and reading a variable value corresponding to the configuration parameters according to the TBDS security configuration item variable.
10. The TBDS Hadoop-based Flink integrated system according to claim 8, wherein the authentication configuration module specifically comprises:
the authentication type judging unit judges the authentication type according to the isTBDS method and the isKerberos method;
the TBDS security authentication unit enters common authentication when judging that the current authentication type is the TBDS, and adds the configuration parameters in the configuration file into Hadoop configuration obtained by the Flank Client;
and the Kerberos security authentication unit enters security authentication when the authentication type is Kerberos, and verifies the Kerberos verification code through a Hadoop Module authentication class and a YarnCluster Descriptor submission class.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011549744.0A CN112558995A (en) | 2020-12-24 | 2020-12-24 | Flink integration method and system based on TBDS Hadoop |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011549744.0A CN112558995A (en) | 2020-12-24 | 2020-12-24 | Flink integration method and system based on TBDS Hadoop |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112558995A true CN112558995A (en) | 2021-03-26 |
Family
ID=75033316
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011549744.0A Pending CN112558995A (en) | 2020-12-24 | 2020-12-24 | Flink integration method and system based on TBDS Hadoop |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112558995A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254106A (en) * | 2021-06-23 | 2021-08-13 | 中国平安人寿保险股份有限公司 | Task execution method and device based on Flink, computer equipment and storage medium |
| CN114816583A (en) * | 2022-05-31 | 2022-07-29 | 以萨技术股份有限公司 | Flink-based data automatic processing method and device and electronic equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109840253A (en) * | 2019-01-10 | 2019-06-04 | 北京工业大学 | Enterprise-level big data platform framework |
| CN110109680A (en) * | 2019-05-14 | 2019-08-09 | 重庆商勤科技有限公司 | Using dispositions method, device and apply dissemination method, server, storage medium |
| CN110297859A (en) * | 2019-06-10 | 2019-10-01 | 恩亿科(北京)数据科技有限公司 | The method and apparatus for handling real time data |
| CN110874272A (en) * | 2020-01-16 | 2020-03-10 | 北京懿医云科技有限公司 | Resource allocation method and device, computer readable storage medium and electronic device |
| US10623281B1 (en) * | 2017-04-18 | 2020-04-14 | EMC IP Holding Company LLC | Dynamically scheduled checkpoints in distributed data streaming system |
| CN111368165A (en) * | 2020-02-25 | 2020-07-03 | 江苏物联网研究发展中心 | Spatio-temporal streaming data integration platform |
-
2020
- 2020-12-24 CN CN202011549744.0A patent/CN112558995A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10623281B1 (en) * | 2017-04-18 | 2020-04-14 | EMC IP Holding Company LLC | Dynamically scheduled checkpoints in distributed data streaming system |
| CN109840253A (en) * | 2019-01-10 | 2019-06-04 | 北京工业大学 | Enterprise-level big data platform framework |
| CN110109680A (en) * | 2019-05-14 | 2019-08-09 | 重庆商勤科技有限公司 | Using dispositions method, device and apply dissemination method, server, storage medium |
| CN110297859A (en) * | 2019-06-10 | 2019-10-01 | 恩亿科(北京)数据科技有限公司 | The method and apparatus for handling real time data |
| CN110874272A (en) * | 2020-01-16 | 2020-03-10 | 北京懿医云科技有限公司 | Resource allocation method and device, computer readable storage medium and electronic device |
| CN111368165A (en) * | 2020-02-25 | 2020-07-03 | 江苏物联网研究发展中心 | Spatio-temporal streaming data integration platform |
Non-Patent Citations (2)
| Title |
|---|
| FRANCESCO VERSACI等: "Kafka interfaces for composable streaming genomics pipelines", 《2018 IEEE EMBS INTERNATIONAL CONFERENCE ON BIOMEDICAL & HEALTH INFORMATICS (BHI)》, 9 April 2018 (2018-04-09), pages 259 * |
| 张庄: "工业大数据分析建模平台研究与实现", 《CNKI优秀硕士学位论文全文库 信息科技辑》, no. 05, 15 May 2021 (2021-05-15), pages 138 - 562 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254106A (en) * | 2021-06-23 | 2021-08-13 | 中国平安人寿保险股份有限公司 | Task execution method and device based on Flink, computer equipment and storage medium |
| CN113254106B (en) * | 2021-06-23 | 2022-04-19 | 中国平安人寿保险股份有限公司 | Task execution method and device based on Flink, computer equipment and storage medium |
| CN114816583A (en) * | 2022-05-31 | 2022-07-29 | 以萨技术股份有限公司 | Flink-based data automatic processing method and device and electronic equipment |
| CN114816583B (en) * | 2022-05-31 | 2024-03-19 | 以萨技术股份有限公司 | Flink-based data automatic processing method and device and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10482257B2 (en) | System and method to enforce the secure boot policy of a platform on a virtual machine | |
| Brittain et al. | Tomcat: The Definitive Guide: The Definitive Guide | |
| CN108681662B (en) | Method and device for installing program | |
| EP3345112B1 (en) | Thresholds on scripts executable by unified extensible firmware interface systems | |
| CN112558995A (en) | Flink integration method and system based on TBDS Hadoop | |
| EP3163489B1 (en) | Token-based control of software installation and operation | |
| US10148440B2 (en) | Binary code authentication | |
| CN103368927B (en) | A kind of security configuration inspecting device and method | |
| CN114064155A (en) | Container-based algorithm calling method, device, equipment and storage medium | |
| CN114385195A (en) | Third-party platform deployment method, third-party platform deployment device, computer equipment and medium | |
| RU2357287C2 (en) | Safe identification of executable file for logical object determining confidence | |
| CN113239308A (en) | Page access method, device, equipment and storage medium | |
| US20230041769A1 (en) | Management system for disk encryption | |
| CN112788017B (en) | Security verification method, device, equipment and medium | |
| CN112364340B (en) | Authority management method, device, equipment and computer readable storage medium | |
| Titze et al. | Preventing library spoofing on android | |
| US20210103651A1 (en) | Registration of dynamically created packaged applications | |
| CN112528286A (en) | Terminal device security detection method, associated device and computer program product | |
| CN111475763A (en) | Webpage running method and device, storage medium and equipment | |
| CN111782230B (en) | Program installation control method and device and electronic equipment | |
| JP7380251B2 (en) | Method and device for monitoring data tampering in virtual machines | |
| Olaoye | Continuous Delivery Pipelines for iOS App Development | |
| Badura et al. | Testing the symbian OS platform security architecture | |
| CN115982697A (en) | Sandbox access method, sandbox system and computer-readable storage medium | |
| Matotek et al. | Configuration Management: By James Turnbull and Dennis Matotek |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |