CN113778392A

CN113778392A - Method and device for realizing streaming development based on configuration

Info

Publication number: CN113778392A
Application number: CN202110287623.1A
Authority: CN
Inventors: 王杰颖
Original assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Current assignee: Beijing Jingdong Century Trading Co Ltd; Beijing Wodong Tianjun Information Technology Co Ltd
Priority date: 2021-03-17
Filing date: 2021-03-17
Publication date: 2021-12-10

Abstract

The invention discloses a method and a device for realizing streaming development based on configuration, and relates to the technical field of computers. One embodiment of the method comprises: acquiring a configuration file generated based on a framework of a data stream type development framework; analyzing the configuration file to obtain the development flow of the application and the operation of the module related to the development flow; and executing the operation of the module related to the development flow according to the development flow to complete the streaming application development. The implementation mode realizes the rapid access of the modular configuration development of the streaming application, the uniform analysis process of the streaming architecture and the modular access; the service flow is clearer and clearer, the code can be repeatedly used according to the configuration, and the service adjustment is simple.

Description

Method and device for realizing streaming development based on configuration

Technical Field

The invention relates to the technical field of computers, in particular to a method and a device for realizing streaming development based on configuration.

Background

With the change of the current life style of people, people pay more and more attention to the acquisition and development of real-time data, and the real-time streaming development also receives more and more attention from people. At present, mainstream technologies for realizing Streaming development mainly include Flink, Storm, Spark Streaming and the like, and for specific real-time data indexes, according to technical module requirements, corresponding process development is realized, and the deployment of the whole process is completed. The structure of the streaming framework is fixed, and each time the streaming framework is used for development, analysis is required to be completed according to the corresponding specification, and task information is submitted. For example: the access development mode of the streaming framework Flink mainly comprises a Source layer, a Mapper layer and a Sink layer, and the data access development needs to access according to the three layers of frameworks to construct a Flink operating environment and finally submit an operating task job.

In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:

the existing scheme for performing demand development by using a streaming framework needs to be developed according to the framework requirement of the streaming framework, a streaming framework analysis flow needs to be written repeatedly, each service analysis chain cannot quickly preview in codes, and when a service is modified or a processing sequence is changed, the codes need to be adjusted correspondingly, so that errors are easy to occur.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for implementing streaming development based on configuration, which can implement fast access, a unified streaming architecture parsing process, and modular access for modular configuration development of streaming application; the service flow is clearer and clearer, the code can be repeatedly used according to the configuration, and the service adjustment is simple.

To achieve the above object, according to an aspect of an embodiment of the present invention, a method for implementing streaming development based on configuration is provided.

A method for implementing streaming development based on configuration, comprising:

acquiring a configuration file generated based on a framework of a data stream type development framework;

analyzing the configuration file to obtain the development flow of the application and the operation of the module related to the development flow;

and executing the operation of the module related to the development flow according to the development flow so as to complete the development of the streaming application.

Optionally, the configuration file is an extensible markup language configuration file, and the development process and the module related to the development process are configured in the form of a tag.

Optionally, the label of the development flow includes common parameter information of a data streaming development framework, and attribute information and execution order of modules involved in the development flow;

the label of the module related to the development process comprises a data source access label, a data conversion operation label and a calculation result summary label.

Optionally, the field attribute of the data source access tag includes:

the label unique identification, the name of the data source, the type of the data source, the operation attribute parameter information of the data flow development framework and the operation implementation class.

Optionally, the field attribute of the data conversion operation tag includes:

the label unique identification, the name of a data conversion operator, the operation attribute parameter information of the data flow development framework and the operation implementation class.

Optionally, the field attribute of the computation result summary tag includes:

the label unique identification, the result summarizing operation name, the result summarizing operation type, the operation attribute parameter information of the data flow type development framework and the operation implementation class.

Optionally, executing, according to the development flow, the operation of the module involved in the development flow to complete the streaming application development includes:

setting public parameter information of a data stream type development framework according to the label of the development process;

acquiring a label of the module according to the attribute information of the module related to the development process;

and analyzing the acquired label of the module and executing an operation implementation class according to the execution sequence of the module related to the development flow so as to complete the streaming application development.

According to another aspect of the embodiments of the present invention, an apparatus for implementing streaming development based on configuration is provided.

An apparatus for implementing streaming development based on configuration, comprising:

the configuration acquisition module is used for acquiring a configuration file generated based on the architecture of the data stream type development framework;

the configuration analysis module is used for analyzing the configuration file to obtain the development flow of the application and the operation of the module related to the development flow;

and the operation execution module is used for executing the operation of the module related to the development flow according to the development flow so as to complete the development of the streaming application.

Optionally, the field attribute of the data source access tag includes:

Optionally, the field attribute of the data conversion operation tag includes:

Optionally, the field attribute of the computation result summary tag includes:

Optionally, the operation execution module is further configured to:

According to another aspect of the embodiments of the present invention, an electronic device for implementing streaming development based on configuration is provided.

An electronic device that implements streaming development based on configuration, comprising: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for implementing streaming development based on configuration, which is provided by the embodiment of the invention.

According to yet another aspect of embodiments of the present invention, a computer-readable medium is provided.

A computer readable medium, on which a computer program is stored, the program, when executed by a processor, implements a method for implementing streaming development based on configuration, provided by an embodiment of the present invention.

One embodiment of the above invention has the following advantages or benefits: acquiring a configuration file generated based on the framework of the data streaming development framework; analyzing the configuration file to obtain the development flow of the application and the operation of the module related to the development flow; executing the operation of the module related to the development flow according to the development flow to complete the stream type application development, realizing the adoption of a stream type processing modularization management mode, generating an xml configuration file according to the framework of a stream type development framework, defining related configurations such as class types and the like required to be processed at each stage, and integrating the whole stream type execution chain to generate a stream type development flow; analyzing the xml configuration file to obtain the development flow of the application and the operation of the module related to the development flow; and finally, executing the class according to the development flow to complete the streaming application development. The service logics corresponding to the streaming type hierarchy modules can be freely combined to complete configuration information, so that the fast access of the modular configuration development of streaming type application, the uniform streaming type architecture analysis process and the modular access are realized; the whole real-time chain is realized in xml configuration, and the service flow is clearer and clearer; corresponding to source, mapper and sink layers, modular development is carried out, and codes can be reused according to configuration; the service adjustment is simple, and the xml configuration information is directly adjusted to realize the change of service logic processing.

Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.

Drawings

The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:

FIG. 1 is a schematic diagram of the main steps of a method for implementing streaming development based on configuration according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of the main modules of an apparatus for implementing streaming development based on configuration according to an embodiment of the present invention;

FIG. 3 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;

fig. 4 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.

Detailed Description

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the prior art, when streaming application development is performed, corresponding code development is performed on each layer included in a framework directly according to a streaming development framework, so that each time of demand development needs to write codes according to the framework requirements of the streaming application, the development is relatively trivial, the whole streaming framework access mode needs to be repeated, when a service is modified or a processing sequence is changed, the codes need to be correspondingly adjusted, and errors are easy to occur.

In order to solve the technical problems in the prior art, the invention provides a device for realizing a method for stream-oriented development based on configuration, which can realize the development and analysis of the automatic configuration of real-time stream-oriented application. The following describes a specific implementation process of the embodiment of the present invention with reference to the drawings.

Fig. 1 is a schematic diagram of main steps of a method for implementing streaming development based on configuration according to an embodiment of the present invention. As shown in fig. 1, the method for implementing streaming development based on configuration according to the embodiment of the present invention mainly includes the following steps S101 to S103.

Step S101: a configuration file generated based on the architecture of the data streaming development framework is obtained. According to an embodiment of the present invention, the configuration file is an XML configuration file (configuration files in other formats may be used according to the application), and the configuration file includes the development flow of the streaming development framework and the modules involved in the development flow. The configuration file may be generated according to the architecture of a streaming development framework for the data. For example: taking a streaming development framework Flink as an example, the framework Flink comprises a Source layer, a Mapper layer and a Sink layer, so that each layer can be individually configured as a module when a configuration file is generated, and development processes of definitions, execution sequences, execution rules and the like of the modules are also configured, that is: the configuration file comprises a development flow streaming of the streaming development framework and a module Source layer, a Mapper layer and a Sink layer related to the development flow. In the embodiment of the invention, the configuration file is set as the flow management template of the streaming development framework in the XML format, so that the modules related to the development flow and the development flow are configured in the form of tags, and the configuration file needs to comprise four tags, namely streaming, source, mapper and sink.

According to the embodiment of the invention, taking a streaming development framework Flink as an example, a label streaming label of a development flow comprises common parameter information of a data streaming development framework, and attribute information and an execution sequence of a module related to the development flow; the label of the module related to the development process comprises a data source access label, a data conversion operation label and a calculation result summary label.

Specifically, according to an embodiment of the present invention, the field attribute of the data source access tag source tag includes: the label is unique identification, the name of the data source, the type of the data source, the operation attribute parameter information of the data flow development framework, the operation implementation class and the like. The source tag, the data source access layer and the xml configuration file can contain different data source tag information, and the tag contains field attributes such as id, name, type, class, slotsharringgroup and parallelisms:

the id field represents the unique identification of the label, and different data sources set unique id information;

the name field represents the name of the source data source;

the type keyword is used for setting data source types, such as hbase, kafaka, text and the like, a property attribute can be set in a source tag to define an independent variable attribute value, for example, the type is a kafaka type, a topic name, a server address and the like can be set, different topic information can be conveniently accessed, and code sharing is realized;

the fields of slotSharingGroup, parallelism and the like are the running attribute parameter information of the flink, are used for dynamically setting the packet parallelism, can not be set, can not set default using the default packet of the flink, and can also increase other public parameter information of the flink according to the requirement. Wherein, the slotsharringgroup is a class used for realizing slot sharing in the Flink, and the slotsks share one slot as much as possible; parallelisms are the number of parallel instances of a task; other flink common parameter information such as: maximum concurrency number maxparalleisms, maximum system delay buffertioutcontrolling Flink at low flow, and the like;

the class field is a specific operation implementation class, encapsulates information for realizing data source processing, acquires data stream DataStream information of the flash according to all parameter information in the source tag and the type of the data source and an analysis access mode, and completes the access of the source data source.

According to the embodiment of the invention, the field attribute of the mapper tag of the data conversion operation tag comprises the following steps: the label unique identification, the name of a data conversion operator, the operation attribute parameter information of the data flow development framework, the operation implementation class and the like. The mapper tag and the Flink DataStream streaming data conversion operator (all operations on data are called as an operator) complete business logic calculation, and can realize the conversion of business streams in the modes of Map, Flatmap, Filter, KeyBy, Reduce, Fold, Aggregate and the like according to business requirements. The mapper tag is a tag defined in xml, and the specific format is, for example: the "mapper id ═ mapper1" name ═ mapper inject1"class ═ the executed class name" slotsharing group ═ "222" parallelism ═ 10 "/>. The mapper layer operates primarily on the streamwise acquired data, such as:

and Map: one-to-one conversion, i.e., one to another;

FlatMap: one-to-many, the data format can be changed;

a Filter: filtering out required data according to conditions;

KeyBy: and data is re-partitioned according to the appointed Key. Putting data of the same Key into the same partition;

reduce: performing rolling aggregation based on the reduce function, and outputting a result after each rolling aggregation to a downstream operator;

fold: performing rolling folding based on the initial value and the FoldFunction, and outputting a result after each rolling folding to a downstream operator;

aggregate: and performing rolling aggregation on the KeyedStream according to the specified field and outputting the result after each rolling aggregation. The default aggregation functions are: sum, min, minBy, max, mabBy, and the like.

The mapper tag can be set with the field attributes of id, name, class, slotsharringgroup, parallelisms, etc.:

the id field represents the unique identification of the label, and different operators set unique id information;

the name field represents the name of the data conversion operator;

the slotsharringgroup and parallelisms fields are the running attribute parameter information of the flink, are used for dynamically setting the packet parallelism, can not set default using the default packet of the flink, and can also increase other public parameter information of the flink according to the requirement. Other flink common parameter information such as: maximum concurrency number maxparalleisms, system maximum delay buffertioutcontrolling Flink at low flow, operator ID, and the like;

the class field is a specific operation implementation class, the encapsulation realizes the processing information of the service logic, the conversion of the flash data stream is realized according to different operators, and the stream calculation of the whole mapper layer is completed.

According to the embodiment of the invention, the field attributes of the calculation result summary tag sink tag comprise: the label unique identification, the result summarizing operation name, the result summarizing operation type, the operation attribute parameter information of the data flow type development framework and the operation implementation class. And the sink label is used for searching the calculated Data result to a certain place through Data source access and calculation of a Data conversion operator, and the label is mainly used for butt joint with the Data sink of the Flink. The sink tag is for example of the form: the "sink id ═ sink" name ═ sink "class ═ executed class name" slotSharingGroup ═ "333" parallelisms ═ "5"/>, which corresponds to the link where the final result falls after the data calculation is completed. The sink tag defines field attributes such as id, name, class, type, slotsharringgroup, parallelisms and the like:

the id field represents the unique identification of the tag, and different sink sets unique id information;

the name field represents the name of the sink;

type keywords are used to set sink types, such as hbase, kafaka, redis, text, and the like. Property attributes can be set in the sink tags to define independent variable attribute values, for example, the type is kafaka type, and topic names, server addresses and the like can be set, so that different topic information can be conveniently accessed, and code sharing is realized;

the slotsharringgroup and parallelism fields are flink running attribute parameter information and are used for dynamically setting the packet parallelism, and the default flank default packet can be used without setting or setting. Other flink public parameter information can be added according to the requirement;

the class field is a specific operation implementation class, different sink processing information is encapsulated, all parameter information in the sink tag is obtained, an access mode is analyzed based on the type of the data source, and finally the output of the data result is achieved.

According to the embodiment of the invention, the label streaming label of the development flow comprises common parameter information of a data streaming development framework, attribute information and execution sequence of modules involved in the development flow, and the like. And (3) streaming tag, configuring source, mapper and sink information for the processing flow of the tag flash, and completing the whole streaming calculation. Reference patterns of streaming tags are for example: < streaming name ═ stream _ chain "source ═ source id1

< | A! Setting a common parameter in the stram stream >

</streaming>>。

The streaming tag contains attributes of name, source, mapper and sink, and meanwhile, attribute information of < property > can be configured in the tag:

the source attribute is an id value of a configured source tag, and the id value is a unique attribute and can be used for directly finding data source information correspondingly;

the mapper attribute is an id value of a configured mapper label, the id value is a unique attribute, and operator calculation logic information is directly found correspondingly. When a plurality of mapper processes exist, commas are used for separating, and the sequence is executed from left to right;

the Sink attribute is an id value of the configured Sink tag, the id value is a unique attribute, and a processing logic for Data Sink Data output is correspondingly and directly found;

name is the name of the whole stream;

the property attribute configured under the streaming label can set parameter information common to the flash, such as checkpoint, to complete checkpoint operation, and set checkpoint time label, and when the streaming label is analyzed, the parameter is obtained. The Checkpoint is the most core function of the Flink implementation fault-tolerant mechanism, and can periodically generate snapshots based on the states of various operations in the Stream according to configuration, so that the state data is periodically stored persistently, and when the Flink program crashes unexpectedly, the Flink program can be selectively recovered from the snapshots when being rerun, so that the state interruption of the program data caused by faults is corrected; checkpoint time refers to the cycle time that the snapshot was generated.

In the configuration file, the corresponding tags such as source, mapper, sink, and the like are developed in a modular manner, and the streaming tags can be configured in a combined manner according to the service requirements corresponding to the classes configured in the xml configuration file.

Step S102: and analyzing the configuration file to obtain the development flow of the application and the operation of the module related to the development flow. By analyzing the XML configuration file, the corresponding development process can be obtained according to the streaming tag, and the operation of the module related to the development process can be obtained according to the tags such as source, mapper and sink.

Step S103: and executing the operation of the module related to the development flow according to the development flow to complete the streaming application development. According to an embodiment of the present invention, the step may specifically include:

setting public parameter information of a data stream type development framework according to a label of a development process;

According to the embodiment of the invention, the streaming tag defines source, map and sink tags, and the Flink frame is executed based on the sequence of source → map → sink during the program running process. Therefore, in the embodiment of the present invention, the process of executing the operation of the module involved in the development flow according to the development flow to complete the development of the streaming application is as follows:

firstly, acquiring a running environment streamExecutionEnvironment of a cluster, and setting cluster public parameter information according to a property attribute configured in a streaming tag. At the beginning of the running of the Flink, firstly, a running environment needs to be acquired, a corresponding calling method provided for the Flink can be directly used, and after the running environment is acquired, the contents in source, mapper and sink are sequentially analyzed, and the whole task submission is completed;

secondly, acquiring the configuration information of the source tag, executing class to acquire data stream information DataStream, and configuring information such as name parallelism, grouping name and the like according to the attribute information configured in the source tag;

thirdly, analyzing the information of the mapper label, executing class, acquiring a corresponding data stream, finishing the interaction of an operator layer, and configuring information such as name parallelism, grouping names and the like according to attribute information configured in the source label;

and finally, analyzing and executing the corresponding calls class according to the sink configuration file information, submitting the job task, and completing the whole task release.

In the above embodiment, a Flink development process is mainly used as an example for description, and stream type job submission is completed based on three dimensions of source, mapper and sink. The streaming development framework from can also be realized according to the xml configuration file, by setting topology information of storm, setting spout and bolt chains, and finally executing submittoplogy according to the from streaming development specification, thereby submitting the whole storm topology information. Similarly, for other data streaming development frameworks, the fast and real-time development of streaming applications can be realized according to a modular configuration method.

The embodiment of the invention mainly realizes the development and analysis of the automatic configuration real-time streaming application. Generating an xml configuration file according to the framework of a stream type development framework by adopting a stream type processing modular management mode, defining related configurations such as class types to be processed at each stage, and integrating a whole stream type execution chain to generate a stream type development flow; analyzing the xml configuration file to obtain the development flow of the application and the operation of the module related to the development flow; and finally, executing the class according to the development flow to complete the streaming application development.

According to the technical scheme of the invention, the service logics corresponding to the streaming type hierarchy modules can be freely combined to complete configuration information, and the rapid access of the modular configuration development of the streaming type application is realized.

Fig. 2 is a schematic diagram of main modules of an apparatus for implementing streaming development based on configuration according to an embodiment of the present invention. As shown in fig. 2, an apparatus 200 for implementing streaming development based on configuration according to an embodiment of the present invention mainly includes a configuration obtaining module 201, a configuration parsing module 202, and an operation executing module 203.

A configuration acquisition module 201, configured to acquire a configuration file generated based on a framework of a data streaming development framework;

a configuration analysis module 202, configured to analyze the configuration file, and obtain a development flow of an application and an operation of a module related to the development flow;

and the operation execution module 203 is used for executing the operation of the module related to the development flow according to the development flow so as to complete the streaming application development.

According to one embodiment of the invention, the configuration file is an extensible markup language configuration file, and the development process and the modules involved in the development process are configured in the form of tags.

According to another embodiment of the invention, the label of the development flow comprises common parameter information of a data streaming development framework, and attribute information and execution sequence of the modules involved in the development flow; the label of the module related to the development process comprises a data source access label, a data conversion operation label and a calculation result summary label.

According to another embodiment of the present invention, the field attribute of the data source access tag includes:

According to another embodiment of the present invention, the field attribute of the data conversion operation tag includes:

According to another embodiment of the present invention, the field attributes of the computation result summary label include:

According to yet another embodiment of the present invention, the operation executing module 203 is further configured to:

According to the technical scheme of the embodiment of the invention, the configuration file generated by the framework based on the data flow development framework is obtained; analyzing the configuration file to obtain the development flow of the application and the operation of the module related to the development flow; executing the operation of the module related to the development flow according to the development flow to complete the stream type application development, realizing the adoption of a stream type processing modularization management mode, generating an xml configuration file according to the framework of a stream type development framework, defining related configurations such as class types and the like required to be processed at each stage, and integrating the whole stream type execution chain to generate a stream type development flow; analyzing the xml configuration file to obtain the development flow of the application and the operation of the module related to the development flow; and finally, executing the class according to the development flow to complete the streaming application development. The service logics corresponding to the streaming type hierarchy modules can be freely combined to complete configuration information, so that the fast access of the modular configuration development of streaming type application, the uniform streaming type architecture analysis process and the modular access are realized; the whole real-time chain is realized in xml configuration, and the service flow is clearer and clearer; corresponding to source, mapper and sink layers, modular development is carried out, and codes can be reused according to configuration; the service adjustment is simple, and the xml configuration information is directly adjusted to realize the change of service logic processing.

Fig. 3 illustrates an exemplary system architecture 300 for a method for implementing streaming development based on configuration or an apparatus for implementing streaming development based on configuration, to which embodiments of the present invention may be applied.

As shown in fig. 3, the system architecture 300 may include

terminal devices

301, 302, 303, a network 304, and a server 305. The network 304 serves as a medium for providing communication links between the

terminal devices

301, 302, 303 and the server 305. Network 304 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal device

301, 302, 303 to interact with the server 305 via the network 304 to receive or send messages or the like. The

terminal devices

301, 302, 303 may have installed thereon various communication client applications, such as a program development class application, an application development class application, a streaming development framework, an instant messaging tool, social platform software, etc. (by way of example only).

The

terminal devices

301, 302, 303 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 305 may be a server providing various services, such as a background management server (for example only) providing support for streaming application development requests issued by users using the

terminal devices

301, 302, 303. The background management server can process the received data such as the streaming application development request and the like to acquire a configuration file generated based on the framework of the data streaming development framework; analyzing the configuration file to obtain the development flow of the application and the operation of the module related to the development flow; and executing the processing such as the operation of the module related to the development flow according to the development flow, and feeding back the processing result (such as the developed streaming application-only an example) to the terminal equipment.

It should be noted that the method for implementing streaming development based on configuration provided by the embodiment of the present invention is generally executed by the server 305, and accordingly, an apparatus for implementing streaming development based on configuration is generally disposed in the server 305.

It should be understood that the number of terminal devices, networks, and servers in fig. 3 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring now to FIG. 4, a block diagram of a computer system 400 suitable for use with a terminal device or server implementing an embodiment of the invention is shown. The terminal device or the server shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.

As shown in fig. 4, the computer system 400 includes a Central Processing Unit (CPU)401 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)402 or a program loaded from a storage section 408 into a Random Access Memory (RAM) 403. In the RAM 403, various programs and data necessary for the operation of the system 400 are also stored. The CPU 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.

The following components are connected to the I/O interface 405: an input section 406 including a keyboard, a mouse, and the like; an output section 407 including a display device such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 408 including a hard disk and the like; and a communication section 409 including a network interface card such as a LAN card, a modem, or the like. The communication section 409 performs communication processing via a network such as the internet. A driver 410 is also connected to the I/O interface 405 as needed. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 410 as necessary, so that a computer program read out therefrom is mounted into the storage section 408 as necessary.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 409, and/or installed from the removable medium 411. The computer program performs the above-described functions defined in the system of the present invention when executed by a Central Processing Unit (CPU) 401.

It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units or modules described in the embodiments of the present invention may be implemented by software, or may be implemented by hardware. The described units or modules may also be provided in a processor, and may be described as: a processor includes a configuration acquisition module, a configuration parsing module, and an operation execution module. Where the names of these units or modules do not in some cases constitute a limitation on the units or modules themselves, for example, the configuration acquisition module may also be described as a "module for acquiring configuration files generated based on the architecture of the data streaming development framework".

As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: acquiring a configuration file generated based on a framework of a data stream type development framework; analyzing the configuration file to obtain the development flow of the application and the operation of the module related to the development flow; and executing the operation of the module related to the development flow according to the development flow so as to complete the development of the streaming application.

The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for implementing streaming development based on configuration, comprising:

2. The method according to claim 1, wherein the configuration file is an extensible markup language configuration file, and the development process and the module involved in the development process are configured in the form of a tag.

3. The method according to claim 2, wherein the label of the development flow comprises common parameter information of a data streaming development framework, and attribute information and execution sequence of the module involved in the development flow;

4. The method of claim 3, wherein the field attribute of the data source access tag comprises:

5. The method of claim 3, wherein the field attributes of the data transformation operation tag comprise:

6. The method of claim 3, wherein the field attributes of the computation result summary tag comprise:

7. The method of claim 3, wherein performing the operations of the modules involved in the development flow according to the development flow to complete the streaming application development comprises:

8. An apparatus for implementing streaming development based on configuration, comprising:

9. An electronic device that implements streaming development based on configuration, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.