CN114579331A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114579331A CN114579331A CN202210199927.7A CN202210199927A CN114579331A CN 114579331 A CN114579331 A CN 114579331A CN 202210199927 A CN202210199927 A CN 202210199927A CN 114579331 A CN114579331 A CN 114579331A
- Authority
- CN
- China
- Prior art keywords
- data processing
- operator
- plug
- data
- processing operator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/541—Interprogram communication via adapters, e.g. between incompatible applications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44521—Dynamic linking or loading; Link editing at or after load time, e.g. Java class loading
- G06F9/44526—Plug-ins; Add-ons
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
- G06F9/548—Object oriented; Remote method invocation [RMI]
Abstract
An embodiment of the present application provides a data processing method, an apparatus, an electronic device, and a storage medium, where the method is used in a big data processing system, and the big data processing system includes: the method comprises the following steps that the data processing operator and an adapter are used for carrying out corresponding encapsulation adaptation on the data processing operator according to interface standards of different computing platforms, and the method comprises the following steps: determining a data processing operator for generating a data processing task; utilizing an adapter to carry out encapsulation adaptation on the data processing operator so as to enable the data processing operator subjected to encapsulation adaptation to conform to the interface standard of a corresponding computing platform; generating a data processing task according to the data processing operator after encapsulation and adaptation; sending the data processing task to the computing platform to enable the computing platform to process. Based on the above embodiment, the data processing operator can be sent to different computing platforms for processing.
Description
Technical Field
The present application relates to the field of big data technologies, and in particular, to a data processing method, an apparatus, an electronic device, and a computer-readable storage medium.
Background
With the rapid development of internet company business, data is also increasing in a blowout manner, and how to provide easy-to-use, stable and efficient real-time data service for business departments is becoming more and more important, so that business operations based on big data real-time calculation begin to be completely open and are more and more put into online use, such as real-time recommendation, double eleven real-time large-screen statistics, real-time anti-fraud and the like.
Although there are many implementations of current big data processing, the big data processing in the prior art is usually bound to a specific computing platform, which causes inconvenience in use and is not easy to expand.
Disclosure of Invention
An object of the embodiments of the present application is to provide a data processing method, an apparatus, an electronic device, and a storage medium, which can implement processing of big data on different platforms.
In a first aspect, an embodiment of the present application provides a data processing method, where the method is used in a big data processing system, and the big data processing system includes: the method comprises the following steps that the data processing operator and an adapter are used for carrying out corresponding encapsulation adaptation on the data processing operator according to interface standards of different computing platforms, and the method comprises the following steps:
determining a data processing operator for generating a data processing task;
performing encapsulation adaptation on the data processing operator by using an adapter so that the data processing operator after encapsulation adaptation meets the interface standard of a corresponding computing platform;
generating a data processing task according to the data processing operator after encapsulation and adaptation;
and sending the data processing task to the computing platform so that the computing platform processes data according to the data processing task.
In the implementation process, firstly, a data processing operator is determined, the data processing operator has the capacity of processing data, in order to process large-scale data, the data processing operator needs to be sent to a computing platform, the computing platform calls the data processing operator to process the data, different computing platforms have different requirements on the data processing operator, therefore, an adapter is used for packaging and adapting the data processing operator, the data processing operator after packaging and adapting meets the interface requirement of the computing platform on the data processing operator, and a data processing task is generated according to the data processing operator after packaging and adapting; the data processing task is sent to the computing platform to be processed, so that the computing platform can call the data processing operator to process the data.
Further, the big data processing system further comprises: the system comprises a plug-in frame and a plug-in frame interface, wherein the plug-in frame interface has an interface standard;
the performing encapsulation adaptation on the data processing operator by using the adapter includes:
transforming the data processing operator according to the interface standard of the plug-in frame interface to obtain a data processing plug-in;
and driving the plug-in framework to call the data processing plug-in through the plug-in framework interface, and performing encapsulation adaptation on the data processing plug-in through the adapter.
In the implementation process, a plug-in frame and a plug-in frame interface are arranged, the plug-in frame interface has a corresponding interface standard of the plug-in frame, a data processing operator is modified according to the interface standard to obtain a data processing plug-in, the plug-in frame can call the data processing plug-in through the interface, and the data plug-in is packaged and adapted through an adapter. Based on the embodiment, the automatic encapsulation and adaptation of the data processing operator can be realized based on the frame, and the encapsulation and adaptation efficiency is improved.
Further, the following constraints are not included in the interface standard of the plug-in framework:
constraints on the input data format and the output data format of the data processing operator.
In the implementation process, the interface standard does not include constraints on the input data type and the output data type of the interface, so that the flexibility of the data processing operator is improved, the coupling degree of the data processing operator and the plug-in frame is reduced, and the reusability of the data processing operator is improved.
Further, after the data processing operator is encapsulated and adapted, the method further includes:
and the drive plug-in framework obtains configuration parameters, and the configuration parameters are used for initializing the data processing operator.
In the implementation process, the data processing operator is initialized by acquiring the configuration parameters, so that the data processing operator can adapt to different data, and the reusability of the data processing operator is further improved.
Further, the generating a data processing task according to the data processing operator after the encapsulation adaptation includes:
generating a directed acyclic graph by taking the data processing operator as a node;
and generating the data processing task according to the directed acyclic graph.
In the implementation process, the directed acyclic graph is generated firstly, and then the data processing task is generated according to the directed acyclic graph, so that the computing platform can call a data processing operator according to the sequence in the directed acyclic graph to process the data, and the processing efficiency of the computing platform is accelerated.
Further, the big data processing system further comprises: the data processing engine calls the data processing operator through an interface of the data processing operator;
the generating of the directed acyclic graph by using the data processing operator as the node comprises:
determining a connection order between data processing operators for generating data processing tasks;
and driving the data processing engine, and generating the directed acyclic graph by taking each data processing operator as a node according to the connection sequence, so that the computing platform calls the data processing operators to process the data according to the connection sequence of the directed acyclic graph.
In the implementation process, the data processing engine calls the data processing operators through interfaces of the data processing operators, and generates the directed acyclic graph by taking each data processing operator as a node according to the connection sequence, so that each data processing operator can be prevented from being manually operated in the process of generating the directed acyclic graph, and the efficiency of generating the directed acyclic graph is improved.
Further, the generating the data processing task according to the directed acyclic graph includes:
adding an input plug-in before a starting node of the directed acyclic graph, and adding an output plug-in after an ending node of the directed acyclic graph to obtain the data processing task, wherein the input plug-in is used for inputting the data into the directed acyclic graph, and the output plug-in is used for outputting the processed data out of the directed acyclic graph.
In the implementation process, an output plug-in and an input plug-in are arranged on the basis of the directed acyclic graph, the output plug-in is used for inputting data into the directed acyclic graph, and the output plug-in is used for outputting the processed data. Based on the embodiment, the computing platform can directly process and output the data through the data processing task.
Further, after the generating a data processing task according to the data processing operator after the encapsulating adaptation, the method further includes:
randomly determining a first data processing operator and a second data processing operator which are connected along the connection sequence of the directed acyclic graph in a plurality of data processing operators according to the directed acyclic graph;
judging whether the output data type of the first data processing operator is matched with the input data type of the second data processing operator;
if not, alarm information is sent out.
In the implementation process, because the computing platform processes data according to the connection order of each node in the directed acyclic graph, and the output data type and the input data type of each data processing operator are not completely the same, when data computation is performed in the order of the directed acyclic graph, if the output data type and the input type of the first data processing operator and the second data processing operator connected in the front and back are not matched, an error occurs in the processing process of the computing platform, and therefore, before a data processing task is generated, whether the output data type of the first data processing operator is matched with the input data type of the second data processing operator is judged, and if not, alarm information is sent.
In a second aspect, an embodiment of the present application provides a data processing apparatus, which is applied to a big data system, where the apparatus includes:
a determining module for determining a data processing operator for generating a data processing task;
the encapsulation adaptation module is used for carrying out encapsulation adaptation on the data processing operator by utilizing an adapter so as to enable the data processing operator subjected to encapsulation adaptation to conform to the interface standard of a corresponding computing platform;
the generating module is used for generating a data processing task according to the data processing operator after the encapsulation and adaptation;
and the sending module is used for sending the data processing task to the computing platform so that the computing platform processes data according to the data processing task.
In a third aspect, an electronic device provided in an embodiment of the present application includes: memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method according to any of the first aspect when executing the computer program.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, cause the computer to perform the method according to any one of the first aspect.
Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the above-described techniques.
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a schematic flowchart of a data processing method according to an embodiment of the present application;
fig. 2 is a schematic flowchart of encapsulation adaptation for a data processing operator according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
Example 1
Referring to fig. 1, an embodiment of the present application provides a data processing method, where the method is used in a big data processing system, and the big data processing system includes: the adapter is used for carrying out corresponding encapsulation adaptation on the data processing operator according to interface standards of different computing platforms, and comprises:
s1: determining a data processing operator for generating a data processing task;
s2: the adapter is used for carrying out encapsulation and adaptation on the data processing operator, so that the encapsulated and adapted data processing operator meets the interface standard of a corresponding computing platform;
s3: generating a data processing task according to the data processing operator after the encapsulation and adaptation;
s4: and sending the data processing task to the computing platform so that the computing platform processes the data according to the data processing task.
In the above embodiments, the data processing operator defines a method for processing data. The computing platform may acquire data through the big data processing system, or may acquire data in other manners, and this embodiment does not limit the specific manner in which the computing platform acquires data.
Illustratively, in the object-oriented programming language, the data processing operators and the adapters are in the form of objects, the adapters are essentially wrappers in a design mode, and the data processing operators can be wrapped by using the adapters, so that the wrapped data processing operators have certain functions and the like, and therefore the interface requirements of the computing platform on the data processing operators are met. It will be appreciated that different computing platforms correspond to different adapters, with one or more adapters corresponding to the same computing platform computing, based on the different interface requirements of the computing platform.
In the implementation process, a data processing operator is firstly determined, the data processing operator has the capability of processing data, in order to process large-scale data, a data processing operator needs to be sent to a computing platform for processing, different computing platforms have different requirements on the data processing operators, so that the data processing operators are packaged and adapted by utilizing the adapters, the data processing operator after encapsulation and adaptation conforms to the interface requirement of the computing platform on the data processing operator, so that the computing platform can call the data processing operator to process the data, and then, and generating a data processing task according to the data processing operator after the encapsulation and adaptation, and sending the data processing task to a computing platform for processing.
Based on the implementation method, the data processing task can be embedded into a common distributed computing framework, such as Flink, or a batch computing framework, such as Spark, or even some data access systems, such as flash, to achieve data access and complete processing at the same time.
Referring to FIG. 2, in one possible implementation, the big data processing system further comprises: the system comprises a plug-in frame and a plug-in frame interface, wherein the plug-in frame interface has an interface standard; s2 includes:
s21: transforming the data processing operator according to the interface standard of the plug-in frame interface to obtain a data processing plug-in;
s22: and the driving plug-in framework calls the data processing plug-in through the plug-in framework interface, and the data processing plug-in is packaged and adapted through the adapter.
In S21, an output data type and an input data type defining a data processing operator are included.
Illustratively, the interface standard may include constraints on and for the type of output data, or a method that requires the inclusion of a particular name. When a data processing operator is written by using an object-oriented programming language, it is also required to indicate that the data processing operator conforms to an interface standard of a certain interface according to the format of the object-oriented programming language during modification.
In the implementation process, a plug-in frame and a plug-in frame interface are arranged, the plug-in frame interface has a corresponding interface standard of the plug-in frame, a data processing operator is modified according to the interface standard to obtain a data processing plug-in, the plug-in frame can call the data processing plug-in through the interface, and the data plug-in is packaged and adapted through an adapter. Based on the embodiment, the automatic encapsulation and adaptation of the data processing operator can be realized based on the frame, and the encapsulation and adaptation efficiency is improved.
In one possible embodiment, the following constraints are not included in the interface standard of the plug-in framework:
constraints on the input data format and output data format of the data processing operator.
Illustratively, when the plug-in framework interface is defined by an object-oriented programming language, the plug-in framework interface and a method in the plug-in framework interface can be defined by a generic method and the like, so as to avoid constraining the output type and the input type of the data processing operator.
In the implementation process, the interface standard does not include the constraints on the input data type and the output data type of the interface, so that the flexibility of the data processing operator is improved, the coupling degree of the data processing operator and the plug-in frame is reduced, and the reusability of the data processing operator is improved.
In a possible implementation, after performing encapsulation adaptation on the data processing operator, the method further includes:
and the drive plug-in framework acquires configuration parameters, and the configuration parameters are used for initializing the data processing operator.
In the implementation process, the data processing operator is initialized by acquiring the configuration parameters, so that the data processing operator can adapt to different data, and the reusability of the data processing operator is further improved.
In a possible implementation manner, an initialization module for initializing the data processing operator is arranged in the data processing operator, and the plug-in framework initializes the data processing operator by calling the module of the data processing operator.
Illustratively, if the data processing operator is implemented in an object-oriented programming language, the corresponding initialization module may be a function of the data processing operator with respect to configuration parameters, or the like. Based on the initialization module, the performance of the data processing operator can be adjusted.
In one possible embodiment, S3 includes: generating a directed acyclic graph by taking the data processing operator as a node; and generating a data processing task according to the directed acyclic graph.
In the implementation process, the directed acyclic graph is generated firstly, and then the data processing task is generated according to the directed acyclic graph, so that the computing platform can call a data processing operator according to the connection sequence in the directed acyclic graph to process the data, and the processing efficiency of the computing platform is accelerated.
In one possible implementation, the big data processing system further comprises: the data processing engine calls the data processing operator through an interface of the data processing operator;
the method for generating the directed acyclic graph by taking the data processing operator as the node comprises the following steps:
determining a connection order between data processing operators for generating data processing tasks;
and driving the data processing engine, and generating the directed acyclic graph by taking each data processing operator as a node according to the connection sequence so that the computing platform calls the data processing operators to process the data according to the connection sequence of the directed acyclic graph.
In the implementation process, the data processing engine calls the data processing operators through interfaces of the data processing operators, and generates the directed acyclic graph by taking each data processing operator as a node according to the connection sequence, so that each data processing operator can be prevented from being manually operated in the process of generating the directed acyclic graph, and the efficiency of generating the directed acyclic graph is improved.
In one possible implementation, generating data processing tasks from a directed acyclic graph includes:
and adding an input plug-in before the start node of the directed acyclic graph, and adding an output plug-in after the end node of the directed acyclic graph to obtain a data processing task, wherein the data input plug-in is used for inputting data into the directed acyclic graph, and the data output plug-in is used for outputting the processed data.
In the implementation process, an output plug-in and an input plug-in are added on the basis of the directed acyclic graph, the output plug-in is used for inputting data into the directed acyclic graph, and the data output plug-in is used for outputting the processed data. Based on the above embodiment, the computing platform can output the data after computing the data through the data processing task.
When the interface standard of the plug-in interface does not include constraints on the output data type and the input data type of the data processing operator, after S3, the method further includes:
randomly determining a first data processing operator and a second data processing operator which are connected along the connection sequence of the directed acyclic graph in a plurality of data processing operators according to the directed acyclic graph;
judging whether the output data type of the first data processing operator is matched with the input data type of the second data processing operator;
if not, alarm information is sent out.
In the implementation process, because the computing platform processes data according to the connection sequence of each node in the directed acyclic graph, and the output data type and the input data type of each data processing operator are not completely the same, when data computation is performed in the sequence of the directed acyclic graph, if the output data type and the input type of the first data processing operator and the second data processing operator connected in the front and back are not matched, an error occurs in the processing process of the computing platform, and therefore, before a data processing task is generated, whether the output data type of the first data processing operator is matched with the input data type of the second data processing operator is judged, and if not, alarm information is sent.
Example 2
Referring to fig. 3, an embodiment of the present application provides a data processing apparatus, which is applied to a big data processing system, and the apparatus includes:
a determining module 1, configured to determine a data processing operator for generating a data processing task;
the encapsulation adaptation module 2 is used for carrying out encapsulation adaptation on the data processing operator by using an adapter so as to enable the data processing operator after encapsulation adaptation to meet the interface standard of a corresponding computing platform;
the generating module 3 is used for generating a data processing task according to the data processing operator after the encapsulation and adaptation;
and the sending module 4 is used for sending the data processing task to the computing platform so that the computing platform processes the data according to the data processing task.
In the implementation process, firstly, a data processing operator is determined, the data processing operator has the capacity of processing data, in order to process large-scale data, the data processing operator needs to be sent to a computing platform for processing, and different computing platforms have different requirements on the data processing operator, therefore, an adapter is used for carrying out encapsulation adaptation on the data processing operator, the data processing operator after encapsulation adaptation meets the interface requirement of the computing platform on the data processing operator, the computing platform can call the data processing operator to process the data, then, a data processing task is generated according to the data processing operator after encapsulation adaptation, and the data processing task is sent to the computing platform for processing. Based on the above embodiment, the data processing operator can be sent to different computing platforms for processing.
In one possible implementation, the big data processing system further comprises: the system comprises a plug-in frame and a plug-in frame interface, wherein the plug-in frame interface has an interface standard; the encapsulation adapting module 2 is also used for transforming a data processing operator according to the interface standard of the plug-in frame interface to obtain a data processing plug-in; and the driving plug-in framework calls the data processing plug-in through the plug-in framework interface, and the data processing plug-in is packaged and adapted through the adapter.
In one possible implementation, the following constraints are not included in the interface standard of the plug-in framework: constraints on the input data format and output data format of the data processing operator.
In a possible implementation manner, the apparatus further includes a configuration module, configured to drive the plug-in framework to obtain a configuration parameter, where the configuration parameter is used to initialize the data processing operator.
In a possible implementation, the generating module 3 is further configured to generate a directed acyclic graph with data processing operators as nodes; and generating a data processing task according to the directed acyclic graph.
In one possible implementation, the big data processing system further comprises: the data processing engine calls the data processing operators through interfaces of the data processing operators, and the generation module 3 is also used for determining a connection sequence between the data processing operators for generating the data processing tasks; and driving the data processing engine, and generating the directed acyclic graph by taking each data processing operator as a node according to the connection sequence so that the computing platform calls the data processing operators to process the data according to the connection sequence of the directed acyclic graph.
In a possible embodiment, the generating module 3 is further configured to add an input plug-in before a start node of the directed acyclic graph and add an output plug-in after an end node of the directed acyclic graph to obtain the data processing task, where the data input plug-in is configured to input data into the directed acyclic graph, and the data output plug-in is configured to output the processed data.
In a possible implementation manner, the apparatus further includes an examination module, configured to arbitrarily determine, according to the directed acyclic graph, a first data processing operator and a second data processing operator, which are connected along a connection order of the directed acyclic graph, from among the plurality of data processing operators; judging whether the output data type of the first data processing operator is matched with the input data type of the second data processing operator; if not, alarm information is sent out.
Example 3
Fig. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure, where fig. 4 is a block diagram of the electronic device. The electronic device may include a processor 41, a communication interface 42, a memory 43, and at least one communication bus 44. Wherein the communication bus 44 is used for realizing direct connection communication of these components. In the embodiment of the present application, the communication interface 42 of the electronic device is used for performing signaling or data communication with other node devices. The processor 41 may be an integrated circuit chip having signal processing capabilities.
The Processor 41 may be a general-purpose Processor including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor 41 may be any conventional processor or the like.
The Memory 43 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like. The memory 43 stores computer readable instructions which, when executed by the processor 41, enable the electronic device to perform the various steps involved in the above-described method embodiments.
Optionally, the electronic device may further include a memory controller, an input output unit.
The memory 43, the memory controller, the processor 41, the peripheral interface, and the input/output unit are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses 44. The processor 41 is adapted to execute executable modules stored in the memory 43, such as software functional modules or computer programs comprised by the electronic device.
The input and output unit is used for providing a task for a user and starting an optional time interval or preset execution time for the task creation so as to realize the interaction between the user and the server. The input/output unit may be, but is not limited to, a mouse, a keyboard, and the like.
It will be appreciated that the configuration shown in fig. 4 is merely illustrative and that the electronic device may include more or fewer components than shown in fig. 4 or have a different configuration than shown in fig. 3. The components shown in fig. 4 may be implemented in hardware, software, or a combination thereof.
The embodiments of the present application further provide a computer-readable storage medium, where instructions are stored on the computer-readable storage medium, and when the instructions are run on a computer, a computer program is executed by a processor to implement the method of the method embodiments, and details are not repeated here to avoid repetition.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative and, for example, the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: u disk, removable hard disk, read only memory, random access memory, magnetic or optical disk, etc. for storing program codes.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Claims (11)
1. A data processing method for use in a big data processing system, the big data processing system comprising: the method comprises the following steps that the data processing operator and an adapter are used for carrying out corresponding encapsulation adaptation on the data processing operator according to interface standards of different computing platforms, and the method comprises the following steps:
determining a data processing operator for generating a data processing task;
performing encapsulation adaptation on the data processing operator by using an adapter so that the data processing operator after encapsulation adaptation meets the interface standard of a corresponding computing platform;
generating a data processing task according to the data processing operator after encapsulation and adaptation;
and sending the data processing task to the computing platform so that the computing platform processes data according to the data processing task.
2. The data processing method of claim 1, wherein the big data processing system further comprises: the system comprises a plug-in frame and a plug-in frame interface, wherein the plug-in frame interface has an interface standard; the method for performing encapsulation adaptation on the data processing operator by using the adapter comprises the following steps:
transforming the data processing operator according to the interface standard of the plug-in frame interface to obtain a data processing plug-in;
and driving the plug-in framework to call the data processing plug-in through the plug-in framework interface, and performing encapsulation adaptation on the data processing plug-in through the adapter.
3. The data processing method of claim 2, wherein the following constraints are not included in the interface standard of the plug-in framework:
constraints on the input data format and the output data format of the data processing operator.
4. The data processing method according to claim 1 or 2, further comprising, after performing encapsulation adaptation on the data processing operator:
and the drive plug-in framework obtains configuration parameters, and the configuration parameters are used for initializing the data processing operator.
5. The data processing method of claim 4, wherein generating the data processing task according to the encapsulated adapted data processing operator comprises:
generating a directed acyclic graph by taking the data processing operator as a node;
and generating the data processing task according to the directed acyclic graph.
6. The data processing method of claim 5, wherein the big data processing system further comprises: the data processing engine calls the data processing operator through an interface of the data processing operator;
the generating of the directed acyclic graph by using the data processing operator as the node comprises:
determining a connection order between data processing operators for generating data processing tasks;
and driving the data processing engine, and generating the directed acyclic graph by taking each data processing operator as a node according to the connection sequence, so that the computing platform calls the data processing operators to process the data according to the connection sequence of the directed acyclic graph.
7. The data processing method of claim 5, wherein the generating the data processing task from the directed acyclic graph comprises:
adding an input plug-in before a starting node of the directed acyclic graph, and adding an output plug-in after an ending node of the directed acyclic graph to obtain the data processing task, wherein the input plug-in is used for inputting the data into the directed acyclic graph, and the output plug-in is used for outputting the processed data out of the directed acyclic graph.
8. The data processing method of claim 5, wherein after the generating of the data processing task according to the encapsulated adapted data processing operator, further comprising:
randomly determining a first data processing operator and a second data processing operator which are connected along the connection sequence of the directed acyclic graph in a plurality of data processing operators according to the directed acyclic graph;
judging whether the output data type of the first data processing operator is matched with the input data type of the second data processing operator;
if not, alarm information is sent out.
9. A data processing apparatus, for use in a big data system, the apparatus comprising:
a determining module for determining a data processing operator for generating a data processing task;
the encapsulation adaptation module is used for carrying out encapsulation adaptation on the data processing operator by utilizing an adapter so as to enable the data processing operator subjected to encapsulation adaptation to conform to the interface standard of a corresponding computing platform;
the generating module is used for generating a data processing task according to the data processing operator after the encapsulation and adaptation;
and the sending module is used for sending the data processing task to the computing platform so that the computing platform processes data according to the data processing task.
10. An electronic device, comprising: memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the data processing method according to any of claims 1 to 8 when executing the computer program.
11. A computer-readable storage medium, having stored thereon instructions which, when executed on a computer, cause the computer to perform the steps of the data processing method of any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210199927.7A CN114579331A (en) | 2022-03-01 | 2022-03-01 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210199927.7A CN114579331A (en) | 2022-03-01 | 2022-03-01 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114579331A true CN114579331A (en) | 2022-06-03 |
Family
ID=81776254
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210199927.7A Pending CN114579331A (en) | 2022-03-01 | 2022-03-01 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114579331A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024001594A1 (en) * | 2022-06-29 | 2024-01-04 | 第四范式（北京）技术有限公司 | Operator development method and apparatus, operator processing method and apparatus, and electronic device, system and storage medium |
-
2022
- 2022-03-01 CN CN202210199927.7A patent/CN114579331A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024001594A1 (en) * | 2022-06-29 | 2024-01-04 | 第四范式（北京）技术有限公司 | Operator development method and apparatus, operator processing method and apparatus, and electronic device, system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109558748B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111143087A (en) | Interface calling method and device, storage medium and server | |
| CN110007926B (en) | Language conversion method and device | |
| CN112835782B (en) | Interface access test method and system | |
| CN111443901B (en) | Java reflection-based service expansion method and device | |
| CN111142933A (en) | Workflow generation method and device, computer equipment and storage medium | |
| CN114579331A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111382084A (en) | Test method and device and electronic equipment | |
| CN112631649A (en) | Intelligent contract management method, device, terminal equipment and medium | |
| CN114443905A (en) | Interface document updating method and device, electronic equipment and readable storage medium | |
| US20140351677A1 (en) | Minimum cut set evaluation system, minimum cut set calculation method, and program | |
| CN113468212A (en) | Event execution method and device and electronic equipment | |
| CN113254014A (en) | Sketch plug-in-based interface development method, device and system | |
| CN110995522A (en) | Information processing method and device | |
| CN113630404B (en) | Protocol message transmission method, device, storage medium and terminal | |
| CN114564480A (en) | Data processing method and device based on Flink platform, electronic equipment and storage medium | |
| CN111324645A (en) | Data processing method and device for block chain | |
| CN113935847A (en) | Online process risk processing method, device, server and medium | |
| CN112579093A (en) | Information pushing method and device and related equipment | |
| CN114020646A (en) | VPN automatic test method, device, electronic equipment and storage medium | |
| CN115526580B (en) | Node state determining method and device, electronic equipment and storage medium | |
| CN117492968B (en) | Native application calling method and device, electronic equipment and readable storage medium | |
| CN114116051B (en) | Processing method, device, equipment and storage medium based on neural network model | |
| CN115032971B (en) | Updating method and device of vehicle diagnosis equipment and server | |
| CN111338686A (en) | Software construction method, equipment, server and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |