CN109726216A - A kind of data processing method and processing system based on directed acyclic graph - Google Patents
A kind of data processing method and processing system based on directed acyclic graph Download PDFInfo
- Publication number
- CN109726216A CN109726216A CN201811644309.9A CN201811644309A CN109726216A CN 109726216 A CN109726216 A CN 109726216A CN 201811644309 A CN201811644309 A CN 201811644309A CN 109726216 A CN109726216 A CN 109726216A
- Authority
- CN
- China
- Prior art keywords
- node
- data
- directed acyclic
- acyclic graph
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The present invention provides a kind of data processing method and processing system based on directed acyclic graph, this method comprises: the first user interface of display, the first user interface includes node, and node includes source point, operator and place point；In response to the selection operation to node, the icon of the destination node of selection is shown in the first display area of the first user interface；In response to the attended operation to destination node, it will be connected including source point, operator with the destination node of place point using node connecting line, and be created as directed acyclic graph；In directed acyclic graph, the data that source point obtains are input to operator by node connecting line, and by operator, treated that data are input to place point to export target data.The present invention can provide visual user interface, for creating the directed acyclic graph that can be used in static data processing or real time data processing, the computer programming sentence using profession is not needed, it is easy to operate.
Description
Technical field
The present invention relates to big data processing field more particularly to a kind of data processing methods and place based on directed acyclic graph
Reason system.
Background technique
Stream calculation is an important component in current data processing field.With traditional data processing system number
Calculation processing difference is carried out again according to being stored in after hard disk or other storage services, stream calculation processing data incoming in real time and real
When the value for embodying data, generally believe that the value of data flow lowers with the passing of time.
Existing stream calculation system uses SQL (Structured Query Language, structured query language) language
Sentence generates corresponding DAG (Directed Acyclic Graph, directed acyclic graph), and being generated with directed acyclic graph includes data
Data flow logic in directed acyclic graph, is distributed to different nodes, professional for user is wanted by the node for flowing logic
Ask higher, and inconvenient.
Summary of the invention
In view of this, the present invention provides a kind of data processing method and processing system based on directed acyclic graph, for solving
Certainly existing stream calculation system needs that the corresponding directed acyclic of data flow can be generated using the computer programming sentence of profession
Figure, professional requirement is high, inconvenient problem.
In order to solve the above technical problems, the present invention provides a kind of data processing method based on directed acyclic graph, comprising:
Show that the first user interface, first user interface include the node for creating directed acyclic graph, the section
Point includes the source point for obtaining the input data of data source, the operator for being handled input data and for exporting mesh
Mark the place point of data；
In response to the selection operation to the node, the icon of the destination node of selection is shown in the first user interface
In first display area；
It will include the source point, operator and place using node connecting line in response to the attended operation to the destination node
The destination node connection of point, is created as directed acyclic graph；
Wherein, in the directed acyclic graph, the data that the source point obtains are input to described by the node connecting line
Operator, and treated that data are input to the place point to export target data the operator.
Optionally, first user interface further includes the second display area, and second display area is for showing institute
State node.
Optionally, the selection operation is to click the clicking operation of the node, alternatively, the node is drawn to described
The drag operation of first display area.
Optionally, the destination node includes output anchor point and/or input anchor point, and the attended operation is from a mesh
Mark the drag operation that the output anchor point of node is pulled to the input anchor point of another destination node.
Optionally, the method also includes:
In response to the configuration operation to the destination node or the node connecting line in first display area, show
Show information configuration interface；
Based on the configuration information that user inputs at the information configuration interface, to the institute shown in first display area
It states destination node or the node connecting line is configured.
Optionally, the configuration information of the destination node includes at least one of the following: title, description information, output field
Structure and parameter configuration.
Optionally, the configuration information of the node connecting line includes at least one of the following: data filtering information and data point
Stream information.
Optionally, the destination node includes output anchor point and/or input anchor point, in response to the company to the destination node
Operation is connect, will include the steps that the source point, operator are connected with the destination node of place point using node connecting line, comprising:
It is shown based on default display mode and exports the matched input anchor point of anchor point with target；
Target input anchor point is connected to from target output anchor point using node connecting line.
Optionally, the method also includes:
In response to the preservation operation of the directed acyclic graph to creation, the directed acyclic graph of creation is converted into json format
Code saves.
Optionally, the method also includes:
In response to the publication operation of the directed acyclic graph to creation, the directed acyclic graph of creation is converted into json format
Code；
It is the corresponding data flow of Flink by the code conversion of the json format；
The data flow is run, stream operation is generated.
Optionally, the method also includes:
In response to the export operation of the directed acyclic graph to creation, the directed acyclic graph of creation is converted into json format
Code, and be compressed file export by the code conversion of the json format.
Optionally, the method also includes:
In response to the import operation of the directed acyclic graph to creation, the directed acyclic graph after file is converted will be imported and be shown in
Current interface.
Optionally, the import operation in response to the directed acyclic graph to creation, it is oriented after file is converted by importing
Acyclic figure is shown in current interface
In response to importing the import operation of the code of the json format of compression, the generation of the json format of the compression is decompressed
Code；
It is directed acyclic graph by the code conversion of the json format after decompression, and covers the directed acyclic graph currently created.
Optionally, the method also includes:
In response to uploading the upload operation of operator, the operator of upload is added to the node for being used to create directed acyclic graph
In.
Optionally, the method also includes:
In response to the debugging operations debugged to the stream operation, the stream operation is debugged, and show debugging
As a result.
Optionally, the method also includes:
In response to checking operation to the log of the stream operation, the log of the stream operation is shown.
Optionally, the method also includes:
Operation is checked in response to the operating status to the stream operation, is obtained in the stream operation at least partly described
The monitoring information of node, and show the monitoring information.
The present invention also provides a kind of data processing systems based on directed acyclic graph, comprising:
Display module, for showing the first user interface, first user interface includes for creating directed acyclic graph
Node, the node includes the source point for obtaining the input data of data source, the calculation for being handled input data
Son and the place point for exporting target data；
First execution module, in response to the selection operation to the node, the icon of the destination node of selection to be shown
It is shown in the first display area of the first user interface；
Second execution module, for that will include using node connecting line in response to the attended operation to the destination node
The source point, operator are connected with the destination node of place point, are created as directed acyclic graph；Wherein, in the directed acyclic graph, institute
It states node connecting line and the data that the source point obtains is input to the operator, and treated that data are input to the operator
The place point is to export target data.
Optionally, first user interface further includes the second display area, and second display area is for showing institute
State node.
Optionally, the selection operation is to click the clicking operation of the node, alternatively, the node is drawn to described
The drag operation of first display area.
Optionally, the destination node includes output anchor point and/or input anchor point, and the attended operation is from a mesh
Mark the drag operation that the output anchor point of node is pulled to the input anchor point of another destination node.
Optionally, the system also includes:
Third execution module, in response to in first display area the destination node or the node connect
The configuration of wiring operates, and shows information configuration interface；
4th execution module, the configuration information for being inputted based on user at the information configuration interface, to described first
The destination node or the node connecting line shown in display area is configured.
Optionally, the configuration information of the destination node includes at least one of the following: title, description information, output field
Structure and parameter configuration.
Optionally, the configuration information of the node connecting line includes at least one of the following: data filtering information and data point
Stream information.
Optionally, the destination node includes that output anchor point and/or input anchor point, second execution module are also used to
It is shown based on default display mode and exports the matched input anchor point of anchor point with target；It is exported using node connecting line from the target
Anchor point is connected to target input anchor point.
Optionally, the system also includes:
5th execution module, for the preservation operation in response to the directed acyclic graph to creation, by the directed acyclic of creation
The code that figure is converted into json format saves.
Optionally, the system also includes:
6th execution module, for the publication operation in response to the directed acyclic graph to creation, by the directed acyclic of creation
Figure is converted into the code of json format；It is the corresponding data flow of Flink by the code conversion of the json format；Run the number
According to stream, stream operation is generated.
Optionally, the system also includes:
7th execution module, for the export operation in response to the directed acyclic graph to creation, by the directed acyclic of creation
Figure is converted into the code of json format, and is compressed file export by the code conversion of the json format.
Optionally, the system also includes:
8th execution module, for the import operation in response to the directed acyclic graph to creation, after file conversion will be imported
Directed acyclic graph be shown in current interface.
Optionally, the system also includes:
8th execution module is further used for the import operation of the code of the json format in response to importing compression,
Decompress the code of the json format of the compression；It is directed acyclic graph by the code conversion of the json format after decompression, and covers
The directed acyclic graph currently created.
Optionally, the system also includes:
The operator of upload is added to for the upload operation in response to uploading operator and is used to create by the 9th execution module
In the node of directed acyclic graph.
Optionally, the system also includes:
Tenth execution module, in response to the debugging operations debugged of stream operation, to the stream operation into
Row debugging, and show debugging result.
Optionally, the system also includes:
11st execution module, for showing the stream operation in response to checking operation to the log of the stream operation
Log.
Optionally, the system also includes:
12nd execution module obtains the stream for checking operation in response to the operating status to the stream operation
To the monitoring information of at least partly described node in operation, and show the monitoring information.
The present invention also provides a kind of processing equipment, including processor, memory and it is stored on the memory and can be
The computer program run on the processor, the computer program are realized above-mentioned based on oriented when being executed by the processor
The step of data processing method of acyclic figure.
The present invention also provides a kind of computer readable storage medium, computer is stored on the computer readable storage medium
The step of program, the computer program realizes the above-mentioned data processing method based on directed acyclic graph when being executed by processor.
The advantageous effects of the above technical solutions of the present invention are as follows:
In an embodiment of the present invention, it is capable of providing visual user interface, can be used in static data for creating
The directed acyclic graph of processing or real time data processing does not need the computer programming sentence using profession, for the profession of user
Property require it is lower and easy to operate.
Detailed description of the invention
Fig. 1 is the flow diagram of the data processing method based on directed acyclic graph of the embodiment of the present invention；
Fig. 2 is the schematic diagram of the first user interface provided in an embodiment of the present invention；
Fig. 3 is the first embodiment for the user interface that the present invention configures an operator；
Fig. 4 is the second embodiment for the user interface that the present invention configures an operator；
Fig. 5 is the schematic diagram at the interface of editor's application message of the embodiment of the present invention；
Fig. 6 is the schematic diagram that the Jar of the embodiment of the present invention assures the interface of reason；
Fig. 7 and Fig. 8 is the debugging window schematic diagram of the stream operation of the embodiment of the present invention；
Fig. 9 is the monitoring window schematic diagram of the stream operation of the embodiment of the present invention；
Figure 10 is the structural schematic diagram of the data processing system based on directed acyclic graph of the embodiment of the present invention；
Figure 11 is the structural schematic diagram of the processing equipment of the embodiment of the present invention.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
Attached drawing, the technical solution of the embodiment of the present invention is clearly and completely described.Obviously, described embodiment is this hair
Bright a part of the embodiment, instead of all the embodiments.Based on described the embodiment of the present invention, ordinary skill
Personnel's every other embodiment obtained, shall fall within the protection scope of the present invention.
To solve existing stream calculation system need that data flow pair can be generated using the computer programming sentence of profession
The directed acyclic graph answered, professional requirement is high, inconvenient problem, referring to FIG. 1, Fig. 1 be the embodiment of the present invention based on
The data processing method of the flow diagram of the data processing method of directed acyclic graph, the directed acyclic graph includes:
Step 11: the first user interface of display, first user interface includes the node for creating directed acyclic graph,
The node includes the source point (source) for obtaining the input data of data source, the calculation for being handled input data
Sub (operator) and the place point (sink) for exporting target data；
So-called directed acyclic graph is as group, arrangement, block chain and a kind of data structure.
Step 12: in response to the selection operation to the node, the icon of the destination node of selection being shown in the first use
In first display area at family interface；
Step 13: will include the source point, calculation using node connecting line in response to the attended operation to the destination node
Son is connected with the destination node of place point, is created as directed acyclic graph；Wherein, in the directed acyclic graph, the node connection
The data that the source point obtains are input to the operator by line, and the operator treated data the are input to place point with
Export target data.In an embodiment of the present invention, it is capable of providing visual user interface, can be used in static state for creating
The directed acyclic graph of data processing or real time data processing does not need the computer programming sentence using profession, for user's
It is professional require it is lower and easy to operate.
The citing of source point, operator and place point is illustrated separately below.
The source point indicates the source that the data of directed acyclic graph flow into, such as may include at least one of following: local
File source (from data are read in local file), the source socket (data are read from socket), the source http (connect from http endpoint
Receive data), the source JDBC is (with JDBC from database (mysql, DB (Database, database) 2) in read data), analogue data
Source (interval output specified data according to the specified time), IBM MQ (reading data from WebSphere MQ) and Kafka
Source (consumption data from specified Kafka theme).
The intermediate node of the operator representation directed acyclic graph, handles input data, such as may include following
It is at least one: the classes such as converging operation, Http operation, machine learning, serializing operation, statistics, stream operation, map function and window
Not.
Wherein, converging operation includes pure window polymerization, key polymerization and window polymerization；Pure window polymerization is by window polymerization note
Record, when polymerization, can add major key, achieve the purpose that be polymerize based on certain keys；Key polymerization carries out field based on Key
Converging operation, addition major key field name configuration need to configure polymerization Id, operator can according to the configuration in major key field name, into
Row polymerization；Window polymerization can add major key by window record polymerization, when polymerization, reach based on certain keys and polymerize
Purpose.Pure window polymerization can be understood as normal window polymerization, and window polymerization has been done on this basis patrols the fractionation of window
Volume.Because the data in a window may be excessive, calculator memory is caused to be not enough to store the data in next window, that
In window polymerization, understands and this window is split into different wickets, each independent calculated result of wicket, and it is final
It merges.
Http operation includes that Http is inquired, and Http inquiry is by sending Http request data, the data that will acquire
It stores in output field.Output field refers to that the column name of data, data are to pass in entire stream in the form of key:value
Defeated, so each data all has one or more key, key is exactly to arrange name here.
Machine learning includes Drools rule, PMML model and Kalman filter；Drools rule uses customized
Drools rule handles data, and the file of upload must be ended up with .jar, and the inside will include Drools rule text
Part after file uploads successfully, can obtain file name as module name automatically；PMML rule is according to the PMML model content of upload
Data are handled, the file of upload must be ended up with .pmml, after file uploads successfully, can obtain file name automatically
As model name；Kalman filter is filtered using Kalman to data are flowed into.One module represents, one rule, than
If a drools file is exactly a module, this module is exactly the rule described in this drools file.
Serializing operation includes Kafka Avro serializing and Kafka Arvo unserializing, and Kafka Avro serializing will
Java Map turns to binary data according to Avro schema sequence, Kafka Arvo unserializing by binary data according to
Avro schema is deserialized as Java Map.
Statistics includes that uniqueness counts, frequency statistics and quantile are estimated；Uniqueness is counted based on field to inflow data
The calculating of value uniqueness is carried out, configuration needs the field name counted, carries out statistics frequency of occurrence according to field name, then configuration statistics
The setting (configuring newly-generated data to be stored in which column, i.e. in which key) that value field name is exported；Frequency statistics
Imprecise computation is executed in a stream, by configuring the field name to be counted, after statistics, is saved in statistical value field name
In；Quantile estimation carries out quantile estimation to data are flowed into based on field and specified fractile, configures the field to be estimated
Title is estimated according to fractile, finally using the quantile field name of configuration as field name when output.I.e. configuration is new
Which column is the data of generation be stored in, i.e. in which key.
Stream operation includes joint, Field-Partitions, value subregion, UDF (user-defined function, user-defined
Function) operator and complicated event matching；Combining is to be joined to a upstream, if there is multiple upstream nodes, by upstream
Output data joins together to be transmitted to downstream again；Field-Partitions are based on certain field and carry out attitude layer, configure option field,
Divisional type is set, subregion is carried out to data according to divisional type and field；It is poly- according to field attribute value progress subregion to be worth subregion
It closes, usually docks window class operator below, configure optionally field attribute, subregion is carried out to data by field；UDF operator
User oneself can define the operator is used to do which operation, if having selected filter type, the class filled in action name
Filter class or its subclass must be inherited from；Complicated event matching carries out matching treatment according to match pattern condition to upstream events
And it returns and meets matched sequence of events set.
Map function includes at least one of the following: Spring rule, window maximum, Attribute Association, obtains city name, item
Part filter, unique key filtering, resolver, input format conversion, output format conversion, attribute is promoted, attribute maps, Cache
Operation；In Spring rule, if regular design conditions are really to execute the rule operation of configuration, when regular design conditions are set
It is defaulted as when being set to * true；Window maximum is that history dominant record will be spilt into downstream, and the field to be compared is arranged, according to
Compare field, the dominant record in the same window is spilt into downstream；Attribute Association is associated with difference by relevant field attribute
Transaction event, configure major key, field name and field value, field name can be matched with field value, and major key is used to extract field
The content of name；It obtains city name and city name is searched by IP, be arranged in input data and represent the field of IP as IP field, city
City's field name represents city fields in output data；Conditional filter carries out garbled data by conditional expression, meets item
The data of part, which will be retained, flows to next operator；Unique key filtering is filtered data according to key, it is ensured that key is identical
Data within a specified time only downstream export one；Resolver is based on value (the value xml/json/csv/ in certain field
Regular Expression), parse the value (number of xml/json/csv/Regular Expression format of specific field
According to)；Output format conversion is to format customization data format and store it in payload, first has to configuration format, selects
It selects with the output of which kind of format as a result, mapping entity in addition can be configured, it can be with mapping field title；Input format conversion is by word
Symbol string, byte, Map set or other kinds of payload are converted to RT event, and (data is a RT event, is represent
The format of data in a stream), a format conversion is carried out to input data, output Schema can be configured for next calculation
Son uses, and can also configure mapping entity, and the field that original field is changed to oneself needs (is changed the field name of data, i.e.,
Column name)；Attribute promotion is to promote complex object attribute in Map into top layer Map, fills in the field that promote top layer, root
The Map of lower layer is extracted into top layer according to field is filled in；Attribute mapping is to be carried out by SpringEL expression formula to field attribute
Map operation, operator can directly operate the column of primary data, and optional includes column to be deleted and column to be transformed, configure wait delete
Except column can delete certain column in primary data, column name in primary data can be changed by configuring column to be transformed, be configured to be transformed
Column need to fill in calculation expression, it is necessary to be the expression formulas of SpringEL grammer；Cache operation is based on Ignite carry out table wound
/ reading/writing data is built, grouping key attribute indicates the column for needing to cache, and object class must be then the class for needing the value cached.Window
Mouth operation includes window sequence and deleting stale data, and window sequence is to be based on field name in the window temporally to stab sequence, can
With option and installment major key, it is polymerize according to major key, is then ranked up further according to the sort key title of configuration；It eliminates expired
Data are to be based on timestamp deleting stale data in the window to carry out the time in the window by extracting the time value in data
The comparison of stamp eliminates expired data.Sort key seeks to compare sequence with the data in which column, and column name is exactly
Here classification key name.
The place point indicates the place of data outflow, may include at least one of following: JDBC for exporting target data
Output, Redis output, Kafka output, IBM MQ are exported, based on the Http output of Http operation, the Socket based on data source
It exports and the local file based on data source exports.Wherein, Http endpoint is sent data to when Http is exported.JDBC output
It is to write data into database, such as MySQL, DB2 with JDBC.Redis output is to write data into Redis.Kafka is exported
Transmit data to Kafka.IBM MQ output is to transmit data to IBM MQ.Socket output is to transmit data to
Socket.Local file output is to write data into local file.
Referring to FIG. 2, Fig. 2 is the schematic diagram of the first user interface provided in an embodiment of the present invention, first user circle
Face is capable of providing the node for creating directed acyclic graph, and the node includes source point, operator and place point.Implementation shown in Fig. 2
In example, first user interface includes the first display area 201 and the second display area 202, and the first display area 201 is used for
Show the directed acyclic graph of creation, second display area 202 is for showing the node.
In embodiment shown in Fig. 2, the first user interface can directly provide the second display area 202, for showing section
Point, so that user be facilitated to select the node for creating directed acyclic graph.Certainly, in some other embodiment of the invention,
Node can not also be directly displayed in first user interface, but the selection of node is provided by other means, for example, providing one
Kind node searching tool, as shown in Fig. 2, the search box in the second display area 202, to be searched by inputting in the search box
Node keyword, node can also be found.Certainly, other modes for providing node of the first user interface are not limited to above-mentioned
Two ways.
In the embodiment of the present invention, in response to the selection operation to the node, the icon of the destination node of selection is shown
In in the first display area of the first user interface, the icon is, for example, box shown in Fig. 2.The selection operation can be with
For the clicking operation for clicking the node, alternatively, the node to be drawn to the drag operation of first display area.It clicks
Operation and the equal very simple of drag operation, and meet user's use habit.
It will include institute using node connecting line in response to the attended operation to the destination node in the embodiment of the present invention
State source point, operator is connected with the destination node of place point, is created as directed acyclic graph, the node connecting line is, for example, institute in Fig. 2
The connecting line with the arrow shown.The attended operation can be for from the output anchor point of a destination node to the defeated of another destination node
Enter the drag operation of anchor point dragging.In the embodiment of the present invention, in addition to the source point and place point, each operator all has an input
Anchor point and an output anchor point；The source point has an output anchor point, realizes connection for the input anchor point with the operator；It is described
Place point has an input anchor point, realizes connection for the output anchor point with the operator.When two nodes connect, inputted as data
The node of side is known as upstream node, and the node as data output side is known as downstream node.In the embodiment of the present invention, the node
Connecting line is formed by drag operation, certainly, can also shape by other means in some other embodiment of the invention
At, such as a connecting line drawing tool is provided, node connecting line etc. is drawn by clicking connecting line drawing tool.
Referring to FIG. 2, the directed acyclic graph of creation completion is shown in the first display area 201 in Fig. 2, a creation
The directed acyclic graph of the data flow of completion includes source point, at least one operator and place point.In other embodiments, a creation is completed
Data flow also may include more than one source point and more than one place point.The section of the directed acyclic graph of display in Fig. 2
In point, local data source is source point, and local file output is place point, other are operator.
In the embodiment of the present invention, the data processing method based on directed acyclic graph can also include:
In response to the configuration operation to the destination node or the node connecting line in first display area, show
Show information configuration interface；
Based on the configuration information that user inputs at the information configuration interface, to the institute shown in first display area
It states destination node or the node connecting line is configured.
Referring to FIG. 2, the first user interface in Fig. 2 includes third display area 203, the third display area 203
For showing information configuration interface.The information configuration interface configured to local file source is shown in Fig. 2.It can from Fig. 2
To find out, user can input the configuration information in local file source in the text input box of offer.
In the embodiment of the present invention, the configuration information of the destination node may include at least one of following: title, description letter
Breath, output field structure (schema) and parameter configuration.
In the embodiment of the present invention, when carrying out information configuration to node connecting line, it is generally basede on the setting of SpringEL grammer
Data select expression formula progress data flow to enter selection.The configuration information of the node connecting line may include at least one of following:
Data filtering information and data diffluent information.In the embodiment of the present invention, the node connecting line will be with its input terminal in addition to having
The data of the node of connection are directed into except the function for the node connecting with its output end, can also have data filtering and/or
The function of data distribution, filtering refer to the data for filtering out certain condition, and shunting refers to the node connecting with its input terminal is defeated
Data out are divided at least two parts, are directed respectively into the different nodes connected from its output end.
When carrying out information configuration to source point, can configure at least one of following: the title of source point, description information, output
Schema and parameter configuration, local file source can with Profile Path and/or file coding format (generally default UTF-8),
The source Socket can configure the address Socket and/or port numbers, and the source Http can configure at least one of following: obtain data when
Between be spaced (necessarily), http endpoint network address URL, http Method type, receive data format (xml, json, form),
The source JDBC can configure at least one of following: driving title, database connection network address, user name, password, for loading data
SQL, analog data source generally need setup time interval and reading value, and IBM MQ can configure at least one of following: host address,
Target port, channel, user id, password, target CCSID, queue management device, queue or subject name, maximum reconnect number,
The source Kafka can configure at least one of following: subscribe to subject, the address kafka, consumption group mark, deviation post of data.
When carrying out information configuration to operator, can configure at least one of following: the title of operator, description information, output
Schema and parameter configuration, its parameter configuration of different operators are also different.It is represented due to different operators at different data
Logic is managed, therefore the configuration of different operators is also not quite similar.Such as window Aggregation Operator, configuration is divided into two parts: a part is window
Relevant configuration, how configuration information is for by data to carry out classified packets, as shown in Figure 3.In Fig. 3, it is one section that major key, which generates class,
Code represents the logic for generating major key, can be customized by the user.Major key and major key, which generate class, one, two choosings
One.Global window is the window classified without major key, and all data belong to same group.Window type refers to different windows
Mouthful, there are tumbling type and slidling type.Window size, that is, window duration.A part is matched for polymerization correlation
It sets, configuration information has indicated how by the data progress converging operation after classified packets, as shown in Figure 4.Grouping is data point
It opens, polymerization is the later data of statistical packet.Such as batch of data 1,2,3,4,5,6 is divided into 2 groups according to certain rule,
It is respectively as follows: 1,2,3 and 4,5,6, then does the converging operation of their accumulated values, then first group of result is 6, second group of knot
Fruit is 15.Polymeric type is just different converging operation, such as accumulated value, maximum value, minimum value, average value, variance etc..It is poly-
Closing attribute is to carry out the converging operation that front is said to the value in which field.Adeditive attribute is certain converging operation needs
Other attributes, such as the converging operation of TopN need specified N is how many.
When clicking through row information configuration to place, can configure at least one of following: title, the description information, output of place point
Schema and parameter configuration, the parameter type of homoclinic point configuration is not different.As local file output can with Profile Path and/
Or file write mode (newly-built or covering), Http output can configure at least one of following: Thread Count, maximum thread, block
Queue size, URL, http method and reception data format.
In the embodiment of the present invention, between node and node and it is not all can connect, such as the data of source point can only flow out,
The data of place point can only flow into, and the data of operator can flow in and out, but cannot form loop, etc., node and node
Between can connect, referred to as nodes match cannot connect between node and node, and referred to as two nodes mismatch.The present invention is real
It applies in example, the matching rule of these nodes can be saved.In order to remind user, in the embodiment of the present invention, the destination node packet
Output anchor point and/or input anchor point are included, it is described in response to the target section in response to the attended operation to the destination node
The attended operation of point will be connected with the destination node of place point including the source point, operator using node connecting line and include:
It is shown based on default display mode and exports the matched input anchor point of anchor point with target；
Target input anchor point is connected to from target output anchor point using node connecting line.
The default display mode of institute, such as can be to be highlighted and export the matched input anchor point of anchor point with target, alternatively, will
With the matched input anchor point amplification display of target output anchor point, etc., if reach differentiation and target to export anchor point matched defeated
Enter anchor point and the purpose with the unmatched input anchor point of target output anchor point.
In above-described embodiment, the creation of directed acyclic graph can be completed, it, can also be to the first user circle after the completion of creation
Directed acyclic graph on face is edited, it is described editor may include: preservation, publication, importing, export, editor application message,
Jar assures reason, automatic placement, multiple selection modes, duplication are pasted, the first user interface of resetting, in shortcut key at least one of.
1, save and issue directed acyclic graph
In the embodiment of the present invention, the data processing method based on directed acyclic graph can also include:
In response to the preservation operation of the directed acyclic graph to creation, the directed acyclic graph of creation is converted into json format
Code saves.
Referring to FIG. 2, the first user interface in Fig. 2 can also provide one the 4th display area 204, the 4th display
A preservation icon can be shown in region 204, saved icon by clicking, can be saved the directed acyclic graph of creation.When with
When preservation icon is clicked at family, directed acyclic graph created can be converted into the code of json format, include stream in the code
Using (real-time streams run in RT are defined by application one by one.Each application is stream application) in each node match
Confidence breath and upstream and downstream information, such as pull a node it is carried out up and down line configuration when, the upstream ID config option of the node
In preserve upstream node ID, similarly, the downstream ID of the node also can be set, if the upstream node ID of the node be 1, under
Swimming node ID is 1, then the ID of the node itself is 2, and the upstream ID corresponds to the input anchor point of the node, and the downstream ID is corresponding
The output anchor point of the node.When as saved icon when the user clicks, the directed acyclic graph in the first user interface can be turned from the background
The code for being changed to json format is saved to database.
In the embodiment of the present invention, directed acyclic graph is converted into the code of json format, json (JavaScript
Object Notation, JS object numbered musical notation) be a kind of lightweight data interchange format.It is by ECMAScript (based on Europe
Suan Ji association formulate js specification) a subset, stored and indicated using the text formatting for being totally independent of programming language
Data.Succinctly and clearly hierarchical structure makes json become ideal data interchange language.It is easy to people to read and write, simultaneously
It is also easy to machine parsing and generates, and effectively promote network transmission efficiency.
In the embodiment of the present invention, the data processing method based on directed acyclic graph can also include:
In response to the publication operation of the directed acyclic graph to creation, the directed acyclic graph of creation is converted into json format
Code；
It is the corresponding data flow of Flink by the code conversion of the json format；
The data flow is run, stream operation is generated.
Referring to FIG. 2, a publication icon can be shown in the 4th display area 204, icon is issued by clicking,
The directed acyclic graph of creation can be issued.Referring to FIG. 2, when issuing icon when the user clicks, by having in the first user interface
The code json code for being converted to json format to acyclic figure is further converted into the corresponding data flow of Flink, and automatic running.
To realize the processing to data, in concrete processing procedure, the data that the node connecting line obtains the source point are inputted
To the operator, and the operator, treated that data are input to the place point to export target data.Flink is a point
The computing engines of cloth, it can be used to do stream process, i.e., handle some real-time streams in real time, generate data in real time
As a result.
The data flow is, for example, flow diagram (JobGraph), and JobGraph is the unique data flow engine institute by Flink
The data structure of the statement operation of identification.
2, the importing and export of directed acyclic graph
Icon and export icon are imported (after issuing referring to FIG. 2, can show in the 4th display area 204
Two icons), by click import icon and export icon, can import icon and export icon directed acyclic graph.
That is, in the embodiment of the present invention, the data processing method based on directed acyclic graph can also include:
In response to the export operation of the directed acyclic graph to creation, the directed acyclic graph of creation is converted into json format
Code, and be compressed file (such as ZIP file) export by the code conversion of the json format.
Transmission etc. so as to realize directed acyclic graph operates.Derived file can be used for newly-built complete application.
In the embodiment of the present invention, the data processing method based on directed acyclic graph can also include:
In response to the import operation of the directed acyclic graph to creation, the directed acyclic graph after file is converted will be imported and be shown in
Current interface.
Further, the import operation in response to the directed acyclic graph to creation will import having after file is converted
Being shown in current interface to acyclic figure can also include:
In response to importing the import operation of the code of the json format of compression, the generation of the json format of the compression is decompressed
Code；
It is directed acyclic graph by the code conversion of the json format after decompression, and covers current in first display area
The directed acyclic graph of creation.
Existing directed acyclic graph is imported into system so as to realize.
In the embodiment of the present invention, is imported and exported with json code.Export is usually to be converted to directed acyclic graph
Json code is simultaneously further compressed into compressed file and exports to database or local.Import be usually upload from RT application derived from
Compressed file, the stream application that covering is currently editing by import operation, and can not restore.
3, application message is edited
First user interface further includes editor's application message icon, clicks editor's application message icon, will pop up one
Messagewindow is edited, as shown in Figure 5.
Wherein, timeliness is determined by the data in stream application, each data corresponding time, if data itself are wrapped
It then handles, can be handled according to consumption time if itself does not include the time, when consumption according to event time containing event time
Between be exactly time that this data enters in system, runing time is the time that this data is handled by system.Watermark interval is
The attribute of Flink itself, there is window converging operation in flink, window according to watermark interval determine window how cutting or every
Which window data belongs to.
Checkpoint configuration is the attribute configuration for adjusting stream application operation and flink itself, there is default value, user
It can also custom-configure.
Restart the attribute configuration that strategy configuration is also flink itself, has default value, user can also custom-configure.
4, Jar assures reason
As shown in Fig. 2, first user interface can also include the icon (one i.e. after setting icon that Jar assures reason
A icon), it clicks Jar and assures reason icon, will pop up Jar and assure reason window, as shown in Figure 6.By interface user can more than
Pass jar file, mainly operator.
That is, in the embodiment of the present invention, the data processing method based on directed acyclic graph can also include:
In response to uploading the upload operation of operator, the operator of upload is added to the node for being used to create directed acyclic graph
In.
That is, except the operator for having deposited system offer, user can also upload other calculations in the embodiment of the present invention
Son, to improve user experience.
5, automatic placement
First user interface further includes the icon of automatic placement, automatic placement icon is clicked, to the first user interface
In mixed and disorderly node and node connecting line carry out automatic placement, help user to set up the directed acyclic of a beauty and specification
Figure.
6, multiple selection modes
As shown in Fig. 2, first user interface further includes icon (the 8th figure i.e. after publication icon of multiple selection modes
Mark), multiple selection modes icon is clicked, multiple nodes in the first user interface can be selected, with choice box in order to replicate, move
Or delete choice.
7, duplication is pasted
As shown in Fig. 2, first user interface further includes icon (the 9th figure i.e. after publication icon that duplication is pasted
Mark), duplication paste icon is clicked, can choose some or all of node and node connecting line in directed acyclic graph, is replicated
To other directed acyclic graph, with all information of node and node connecting line, including configuration information when duplication.
8, the first user interface is reset
First user interface can also include the icon of the first user interface of resetting, for the first user interface to contract
Ratio is put, at the same time it can also which directed acyclic graph to be shown centered on.
9, shortcut key
First user interface can also include shortcut key, including amplification, reduce, reform, recalling, replicating and/or
Prompt operation is supported in stickup etc., provides night mode, convenient to be shown in projection or large-size screen monitors.
In the embodiment of the present invention, after having created directed acyclic graph, directed acyclic graph can also be debugged, be checked
Log and/or monitoring.
1, it debugs
Directed acyclic graph creation complete after, can stream operation corresponding to the directed acyclic graph of creation debug,
In order to developer debugging window in be currently running stream operation input test data, observe output result whether
Meet expection, to pass through the modification stream operation of the configuration information of update operator.
Debugging window can be as shown in FIG. 7 and 8.
I.e. in the embodiment of the present invention, the data processing method based on directed acyclic graph can also include:
In response to the debugging operations debugged to the stream operation, the stream operation is debugged, and show debugging
As a result.
2, log is checked
After directed acyclic graph creation is completed, the log of the corresponding stream operation of directed acyclic graph can be checked, in order to
Exploitation or operation maintenance personnel pass through the real-time running state for checking the log observation stream operation exported in operation in stream operation.Such as
In Risk Monitoring, risk information can be known in real time.
That is, in the embodiment of the present invention, the data processing method based on directed acyclic graph can also include:
In response to checking operation to the log of the stream operation, the log of the stream operation is shown.
3, it monitors
After directed acyclic graph creation is completed, it can be monitored with the treatment process of each operator in convection current operation,
In order to develop or operation maintenance personnel stream operation in by check flink monitoring information observe stream operation real time execution shape
State.As shown in Figure 9.
I.e. in the embodiment of the present invention, the data processing method based on directed acyclic graph can also include:
Operation is checked in response to the operating status to the stream operation, is obtained in the stream operation at least partly described
The monitoring information of node, and show the monitoring information.
Based on the same inventive concept, referring to FIG. 10, the embodiment of the present invention also provides a kind of number based on directed acyclic graph
According to processing system, comprising:
Display module 101, for showing the first user interface, first user interface includes for creating directed acyclic
The node of figure, the node include source point for obtaining the input data of data source, for being handled input data
Operator and place point for exporting target data；
First execution module 102, in response to the selection operation to the node, by the icon of the destination node of selection
It is shown in the first display area of the first user interface；
Second execution module 103, for that will be wrapped using node connecting line in response to the attended operation to the destination node
Include the source point, operator is connected with the destination node of place point, be created as directed acyclic graph；Wherein, in the directed acyclic graph,
The data that the source point obtains are input to the operator by the node connecting line, and treated that data input the operator
To the place point to export target data.
Optionally, first user interface further includes the second display area, and second display area is for showing institute
State node.
Optionally, the selection operation is to click the clicking operation of the node, alternatively, the node is drawn to described
The drag operation of first display area.
Optionally, the destination node includes output anchor point and/or input anchor point, and the attended operation is from a mesh
Mark the drag operation that the output anchor point of node is pulled to the input anchor point of another destination node.
Optionally, the system also includes:
Third execution module, in response to in first display area the destination node or the node connect
The configuration of wiring operates, and shows information configuration interface；
4th execution module, the configuration information for being inputted based on user at the information configuration interface, to described first
The destination node or the node connecting line shown in display area is configured.
Optionally, the configuration information of the destination node includes at least one of the following: title, description information, output field
Structure and parameter configuration.
Optionally, the configuration information of the node connecting line includes at least one of the following: data filtering information and data point
Stream information.
Optionally, the destination node includes that output anchor point and/or input anchor point, second execution module are also used to
It is shown based on default display mode and exports the matched input anchor point of anchor point with target；It is exported using node connecting line from the target
Anchor point is connected to target input anchor point.
Optionally, the system also includes:
5th execution module, for the preservation operation in response to the directed acyclic graph to creation, by the directed acyclic of creation
The code that figure is converted into json format saves.
Optionally, the system also includes:
6th execution module, for the publication operation in response to the directed acyclic graph to creation, by the directed acyclic of creation
Figure is converted into the code of json format；It is the corresponding data flow of Flink by the code conversion of the json format；Run the number
According to stream, stream operation is generated.
Optionally, the system also includes:
7th execution module, for the export operation in response to the directed acyclic graph to creation, by the directed acyclic of creation
Figure is converted into the code of json format, and is compressed file export by the code conversion of the json format.
Optionally, the system also includes:
8th execution module, for the import operation in response to the directed acyclic graph to creation, after file conversion will be imported
Directed acyclic graph be shown in current interface.
Optionally, the system also includes:
8th execution module is further used for the import operation of the code of the json format in response to importing compression,
Decompress the code of the json format of the compression；It is directed acyclic graph by the code conversion of the json format after decompression, and covers
The directed acyclic graph currently created.
Optionally, the system also includes:
The operator of upload is added to for the upload operation in response to uploading operator and is used to create by the 9th execution module
In the node of directed acyclic graph.
Optionally, the system also includes:
Tenth execution module, in response to the debugging operations debugged of stream operation, to the stream operation into
Row debugging, and show debugging result.
Optionally, the system also includes:
11st execution module, for showing the stream operation in response to checking operation to the log of the stream operation
Log.
Optionally, the system also includes:
12nd execution module obtains the stream for checking operation in response to the operating status to the stream operation
To the monitoring information of at least partly described node in operation, and show the monitoring information.
Figure 11 is please referred to, Figure 11 is the structural schematic diagram of the processing equipment of one embodiment of the invention, the processing equipment 110 packet
It includes: processor 111 and memory 112.In embodiments of the present invention, processing equipment 110 further include: be stored on memory 112
And the computer program that can be run on processor 111, computer program realize following steps when being executed by processor 111:
Show that the first user interface, first user interface include the node for creating directed acyclic graph, the section
Point includes the source point for obtaining the input data of data source, the operator for being handled input data and for exporting mesh
Mark the place point of data；
In response to the selection operation to the node, the icon of the destination node of selection is shown in the first user interface
In first display area；
It will include the source point, operator and place using node connecting line in response to the attended operation to the destination node
The destination node connection of point, is created as directed acyclic graph；
Wherein, in the directed acyclic graph, the data that the source point obtains are input to described by the node connecting line
Operator, and treated that data are input to the place point to export target data the operator.It is total that processor 111 is responsible for management
Line architecture and common processing, memory 112 can store the used data when executing operation of processor 111.
Optionally, first user interface further includes the second display area, and second display area is for showing institute
State node.
Optionally, the attended operation is from the output anchor point of a destination node to the defeated of another destination node
Enter the drag operation of anchor point dragging.
Optionally, the destination node includes output anchor point and/or input anchor point, and the attended operation is from a mesh
The drag operation that the anchor point of mark node is pulled to the anchor point of another destination node.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
In response to the configuration operation to the destination node or the node connecting line in first display area, show
Show information configuration interface；
Based on the configuration information that user inputs at the information configuration interface, to the institute shown in first display area
It states destination node or the node connecting line is configured.
Optionally, the configuration information of the destination node includes at least one of the following: title, description information, output field
Structure and parameter configuration.
Optionally, the configuration information of the node connecting line includes at least one of the following: data filtering information and data point
Stream information.
Optionally, the destination node includes that output anchor point and/or input anchor point, computer program are held by processor 111
It can also be achieved following steps when row:
It is shown based on default display mode and exports the matched input anchor point of anchor point with target；
Target input anchor point is connected to from target output anchor point using node connecting line.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
In response to the preservation operation of the directed acyclic graph to creation, the directed acyclic graph of creation is converted into json format
Code saves.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
In response to the publication operation of the directed acyclic graph to creation, the directed acyclic graph of creation is converted into json format
Code；
It is the corresponding data flow of Flink by the code conversion of the json format；
The data flow is run, stream operation is generated.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
In response to the export operation of the directed acyclic graph to creation, the directed acyclic graph of creation is converted into json format
Code, and be compressed file export by the code conversion of the json format.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
In response to the import operation of the directed acyclic graph to creation, the directed acyclic graph after file is converted will be imported and be shown in
Current interface.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
In response to importing the import operation of the code of the json format of compression, the generation of the json format of the compression is decompressed
Code；
It is directed acyclic graph by the code conversion of the json format after decompression, and covers current in first display area
The directed acyclic graph of creation.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
In response to uploading the upload operation of operator, the operator of upload is added to the node for being used to create directed acyclic graph
In.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
In response to the debugging operations debugged to the stream operation, the stream operation is debugged, and show debugging
As a result.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
In response to checking operation to the log of the stream operation, the log of the stream operation is shown.
Optionally, following steps be can also be achieved when computer program is executed by processor 111:
Operation is checked in response to the operating status to the stream operation, is obtained in the stream operation at least partly described
The monitoring information of node, and show the monitoring information.
The embodiment of the present invention also provides a kind of computer readable storage medium, stores on the computer readable storage medium
Computer program, the computer program realize that the above-mentioned data processing method based on directed acyclic graph is real when being executed by processor
Each process of example is applied, and identical technical effect can be reached, to avoid repeating, which is not described herein again.Wherein, the calculating
Machine readable storage medium storing program for executing, such as read-only memory (Read-Only Memory, ROM), random access memory (Random
Access Memory, RAM), magnetic or disk etc..
The above is a preferred embodiment of the present invention, it is noted that for those skilled in the art
For, without departing from the principles of the present invention, it can also make several improvements and retouch, these improvements and modifications
It should be regarded as protection scope of the present invention.
Claims (10)
1. a kind of data processing method based on directed acyclic graph characterized by comprising
Show that the first user interface, first user interface include the node for creating directed acyclic graph, the node packet
Include the source point of the input data for obtaining data source, the operator for being handled input data and for exporting number of targets
According to place point；
In response to the selection operation to the node, the icon of the destination node of selection is shown in the first of the first user interface
In display area；
It will include the source point, operator and place point using node connecting line in response to the attended operation to the destination node
Destination node connection, is created as directed acyclic graph；
Wherein, in the directed acyclic graph, the data that the source point obtains are input to the operator by the node connecting line,
And treated that data are input to the place point to export target data the operator.
2. the method as described in claim 1, which is characterized in that first user interface further includes the second display area, institute
The second display area is stated for showing the node.
3. method according to claim 1 or 2, which is characterized in that the selection operation is to click the click behaviour of the node
Make, alternatively, the node to be drawn to the drag operation of first display area.
4. the method as described in claim 1, which is characterized in that the destination node includes exporting anchor point and/or input anchor point,
The attended operation is to drag from the output anchor point of a destination node to what the input anchor point of another destination node pulled
Drag operation.
5. the method as described in claim 1, which is characterized in that further include:
In response to the configuration operation to the destination node or the node connecting line in first display area, display letter
Cease configuration interface；
Based on the configuration information that user inputs at the information configuration interface, to the mesh shown in first display area
Mark node or the node connecting line are configured.
6. a kind of data processing system based on directed acyclic graph characterized by comprising
Display module, for showing the first user interface, first user interface includes the section for creating directed acyclic graph
Point, the node include the source point for obtaining the input data of data source, the operator for being handled input data and
For exporting the place point of target data；
First execution module, in response to the selection operation to the node, the icon of the destination node of selection to be shown in
In first display area of the first user interface；
Second execution module, for that will include described using node connecting line in response to the attended operation to the destination node
Source point, operator are connected with the destination node of place point, are created as directed acyclic graph；Wherein, in the directed acyclic graph, the section
The data that the source point obtains are input to the operator by point connecting line, and treated described in data are input to the operator
Place point is to export target data.
7. system as claimed in claim 6, which is characterized in that first user interface further includes the second display area, institute
The second display area is stated for showing the node.
8. system as claimed in claims 6 or 7, which is characterized in that the selection operation is to click the click behaviour of the node
Make, alternatively, the node to be drawn to the drag operation of first display area.
9. system as claimed in claim 6, which is characterized in that the destination node includes exporting anchor point and/or input anchor point,
The attended operation is to drag from the output anchor point of a destination node to what the input anchor point of another destination node pulled
Drag operation.
10. system as claimed in claim 6, which is characterized in that further include:
Third execution module, in response to the destination node or the node connecting line in first display area
Configuration operation, show information configuration interface；
4th execution module, the configuration information for being inputted based on user at the information configuration interface, to first display
The destination node or the node connecting line shown in region is configured.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811644309.9A CN109726216A (en) | 2018-12-29 | 2018-12-29 | A kind of data processing method and processing system based on directed acyclic graph |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811644309.9A CN109726216A (en) | 2018-12-29 | 2018-12-29 | A kind of data processing method and processing system based on directed acyclic graph |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109726216A true CN109726216A (en) | 2019-05-07 |
Family
ID=66299505
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811644309.9A Pending CN109726216A (en) | 2018-12-29 | 2018-12-29 | A kind of data processing method and processing system based on directed acyclic graph |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109726216A (en) |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110825511A (en) * | 2019-11-07 | 2020-02-21 | 北京集奥聚合科技有限公司 | Operation flow scheduling method based on modeling platform model |
| CN110852596A (en) * | 2019-11-06 | 2020-02-28 | 无锡功恒精密机械制造有限公司 | Process design method and design module |
| CN110956272A (en) * | 2019-11-01 | 2020-04-03 | 第四范式（北京）技术有限公司 | Method and system for realizing data processing |
| CN111078211A (en) * | 2019-12-26 | 2020-04-28 | 北京每日优鲜电子商务有限公司 | Method, device, equipment and storage medium for creating data processing tool |
| CN111209268A (en) * | 2020-01-13 | 2020-05-29 | 北京明略软件系统有限公司 | Directed acyclic graph configuration method, data processing method, device and configuration platform |
| CN111324345A (en) * | 2020-03-19 | 2020-06-23 | 北京奇艺世纪科技有限公司 | Data processing mode generation method, data processing method and device and electronic equipment |
| CN111506779A (en) * | 2020-04-20 | 2020-08-07 | 东云睿连(武汉)计算技术有限公司 | Object version and associated information management method and system facing data processing |
| CN111552470A (en) * | 2019-12-31 | 2020-08-18 | 远景智能国际私人投资有限公司 | Data analysis task creation method and device in Internet of things and storage medium |
| CN111951010A (en) * | 2020-07-26 | 2020-11-17 | 中国建设银行股份有限公司 | Business decision method and device based on rule engine, electronic equipment and readable storage medium |
| CN112148926A (en) * | 2019-06-28 | 2020-12-29 | 京东数字科技控股有限公司 | Graph data stream processing method, processing device and storage medium |
| CN112306502A (en) * | 2019-07-31 | 2021-02-02 | 上海华为技术有限公司 | Code generation method and device |
| CN112347054A (en) * | 2020-11-10 | 2021-02-09 | 北京明略软件系统有限公司 | Personnel information comparison method, system, electronic equipment and computer readable storage medium |
| CN112506497A (en) * | 2020-11-30 | 2021-03-16 | 北京九章云极科技有限公司 | Data processing method and data processing system |
| CN112749194A (en) * | 2020-06-03 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Visualized data processing method and device, electronic equipment and readable storage medium |
| CN113313419A (en) * | 2021-06-23 | 2021-08-27 | 中国农业银行股份有限公司 | Information system window change risk obtaining method and device |
| CN113792067A (en) * | 2021-11-16 | 2021-12-14 | 全景智联(武汉)科技有限公司 | System and method for automatically generating SQL (structured query language) based on recursive algorithm |
| KR20220002070A (en) * | 2020-06-30 | 2022-01-06 | 베이징 바이두 넷컴 사이언스 앤 테크놀로지 코., 엘티디. | Streaming computing method and apparatus based on dag interaction |
| CN116050713A (en) * | 2023-03-13 | 2023-05-02 | 深圳迅策科技有限公司 | Index calculation method, apparatus, electronic device and computer program product |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101859313A (en) * | 2009-04-08 | 2010-10-13 | 索尼公司 | Messaging device and method and program thereof |
| CN102508706A (en) * | 2011-11-18 | 2012-06-20 | 北京航空航天大学 | Multi-source data integrating platform and establishing method thereof |
| CN103677763A (en) * | 2012-08-30 | 2014-03-26 | 中国科学院软件研究所 | Graphical programming source file storage and analytic method |
| CN105303397A (en) * | 2015-09-28 | 2016-02-03 | 百度在线网络技术（北京）有限公司 | Information recommendation method, system, server end and client |
| CN105843630A (en) * | 2016-06-08 | 2016-08-10 | 江西洪都航空工业集团有限责任公司 | Method for graphical programming development based on robot |
| US20170017903A1 (en) * | 2015-02-11 | 2017-01-19 | Skytree, Inc. | User Interface for a Unified Data Science Platform Including Management of Models, Experiments, Data Sets, Projects, Actions, Reports and Features |
| CN108830383A (en) * | 2018-05-30 | 2018-11-16 | 第四范式（北京）技术有限公司 | For showing the method and system of machine learning model building process |
| CN108898229A (en) * | 2018-06-26 | 2018-11-27 | 第四范式（北京）技术有限公司 | For constructing the method and system of machine learning modeling process |
| CN109033109A (en) * | 2017-06-09 | 2018-12-18 | 杭州海康威视数字技术股份有限公司 | Data processing method and system |
-
2018
- 2018-12-29 CN CN201811644309.9A patent/CN109726216A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101859313A (en) * | 2009-04-08 | 2010-10-13 | 索尼公司 | Messaging device and method and program thereof |
| CN102508706A (en) * | 2011-11-18 | 2012-06-20 | 北京航空航天大学 | Multi-source data integrating platform and establishing method thereof |
| CN103677763A (en) * | 2012-08-30 | 2014-03-26 | 中国科学院软件研究所 | Graphical programming source file storage and analytic method |
| US20170017903A1 (en) * | 2015-02-11 | 2017-01-19 | Skytree, Inc. | User Interface for a Unified Data Science Platform Including Management of Models, Experiments, Data Sets, Projects, Actions, Reports and Features |
| CN105303397A (en) * | 2015-09-28 | 2016-02-03 | 百度在线网络技术（北京）有限公司 | Information recommendation method, system, server end and client |
| CN105843630A (en) * | 2016-06-08 | 2016-08-10 | 江西洪都航空工业集团有限责任公司 | Method for graphical programming development based on robot |
| CN109033109A (en) * | 2017-06-09 | 2018-12-18 | 杭州海康威视数字技术股份有限公司 | Data processing method and system |
| CN108830383A (en) * | 2018-05-30 | 2018-11-16 | 第四范式（北京）技术有限公司 | For showing the method and system of machine learning model building process |
| CN108898229A (en) * | 2018-06-26 | 2018-11-27 | 第四范式（北京）技术有限公司 | For constructing the method and system of machine learning modeling process |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112148926A (en) * | 2019-06-28 | 2020-12-29 | 京东数字科技控股有限公司 | Graph data stream processing method, processing device and storage medium |
| CN112306502A (en) * | 2019-07-31 | 2021-02-02 | 上海华为技术有限公司 | Code generation method and device |
| CN110956272A (en) * | 2019-11-01 | 2020-04-03 | 第四范式（北京）技术有限公司 | Method and system for realizing data processing |
| CN110956272B (en) * | 2019-11-01 | 2023-08-08 | 第四范式（北京）技术有限公司 | Method and system for realizing data processing |
| CN110852596A (en) * | 2019-11-06 | 2020-02-28 | 无锡功恒精密机械制造有限公司 | Process design method and design module |
| CN110825511A (en) * | 2019-11-07 | 2020-02-21 | 北京集奥聚合科技有限公司 | Operation flow scheduling method based on modeling platform model |
| CN111078211A (en) * | 2019-12-26 | 2020-04-28 | 北京每日优鲜电子商务有限公司 | Method, device, equipment and storage medium for creating data processing tool |
| CN111552470A (en) * | 2019-12-31 | 2020-08-18 | 远景智能国际私人投资有限公司 | Data analysis task creation method and device in Internet of things and storage medium |
| CN111552470B (en) * | 2019-12-31 | 2023-09-12 | 远景智能国际私人投资有限公司 | Data analysis task creation method, device and storage medium in Internet of Things |
| CN111209268A (en) * | 2020-01-13 | 2020-05-29 | 北京明略软件系统有限公司 | Directed acyclic graph configuration method, data processing method, device and configuration platform |
| CN111324345A (en) * | 2020-03-19 | 2020-06-23 | 北京奇艺世纪科技有限公司 | Data processing mode generation method, data processing method and device and electronic equipment |
| CN111506779A (en) * | 2020-04-20 | 2020-08-07 | 东云睿连(武汉)计算技术有限公司 | Object version and associated information management method and system facing data processing |
| CN112749194A (en) * | 2020-06-03 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Visualized data processing method and device, electronic equipment and readable storage medium |
| JP2022013649A (en) * | 2020-06-30 | 2022-01-18 | ベイジン バイドゥ ネットコム サイエンス テクノロジー カンパニー リミテッド | Streaming computing method and device based on dag interaction |
| KR20220002070A (en) * | 2020-06-30 | 2022-01-06 | 베이징 바이두 넷컴 사이언스 앤 테크놀로지 코., 엘티디. | Streaming computing method and apparatus based on dag interaction |
| JP7197620B2 (en) | 2020-06-30 | 2022-12-27 | ベイジン バイドゥ ネットコム サイエンス テクノロジー カンパニー リミテッド | Streaming computing method and apparatus based on DAG interactions |
| KR102528210B1 (en) * | 2020-06-30 | 2023-05-02 | 베이징 바이두 넷컴 사이언스 앤 테크놀로지 코., 엘티디. | Streaming computing method and apparatus based on dag interaction |
| CN111951010A (en) * | 2020-07-26 | 2020-11-17 | 中国建设银行股份有限公司 | Business decision method and device based on rule engine, electronic equipment and readable storage medium |
| CN112347054A (en) * | 2020-11-10 | 2021-02-09 | 北京明略软件系统有限公司 | Personnel information comparison method, system, electronic equipment and computer readable storage medium |
| CN112506497B (en) * | 2020-11-30 | 2021-08-24 | 北京九章云极科技有限公司 | Data processing method and data processing system |
| CN112506497A (en) * | 2020-11-30 | 2021-03-16 | 北京九章云极科技有限公司 | Data processing method and data processing system |
| CN113313419A (en) * | 2021-06-23 | 2021-08-27 | 中国农业银行股份有限公司 | Information system window change risk obtaining method and device |
| CN113313419B (en) * | 2021-06-23 | 2023-10-10 | 中国农业银行股份有限公司 | Method and device for acquiring window change risk of information system |
| CN113792067A (en) * | 2021-11-16 | 2021-12-14 | 全景智联(武汉)科技有限公司 | System and method for automatically generating SQL (structured query language) based on recursive algorithm |
| CN116050713A (en) * | 2023-03-13 | 2023-05-02 | 深圳迅策科技有限公司 | Index calculation method, apparatus, electronic device and computer program product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109726216A (en) | A kind of data processing method and processing system based on directed acyclic graph | |
| US11429600B2 (en) | Loading queries using search points | |
| US11698901B1 (en) | Interactive data field analyzer | |
| US11238033B1 (en) | Interactive location queries for raw machine data | |
| US11625404B2 (en) | Multi-phased execution of a search query | |
| US10853383B2 (en) | Interactive parallel coordinates visualizations | |
| US10853382B2 (en) | Interactive punchcard visualizations | |
| US11768875B2 (en) | Monitoring system control interface for asset tree determination | |
| US20210133634A1 (en) | Efficiently executing commands at external computing services | |
| CN108038222B (en) | System of entity-attribute framework for information system modeling and data access | |
| US11676092B1 (en) | Graphical user interface with hybrid role-based access control | |
| CN106682097A (en) | Method and device for processing log data | |
| US11037342B1 (en) | Visualization modules for use within a framework for displaying interactive visualizations of event data | |
| CN110007913A (en) | Visual flow chart of data processing setting method, device, equipment and storage medium | |
| CN106682096A (en) | Method and device for log data management | |
| WO2014079304A1 (en) | Report creation method, device and system | |
| CN106648859A (en) | Task scheduling method and device | |
| US20210263717A1 (en) | Key-based logging for processing of structured data items with executable logic | |
| US10628603B1 (en) | Graphical user interface for configuring a cross-silo enterprise data acquisition, reporting and analysis system | |
| CN106708965A (en) | Data processing method and apparatus | |
| CN106682099A (en) | Data storage method and device | |
| CN106681808A (en) | Task scheduling method and device | |
| CN105824944B (en) | A kind of XML configuration data maintaining method and system based on scripting language | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN109033157B (en) | Complex data retrieval method and system based on user-defined search condition tree |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190507 |
|
| RJ01 | Rejection of invention patent application after publication |