CN110727729A - Method and device for realizing intelligent operation - Google Patents
Method and device for realizing intelligent operation Download PDFInfo
- Publication number
- CN110727729A CN110727729A CN201810717155.5A CN201810717155A CN110727729A CN 110727729 A CN110727729 A CN 110727729A CN 201810717155 A CN201810717155 A CN 201810717155A CN 110727729 A CN110727729 A CN 110727729A
- Authority
- CN
- China
- Prior art keywords
- computation
- data
- calculation
- tree
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000004364 calculation method Methods 0.000 claims abstract description 141
- 238000007726 management method Methods 0.000 claims description 37
- 238000004422 calculation algorithm Methods 0.000 claims description 21
- 238000010586 diagram Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 claims description 13
- 238000013523 data management Methods 0.000 claims description 10
- 230000002085 persistent effect Effects 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000013461 design Methods 0.000 abstract description 4
- 238000012800 visualization Methods 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 24
- 238000011161 development Methods 0.000 description 5
- 238000010276 construction Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 3
- 230000001419 dependent effect Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000007418 data mining Methods 0.000 description 2
- 238000013515 script Methods 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000007723 transport mechanism Effects 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
Images
Abstract
The invention discloses a method and a device for realizing intelligent operation. The disclosed method comprises: providing a GUI; displaying selectable data sources and selectable intelligent operation operations to a user through a GUI; receiving a calculation graph which is input by a user and comprises a selected data source, a selected intelligent operation and a connection relation between the selected data source and the selected intelligent operation through a GUI (graphical user interface); acquiring input data from a selected data source based on a calculation graph, and performing intelligent operation based on the input data and selected intelligent operation and connection relation to obtain intermediate calculation results and final calculation results of all levels; and displaying the final calculation result and/or the intermediate calculation results of each stage to a user through a GUI. The disclosed solution enables a user to use a GUI interface for design of a visualization computation graph and for various forms of output of desired data.
Description
Technical Field
The invention relates to the technical field of computer technology and data mining, in particular to a method and a device for realizing intelligent operation.
Background
With the development and popularization of computer technology, people often need to use computers to perform various operations in daily life and work, and the operations usually involve a large amount of data. How to process and analyze the massive user data to obtain valuable data analysis results about user habits, network service operation information and the like is always the key point of data mining research of people.
Taking a network service provider as an example, the service network provided by the network service provider involves many servers and communication links, and a large amount of new data is generated every moment. If these massive raw data (e.g., log data, billing bandwidth data, reuse rate data, node bandwidth data, node coverage quality data, etc.) are analyzed, very valuable information is often obtained on how to better improve quality of service and save costs.
When a network facilitator mines and analyzes raw data relating to a plurality of different platforms, the following methods are currently commonly employed: collecting and aggregating the original data of each platform through the respective API interfaces of the plurality of platforms, and providing the related original data to the outside through a unified interface; when the original data volume is large or the aggregation algorithm is complex, professional developers are usually required to compile scripts to obtain API data and compile algorithms to process, development cost is high, and maintenance difficulty is high.
In order to solve the above problems, a new technical solution needs to be proposed.
Disclosure of Invention
The method for realizing intelligent operation comprises the following steps:
providing a GUI;
displaying selectable data sources and selectable intelligent operation operations to a user through a GUI;
receiving a calculation graph which is input by a user and comprises a selected data source, a selected intelligent operation and a connection relation between the selected data source and the selected intelligent operation through a GUI (graphical user interface);
acquiring input data from a selected data source based on a calculation graph, and performing intelligent operation based on the input data and selected intelligent operation and connection relation to obtain intermediate calculation results and final calculation results of all levels;
the final calculation results and/or intermediate calculation results at various levels are displayed to the user via the GUI,
wherein the selectable intelligent operation comprises: basic mathematical operation, custom domain-specific computation functions, user input including at least one of: clicking and dragging through a mouse, selecting through a shortcut key, and moving and positioning through a direction key.
According to the method for realizing intelligent operation, the steps of acquiring input data from a selected data source based on a calculation graph, and carrying out intelligent operation based on the input data, the selected intelligent operation and the connection relation to obtain intermediate calculation results and final calculation results of each level comprise:
generating a calculation flow tree corresponding to the calculation graph based on the calculation graph, performing persistent storage, and converting the calculation flow tree into an engine instruction;
constructing a data engine tree based on the engine instructions;
based on the state information of the data source node and the intermediate calculation node in the data engine tree and the dependency relationship among the nodes, the dispatching operation is carried out to obtain intermediate calculation results and final calculation results of each level,
wherein, the intermediate calculation results and the final calculation results of each stage can be multiplexed.
According to the method for realizing intelligent operation, the data engine tree is used for describing all computing nodes related to the selected intelligent operation, the operation sequence and the operation dependent computing process, the data engine tree is used for registering each computing node in the state pool by driving the scheduling center, the computing process is managed through the computing node state information in the state pool, when all child nodes of the computing nodes in the computing process are computed, the computing process tree obtains the information that the computing nodes can start to compute, and the computing process tree is also used for managing parallel computing among independent data and pipeline management for finishing the data processing process.
According to the method for realizing intelligent operation, the state pool is also used for index management and data management, the index management is used for generating the node unique ID for the computing node, and the data engine tree manages the computing node through the node unique ID.
According to the method for realizing intelligent operation, the step of carrying out scheduling operation based on the state information of the data source node and the intermediate computing node in the data engine tree and the dependency relationship among the nodes to obtain intermediate computing results and final computing results of each level comprises the following steps:
and the driving scheduling center issues a scheduling task ID, and the driving management module automatically searches corresponding data and algorithm for calculation based on a selected data source corresponding to the scheduling task ID and the unique ID of the selected intelligent operation to obtain intermediate calculation results and final calculation results of each level.
The device for realizing intelligent operation according to the invention comprises:
the front-end data diagram configuration and display module is used for providing a GUI (graphical user interface), displaying selectable data sources and selectable intelligent operation operations to a user through the GUI, receiving a calculation diagram which is input by the user and comprises selected data sources, selected intelligent operation operations and a connection relation between the selected data sources and the selected intelligent operation operations through the GUI, and displaying a final calculation result and/or intermediate calculation results of each level to the user through the GUI;
the intelligent operation module is used for acquiring input data from a selected data source based on the calculation graph, performing intelligent operation based on the input data and the selected intelligent operation and connection relation to obtain intermediate calculation results and final calculation results of each level,
wherein the selectable intelligent operation comprises: basic mathematical operation, custom domain-specific computation functions, user input including at least one of: clicking and dragging through a mouse, selecting through a shortcut key, and moving and positioning through a direction key.
According to the device for realizing intelligent operation of the invention, the intelligent operation module comprises:
the calculation flow tree generation and conversion module is used for generating a calculation flow tree corresponding to the calculation graph based on the calculation graph, performing persistent storage and converting the calculation flow tree into an engine instruction;
the data engine tree generating module is used for constructing a data engine tree based on the engine instruction;
a scheduling operation module for performing scheduling operation based on the state information of the data source node and the intermediate calculation node in the data engine tree and the dependency relationship among the nodes to obtain intermediate calculation results and final calculation results of each level,
wherein, the intermediate calculation results and the final calculation results of each stage can be multiplexed.
According to the device for realizing intelligent operation, the data engine tree is used for describing all computing nodes related to the selected intelligent operation, the operation sequence and the operation dependent computing process, the data engine tree registers each computing node in the state pool by driving the scheduling center, the computing process is managed through the computing node state information in the state pool, when all sub-nodes of the computing nodes in the computing process are computed, the computing process tree obtains the information that the computing nodes can start to compute, and the computing process tree is also used for managing parallel computing among independent data and pipeline management for finishing the data processing process.
According to the device for realizing intelligent operation, the state pool is also used for index management and data management, the index management is used for generating the node unique ID for the computing node, and the data engine tree manages the computing node through the node unique ID.
According to the device for realizing intelligent operation of the invention, the scheduling operation module further comprises:
the driving scheduling center is used for issuing scheduling task ID;
and the drive management module is used for automatically searching corresponding data and algorithm for calculation based on the selected data source corresponding to the scheduling task ID and the unique ID of the selected intelligent operation to obtain intermediate calculation results and final calculation results of all levels.
According to the above-described aspect of the present invention, the user can design the visual computation graph and obtain various forms of output of desired data using the GUI interface.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention. In the drawings, like reference numerals are used to indicate like elements. The drawings in the following description are directed to some, but not all embodiments of the invention. For a person skilled in the art, other figures can be derived from these figures without inventive effort.
Fig. 1 schematically shows a schematic flow diagram of a method of implementing intelligent operations according to the invention.
Fig. 2 schematically shows a block schematic of an apparatus for performing intelligent operations according to the present invention.
FIG. 3 illustratively depicts a schematic view of a GUI provided by the front end data diagram configuration and display module in accordance with the present invention.
Fig. 4 schematically shows a block diagram of one example of an apparatus for implementing intelligent operations according to the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.
Fig. 1 schematically shows a schematic flow diagram of a method of implementing intelligent operations according to the invention.
As shown in fig. 1, the method for implementing intelligent operation according to the present invention includes:
step S102: providing a GUI;
step S104: displaying selectable data sources and selectable intelligent operation operations to a user through a GUI;
step S106: receiving a calculation graph which is input by a user and comprises a selected data source, a selected intelligent operation and a connection relation between the selected data source and the selected intelligent operation through a GUI (graphical user interface);
step S108: acquiring input data from a selected data source based on a calculation graph, and performing intelligent operation based on the input data and selected intelligent operation and connection relation to obtain intermediate calculation results and final calculation results of all levels;
step S110: the final calculation results and/or intermediate calculation results at various levels are displayed to the user via the GUI,
wherein the selectable intelligent operation comprises: basic mathematical operations, custom domain-specific computational functions (e.g., operations or functions related to network bandwidth management), user inputs including at least one of: clicking and dragging through a mouse, selecting through a shortcut key, and moving and positioning through a direction key.
For example, the operations associated with the custom domain-specific computation function may be the bandwidth splitting, bandwidth-data-alignment computation operations, etc. shown in FIG. 3.
Alternatively, step S108 may include the steps of:
generating a calculation flow tree corresponding to the calculation graph based on the calculation graph, performing persistent storage, and converting the calculation flow tree into an engine instruction;
constructing a data engine tree based on the engine instructions;
based on the state information of the data source node and the intermediate calculation node in the data engine tree and the dependency relationship among the nodes, the dispatching operation is carried out to obtain intermediate calculation results and final calculation results of each level,
wherein, the intermediate calculation results and the final calculation results of each stage can be multiplexed.
Optionally, the data engine tree is a description of a computation flow including all computation nodes, an operation sequence and operation dependencies related to the selected intelligent operation, the data engine tree registers each computation node in the state pool by driving the scheduling center, the computation flow is managed through computation node state information in the state pool, when all child nodes of the computation node in the computation flow are computed, the computation flow tree obtains information that the computation node can start computation, and the computation flow tree is further used for managing parallel computation between independent data and pipeline management for completing a data processing process.
Optionally, the state pool is further used for index management and data management, the index management is used for generating node unique IDs for the computing nodes, and the data engine tree manages the computing nodes through the node unique IDs.
Optionally, the step of performing scheduling operation based on the state information of the data source node and the intermediate computation node in the data engine tree and the dependency relationship between the nodes to obtain intermediate computation results and final computation results at each level includes:
and the driving scheduling center issues a scheduling task ID, and the driving management module automatically searches corresponding data and algorithm for calculation based on a selected data source corresponding to the scheduling task ID and the unique ID of the selected intelligent operation to obtain intermediate calculation results and final calculation results of each level.
Fig. 2 schematically shows a block schematic of an apparatus 200 for performing intelligent operations according to the present invention.
As shown in fig. 2, the apparatus 200 for implementing intelligent operation according to the present invention includes:
a front-end data diagram configuration and display module 201, configured to provide a GUI, display selectable data sources and selectable intelligent operation operations to a user through the GUI, receive a computation diagram, which is input by the user and includes a selected data source, a selected intelligent operation, and a connection relationship between the selected data source and the selected intelligent operation through the GUI, and display a final computation result and/or intermediate computation results of each stage to the user through the GUI;
an intelligent operation module 203 for obtaining input data from the selected data source based on the calculation graph, performing intelligent operation based on the input data and the selected intelligent operation and connection relation to obtain intermediate calculation results and final calculation results at each level,
wherein the selectable intelligent operation comprises: basic mathematical operation, custom domain-specific computation functions, user input including at least one of: clicking and dragging through a mouse, selecting through a shortcut key, and moving and positioning through a direction key.
Optionally, the intelligent operation module 203 may include:
the calculation flow tree generation and conversion module is used for generating a calculation flow tree corresponding to the calculation graph based on the calculation graph, performing persistent storage and converting the calculation flow tree into an engine instruction;
the data engine tree generating module is used for constructing a data engine tree based on the engine instruction;
a scheduling operation module for performing scheduling operation based on the state information of the data source node and the intermediate calculation node in the data engine tree and the dependency relationship among the nodes to obtain intermediate calculation results and final calculation results of each level,
wherein, the intermediate calculation results and the final calculation results of each stage can be multiplexed.
Optionally, the data engine tree is a description of a computation flow including all computation nodes, an operation sequence and operation dependencies related to the selected intelligent operation, the data engine tree registers each computation node in the state pool by driving the scheduling center, the computation flow is managed through computation node state information in the state pool, when all child nodes of the computation node in the computation flow are computed, the computation flow tree obtains information that the computation node can start computation, and the computation flow tree is further used for managing parallel computation between independent data and pipeline management for completing a data processing process.
Optionally, the state pool is further used for index management and data management, the index management is used for generating node unique IDs for the computing nodes, and the data engine tree manages the computing nodes through the node unique IDs.
Optionally, the scheduling operation module further includes:
the driving scheduling center is used for issuing scheduling task ID;
and the drive management module is used for automatically searching corresponding data and algorithm for calculation based on the selected data source corresponding to the scheduling task ID and the unique ID of the selected intelligent operation to obtain intermediate calculation results and final calculation results of all levels.
In order to make the GUI used according to the above technical solution of the present invention more clearly understood by those skilled in the art, the following description will be made with reference to a specific example.
Fig. 3 exemplarily shows a schematic view of a GUI provided by the front-end data diagram configuration and display module 201 according to the present invention.
As described above, the front-end data diagram configuration and display module 201 is used to provide a GUI, for example, to provide visualization controls (including the optional data sources, optional intelligent computing operations, connection relationships described above) to assist a user in building a computing diagram.
As shown in fig. 3, the front end (i.e., the GUI described above) includes elements (zones), calculation zones, and (element) details (zones). The function of each region is described as follows:
1. the element area provides all data source pools (including all data source elements, namely the optional data sources), a computing pool (including all computing elements, namely the optional intelligent operation operations), all display elements (including all display forms of result data, namely the display tool, currently providing excel table export, webpage chart display and API interface export), and a storage element pool (including all storage modes of intermediate data and result data to be stored, namely the storage tools, including a back-end cache, a permanent cache, a page cache, a timing cache and the like).
2. The component detail area provides detailed configuration items (i.e., the above configuration parameters) of all components, such as a limitation condition when the data source pulls data (e.g., a start time, a time granularity of data, a format of data, etc.), a constant item configuration when the computing component calculates, and so on.
3. The calculation area provides visualization display and editing capacity of the whole calculation graph, the calculation graph is formed by dragging the elements to the calculation area, and a user can clearly know the specific operation process of the whole data.
In order to make the apparatus 200 for implementing intelligent operations according to the present invention more obvious for those skilled in the art, the following description will be made with reference to a specific example.
Fig. 4 schematically shows a block diagram of one example of an apparatus 200 for implementing intelligent operations according to the present invention.
As shown in fig. 4, this example includes an intermediate layer (module), an operand (module), a state pool (module), and a drive management (module). The specific structure and function of the various modules of this example are described below:
one, middle layer (contained in the intelligent operation module 203)
The middle layer includes modules such as algorithm analysis (included in the smart computation module 203 and corresponding to the data engine tree generation module), process tree management (included in the smart computation module 203 and corresponding to the computational process tree generation and conversion module), and authority authentication (included in the smart computation module 203), which are started simultaneously after the completion of the computation graph construction (i.e., receiving input from the "front end" in fig. 4).
1. The algorithm analysis module is responsible for converting each element into an instruction recognizable by the engine to construct a data engine tree.
2. The flow tree management module is responsible for carrying out persistent storage on the configured calculation flow tree, and is convenient to modify or serve as a data source to construct a more complex calculation flow tree.
3. The authority authentication module is responsible for verifying data authority, traversing each element, judging whether the user account number for creating the calculation graph has the authority of the element, and finally confirming whether the expected result of the calculation graph constructed by the user can be provided for the user, if the expected result cannot be provided, finishing the ongoing calculation task by asynchronous striking (the calculation task is constructed and starts to run before the authority authentication is finished, and when the authority authentication module determines that the user has no authority, finishing the calculation task, namely, the authority authentication and the task calculation are asynchronously carried out), and returning the authority authentication result to the user.
Second, the operand (contained in the intelligent operation module 203)
The operators include an engine switch (included in the intelligent operation module 203 and corresponding to the data engine tree generation module), a data engine tree (included in the intelligent operation module 203 and corresponding to the data engine tree generation module), and a driving scheduling center (included in the intelligent operation module 203 and corresponding to the scheduling operation module).
1. The engine transformation is responsible for generating the data engine tree through the construction instructions passed back by the front end.
2. The data engine tree is essentially a description of a computing process composed of all computing nodes, computing sequence and computing dependency, for example, the data engine tree can manage the whole computing process by driving a scheduling center to register each computing node in a state pool and through state information of the computing nodes in the state pool. For example, when all children of a compute node in a compute flow graph have been computed, the flow tree derives that the node can begin computing. Parallel calculation among all independent data can be realized through flow tree management, and pipeline management of a data processing process is completed.
3. The driver scheduling center registers all the computing nodes on the data engine tree with the state pool, and calls corresponding drivers from a data source pool (namely, a source pool) or an algorithm pool to start computing according to the attribute of each node.
Third, status pool (included in the intelligent operation module 203, corresponding to the status information of the nodes required by the scheduling operation module and the source of the dependency relationship between each node)
The state pool includes index management, state management, and data management.
1. The index management is responsible for generating unique IDs for all the computing nodes, and the data engine tree manages the computing nodes through the unique IDs.
2. The state management is responsible for maintaining the current states of all the computing nodes (the node states include non-start, activation, success and failure, the non-start indicates that all the dependent nodes of the node are not computed, the activation indicates that the node starts to compute, the success indicates that the node completes computing, the result is stored in the data management and can be called by the upper node, the failure indicates that the node fails computing, and meanwhile, the whole data engine tree is set to be in a failure state to stop all ongoing computing), and the data engine tree schedules computing tasks according to the states.
3. Data management is responsible for maintaining the results of all completed compute nodes, equivalent to a repository of all intermediate and expected data in the pipeline. The multiplexing of the calculation results is convenient (for example, the calculation of a plurality of father nodes depends on one child node for calculation at the same time, the child node only needs to be calculated once, after the calculation is completed, the state is set to be successful in the state pool, and all father nodes can use the calculation results, through the unique number, the multiplexing in a multi-path calculation task can be realized, and in the common extension recursion, the child node needs to be calculated once in each father node), and the calculation process is accelerated.
Fourth, drive management (implicitly contained in the intelligent operation module 203)
Drive management divides all drives into two types: the system comprises a data source drive (corresponding to the intelligent operation module 203 for acquiring input data from a selected data source based on a calculation graph) and an algorithm drive (corresponding to the intelligent operation module 203 for performing intelligent operation based on the input data and the selected intelligent operation and connection relation), wherein the data source drive is stored in a data source pool, the algorithm drive is stored in an algorithm pool, a drive scheduling center issues scheduling tasks, and the drive management can automatically find the corresponding data source or algorithm to be applied to calculation according to the unique ID (fixed value is stored in the drive management center) of the data source or algorithm.
In order to make the specific operation of the above method for implementing intelligent operations according to the present invention more clear to those skilled in the art, the following description will be made with reference to a specific example.
The embodiment of the method for realizing intelligent operation comprises the following steps:
suppose that a user needs to acquire the daily alarm times of a certain network service node, two acquisition modes are desired, and the user can acquire the alarm times through an API (application programming interface) or directly view the alarm times through a webpage chart.
1. The user drags the required data source element to the calculation area through the webpage.
2. The user drags the desired algorithm element to the calculation area.
3. The user links all the data source elements and algorithm elements according to the computational logic and names the desired result elements. (Steps 1-3 correspond to steps S102 to S110 described above)
4. The user drags the presentation element to the calculation area, linking the desired result with the presentation element.
5. The user drags the storage element to the computing area, linking all the data desired to be stored to the storage element.
6. The user configures the detailed parameters of each element.
7. The user saves the computing configuration.
8. The system carries out authority authentication on each computational element, the system generates a flow tree to carry out persistent storage (generating data of a flow tree description file permanently stored at the rear end), and the system converts the flow tree into an engine instruction to send to a construction engine.
9. The construction engine recursively constructs a tree of data engines from instructions, wherein a child node is a source if it is empty and a parent node is expected data if it is empty.
10. And the driving dispatching center sends the data engine tree nodes to the state pool.
11. The state pool computes a unique ID, queries or initializes the state.
12. And the driving scheduling center performs operation according to the flow description scheduling algorithm of the engine tree and the data source.
13. And storing the generated intermediate result in a data management module of the state pool, and changing the state of the corresponding node into a finished state.
14. And (3) solving each calculation node layer by layer (each node and child nodes thereof can be regarded as a subtree, when no shared node exists between the subtrees, the calculation is directly carried out to the level with the shared node without waiting at all), and generating an expected result and returning the expected result to the front end. (Steps 9-14 correspond to the 3 specific steps included in step S108 described above)
According to the above-described aspect of the present invention, the user can use the GUI interface to design (i.e., input or configure) the visual computation graph and obtain various forms of output of desired data. The developer does not need to have the capability of developing by using a script language, only needs to master a common data analysis algorithm and know the design (namely, input or configuration) method of the visual computation graph, and can complete the configuration of the computation graph only by dragging a control by using a mouse in the whole process. The development difficulty is reduced, and the development cost is saved.
According to the technical scheme of the invention, the data among a plurality of platforms can be acquired quickly, efficiently and at low cost, and are aggregated and calculated, so that the efficiency of data-based operation is improved.
According to the technical scheme of the invention, the operation of multi-form output of expected data from the configuration of the visualized computation graph can be realized, the development workload related to data analysis is reduced through automatic acquisition and intelligent operation of the data, and the method has functions (such as visualized computation graph configuration, joint scheduling of algorithm and data source, multiplexing of intermediate results and the like) which are not provided by the existing data integration mode.
The above-described aspects may be implemented individually or in various combinations, and such variations are within the scope of the present invention.
It will be understood by those of ordinary skill in the art that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
Finally, it should be noted that: the above examples are only for illustrating the technical solutions of the present invention, and are not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for implementing intelligent operations, comprising:
providing a GUI;
displaying selectable data sources and selectable intelligent operation operations to a user through the GUI;
receiving, through the GUI, a user-input computation graph including a selected data source, a selected intelligent operation, and a connection relationship between the selected data source and the selected intelligent operation;
acquiring input data from the selected data source based on the calculation graph, and performing intelligent operation based on the input data, the selected intelligent operation and the connection relation to obtain intermediate calculation results and final calculation results of all levels;
displaying the final calculation results and/or intermediate calculation results of each stage to a user through the GUI,
wherein the selectable smart operation comprises: basic mathematical operation, custom domain-specific computation functions, said user input comprising at least one of: clicking and dragging through a mouse, selecting through a shortcut key, and moving and positioning through a direction key.
2. The method according to claim 1, wherein the step of obtaining input data from the selected data source based on the computation graph, performing intelligent computation based on the input data, the selected intelligent computation operation and the connection relationship, and obtaining intermediate computation results and final computation results at each stage comprises:
generating a calculation flow tree corresponding to the calculation graph based on the calculation graph, performing persistent storage, and converting the calculation flow tree into an engine instruction;
constructing a data engine tree based on the engine instructions;
performing scheduling operation based on the state information of the data source node and the intermediate computing node in the data engine tree and the dependency relationship among the nodes to obtain intermediate computing results and final computing results of each level,
wherein, the intermediate calculation results and the final calculation results of each stage can be multiplexed.
3. The method for implementing intelligent operations as claimed in claim 2, wherein the data engine tree is a description of a computation flow including all computation nodes, computation sequences and computation dependencies involved in the selected intelligent operation, the data engine tree registers each computation node in a state pool by driving a scheduling center, the computation flow is managed by computation node state information in the state pool, the computation flow tree obtains information that a computation node in the computation flow can start computation when all child nodes of the computation node have completed computation, and the computation flow tree is also used for managing parallel computation between independent data and pipeline management for completing data processing.
4. The method of implementing intelligent operations of claim 3, wherein the state pool is further used for index management and data management, the index management being used to generate node unique IDs for compute nodes through which the data engine tree manages the compute nodes.
5. The method according to claim 3, wherein the step of performing scheduling operation based on the state information of the data source node and the intermediate computation node in the data engine tree and the dependency relationship among the nodes to obtain the intermediate computation result and the final computation result of each level comprises:
the driving scheduling center issues a scheduling task ID, and the driving management module automatically searches corresponding data and algorithm for calculation based on a selected data source corresponding to the scheduling task ID and the unique ID of the selected intelligent operation to obtain intermediate calculation results and final calculation results of each level.
6. An apparatus for performing intelligent operations, comprising:
the front-end data diagram configuration and display module is used for providing a GUI (graphical user interface), displaying selectable data sources and selectable intelligent operation operations to a user through the GUI, receiving a calculation diagram which is input by the user and comprises a selected data source, a selected intelligent operation and a connection relation between the selected data source and the selected intelligent operation through the GUI, and displaying a final calculation result and/or intermediate calculation results of each level to the user through the GUI;
an intelligent operation module for obtaining input data from the selected data source based on the calculation graph, performing intelligent operation based on the input data, the selected intelligent operation and the connection relation, and obtaining intermediate calculation results and final calculation results of each level,
wherein the selectable smart operation comprises: basic mathematical operation, custom domain-specific computation functions, said user input comprising at least one of: clicking and dragging through a mouse, selecting through a shortcut key, and moving and positioning through a direction key.
7. The apparatus for performing intelligent operations of claim 6, wherein said intelligent operations module comprises:
the calculation flow tree generation and conversion module is used for generating a calculation flow tree corresponding to the calculation graph based on the calculation graph, performing persistent storage and converting the calculation flow tree into an engine instruction;
the data engine tree generating module is used for constructing a data engine tree based on the engine instruction;
a scheduling operation module for performing scheduling operation based on the state information of the data source node and the intermediate calculation node in the data engine tree and the dependency relationship among the nodes to obtain the intermediate calculation result and the final calculation result of each level,
wherein, the intermediate calculation results and the final calculation results of each stage can be multiplexed.
8. The apparatus for implementing intelligent operations as recited in claim 7, wherein the data engine tree is a description of a computation flow including all computation nodes, computation orders and computation dependencies involved in the selected intelligent operation, the data engine tree is used for managing the computation flow by driving a scheduling center to register each computation node in a state pool through computation node state information in the state pool, the computation flow tree is used for obtaining information that the computation node can start computation when all child nodes of the computation node in the computation flow are computed, and the computation flow tree is also used for managing parallel computation between independent data and pipeline management for completing data processing.
9. The apparatus for implementing intelligent operations of claim 8, wherein the state pool is further used for index management and data management, the index management is used to generate node unique IDs for compute nodes, and the data engine tree manages the compute nodes by the node unique IDs.
10. The apparatus for performing intelligent operations of claim 8, wherein said scheduling operations module further comprises:
the driving scheduling center is used for issuing scheduling task ID;
and the drive management module is used for automatically searching corresponding data and algorithm for calculation based on the selected data source corresponding to the scheduling task ID and the unique ID of the selected intelligent operation to obtain intermediate calculation results and final calculation results of each level.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810717155.5A CN110727729A (en) | 2018-06-29 | 2018-06-29 | Method and device for realizing intelligent operation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810717155.5A CN110727729A (en) | 2018-06-29 | 2018-06-29 | Method and device for realizing intelligent operation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110727729A true CN110727729A (en) | 2020-01-24 |
Family
ID=69216795
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810717155.5A Pending CN110727729A (en) | 2018-06-29 | 2018-06-29 | Method and device for realizing intelligent operation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110727729A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112333292A (en) * | 2021-01-06 | 2021-02-05 | 苏州光格设备有限公司 | Electric power internet of things gateway edge calculation method |
| CN116302513A (en) * | 2023-02-28 | 2023-06-23 | 易方达基金管理有限公司 | Quantization factor processing method, quantization factor processing device, computer equipment and readable storage medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120079461A1 (en) * | 2010-09-29 | 2012-03-29 | Rockwell Automation Technologies, Inc. | Extensible device object model |
| CN104268428A (en) * | 2014-10-14 | 2015-01-07 | 国家电网公司 | Visual configuration method for index calculation |
| CN104573063A (en) * | 2015-01-23 | 2015-04-29 | 四川中科腾信科技有限公司 | Data analysis method based on big data |
| CN104915341A (en) * | 2014-03-10 | 2015-09-16 | 中国科学院沈阳自动化研究所 | Visual multi-database ETL integration method and system |
| CN105550268A (en) * | 2015-12-10 | 2016-05-04 | 江苏曙光信息技术有限公司 | Big data process modeling analysis engine |
| CN105912588A (en) * | 2016-03-31 | 2016-08-31 | 中国农业银行股份有限公司 | Visualization processing method and system for big data based on memory calculations |
| CN106202192A (en) * | 2016-06-28 | 2016-12-07 | 浪潮软件集团有限公司 | Workflow-based big data analysis method |
| CN106407413A (en) * | 2016-09-23 | 2017-02-15 | 浪潮软件集团有限公司 | Operation container suitable for distributed algorithm and flow chart creation method |
| CN107526832A (en) * | 2017-09-05 | 2017-12-29 | 江苏电力信息技术有限公司 | A kind of method for building the big data business model that technology is pulled based on the page |
| CN107533453A (en) * | 2015-03-06 | 2018-01-02 | 思科技术公司 | System and method for generating data visualization application |
| CN108170696A (en) * | 2017-06-08 | 2018-06-15 | 国云科技股份有限公司 | A kind of method of data mining |
-
2018
- 2018-06-29 CN CN201810717155.5A patent/CN110727729A/en active Pending
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120079461A1 (en) * | 2010-09-29 | 2012-03-29 | Rockwell Automation Technologies, Inc. | Extensible device object model |
| CN104915341A (en) * | 2014-03-10 | 2015-09-16 | 中国科学院沈阳自动化研究所 | Visual multi-database ETL integration method and system |
| CN104268428A (en) * | 2014-10-14 | 2015-01-07 | 国家电网公司 | Visual configuration method for index calculation |
| CN104573063A (en) * | 2015-01-23 | 2015-04-29 | 四川中科腾信科技有限公司 | Data analysis method based on big data |
| CN107533453A (en) * | 2015-03-06 | 2018-01-02 | 思科技术公司 | System and method for generating data visualization application |
| CN105550268A (en) * | 2015-12-10 | 2016-05-04 | 江苏曙光信息技术有限公司 | Big data process modeling analysis engine |
| CN105912588A (en) * | 2016-03-31 | 2016-08-31 | 中国农业银行股份有限公司 | Visualization processing method and system for big data based on memory calculations |
| CN106202192A (en) * | 2016-06-28 | 2016-12-07 | 浪潮软件集团有限公司 | Workflow-based big data analysis method |
| CN106407413A (en) * | 2016-09-23 | 2017-02-15 | 浪潮软件集团有限公司 | Operation container suitable for distributed algorithm and flow chart creation method |
| CN108170696A (en) * | 2017-06-08 | 2018-06-15 | 国云科技股份有限公司 | A kind of method of data mining |
| CN107526832A (en) * | 2017-09-05 | 2017-12-29 | 江苏电力信息技术有限公司 | A kind of method for building the big data business model that technology is pulled based on the page |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112333292A (en) * | 2021-01-06 | 2021-02-05 | 苏州光格设备有限公司 | Electric power internet of things gateway edge calculation method |
| CN112333292B (en) * | 2021-01-06 | 2021-05-04 | 苏州光格科技股份有限公司 | Electric power internet of things gateway edge calculation method |
| CN116302513A (en) * | 2023-02-28 | 2023-06-23 | 易方达基金管理有限公司 | Quantization factor processing method, quantization factor processing device, computer equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2956078C (en) | Data lineage summarization | |
| CN107430611B (en) | Filtering data lineage graph | |
| US8060544B2 (en) | Representation of data transformation processes for parallelization | |
| CN112199086A (en) | Automatic programming control system, method, device, electronic device and storage medium | |
| US20190146763A1 (en) | Key-based logging for processing of structured data items with executable logic | |
| CN108595316A (en) | Life cycle management method, manager, equipment and the medium of Distributed Application | |
| US10592304B2 (en) | Suggesting application programming interfaces based on feature and context analysis | |
| CN111427865A (en) | Method and device for managing objects of database | |
| CN110727729A (en) | Method and device for realizing intelligent operation | |
| CN116560626A (en) | Data processing method, system, equipment and storage medium based on custom rules | |
| US9720689B2 (en) | Context-specific view of a hierarchical data structure | |
| CN101495978B (en) | Reduction of message flow between bus-connected consumers and producers | |
| CN110019207B (en) | Data processing method and device and script display method and device | |
| US20220083881A1 (en) | Automated analysis generation for machine learning system | |
| CN113495723B (en) | Method, device and storage medium for calling functional component | |
| Behringer et al. | Towards interactive data processing and analytics-putting the human in the center of the loop | |
| US11567758B2 (en) | Configuration properties management for software | |
| Yongsiriwit | Modeling and mining business process variants in cloud environments | |
| Merah et al. | Design of ATL rules for transforming UML 2 communication diagrams into buchi automata | |
| CN112860725A (en) | SQL automatic generation method and device, storage medium and electronic equipment | |
| CN116382666B (en) | Project execution method and system based on artificial intelligence model | |
| US20230316188A1 (en) | Workflow transformation framework | |
| US10956416B2 (en) | Data schema discovery with query optimization | |
| CN114416574A (en) | Test data generation method, device, medium and equipment | |
| CN117032673A (en) | Lightweight coded service arrangement method, system and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200124 |
|
| RJ01 | Rejection of invention patent application after publication |