KR20210057656A - workflow engine framework for cross-domain extension - Google Patents

Abstract

The present invention relates to a workflow engine framework of a cross-domain extension-type. The workflow engine framework of a cross-domain extension type (hereinafter, referred to as "workflow engine framework") executed by a computing device including a processor, comprises: a system configuration unit for generating a plurality of engines in accordance with workflow specifications defined by a user; and a system control unit for controlling the execution of the engines. The engines include a first engine for executing a first workflow of a first domain and a second engine for executing a second workflow of a second domain. The workflow engine framework further comprises an application storage for transmitting a result of processing data performed in the first engine to the second engine for fusion of the first and second domains. Accordingly, a wide range of intelligent services can be provided in various fields.

Description

Workflow engine framework for cross-domain extension

The present invention relates to a configuration of a workflow for fusing different business domains, destination domains, or spatial domains.

The smart manufacturing industry, which has recently attracted attention, is based on a cross-domain intelligent service that derives optimal results by linking different tasks or different purposes performed in different physical spaces.

In order to provide a cross-domain intelligent service, different spatial domains, different business domains, and/or different target domains need to be fused, and at the same time, a workflow configuration adaptive to changes in data generation status or environment is required.

In addition, there is a need for an orchestration technique for organically executing and controlling different spatial domains, different business domains, and/or different target domains.

The present invention is a system for constructing a cross-domain extended workflow that can be commonly applied to different spatial domains, different business domains, and/or different purpose domains by extending the workflow configured within one domain, and its object. Its purpose is to provide an orchestration method.

The above-described and other objects, advantages, and features of the present invention, and methods of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings.

A cross-domain extended workflow engine framework according to an aspect of the present invention for achieving the above object is a cross-domain extended workflow engine framework executed by a computing device including a processor (hereinafter, referred to as'workflow engine framework. '), the workflow engine framework comprising a system configuration unit that generates a plurality of engines according to a workflow specification defined by a user, and a system control unit that controls execution of the plurality of engines, wherein the plurality of The engine includes a first engine executing a first workflow in a first domain and a second engine executing a second workflow in a second domain, and the workflow engine framework includes the first domain and the first workflow. For the convergence of the two domains, it further includes an application storage for transmitting a result of processing data performed by the first engine to the second engine.

A cross-domain extensible workflow engine framework according to another aspect of the present invention is a cross-domain extensible workflow engine framework (hereinafter referred to as'workflow engine framework') executed by a computing device including a processor. The workflow engine framework comprising a system configuration unit that generates a plurality of engines according to a workflow specification defined by and a system control unit that controls execution of the plurality of engines, wherein the plurality of engines comprises a first domain First and second engines for executing the first workflow of; And a third engine executing a second workflow of a second domain, wherein the workflow engine framework includes a result of processing the second engine for fusion of the first domain and the second domain. It further includes an application storage configured as a result of processing of the first workflow and transmitted to the third engine.

A cross-domain extensible workflow engine framework according to another aspect of the present invention is a cross-domain extensible workflow engine framework (hereinafter referred to as'workflow engine framework') executed by a computing device including a processor. The workflow engine framework including a system configuration unit that generates a plurality of engines according to a workflow specification defined by and a system control unit that controls execution of the plurality of engines, wherein the plurality of engines is a first edge A first engine that executes a first workflow of a first edge domain in a node, a second engine that executes a second workflow of a core domain in a core node, and a third workflow of a second edge domain in a second edge node. A third engine to be executed, and the workflow engine framework, for fusion of the first domain and the second domain, transmits a result of processing data performed by the first engine to the second engine. 1 shared storage; And a second shared storage for transferring the processing result of data performed by the second engine to the third engine for fusion of the second domain and the third domain.

According to the present invention, there is an advantage of defining, executing, and controlling a workflow between one or more physical space domains, one or more business domains, and one or more target domains by a unified workflow configuration method.

In addition, according to the present invention, in order to provide an intelligent service that requires smart production, manufacturing, and processing, a cooperative workflow between physical spatial domains, a workflow that can be fused between different tasks or different purpose domains, and data generation status B. By providing orchestration technology to construct a workflow adaptive to changes in the environment and organically execute/control it, it is possible to provide a wide range of intelligent services in various fields such as smart city, smart farm, and smart public.

1 is a block diagram showing components of a cross-domain extended workflow engine framework according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a GUI screen for explaining the configuration of the editing unit shown in FIG. 1.
3 is a block diagram of an engine for executing a cross-domain extended workflow according to an embodiment of the present invention.
4 is a diagram illustrating a configuration of a single engine based on a file system according to an embodiment of the present invention.
FIG. 5 is a configuration diagram of an engine for configuring a cross-domain extended workflow using a single engine configuration based on a file system shown in FIG. 4.
6 is a configuration diagram of an engine showing another example of the configuration of the engine shown in FIG. 5.
7 is a diagram illustrating a method of providing a cooperative intelligent service based on a cross-domain extended workflow according to an embodiment of the present invention.
FIG. 8 is a configuration diagram of an engine (engine container) for defining a converged intelligent service that combines services provided by different business and target domains according to an embodiment of the present invention.
9 is a block diagram of an engine (engine container) for defining a converged intelligent service that combines services provided by different business and target domains according to another embodiment of the present invention.
10 to 14 are diagrams for explaining an internal processing procedure of the engine for the converged intelligence service shown in FIGS. 8 and 9.
FIG. 15 is a diagram for explaining a workflow service scenario related to pipe leakage detection using an engine configuration including an operator having an aggregate function shown in FIG. 13.
FIG. 16 is a diagram for explaining a workflow service scenario related to conversation generation by using an engine configuration configured to include an operator having a lookup table shown in FIG. 14.
17 is a diagram illustrating an orchestration method of executing and controlling one or more workflows for periodically updating a prediction model related to a converged intelligence service according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in the present specification are exemplified only for the purpose of describing the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention are It may be implemented in various forms and is not limited to the embodiments described herein.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by these embodiments.

1 is a block diagram of a cross-domain extended workflow engine framework according to an embodiment of the present invention.

Referring to FIG. 1, the cross-domain extended workflow engine framework 100 of the present invention may be implemented as a software module programmed or executed by a computing device equipped with a processor, or a hardware module including such a software module.

The cross-domain extended workflow engine framework (100, hereinafter, referred to as'engine framework') is basically a front end unit 10, a resource management unit 20, a system configuration unit 30, a system control unit 40, and It includes an application unit 50. In Fig. 1, the editing section 5 is further shown.

The editing unit 5 may be a software module providing a function of creating a structure and specification of a task (workflow) to be executed, or a hardware module incorporating such a software module.

The front end unit 10 includes a software module or such a software module that provides a function of transferring the structure of the work (workflow) and the specification of the work (workflow) created by the editing unit 5 to the engine framework 100. It can be a built-in hardware module.

The resource management unit 20 may be a software module providing a function of registering, maintaining, and managing various resources required to define a workflow, or a hardware module incorporating such a software module.

The resources required to define the workflow are, for example, related to attributes for defining each engine component on the workflow specification (a specification consisting of one or more engines constituting the workflow) that the user will create through the editing unit 10. It may include an attribute specification component, an engine component corresponding to a component of an engine to perform workflow execution, and a previously created workflow specification instance.

The system configuration unit 30 has a function of assembling a workflow attribute specification, dynamically assembling engine components required for workflow execution, and creating an engine instance (hereinafter, used as an'engine') that performs the workflow. It may be a software module to provide or a hardware module incorporating it.

The system control unit 40 is a software module that provides a function of controlling the execution of the engine generated by the resource management unit 20, the system configuration unit 30, the application unit 50, and the system configuration unit 30, or a built-in software module thereof. It can be one hardware module.

The system configuration unit 30 and the system control unit 40 may be integrated into one module, and in this case, the integrated unit may be referred to as an “operation control unit” or an “orchestration unit”.

The application unit 50 manages engine instances created to execute the workflow specification created through the editing unit 5, stores data collected or processed according to the execution of the engine instances, and controls access to the stored data. It may be a software module providing a function or a hardware module incorporating it.

In the workflow engine framework 100, a user defines a workflow composed of one or more engines to create a desired system. In this case, the workflow consists of definitions for one or more engines.

The definition of an engine may be a combination of an engine component container to contain engine components and an engine component to be contained in the engine component container.

The engine component container may be a combination of one or more reader components, writer components, runner components, operator components, and controller components.

One or more input device components, writer components, executor components, handler components, and controller components are each made of a combination of a property specification component for defining properties representing the properties of the component and an execution component corresponding to the actual implementation of the components.

Execution components correspond to classes such as Java, C++, etc., and attribute specification components correspond to constructor parameters that can be included in the constructor of the class or classes that contain constructor parameters.

In the present invention, by defining a workflow for one or more engines created in this manner, a workflow that can be fused between different tasks or/and different target domains and a workflow that is adaptive to changes in data generation state or environment can be configured. I can.

Hereinafter, each of the components shown in FIG. 1 will be described in more detail.

The editing unit 5 defines a detailed domain for a desired task and a task structure (workflow structure) to be processed for each domain.

In addition, the editing unit 5 provides a function of defining the workflow execution flow by selecting the engine component(s) according to the defined work structure, and creating a specification (work specification or workflow specification) of each engine component.

To this end, the editing unit 5 can inquire the attribute specification component and the engine component from the framework (especially, the resource management unit 20).

Here, the'work structure' is a structured process of data collection, data processing, and learning from a specific IoT platform, and sequentially processing them may correspond to a workflow execution flow.

'Specifications of components (task specification or workflow specification)' means from which device to collect data, and how to connect the data in order to perform work on where to get data from, what process to go through, and where to export it. It means specific definitions for each element component, such as what is it, whether to put the received data in memory or to put it in the storage, what is the memory information, where is the storage location, etc.

Users (data scientists, model developers, business developers, etc.) define an engine component constituting a workflow and a property specification component that can define parameters that determine the characteristics of the engine component according to the convention through the editing unit (5). You can also define and edit pairs of component and attribute specifications.

An example of a GUI screen 71 for explaining the configuration of the editing unit 5 is shown in FIG. 2.

2, the GUI screen 71 includes a function menu 72, an engine type selection unit 73, a component selection unit 74, a component property inquiry/editing unit 75, and a workflow instance storage/retrieval unit ( 76).

The function menu 72 is a menu for selecting various functions that can be provided by the editing unit 5, for example, New (creating a new workflow), Open (recalling a saved workflow), and Save (saving a workflow). It can be composed of a menu that selects functions such as, Run (running workflow), Result (viewing execution results), and Help (help).

The engine type selection unit 73 provides a function of presenting various engine types and allowing a user to select necessary ones among them.

As the engine type, for example, there may be a real-time stream processing engine, a batch analysis engine, an on-demand data processing engine, an evaluation engine, a stream machine learning prediction engine, and a demand fusion serving engine.

The component selection unit 74 provides a list of various engine components for each component type so that a user can select a component type and an engine component for each type. Table 1 below is an example of a component type and an engine component list presented by the component selection unit 74.

Component type Engine component Reader (Input Method Editor) FileReader
HttpServerReader
Kafka Reader
MongodbReader
... Writer (writer) FileWriter
KafkaWriter
MongodbReader
... Controller SparkSessionController
SparkSessionStreamController
... Runner SparkRunnerTensoflowRunner
... Operator MinMaxScaler
Aggregator
...

The component property selection/editing unit 75 presents properties of the engine component selected by the component selection unit 74, and the user can search for properties and select and edit them.

The workflow instance selection unit 76 provides a function of displaying a list in which a previously produced workflow is stored. Among these, a workflow desired to be recycled may be selected and re-edited, or an execution request may be made to the framework 100 as it is. The unit of recycling may be the entire workflow, or it may be reused by a single engine included in the workflow and used for editing or execution.

When a workflow specification file is created in the editing unit 5, the workflow specification file is submitted (input) to the system configuration unit 30 in the engine framework 100. In this case, the front end unit 10 may serve to receive the workflow specification and transmit it to the system configuration unit 30.

The front end unit 10 serves to mediate execution of a process according to a user request, and serves to respond to various requests such as user management or storage management.

The front end 10 may be, for example, a general application server, a web application server providing a web-based REST API, or a system including a general socket communication-based listener module.

In some cases, the front end 77 may be executed on a network different from that of the editing unit 10 or the back end constituting the framework 100.

Referring back to FIG. 1, the resource management unit 20 in the engine framework 100 provides a function of registering, maintaining, and managing various resources required to define a workflow, and at the same time managing components required for execution of the workflow. Provides a function.

To this end, the resource management unit 20 includes an application asset registration management unit 22, an engine component unit 24, and a workflow instance specification management unit 26.

The application asset registration management unit 22 registers, maintains, and manages resources or additional information to be used to construct a workflow.

The application assets managed by the application asset registration management unit 22 are, for example, a dataset to be used for machine running or data analysis, a training code for training machine learning, and generated through training. Machine learning/deep learning based prediction models or other prediction algorithms, algorithms or codes implemented in a specific language such as Python, container images, service APIs that exist inside or outside, linked platform information, authentication for linked systems It includes information necessary for (authentication), data origin information, data subscription information, etc.

The engine component unit 24 provides a space in which the attribute specification component and the engine component managed by the resource management unit 20 are physically or virtually stored.

The workflow instance specification management unit 26 manages a previously created and stored workflow specification instance. The workflow specification instance management unit 26 stores and manages the workflow specification instance to be used later at the request of the editing unit 5 (for example, through the workflow instance selection unit 76 of FIG. 2 ).

Other than that, although not shown in the drawings, the resource management unit 20 performs the attribute specification component management unit and the attribute specification component management unit that manages (updates) the attribute specification components including attribute specifications that determine component properties or attributes of the workflow instance It may be configured to further include an engine component management unit for managing (including updating) components (engine components) for and the list thereof.

The system configuration unit 30 provides a function of creating a component necessary for execution of the created workflow. The system configuration unit 30 constructs necessary engine component containers according to the work structure (workflow structure) and work specification (workflow specification) input from the front end 10 to create an engine instance.

To this end, the system configuration unit 30 includes a workflow configuration unit 34 and a log management unit 36.

The workflow configuration unit 34 includes a work structure (workflow structure) and a work specification (workflow structure) received from the editing unit 5 through the front end unit 10, based on the resources managed by the resource management unit 20. A series of engine components are configured through the process of binding the attribute specification component corresponding to the specification) and the engine component, and a workflow execution platform based on the series of engine components is formed.

The attribute specification component bound to the engine component includes, for example, Google's Protobuf message object, Scala's Case Class, Java's Property object, and the like.

The workflow configuration unit 34 executes the workflow by binding the engine component constituting the workflow and the property specification component defining the parameters that determine the characteristics of the engine component, and binding the instances of the generated engine components to the engine container. A workflow execution instance unit 52 that includes a series of engine instances for dynamic is configured.

The engine created to execute a workflow is distributed and executed on other computing devices connected via a network as a single independent program package.

The log management unit 36 manages log information generated according to the execution of the engine constituting the workflow, and provides the log information to the requestor.

The system control unit 40 drives engine instances (engines) generated by the system configuration unit 30 and also serves to terminate the driven engine instances (engines). That is, when the system control unit 40 requests the workflow execution instance unit 52 to execute the workflow, the workflow is executed.

In addition, the system controller 40 may control a specific engine generated to be executed in a specific domain and an engine generated to be executed in a specific domain different from each other to be executed in a pipeline manner. This engine configuration will be described in detail below.

The application unit 50 includes a workflow execution instance unit 52, an application storage 54, and an application access control unit 56.

The workflow execution instance unit 52 is configured to include instances created to execute the workflow specification (spec) defined through the editing unit 5.

The application storage 54 provides a service for storing data collected or processed on the workflow. In this case, the application storage unit 54 may be in or outside a specific domain.

The application access control unit 56 controls the access of the engines constituting the workflow of different domains to the application storage 54.

The engine framework 100 according to the present invention extends the configuration and procedure of the workflow configured in one domain using the application storage 54 and the application access control unit 56, so that different spatial domains and different tasks You can configure a cross-domain extensible workflow that can be commonly applied to domains or different target domains. An example of a cross-domain extension workflow configuration using the application storage 54 and the application access control unit 56 will be described in detail below.

In addition, the workflow engine framework 100 according to the present invention, a core node, one or more hierarchical nodes such as a cloud, a plurality of edge nodes of the same layer, or nodes within a group consisting of a core node cooperate to achieve one purpose. Provides a method of constructing and executing a workflow to achieve.

In addition, the workflow engine framework 100 according to the present invention may configure and execute a workflow for achieving an adaptive intelligent service workflow execution such as periodic machine learning model tuning and serving model update.

In addition, the workflow engine framework 100 according to the present invention may configure and execute a workflow for a service pipeline for generating a complex API by connecting one or more APIs.

Hereinafter, an engine configuration (engine container configuration) for configuring a cross-domain extended workflow according to an embodiment of the present invention will be described.

3 is a block diagram of an engine for configuring a cross-domain extended workflow according to an embodiment of the present invention.

Referring to FIG. 3, in order to configure a cross-domain extended workflow according to an embodiment, an application storage 54 and an application access control unit 56 shown in FIG. 1 are used.

The first engine 60 configures (executes) a first workflow of domain #1, and the second engine 70 configures (executes) a second workflow of domain #2.

The first engine 60 includes a controller 61, a reader 62, a plurality of unit operators 63, 64 and 65, a runner 66 and a writer. ) (67). The second engine 70 is configured to include a controller 71, an input unit 72, a plurality of unit processors 73, 74 and 75, an executor 76 and an output unit 77.

The readers 62 and 72 have a function of reading data from a data source under the control of the controller 61. Here, the data source may be a source of various logical driver concepts such as in-memory buffer/cache, file system, messaging system, database, and network.

The writers 67 and 77 have a function of writing data to a data destination (not shown) of any one type of in-memory buffer, cache memory, file system, messaging system, database, and network driver.

The plurality of unit processors 63, 64 and 65/73, 74 and 75 are dependently connected, and the input unit 62 processes the data read from the data storage in a pipelining manner. Here, the data processing performed in each unit processor includes, for example, data purification, consolidation, reduction and/or conversion processing. Data processing processes such as refining, consolidation, reduction, and transformation are introduced in detail in Data Mining Technology, and descriptions thereof are replaced with known technologies.

The executors 66 and 76 may be all programs required for data processing or external platforms/frameworks. The executor (66 or 76) is, for example, Spark for processing big data, Caffe for deep learning analysis, a deep learning platform such as Tensorflow, a connector for interworking with or executing a knowledge processing engine such as Jena, It may be a REST server or a session manager.

Controllers 61 and 71 control inputs 62 or 72, multiple unit processors 63, 64 and 65 or 73, 74 and 75, executors 66 or 76, and outputs 67 or 77.

Controller 61 or 71, as shown in Figure 3, each engine is an input device (62 or 72), a plurality of unit processors (63, 64 and 65 or 73, 74 and 75), an executor (66 or 76) , When configured with an output unit (67 or 77), a sequential processing method that transfers data processed by each configuration to the next configuration according to a pipelining method, a simultaneous processing method that processes data simultaneously in each configuration, a sequential processing method and simultaneous processing The above configurations can be controlled by a method that combines methods or the like.

The first engine 60 processes data read from a data source through the input device 62, and outputs the processed data to the application storage 54 through the output device 67. To do. At this time, the writer 67 has access rights and means for accessing the application storage 54.

The writer 67 can be assigned access rights to access the application storage 54 through authentication, and to be assigned the access rights, the writer 67 is, for example, OAuth ( A token that can be accessed through an API to the application storage 54 can be used based on an authentication method such as Open Authentication (open authentication).

The input device 72 of the second engine 70, like the output device 67 of the first engine 60, has the same access rights as a token, and can access the application storage 54 based on these access rights. have.

The input unit 72 of the second engine 70 reads data processed by the first engine 60 from the application storage 54, and a plurality of unit processors 73, 74, and 75 of the second engine 70 ) Is a second workflow for performing a data processing process on the data processed by the first engine 60 in a pipelined manner, and outputting the processing result to a data destination through the output unit 77. Run.

The application access control unit 56 performs an authentication process for the first engine 60 or the output unit 67 and the second engine 70 or the input unit 72, and according to the authentication result, the first engine 60 or It performs a role of allocating access rights to the application storage 54 to the output unit 67 and the second engine 70 or the input unit 72.

As described above, in an embodiment of the present invention, in order to configure a cross-domain extended workflow, the first engine 60 constituting the first workflow of domain #1 and the second engine constituting the second workflow of domain #2 ( 70) is organically combined through the application storage (54 in FIG. 1), it is possible to execute a workflow capable of fusion between different spatial domains, different business domains, and/or different target domains.

Hereinafter, a configuration of an engine (engine container) constituting a cross-domain extended workflow based on a file system according to another embodiment of the present invention will be described. For better understanding, a description will be made of the configuration of a single engine that moves, copies, and deletes data based on a file system.

4 is a diagram illustrating a configuration of a single engine based on a file system according to an embodiment of the present invention.

4, a file system-based engine (or engine container) according to an embodiment of the present invention includes a controller 81, an input device 82, a plurality of unit processors 83, 84, and 85, and an executor ( 86) and an output unit 87.

The input device 82 reads file lists from a data source under the control of the controller 81.

The unit processors 83, 84, 85 process the data processing process for the file list input from the input unit 82 in a pipeline manner, and then input the processed file lists to the output unit 87.

For example, the unit processor 83 filters to select a file to be moved to a data destination from the list of files input from the input unit 82, and the unit processor 84 is filtered by the unit processor 83. The file name of the file is converted into a new file name, and the unit processor 85 changes different directory paths of files included in the file list processed by the unit processors 83 and 84 into the same single directory path.

The output unit 87 moves or copies a list of files processed by the unit processors 83, 84, and 85 and files corresponding to the list of files from a data source to a data destination according to a defined movement path. It can be defined to have a function.

In addition, the output unit 87 deletes the file list processed by the unit processors 83, 84 and 85 and the files corresponding to the file list without moving or copying from the data source to the data destination. It can be defined to have a function.

FIG. 5 is a configuration diagram of an engine for configuring a cross-domain extended workflow using a single engine configuration based on a file system shown in FIG. 4, and FIG. 6 is an engine showing another example of the engine configuration shown in FIG. It is a configuration diagram of.

First, referring to FIG. 5, the first and second engines 90 and 110 configure (execute or perform) the first workflow of domain #1, and the third engine 120 is Configure (execute or execute) the second workflow of domain #2 that defines the space. At this time, the application storage 54 and the application access control unit 56 are executed in domain #2.

Each of the first to third engines (90, 110, 120) is a controller (91, 111, 121), input devices (92, 112, 122), unit processors (93, 113, 123), executors (94, 114, 124). ) And an output unit 95.

At least one of the first to third engines 90, 110, and 120 is configured as a single engine based on the file system shown in FIG. 4. That is, all of the first to third engines 90, 110, and 120 may be configured as a single engine based on a file system, or only partially may be configured as a single engine based on a file system. Of course, all of the first to third engines 90, 110, and 120 may be configured with a single engine 60 or 70 shown in FIG. 3.

In the engine configuration of FIG. 5, the second engine 110 is configured as a single engine based on a file system, and the first and third engines 90 and 120 are the single engines 60 or 70 (' It consists of a single engine based on data').

The description of the components included in the second engine 110 is replaced by a description of each component included in the single engine based on the file system of FIG. 4, and is included in each of the first and third engines 90 and 120 The description of the configured configurations is replaced with a description of each configuration included in the data-based single engine 60 or 70 shown in FIG. 3.

In addition, unlike FIGS. 3 and 4 in FIG. 5, each engine includes only one unit processor 93, 113, and 123, but this is for simplification of the drawing and is not intended to limit the present invention.

Hereinafter, a method of transmitting a processing result of the first workflow of domain #1 to the domain #2 for fusion of the first workflow of domain #1 and the second workflow of domain #2 will be described.

First, the first engine 90 of domain #1 performs a data processing process on the data (file list and file) read from the data source (A), and the processed data (file list and file) is converted into intermediate data. Output to destination (B).

Then, the second engine 110 reads the file list from the intermediate data destination (B), extracts (filters) the file list to be delivered to domain #2 from the read file list, and is executed in the second domain. Output to the application storage 54.

At this time, the application access control unit 56 of domain #2 checks whether the output unit 115 of the second engine 100 running in domain #1 has access rights to access the application storage 54, and has access rights. If it is verified that there is, the output unit 115 of the second engine 110 accesses the application storage 54 and a list of files extracted (filtered) by the unit processor 113 (hereinafter, the processing result of the first workflow ) To the application storage 54.

Then, the third engine 120 of domain #2 reads the processing result of the first workflow processed by the second engine from the application storage 54, and defines the processing result of the read first workflow. The processed data is processed and output to the final data destination (C).

Next, the engine configuration of FIG. 6 has the biggest difference from the engine configuration of FIG. 5 in that the application storage 54 and the application access control unit 56 are executed in domain #1. In the following description, it is assumed that the output device of the first engine 90 and the input device 112 of the second engine 110 have access rights to the application storage 54.

6, the first engine 90 of domain #1 performs a data processing process on the data (file list and file) read from the data source (A), and then returns the processing result to the domain #1. The application storage 54 is accessed and output to the application storage 54.

Thereafter, the second engine 110 of the domain #2 accesses the application storage 54 of the domain #1 through the input device 112, and the processing result processed by the first engine 90 from the application storage 54 Read.

Thereafter, the second engine 110 extracts (filters) a required file list from among the file lists included in the processing result read from the application storage 54, and then transfers the extracted file list and files to the intermediate data destination (B). ) To move or copy.

Thereafter, the third engine 120 reads the moved or copied data (file list and file) to the intermediate data destination B through the input device 122, and then processes the data in a predetermined manner through the unit processor 123. After performing the process, it is output to the final data destination (C) through the output unit 125.

7 is a diagram illustrating a method of providing a cooperative intelligent service based on a cross-domain extended workflow according to an embodiment of the present invention.

Referring to FIG. 7, a cooperative intelligent service according to an embodiment of the present invention is a cross domain for achieving one purpose by cooperating with an edge domain and a core domain based on the engine configuration described in FIGS. 3 to 6. Build and run the extensible workflow.

In order to provide collaborative intelligence services, the system environment in which the workflow engine framework that can configure and execute the workflow of edge domain #1, the workflow of edge domain #2, and the workflow of core domain is installed and operated. Is assumed.

The first edge node 170 of edge domain #1 executes workflow #1 for collecting and processing data from the data origin 160. Here, the first edge node 170 may be referred to as an engine that executes the first workflow of the edge domain # in the first edge node 170.

Workflow #1 is moved to the shared storage 180 (application storage) to share the engine processing the data collected from the data source 160 and the processing result (data file) with the core node 190 of the core domain. Or, it is composed of an engine based on the file system to be copied.

The core node 190 of the core domain uses the processing result (data file) by the first edge node 170 from the shared storage 180 as training data, and performs workflow #2 for fine tuning the learning model. Run. Here, the core node 190 may be referred to as an engine that executes the workflow #2 of the core domain in the core node 190.

Workflow #2 is an engine that reads the processing result (data file) of workflow #1 from the shared storage 180 as training data, and an engine that performs learning for fine tuning using the read training data (data file). , It is composed of an engine that moves the new version of the learning model generated (updated) according to the learning result to the shared storage 200.

The second edge node 210 of edge domain #3 executes a workflow #3 of performing a prediction service using a new version of the learning model loaded from the shared storage 200. Here, the second edge node 210 may be referred to as an engine that executes workflow #3 of edge domain #3.

Workflow #3 is an engine that moves a new version of the learning model from the shared storage 200 to the second edge node, and serves to provide a prediction service using the new version of the learning model moved from the shared storage 200. It consists of a serving engine that performs.

As such, it is possible to provide a cooperative intelligent service by performing a workflow in cooperation between the edge node and the core node.

8 is a block diagram of an engine (engine container) for defining a converged intelligent service in which services provided by different business and target domains are fused according to an embodiment of the present invention.

Referring to FIG. 8, an engine 300 for defining a converged intelligent service that combines services provided by different business and purpose domains includes a controller 301, a plurality of operators 302 to 306, and an executor 307. It is configured to include, and these components are similar to those described with reference to FIGS. 3 and 4.

However, the engine 300 of FIG. 8 is configured for a function of serving service execution in a workflow service, and in this regard, may be referred to as a serving engine.

The controller 301 controls the operation of a number of operators 302-306 and executors 307.

The plurality of operators 302 to 306 interlock with a plurality of application programming interfaces (APIs) 410 to 430 and the knowledge base 440, respectively. Each operator can be viewed as a configuration similar to the unit processor illustrated in FIGS. 3 to 6.

The executor 307 may be a Representational State Transfer (REST) server or interworking with a REST server, and sequentially calls the operators 302 to 306 according to the request of the client 400.

When calling the executor 307, each operator calls the REST API using the URL information transmitted from the executor 307 or the URL information they know, and processes the data provided through the called REST API. Then, it operates in a pipeline method that passes it to the next operator.

The processing result of the final operator 306 is returned back to the client 400 through the executor 307.

9 is a block diagram of an engine (engine container) for defining a converged intelligent service in which services provided by different business and target domains are fused according to another embodiment of the present invention.

Referring to FIG. 9, the engine 300' according to another embodiment of the present invention further includes a reader 309 and a writer 310, as shown in FIG. It is the same as the engine configuration.

The engine 300' includes, for example, an input device 309 that reads data from a message queue, Kafka, etc., and operators 302 to 306 that perform ontology-based inference on the read data. , By being configured to include an executor 307 that serves the reasoning result through a REST interface and an output device 310 that outputs the reasoning result to a message queue, Kafka, etc., it is possible to provide a converged intelligence service. .

10 to 14 are diagrams for explaining an internal processing procedure of the engine for the converged intelligence service shown in FIGS. 8 and 9.

First, the internal processing procedure of the engine for the converged intelligence service of FIG. 10 is an internal processing procedure performed in an independent case where the processing results of operators do not affect each other.

First, the executor 307 in the REST server calls the first operator 302 according to the fusion intelligent service request from the client 400 (2), and the called first operator 302 is the corresponding RESTful API 450 ( Or, it transmits the query data (③) to an external server), and receives the response data (④) from the RESTful API (450).

The processing result of the first operator 302 is an input field in which an input parameter used for a query is recorded, a data field in which data is recorded, and routing information indicating a query address is a route field and It consists of a data format consisting of an output field in which the response data (④) is recorded.

The arrangement order of these fields may be different for each operator, and information recorded in the input field, data field, and route field may be used as main information of the query data.

When the first operator 302 completes the query/response procedure, the second operator 303 delivers the query data (⑤) to the corresponding RESTful API 460, and the response data (⑥) to the corresponding RESTful API (460). ) And outputs the processing result reflecting the received response data.

When the second operator completes the query/response procedure, the third operator 304 delivers the query data (⑦) to the RESTful API 470, and receives the response data (⑧) from the RESTful API 470, Outputs the processing result reflecting the received response data.

When the query/response procedures of all operators 302, 303, and 304 are completed, the executor 307 collects processing results output by each operator and returns them to the client 400 (⑨).

The internal processing procedure of the engine for providing the converged intelligence service of FIG. 11 is the internal processing procedure shown in FIG. 10 in that each operator supports a chaining method in which the processing result of the previous operator is reflected in its own query. There is a difference from the processing procedure.

Referring to FIG. 11, the first operator 302 called by the executor 307 according to the service request (①) of the client 400 configures input parameters, routing information, and data as query data, and a corresponding RESTful API Pass it to (450) (③).

The first operator 302 receives response data for the query data, records the response data in an output field of the query data, and updates the output field.

When the update of the output field is completed, the first operator 302 outputs a processing result consisting of input parameters, routing information, and response data to the second operator 303.

The second operator 303 updates the query data of itself 303 using the processing result (input parameters, routing information, data, and response data of the RESTful API 450) output from the first operator 303.

Thereafter, the second operator 303 transmits the updated query data ⑤ to the corresponding RESTful API 460 (or external server), and the updated query data ⑤ from the RESTful API 460 It receives response data (⑥) and updates the output field with the response data (⑥).

When the update of the output field is completed, the second operator 303 outputs the processing result to the third operator 304.

Similarly, the third operator 304 updates its own query data by using the query data included in the processing result input from the second operator 303, and transfers the updated query data (⑦) to the corresponding RESTful API 470. It transmits, receives response data (8) for the updated query data, and updates an output field with the response data.

When the update of the output field is completed, the third operator 304 transfers the processing result to the executor 307, and the executor 307 returns the final result (⑨) transferred from the third operator 304 to the client. .

The internal processing procedure of the engine for providing the converged intelligence service of FIG. 12 is an internal processing procedure performed when an input device for reading a data field and a writer for outputting a processed data field are further included.

The client 400 transmits a service request message including only the remaining data (Input/Route) excluding the actual data included in the query data to the executor 307 (①).

The executor 307 outputs data (Input/Route) included in the service request message to the input device 309.

The input device 309 (Reader) reads data from the data source 12 and updates the data field. At this time, the data read from the data source 12 by the input device 309 may have an arrangement that specifies the data for each operator.

Next, the operator 303 generates query data including data read from the input device 309 (Reader) and delivers it to the RESTful API 460 (⑤), and the response data for the query data is transmitted from the RESTful API 460. Receive (⑥).

After the inquiry and response procedure of the operator 303 is completed, the operator 303 outputs the processing result including the inquiry data and the response data to the output unit 310 (7).

The output unit 310 outputs a part or all of the processing result received from the operator 303 to the data destination 13.

Finally, the executor 307 transmits the processing result of the operator 303 received from the output device 310 to the client 400.

The internal processing procedure of the engine for providing the converged intelligence service of FIG. 13 is an internal processing procedure when the last operator 304 performs a function of aggregating outputs.

In response to the client's request (①), the executor 307 sequentially calls the operators 302 and 303 (②), and the sequentially called operators 302 and 303 correspond to the corresponding RESTful APIs 450 and 460 ), each of the query data is delivered (③, ⑤).

Each operator receives the response data for the query data (④, ⑥), and updates the output field using the response data.

The last operator 304 having the output aggregation function uses the aggregation method (average, sum, And, Or, Min, Max, Percentile, etc.) defined as a property to be included in the processing results of the previous operators (302, 303). The response data (4, 6) of the data 450 and 460 are aggregated, and an output field included in the processing result of the self 304 is updated with the aggregated result.

The executor 307 returns the processing result aggregated by the last operator 304 to the client 400 (⑨).

The internal processing procedure of the engine for providing the converged intelligence service of FIG. 14 is an internal processing procedure in the case of including operators 303 having a lookup table for a URL indicating a query address.

First, according to the client's request (①), the first operator 302 called by the executor 307 delivers the query data to the corresponding API 450 (or external server), and the API 450 (or external server) Server) to include the response data (④) for the query data, and transmits the result to the second operator 303.

The second operator 303 has a lookup table for URLs, and uses the data (response data (④) of 450) recorded in the output field constituting the processing result of the first operator 302 as an index, and the output field The Url value mapped to the recorded data (response data (4) of 450) is extracted from the lookup table.

Thereafter, the second operator 303 constructs the processing result to include the Url value extracted from the lookup table, and transmits it to the third operator 304. Here, the Url value extracted by the second operator 303 using the lookup table is used for updating routing information of the third operator 304. That is, the third operator 304 transmits the query data of itself 304 to the external servers 460 and 470 indicated by the Url value extracted by the second operator 303.

The last operator 304 transmits the query data to the APIs 460 and 470 using the routing information updated by the extracted URL, receives the response data, and executes the processing result including the response data. ).

The executor 307 returns the processing result transmitted from the last operator 304 to the client 400 (⑨).

FIG. 15 is a diagram illustrating a workflow service scenario related to pipe leakage detection using an engine configuration including an operator having an aggregate function shown in FIG. 13.

The workflow service scenario related to leak detection shown in FIG. 15 consists of an engine that provides a response by configuring one or more intelligent APIs in an ensemble manner.

The service client 500 calls an API for predicting leakage by collecting sound signals of the pipe, and the executor 600 serving the called API sequentially calls the operators 610, 620, and 630.

Each of the operators 610, 620, and 630 called by the executor 600 constructs query data related to the pipe leak and transmits it to their respective external servers 710, 720, 730.

The external servers 610', 620', and 630' determine whether there is a leak by using a predictive model such as a leak detection algorithm, and the result of the determination is composed of response data to each of the operators 610, 620, and 630. Deliver.

The last operator 640 aggregates the processing results of each operator 610, 620, 630, and finally determines whether there is a pipe leak using the aggregated results, and provides the final determination result through the executor 600. It returns to the client 500.

FIG. 16 is a diagram for describing a workflow service scenario related to conversation generation by using an engine configuration configured to include an operator having a lookup table shown in FIG. 14.

Referring to FIG. 16, referring to FIG. 16, the first operator 710 in the engine receives a sentence from the client 500 through the executor 700, and generates query data for requesting topic classification of the received sentence. Thus, the query data is transferred to an external server 710' that serves a classifier model for topic extraction, and response data including topic information extracted by the classifier model from the external server 710' Receive.

The first operator 710 configures the processing result to include the response data received from the external server 710', and transmits it to the second operator 720.

The second operator 720 extracts the URL address mapped to the classification result (topic information) included in the processing result transmitted from the first operator 710 from the lookup table, configures the processing result to include the URL address, and configures the last operator. Forward to (730).

The last operator 730 generates query data using the URL address included in the processing result of the second operator 720, and transmits the query data to the external servers 720' and 730' indicated by the URL address. In this case, the external servers 720 ′ and 730 ′ may serve a conversation generation model that predicts a response sentence.

The external servers 720 ′ and 730 ′ configure the response sentence as response data and deliver it to the last operator 730, and the last operator 730 transmits the response sentence to the service client through the response sentence executor 700 included in the response data. 500).

As described above, when the service client 500 inputs a sentence, the address of the sentence generation API according to the topic classification result, which is the result of the initial operator's query, is selectively determined, a response sentence suitable for the context of the sentence is generated, and the response is service It is returned to the client 500.

17 is a diagram illustrating an orchestration method of executing and controlling one or more workflows for periodically updating a prediction model related to a converged intelligence service according to an embodiment of the present invention.

In the orchestration method of the present invention, one or more workflow pipelines for updating a prediction model in service by performing real-time processing of time series data and periodically fine-tuning a time series prediction model are continuously, sequentially, and periodically executed. As a result, it is possible to provide continuous prediction services.

Referring to FIG. 17, workflow #1 for real-time data processing (①) is configured. It runs on a real-time stream framework.

Workflow #1 is a real-time stream engine in the form of a mini batch that performs feature engineering to create training data (training data) using the engine for real-time purification of data, and the refined data delivered from the engine. It is configured by a time series stream file generation engine for generating the learning data as a file stream.

Workflow #2 for real-time inference serving (②) is by engine #1 and engine #1 that receive real-time data processed by execution of workflow #1 in a stream manner and perform inference for prediction. It is composed of engine #2 that constructs an evaluation dataset to evaluate the accuracy of prediction results, and generates the evaluation dataset in the form of a file stream.

Engine #2 may select specific prediction results from among prediction results performed by engine #1 in various ways, and configure the selected specific prediction results as an evaluation dataset. The results inferred by the engine #1 are output to a stream-only message queue for delivery to the client.

Workflow #3 is a workflow for periodically fine-tuning a predictive model using new data.

This workflow #3 is an engine #1 that extracts time series data for fine-tuning at a specific reference time from the list of real-time streaming files generated by workflow #1, and the ground-truth of prediction results for verification from time series data. It consists of engine #2 that automatically maps values, learning engine #3 for fine-tuning model parameters, and engine #4 that deletes temporarily created files for training.

Workflow #4 periodically evaluates the prediction service using the evaluation dataset generated by engine #2 of workflow #2.

This workflow #4 is the engine #1 that extracts the prediction result at a specific point in time from the evaluation dataset generated by the engine #2 of workflow #2, the label value obtained by labeling the evaluation dataset, and the specific point in time. Engine #2 that compares and evaluates the prediction results extracted from engine #2, engine #3 that determines whether to update the prediction model based on the evaluation result by engine #2, and deletes the evaluation dataset after evaluation is completed It consists of #4.

This orchestration method maintains a real-time streaming processing pipeline by running workflows #1 and #2 together for real-time prediction.

Workflows #3 and #4 are driven by specific cycles, fine-tuning the learning model from the data collected according to the cycle, and periodically evaluate whether or not the optimal performance is achieved.

The cross-domain extended workflow engine framework according to the present invention may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component.

For example, the components described in the embodiments are a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU). ), a microprocessor, or any other device capable of executing and responding to instructions.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to operate as desired or processed independently or collectively. You can command the device.

Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave.

The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

So far, the present invention has been looked at based on examples. Those of ordinary skill in the art to which the present invention pertains will appreciate that the present invention can be implemented in variously modified or modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an exemplary point of view for description and not a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims (17)

As a cross-domain extended workflow engine framework (hereinafter referred to as'workflow engine framework') executed by a computing device including a processor, a system configuration unit that generates a plurality of engines according to a workflow specification defined by a user And, as the workflow engine framework including a system control unit for controlling the execution of the plurality of engines,
The plurality of engines include a first engine executing a first workflow in a first domain and a second engine executing a second workflow in a second domain,
The workflow engine framework,
Application storage that delivers the result of data processing performed by the first engine to the second engine for the fusion of the first domain and the second domain
Cross-domain extensible workflow engine framework that further comprises. In claim 1,
The first engine,
The application of an input device that reads data from a data source, a plurality of unit processors that process the data read by the input device, and the processing result (hereinafter referred to as'first processing result') processed by the plurality of unit processors. Contains a writer to output to the storage,
The second engine,
An input device that reads the first processing result from the application storage, a plurality of unit processors that process the first processing result read from the input device, and a second processing result processed by the plurality of unit processors are a data destination. Output to
Cross-domain extensible workflow engine framework that includes a. In claim 1,
The cross-domain extended workflow engine framework further comprises an application access control unit that allocates access rights to the application storage to the first and second engines through an authentication process for the first and second engines. In paragraph 3,
The authentication process is a cross-domain extended workflow engine framework that is an authentication process based on open authentication. As a cross-domain extended workflow engine framework (hereinafter referred to as'workflow engine framework') executed by a computing device including a processor, a system configuration unit that generates a plurality of engines according to a workflow specification defined by a user And, as the workflow engine framework including a system control unit for controlling the execution of the plurality of engines,
The plurality of engines,
First and second engines executing a first workflow in the first domain; And
A third engine that executes a second workflow in the second domain,
The workflow engine framework,
For fusion of the first domain and the second domain, an application storage configured to configure the processing result of the second engine as the processing result of the first workflow and deliver it to the third engine
Cross-domain extensible workflow engine framework that further comprises. In clause 5,
The first engine,
A first input unit for reading data from a data source, a first unit processor for processing the data read by the first input unit, and the processing result (hereinafter referred to as'first processing result') processed by the first unit processor Including a first output unit for outputting to an intermediate destination in the first domain,
The second engine,
A second input unit that reads the first processing result from the intermediate destination, a second unit processor that processes the first processing result read from the second input unit, and the second processing result processed by the second unit processor, Second writer to output application storage
Cross-domain extensible workflow engine framework that includes a. In clause 5,
The third engine,
A third input device that reads the processing result of the second engine (hereinafter,'second processing result') from the application storage, and generates a third processing result by processing the second processing result read from the third input device. A cross-domain extended workflow engine framework comprising a three unit processor and a third output unit for outputting the third processing result to a data destination. In clause 5,
The second engine receives the processing result of the first engine,
For fusion of the first workflow of the first domain and the second workflow of the second domain, the application to the second and third engines through an authentication process for the second and third engines Cross-domain extensible workflow engine framework further comprising an application access control unit for allocating access rights to the storage. In clause 5,
The second engine,
A second input unit for reading the first processing result from an intermediate data destination receiving the processing result of the first engine (hereinafter,'first processing result');
Filters a file to be moved to a final data destination among the file list included in the first processing result input from the second input device, converts the file name of the filtered file to a new file name, and the file included in the file list A second unit processor for changing different directory paths of the files into one directory path; And
A second output unit that moves or copies a list of files processed by a unit processor and files included in the list of files to the application storage
Cross-domain extensible workflow engine framework that includes a. In claim 9,
The workflow engine framework,
The cross-domain extended workflow engine framework further comprises an application access control unit for verifying whether the second output device has access rights to the application storage. In claim 10,
The application access control unit,
A cross-domain extended workflow engine framework that performs an authentication process based on Open Authentication for the second output device to verify whether the second output device has access rights to the application storage. As a cross-domain extended workflow engine framework (hereinafter referred to as'workflow engine framework') executed by a computing device including a processor, a system configuration unit that generates a plurality of engines according to a workflow specification defined by a user And, as the workflow engine framework including a system control unit for controlling the execution of the plurality of engines,
The plurality of engines include a first engine executing a first workflow of a first edge domain at a first edge node, a second engine executing a second workflow of a core domain at a core node, and a second engine at a second edge node. Including a third engine to execute a third workflow of the edge domain,
The workflow engine framework,
A first shared storage for transferring a result of processing data performed by a first engine to the second engine for fusion of the first domain and the second domain; And
A second shared storage configured to transmit a result of processing data performed by the second engine to the third engine for fusion of the second domain and the third domain;
Cross-domain extensible workflow engine framework that further comprises. In claim 12,
The first engine,
Engine A that processes data collected from the data origin; And
Engine B for moving or copying the processing result of the engine A to the first shared storage in order to share the processing result of the engine A with the core node
Cross-domain extensible workflow engine framework that includes a. In claim 12,
The second engine,
A cross-domain extended workflow engine framework that executes the second workflow for fine tuning a learning model by using a result of processing by a first edge node from the first shared storage as training data. In clause 14,
The second engine,
An engine A reading the processing result of the first engine from the first shared storage as training data;
An engine B that performs learning for fine tuning using the training data read from the engine A; And
Engine C for moving the learning model of the new version created or updated according to the learning result of engine B to the second shared storage
Cross-domain extensible workflow engine framework that includes a. In claim 12,
The third engine,
A cross-domain extended workflow engine framework that executes the third workflow for performing a prediction service using a new version of the learning model loaded from the second shared storage. In paragraph 16,
The third engine,
An engine for moving the new version of the learning model from the second shared storage to a second edge node; And
A serving engine that performs serving to provide a prediction service using a new version of the learning model moved from the second shared storage
It will include a cross-domain extensible workflow engine framework.

Images