US20230134573A1 - Automated generation of dependency hierarchy based on input and output requirements of information - Google Patents
Automated generation of dependency hierarchy based on input and output requirements of information Download PDFInfo
- Publication number
- US20230134573A1 US20230134573A1 US18/049,570 US202218049570A US2023134573A1 US 20230134573 A1 US20230134573 A1 US 20230134573A1 US 202218049570 A US202218049570 A US 202218049570A US 2023134573 A1 US2023134573 A1 US 2023134573A1
- Authority
- US
- United States
- Prior art keywords
- module
- output
- dataset
- modules
- input
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/40—Transformation of program code
- G06F8/41—Compilation
- G06F8/43—Checking; Contextual analysis
- G06F8/433—Dependency analysis; Data or control flow analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/448—Execution paradigms, e.g. implementations of programming paradigms
- G06F9/4494—Execution paradigms, e.g. implementations of programming paradigms data driven
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/10—Requirements analysis; Specification techniques
Definitions
- the present disclosure relates generally to software technology, and more particularly, to systems and methods for automatically generating a dependency graph based on input and output requirements of information.
- a software deployment is the process for deploying, configuring, and updating software applications to help ensure maximum optimization, security, and compatibility within a computing network.
- a deployment is often scheduled to take place at times considered to be the least intrusive by an organization's network administrator. For major rollouts, this can result in staggered releases to help minimize interruptions to employee productivity and reduce strain upon the computing network, including providing post-deployment support.
- FIG. 1 is a block diagram depicting an example environment for automatically generating a dependency graph based on input and output requirements of information, according to some embodiments;
- FIG. 2 is a block diagram depicting an example of the module 126 in FIG. 1 , according to some embodiments;
- FIG. 3 is a flow diagram depicting a method for confirming a validity of an input dataset, according to some embodiments
- FIG. 4 is a block diagram of a dependency graph of modules, according to some embodiments.
- FIG. 5 A is a block diagram depicting an example of the module management system 102 in FIG. 1 , according to some embodiments;
- FIG. 5 B is a block diagram depicting an example of the computing system 124 of the environment in FIG. 1 , according to some embodiments.
- FIG. 6 is a flow diagram depicting a method for handling connection pool sizing with heterogeneous concurrency, according to some embodiments.
- a software deployment is the process for deploying, configuring, and updating software applications to help ensure maximum optimization, security, and compatibility within a computing network.
- a deployment is often scheduled to take place at times considered to be the least intrusive by an organization's network administrator. For major rollouts, this can result in staggered releases to help minimize interruptions to employee productivity and reduce strain upon the computing network, including providing post-deployment support.
- Software deployments are the entry point for a customer to access an organization's cloud computing service.
- a new deployment is required, either due to customer demand, or because of adding a new customer, a manual procedure must be followed to bring the new deployment to life.
- This manual procedure is often a rigid, pre-defined order of operations that are carried out by various teams of developers, each working together to bring the software deployment online.
- a module management system may obtain a list of a plurality of modules executing on one or more computing systems, where the plurality of modules may be associated with a plurality of input requirements and a plurality of output requirements. Each module may be configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements. The module management system may determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements.
- the module management system may establish a plurality of connections between the plurality of modules based on the execution order.
- the module management system may receive a first output dataset from a first module of the plurality of modules.
- the module management system may use one or more of the plurality of connections to redirect the first output dataset to a second module of the plurality of modules to cause the second module to generate a second output dataset based on the first output dataset.
- Each module may include one or more operations of a process or program, such that the execution of each of the modules according to the execution order causes the modules to execute the operations of the process or program, as a whole.
- the embodiments of the present disclosure determine the type of information that each operation needs in order to do its work, and finds what other operations that could provide the information. It then ensures that the operations are executed in the correct order to satisfy each operation's dependencies and marshals the data going into and out of each operation so that the operation itself does not have to know anything about the providing operation's data format. If any preceding operation's data output changes for any reason, then the embodiments of the present disclosure know to re-evaluate any dependent operation.
- Benefits of using the one or more embodiments of the present disclosure for automatically generating a dependency graph may include a reduction in networking resources needed to execute a process, as well as, a decrease in network congestion and power consumption for the overall network infrastructure.
- FIG. 1 is a block diagram depicting an example environment for automatically generating a dependency graph based on input and output requirements of information, according to some embodiments.
- the environment 100 includes a computing system 124 a , a computing system 124 b , and a computing system 124 c (collectively referred as, computing systems 124 ).
- the environment 100 includes modules 126 a , 126 b , 126 c , 126 d , 126 e , 126 f , 126 g , 126 h , 126 i , 126 j , 126 k , 126 l , 126 m , 126 n , 126 o , 126 p (collectively referred as, modules 126 ).
- the computing system 124 a may be configured to execute and/or store one or more of modules 126 a , 126 b , 126 c , 126 d ; the computing system 124 b may be configured to execute and/or store one or more of modules 126 e , 126 f , 126 g , 126 h ; and the computing system 124 c may be configured to execute and/or store one or more of modules 126 i , 126 j , 126 k , 126 l.
- a computing system 124 may be any suitable type of computing device or machine that has a processing device, for example, server computers (e.g., an application server, a catalog server, a communications server, a computing server, a database server, a file server, a game server, a mail server, a media server, a proxy server, a virtual server, a web server), desktop computers, laptop computers, tablet computers, smartphones, set-top boxes, a graphics processing unit (GPU), etc.
- a computing device may comprise a single machine or may include multiple interconnected machines (e.g., multiple servers configured in a cluster).
- a computing system 124 may be any type of cloud computing platform (e.g., Amazon Web Services, Microsoft Azure, Google Cloud Platform, etc.).
- the computing system 124 may be an Infrastructure-as-a-Service (IaaS) that provides users with compute (e.g., processing), networking, and storage resources.
- the computing system 124 may be a Platform-as-a-Service (PaaS) that provides users with a platform on which applications can run, as well as the information technology (IT) infrastructure for it to run.
- IaaS Infrastructure-as-a-Service
- PaaS Platform-as-a-Service
- the computing system 124 may be a Software-as-a-Service (SaaS) that provides users with a cloud application, the platform on which it runs, and the platform's underlying infrastructure.
- the computing system 124 may be a Function-as-a-Service (FaaS) that is an event-driven execution model that lets the developers build, run, and manage application packages as functions without maintaining the infrastructure.
- SaaS Software-as-a-Service
- FaaS Function-as-a-Service
- a computing system 124 may include a one or more virtual machines (VM) that execute on a hypervisor which executes on top of an operating system (OS) of the computing system 124 to provide a virtual environment.
- the hypervisor may manage system sources (including access to hardware devices, such as processing devices, memories, storage devices).
- the hypervisor may also emulate the hardware (or other physical resources) which may be used by the VMs to execute software/applications.
- a virtual environment may be a container that may execute on a container engine which executes on top of the OS for a computing device.
- a container engine may allow different containers to share the OS of a computing device (e.g., the OS kernel, binaries, libraries, etc.).
- the environment 100 includes a module management system 102 for managing the order in which the modules 126 are executed and/or directing the communication (e.g., input datasets, output datasets, etc.) between the modules 126 .
- the module management system 102 may be configured to execute and/or store one or more of modules 126 m , 126 n , 126 o , 126 p .
- the module management system 102 is communicably coupled to each of the computing systems 124 via a communication network 120 .
- the environment 100 includes a centralized database 114 and/or internal cache memory (not shown in FIG. 1 ) for storing information associated with the modules 126 .
- FIG. 1 shows only a select number of computing devices (e.g., computing systems 124 and module management system 102 ), components (e.g., modules 126 ), and storage devices (e.g., centralized databases 114 ); the environment 100 may include any number of computing devices, components and/or storage devices that are interconnected in any arrangement to facilitate the exchange of data between the computing devices, components and/or storage devices.
- Each computing system 124 may execute any number (e.g., 1, 10s, 100s, 1000s, etc.) of modules 126 on its respective processing device.
- the communication network 120 may be a public network (e.g., the internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof.
- communication network 120 may include a wired or a wireless infrastructure, which may be provided by one or more wireless communications systems, such as wireless fidelity (Wi-Fi) connectivity to the communication network 120 and/or a wireless carrier system that can be implemented using various data processing equipment, communication towers (e.g. cell towers), etc.
- the communication network 120 may carry communications (e.g., data, message, packets, frames, etc.) between any other the computing device.
- a program or a process may include a plurality of operations (e.g., tasks, procedures, instructions).
- the module management system 102 may be configured to divide the plurality of operations of the program into sets of one or more operations, where each set may be included in a module 126 .
- the module management system 102 may be configured to divide the operations of a program into one or more modules 126 that are independent and interchangeable, such that each module 126 may contain everything necessary to execute portions of the desired functionality of the program.
- a module 126 may include a requires component that stores an input requirement.
- a module 126 may include a provides component that stores an output requirement.
- a module 126 may include an execute component that includes the one or more operations of the module 126 .
- the input requirement may be an input dataset (e.g., one or more fragments or segments of information) of a particular type that the module 126 needs to consume (e.g., process) in order to execute the one or more operations of the execute component.
- the output requirement may be an output dataset of a particular type that the module 126 generates (e.g., computes) based on the input dataset of the particular type.
- a particular type of an output dataset may correspond to code, an instruction, a data structure, a variable, a constant, a pointer (e.g., a reference to a location in memory), a list, a schema, a table, an object, and the like.
- an output dataset may differ from another output dataset in at least one of a data format, a data value, or a data type.
- a module 126 may include a confirm component that is configured to verify (e.g., confirm) that the information that it receives from another entity (e.g., another module 126 , the module management system 102 ) conforms to the input requirements of the module 126 .
- a second module e.g., module 126 b
- may receive an input dataset from a first module e.g., module 126 a
- confirm whether the input dataset conforms e.g., is compliant
- the confirm component of the second module may be configured to allow the second module to generate an output dataset based on the input dataset, send the output dataset to another entity, and/or store the output dataset in memory/storage. If the input dataset of the second module does not conform to the input requirements, then the confirm component of the second module may be configured to prevent the second module from generating the output dataset, sending the output dataset to another entity, and/or storing the output dataset in memory/storage.
- the confirm component of a module 126 may be configured to verify that the information that it receives from another entity conforms to the input requirements of the module 126 responsive to determining an occurrence of a triggering event.
- a triggering event may occur when the module 126 receives information (e.g., an input dataset), when any module 126 in the dependency graph changes state, and/or when there is an elapse of a predetermined amount of time (e.g., seconds, minutes, hours, days, etc.).
- the confirm component of a module 126 may determine that a first dataset from a first module (e.g., module 126 a ) and a second dataset from a second module (e.g., module 126 b ) are each compliant to the input requirements of the module 126 even though the two datasets may be different and/or originated from different modules 126 .
- a third module e.g., module 126 c
- the confirm components of the third module may determine that the table of numbers and the string of alphanumeric characters each conforms to the input requirements of the third module.
- a module 126 may be configured to receive an input dataset and determine whether the module 126 has a capability to generate an output dataset of a particular output requirement based on the input dataset. In some embodiments, a module 126 may be configured to send a message to the module management system 102 indicating the capability of the module 126 to generate the output dataset of the particular output requirement based on the input dataset. In some embodiments, the module management system 102 may be configured to modify an execution order (e.g., a workflow) of a plurality of modules responsive to determining that the message indicates that the module 126 lacks the capability (e.g., computing resources, storage resources, memory resources, etc.) to generate the output dataset of the particular output requirement based on the input dataset. For example, the module management system 102 may be configured to modify an execution order of the plurality of modules by replacing module 126 a with module 126 b responsive to determining from the message that the module 126 lacks the processing capability to generate an output dataset.
- an execution order e.g., a workflow
- a child module is a module 126 that has its input connected to the output of another module 126 , referred to as a parent module.
- a child module depends on the parent module because the child module needs the output dataset from the parent module in order to generate its own output dataset.
- the module management system 102 may be configured to seamlessly replace (e.g., swap, switch) a parent module with a different parent module without the child module ever even knowing that the parent module was replaced.
- module 126 a may be the parent module to module 126 b because there is a connection between the output of module 126 a and the input of module 126 b .
- the module management system 102 may replace the module 126 a with module 126 c by removing the connection from module 126 a and re-attaching the connection to module 126 c such that the connection now exists between module 126 a and module 126 c ; thereby making module 126 c the parent module to module 126 a.
- the confirm component of a module 126 may be configured to verify (e.g., confirm) the validity of the output dataset by confirming the state of the module 126 or whether the state of the module 126 has changed. For example, if the confirm component of module 126 a determines that module 126 a was in an erred state while generating the output dataset or after generating the output dataset, then the confirm component may determine that the output dataset is invalid. If the confirm component of module 126 a determines that module 126 a was in an error-free state while generating the output dataset and stayed in the error-free state after generating the output dataset, then the confirm component may be configured to determine that the output dataset is valid.
- the confirm component of a module 126 determines that the module 126 was in an error-free state while generating the output dataset and stayed in the error-free state after generating the output dataset, then the confirm component may be configured to allow the module 126 to generate the output dataset, send the output dataset to another entity (e.g., another module 126 , the module management system 102 ), and/or store the output dataset in memory/storage.
- another entity e.g., another module 126 , the module management system 102
- the confirm component of a module 126 determines that the module 126 was in an erred state while generating the output dataset, then the confirm component may be configured to prevent module 126 from generating the output dataset, sending the output dataset to another entity (e.g., another module 126 , the module management system 102 ), and/or storing the output dataset in memory/storage.
- another entity e.g., another module 126 , the module management system 102
- the confirm component of a module 126 may be configured to generate a message (e.g., a flag) indicating that the output dataset is invalid and/or that a subset (e.g., one or more) of a plurality of modules 126 associated with a program need to re-generate their output datasets.
- the confirm component of a module 126 may be configured to send the message to the module management system 102 .
- the module management system 102 may be configured to identify a subset of modules 126 of the plurality of modules 126 whose respective input dataset depends on (or is affected by) the state or change in state of the module 126 .
- an output of a first module may be coupled to an input of a second module (e.g., module 126 b ), whose output is coupled to an input of a third module (e.g., module 126 c ).
- the first module 126 may generate a first output dataset and provide the first output dataset to the second module 126 to cause the second module 126 to generate a second output dataset based on the first output dataset and provide the second output dataset to the third module 126 .
- the module management system 102 determines that the first output dataset is invalid, then the module management system 102 may determine that the second module 126 and the third module 126 need to re-generate their output datasets because they are dependent on the first output dataset.
- a module 126 may be an erred state, an error-free state, an active state, or an inactive state. In some embodiments, a module 126 may be in an erred state if the module 126 is executing incorrectly and/or producing unexpected (e.g., undesired, unpredictable) results. In some embodiments, a module 126 may be in an error-free state if the module 126 is executing correctly and producing expected (e.g., desired, predictable) results. In some embodiments, an active state may be where a module 126 is executing on a processing device of a computing system 124 , but not consuming an input dataset and/or generating an output dataset. In some embodiments, an inactive state may be where a module 126 is not executing (e.g., not yet launched) on a processing device of a computing system 124 .
- the module management system 102 may be configured to send a request to each of the computing systems 124 for a list (e.g., one or more) of module identifiers corresponding to one or more modules 126 that are executing and/or stored on the computing systems 124 .
- each of the computing systems 124 may generate a list of module identifiers corresponding to the one or more modules 126 executing on their respective processors and/or stored in their respective memory/storage systems.
- computing system 124 a may generate a first list of module identifiers corresponding to modules 126 a - 126 d
- computing system 124 b may generate a second list of module identifiers corresponding to modules 126 e - 126 h
- computing system 124 c may generate a third list of module identifiers corresponding to modules 126 i - 126 l
- a computing system 124 may generate a list of module identifiers for the modules 126 that are currently executing on a processing device of the computing system 124 and/or stored on a memory of the computing system 124 .
- a computing system 124 may generate a list of module identifiers for the modules 126 that are currently executing on a processing device of the computing system 124 , but exclude the modules from the list that are currently not executing on the processing device of the computing system 124 .
- Each of the computing systems 124 may send their respective list of module identifiers to the module management system 102 . Since each computing system 124 may execute its own set of modules, their respective lists of module identifiers refer to different sets of modules 126 .
- a computing system 124 may be configured to send a list of module identifiers to the module management system 102 without having to receive a request from the module management system 102 .
- a computing system 124 may be configured to send a list of module identifiers to the module management system 102 responsive to determining an occurrence of a triggering event.
- a triggering event may be when there is an elapse of a predetermined amount of time (e.g., hours, days, weeks), when a module 126 is executed (e.g., launched), when a module is terminated, and/or when a module 126 in the dependency graph changes state.
- the module management system 102 may be configured to generate a list of module identifiers corresponding to the one or more modules 126 executing on a processor of the module management system 102 and/or stored in a memory/storage system of the module management system 102 .
- the module management system 102 may generate a list of module identifiers corresponding to modules 126 m - 126 p.
- the module management system 102 may be configured to aggregate (e.g., combine, collect) each of the lists of module identifiers into a single list of module identifiers.
- the module management system 102 may be configured to store the list of module identifiers in memory/storage of the module management system 102 to be retrieved by the module management system 102 at a later time.
- a list of module identifiers may correspond to a plurality of modules 126 associated with a plurality of input requirements and a plurality of output requirements.
- Each module 126 of the plurality of modules 126 may be configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements.
- two or more of the input requirements of the plurality of input requirements may be identical or different.
- a first module e.g., module 126 a
- a second module e.g., modules 126 b
- an input requirement indicative of a table of numbers may also include an input requirement indicative of a table of numbers.
- a first module (e.g., module 126 a ) may include an input requirement indicative of a table of numbers and a second module (e.g., modules 126 b ) may include an input requirement indicative of a string of alphanumeric characters.
- two or more of the output requirements of the plurality of output requirements may be identical or different.
- a first module e.g., module 126 a
- a second module e.g., module 126 a
- a third module e.g., module 126 b
- the first module may send a set of numbers to a second module (e.g., module 126 b ) and a third module (e.g., module 126 c ) to cause the second module to generate a string of alphanumeric characters based on the set of numbers and the third module to generate a table of numbers based on the set of numbers.
- a second module e.g., module 126 b
- a third module e.g., module 126 c
- the module management system 102 may be configured to determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements that are associated with the list of module identifiers.
- An execution order (e.g., a workflow) is the order in which a plurality of modules that are associated with a process or program are executed.
- the module management system 102 may be configured to select a module identifier from the list of module identifiers and determine that the module 126 corresponding to the selected module identifier includes a particular input requirement.
- the module management system 102 may be configured to scan (e.g., search) the list of module identifiers to determine whether there are other modules 126 (e.g., one or more) that include an output requirement that conforms to the input requirement of the selected module.
- the module management system 102 may be configured to establish, in response to the determination, a connection between the outputs of each of the other modules 126 (e.g., the ones with conforming output requirements) and the input of the selected module, such that the selected module receives one or more input datasets that conforms to its input requirements.
- the module management system 102 may scan the list of module identifiers to determine that a third module (e.g., module 126 c ) includes an input requirement indicative of a table of numbers.
- the module management system 102 may scan the list of module identifiers to determine that a first module (e.g., module 126 a ) includes an output requirement indicative of a table of numbers and a second module (e.g., module 126 b ) includes an output requirement indicative of a function pointer.
- the module management system 102 may establish a connection between the output of the first module 126 and the input of the third module 126 , but not establish a connection between the output of the second module 126 and the input of the third module 126 .
- the module management system 102 may be configured to establish connections between outputs of one or more modules 126 and inputs of one or more modules 126 by maintaining a plurality of mappings (e.g., linkages, associations) between the identifiers of the modules in memory/storage.
- Each module 126 may be configured to provide its output dataset (shown in FIG. 1 as, “Module Output Data”) to the module management system 102 .
- the module management system 102 may be configured to redirect (shown in FIG. 1 as, “Redirected Module Output Data”) the output dataset that it receives from each module 126 to the inputs of one or more of the other modules 126 based on the plurality of mappings (sometimes referred to as, plurality of connections).
- the module management system 102 may update the plurality of mappings to indicate that the output of a first module (e.g., module 126 a ) is mapped to the input of a second module (e.g., module 126 b ). If the module management system 102 receives an output dataset from the first module, then the module management system 102 may determine that the plurality of mappings indicate that the output of the first module is mapped to the input of the second module, and in response, redirect the output dataset from the first module to the input of the second module.
- an execution order may include (or indicate) a plurality of mappings associated with a plurality of modules.
- the module management system 102 may receive a plurality of output datasets from a plurality of modules that are connected (e.g., according to the plurality of mappings) to an input of a single module 126 .
- the module management system 102 may be configured to select the output dataset that should be redirected based on the availability of the dataset. For example, the module management system 102 may receive a first output dataset from a first module (e.g., module 126 a ) during a first timeslot and a second output dataset from a second module (e.g., module 126 b ) during a second timeslot, where the first timeslot is earlier than the second timeslot.
- the module management system 102 may determine that both datasets could be redirected to the third module because the plurality of mappings indicate that the output of the first module and the output of the second module are each connected to the input of a third module (e.g., module 126 c ). In this instance, the module management system 102 implements a tie-breaking approach by selecting the first dataset instead of the second dataset because the first dataset arrived at the module management system 102 before the second dataset. The module management system 102 redirects the first dataset to the input of a third module and without redirecting the second dataset to the input of the third module.
- a third module e.g., module 126 c
- the module management system 102 may include the same functionality as the confirm component of a module 126 .
- the module management system 102 may be configured to receive an output dataset from a first module (e.g., module 126 a ) and verify (e.g., confirm) whether the output dataset conforms to the input requirements of a second module (e.g., module 126 b ).
- the module management system 102 may be configured to receive an output dataset from the first module and verify the validity of the output data by confirming the state of the first module or whether the state of the first module has changed.
- the module management system 102 may be configured to simultaneously confirm all modules 126 of an execution order or dependency graph.
- the module management system 102 may be configured to modify (e.g., adjust, change) an execution order of the plurality of modules when the plurality of modules is in a non-executing state or when the plurality of modules is in an executing state (e.g., during runtime). In some embodiments, the module management system 102 may be configured to modify an execution order of the plurality of modules by replacing a module 126 of the plurality of modules 126 with a different module 126 . In some embodiments, the module management system 102 may be configured to modify an execution order of the plurality of modules 126 responsive to determining that one or more of the modules 126 of the execution order is in (or changed to) an error-state.
- the computing systems 124 , the modules 126 , and/or the module management system may each include their own interface, but each interface corresponds to a common interface type that uses a standardized protocol for communicating the output datasets.
- the common interface may allow a first module (e.g., module 126 a ) to generate and send an output dataset of a particular output requirement to a second module (e.g., module 126 b ), such that the particular output requirement conforms to the input requirement of the second module.
- the common interface may allow the module management system 102 (or a system administrator of the environment 100 ) to modify a module 126 of an execution order (e.g., workflow) without impacting the performance of any of the other modules 126 of an executive order.
- a module 126 of an execution order e.g., workflow
- an output of a first module e.g., module 126 a
- a third module e.g., module 126 c
- the first module may send a first output dataset of a particular output requirement to the input of the third module
- the third module may confirm that the first output dataset conforms to the input requirement of the third module.
- the module management system 102 may then replace the first module with a second module (e.g., module 126 c ), such that the second module may send a second output dataset of the particular output requirement to the input of the third module.
- the third module may confirm that the second output dataset conforms to the input requirement of the third module even though the first module was replaced with the second module because each of the modules use common interfaces.
- the common interface may allow a first module (e.g., module 126 a ) to send an output dataset to a second module (e.g., module 126 b ), where the second module is unaware that the first module was the module that generated and sent the output dataset.
- the module management system 102 may be configured to establish any combination of connections between any of the modules 126 executing on the computing system 124 a , any of the modules 126 executing on the computing system 124 b , and/or any of the modules 126 executing on the computing system 124 c .
- module management system 102 may establish a connection between the output of module 126 a and the input of module 126 b , a connection between the output of module 126 a and the input of module 126 g , and/or a connection between the output of module 126 a and the input of module 126 j .
- the module management system 102 may be configured to establish direct connections between the modules 126 , such that the output dataset from a first module 126 may pass to the input of a second module 126 without having to rely on a computing device (e.g., the module management system 102 ) to receive the output dataset from the first module 126 and redirect the output dataset to the second module 126 .
- a computing device e.g., the module management system 102
- FIG. 2 is a block diagram depicting an example of the module 126 in FIG. 1 , according to some embodiments.
- the module 126 may include a requires component 204 that stores an input requirement.
- the module 126 may include a provides component 210 that stores an output requirement.
- the module 126 may include an execute component 208 that stores one or more operations of the module 126 .
- the input requirement may be an input dataset (e.g., one or more fragments or segments of information) of a particular type that the module 126 needs to consume in order to execute the one or more operations of the execute component 208 .
- the output requirement may be an output dataset of a particular type that the module 126 generates based on the input dataset of the particular type.
- a particular type may correspond to code, an instruction, a data structure, a variable, a constant, a pointer (e.g., a reference to a location in memory), a list, a schema, a table, an object, and the like.
- an output dataset may differ from another output dataset in at least one of a data format, a data value, or a data type.
- the module 126 includes a confirm component 206 that may be configured to verify (e.g., confirm) that the output dataset (e.g., this becomes the input dataset for the module 126 ) that it receives from another entity (e.g., another module 126 , the module management system 102 ) conforms to the input requirements of the module 126 .
- the confirm component 206 of the module 126 may be configured to verify the validity of the output dataset (e.g., this becomes the input dataset for the module 126 ) by confirming the state of the module 126 or whether the state of the module 126 has changed.
- the module 126 may be configured to determine whether to process an input dataset based on whether the input dataset conforms to the input requirements of the module 126 .
- FIG. 3 is a flow diagram depicting a method for confirming a validity of an input dataset, according to some embodiments.
- the method 300 may be performed by the module 126 in FIG. 1 . Although the method 300 is described with respect to the module 126 , one or more of the blocks may be performed by the module management system 102 in FIG. 1 .
- the method 300 illustrates example functions used by various embodiments. Although specific function blocks (“blocks”) are disclosed in method 300 , such blocks are examples. That is, embodiments are well suited to performing various other blocks or variations of the blocks recited in method 300 . It is appreciated that the blocks in method 300 may be performed in an order different than presented, and that not all of the blocks in method 300 may be performed.
- the method 300 includes the block 302 of starting, by the module management system 102 , a module 126 to cause the module 126 to execute on a processing device of the computing system 124 .
- the method 300 includes the block 304 of receiving, by the module 126 , information.
- the method 300 includes the block 306 of determining, by the module 126 , whether the information conforms to the input requirements of the requires component 204 . If the module 126 determines that the information conforms to the input requirements of the requires component 204 , then the module 126 proceeds to block 306 ; otherwise the module 126 proceeds to block 304 , to repeat block 304 .
- the method 300 includes the block 308 of running, by the module 126 , one or more confirm tests.
- the method 300 includes the block 310 of determining, by the module 126 , whether the confirm tests succeeded (e.g., passed). If the confirm tests succeeded, then the module 126 proceeds to block 320 to store the information in a memory/storage and proceed to block 322 to report that the module is a success. If the confirm tests did not succeed, then the module 126 proceeds to block 312 .
- the method 300 includes the block 312 of running, by the module 126 , executed tasks.
- the method 300 includes the block 314 of determining whether an error occurred as a result of running the executed tasks. If an error occurred, then the module 126 proceeds to block 318 to report the error and proceed to block 304 , to repeat block 304 . If an error did not occur, then the module 126 proceeds to block 316 to store the information in a memory/storage and proceed to block 308 , to repeat block 308 .
- FIG. 4 is a block diagram of a dependency graph (sometimes referred to as, an execution order, a workflow, or a dependency tree) of modules, according to some embodiments.
- the dependency graph 400 shows the order in which a plurality of modules receive information (e.g., input datasets), execute tasks, and provide information (e.g., output datasets) based on their connections.
- the graph 400 includes modules 426 a , 426 b , 426 c , 426 d , 426 e , 426 f , 426 g , 426 h (collectively referred to as, modules 426 ). In some embodiments, any of the modules 426 may be the module 126 in FIG. 1 .
- the graph 400 includes connection 402 , 404 , 406 , 408 , 410 , 412 , 414 (collectively referred to as, connections 402 - 414 ).
- an output of module 426 a is coupled to module 426 b via connection 402 .
- An output of module 426 b is coupled to an input of module 426 c via connection 404 , an input of module 426 d via connection 406 , and an input of module 426 e via connection 408 .
- An output of module 426 c is coupled to an input of module 426 f via connection 410 .
- An output of module 426 f is coupled to an input of module 426 g via connection 412 and an input of module 426 h via connection 414 .
- the module 426 a sends an output dataset to module 426 b via connection 402 , causing the module 426 b to generate an output dataset based on the input dataset.
- the module 426 b sends its output dataset to module 426 c via connection 404 , causing the module 426 c to generate an output dataset based on the input dataset.
- the module 426 b sends its output dataset to module 426 d via connection 406 , causing the module 426 d to generate an output dataset based on the input dataset.
- the module 426 b sends its output dataset to module 426 e via connection 408 , causing the module 426 e to generate an output dataset based on the input dataset.
- the module 426 c sends its output dataset to module 426 f via connection 410 , causing module 426 f to generate an output dataset based on the input dataset.
- the module 426 f sends its output dataset to module 426 g via connection 412 , causing module 426 g to generate an output dataset based on the input dataset.
- the module 426 f sends its output dataset to module 426 h via connection 414 , causing module 426 h to generate an output dataset based on the input dataset.
- each of the connections 402 - 414 shown in FIG. 4 may be a direct connection between an output of a first module 426 and an input of a second module 426 , such that the output dataset from a first module 426 passes to the input of second module 426 without having to rely on a computing device (e.g., the module management system 102 ) to receive the output dataset from the first module 426 and redirect the output dataset to the second module 426 .
- a computing device e.g., the module management system 102
- each of the connections 402 - 414 shown in FIG. 4 may be an indirect connection between an output of a first module 426 and an input of a second module 426 , such that the output dataset from a first module 426 passes to the input of second module 426 because a computing device (e.g., the module management system 102 ) must receive the output dataset from the first module 426 and redirect the output dataset to the second module 426 .
- the module management system 102 redirects the output data from a first module 426 to an input of the second module 426 based on an execution order or a plurality of mappings associated with modules 426 .
- the module management system 102 determines the execution order or the plurality of mappings for the modules 426 based on the plurality of input requirements associated with modules 426 and/or the plurality of output requirements associated with modules 426 .
- FIG. 5 A is a block diagram depicting an example of the module management system 102 in FIG. 1 , according to some embodiments. While various devices, interfaces, and logic with particular functionality are shown, it should be understood that the module management system 102 includes any number of devices and/or components, interfaces, and logic for facilitating the functions described herein. For example, the activities of multiple devices may be combined as a single device and implemented on a same processing device (e.g., processing device 502 a ), as additional devices and/or components with additional functionality are included.
- processing device 502 a e.g., processing device 502 a
- the module management system 102 includes a processing device 502 a (e.g., general purpose processor, a PLD, etc.), which may be composed of one or more processors, and a memory 504 a (e.g., synchronous dynamic random access memory (DRAM), read-only memory (ROM)), which may communicate with each other via a bus (not shown).
- a processing device 502 a e.g., general purpose processor, a PLD, etc.
- a memory 504 a e.g., synchronous dynamic random access memory (DRAM), read-only memory (ROM)
- DRAM synchronous dynamic random access memory
- ROM read-only memory
- the processing device 502 a may be provided by one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like.
- processing device 502 a may include a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets.
- the processing device 502 a may comprise one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like.
- ASIC application specific integrated circuit
- FPGA field programmable gate array
- DSP digital signal processor
- the processing device 502 a may be configured to execute the operations described herein, in accordance with one or more aspects of the present disclosure, for performing the operations and steps discussed herein.
- the memory 504 a (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Non-volatile RAM (NVRAM), Flash Memory, hard disk storage, optical media, etc.) of processing device 502 a stores data and/or computer instructions/code for facilitating at least some of the various processes described herein.
- the memory 504 a includes tangible, non-transient volatile memory, or non-volatile memory.
- the memory 504 a stores programming logic (e.g., instructions/code) that, when executed by the processing device 502 a , controls the operations of the module management system 102 .
- the processing device 502 a and the memory 504 a form various processing devices and/or circuits described with respect to module management system 102 .
- the instructions include code from any suitable computer programming language such as, but not limited to, C, C++, C#, Java, JavaScript, VBScript, Perl, HTML, XML, Python, TCL, and Basic.
- the processing device 502 a may execute a module management component 510 a .
- the module management component 510 a may be configured to obtain a list of a plurality of modules 126 executing on one or more computing systems 124 .
- the plurality of modules 126 may be associated with a plurality of input requirements and a plurality of output requirements.
- each module 126 may be configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements.
- the module management component 510 a may be configured to determine an execution order of the plurality of modules 126 based on the plurality of input requirements and the plurality of output requirements. In some embodiments, the module management component 510 a may be configured to establish a plurality of connections between the plurality of modules 126 based on the execution order.
- the module management component 510 a may be configured to receive a first output dataset from a first module 126 of the plurality of modules 126 . In some embodiments, the module management component 510 a may be configured to redirect, based on the execution order, the first output dataset to a second module 126 of the plurality of modules 126 to cause the second module 126 to generate a second output dataset based on the first output dataset.
- the module management component 510 a may be configured to receive the second output dataset from the second module 126 of the plurality of modules 126 . In some embodiments, the module management component 510 a may be configured to redirect, based on the execution order, the second output dataset to a third module 126 of the plurality of modules to cause the third module 126 to generate a third output dataset based on the second output dataset. In some embodiments, the module management component 510 a may be configured to redirect, based on the execution order, the second output dataset to a fourth module 126 of the plurality of modules 126 to cause the fourth module 126 to generate a fourth output dataset based on the second output dataset.
- a first connection of the plurality of connections is between a first module 126 of the plurality of modules and a third module 126 of the plurality of modules 126 .
- a second connection of the plurality of connections is between a second module 126 of the plurality of modules 126 and the third module 126 of the plurality of modules 126 .
- the module management component 510 a may be configured to determine that a first output dataset that is generated by the first module 126 is available before a second output dataset that is generated by the second module 126 .
- the module management component 510 a may be configured to redirect, responsive to determining that the first output dataset is available before the second output dataset, the first output dataset to the third module 126 of the plurality of modules 126 without redirecting the second output dataset to the third module 126 of the plurality of modules 126 .
- the first module 126 was in a first state when generating the first output dataset. In some embodiments, the second module 126 was in a second state when generating the second output dataset. In some embodiments, the module management component 510 a may be configured to determine that the first module 126 generated the first output dataset while in an error state. In some embodiments, the module management component 510 a may be configured to determine, responsive to determining that the first module 126 generated the first output dataset while in an error state, that the second module 126 generated the second output dataset while in an error-free state.
- the module management component 510 a may be configured to redirect, responsive to determining that the second module 126 generated the second output dataset while in an error-free state, the second output dataset to the third module 126 of the plurality of modules 126 without redirecting the first output dataset to the third module 126 of the plurality of modules 126 .
- the module management component 510 a may be configured to determine a change in a first module 126 of the plurality of modules 126 from an error-free state to an error state. In some embodiments, the module management component 510 a may be configured to identify a subset of modules 126 of the plurality of modules 126 whose respective input dataset depends on the change. In some embodiments, the module management component 510 a may be configured to cause each of the subset of modules 126 to re-generate the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- the change in the first module 126 of the plurality of modules 126 from the error-free state to the error state occurs after the first module 126 of the plurality of modules 126 generates the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- the module management component 510 a may be configured to determine, by the first module 126 of the plurality of modules 126 , that a state of the first module 126 changed during or after the generation of the first output dataset. In some embodiments, the module management component 510 a may be configured to generate, by the first module 126 , a flag indicating an invalidity of the first output dataset or a requirement for a subset of modules 126 to re-generate a plurality of output datasets.
- the module management component 510 a may be configured to replace a first module 126 of the plurality of modules 126 with a third module 126 of the plurality of modules 126 . In some embodiments, the module management component 510 a may be configured to receive a third output dataset from the third module 126 of the plurality of modules 126 . In some embodiments, the module management component 510 a may be configured to determine a compliance of the second output dataset to the input requirements of the second module 126 of the plurality of modules 126 . In some embodiments, the module management component 510 a may be configured to redirect, responsive to determining the compliance of the second output dataset to the input requirements of the second module 126 , the third output dataset to the second module 126 of the plurality of modules 126 .
- the first module 126 sends the first output dataset to the processing device using a first interface of the first module 126
- the third module 126 sends the third output dataset to the processing device using a second interface.
- the first interface and the second interface correspond to a common interface type.
- the second output dataset is different from the first output dataset in at least one of a data format, a data value, or a data type.
- the second module 126 is unaware that the first module 126 of the plurality of modules 126 generated the first output dataset.
- the module management component 510 a may be configured to determine a capability to generate the output dataset of a respective output requirement of the plurality of output requirements based on the input dataset of a respective input requirement of the plurality of input requirements, and send a message to the processing device indicative of the capability.
- the processing device 502 a may execute one or more modules 126 in FIG. 1 .
- the module management system 102 includes a network interface 506 a configured to establish a communication session with a computing device for sending and receiving data over the communication network 120 to the computing device.
- the network interface 506 A includes a cellular transceiver (supporting cellular standards), a local wireless network transceiver (supporting 802.11X, ZigBee, Bluetooth, Wi-Fi, or the like), a wired network interface, a combination thereof (e.g., both a cellular transceiver and a Bluetooth transceiver), and/or the like.
- the module management system 102 includes a plurality of network interfaces 506 a of different types, allowing for connections to a variety of networks, such as local area networks (public or private) or wide area networks including the Internet, via different sub-networks.
- networks such as local area networks (public or private) or wide area networks including the Internet, via different sub-networks.
- the module management system 102 includes an input/output device 505 a configured to receive user input from and provide information to a user.
- the input/output device 505 a is structured to exchange data, communications, instructions, etc. with an input/output component of the module management system 102 .
- input/output device 505 a may be any electronic device that conveys data to a user by generating sensory information (e.g., a visualization on a display, one or more sounds, tactile feedback, etc.) and/or converts received sensory information from a user into electronic signals (e.g., a keyboard, a mouse, a pointing device, a touch screen display, a microphone, etc.).
- the one or more user interfaces may be internal to the housing of the module management system 102 , such as a built-in display, touch screen, microphone, etc., or external to the housing of the module management system 102 , such as a monitor connected to the module management system 102 , a speaker connected to the module management system 102 , etc., according to various embodiments.
- the module management system 102 includes communication circuitry for facilitating the exchange of data, values, messages, and the like between the input/output device 505 a and the components of the module management system 102 .
- the input/output device 505 a includes machine-readable media for facilitating the exchange of information between the input/output device 505 a and the components of the module management system 102 .
- the input/output device 505 a includes any combination of hardware components (e.g., a touchscreen), communication circuitry, and machine-readable media.
- the module management system 102 includes a device identification component 507 a (shown in FIG. 5 A as device ID component 507 a ) configured to generate and/or manage a device identifier associated with the module management system 102 .
- the device identifier may include any type and form of identification used to distinguish the module management system 102 from other computing devices.
- the device identifier may be cryptographically generated, encrypted, or otherwise obfuscated by any device and/or component of module management system 102 .
- the module management system 102 may include the device identifier in any communication (e.g., establish connection request, resource request) that the module management system 102 sends to a computing device.
- the module management system 102 includes a bus (not shown in FIG. 5 A ), such as an address/data bus or other communication mechanism for communicating information, which interconnects the devices and/or components of module management system 102 , such as processing device 502 a , network interface 506 a , input/output device 505 a , device ID component 507 a , module management component 510 a , and modules 126 .
- a bus such as an address/data bus or other communication mechanism for communicating information, which interconnects the devices and/or components of module management system 102 , such as processing device 502 a , network interface 506 a , input/output device 505 a , device ID component 507 a , module management component 510 a , and modules 126 .
- module management system 102 may be implemented with the processing device 502 a .
- the module management system 102 may be implemented as a software application stored within the memory 504 a and executed by the processing device 502 a . Accordingly, such embodiment can be implemented with minimal or no additional hardware costs.
- any of these above-recited devices and/or components rely on dedicated hardware specifically configured for performing operations of the devices and/or components.
- FIG. 5 B is a block diagram depicting an example of the computing system 124 of the environment in FIG. 1 , according to some embodiments. While various devices, interfaces, and logic with particular functionality are shown, it should be understood that the computing system 124 includes any number of devices and/or components, interfaces, and logic for facilitating the functions described herein. For example, the activities of multiple devices may be combined as a single device and implemented on a same processing device (e.g., processing device 502 b ), as additional devices and/or components with additional functionality are included.
- processing device 502 b e.g., processing device 502 b
- the computing system 124 includes a processing device 502 b (e.g., general purpose processor, a PLD, etc.), which may be composed of one or more processors, and a memory 504 b (e.g., synchronous dynamic random access memory (DRAM), read-only memory (ROM)), which may communicate with each other via a bus (not shown).
- the processing device 502 b includes identical or nearly identical functionality as processing device 502 a in FIG. 5 a , but with respect to devices and/or components of the computing system 124 instead of devices and/or components of the module management system 102 .
- the memory 504 b of processing device 502 b stores data and/or computer instructions/code for facilitating at least some of the various processes described herein.
- the memory 504 b includes identical or nearly identical functionality as memory 504 a in FIG. 5 A , but with respect to devices and/or components of the computing system 124 instead of devices and/or components of the module management system 102 .
- the processing device 502 b may execute a client component 510 b that may be configured to launch a module 126 to cause the module 126 to execute on the processing device 02 b .
- the client component 510 b may be configured to receive a request from the module management system 102 for a list of module identifiers corresponding to one or more modules 126 that are executing and/or stored on the computing systems 124 .
- the client component 510 b may be configured to generate the list of module identifiers and send the list to the module management system 102 .
- the processing device 502 b may execute one or more modules 126 that may be configured to receive an output dataset from another entity (e.g., another module 126 , the module management system 102 ). In some embodiments, the module 126 may be configured to generate an output dataset of an output requirement based on an input dataset of an input requirement. In some embodiments, the module 126 may be configured to send the output dataset to another entity (e.g., another module 126 , the module management system 102 ).
- another entity e.g., another module 126 , the module management system 102 .
- the module 126 may be configured to verify that the information that the module 126 receives from another entity (e.g., another module 126 , the module management system 102 ) conforms to the input requirements of the module 126 . In some embodiments, the module 126 may be configured to verify the validity of an output dataset by confirming the state of a module 126 or whether the state of a module 126 has changed. In some embodiments, the module 126 may be configured to receive an input dataset and determine whether the module 126 has a capability to generate an output dataset of a particular output requirement based on the input dataset.
- the computing system 124 includes a network interface 506 b configured to establish a communication session with a computing device for sending and receiving data over a network to the computing device. Accordingly, the network interface 506 b includes identical or nearly identical functionality as network interface 506 a in FIG. 2 A , but with respect to devices and/or components of the computing system 124 instead of devices and/or components of the module management system 102 .
- the computing system 124 includes an input/output device 505 b configured to receive user input from and provide information to a user.
- the input/output device 505 b is structured to exchange data, communications, instructions, etc. with an input/output component of the remote server 108 .
- the input/output device 505 b includes identical or nearly identical functionality as input/output device 505 a in FIG. 5 A , but with respect to devices and/or components of the computing system 124 instead of devices and/or components of the module management system 102 .
- the computing system 124 includes a device identification component 507 b (shown in FIG. 5 B as device ID component 507 b ) configured to generate and/or manage a device identifier associated with the computing system 124 .
- the device ID component 507 b includes identical or nearly identical functionality as device ID component 507 a in FIG. 5 A , but with respect to devices and/or components of the computing system 124 instead of devices and/or components of the module management system 102 .
- the computing system 124 includes a bus (not shown), such as an address/data bus or other communication mechanism for communicating information, which interconnects the devices and/or components of the computing system 124 , such as processing device 502 b , network interface 506 b , input/output device 505 b , device ID component 507 b , the client component 510 b , and the modules 126 .
- a bus such as an address/data bus or other communication mechanism for communicating information, which interconnects the devices and/or components of the computing system 124 , such as processing device 502 b , network interface 506 b , input/output device 505 b , device ID component 507 b , the client component 510 b , and the modules 126 .
- computing system 124 may be implemented with the processing device 502 b .
- the computing system 124 may be implemented as a software application stored within the memory 504 b and executed by the processing device 502 b . Accordingly, such embodiment can be implemented with minimal or no additional hardware costs.
- any of these above-recited devices and/or components rely on dedicated hardware specifically configured for performing operations of the devices and/or components.
- FIG. 6 is a flow diagram depicting a method for handling connection pool sizing with heterogeneous concurrency, according to some embodiments.
- Method 600 may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, a processor, a processing device, a central processing unit (CPU), a system-on-chip (SoC), etc.), software (e.g., instructions running/executing on a processing device), firmware (e.g., microcode), or a combination thereof.
- method 600 may be performed by a module management system, such as module management system 102 in FIG. 1 .
- method 600 may be performed by a computing system, such as computing system 124 in FIG. 1 .
- method 600 may be performed by a module 126 executing on the computing system 124 .
- method 600 illustrates example functions used by various embodiments. Although specific function blocks (“blocks”) are disclosed in method 600 , such blocks are examples. That is, embodiments are well suited to performing various other blocks or variations of the blocks recited in method 600 . It is appreciated that the blocks in method 600 may be performed in an order different than presented, and that not all of the blocks in method 600 may be performed.
- the method 600 includes the block 602 of obtaining, by a processing device, a list of a plurality of modules executing on one or more computing systems, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module is configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements.
- the method 600 includes the block 604 of determining, by the processing device, an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements.
- the method 600 includes the block 606 of establishing, by the processing device, a plurality of connections between the plurality of modules based on the execution order.
- Example 1 is a method.
- the method includes obtaining, by a processing device, a list of a plurality of modules executing on one or more processing devices, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module is configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements.
- the method includes determining, by the processing device, an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements.
- the method includes establishing, by the processing device, a plurality of connections between the plurality of modules based on the execution order.
- Example 2 is a method as in Example 1, further comprising receiving, by the processing device, a first output dataset from a first module of the plurality of modules; and redirecting, by the processing device based on the execution order, the first output dataset to a second module of the plurality of modules to cause the second module to generate a second output dataset based on the first output dataset.
- Example 3 is a method as in any of Examples 1-2, further comprising receiving, by the processing device, the second output dataset from the second module of the plurality of modules; redirecting, by the processing device based on the execution order, the second output dataset to a third module of the plurality of modules to cause the third module to generate a third output dataset based on the second output dataset; and redirecting, by the processing device based on the execution order, the second output dataset to a fourth module of the plurality of modules to cause the fourth module to generate a fourth output dataset based on the second output dataset.
- Example 4 is a method as in any of Examples 1-3, wherein a first connection of the plurality of connections is between a first module of the plurality of modules and a third module of the plurality of modules, wherein a second connection of the plurality of connections is between a second module of the plurality of modules and the third module of the plurality of modules, and further comprising determining, by the processing device, that a first output dataset that is generated by the first module is available before a second output dataset that is generated by the second module.
- Example 5 is a method as in any of Examples 1-4, further comprising redirecting, by the processing device responsive to determining that the first output dataset is available before the second output dataset, the first output dataset to the third module of the plurality of modules without redirecting the second output dataset to the third module of the plurality of modules.
- Example 6 is a method as in any of Examples 1-5, wherein the first module was in a first state when generating the first output dataset, and wherein the second module was in a second state when generating the second output dataset, and further comprising determining, by the processing device, that the first module generated the first output dataset while in an error state; determining, by the processing device responsive to determining that the first module generated the first output dataset while in an error state, that the second module generated the second output dataset while in an error-free state; and redirecting, by the processing device responsive to determining that the second module generated the second output dataset while in an error-free state, the second output dataset to the third module of the plurality of modules without redirecting the first output dataset to the third module of the plurality of modules.
- Example 7 is a method as in any of Examples 1-6, further comprising determining, by the processing device, a change in a first module of the plurality of modules from an error-free state to an error state; identifying, by the processing device, a subset of modules of the plurality of modules whose respective input dataset depends on the change; and causing each of the subset of modules to re-generate the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- Example 8 is a method as in any of Examples 1-7, wherein the change in the first module of the plurality of modules from the error-free state to the error state occurs after the first module of the plurality of modules generates the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- Example 9 is a method as in any of Examples 1-8, further comprising determining, by the first module of the plurality of modules, that a state of the first module changed during or after the generation of the first output dataset; and generating, by the first module, a flag indicating an invalidity of the first output dataset or a requirement for a subset of modules to re-generate a plurality of output datasets.
- Example 10 is a method as in any of Examples 1-9, further comprising replacing, by the processing device, a first module of the plurality of modules with a third module of the plurality of modules; receiving, by the processing device, a third output dataset from the third module of the plurality of modules; determining, by the processing device, a compliance of the third output dataset to the input requirements of the second module of the plurality of modules; and redirecting, by the processing device responsive to determining the compliance of the second output dataset to the input requirements of the second module, the third output dataset to the second module of the plurality of modules.
- Example 11 is a method as in any of Examples 1-10, wherein the first module sends the first output dataset to the processing device using a first interface of the first module, and the third module sends the third output dataset to the processing device using a second interface, and wherein the first interface and the second interface correspond to a common interface type.
- Example 12 is a method as in any of Examples 1-11, wherein the second output dataset is different from the first output dataset in at least one of a data format, a data value, or a data type.
- Example 13 is a method as in any of Examples 1-12, wherein the second module is unaware that the first module of the plurality of modules generated the first output dataset.
- Example 14 is a method as in any of Examples 1-13, wherein each module is configured to determine a capability to generate the output dataset of a respective output requirement of the plurality of output requirements based on the input dataset of a respective input requirement of the plurality of input requirements.
- Example 15 is a method as in any of Examples 1-14, wherein each module is configured to send a message to the processing device that is indicative of the capability.
- Example 16 is a system.
- the system includes a memory; and a processing device, operatively coupled to the memory, to obtain a list of a plurality of modules executing on one or more processing devices, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module is configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements.
- the processing device is further to determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements.
- the processing device is further to establish a plurality of connections between the plurality of modules based on the execution order.
- Example 17 is a system as in Example 16, wherein the processing device is further to: receive a first output dataset from a first module of the plurality of modules; and redirect, based on the execution order, the first output dataset to a second module of the plurality of modules to cause the second module to generate a second output dataset based on the first output dataset.
- Example 18 is a system as in any of Examples 16-17, wherein the processing device is further to receive the second output dataset from the second module of the plurality of modules; redirect, based on the execution order, the second output dataset to a third module of the plurality of modules to cause the third module to generate a third output dataset based on the second output dataset; and redirect, based on the execution order, the second output dataset to a fourth module of the plurality of modules to cause the fourth module to generate a fourth output dataset based on the second output dataset.
- Example 19 is a system as in any of Examples 16-18, wherein a first connection of the plurality of connections is between a first module of the plurality of modules and a third module of the plurality of modules, wherein a second connection of the plurality of connections is between a second module of the plurality of modules and the third module of the plurality of modules, and wherein the processing device is further to determine that a first output dataset that is generated by the first module is available before a second output dataset that is generated by the second module.
- Example 20 is a system as in any of Examples 16-19, wherein the processing device is further to redirect, responsive to determining that the first output dataset is available before the second output dataset, the first output dataset to the third module of the plurality of modules without redirecting the second output dataset to the third module of the plurality of modules.
- Example 21 is a system as in any of Examples 16-20, wherein the first module was in a first state when generating the first output dataset, and wherein the second module was in a second state when generating the second output dataset, and wherein the processing device is further to determine that the first module generated the first output dataset while in an error state; determine, responsive to determining that the first module generated the first output dataset while in an error state, that the second module generated the second output dataset while in an error-free state; and redirect, responsive to determining that the second module generated the second output dataset while in an error-free state, the second output dataset to the third module of the plurality of modules without redirecting the first output dataset to the third module of the plurality of modules.
- Example 22 is a system as in any of Examples 16-21, wherein the processing device is further to determine a change in a first module of the plurality of modules from an error-free state to an error state; identify a subset of modules of the plurality of modules whose respective input dataset depends on the change; and cause each of the subset of modules to re-generate the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- Example 23 is a system as in any of Examples 16-22, wherein the change in the first module of the plurality of modules from the error-free state to the error state occurs after the first module of the plurality of modules generates the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- Example 24 is a system as in any of Examples 16-23, wherein the processing device is further to determine, by the first module of the plurality of modules, that a state of the first module changed during or after the generation of the first output dataset; and generate, by the first module, a flag indicating an invalidity of the first output dataset or a requirement for a subset of modules to re-generate a plurality of output datasets.
- Example 25 is a system as in any of Examples 16-24, wherein the processing device is further to replace a first module of the plurality of modules with a third module of the plurality of modules; receive a third output dataset from the third module of the plurality of modules; determine a compliance of the third output dataset to the input requirements of the second module of the plurality of modules; and redirect, responsive to determining the compliance of the second output dataset to the input requirements of the second module, the third output dataset to the second module of the plurality of modules.
- Example 26 is a system as in any of Examples 16-25, wherein the first module sends the first output dataset to the processing device using a first interface of the first module, and the third module sends the third output dataset to the processing device using a second interface, and wherein the first interface and the second interface correspond to a common interface type.
- Example 27 is a system as in any of Examples 16-26, wherein the second output dataset is different from the first output dataset in at least one of a data format, a data value, or a data type.
- Example 28 is a system as in any of Examples 16-27, wherein the second module is unaware that the first module of the plurality of modules generated the first output dataset.
- Example 29 is a system as in any of Examples 16-28, wherein each module is configured to determine a capability to generate the output dataset of a respective output requirement of the plurality of output requirements based on the input dataset of a respective input requirement of the plurality of input requirements, and send a message to the processing device that is indicative of the capability.
- Example 30 is a non-transitory computer-readable medium storing instructions that, when execute by a processing device, cause the processing device to obtain, by a processing device, a list of a plurality of modules executing on one or more processing devices, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module is configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements; determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements; and establish a plurality of connections between the plurality of modules based on the execution order.
- terms such as “obtaining,” “determining,” “establishing,” or the like refer to actions and processes performed or implemented by computing devices that manipulates and transforms data represented as physical (electronic) quantities within the computing device's registers and memories into other data similarly represented as physical quantities within the computing device memories or registers or other such information storage, transmission or display devices.
- the terms “first,” “second,” “third,” “fourth,” etc., as used herein are meant as labels to distinguish among different elements and may not necessarily have an ordinal meaning according to their numerical designation.
- Examples described herein also relate to an apparatus for performing the operations described herein.
- This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computing device selectively programmed by a computer program stored in the computing device.
- a computer program may be stored in a computer-readable non-transitory storage medium.
- Various units, circuits, or other components may be described or claimed as “configured to” or “configurable to” perform a task or tasks.
- the phrase “configured to” or “configurable to” is used to connote structure by indicating that the units/circuits/components include structure (e.g., circuitry) that performs the task or tasks during operation.
- the unit/circuit/component can be said to be configured to perform the task, or configurable to perform the task, even when the specified unit/circuit/component is not currently operational (e.g., is not on).
- the units/circuits/components used with the “configured to” or “configurable to” language include hardware--for example, circuits, memory storing program instructions executable to implement the operation, etc. Reciting that a unit/circuit/component is “configured to” perform one or more tasks, or is “configurable to” perform one or more tasks, is expressly intended not to invoke 35 U.S.C. 112, sixth paragraph, for that unit/circuit/component. Additionally, “configured to” or “configurable to” can include generic structure (e.g., generic circuitry) that is manipulated by software and/or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue.
- generic structure e.g., generic circuitry
- firmware e.g., an FPGA or a general-purpose processor executing software
- Configured to may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks.
- a manufacturing process e.g., a semiconductor fabrication facility
- devices e.g., integrated circuits
- Configurable to is expressly intended not to apply to blank media, an unprogrammed processor or unprogrammed generic computer, or an unprogrammed programmable logic device, programmable gate array, or other unprogrammed device, unless accompanied by programmed media that confers the ability to the unprogrammed device to be configured to perform the disclosed function(s).
Abstract
A system and method for automatically routing datasets based on input and output requirements of information is disclosed. The example method includes obtaining a list of modules executing on one or more processing devices, each module to generate an output dataset of a respective output requirement based on an input dataset of a respective input requirement. The method also includes establishing connections between the modules based on the input requirements and output requirements, including a first connection between a first module and a third module and a second connection between a second module and the third module. The method also includes receiving a first output dataset from the first module and receiving a second output dataset from the second module and, responsive to determining that the first module reported an error, routing the second output dataset to the third module without routing the first output dataset to the third module.
Description
- This application is a continuation of U.S. patent application Ser. No. 17/696,195, entitled “AUTOMATED GENERATION OF DEPENDENCY HIERARCHY BASED ON INPUT AND OUTPUT REQUIREMENTS OF INFORMATION”, filed Mar. 16, 2022, which is a continuation of U.S. patent application Ser. No. 17/515,239, entitled “AUTOMATED GENERATION OF DEPENDENCY GRAPH BASED ON INPUT AND OUTPUT REQUIREMENTS OF INFORMATION, filed Oct. 29, 2021, issued as U.S. Pat. No. 11,307,852 on Apr. 19, 2022, the disclosure of which is incorporated herein by reference in its entirety.
- The present disclosure relates generally to software technology, and more particularly, to systems and methods for automatically generating a dependency graph based on input and output requirements of information.
- A software deployment is the process for deploying, configuring, and updating software applications to help ensure maximum optimization, security, and compatibility within a computing network. A deployment is often scheduled to take place at times considered to be the least intrusive by an organization's network administrator. For major rollouts, this can result in staggered releases to help minimize interruptions to employee productivity and reduce strain upon the computing network, including providing post-deployment support.
- The described embodiments and the advantages thereof may best be understood by reference to the following description taken in conjunction with the accompanying drawings. These drawings in no way limit any changes in form and detail that may be made to the described embodiments by one skilled in the art without departing from the spirit and scope of the described embodiments.
-
FIG. 1 is a block diagram depicting an example environment for automatically generating a dependency graph based on input and output requirements of information, according to some embodiments; -
FIG. 2 is a block diagram depicting an example of themodule 126 inFIG. 1 , according to some embodiments; -
FIG. 3 is a flow diagram depicting a method for confirming a validity of an input dataset, according to some embodiments; -
FIG. 4 is a block diagram of a dependency graph of modules, according to some embodiments; -
FIG. 5A is a block diagram depicting an example of themodule management system 102 inFIG. 1 , according to some embodiments; -
FIG. 5B is a block diagram depicting an example of thecomputing system 124 of the environment inFIG. 1 , according to some embodiments; and -
FIG. 6 is a flow diagram depicting a method for handling connection pool sizing with heterogeneous concurrency, according to some embodiments. - A software deployment is the process for deploying, configuring, and updating software applications to help ensure maximum optimization, security, and compatibility within a computing network. A deployment is often scheduled to take place at times considered to be the least intrusive by an organization's network administrator. For major rollouts, this can result in staggered releases to help minimize interruptions to employee productivity and reduce strain upon the computing network, including providing post-deployment support.
- Software deployments are the entry point for a customer to access an organization's cloud computing service. When a new deployment is required, either due to customer demand, or because of adding a new customer, a manual procedure must be followed to bring the new deployment to life. This manual procedure is often a rigid, pre-defined order of operations that are carried out by various teams of developers, each working together to bring the software deployment online.
- However, team collaboration in software deployment often leads to bottlenecks. That is, each team of developers is dependent on the output of the other teams. Therefore, if a first team of developers is delayed in providing their output (e.g., code, instructions, data, etc.) to a second team of developers, then the second team must wait before they can begin developing their portion of the project. This leads to unnecessary delays and inefficiencies in software development and deployment.
- Aspects of the present disclosure address the above-noted and other deficiencies by automatically generating a dependency graph (e.g., a workflow) based on input and output requirements of information. As discussed in greater detail below, a module management system may obtain a list of a plurality of modules executing on one or more computing systems, where the plurality of modules may be associated with a plurality of input requirements and a plurality of output requirements. Each module may be configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements. The module management system may determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements. The module management system may establish a plurality of connections between the plurality of modules based on the execution order. The module management system may receive a first output dataset from a first module of the plurality of modules. The module management system may use one or more of the plurality of connections to redirect the first output dataset to a second module of the plurality of modules to cause the second module to generate a second output dataset based on the first output dataset. Each module may include one or more operations of a process or program, such that the execution of each of the modules according to the execution order causes the modules to execute the operations of the process or program, as a whole.
- Thus, the embodiments of the present disclosure determine the type of information that each operation needs in order to do its work, and finds what other operations that could provide the information. It then ensures that the operations are executed in the correct order to satisfy each operation's dependencies and marshals the data going into and out of each operation so that the operation itself does not have to know anything about the providing operation's data format. If any preceding operation's data output changes for any reason, then the embodiments of the present disclosure know to re-evaluate any dependent operation.
- Benefits of using the one or more embodiments of the present disclosure for automatically generating a dependency graph (e.g., a workflow) based on input and outputs requirements of information may include a reduction in networking resources needed to execute a process, as well as, a decrease in network congestion and power consumption for the overall network infrastructure.
-
FIG. 1 is a block diagram depicting an example environment for automatically generating a dependency graph based on input and output requirements of information, according to some embodiments. Theenvironment 100 includes acomputing system 124 a, acomputing system 124 b, and acomputing system 124 c (collectively referred as, computing systems 124). Theenvironment 100 includesmodules computing system 124 a may be configured to execute and/or store one or more ofmodules computing system 124 b may be configured to execute and/or store one or more ofmodules computing system 124 c may be configured to execute and/or store one or more ofmodules - A
computing system 124 may be any suitable type of computing device or machine that has a processing device, for example, server computers (e.g., an application server, a catalog server, a communications server, a computing server, a database server, a file server, a game server, a mail server, a media server, a proxy server, a virtual server, a web server), desktop computers, laptop computers, tablet computers, smartphones, set-top boxes, a graphics processing unit (GPU), etc. In some examples, a computing device may comprise a single machine or may include multiple interconnected machines (e.g., multiple servers configured in a cluster). - In some embodiments, a
computing system 124 may be any type of cloud computing platform (e.g., Amazon Web Services, Microsoft Azure, Google Cloud Platform, etc.). In some embodiments, thecomputing system 124 may be an Infrastructure-as-a-Service (IaaS) that provides users with compute (e.g., processing), networking, and storage resources. In some embodiments, thecomputing system 124 may be a Platform-as-a-Service (PaaS) that provides users with a platform on which applications can run, as well as the information technology (IT) infrastructure for it to run. In some embodiments, thecomputing system 124 may be a Software-as-a-Service (SaaS) that provides users with a cloud application, the platform on which it runs, and the platform's underlying infrastructure. In some embodiments, thecomputing system 124 may be a Function-as-a-Service (FaaS) that is an event-driven execution model that lets the developers build, run, and manage application packages as functions without maintaining the infrastructure. - In some embodiments, a
computing system 124 may include a one or more virtual machines (VM) that execute on a hypervisor which executes on top of an operating system (OS) of thecomputing system 124 to provide a virtual environment. The hypervisor may manage system sources (including access to hardware devices, such as processing devices, memories, storage devices). The hypervisor may also emulate the hardware (or other physical resources) which may be used by the VMs to execute software/applications. In another embodiment, a virtual environment may be a container that may execute on a container engine which executes on top of the OS for a computing device. For example, a container engine may allow different containers to share the OS of a computing device (e.g., the OS kernel, binaries, libraries, etc.). - The
environment 100 includes amodule management system 102 for managing the order in which themodules 126 are executed and/or directing the communication (e.g., input datasets, output datasets, etc.) between themodules 126. As shown inFIG. 1 , themodule management system 102 may be configured to execute and/or store one or more ofmodules module management system 102 is communicably coupled to each of thecomputing systems 124 via acommunication network 120. Theenvironment 100 includes acentralized database 114 and/or internal cache memory (not shown inFIG. 1 ) for storing information associated with themodules 126. - Although
FIG. 1 shows only a select number of computing devices (e.g.,computing systems 124 and module management system 102), components (e.g., modules 126), and storage devices (e.g., centralized databases 114); theenvironment 100 may include any number of computing devices, components and/or storage devices that are interconnected in any arrangement to facilitate the exchange of data between the computing devices, components and/or storage devices. Eachcomputing system 124 may execute any number (e.g., 1, 10s, 100s, 1000s, etc.) ofmodules 126 on its respective processing device. - The
communication network 120 may be a public network (e.g., the internet), a private network (e.g., a local area network (LAN) or wide area network (WAN)), or a combination thereof. In one embodiment,communication network 120 may include a wired or a wireless infrastructure, which may be provided by one or more wireless communications systems, such as wireless fidelity (Wi-Fi) connectivity to thecommunication network 120 and/or a wireless carrier system that can be implemented using various data processing equipment, communication towers (e.g. cell towers), etc. Thecommunication network 120 may carry communications (e.g., data, message, packets, frames, etc.) between any other the computing device. - A program or a process may include a plurality of operations (e.g., tasks, procedures, instructions). The
module management system 102 may be configured to divide the plurality of operations of the program into sets of one or more operations, where each set may be included in amodule 126. In other words, themodule management system 102 may be configured to divide the operations of a program into one ormore modules 126 that are independent and interchangeable, such that eachmodule 126 may contain everything necessary to execute portions of the desired functionality of the program. - A
module 126 may include a requires component that stores an input requirement. Amodule 126 may include a provides component that stores an output requirement. Amodule 126 may include an execute component that includes the one or more operations of themodule 126. The input requirement may be an input dataset (e.g., one or more fragments or segments of information) of a particular type that themodule 126 needs to consume (e.g., process) in order to execute the one or more operations of the execute component. The output requirement may be an output dataset of a particular type that themodule 126 generates (e.g., computes) based on the input dataset of the particular type. A particular type of an output dataset may correspond to code, an instruction, a data structure, a variable, a constant, a pointer (e.g., a reference to a location in memory), a list, a schema, a table, an object, and the like. In some embodiments, an output dataset may differ from another output dataset in at least one of a data format, a data value, or a data type. - A
module 126 may include a confirm component that is configured to verify (e.g., confirm) that the information that it receives from another entity (e.g., anothermodule 126, the module management system 102) conforms to the input requirements of themodule 126. For example, a second module (e.g.,module 126 b) may receive an input dataset from a first module (e.g.,module 126 a), and in response to receiving the input dataset, confirm whether the input dataset conforms (e.g., is compliant) to the input requirements of the second module. If the input dataset conforms to the input requirements, then the confirm component of the second module may be configured to allow the second module to generate an output dataset based on the input dataset, send the output dataset to another entity, and/or store the output dataset in memory/storage. If the input dataset of the second module does not conform to the input requirements, then the confirm component of the second module may be configured to prevent the second module from generating the output dataset, sending the output dataset to another entity, and/or storing the output dataset in memory/storage. - The confirm component of a
module 126 may be configured to verify that the information that it receives from another entity conforms to the input requirements of themodule 126 responsive to determining an occurrence of a triggering event. In some embodiments, a triggering event may occur when themodule 126 receives information (e.g., an input dataset), when anymodule 126 in the dependency graph changes state, and/or when there is an elapse of a predetermined amount of time (e.g., seconds, minutes, hours, days, etc.). - In some embodiments, the confirm component of a
module 126 may determine that a first dataset from a first module (e.g.,module 126 a) and a second dataset from a second module (e.g.,module 126 b) are each compliant to the input requirements of themodule 126 even though the two datasets may be different and/or originated fromdifferent modules 126. For example, a third module (e.g.,module 126 c) may receive a table of numbers from a first module (e.g.,module 126 a) and a string of alphanumeric characters from a second module (e.g.,module 126 b). The confirm components of the third module may determine that the table of numbers and the string of alphanumeric characters each conforms to the input requirements of the third module. - In some embodiments, a
module 126 may be configured to receive an input dataset and determine whether themodule 126 has a capability to generate an output dataset of a particular output requirement based on the input dataset. In some embodiments, amodule 126 may be configured to send a message to themodule management system 102 indicating the capability of themodule 126 to generate the output dataset of the particular output requirement based on the input dataset. In some embodiments, themodule management system 102 may be configured to modify an execution order (e.g., a workflow) of a plurality of modules responsive to determining that the message indicates that themodule 126 lacks the capability (e.g., computing resources, storage resources, memory resources, etc.) to generate the output dataset of the particular output requirement based on the input dataset. For example, themodule management system 102 may be configured to modify an execution order of the plurality of modules by replacingmodule 126 a withmodule 126 b responsive to determining from the message that themodule 126 lacks the processing capability to generate an output dataset. - A child module is a
module 126 that has its input connected to the output of anothermodule 126, referred to as a parent module. A child module depends on the parent module because the child module needs the output dataset from the parent module in order to generate its own output dataset. Themodule management system 102 may be configured to seamlessly replace (e.g., swap, switch) a parent module with a different parent module without the child module ever even knowing that the parent module was replaced. For example,module 126 a may be the parent module tomodule 126 b because there is a connection between the output ofmodule 126 a and the input ofmodule 126 b. While in this configuration, themodule management system 102 may replace themodule 126 a withmodule 126 c by removing the connection frommodule 126 a and re-attaching the connection tomodule 126 c such that the connection now exists betweenmodule 126 a andmodule 126 c; thereby makingmodule 126 c the parent module tomodule 126 a. - The confirm component of a
module 126 may be configured to verify (e.g., confirm) the validity of the output dataset by confirming the state of themodule 126 or whether the state of themodule 126 has changed. For example, if the confirm component ofmodule 126 a determines thatmodule 126 a was in an erred state while generating the output dataset or after generating the output dataset, then the confirm component may determine that the output dataset is invalid. If the confirm component ofmodule 126 a determines thatmodule 126 a was in an error-free state while generating the output dataset and stayed in the error-free state after generating the output dataset, then the confirm component may be configured to determine that the output dataset is valid. - If the confirm component of a
module 126 determines that themodule 126 was in an error-free state while generating the output dataset and stayed in the error-free state after generating the output dataset, then the confirm component may be configured to allow themodule 126 to generate the output dataset, send the output dataset to another entity (e.g., anothermodule 126, the module management system 102), and/or store the output dataset in memory/storage. If the confirm component of amodule 126 determines that themodule 126 was in an erred state while generating the output dataset, then the confirm component may be configured to preventmodule 126 from generating the output dataset, sending the output dataset to another entity (e.g., anothermodule 126, the module management system 102), and/or storing the output dataset in memory/storage. - If the confirm component of a
module 126 determines that themodule 126 was in an erred state after generating the output dataset, then the confirm component may be configured to generate a message (e.g., a flag) indicating that the output dataset is invalid and/or that a subset (e.g., one or more) of a plurality ofmodules 126 associated with a program need to re-generate their output datasets. The confirm component of amodule 126 may be configured to send the message to themodule management system 102. In response to receiving the message, themodule management system 102 may be configured to identify a subset ofmodules 126 of the plurality ofmodules 126 whose respective input dataset depends on (or is affected by) the state or change in state of themodule 126. - For example, an output of a first module (e.g.,
module 126 a) may be coupled to an input of a second module (e.g.,module 126 b), whose output is coupled to an input of a third module (e.g.,module 126 c). In this configuration, thefirst module 126 may generate a first output dataset and provide the first output dataset to thesecond module 126 to cause thesecond module 126 to generate a second output dataset based on the first output dataset and provide the second output dataset to thethird module 126. Now, if themodule management system 102 determines that the first output dataset is invalid, then themodule management system 102 may determine that thesecond module 126 and thethird module 126 need to re-generate their output datasets because they are dependent on the first output dataset. - In some embodiments, a
module 126 may be an erred state, an error-free state, an active state, or an inactive state. In some embodiments, amodule 126 may be in an erred state if themodule 126 is executing incorrectly and/or producing unexpected (e.g., undesired, unpredictable) results. In some embodiments, amodule 126 may be in an error-free state if themodule 126 is executing correctly and producing expected (e.g., desired, predictable) results. In some embodiments, an active state may be where amodule 126 is executing on a processing device of acomputing system 124, but not consuming an input dataset and/or generating an output dataset. In some embodiments, an inactive state may be where amodule 126 is not executing (e.g., not yet launched) on a processing device of acomputing system 124. - The
module management system 102 may be configured to send a request to each of thecomputing systems 124 for a list (e.g., one or more) of module identifiers corresponding to one ormore modules 126 that are executing and/or stored on thecomputing systems 124. In response to receiving the request, each of thecomputing systems 124 may generate a list of module identifiers corresponding to the one ormore modules 126 executing on their respective processors and/or stored in their respective memory/storage systems. For example,computing system 124 a may generate a first list of module identifiers corresponding tomodules 126 a-126 d,computing system 124 b may generate a second list of module identifiers corresponding tomodules 126 e-126 h, andcomputing system 124 c may generate a third list of module identifiers corresponding tomodules 126 i-126 l. In some embodiments, acomputing system 124 may generate a list of module identifiers for themodules 126 that are currently executing on a processing device of thecomputing system 124 and/or stored on a memory of thecomputing system 124. In some embodiments, acomputing system 124 may generate a list of module identifiers for themodules 126 that are currently executing on a processing device of thecomputing system 124, but exclude the modules from the list that are currently not executing on the processing device of thecomputing system 124. Each of thecomputing systems 124 may send their respective list of module identifiers to themodule management system 102. Since eachcomputing system 124 may execute its own set of modules, their respective lists of module identifiers refer to different sets ofmodules 126. - In some embodiments, a
computing system 124 may be configured to send a list of module identifiers to themodule management system 102 without having to receive a request from themodule management system 102. In some embodiments, acomputing system 124 may be configured to send a list of module identifiers to themodule management system 102 responsive to determining an occurrence of a triggering event. In some embodiments, a triggering event may be when there is an elapse of a predetermined amount of time (e.g., hours, days, weeks), when amodule 126 is executed (e.g., launched), when a module is terminated, and/or when amodule 126 in the dependency graph changes state. - In some embodiments, the
module management system 102 may be configured to generate a list of module identifiers corresponding to the one ormore modules 126 executing on a processor of themodule management system 102 and/or stored in a memory/storage system of themodule management system 102. For example, themodule management system 102 may generate a list of module identifiers corresponding tomodules 126 m-126 p. - The
module management system 102 may be configured to aggregate (e.g., combine, collect) each of the lists of module identifiers into a single list of module identifiers. Themodule management system 102 may be configured to store the list of module identifiers in memory/storage of themodule management system 102 to be retrieved by themodule management system 102 at a later time. A list of module identifiers may correspond to a plurality ofmodules 126 associated with a plurality of input requirements and a plurality of output requirements. - Each
module 126 of the plurality ofmodules 126 may be configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements. In some embodiments, two or more of the input requirements of the plurality of input requirements may be identical or different. For example, a first module (e.g.,module 126 a) may include an input requirement indicative of a table of numbers and a second module (e.g.,modules 126 b) may also include an input requirement indicative of a table of numbers. Alternatively, a first module (e.g.,module 126 a) may include an input requirement indicative of a table of numbers and a second module (e.g.,modules 126 b) may include an input requirement indicative of a string of alphanumeric characters. - In some embodiments, two or more of the output requirements of the plurality of output requirements may be identical or different. For example, a first module (e.g.,
module 126 a) may send a set of numbers to a second module (e.g.,module 126 a) and a third module (e.g.,module 126 b) to cause the second module to generate a string of alphanumeric characters based on the set of numbers and the third module to generate a string of alphanumeric characters based on the set of numbers. Alternatively, the first module (e.g.,module 126 a) may send a set of numbers to a second module (e.g.,module 126 b) and a third module (e.g.,module 126 c) to cause the second module to generate a string of alphanumeric characters based on the set of numbers and the third module to generate a table of numbers based on the set of numbers. - The
module management system 102 may be configured to determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements that are associated with the list of module identifiers. An execution order (e.g., a workflow) is the order in which a plurality of modules that are associated with a process or program are executed. To determine the execution order, themodule management system 102 may be configured to select a module identifier from the list of module identifiers and determine that themodule 126 corresponding to the selected module identifier includes a particular input requirement. Themodule management system 102 may be configured to scan (e.g., search) the list of module identifiers to determine whether there are other modules 126 (e.g., one or more) that include an output requirement that conforms to the input requirement of the selected module. Themodule management system 102 may be configured to establish, in response to the determination, a connection between the outputs of each of the other modules 126 (e.g., the ones with conforming output requirements) and the input of the selected module, such that the selected module receives one or more input datasets that conforms to its input requirements. - For example, the
module management system 102 may scan the list of module identifiers to determine that a third module (e.g.,module 126 c) includes an input requirement indicative of a table of numbers. Themodule management system 102 may scan the list of module identifiers to determine that a first module (e.g.,module 126 a) includes an output requirement indicative of a table of numbers and a second module (e.g.,module 126 b) includes an output requirement indicative of a function pointer. In response to the determination, themodule management system 102 may establish a connection between the output of thefirst module 126 and the input of thethird module 126, but not establish a connection between the output of thesecond module 126 and the input of thethird module 126. - The
module management system 102 may be configured to establish connections between outputs of one ormore modules 126 and inputs of one ormore modules 126 by maintaining a plurality of mappings (e.g., linkages, associations) between the identifiers of the modules in memory/storage. Eachmodule 126 may be configured to provide its output dataset (shown inFIG. 1 as, “Module Output Data”) to themodule management system 102. Themodule management system 102 may be configured to redirect (shown inFIG. 1 as, “Redirected Module Output Data”) the output dataset that it receives from eachmodule 126 to the inputs of one or more of theother modules 126 based on the plurality of mappings (sometimes referred to as, plurality of connections). For example, themodule management system 102 may update the plurality of mappings to indicate that the output of a first module (e.g.,module 126 a) is mapped to the input of a second module (e.g.,module 126 b). If themodule management system 102 receives an output dataset from the first module, then themodule management system 102 may determine that the plurality of mappings indicate that the output of the first module is mapped to the input of the second module, and in response, redirect the output dataset from the first module to the input of the second module. In some embodiments, an execution order may include (or indicate) a plurality of mappings associated with a plurality of modules. - The
module management system 102 may receive a plurality of output datasets from a plurality of modules that are connected (e.g., according to the plurality of mappings) to an input of asingle module 126. In this instance, themodule management system 102 may be configured to select the output dataset that should be redirected based on the availability of the dataset. For example, themodule management system 102 may receive a first output dataset from a first module (e.g.,module 126 a) during a first timeslot and a second output dataset from a second module (e.g.,module 126 b) during a second timeslot, where the first timeslot is earlier than the second timeslot. Themodule management system 102 may determine that both datasets could be redirected to the third module because the plurality of mappings indicate that the output of the first module and the output of the second module are each connected to the input of a third module (e.g.,module 126 c). In this instance, themodule management system 102 implements a tie-breaking approach by selecting the first dataset instead of the second dataset because the first dataset arrived at themodule management system 102 before the second dataset. Themodule management system 102 redirects the first dataset to the input of a third module and without redirecting the second dataset to the input of the third module. - In some embodiments, the
module management system 102 may include the same functionality as the confirm component of amodule 126. For example, themodule management system 102 may be configured to receive an output dataset from a first module (e.g.,module 126 a) and verify (e.g., confirm) whether the output dataset conforms to the input requirements of a second module (e.g.,module 126 b). Additionally, themodule management system 102 may be configured to receive an output dataset from the first module and verify the validity of the output data by confirming the state of the first module or whether the state of the first module has changed. In some embodiments, themodule management system 102 may be configured to simultaneously confirm allmodules 126 of an execution order or dependency graph. - In some embodiments, the
module management system 102 may be configured to modify (e.g., adjust, change) an execution order of the plurality of modules when the plurality of modules is in a non-executing state or when the plurality of modules is in an executing state (e.g., during runtime). In some embodiments, themodule management system 102 may be configured to modify an execution order of the plurality of modules by replacing amodule 126 of the plurality ofmodules 126 with adifferent module 126. In some embodiments, themodule management system 102 may be configured to modify an execution order of the plurality ofmodules 126 responsive to determining that one or more of themodules 126 of the execution order is in (or changed to) an error-state. - The
computing systems 124, themodules 126, and/or the module management system may each include their own interface, but each interface corresponds to a common interface type that uses a standardized protocol for communicating the output datasets. The common interface may allow a first module (e.g.,module 126 a) to generate and send an output dataset of a particular output requirement to a second module (e.g.,module 126 b), such that the particular output requirement conforms to the input requirement of the second module. - The common interface may allow the module management system 102 (or a system administrator of the environment 100) to modify a
module 126 of an execution order (e.g., workflow) without impacting the performance of any of theother modules 126 of an executive order. For example, an output of a first module (e.g.,module 126 a) may be connected to an input of a third module (e.g.,module 126 c). In this configuration, the first module may send a first output dataset of a particular output requirement to the input of the third module, and the third module may confirm that the first output dataset conforms to the input requirement of the third module. Themodule management system 102 may then replace the first module with a second module (e.g.,module 126 c), such that the second module may send a second output dataset of the particular output requirement to the input of the third module. The third module may confirm that the second output dataset conforms to the input requirement of the third module even though the first module was replaced with the second module because each of the modules use common interfaces. The common interface may allow a first module (e.g.,module 126 a) to send an output dataset to a second module (e.g.,module 126 b), where the second module is unaware that the first module was the module that generated and sent the output dataset. - The
module management system 102 may be configured to establish any combination of connections between any of themodules 126 executing on thecomputing system 124 a, any of themodules 126 executing on thecomputing system 124 b, and/or any of themodules 126 executing on thecomputing system 124 c. For example,module management system 102 may establish a connection between the output ofmodule 126 a and the input ofmodule 126 b, a connection between the output ofmodule 126 a and the input ofmodule 126 g, and/or a connection between the output ofmodule 126 a and the input of module 126 j. In some embodiments, themodule management system 102 may be configured to establish direct connections between themodules 126, such that the output dataset from afirst module 126 may pass to the input of asecond module 126 without having to rely on a computing device (e.g., the module management system 102) to receive the output dataset from thefirst module 126 and redirect the output dataset to thesecond module 126. -
FIG. 2 is a block diagram depicting an example of themodule 126 inFIG. 1 , according to some embodiments. Themodule 126 may include a requirescomponent 204 that stores an input requirement. Themodule 126 may include a provides component 210 that stores an output requirement. Themodule 126 may include an executecomponent 208 that stores one or more operations of themodule 126. The input requirement may be an input dataset (e.g., one or more fragments or segments of information) of a particular type that themodule 126 needs to consume in order to execute the one or more operations of the executecomponent 208. The output requirement may be an output dataset of a particular type that themodule 126 generates based on the input dataset of the particular type. A particular type may correspond to code, an instruction, a data structure, a variable, a constant, a pointer (e.g., a reference to a location in memory), a list, a schema, a table, an object, and the like. In some embodiments, an output dataset may differ from another output dataset in at least one of a data format, a data value, or a data type. - The
module 126 includes aconfirm component 206 that may be configured to verify (e.g., confirm) that the output dataset (e.g., this becomes the input dataset for the module 126) that it receives from another entity (e.g., anothermodule 126, the module management system 102) conforms to the input requirements of themodule 126. Theconfirm component 206 of themodule 126 may be configured to verify the validity of the output dataset (e.g., this becomes the input dataset for the module 126) by confirming the state of themodule 126 or whether the state of themodule 126 has changed. Themodule 126 may be configured to determine whether to process an input dataset based on whether the input dataset conforms to the input requirements of themodule 126. -
FIG. 3 is a flow diagram depicting a method for confirming a validity of an input dataset, according to some embodiments. Themethod 300 may be performed by themodule 126 inFIG. 1 . Although themethod 300 is described with respect to themodule 126, one or more of the blocks may be performed by themodule management system 102 inFIG. 1 . Themethod 300 illustrates example functions used by various embodiments. Although specific function blocks (“blocks”) are disclosed inmethod 300, such blocks are examples. That is, embodiments are well suited to performing various other blocks or variations of the blocks recited inmethod 300. It is appreciated that the blocks inmethod 300 may be performed in an order different than presented, and that not all of the blocks inmethod 300 may be performed. - As shown in
FIG. 3 , themethod 300 includes theblock 302 of starting, by themodule management system 102, amodule 126 to cause themodule 126 to execute on a processing device of thecomputing system 124. Themethod 300 includes theblock 304 of receiving, by themodule 126, information. Themethod 300 includes theblock 306 of determining, by themodule 126, whether the information conforms to the input requirements of the requirescomponent 204. If themodule 126 determines that the information conforms to the input requirements of the requirescomponent 204, then themodule 126 proceeds to block 306; otherwise themodule 126 proceeds to block 304, to repeatblock 304. - The
method 300 includes theblock 308 of running, by themodule 126, one or more confirm tests. Themethod 300 includes theblock 310 of determining, by themodule 126, whether the confirm tests succeeded (e.g., passed). If the confirm tests succeeded, then themodule 126 proceeds to block 320 to store the information in a memory/storage and proceed to block 322 to report that the module is a success. If the confirm tests did not succeed, then themodule 126 proceeds to block 312. - The
method 300 includes theblock 312 of running, by themodule 126, executed tasks. Themethod 300 includes theblock 314 of determining whether an error occurred as a result of running the executed tasks. If an error occurred, then themodule 126 proceeds to block 318 to report the error and proceed to block 304, to repeatblock 304. If an error did not occur, then themodule 126 proceeds to block 316 to store the information in a memory/storage and proceed to block 308, to repeatblock 308. -
FIG. 4 is a block diagram of a dependency graph (sometimes referred to as, an execution order, a workflow, or a dependency tree) of modules, according to some embodiments. Thedependency graph 400 shows the order in which a plurality of modules receive information (e.g., input datasets), execute tasks, and provide information (e.g., output datasets) based on their connections. Thegraph 400 includesmodules module 126 inFIG. 1 . Thegraph 400 includesconnection - As shown, an output of
module 426 a is coupled tomodule 426 b via connection 402. An output ofmodule 426 b is coupled to an input ofmodule 426 c viaconnection 404, an input ofmodule 426 d viaconnection 406, and an input ofmodule 426 e viaconnection 408. An output ofmodule 426 c is coupled to an input ofmodule 426 f viaconnection 410. An output ofmodule 426 f is coupled to an input ofmodule 426 g viaconnection 412 and an input ofmodule 426 h viaconnection 414. - According to the arrangement of
graph 400, themodule 426 a sends an output dataset tomodule 426 b via connection 402, causing themodule 426 b to generate an output dataset based on the input dataset. Themodule 426 b sends its output dataset tomodule 426 c viaconnection 404, causing themodule 426 c to generate an output dataset based on the input dataset. Themodule 426 b sends its output dataset tomodule 426 d viaconnection 406, causing themodule 426 d to generate an output dataset based on the input dataset. Themodule 426 b sends its output dataset tomodule 426 e viaconnection 408, causing themodule 426 e to generate an output dataset based on the input dataset. Themodule 426 c sends its output dataset tomodule 426 f viaconnection 410, causingmodule 426 f to generate an output dataset based on the input dataset. Themodule 426 f sends its output dataset tomodule 426 g viaconnection 412, causingmodule 426 g to generate an output dataset based on the input dataset. Themodule 426 f sends its output dataset tomodule 426 h viaconnection 414, causingmodule 426 h to generate an output dataset based on the input dataset. - In some embodiments, each of the connections 402-414 shown in
FIG. 4 may be a direct connection between an output of a first module 426 and an input of a second module 426, such that the output dataset from a first module 426 passes to the input of second module 426 without having to rely on a computing device (e.g., the module management system 102) to receive the output dataset from the first module 426 and redirect the output dataset to the second module 426. - In some embodiments, each of the connections 402-414 shown in
FIG. 4 may be an indirect connection between an output of a first module 426 and an input of a second module 426, such that the output dataset from a first module 426 passes to the input of second module 426 because a computing device (e.g., the module management system 102) must receive the output dataset from the first module 426 and redirect the output dataset to the second module 426. In some embodiments, themodule management system 102 redirects the output data from a first module 426 to an input of the second module 426 based on an execution order or a plurality of mappings associated with modules 426. In some embodiments, themodule management system 102 determines the execution order or the plurality of mappings for the modules 426 based on the plurality of input requirements associated with modules 426 and/or the plurality of output requirements associated with modules 426. -
FIG. 5A is a block diagram depicting an example of themodule management system 102 inFIG. 1 , according to some embodiments. While various devices, interfaces, and logic with particular functionality are shown, it should be understood that themodule management system 102 includes any number of devices and/or components, interfaces, and logic for facilitating the functions described herein. For example, the activities of multiple devices may be combined as a single device and implemented on a same processing device (e.g.,processing device 502 a), as additional devices and/or components with additional functionality are included. - The
module management system 102 includes aprocessing device 502 a (e.g., general purpose processor, a PLD, etc.), which may be composed of one or more processors, and amemory 504 a (e.g., synchronous dynamic random access memory (DRAM), read-only memory (ROM)), which may communicate with each other via a bus (not shown). - The
processing device 502 a may be provided by one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. In some embodiments,processing device 502 a may include a complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. In some embodiments, theprocessing device 502 a may comprise one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Theprocessing device 502 a may be configured to execute the operations described herein, in accordance with one or more aspects of the present disclosure, for performing the operations and steps discussed herein. - The
memory 504 a (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Non-volatile RAM (NVRAM), Flash Memory, hard disk storage, optical media, etc.) ofprocessing device 502 a stores data and/or computer instructions/code for facilitating at least some of the various processes described herein. Thememory 504 a includes tangible, non-transient volatile memory, or non-volatile memory. Thememory 504 a stores programming logic (e.g., instructions/code) that, when executed by theprocessing device 502 a, controls the operations of themodule management system 102. In some embodiments, theprocessing device 502 a and thememory 504 a form various processing devices and/or circuits described with respect tomodule management system 102. The instructions include code from any suitable computer programming language such as, but not limited to, C, C++, C#, Java, JavaScript, VBScript, Perl, HTML, XML, Python, TCL, and Basic. - The
processing device 502 a may execute amodule management component 510 a. In some embodiments, themodule management component 510 a may be configured to obtain a list of a plurality ofmodules 126 executing on one ormore computing systems 124. In some embodiments, the plurality ofmodules 126 may be associated with a plurality of input requirements and a plurality of output requirements. In some embodiments, eachmodule 126 may be configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements. - In some embodiments, the
module management component 510 a may be configured to determine an execution order of the plurality ofmodules 126 based on the plurality of input requirements and the plurality of output requirements. In some embodiments, themodule management component 510 a may be configured to establish a plurality of connections between the plurality ofmodules 126 based on the execution order. - In some embodiments, the
module management component 510 a may be configured to receive a first output dataset from afirst module 126 of the plurality ofmodules 126. In some embodiments, themodule management component 510 a may be configured to redirect, based on the execution order, the first output dataset to asecond module 126 of the plurality ofmodules 126 to cause thesecond module 126 to generate a second output dataset based on the first output dataset. - In some embodiments, the
module management component 510 a may be configured to receive the second output dataset from thesecond module 126 of the plurality ofmodules 126. In some embodiments, themodule management component 510 a may be configured to redirect, based on the execution order, the second output dataset to athird module 126 of the plurality of modules to cause thethird module 126 to generate a third output dataset based on the second output dataset. In some embodiments, themodule management component 510 a may be configured to redirect, based on the execution order, the second output dataset to afourth module 126 of the plurality ofmodules 126 to cause thefourth module 126 to generate a fourth output dataset based on the second output dataset. - In some embodiments, a first connection of the plurality of connections is between a
first module 126 of the plurality of modules and athird module 126 of the plurality ofmodules 126. In some embodiments, a second connection of the plurality of connections is between asecond module 126 of the plurality ofmodules 126 and thethird module 126 of the plurality ofmodules 126. In some embodiments, themodule management component 510 a may be configured to determine that a first output dataset that is generated by thefirst module 126 is available before a second output dataset that is generated by thesecond module 126. - In some embodiments, the
module management component 510 a may be configured to redirect, responsive to determining that the first output dataset is available before the second output dataset, the first output dataset to thethird module 126 of the plurality ofmodules 126 without redirecting the second output dataset to thethird module 126 of the plurality ofmodules 126. - In some embodiments, the
first module 126 was in a first state when generating the first output dataset. In some embodiments, thesecond module 126 was in a second state when generating the second output dataset. In some embodiments, themodule management component 510 a may be configured to determine that thefirst module 126 generated the first output dataset while in an error state. In some embodiments, themodule management component 510 a may be configured to determine, responsive to determining that thefirst module 126 generated the first output dataset while in an error state, that thesecond module 126 generated the second output dataset while in an error-free state. In some embodiments, themodule management component 510 a may be configured to redirect, responsive to determining that thesecond module 126 generated the second output dataset while in an error-free state, the second output dataset to thethird module 126 of the plurality ofmodules 126 without redirecting the first output dataset to thethird module 126 of the plurality ofmodules 126. - In some embodiments, the
module management component 510 a may be configured to determine a change in afirst module 126 of the plurality ofmodules 126 from an error-free state to an error state. In some embodiments, themodule management component 510 a may be configured to identify a subset ofmodules 126 of the plurality ofmodules 126 whose respective input dataset depends on the change. In some embodiments, themodule management component 510 a may be configured to cause each of the subset ofmodules 126 to re-generate the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements. - In some embodiments, the change in the
first module 126 of the plurality ofmodules 126 from the error-free state to the error state occurs after thefirst module 126 of the plurality ofmodules 126 generates the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements. - In some embodiments, the
module management component 510 a may be configured to determine, by thefirst module 126 of the plurality ofmodules 126, that a state of thefirst module 126 changed during or after the generation of the first output dataset. In some embodiments, themodule management component 510 a may be configured to generate, by thefirst module 126, a flag indicating an invalidity of the first output dataset or a requirement for a subset ofmodules 126 to re-generate a plurality of output datasets. - In some embodiments, the
module management component 510 a may be configured to replace afirst module 126 of the plurality ofmodules 126 with athird module 126 of the plurality ofmodules 126. In some embodiments, themodule management component 510 a may be configured to receive a third output dataset from thethird module 126 of the plurality ofmodules 126. In some embodiments, themodule management component 510 a may be configured to determine a compliance of the second output dataset to the input requirements of thesecond module 126 of the plurality ofmodules 126. In some embodiments, themodule management component 510 a may be configured to redirect, responsive to determining the compliance of the second output dataset to the input requirements of thesecond module 126, the third output dataset to thesecond module 126 of the plurality ofmodules 126. - In some embodiments, the
first module 126 sends the first output dataset to the processing device using a first interface of thefirst module 126, and thethird module 126 sends the third output dataset to the processing device using a second interface. In some embodiments, the first interface and the second interface correspond to a common interface type. In some embodiments, the second output dataset is different from the first output dataset in at least one of a data format, a data value, or a data type. In some embodiments, thesecond module 126 is unaware that thefirst module 126 of the plurality ofmodules 126 generated the first output dataset. In some embodiments, themodule management component 510 a may be configured to determine a capability to generate the output dataset of a respective output requirement of the plurality of output requirements based on the input dataset of a respective input requirement of the plurality of input requirements, and send a message to the processing device indicative of the capability. - In some embodiments, the
processing device 502 a may execute one ormore modules 126 inFIG. 1 . - The
module management system 102 includes anetwork interface 506 a configured to establish a communication session with a computing device for sending and receiving data over thecommunication network 120 to the computing device. Accordingly, the network interface 506A includes a cellular transceiver (supporting cellular standards), a local wireless network transceiver (supporting 802.11X, ZigBee, Bluetooth, Wi-Fi, or the like), a wired network interface, a combination thereof (e.g., both a cellular transceiver and a Bluetooth transceiver), and/or the like. In some embodiments, themodule management system 102 includes a plurality ofnetwork interfaces 506 a of different types, allowing for connections to a variety of networks, such as local area networks (public or private) or wide area networks including the Internet, via different sub-networks. - The
module management system 102 includes an input/output device 505 a configured to receive user input from and provide information to a user. In this regard, the input/output device 505 a is structured to exchange data, communications, instructions, etc. with an input/output component of themodule management system 102. Accordingly, input/output device 505 a may be any electronic device that conveys data to a user by generating sensory information (e.g., a visualization on a display, one or more sounds, tactile feedback, etc.) and/or converts received sensory information from a user into electronic signals (e.g., a keyboard, a mouse, a pointing device, a touch screen display, a microphone, etc.). The one or more user interfaces may be internal to the housing of themodule management system 102, such as a built-in display, touch screen, microphone, etc., or external to the housing of themodule management system 102, such as a monitor connected to themodule management system 102, a speaker connected to themodule management system 102, etc., according to various embodiments. In some embodiments, themodule management system 102 includes communication circuitry for facilitating the exchange of data, values, messages, and the like between the input/output device 505 a and the components of themodule management system 102. In some embodiments, the input/output device 505 a includes machine-readable media for facilitating the exchange of information between the input/output device 505 a and the components of themodule management system 102. In still another embodiment, the input/output device 505 a includes any combination of hardware components (e.g., a touchscreen), communication circuitry, and machine-readable media. - The
module management system 102 includes adevice identification component 507 a (shown inFIG. 5A asdevice ID component 507 a) configured to generate and/or manage a device identifier associated with themodule management system 102. The device identifier may include any type and form of identification used to distinguish themodule management system 102 from other computing devices. In some embodiments, to preserve privacy, the device identifier may be cryptographically generated, encrypted, or otherwise obfuscated by any device and/or component ofmodule management system 102. In some embodiments, themodule management system 102 may include the device identifier in any communication (e.g., establish connection request, resource request) that themodule management system 102 sends to a computing device. - The
module management system 102 includes a bus (not shown inFIG. 5A ), such as an address/data bus or other communication mechanism for communicating information, which interconnects the devices and/or components ofmodule management system 102, such asprocessing device 502 a,network interface 506 a, input/output device 505 a,device ID component 507 a,module management component 510 a, andmodules 126. - In some embodiments, some or all of the devices and/or components of
module management system 102 may be implemented with theprocessing device 502 a. For example, themodule management system 102 may be implemented as a software application stored within thememory 504 a and executed by theprocessing device 502 a. Accordingly, such embodiment can be implemented with minimal or no additional hardware costs. In some embodiments, any of these above-recited devices and/or components rely on dedicated hardware specifically configured for performing operations of the devices and/or components. -
FIG. 5B is a block diagram depicting an example of thecomputing system 124 of the environment inFIG. 1 , according to some embodiments. While various devices, interfaces, and logic with particular functionality are shown, it should be understood that thecomputing system 124 includes any number of devices and/or components, interfaces, and logic for facilitating the functions described herein. For example, the activities of multiple devices may be combined as a single device and implemented on a same processing device (e.g.,processing device 502 b), as additional devices and/or components with additional functionality are included. - The
computing system 124 includes aprocessing device 502 b (e.g., general purpose processor, a PLD, etc.), which may be composed of one or more processors, and amemory 504 b (e.g., synchronous dynamic random access memory (DRAM), read-only memory (ROM)), which may communicate with each other via a bus (not shown). Theprocessing device 502 b includes identical or nearly identical functionality asprocessing device 502 a inFIG. 5 a , but with respect to devices and/or components of thecomputing system 124 instead of devices and/or components of themodule management system 102. - The
memory 504 b ofprocessing device 502 b stores data and/or computer instructions/code for facilitating at least some of the various processes described herein. Thememory 504 b includes identical or nearly identical functionality asmemory 504 a inFIG. 5A , but with respect to devices and/or components of thecomputing system 124 instead of devices and/or components of themodule management system 102. - The
processing device 502 b may execute aclient component 510 b that may be configured to launch amodule 126 to cause themodule 126 to execute on the processing device 02 b. In some embodiments, theclient component 510 b may be configured to receive a request from themodule management system 102 for a list of module identifiers corresponding to one ormore modules 126 that are executing and/or stored on thecomputing systems 124. In some embodiments, theclient component 510 b may be configured to generate the list of module identifiers and send the list to themodule management system 102. - The
processing device 502 b may execute one ormore modules 126 that may be configured to receive an output dataset from another entity (e.g., anothermodule 126, the module management system 102). In some embodiments, themodule 126 may be configured to generate an output dataset of an output requirement based on an input dataset of an input requirement. In some embodiments, themodule 126 may be configured to send the output dataset to another entity (e.g., anothermodule 126, the module management system 102). - In some embodiments, the
module 126 may be configured to verify that the information that themodule 126 receives from another entity (e.g., anothermodule 126, the module management system 102) conforms to the input requirements of themodule 126. In some embodiments, themodule 126 may be configured to verify the validity of an output dataset by confirming the state of amodule 126 or whether the state of amodule 126 has changed. In some embodiments, themodule 126 may be configured to receive an input dataset and determine whether themodule 126 has a capability to generate an output dataset of a particular output requirement based on the input dataset. - The
computing system 124 includes anetwork interface 506 b configured to establish a communication session with a computing device for sending and receiving data over a network to the computing device. Accordingly, thenetwork interface 506 b includes identical or nearly identical functionality asnetwork interface 506 a inFIG. 2A , but with respect to devices and/or components of thecomputing system 124 instead of devices and/or components of themodule management system 102. - The
computing system 124 includes an input/output device 505 b configured to receive user input from and provide information to a user. In this regard, the input/output device 505 b is structured to exchange data, communications, instructions, etc. with an input/output component of the remote server 108. The input/output device 505 b includes identical or nearly identical functionality as input/output device 505 a inFIG. 5A , but with respect to devices and/or components of thecomputing system 124 instead of devices and/or components of themodule management system 102. - The
computing system 124 includes adevice identification component 507 b (shown inFIG. 5B asdevice ID component 507 b) configured to generate and/or manage a device identifier associated with thecomputing system 124. Thedevice ID component 507 b includes identical or nearly identical functionality asdevice ID component 507 a inFIG. 5A , but with respect to devices and/or components of thecomputing system 124 instead of devices and/or components of themodule management system 102. - The
computing system 124 includes a bus (not shown), such as an address/data bus or other communication mechanism for communicating information, which interconnects the devices and/or components of thecomputing system 124, such asprocessing device 502 b,network interface 506 b, input/output device 505 b,device ID component 507 b, theclient component 510 b, and themodules 126. - In some embodiments, some or all of the devices and/or components of
computing system 124 may be implemented with theprocessing device 502 b. For example, thecomputing system 124 may be implemented as a software application stored within thememory 504 b and executed by theprocessing device 502 b. Accordingly, such embodiment can be implemented with minimal or no additional hardware costs. In some embodiments, any of these above-recited devices and/or components rely on dedicated hardware specifically configured for performing operations of the devices and/or components. -
FIG. 6 is a flow diagram depicting a method for handling connection pool sizing with heterogeneous concurrency, according to some embodiments.Method 600 may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, a processor, a processing device, a central processing unit (CPU), a system-on-chip (SoC), etc.), software (e.g., instructions running/executing on a processing device), firmware (e.g., microcode), or a combination thereof. In some embodiments,method 600 may be performed by a module management system, such asmodule management system 102 inFIG. 1 . In some embodiments,method 600 may be performed by a computing system, such ascomputing system 124 inFIG. 1 . In some embodiments,method 600 may be performed by amodule 126 executing on thecomputing system 124. - With reference to
FIG. 6 ,method 600 illustrates example functions used by various embodiments. Although specific function blocks (“blocks”) are disclosed inmethod 600, such blocks are examples. That is, embodiments are well suited to performing various other blocks or variations of the blocks recited inmethod 600. It is appreciated that the blocks inmethod 600 may be performed in an order different than presented, and that not all of the blocks inmethod 600 may be performed. - As shown in
FIG. 6 , themethod 600 includes the block 602 of obtaining, by a processing device, a list of a plurality of modules executing on one or more computing systems, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module is configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements. Themethod 600 includes theblock 604 of determining, by the processing device, an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements. Themethod 600 includes theblock 606 of establishing, by the processing device, a plurality of connections between the plurality of modules based on the execution order. - The following examples pertain to further embodiments.
- Example 1 is a method. The method includes obtaining, by a processing device, a list of a plurality of modules executing on one or more processing devices, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module is configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements. The method includes determining, by the processing device, an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements. The method includes establishing, by the processing device, a plurality of connections between the plurality of modules based on the execution order.
- Example 2 is a method as in Example 1, further comprising receiving, by the processing device, a first output dataset from a first module of the plurality of modules; and redirecting, by the processing device based on the execution order, the first output dataset to a second module of the plurality of modules to cause the second module to generate a second output dataset based on the first output dataset.
- Example 3 is a method as in any of Examples 1-2, further comprising receiving, by the processing device, the second output dataset from the second module of the plurality of modules; redirecting, by the processing device based on the execution order, the second output dataset to a third module of the plurality of modules to cause the third module to generate a third output dataset based on the second output dataset; and redirecting, by the processing device based on the execution order, the second output dataset to a fourth module of the plurality of modules to cause the fourth module to generate a fourth output dataset based on the second output dataset.
- Example 4 is a method as in any of Examples 1-3, wherein a first connection of the plurality of connections is between a first module of the plurality of modules and a third module of the plurality of modules, wherein a second connection of the plurality of connections is between a second module of the plurality of modules and the third module of the plurality of modules, and further comprising determining, by the processing device, that a first output dataset that is generated by the first module is available before a second output dataset that is generated by the second module.
- Example 5 is a method as in any of Examples 1-4, further comprising redirecting, by the processing device responsive to determining that the first output dataset is available before the second output dataset, the first output dataset to the third module of the plurality of modules without redirecting the second output dataset to the third module of the plurality of modules.
- Example 6 is a method as in any of Examples 1-5, wherein the first module was in a first state when generating the first output dataset, and wherein the second module was in a second state when generating the second output dataset, and further comprising determining, by the processing device, that the first module generated the first output dataset while in an error state; determining, by the processing device responsive to determining that the first module generated the first output dataset while in an error state, that the second module generated the second output dataset while in an error-free state; and redirecting, by the processing device responsive to determining that the second module generated the second output dataset while in an error-free state, the second output dataset to the third module of the plurality of modules without redirecting the first output dataset to the third module of the plurality of modules.
- Example 7 is a method as in any of Examples 1-6, further comprising determining, by the processing device, a change in a first module of the plurality of modules from an error-free state to an error state; identifying, by the processing device, a subset of modules of the plurality of modules whose respective input dataset depends on the change; and causing each of the subset of modules to re-generate the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- Example 8 is a method as in any of Examples 1-7, wherein the change in the first module of the plurality of modules from the error-free state to the error state occurs after the first module of the plurality of modules generates the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- Example 9 is a method as in any of Examples 1-8, further comprising determining, by the first module of the plurality of modules, that a state of the first module changed during or after the generation of the first output dataset; and generating, by the first module, a flag indicating an invalidity of the first output dataset or a requirement for a subset of modules to re-generate a plurality of output datasets.
- Example 10 is a method as in any of Examples 1-9, further comprising replacing, by the processing device, a first module of the plurality of modules with a third module of the plurality of modules; receiving, by the processing device, a third output dataset from the third module of the plurality of modules; determining, by the processing device, a compliance of the third output dataset to the input requirements of the second module of the plurality of modules; and redirecting, by the processing device responsive to determining the compliance of the second output dataset to the input requirements of the second module, the third output dataset to the second module of the plurality of modules.
- Example 11 is a method as in any of Examples 1-10, wherein the first module sends the first output dataset to the processing device using a first interface of the first module, and the third module sends the third output dataset to the processing device using a second interface, and wherein the first interface and the second interface correspond to a common interface type.
- Example 12 is a method as in any of Examples 1-11, wherein the second output dataset is different from the first output dataset in at least one of a data format, a data value, or a data type.
- Example 13 is a method as in any of Examples 1-12, wherein the second module is unaware that the first module of the plurality of modules generated the first output dataset.
- Example 14 is a method as in any of Examples 1-13, wherein each module is configured to determine a capability to generate the output dataset of a respective output requirement of the plurality of output requirements based on the input dataset of a respective input requirement of the plurality of input requirements.
- Example 15 is a method as in any of Examples 1-14, wherein each module is configured to send a message to the processing device that is indicative of the capability.
- Example 16 is a system. The system includes a memory; and a processing device, operatively coupled to the memory, to obtain a list of a plurality of modules executing on one or more processing devices, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module is configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements. The processing device is further to determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements. The processing device is further to establish a plurality of connections between the plurality of modules based on the execution order.
- Example 17 is a system as in Example 16, wherein the processing device is further to: receive a first output dataset from a first module of the plurality of modules; and redirect, based on the execution order, the first output dataset to a second module of the plurality of modules to cause the second module to generate a second output dataset based on the first output dataset.
- Example 18 is a system as in any of Examples 16-17, wherein the processing device is further to receive the second output dataset from the second module of the plurality of modules; redirect, based on the execution order, the second output dataset to a third module of the plurality of modules to cause the third module to generate a third output dataset based on the second output dataset; and redirect, based on the execution order, the second output dataset to a fourth module of the plurality of modules to cause the fourth module to generate a fourth output dataset based on the second output dataset.
- Example 19 is a system as in any of Examples 16-18, wherein a first connection of the plurality of connections is between a first module of the plurality of modules and a third module of the plurality of modules, wherein a second connection of the plurality of connections is between a second module of the plurality of modules and the third module of the plurality of modules, and wherein the processing device is further to determine that a first output dataset that is generated by the first module is available before a second output dataset that is generated by the second module.
- Example 20 is a system as in any of Examples 16-19, wherein the processing device is further to redirect, responsive to determining that the first output dataset is available before the second output dataset, the first output dataset to the third module of the plurality of modules without redirecting the second output dataset to the third module of the plurality of modules.
- Example 21 is a system as in any of Examples 16-20, wherein the first module was in a first state when generating the first output dataset, and wherein the second module was in a second state when generating the second output dataset, and wherein the processing device is further to determine that the first module generated the first output dataset while in an error state; determine, responsive to determining that the first module generated the first output dataset while in an error state, that the second module generated the second output dataset while in an error-free state; and redirect, responsive to determining that the second module generated the second output dataset while in an error-free state, the second output dataset to the third module of the plurality of modules without redirecting the first output dataset to the third module of the plurality of modules.
- Example 22 is a system as in any of Examples 16-21, wherein the processing device is further to determine a change in a first module of the plurality of modules from an error-free state to an error state; identify a subset of modules of the plurality of modules whose respective input dataset depends on the change; and cause each of the subset of modules to re-generate the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- Example 23 is a system as in any of Examples 16-22, wherein the change in the first module of the plurality of modules from the error-free state to the error state occurs after the first module of the plurality of modules generates the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
- Example 24 is a system as in any of Examples 16-23, wherein the processing device is further to determine, by the first module of the plurality of modules, that a state of the first module changed during or after the generation of the first output dataset; and generate, by the first module, a flag indicating an invalidity of the first output dataset or a requirement for a subset of modules to re-generate a plurality of output datasets.
- Example 25 is a system as in any of Examples 16-24, wherein the processing device is further to replace a first module of the plurality of modules with a third module of the plurality of modules; receive a third output dataset from the third module of the plurality of modules; determine a compliance of the third output dataset to the input requirements of the second module of the plurality of modules; and redirect, responsive to determining the compliance of the second output dataset to the input requirements of the second module, the third output dataset to the second module of the plurality of modules.
- Example 26 is a system as in any of Examples 16-25, wherein the first module sends the first output dataset to the processing device using a first interface of the first module, and the third module sends the third output dataset to the processing device using a second interface, and wherein the first interface and the second interface correspond to a common interface type.
- Example 27 is a system as in any of Examples 16-26, wherein the second output dataset is different from the first output dataset in at least one of a data format, a data value, or a data type.
- Example 28 is a system as in any of Examples 16-27, wherein the second module is unaware that the first module of the plurality of modules generated the first output dataset.
- Example 29 is a system as in any of Examples 16-28, wherein each module is configured to determine a capability to generate the output dataset of a respective output requirement of the plurality of output requirements based on the input dataset of a respective input requirement of the plurality of input requirements, and send a message to the processing device that is indicative of the capability.
- Example 30 is a non-transitory computer-readable medium storing instructions that, when execute by a processing device, cause the processing device to obtain, by a processing device, a list of a plurality of modules executing on one or more processing devices, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module is configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements; determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements; and establish a plurality of connections between the plurality of modules based on the execution order.
- Unless specifically stated otherwise, terms such as “obtaining,” “determining,” “establishing,” or the like, refer to actions and processes performed or implemented by computing devices that manipulates and transforms data represented as physical (electronic) quantities within the computing device's registers and memories into other data similarly represented as physical quantities within the computing device memories or registers or other such information storage, transmission or display devices. Also, the terms “first,” “second,” “third,” “fourth,” etc., as used herein are meant as labels to distinguish among different elements and may not necessarily have an ordinal meaning according to their numerical designation.
- Examples described herein also relate to an apparatus for performing the operations described herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computing device selectively programmed by a computer program stored in the computing device. Such a computer program may be stored in a computer-readable non-transitory storage medium.
- The methods and illustrative examples described herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will appear as set forth in the description above.
- The above description is intended to be illustrative, and not restrictive. Although the present disclosure has been described with references to specific illustrative examples, it will be recognized that the present disclosure is not limited to the examples described. The scope of the disclosure should be determined with reference to the following claims, along with the full scope of equivalents to which the claims are entitled.
- As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “includes”, and/or “including”, when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Therefore, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.
- It should also be noted that in some alternative implementations, the functions/acts noted may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed substantially concurrently or may sometimes be executed in the reverse order, depending upon the functionality/acts involved.
- Although the method operations were described in a specific order, it should be understood that other operations may be performed in between described operations, described operations may be adjusted so that they occur at slightly different times or the described operations may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing.
- Various units, circuits, or other components may be described or claimed as “configured to” or “configurable to” perform a task or tasks. In such contexts, the phrase “configured to” or “configurable to” is used to connote structure by indicating that the units/circuits/components include structure (e.g., circuitry) that performs the task or tasks during operation. As such, the unit/circuit/component can be said to be configured to perform the task, or configurable to perform the task, even when the specified unit/circuit/component is not currently operational (e.g., is not on). The units/circuits/components used with the “configured to” or “configurable to” language include hardware--for example, circuits, memory storing program instructions executable to implement the operation, etc. Reciting that a unit/circuit/component is “configured to” perform one or more tasks, or is “configurable to” perform one or more tasks, is expressly intended not to invoke 35 U.S.C. 112, sixth paragraph, for that unit/circuit/component. Additionally, “configured to” or “configurable to” can include generic structure (e.g., generic circuitry) that is manipulated by software and/or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue. “Configured to” may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks. “Configurable to” is expressly intended not to apply to blank media, an unprogrammed processor or unprogrammed generic computer, or an unprogrammed programmable logic device, programmable gate array, or other unprogrammed device, unless accompanied by programmed media that confers the ability to the unprogrammed device to be configured to perform the disclosed function(s).
- The foregoing description, for the purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the embodiments of the present disclosure to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the embodiments and its practical applications, to thereby enable others skilled in the art to best utilize the embodiments and various modifications as may be suited to the particular use contemplated. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the embodiments of the present disclosure are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.
Claims (30)
1. A method comprising:
obtaining, by a processing device, a list of a plurality of modules executing on one or more processing devices, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements;
establishing, by the processing device, a plurality of connections between the plurality of modules based on the plurality of input requirements and the plurality of output requirements, wherein the plurality of connections comprises a first connection between a first module and a third module, and a second connection between a second module and the third module;
receiving a first output dataset from the first module and receiving a second output dataset from the second module; and
responsive to determining that the first module has reported an error, routing the second output dataset to the third module of the plurality of modules without routing the first output dataset to the third module of the plurality of modules.
2. The method of claim 1 , wherein establishing the plurality of connections between the plurality of modules based on the plurality of input requirements and the plurality of output requirements comprises:
determining an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements; and
establishing the plurality of connections between the plurality of modules based on the execution order.
3. The method of claim 1 , wherein establishing the plurality of connections between the plurality of modules based on the plurality of input requirements and the plurality of output requirements comprises:
generating a dependency hierarchy of the plurality of modules based on the plurality of input requirements and the plurality of output requirements; and
establishing the plurality of connections between the plurality of modules based on the dependency hierarchy.
4. The method of claim 1 , further comprising:
generating a third output dataset by the third module based on the second output dataset; and
routing, based on the plurality of connections, the third output dataset to a fourth module of the plurality of modules to cause the fourth module to generate a fourth output dataset based on the third output dataset.
5. The method of claim 1 , further comprising:
responsive to determining that the first module has reported the error, identifying a subset of modules of the plurality of modules whose respective input dataset depends on the first output dataset of the first module; and
causing each of the subset of modules to re-generate the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
6. The method of claim 1 , further comprising:
determining, by the first module, that a state of the first module changed during or after generating the first output dataset; and
reporting the error, by the first module, by generating a flag indicating an invalidity of the first output dataset caused by the error.
7. The method of claim 1 , wherein the first module sends the first output dataset to the processing device using a first interface of the first module, and the second module sends the second output dataset to the processing device using a second interface, and wherein the first interface and the second interface correspond to a common interface type.
8. The method of claim 1 , wherein the third module does not receive information about a source of the second output dataset.
9. The method of claim 1 , wherein each of the plurality of modules is configured to determine a capability to generate the output dataset of a respective output requirement of the plurality of output requirements based on the input dataset of a respective input requirement of the plurality of input requirements.
10. The method of claim 9 , wherein each of the plurality of modules is configured to send a message to the processing device that is indicative of the capability.
11. A system comprising:
a memory; and
a processing device, operatively coupled to the memory, to:
obtain a list of a plurality of modules executing on one or more processing devices, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements;
establish a plurality of connections between the plurality of modules based on the plurality of input requirements and the plurality of output requirements, wherein the plurality of connections comprises a first connection between a first module and a third module, and a second connection between a second module and the third module;
receive a first output dataset from the first module and receiving a second output dataset from the second module; and
responsive to an error reported by the first module, route the second output dataset to the third module of the plurality of modules without routing the first output dataset to the third module of the plurality of modules.
12. The system of claim 11 , wherein to establish the plurality of connections between the plurality of modules based on the plurality of input requirements and the plurality of output requirements, the processing device is to:
determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements; and
establish the plurality of connections between the plurality of modules based on the execution order.
13. The system of claim 11 , wherein to establish the plurality of connections between the plurality of modules based on the plurality of input requirements and the plurality of output requirements, the processing device is to:
generate a dependency hierarchy of the plurality of modules based on the plurality of input requirements and the plurality of output requirements; and
establish the plurality of connections between the plurality of modules based on the dependency hierarchy.
14. The system of claim 11 , wherein the processing device is further to:
receive a third output dataset generated by the third module based on the second output dataset; and
route, based on the plurality of connections, the third output dataset to a fourth module of the plurality of modules to cause the fourth module to generate a fourth output dataset based on the third output dataset.
15. The system of claim 11 , wherein the processing device is further to:
responsive to determining that the first module has reported the error, identify a subset of modules of the plurality of modules whose respective input dataset depends on the first output dataset of the first module; and
cause each of the subset of modules to re-generate the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
16. The system of claim 11 , wherein the processing device is further to:
receive, from the first module, a flag indicating an invalidity of the first output dataset caused by the error, wherein the flag is generated by the first module in response to a state change of the first module that occurred during or after generation of the first output dataset.
17. The system of claim 11 , wherein the first module is to send the first output dataset to the processing device using a first interface of the first module, and the second module is to send the second output dataset to the processing device using a second interface, and wherein the first interface and the second interface correspond to a common interface type.
18. The system of claim 11 , wherein the third module does not receive information about a source of the second output dataset.
19. The system of claim 11 , wherein each of the plurality of modules is configured to determine a capability to generate the output dataset of a respective output requirement of the plurality of output requirements based on the input dataset of a respective input requirement of the plurality of input requirements.
20. The system of claim 19 , wherein each of the plurality of modules is configured to send a message to the processing device that is indicative of the capability.
21. A non-transitory computer-readable medium storing instructions that, when execute by a processing device, cause the processing device to:
obtain a list of a plurality of modules executing on one or more processing devices, the plurality of modules associated with a plurality of input requirements and a plurality of output requirements, each module configured to generate an output dataset of a respective output requirement of the plurality of output requirements based on an input dataset of a respective input requirement of the plurality of input requirements;
establish a plurality of connections between the plurality of modules based on the plurality of input requirements and the plurality of output requirements, wherein the plurality of connections comprises a first connection between a first module and a third module, and a second connection between a second module and the third module;
receive a first output dataset from the first module and receiving a second output dataset from the second module; and
responsive to an error reported by the first module, route the second output dataset to the third module of the plurality of modules without routing the first output dataset to the third module of the plurality of modules.
22. The non-transitory computer-readable medium of claim 21 , wherein the instructions to cause the processing device to establish the plurality of connections between the plurality of modules based on the plurality of input requirements and the plurality of output requirements, cause the processing device is to:
determine an execution order of the plurality of modules based on the plurality of input requirements and the plurality of output requirements; and
establish the plurality of connections between the plurality of modules based on the execution order.
23. The non-transitory computer-readable medium of claim 21 , wherein the instructions to cause the processing device to establish the plurality of connections between the plurality of modules based on the plurality of input requirements and the plurality of output requirements, cause the processing device is to:
generate a dependency hierarchy of the plurality of modules based on the plurality of input requirements and the plurality of output requirements; and
establish the plurality of connections between the plurality of modules based on the dependency hierarchy.
24. The non-transitory computer-readable medium of claim 21 , wherein the instructions further cause the processing device to:
receive a third output dataset generated by the third module based on the second output dataset; and
route, based on the plurality of connections, the third output dataset to a fourth module of the plurality of modules to cause the fourth module to generate a fourth output dataset based on the third output dataset.
25. The non-transitory computer-readable medium of claim 21 , wherein the instructions further cause the processing device to:
responsive to determining that the first module has reported the error, identify a subset of modules of the plurality of modules whose respective input dataset depends on the first output dataset of the first module; and
cause each of the subset of modules to re-generate the output dataset of the respective output requirement of the plurality of output requirements based on the input dataset of the respective input requirement of the plurality of input requirements.
26. The non-transitory computer-readable medium of claim 21 , wherein the instructions further cause the processing device to:
receive, from the first module, a flag indicating an invalidity of the first output dataset caused by the error, wherein the flag is generated by the first module in response to a state change of the first module that occurred during or after generation of the first output dataset.
27. The non-transitory computer-readable medium of claim 21 , wherein the first module is to send the first output dataset to the processing device using a first interface of the first module, and the second module is to send the second output dataset to the processing device using a second interface, and wherein the first interface and the second interface correspond to a common interface type.
28. The non-transitory computer-readable medium of claim 21 , wherein the third module does not receive information about a source of the second output dataset.
29. The non-transitory computer-readable medium of claim 21 , wherein each of the plurality of modules is configured to determine a capability to generate the output dataset of a respective output requirement of the plurality of output requirements based on the input dataset of a respective input requirement of the plurality of input requirements.
30. The non-transitory computer-readable medium of claim 29 , wherein each of the plurality of modules is configured to send a message to the processing device that is indicative of the capability.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/049,570 US11625242B1 (en) | 2021-10-29 | 2022-10-25 | Automated generation of dependency hierarchy based on input and output requirements of information |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/515,239 US11307852B1 (en) | 2021-10-29 | 2021-10-29 | Automated generation of dependency graph based on input and output requirements of information |
US17/696,195 US11494182B1 (en) | 2021-10-29 | 2022-03-16 | Automated generation of dependency hierarchy based on input and output requirements of information |
US18/049,570 US11625242B1 (en) | 2021-10-29 | 2022-10-25 | Automated generation of dependency hierarchy based on input and output requirements of information |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/696,195 Continuation US11494182B1 (en) | 2021-10-29 | 2022-03-16 | Automated generation of dependency hierarchy based on input and output requirements of information |
Publications (2)
Publication Number | Publication Date |
---|---|
US11625242B1 US11625242B1 (en) | 2023-04-11 |
US20230134573A1 true US20230134573A1 (en) | 2023-05-04 |
Family
ID=81187218
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/515,239 Active US11307852B1 (en) | 2021-10-29 | 2021-10-29 | Automated generation of dependency graph based on input and output requirements of information |
US17/696,195 Active US11494182B1 (en) | 2021-10-29 | 2022-03-16 | Automated generation of dependency hierarchy based on input and output requirements of information |
US18/049,570 Active US11625242B1 (en) | 2021-10-29 | 2022-10-25 | Automated generation of dependency hierarchy based on input and output requirements of information |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/515,239 Active US11307852B1 (en) | 2021-10-29 | 2021-10-29 | Automated generation of dependency graph based on input and output requirements of information |
US17/696,195 Active US11494182B1 (en) | 2021-10-29 | 2022-03-16 | Automated generation of dependency hierarchy based on input and output requirements of information |
Country Status (1)
Country | Link |
---|---|
US (3) | US11307852B1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11307852B1 (en) * | 2021-10-29 | 2022-04-19 | Snowflake Inc. | Automated generation of dependency graph based on input and output requirements of information |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7644334B2 (en) * | 2006-11-27 | 2010-01-05 | Honeywell International, Inc. | Requirements-based test generation |
US10007594B2 (en) * | 2015-07-21 | 2018-06-26 | International Business Machines Corporation | Proactive cognitive analysis for inferring test case dependencies |
US11392393B2 (en) * | 2018-02-08 | 2022-07-19 | Sap Se | Application runtime configuration using design time artifacts |
US10768908B1 (en) * | 2019-02-25 | 2020-09-08 | Microsoft Technology Licensing, Llc | Workflow engine tool |
US11044139B1 (en) * | 2020-09-29 | 2021-06-22 | Atlassian Pty Ltd | Apparatuses, methods, and computer program products for dynamic generation and traversal of object dependency data structures |
US11307852B1 (en) * | 2021-10-29 | 2022-04-19 | Snowflake Inc. | Automated generation of dependency graph based on input and output requirements of information |
-
2021
- 2021-10-29 US US17/515,239 patent/US11307852B1/en active Active
-
2022
- 2022-03-16 US US17/696,195 patent/US11494182B1/en active Active
- 2022-10-25 US US18/049,570 patent/US11625242B1/en active Active
Also Published As
Publication number | Publication date |
---|---|
US11307852B1 (en) | 2022-04-19 |
US11494182B1 (en) | 2022-11-08 |
US11625242B1 (en) | 2023-04-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11442942B2 (en) | Modified representational state transfer (REST) application programming interface (API) including a customized GraphQL framework | |
US11093232B2 (en) | Microservice update system | |
US10146599B2 (en) | System and method for a generic actor system container application | |
JP6549787B2 (en) | Method and apparatus for deploying network services | |
US20200409671A1 (en) | System and method for autowiring of a microservice architecture | |
CN105593817B (en) | Method and system for flexible node composition on local or distributed computer system | |
US11537367B1 (en) | Source code conversion from application program interface to policy document | |
WO2019034066A1 (en) | Page update method and apparatus | |
US11625242B1 (en) | Automated generation of dependency hierarchy based on input and output requirements of information | |
CN112559143A (en) | Task scheduling method and system and computing device | |
EP4124950A1 (en) | Exposing a cloud api based on supported hardware | |
US20220391748A1 (en) | Method and system for application programming interface based container service for supporting multiple machine learning applications | |
US20220036206A1 (en) | Containerized distributed rules engine | |
US11755297B2 (en) | Compiling monoglot function compositions into a single entity | |
US20150212834A1 (en) | Interoperation method of newtork device performed by computing device including cloud operating system in could environment | |
US11620126B2 (en) | Dynamic multiple repository package management through continuous integration | |
US20230069124A1 (en) | Schema based type-coercion for structured documents | |
US20220342646A1 (en) | System and method for packaging standalone application modules into re-usable application and infrastructure resources | |
US10262054B2 (en) | Database and service upgrade without downtime | |
US20230246916A1 (en) | Service map conversion with preserved historical information | |
US11474845B2 (en) | System and method for versioned script management | |
US20230325169A1 (en) | Rebasing image layers utilising a repository-based strategy | |
Morman et al. | The Future of GNU Radio: Heterogeneous Computing, Distributed Processing, and Scheduler-as-a-Plugin | |
US11449320B2 (en) | Mechanism for deploying legacy applications on new generation hyperscalers | |
US20220391223A1 (en) | Adding expressiveness to plugin extensions using integration with operators |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: SNOWFLAKE INC., MONTANA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HESS, ALEXANDER;HARDIE, TERRY MARC;SIGNING DATES FROM 20211117 TO 20211203;REEL/FRAME:061954/0317 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |