HK1237462B - Managing parameter sets - Google Patents

Description

Management parameter sets

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. application serial number 62/026,228, filed on July 18, 2014.

Technical Field

This specification refers to management parameter sets.

Background Art

In data processing systems, certain types of users often desire access to reports that document the lineage of data as it passes through the system. Generally speaking, such "data lineage" reports can be used to mitigate risk, verify compliance requirements, streamline business processes, and protect data, among many other purposes. Ensuring that data lineage reports are accurate and complete is crucial.

Summary of the Invention

In one aspect, generally speaking, managing parameter value sets, lineage information reflecting relationships between instances of a general purpose computer program instantiated using those parameter value sets enables generation of more accurate and complete data lineage reports.

In another aspect, in general, a method for managing parameter value sets includes receiving a plurality of parameter value sets for a general-purpose computer program, and processing log entries associated with execution of instances of the general-purpose computer program, each instance of the general-purpose computer program being associated with one or more parameter values. The processing includes analyzing the general-purpose computer program to classify each of one or more parameters associated with the general-purpose computer program as a member of a first class of parameters or a second class of parameters; processing the log entries associated with the execution of a first instance of the general-purpose computer program to form a particular parameter value set, wherein the processing includes including in the particular set any values of the parameters appearing in the log entries that are classified as members of the first class and excluding from the particular set any values of the parameters appearing in the log entries that are classified as members of the second class; and determining whether to add the particular parameter value set to the plurality of parameter value sets based on a comparison of a first identifier of the particular parameter value set with identifiers of at least some of the plurality of parameter value sets.

Various aspects can include one or more of the following features.

Processing the log entry includes classifying the parameter based on whether the parameter affects data lineage associated with the general computer program.

The comparison of the first identifier of the specific parameter value set with the identifiers of at least some of the multiple parameter value sets includes: determining the first identifier based on the specific parameter value set and the identifier of the general computer program; determining multiple second identifiers, each second identifier corresponding to a parameter value set of at least some of the multiple parameter value sets; and comparing the first identifier with each of the multiple second identifiers to determine whether the first identifier matches any of the multiple second identifiers.

Determining whether to add the specific parameter value set to the multiple parameter value sets includes: if no second identifier matches the first identifier, determining to add the specific parameter value set to the multiple parameter value sets.

Determining the first identifier includes computing an identification string from the contents of the particular parameter value set, and determining the plurality of second identifiers includes computing identification strings from the contents of at least some of the plurality of parameter value sets.

Determining the first identifier includes forming a concatenation of one or more of: the identifier of the general computer program, a name-value pair of the specific parameter value set, a function prototype of the general computer program, and a project scope of the first instance of the general computer program.

Determining the first identifier includes applying a data mapping function to one or more of: an identifier of the general computer program, name-value pairs of the particular set of parameter values, function prototypes of the general computer program, and a project scope of the first instance of the general computer program.

The data mapping function includes a hash function.

The first category of parameters includes parameters that affect the logical operation of the general-purpose computer program, and the second category of parameters includes parameters that do not affect the logical operation of the general-purpose computer program.

The general computer program is specified as a dataflow graph including nodes representing data processing operations and links between the nodes representing the flow of data elements between the data processing operations.

For each of the plurality of parameters, the analyzing includes automatically classifying the parameter or accepting a user-defined classification for the parameter.

Automatically classifying the parameter includes: initially classifying the parameter as belonging to the first category of parameters; determining a number of unique values of the parameter in multiple executions of instances of the general computer program; and reclassifying the parameter as belonging to the second category of parameters if the number of unique values of the parameter exceeds a predetermined threshold.

Automatically classifying the parameter includes: initially classifying the parameter as belonging to the first category of parameters; determining whether changes in the value of the parameter during multiple executions of the instance of the general-purpose computer program affect data lineage associated with the general-purpose computer program; and if the changes in the value of the parameter do not affect the data lineage, reclassifying the parameter as belonging to the second category of parameters.

The method also includes forming an association between a log entry associated with execution of the first instance of the general purpose computer program and the particular set of parameter values.

The log entry associated with the execution of the first instance of the general-purpose computer program includes a log entry of an execution command for instantiating the general-purpose computer program, the log entry including one or more parameter values provided as arguments to the execution command.

The log entries associated with the execution of the first instance of the general-purpose computer program also include one or more of: an indication of the project in which the first instance executed, an indication of internal parameters of the first instance, and an indication of environment settings, global variables, and configuration variables used by the first instance.

The method also includes: processing all multiple parameter value sets for multiple general-purpose computer programs and all multiple log entries associated with the execution of instances of at least some of the multiple general-purpose computer programs to form a data lineage report, wherein all multiple parameter value sets include: expanded multiple parameter value sets for the general-purpose computer programs, and all multiple log entries associated with the execution of instances of at least some of the multiple general-purpose computer programs include: log entries of the execution of the first instance of the general-purpose computer program, including its association with the specific parameter value set.

Forming a data lineage report includes: for each parameter value set among all the multiple parameter value sets of the multiple general-purpose computer programs, processing all the multiple log entries associated with the execution of instances of the at least some of the multiple general-purpose computer programs to identify all log entries associated with the execution of the instances of the general-purpose computer programs corresponding to the parameter value set, and identifying the most recent instantiation time of the general-purpose computer program from the identified log entries associated with the execution of the instances of the general-purpose computer program; and determining whether to include the parameter value set in the data lineage report based on the most recent instantiation time of the general-purpose computer program.

Determining whether to include the parameter value set in the data lineage report based on the most recent instantiation time of the general computer program includes: comparing the most recent instantiation time with a predetermined time interval, and including the parameter value set in the data lineage report if the most recent instantiation time of the general computer program is within the predetermined time interval.

Forming the data lineage report includes: for each parameter value set in all the plurality of parameter value sets of the plurality of general-purpose computer programs, processing all the plurality of log entries associated with the execution of instances of at least some of the plurality of general-purpose computer programs to determine the number of log entries associated with the execution of the instance of the general-purpose computer program corresponding to the parameter value set, and determining whether to include the parameter value set in the data lineage report based on the number of log entries associated with the execution of the instance of the general-purpose computer program.

In another aspect, in general, software stored in non-transitory form on a computer-readable medium for managing parameter value sets includes instructions for causing a computing system to: receive a plurality of parameter value sets for a general-purpose computer program; process log entries associated with execution of instances of the general-purpose computer program, each instance of the general-purpose computer program being associated with one or more parameter values; and augment the plurality of parameter value sets based on the processing. The processing includes: analyzing the general-purpose computer program to classify each of the one or more parameters associated with the general-purpose computer program as a member of a first class of parameters or a second class of parameters; processing log entries associated with the execution of a first instance of the general-purpose computer program to form a particular parameter value set, wherein the processing includes: including in the particular set any values of the parameters appearing in the log entries that are classified as members of the first class and excluding from the particular set any values of the parameters appearing in the log entries that are classified as members of the second class; and determining whether to add the particular parameter value set to the plurality of parameter value sets based on a comparison of a first identifier of the particular parameter value set with identifiers of at least some of the plurality of parameter value sets.

In another aspect, in general, a computing system for managing parameter value sets includes: an input device or port for receiving a plurality of parameter value sets for a general-purpose computer program, and at least one processor for processing log entries associated with execution of instances of the general-purpose computer program, each instance of the general-purpose computer program being associated with one or more parameter values. The processing includes: analyzing the general-purpose computer program to classify each of one or more parameters associated with the general-purpose computer program as a member of a first class of parameters or a second class of parameters; processing the log entries associated with the execution of a first instance of the general-purpose computer program to form a particular parameter value set, wherein the processing includes: including in the particular set any values of the parameters appearing in the log entries that are classified as members of the first class and excluding from the particular set any values of the parameters appearing in the log entries that are classified as members of the second class; and determining whether to add the particular parameter value set to the plurality of parameter value sets based on a comparison of a first identifier of the particular parameter value set with identifiers of at least some of the plurality of parameter value sets.

In another aspect, in general, a computing system for managing parameter value sets includes: means for receiving a plurality of parameter value sets for a general-purpose computer program; and means for processing log entries associated with execution of instances of the general-purpose computer program, each instance of the general-purpose computer program being associated with one or more parameter values. The processing includes: analyzing the general-purpose computer program to classify each of one or more parameters associated with the general-purpose computer program as a member of a first class of parameters or a second class of parameters; processing log entries associated with the execution of a first instance of the general-purpose computer program to form a particular parameter value set. The processing includes: including in the particular set any values of the parameters appearing in the log entries that are classified as members of the first class, and excluding from the particular set any values of the parameters appearing in the log entries that are classified as members of the second class; and determining whether to add the particular parameter value set to the plurality of parameter value sets based on a comparison of a first identifier of the particular parameter value set with identifiers of at least some of the plurality of parameter value sets.

On the other hand, in general, a method for managing parameter value sets includes: receiving a general computer program; receiving a first parameter value set; generating an executable instance of the general computer program by instantiating the general computer program according to the first parameter value set; receiving data from one or more data sets; executing the executable instance of the general computer program to process at least some of the received data; generating a log entry for the executable instance of the general computer program, the log entry including at least some parameter values of the first parameter value set; storing the log entry; receiving the log entry; processing the log entry to form a specific parameter value set, wherein the processing includes: extracting at least some parameter values of the first parameter value set from the log entry, and forming the specific parameter value set from the extracted parameter values; and determining whether to add the specific parameter value set to the multiple pre-existing parameter value sets based on comparing a first identifier of the specific parameter value set with identifiers of at least some of the pre-existing parameter value sets in the multiple pre-existing parameter value sets.

Various aspects can include one or more of the following features.

The comparison of the identifier of the specific parameter value set with the identifiers of at least some of the pre-existing parameter value sets in a plurality of pre-existing parameter value sets includes: determining the first identifier based on the specific parameter value set and the identifier of the general computer program; determining a plurality of second identifiers, each second identifier corresponding to each pre-existing parameter value set in the at least some of the pre-existing parameter value sets; and comparing the first identifier with each of the plurality of second identifiers to determine whether the first identifier matches any of the plurality of second identifiers.

Determining whether to add the specific parameter value set to the plurality of pre-existing parameter value sets includes determining to add the specific parameter value set to the plurality of pre-existing parameter value sets if no second identifier matches the first identifier.

Determining the first identifier comprises computing an identification string from the contents of the particular parameter value set, and determining the plurality of second identifiers comprises computing identification strings from the contents of at least some of the plurality of pre-existing parameter value sets.

Determining the first identifier includes forming a concatenation of one or more of: the identifier of the general computer program, a name-value pair of the specific parameter value set, a function prototype of the general computer program, and an item scope of an executable instance of the general computer program.

Determining the first identifier includes applying a data mapping function to one or more of: an identifier of the general computer program, name-value pairs of the specific set of parameter values, function prototypes of the general computer program, and a project scope of an executable instance of the general computer program.

The data mapping function includes a hash function.

The method also includes analyzing the general purpose computer program to classify each of one or more parameters associated with the general purpose computer program as a member of a first class of parameters or a second class of parameters.

Processing the log entry to form a specific set of parameter values also includes: including in the specific set any extracted parameter values that appear in the log entry and are classified as members of the first class, and excluding from the specific set any extracted parameter values that appear in the log entry and are classified as members of the second class.

The first category of parameters includes parameters that affect the logical operation of the general-purpose computer program, and the second category of parameters includes parameters that do not affect the logical operation of the general-purpose computer program.

On the other hand, in general, a software stored in a non-transitory form on a computer-readable medium is provided for managing parameter value sets, the software comprising instructions for causing a computing system to: receive a general-purpose computer program; receive a first parameter value set; generate an executable instance of the general-purpose computer program by instantiating the general-purpose computer program according to the first parameter value set; receive data from one or more data sets; execute the executable instance of the general-purpose computer program to process at least some of the received data; generate a log entry for the executable instance of the general-purpose computer program, the log entry comprising at least some parameter values of the first parameter value set; store the log entry; receive the log entry; process the log entry to form a specific parameter value set, wherein the processing comprises: extracting at least some parameter values of the first parameter value set from the log entry and forming the specific parameter value set from the extracted parameter values; and determining whether to add the specific parameter value set to the multiple pre-existing parameter value sets based on a comparison of a first identifier of the specific parameter value set with identifiers of at least some of the pre-existing parameter value sets in the multiple pre-existing parameter value sets.

In another aspect, in general, a system for managing a set of parameter values comprises: a first computing system comprising: a first input device or port for receiving a general purpose computer program, a first set of parameter values, and data from one or more data sets; a first set of one or more processors configured to: generate an executable instance of the general purpose computer program by instantiating the general purpose computer program according to the first set of parameter values; execute the executable instance of the general purpose computer program to process at least some of the received data; generate a log entry for the executable instance of the general purpose computer program, the log entry including at least some of the parameter values of the first set of parameter values; a first output device or port; a port for storing the log entry in a storage device; the second computing system comprising: a second input device or port for receiving the log entry; a second group of one or more processors configured to: process the log entry to form a specific parameter value set, wherein the processing comprises: extracting at least some of the parameter values of the first parameter value set from the log entry and forming the specific parameter value set from the extracted parameter values; and determining whether to add the specific parameter value set to the multiple pre-existing parameter value sets based on a comparison of a first identifier of the specific parameter value set with identifiers of at least some of the pre-existing parameter value sets in the multiple pre-existing parameter value sets.

Various aspects can include one or more of the following advantages.

By discovering parameter sets using the methods described herein and augmenting existing parameter sets with the discovered parameter sets, a data lineage report generated using the augmented set of existing parameter sets more accurately represents the true data lineage of the data processing system. In particular, previously ignored portions of the data lineage of the data processing system are included in the data lineage report.

In some examples, the results of the parameter set discovery method can also be used to augment the log entries of the execution of the instance of the computer program (i.e., augment the log entries with information about the discovered parameter sets). The augmented log entries can be advantageously used to verify the logical connections between the computer program and/or data sets corresponding to the physical connections. The results of this verification ensure that the data lineage presented to the user shows the correct lineage relationship between the computer program and its inputs and outputs.

Other features and advantages of the invention will be apparent from the following description and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a block diagram of a system for discovering parameter sets.

FIG2 is a data flow diagram including subgraphs and their associated parameter sets.

FIG3 is a runtime configuration of the data flow graph of FIG2.

FIG4 is a static analysis configuration of the data flow graph of FIG2 .

FIG5 is a flow chart of a method for discovering parameter sets.

FIG6 is the first half of exemplary operations of a method for discovering parameter sets.

FIG7 is the second half of exemplary operations of a method for discovering parameter sets.

FIG8 is an example data flow graph including a first subgraph and a second subgraph.

FIG9 shows the data flow diagram of FIG8 , where the logical data set is parsed into a physical data set.

FIG. 10 illustrates a data lineage report for the data flow diagram of FIG. 8 .

FIG. 11 is an example data flow graph including a first subgraph and a second subgraph and having a repeated logical data set.

FIG12 shows the example data flow diagram of FIG11 , where the logical data sets are parsed into physical data sets.

FIG. 13 illustrates a data lineage report including data lineage breaks for the data flow diagram of FIG. 11 .

FIG14 illustrates a first technique for mitigating the impact of data lineage breaks in data lineage reporting.

FIG15 illustrates a second technique for mitigating the impact of data lineage breaks in data lineage reporting.

FIG16 illustrates a third technique for mitigating the impact of data lineage breaks in data lineage reporting.

FIG17 illustrates a fourth technique for mitigating the impact of data lineage outages in data lineage reporting.

FIG18 illustrates a fifth technique for mitigating the impact of data lineage outages in data lineage reporting.

FIG19 illustrates a sixth technique for mitigating the impact of data lineage outages in data lineage reporting.

DETAILED DESCRIPTION

FIG1 shows an example of a data processing system 100 that can use the parameter set discovery techniques described herein. The system includes a development environment 118, which, in some implementations, is a system for developing applications as data flow graphs 117, which include vertices (representing data processing components or data sets) connected by directed links between the vertices (representing flows of work elements, i.e., data). For example, such an environment is described in more detail in U.S. Publication No. 2007/0011668, entitled “Managing Parameters for Graph-Based Applications,” which is incorporated herein by reference. A system for performing such graph-based computations is described in U.S. Patent No. 5,966,072, entitled “Executing Computations Expressed as Graphs,” which is incorporated herein by reference. A data flow graph 117 created according to this system provides methods for inputting and outputting information to and from various processes represented by graph components, for moving information between processes, and for defining the order in which processes are run. The system includes an algorithm for selecting an inter-process communication method from any available method (e.g., a communication path based on the graph's links may use TCP/IP or UNIX domain sockets, or use shared memory to pass data between processes). The data flow graph 117 created by a developer 120 using the development environment 118 may be stored in a data storage system 116 accessible to the development environment 118 for subsequent use by other modules of the system 100.

Execution environment 104 includes parameter parsing module 106 and execution module 112. For example, execution environment 104 can reside on one or more general-purpose computers under the control of a suitable operating system, such as a version of the UNIX operating system. For example, execution environment 104 can include a multi-node parallel computing environment, including a configuration of a computer system using multiple central processing units (CPUs) or processor cores, which is local (e.g., a multi-processor system such as a symmetric multi-processing (SMP) computer), or locally distributed (e.g., multiple processors coupled as a cluster or massively parallel processing (MPP) system), or remote, or remotely distributed (e.g., multiple processors coupled via a local area network (LAN) and/or a wide area network (WAN), or any combination thereof).

The parameter parsing module 106 receives a specification of a data flow graph 117 from the data storage system 116 and parses the parameters of the data flow graph 117 (as described in more detail below) to prepare the data flow graph 117 for execution by the execution module 112. The execution module 112 receives the prepared data flow graph 117 from the parameter parsing module 106 and uses the prepared data flow graph 117 to process data from the data source 102 and generate output data 114. The output data 114 can be stored back in the data source 102 or in a data storage system 116 accessible by the execution environment 104, or used in other ways. In general, the data source 102 can include one or more data sources, such as storage devices or connections to online data streams, each of which can store or provide data in any of a variety of formats (e.g., database tables, spreadsheet files, plain text files, or native formats used by mainframes).

The storage device providing data source 102 may be local to execution environment 104, such as stored on a storage medium connected to the computer hosting execution environment 104 (e.g., hard drive 108), or may be remote from execution environment 104, such as hosted on a remote system (e.g., host computer 110) that communicates with the computer hosting execution environment 104 via a remote connection (e.g., provided by a cloud computing infrastructure).

System 100 also includes a metadata environment module 119 accessible to enterprise users 121 (e.g., data architects or business users). Metadata environment module 119 includes a data lineage module 115 that processes data flow graph 117 (or metadata representing data flow graph 117 and its referenced input and output datasets) to generate data lineage for data flow graph 117. Enterprise users 121 can review data for reasons such as validation and compliance checking of data flow graph 117. Data lineage information about a particular data item (e.g., a dataset or a field within a dataset) is based on dependencies arising from processing performed by a data processing system, and the term "data lineage" as used herein generally refers to a collection of related data items and the processing entities that consume or generate those data items. A data lineage report (also known as a data lineage diagram) can include a graphical representation of data lineage in the form of a diagram with nodes representing data items and processing entities and links representing dependencies between them. Some systems capable of generating and displaying data lineage reports can automatically provide end-to-end data lineage from the ultimate data source at the upstream end to the final data produced at the downstream end. Nodes on a path upstream from a particular data item are sometimes referred to as "dependencies" of that data item, and nodes on a path downstream from a particular data item are sometimes referred to as "influences" of that data item. Although, as used herein, "data lineage" is sometimes used to refer only to upstream dependencies, "data lineage" can refer to both upstream dependencies and/or downstream influences as appropriate to a particular context.

1 Data Flow Diagram Overview

2, there is shown an example of a data flow graph 217 generated using the development environment 118 of FIG 1. FIG 1 includes a first subgraph 202 named gather.mp and a second subgraph 204 named process.mp.

The first subgraph 202 receives as input a first logical dataset DS1 206 and a second logical dataset DS2 208, processes the data from the first logical dataset 206 and the second logical dataset DS2 208, and writes the processed results to a third logical dataset DS3 210. The second subgraph 204 receives as input a fourth logical dataset DS4 212 (which happens to point to the same physical file as the third logical dataset 210), processes the data from the fourth logical dataset 212, and writes the processed results of the fourth logical dataset 212 to a table 214.

Each of the four logical data sets 206, 208, 210, and 212 is associated with a parameterized path that resolves to a physical file path at runtime. Specifically, the first logical data set 206 is identified using the parameterized path /${FEED}/inv_${DATE}.dat, the second logical data set 208 is identified using the parameterized path /${FEED}/cust_${DATE}.dat, the third logical data set 210 is identified using the parameterized path /trans_${DATE}.dat, and the fourth logical data set 212 is identified using the parameterized path /trans_${DATE}.dat.

The first subgraph 202 receives two parameters, P1=FEED and P2=DATE, as arguments, and, as described in more detail below, uses the parameters to resolve paths to the corresponding physical locations of the first logical dataset 206, the second logical dataset 208, and the third logical dataset 210, replacing the FEED and DATE placeholders in the parameterized paths with the values of the received FEED and DATE parameters. Additionally, the first subgraph 202 includes a "statically analyzed" value for the DATE parameter. As described in more detail below, the statically analyzed value for the DATE parameter is a placeholder value that is used as the parameter value during static analysis of the dataflow graph 217 (i.e., when determining the data lineage of the dataflow graph 217).

Similarly, the second subgraph 104 receives a single parameter, P1=DATE, and uses this parameter to resolve a path to the physical location of the fourth logical dataset 212 by replacing the DATE placeholder in the parameterized path of the fourth logical dataset 212 with the value of the received DATE parameter. Additionally, the second subgraph 204 includes a "statically analyzed" value for the DATE parameter. As described in more detail below, the statically analyzed value for the DATE parameter is a placeholder value that is used as a parameter value during static analysis of the dataflow graph 217 (i.e., when determining the data lineage of the dataflow graph 217).

Because the operation of dataflow graph 217 and its subgraphs depends on the parameters they receive, the dataflow graph and its subgraphs are sometimes referred to as "generic" dataflow graphs or "generic" computer programs.

1.1 Parameters

In general, the above parameters can be specified as either "design-time" parameters or "run-time" parameters. Design-time parameters affect the logical operation of their associated dataflow graph, except for their use in path resolution, as described above. In contrast, run-time parameters are provided to the graph on a job-by-job basis and do not affect the graph's logical operation. In some examples, the logical operation of a dataflow graph refers to both the graph's functionality and the logical data sets used by the graph.

In Figure 2, the FEED parameter is a design-time parameter that affects the logical operation of the gather.mp subgraph. For example, for one value of the FEED parameter, sort component 216 in first subgraph 202 may sort the data it receives in ascending order, while a different value of the FEED parameter may cause sort component 216 to sort the data in descending order. In some examples, a dataflow graph that includes a design-time parameter is referred to as a "generic graph" because its logical operation changes based on the value of the provided design-time parameter.

The DATE parameter is a run-time parameter that has no effect on the logical operation of the subgraph 202 and is provided on a job-by-job basis.

1.2 Parameter Set

In some examples, common parameter sets for dataflow graphs are stored as "parameter sets" (sometimes called "psets") that can be saved to disk and easily reused. For example, in Figure 2, the first subgraph 202 has three psets associated with it, PSET_mexico 218, PSET_canada 220, and PSET_usa 222. PSET_mexico 218 includes a common FEED parameter value "mexico" and a common DATE parameter value "today()," which is a function that returns today's date. PSET_canada 220 includes a common FEED parameter value, "canada," and a common DATE parameter value "today()." PSET_usa 222 includes a common FEED parameter value, "usa," and a common DATE parameter value "today()."

Similarly, the second subgraph 204 has a single pset associated with it, PSET 223. PSET 223 includes the commonly used DATE parameter value "today()", which is a function that returns today's date.

2 parameter resolution module

In some examples, before the dataflow graph 117 is executed by the execution module 112, the parameter parsing module 106 of FIG1 identifies one or more psets associated with the dataflow graph 117 (and its associated subgraphs 202, 204) and determines the number of unique design-time parameters in the one or more psets. For each unique design-time parameter of a given dataflow graph, the parameter parsing module 106 instantiates a separate executable instance of the dataflow graph. For example, referring to FIG3, for the dataflow graph 217 of FIG2, three instances of the first subgraph 202, gather.mp, are instantiated (PSET_mexico->gather.mp 202a, PSET_canada->gather.mp 202b, and PSET_usa->gather.mp 202c), each instance being configured according to a different one of the following three unique FEED parameters in the pset of FIG2: mexico, canada, and usa. Since the second subgraph 204 is associated with only a single pset 223 that does not include any design-time parameters, only a single instance of the second subgraph 204 (process.mp 204a) is instantiated at execution time.

Once the appropriate instances of the subgraphs 202, 204 are instantiated by the parameter parsing module 106, the parameter parsing module 106 replaces the parameter value placeholders in the parameterized path of the dataset with the actual parameter values from the pset, resolving the path to the physical location of the dataset. For example, for the PSET_mexico->gather.mp instance 202a of the first subgraph 202, the path of the first dataset 206 is resolved to //mexico/inv_031014 because the FEED parameter value is 'mexico' and the DATE parameter value is '031014'.

Once the parameter parsing module 106 has instantiated the data flow graph 217 including its subgraphs 202 and 204 and has resolved the physical paths to the data sets of the data flow graph 217, the data flow graph 217 is ready for execution by the execution module 112. During execution, the three instances 202a, 202b, and 202c of the first subgraph 202 read data from their respective input data sets, process the data, and store the processed data in the /trans_031014.dat physical file. Since the input data set (e.g., DS4 212) for the instance 204a of the second subgraph 202 is resolved to the same physical file as the output data set of the first subgraph, the /trans_031014.dat physical file is read by the instance of process.mp and then processed and stored in the table 214.

3Data Lineage Module

4 , in some examples, rather than executing the data flow graph 217, the data architect or business user 121 of FIG1 may need to examine the lineage of data as it passes through the data flow graph 217. To this end, the data lineage module 115 of FIG1 is configured to analyze the data flow graph 217 to generate a data lineage report for presentation to the data architect or business user 121.

In some examples, as a first step in determining the data lineage of the dataflow graph 217, the data lineage module 115 identifies the various subgraphs 202, 204 of the dataflow graph 217. For each identified subgraph 202, 204, the data lineage module 115 identifies one or more psets 218, 220, 222, 223 associated with the subgraph 202, 204, and then determines the number of unique design-time parameters in the one or more psets 218, 220, 222, 223 for the subgraph 202, 204. For each unique design-time parameter, the parameter resolution module instantiates a separate instance of the subgraph 202, 204.

In some examples, the data lineage module 115 operates under the assumption that the actual physical files and the data they store are irrelevant to the data lineage analysis. Therefore, any runtime parameter values used to resolve the physical location of the data set are unnecessary and can be replaced with placeholder values. As described above, for each runtime parameter associated with a subgraph, a corresponding placeholder, static analysis parameter value is included in the subgraph. For example, in Figure 2, because both dataflow graphs 202 and 204 include a DATE runtime parameter, they also include a placeholder, static analysis parameter value of 'MMDDYY'.

When the data lineage module 115 analyzes the data flow graph 217 to determine data lineage, all instances of the DATE parameter in the data flow graph are replaced with the 'MMDDYY' placeholder value, thereby creating a temporary dataset object 452 as shown in Figure 4. The interconnections between the various subgraph instances and the temporary dataset objects are then identified and presented to the data architect or business user as data lineage. For example, the analysis of instances 202a, 202b, 202c of the first subgraph 202 indicates that all instances of the first subgraph 202 write data to the dataset represented by the /trans_MMDDYY.dat dataset object. The analysis then indicates that instance 204a of the second data flow graph 204 reads from the dataset represented by the /trans_MMDDYY.dat dataset object. Based on this information, the data lineage of the data flow graph 217 indicates that the outputs of instances 202a, 202b, 202c of the first subgraph 202 are fed into the inputs of instance 204a of the second subgraph 204.

4 Logical PSET discovery and creation methods

In some examples, a given dataflow graph is executed using an execute command that receives parameter values as arguments provided to the execute command, rather than from a previously stored pset. Because the above method uses only stored psets to determine data lineage, the psets associated with parameter values derived from the arguments provided to the execute command used to execute the dataflow graph are not represented in the data lineage. This may result in incomplete or incorrect data lineage being presented to enterprise architects or auditors.

FIG5 is a flow diagram illustrating a method for augmenting a repository of existing logical parameter sets (psets) of a dataflow graph with created logical psets based on parameter sets identified in a log associated with the execution of an instance of the dataflow graph. In some examples, the method depicted in FIG5 is implemented by the data lineage module 115 of FIG1 .

4.1 Chart Parameters

Initially, an example of a dataflow graph (e.g., the first subgraph 202 of FIG. 1 ) includes two parameters (P ₁ and P ₂ ), each of which can be designated as a “design time” parameter or a “run time” parameter. As described above, design time parameters are parameters that affect the logical operation of the graph (e.g., can change the transformations performed by the graph), while run time parameters are parameters that change on a job-by-job basis (e.g., dates) and do not affect the logical operation of the graph.

4.2 Parameter classification

The graph 202 is provided to a parameter classification step 424, which analyzes the parameters of the graph 202 to generate a parameter classification result 426. In the parameter classification result 426, each parameter is classified as a design-time parameter or a run-time parameter. In the exemplary case shown in the flowchart, _P1 is classified as a design-time parameter and _P2 is classified as a run-time parameter.

In some examples, parameters of a dataflow diagram are pre-classified (e.g., by a user) as either design-time parameters or runtime parameters. In other examples (e.g., for legacy dataflow diagrams), parameters for a dataflow diagram are not pre-classified as either design-time parameters or runtime parameters. In this case, the parameter classification step 424 may assume that all parameters are design-time parameters. In a subsequent reclassification step, if a given parameter is determined to have a large (e.g., above a given threshold) number of unique values in a set of log entries (e.g., the job log data store described below), then the given parameter may be reclassified as a runtime parameter. Alternatively, the reclassification may be based on a data lineage sensitivity analysis. Specifically, if a parameter can take on a variety of different values without changing the data lineage within the dataflow diagram (i.e., the impact or dependencies of data sets or components within the dataflow diagram), then the parameter may be classified as a runtime parameter. For example, if the associated record format or other characteristics of the data sets in the diagram (e.g., DS1, DS2, DS3 in FIG3 ) are not affected by the various values of the parameter, then the parameter may be reclassified as a runtime parameter. Variations of this data lineage sensitivity analysis may be used, such as a more comprehensive data lineage sensitivity analysis that includes addressing all internal impacts and dependencies, and a more limited data lineage sensitivity analysis that includes addressing only impacts and dependencies associated with the dataset record format.

In some examples (e.g., for legacy data flow graphs), a parameter may include both a design-time portion and a runtime portion. For example, the file name parameter "/mexico/inv_031014.dat" may be a hybrid parameter because it includes a design-time portion (i.e., "mexico") and a runtime portion (i.e., "031014"). In such an example, a user may provide a regular expression or some other type of string parsing rule that is used by parameter classification step 424 to extract and classify the corresponding design-time and runtime parameters from the hybrid parameter.

4.3 Job Log Data Storage

The method utilizes a job log data store 428 that includes a plurality of job log entries 429, each of which includes information associated with the execution of an instance of a dataflow graph 202. Among other information, at least some of the job log entries include a record of an execute command used to instantiate the dataflow graph 202. The execute command for a given job log entry includes a graph name and parameter values provided as arguments to the execute command. Typically, at least some of the job log entries in the job log data store 428 instantiate the dataflow graph without accessing any parameter set, but rather receive parameter values as arguments to the execute command.

4.4 Processing Loop

The job log data store 428 and the parameter classification results 426 are provided to a processing loop 430 which, for each job log entry 429 in the job log data store 428, generates a new logical pset for the chart execution command, determines whether the new logical pset already exists in a repository 448 of existing logical psets, and adds the new logical pset to the repository 448 if it does not already exist.

4.4.1 Initial command line logic pset construction

Within processing loop 430, parameter classification result 426 and job log entry J _n 432 from job log data store 428 are provided to logical pset construction step 434, which analyzes job log entry 432 based on parameter classification result 426 to generate logical pset 436. To do so, logical pset construction step 434 analyzes the graph execution command included in job log entry 432 to extract parameter values included as arguments to the graph execution command. Logical pset construction step 434 also extracts the project scope included in job log entry 432. In some examples, the project scope includes an indication of the project in which the dataflow graph is executed, an indication of internal parameters of the dataflow graph, and an indication of environment settings, global variables, and configuration variables used by the dataflow graph.

Logical pset construction step 434 automatically includes the extracted project scopes in logical pset 436. Logical pset construction step 434 then matches each extracted parameter value with a corresponding parameter in parameter classification results 426. If logical pset construction step 434 determines that the extracted parameter value corresponds to a design-time parameter in parameter classification results 426, logical pset construction step 434 includes the value of the extracted design-time parameter in logical pset 436. If logical pset construction step 434 determines that the extracted parameter value corresponds to a run-time parameter in parameter classification results 426, the extracted parameter value is not included in logical pset 436.

4.4.2pset signature string calculation

The logical pset 436 is provided to a pset signature string calculation step 442, which calculates a logical pset signature string 444 based on the project scope and the parameter values in the logical pset 436. In some examples, the pset signature string 444 is calculated by serializing the project scope of the logical pset 436, the name/value pairs of the parameters in the logical pset 436, and the prototype of the data flow graph associated with the logical pset 436. In other examples, the pset signature string 444 is calculated by applying a hash function or some other data mapping algorithm to the logical pset 436.

4.4.3pset Signature String Search

The pset signature string 444 is provided to a pset signature search step 446 along with the pset signature strings of all existing logical psets in a repository of existing logical psets 448. For each existing logical pset, the pset signature string pset of the existing logical pset is compared to the pset signature string 444. If the pset signature string 444 matches at least one of the pset signature strings of the existing logical psets, no action is required because the logical pset instantiated by the execute command for the graph 432 already exists in the repository of existing logical psets 448.

In some examples, the pset signature strings for all existing logical psets in the repository of existing logical psets 448 are stored alongside the existing logical psets in the repository 448. In other examples, the signature strings for existing logical psets are calculated automatically (on the fly) and on an as-needed basis.

4.4.4 Adding a new logical pset

Otherwise, if none of the signature strings of the existing logical psets matches the pset signature string 444 , the logical pset 436 and its signature string 444 are added as a new logical pset to the repository 448 of existing logical psets via a new logical pset addition step 450 .

4.5 Examples

6 and 7 , the exemplary operations of the logical PSET discovery and creation method of FIG4 are applied to the first subgraph 202 of FIG2 . The first subgraph 202 of FIG2 includes two parameters, P1=FEED and P2=DATE. The first subgraph 202 is provided to a parameter classification step 424 , where the parameters are classified as either "design-time" parameters or "run-time" parameters, generating a parameter classification result 426 . The parameter classification result 426 indicates that the P1 (FEED) parameter is a design-time parameter and the P2 (DATE) parameter is a run-time parameter.

The parameter classification results 426 and the job log data store 428 are provided to the logical pset construction step 434. In the example of Figure 6, the job log data store 428 includes four job log entries that include information associated with the execution of an instance (i.e., gather.mp) of the first subgraph 202. Each job log entry includes an execution command that receives values for the DATE and FEED parameters as arguments.

The logical pset construction step 434 creates a different logical pset 436 for each job log entry in the job log data store 428. Because the P1 (FEED) parameter is a design-time parameter, the value of the P1 parameter (e.g., mexico, usa, canada, or hong kong) that was provided as an argument to the execute command is included in each of the logical psets 436. Because the P2 (DATE) parameter is a run-time parameter, the value of the P2 parameter that was provided as an argument to the execute command is not included in the logical psets 436. Each logical pset 436 includes the project scope of its corresponding instance of the first subgraph 202.

7 , the logical psets 436 are provided to a pset signature string calculation step 442 , which calculates a different logical pset signature string 444 for each logical pset 436 .

The logical pset signature string 444 and a set of logical pset signature strings for existing psets 447 in the repository of existing psets 448 are provided to a search step 446. As shown in the scenario of FIG2 , there are three existing psets associated with the first subgraph 202: one for the mexicoFEED parameter, one for the usaFEED parameter, and one for the canadaFEED parameter. Thus, the set of logical pset signature strings 444 for existing psets 447 includes a string for each existing pset associated with the first subgraph 202.

Search step 446 searches for the presence of each logical pset signature string 444 in the set of logical pset signature strings of an existing pset 447. In this example, the result generated by search step 446 is that the only logical pset signatures not included in the set of logical pset signature strings of an existing pset 447 are the logical pset signature strings associated with the logical pset having the FEED parameter value 'hong kong'.

The result of the search step 446 and the logical pset 436 including the ‘hong kong’ feed parameters are provided to the logical pset addition step 450, which adds the logical pset including the ‘hong kong’ feed parameters and its corresponding logical pset signature string 444 to the repository 448 of existing logical psets.

By adding the new logical pset to the repository, the 'hong kong' instance of the first subgraph 202 that was previously ignored in the data lineage result is represented in the data lineage result.

Note that while in the above examples the static analysis values of the runtime parameters are described as being stored in the dataflow graph itself, in some examples the static analysis values of the runtime parameters may be maintained in one or more psets associated with the dataflow graph.

In some examples, certain design-time parameter values are derived from sources that are not necessarily present at static analysis time (e.g., from a database). However, in some examples, the job log entries stored in the job log data store include the values of all parameters resolved for that particular job. At static analysis time, the stored parameter values can be used in place of parameter values derived from sources that are not present at static analysis time.

In some examples, job log entries in the job log data store include all parsed parameters of the data flow graph, logs of all files read and written by the data flow graph, and performance tracking information. In some examples, job log entries in the job log data store are augmented with any logical parameter sets discovered by the method of FIG. 4 . In some examples, augmenting the job log entries in the job log data store with the discovered logical parameter sets includes forming an association between the job log entries and the discovered logical parameter sets. The augmented job log entries in the job log data store can be utilized to provide various forms of information to data architects or business users. In some examples, the augmented job log entries can be analyzed to ensure that logically connected data flow graphs are also physically connected. In some examples, the augmented job log entries can be analyzed to determine which physical data sets correspond to which logical data set instances. In some examples, the augmented job log entries can be analyzed to identify data sets with the same physical file name but associated with different static analysis parameters. In such examples, inconsistencies can be presented to a user for manual remediation or can be automatically remediated. In some examples, a data lineage report can include an indication of the inconsistency and whether it has been automatically remediated.

In some examples, the augmented job log entries can be used by the data lineage module to filter the data lineage report by frequency and/or recency. For example, the metadata environment module can maintain several data flow diagrams and psets that are no longer executed by the execution module. Such data flow diagrams and psets can be left in place in case they are needed later. However, unexecuted data flow diagrams and psets may cause unnecessary clutter in the data lineage report. To reduce the clutter, the augmented job log entries can be analyzed to determine which data flow diagrams and/or psets are used infrequently and/or have not been used recently. Based on this frequency and recency information, infrequently and non-recently executed data flow diagrams and psets (e.g., data flow diagrams that have not been run in the past year) can be filtered out of the data lineage report before being presented to the enterprise user.

In some examples, a logical pset may exist for a given dataflow graph (e.g., a pset including FEED=USA), but one or more jobs that invoke the dataflow graph do so by providing parameter values directly to the dataflow graph rather than utilizing an existing pset. In this case, the association maintained between the jobs and the logical psets accessed by the jobs (e.g., via a signature associated with the job) can be used to group job log entries based on their associated logical psets. Based on the grouping, any job that is instantiated by directly invoking the graph rather than utilizing an existing pset can be identified as being associated with the logical pset and its parameters.

In some examples, each job log entry for a dataflow graph includes, among other information, a list of parameter values for all parsings used to execute the dataflow graph associated with the job log entry. Once multiple job log entries have been accumulated, the parameter values for the parsings included in the job log entries can be compared to identify various "design time instances" of the dataflow graph. For example, some parsed parameters in the job log entry may be represented by only a few values across all the job log entries, while some other parsed parameters may be represented by many different values across all the job log entries. Those parsed parameters that are represented by only a few values in the job log entries may be "design time" parameters, and other parsed parameters that are represented by many different values in the job log entries may be "run time parameters". Any instances of dataflow graphs that share a unique combination of "design time parameters" are grouped together and are considered to all be "design time instances" of the dataflow graph. The data lineage module includes the different design time instances of the dataflow graph in the data lineage report.

5 Duplicate Logical Dataset Discovery and Mitigation Methods

5.1 Overview

Typically, the input and output data sets (eg, databases or data tables) of a given data flow graph are specified as logical data sets in the data flow graph. In some examples, each logical data set is associated with an identifier such as a logical file name.

Before executing a dataflow graph, it is prepared for execution, including parsing each logical data set into a corresponding physical data set (e.g., a file on disk). In some examples, each physical data set is associated with an identifier such as a physical file name (e.g., "summary.dat"). The parameter parsing process can successfully parse a logical data set into its corresponding physical data set even if the logical file name of the logical data set is different from the physical file name of the corresponding physical data set.

When determining a data lineage report for a data flow graph that includes two or more subgraphs, the lineage relationships between the subgraphs are determined based, at least in part, on the logical file names of the input and output logical data sets of the two or more subgraphs. To this end, the correctness of the lineage relationships requires that any input and output logical data sets of two or more subgraphs that reference a given physical data set share the same logical file name. In practice, if a first subgraph writes to a given physical data set and a second subgraph subsequently reads from the given physical data set, but the logical file name of the output logical data set of the first subgraph does not match the logical file name of the input logical data set of the second subgraph, then it will be recognized that there is no lineage relationship between the two subgraphs. In some examples, two logical data sets that resolve to the same physical data set but have mismatching logical file names are referred to as "duplicate logical data sets."

As described in detail below, duplicate logical data sets in a data flow graph can be identified and presented to the user. The user can then choose to address the duplicate logical data sets in a variety of ways.

5.2 Example of a logical dataset without duplication

8 , an example of a dataflow graph 817 generated using the development environment 118 of FIG. 1 includes a first subgraph 802 named gather.mp and a second subgraph 804 named process.mp.

The first subgraph 802 receives as input a first logical dataset _DL1 806 having a logical file name of "Acct_1.dat" and a second logical dataset _DL2 808 having a logical file name of "Acct_2.dat." The first subgraph 802 processes data from the first logical dataset 806 and the second logical dataset 808 and writes the processed results to a third logical dataset _DL3 810 having a logical file name of "Acct_summ.dat." The second subgraph 804 receives as input a third logical dataset _DL3 810 having a logical file name of "Acct_summ.dat," processes the data from the third logical dataset 810, and writes the processed results to a table 814. Note that the third logical dataset 810 used by both the first subgraph 802 and the second subgraph 804 has the same logical file name in both subgraphs 802 and 804.

9 , when the data flow graph 817 is parsed before execution, the logical data sets are parsed into their corresponding physical data sets. For example, the first logical data set 806 is parsed into a first physical data set _DP1 814 having a physical file name of "Acct_1.dat," the second logical data set 808 is parsed into a second physical data set DP2 816 having a physical file name of " _{Acct_2.dat} ," and the third logical data set 810 is parsed into a third physical data set _DP3 818 having a physical file name of "summary.dat."

10 , a data lineage report 1017 for a data flow graph includes the first subgraph 1002, the second subgraph 1004, the first logical dataset 1006, the second logical dataset 1008, and the third logical dataset 1010. The data lineage report 1017 also includes a first lineage relationship 1018 between the first logical dataset 1006 and an input of the first subgraph 1002, a second lineage relationship 1020 between the second logical dataset 1008 and an input of the first subgraph 1002, a third lineage relationship 1022 between the output of the first subgraph 1002 and the third logical dataset 1010, and a fourth lineage relationship 1024 between the third logical dataset 1010 and the second subgraph 1004. Note that in this case, the data lineage report 1017 is correct because the same logical dataset (i.e., the third logical dataset _DL3 810) with the same logical file name (i.e., "Acct_summ.dat") exists at the output of the first subgraph 802 and at the input of the second subgraph 804.

5.3 Example of a logical dataset with duplication

11 , which is another example of a data flow graph 1117 generated using the development environment 118 of FIG. 1 , includes a first subgraph 1102 named gather.mp and a second subgraph 1104 named process.mp.

The first subgraph 1102 receives as input a first logical dataset _DL1 1106 having a logical file name "Acct_1.dat" and a second logical dataset _DL2 1108 having a logical file name "Acct_2.dat." The first subgraph 1102 processes the data from the first logical dataset 1106 and the second logical dataset DL2 1108 and writes the processed results to a third logical dataset _DL3 1110 having a logical file name "Acct_summ.dat." The second subgraph 1104 receives as input a fourth logical dataset _DL4 1111 having a logical file name "Acct-summ.dat," processes the data from the fourth logical dataset 1111, and writes the processed results to the table 814. Note that the logical file name of the third logical dataset 1110 (i.e., "Acct_summ.dat") is different from the logical file name of the fourth logical dataset 1111 (i.e., "Acct-summ.dat").

12 , when the data flow graph 1117 is parsed before execution, the logical data sets are parsed into their corresponding physical data sets. For example, the first logical data set 1106 is parsed into a first physical data set _DP1 1114 having a physical file name "Acct_1.dat," the second logical data set 1108 is parsed into a second physical data set _DP2 1116 having a physical file name "Acct_2.dat," and the third logical data set 1110 and the fourth logical data set 1111 are both parsed into a third physical data set _DP3 1218 having a physical file name "summary.dat." Note that the third logical data set 1110 and the fourth logical data set 1111 are duplicate logical data sets because they each point to the same physical data set (i.e., the third physical data set 1218).

13 , a data lineage report 1317 for a data flow graph includes the first subgraph 1102, the second subgraph 1104, the first logical dataset 1106, the second logical dataset 1108, the third logical dataset 1110, and the fourth logical dataset 1111. The data lineage report 1317 also includes a first lineage relationship 1318 between the first logical dataset 1106 and an input of the first subgraph 1102, a second lineage relationship 1320 between the second logical dataset 1108 and an input of the first subgraph 1102, a third lineage relationship 1322 between an output of the first subgraph 1102 and the third logical dataset 1110, and a fourth lineage relationship 1324 between the fourth logical dataset 1111 and the second subgraph 1104.

Note that in this case, data lineage report 1317 is incorrect because two different logical datasets with different logical file names (i.e., third logical dataset 1110 and fourth logical dataset 1111) refer to the same physical dataset (i.e., third physical dataset 1218). Specifically, third logical dataset _DL3 1110, with the logical file name "Acct_summ.dat," exists at the output of first subgraph 1102, while fourth logical dataset 1111, with the logical file name "Acct_summ.dat," exists at the input of second subgraph 1104. Data lineage report 1317 represents third logical dataset 1110 and fourth logical dataset 1111 as separate datasets without any lineage relationship to each other. Therefore, data lineage report 1317 incorrectly includes a break in the data lineage between third logical dataset 1110 and fourth logical dataset 1111.

5.4 Discovery of Duplicate Logical Datasets

In some examples, repeated logical data sets in a data flow graph can be discovered by analyzing runtime artifacts generated by the execution of the data flow graph (e.g., job log 429 of FIG. 5 ). In particular, a job log is generated each time a data flow graph is executed.

The job log includes information associated with the execution of a data flow graph, including the graph instance name and, for each dataset component in the graph, the physical dataset it accesses and the type of access (read or write). The graph instance can be examined to determine the logical dataset name for each dataset component. By matching the graph instance and dataset component names, the system is able to map logical dataset names to physical dataset names.

To identify duplicate logical data sets, the job log is analyzed to identify any logical-to-physical data set mappings where a first logical data set that is mapped is different from a second logical data set that is mapped. Any logical-to-physical data set mapping where the first logical data set and the second logical data set are different is classified as a duplicate logical data set.

The identified duplicate logical data sets are presented to the user, who decides whether to correct the duplicate logical data sets or automatically mitigate them.

5.4.1 Example of duplicate logical dataset discovery

Referring again to FIG12 , when the parsed dataflow graph 1117 is executed, a job log for the dataflow graph execution is generated. The job log includes a single logical-to-physical dataset mapping corresponding to the flow between the first subgraph 1102 and the second subgraph 1104. The logical-to-physical dataset mapping includes an identifier _DL3 1110 of the third logical dataset at the output of the first subgraph 1104, an identifier _DL4 1111 of the fourth logical dataset at the input of the second subgraph 1106, and an identifier of the third physical dataset 1218.

Since the third logical dataset 1110 and the fourth logical dataset 1111 are different logical datasets (e.g., logical datasets with different logical file names) pointing to the same physical dataset (i.e., the third physical dataset 1218), the third logical dataset 1110 and the fourth logical dataset 1111 are classified as duplicate logical datasets.

Note that although the simple example above includes identification of a single pair of duplicate logical data sets from a single job log, in an actual implementation of a data processing system including the above duplicate logical data set discovery method, several pairs of duplicate logical data sets may be identified using multiple job logs.

5.5 Duplicated Logical Dataset Mitigation

As described above, duplicate logical data sets can cause disruptions in data lineage reporting. Once duplicate logical data sets are identified, a number of different approaches can be employed to eliminate the duplicate logical data sets or mitigate their impact on data lineage reporting. In some examples, the identified duplicate logical data sets are presented to the user, for example, in a spreadsheet format. The user can then edit a data flow diagram that includes the duplicate logical data sets to eliminate the duplicate logical data sets (e.g., by ensuring that in a given data flow diagram, a given physical data set is referenced only by a single logical data set). In other examples, the user can mark a pair of duplicate logical data sets as equivalent. In this way, the user does not need to make any changes to the data flow diagram. In other examples, duplicate logical data set pairs can be automatically marked as equivalent.

When a pair of duplicate logical datasets are marked as equivalent, there are several ways to display the equivalence in a data lineage report. In one approach, the physical datasets referenced by the duplicate logical datasets are shown as connected to the duplicate logical datasets in the data lineage report. For example, as shown in FIG14 , the third physical dataset _DP3 1218 is included in data lineage report 1317. The third logical dataset _DL3 1110 and the fourth logical dataset _DL4 1111 are both shown as connected to the third physical dataset 1218 via lineage relationships 1450 and 1452.

In another approach, the logical data sets of a pair of duplicate logical data sets are shown as connected to each other in the data lineage report via a lineage relationship. For example, in FIG15 , the third logical data set _DL3 1110 is shown as connected to the fourth logical data set _DL4 1111 via a lineage relationship 1550 in the data lineage report 1317.

In another approach, the pair of duplicate logical data sets is represented by a combined logical data set in a data lineage report. For example, as shown in FIG16 , the pair of duplicate logical data sets is represented by a combined logical data set D _LR 1654 in a data lineage report 1317 .

In another approach, one of the pair of duplicate logical datasets is selected to represent the pair of duplicate logical datasets in the data lineage report. For example, as shown in FIG17 , the fourth logical dataset _DL4 1111 represents a pair of duplicate logical datasets in the data lineage report 1317 .

In another approach, the pair of duplicate logical datasets and the combined logical dataset representation of the pair of duplicate logical datasets are included in the data lineage report. A unique configuration of the lineage relationship between the pair of duplicate logical datasets and the combined logical dataset representation is shown in the data lineage diagram. For example, as shown in FIG18 , data lineage report 1317 includes a combined logical dataset representation of a pair of duplicate logical datasets D _LR, 1854, a third logical dataset D _L3 1110, and a fourth logical dataset D _L4 1111. Combined logical dataset 1854 is shown as having a direct lineage relationship with first subgraph 1102 and second subgraph 1104. Combined logical dataset 1845 is also shown as having an indirect lineage relationship with first subgraph 1102 via third logical dataset 1110, and an indirect lineage relationship with second subgraph 1104 via fourth logical dataset 1111.

In another approach, a pair of duplicate logical datasets are included in a data lineage report. A unique configuration of the lineage relationship between the logical datasets of the pair of duplicate logical datasets is shown in the data lineage diagram. For example, as shown in FIG19 , data lineage report 1317 includes a third logical dataset _DL3 1110 and a fourth logical dataset _DL4 1111. The fourth logical dataset 1111 is shown as having a direct lineage relationship with the first subgraph 1102 and the second subgraph 1104. The third logical dataset _DL3 1110 is shown as having a direct lineage relationship with the first subgraph 1102 and an indirect lineage relationship with the second subgraph 1104 via the fourth logical dataset 1111.

Note that in some examples, the mitigation methods described above are shown in the data lineage report in dashed lines, bold lines, or in another alternative manner so that it is clear to the user of the data lineage report that the mitigation methods have been applied to the data lineage report.

Note that while the above method for discovering and mitigating duplicate logical datasets is described using a scenario where one component writes to a physical dataset and another component reads from that physical dataset, other scenarios can also result in duplicate logical datasets. For example, a pair of duplicate logical datasets can be generated by two different logical datasets reading from the same physical dataset. Similarly, a pair of duplicate logical datasets can be generated by two different logical datasets writing to the same physical dataset.

The above methods may incorporate features from various other methods for managing and presenting data lineage information and for managing dataset objects, as described in more detail in U.S. application serial number 12/393,765 filed on February 26, 2009, U.S. application serial number 13/281,039 filed on October 25, 2011, and U.S. provisional application serial number 62/028,485 filed on July 24, 2014, all of which are incorporated herein by reference.

The above methods can be implemented, for example, using a programmable computing system that executes appropriate software instructions, or can be implemented in suitable hardware such as a field programmable gate array (FPGA) or some hybrid form. For example, in a programming method, the software can include processes in one or more computer programs executed on one or more programming or programmable computing systems (which can be various architectures such as distributed, client/server or grid), each computing system including at least one processor, at least one data storage system (including volatile and/or non-volatile memory and/or storage elements), at least one user interface (for receiving input using at least one input device or port, and for providing output using at least one output device or port). The software can include, for example, one or more modules of a larger program that provides services related to the design, configuration and execution of data flow graphs. The modules of the program (e.g., elements of a data flow graph) can be implemented as data structures or other organized data that conform to a data model stored in a data warehouse.

The software can be provided on a tangible, non-transitory medium such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computing system or device), or can be delivered (e.g., encoded in a propagated signal) to a computing system where it is executed via a communication medium of a network. Some or all of the processing can be performed on a dedicated computer or using dedicated hardware such as a coprocessor or field programmable gate array (FPGA) or a specific application-specific integrated circuit (ASIC). The processing can be implemented in a distributed manner, where different parts of the calculations specified by the software are performed by different computing elements. Each such computer program is preferably stored or downloaded onto a computer-readable storage medium (e.g., solid-state memory or medium, or magnetic or optical medium) of a storage device accessible by a general-purpose or special-purpose programmable computer, for configuring and operating the computer when the computer reads the storage device medium to perform the processing described herein. The system of the present invention can also be considered to be embodied as a tangible, non-transitory medium configured with a computer program, wherein the medium so configured causes the computer to operate in a specific and predefined manner to perform one or more processing steps.

Several embodiments of the present invention have been described. However, it should be understood that the foregoing description is intended to illustrate, not to limit, the scope of the present invention, which is defined by the scope of the appended claims. Therefore, other embodiments are also within the scope of the appended claims. For example, various modifications may be made without departing from the scope of the present invention. In addition, some of the steps described above may be order-independent and, therefore, may be performed in an order different from that described.

Claims (31)

1. A method for managing a set of parameter values, the method comprising: It receives a set of multiple parameter values used in general-purpose computer programs, and Processing log entries associated with the execution of instances of the general-purpose computer program, each instance of the general-purpose computer program being associated with one or more parameter values, the processing includes: Analyze the general computer program to classify each of one or more parameters associated with the general computer program as a member of a first type of parameter or a second type of parameter; Processing log entries associated with the execution of the first instance of the general-purpose computer program to form a specific set of parameter values, wherein the processing includes: The specific set includes any value of the parameter that appears in the log entry and is classified as a member of the first type of parameter, and Exclude any values of parameters belonging to members classified as the second type of parameter from the specific set that appear in the log entries; and Whether to add the specific set of parameter values to the plurality of parameter value sets is determined by comparing a first identifier of the specific set of parameter values with the identifiers of at least some of the plurality of parameter value sets. The comparison between the first identifier of the specific set of parameter values and the identifiers of at least some of the plurality of parameter value sets includes: The first identifier is determined based on the specific set of parameter values and the identifier of the general computer program; Determine a plurality of second identifiers, each second identifier corresponding to a parameter value set of at least some of the plurality of parameter value sets; and The first identifier is compared with each of the plurality of second identifiers to determine whether the first identifier matches any of the plurality of second identifiers, wherein determining whether to add the specific set of parameter values to the plurality of parameter value sets includes: if no second identifier matches the first identifier, then determining to add the specific set of parameter values to the plurality of parameter value sets. 2. The method of claim 1, wherein processing the log entries includes classifying the parameters based on whether they affect data continuity associated with the general computer program. 3. The method of claim 1, wherein determining the first identifier comprises: calculating an identifier string from the contents of the specific set of parameter values, and determining the plurality of second identifiers comprises: calculating a plurality of identifier strings from the contents of at least some of the sets of parameter values in the plurality of sets of parameter values. 4. The method of claim 1, wherein determining the first identifier comprises a concatenation of one or more of the following: the identifier of the general-purpose computer program, name-value pairs of the particular set of parameter values, function prototypes of the general-purpose computer program, and the item scope of a first instance of the general-purpose computer program. 5. The method of claim 1, wherein determining the first identifier comprises applying a data mapping function to one or more of: the identifier of the general-purpose computer program, name-value pairs of the particular set of parameter values, a function prototype of the general-purpose computer program, and the item scope of a first instance of the general-purpose computer program. 6. The method of claim 5, wherein the data mapping function comprises a hash function. 7. The method of claim 1, wherein the first type of parameters includes parameters that affect the logical operation of the general-purpose computer program, and the second type of parameters includes parameters that do not affect the logical operation of the general-purpose computer program. 8. The method of claim 1, wherein the general computer program is designated as a data flow graph, the data flow graph comprising nodes representing data processing operations and links between the nodes representing data element flows between the data processing operations. 9. The method of claim 1, wherein for each of the plurality of parameters, the analysis comprises: automatically classifying the parameter or accepting a user-defined classification of the parameter. 10. The method of claim 9, wherein automatically classifying the parameter comprises: initially classifying the parameter as belonging to the first type of parameter; determining the number of unique values of the parameter in multiple executions of an instance of the general computer program; and if the number of unique values of the parameter exceeds a predetermined threshold, reclassifying the parameter as belonging to the second type of parameter. 11. The method of claim 9, wherein automatically classifying the parameter comprises: initially classifying the parameter as belonging to the first type of parameter; determining whether changes in the value of the parameter during multiple executions of an instance of the general computer program affect data continuity associated with the general computer program; and if changes in the value of the parameter do not affect the data continuity, reclassifying the parameter as belonging to the second type of parameter. 12. The method of claim 1, further comprising: forming an association between log entries associated with the execution of the first instance of the general-purpose computer program and the specific set of parameter values. 13. The method of claim 1, wherein the log entry associated with the execution of the first instance of the general-purpose computer program comprises: a log entry for an execution command for instantiating the general-purpose computer program, the log entry including one or more parameter values provided as arguments to the execution command. 14. The method of claim 13, wherein the log entries associated with the execution of the first instance of the general-purpose computer program further include one or more of the following: indications of items executed therein by the first instance, indications of internal parameters of the first instance, and indications of environment settings, global variables, and configuration variables used by the first instance. 15. The method of claim 9, further comprising: processing all plurality of parameter value sets for a plurality of general-purpose computer programs and all plurality of log entries associated with the execution of instances of at least some of the plurality of general-purpose computer programs to form a data lineage report, wherein the plurality of parameter value sets include: an extended plurality of parameter value sets of the general-purpose computer programs, and the plurality of log entries associated with the execution of instances of at least some of the plurality of general-purpose computer programs include: log entries for the execution of a first instance of the general-purpose computer program, including its association with the particular set of parameter value sets. 16. The method of claim 15, wherein forming a data lineage report comprises: for each of the plurality of parameter value sets in all the plurality of general-purpose computer programs, Process all log entries associated with the execution of at least some instances of the plurality of general-purpose computer programs to identify all log entries associated with the execution of instances of general-purpose computer programs corresponding to the set of parameter values, and identify the most recent instantiation time of the general-purpose computer program from the identified log entries associated with the execution of instances of the general-purpose computer program; and Whether to include the set of parameter values in the data lineage report is determined based on the most recent instantiation time of the general computer program. 17. The method of claim 16, wherein determining whether to include the set of parameter values in the data lineage report based on the most recent instantiation time of the general computer program comprises: comparing the most recent instantiation time with a predetermined time interval, and if the most recent instantiation time of the general computer program is within the predetermined time interval, then including the set of parameter values in the data lineage report. 18. The method of claim 15, wherein forming the data lineage report comprises: for each of the plurality of parameter value sets in all plurality of parameter value sets of the plurality of general-purpose computer programs, Process all log entries associated with the execution of instances of at least some of the plurality of general-purpose computer programs to determine the number of log entries associated with the execution of instances of general-purpose computer programs corresponding to the set of parameter values, and Whether to include the set of parameter values in the data lineage report is determined based on the number of log entries associated with the execution of an instance of the general-purpose computer program. 19. A computer-readable medium comprising instructions for managing a set of parameter values, the instructions being configured to cause a computing system to: It receives a set of multiple parameter values used in general-purpose computer programs, and Processing log entries associated with the execution of instances of the general-purpose computer program, each instance of the general-purpose computer program being associated with one or more parameter values, and expanding the plurality of parameter value sets based on the processing, the processing including: Analyze the general computer program to classify each of one or more parameters associated with the general computer program as a member of a first type of parameter or a second type of parameter; Processing log entries associated with the execution of the first instance of the general-purpose computer program to form a specific set of parameter values, wherein the processing includes: The specific set includes any value of the parameter that appears in the log entry and is classified as a member of the first type of parameter, and Exclude any values of parameters belonging to members classified as the second type of parameter from the specific set that appear in the log entries; and Whether to add the specific set of parameter values to the plurality of parameter value sets is determined by comparing a first identifier of the specific set of parameter values with the identifiers of at least some of the plurality of parameter value sets. The comparison between the first identifier of the specific set of parameter values and the identifiers of at least some of the plurality of parameter value sets includes: The first identifier is determined based on the specific set of parameter values and the identifier of the general computer program; Determine a plurality of second identifiers, each second identifier corresponding to a parameter value set of at least some of the plurality of parameter value sets; and The first identifier is compared with each of the plurality of second identifiers to determine whether the first identifier matches any one of the plurality of second identifiers. Determining whether to add the specific set of parameter values to the plurality of parameter value sets includes: if no second identifier matches the first identifier, then determining to add the specific set of parameter values to the plurality of parameter value sets. 20. A computing system for managing a set of parameter values, the computing system comprising: An input device or port for receiving a set of multiple parameter values for a general-purpose computer program, and At least one processor is configured to process log entries associated with the execution of instances of the general-purpose computer program, each instance of the general-purpose computer program being associated with one or more parameter values, the processing comprising: Analyze the general computer program to classify each of one or more parameters associated with the general computer program as a member of a first type of parameter or a second type of parameter; Processing log entries associated with the execution of the first instance of the general-purpose computer program to form a specific set of parameter values, wherein the processing includes: The specific set includes any value of the parameter that appears in the log entry and is classified as a member of the first type of parameter, and Exclude any values of parameters belonging to members classified as the second type of parameter from the specific set that appear in the log entries; and Whether to add the specific set of parameter values to the plurality of parameter value sets is determined by comparing a first identifier of the specific set of parameter values with the identifiers of at least some of the plurality of parameter value sets. The comparison between the first identifier of the specific set of parameter values and the identifiers of at least some of the plurality of parameter value sets includes: The first identifier is determined based on the specific set of parameter values and the identifier of the general computer program; Determine a plurality of second identifiers, each second identifier corresponding to a parameter value set of at least some of the plurality of parameter value sets; and The first identifier is compared with each of the plurality of second identifiers to determine whether the first identifier matches any one of the plurality of second identifiers. Determining whether to add the specific set of parameter values to the plurality of parameter value sets includes: if no second identifier matches the first identifier, then determining to add the specific set of parameter values to the plurality of parameter value sets. 21. A computing system for managing a set of parameter values, the computing system comprising: A means for receiving a set of multiple parameter values for a general-purpose computer program, and A means for processing log entries associated with the execution of instances of the general-purpose computer program, each instance of the general-purpose computer program being associated with one or more parameter values, the processing comprising: Analyze the general computer program to classify each of one or more parameters associated with the general computer program as a member of a first type of parameter or a second type of parameter; Processing log entries associated with the execution of the first instance of the general-purpose computer program to form a specific set of parameter values, wherein the processing includes: The specific set includes any value of the parameter that appears in the log entry and is classified as a member of the first type of parameter, and Exclude any values of parameters belonging to members classified as the second type of parameter from the specific set that appear in the log entries; and Whether to add the specific set of parameter values to the plurality of parameter value sets is determined by comparing a first identifier of the specific set of parameter values with the identifiers of at least some of the plurality of parameter value sets. The comparison between the first identifier of the specific set of parameter values and the identifiers of at least some of the plurality of parameter value sets includes: The first identifier is determined based on the specific set of parameter values and the identifier of the general computer program; Determine a plurality of second identifiers, each second identifier corresponding to a parameter value set of at least some of the plurality of parameter value sets; and The first identifier is compared with each of the plurality of second identifiers to determine whether the first identifier matches any one of the plurality of second identifiers. Determining whether to add the specific set of parameter values to the plurality of parameter value sets includes: if no second identifier matches the first identifier, then determining to add the specific set of parameter values to the plurality of parameter value sets. 22. A method for managing a set of parameter values, the method comprising: Receive general-purpose computer programs; Receive the set of first parameter values; An executable instance of the general-purpose computer program is generated by instantiating the general-purpose computer program according to the first set of parameter values. Receive data from one or more datasets; An executable instance of the general-purpose computer program is executed to process at least some of the received data; Log entries for generating executable instances of the general-purpose computer program are generated, the log entries including at least some parameter values from the first set of parameter values; Store the log entries; Receive the log entry; Processing the log entries to form a specific set of parameter values, wherein the processing includes: extracting at least some parameter values from the log entries of the first set of parameter values, and forming the specific set of parameter values from the extracted parameter values; and Whether to add the specific set of parameter values to the plurality of pre-existing parameter value sets is determined by comparing a first identifier of the specific set of parameter values with the identifiers of at least some of the pre-existing parameter value sets. The comparison between the identifier of the specific set of parameter values and the identifiers of at least some of the multiple pre-existing sets of parameter values includes: The first identifier is determined based on the specific set of parameter values and the identifier of the general computer program; Determine a plurality of second identifiers, each second identifier corresponding to one of the at least some pre-existing sets of parameter values; and The first identifier is compared with each of the plurality of second identifiers to determine whether the first identifier matches any one of the plurality of second identifiers. Determining whether to add the specific set of parameter values to the plurality of pre-existing sets of parameter values includes: if no second identifier matches the first identifier, then determining to add the specific set of parameter values to the plurality of pre-existing sets of parameter values. 23. The method of claim 22, wherein determining the first identifier comprises: calculating an identifier string from the contents of the particular set of parameter values, and determining the plurality of second identifiers comprises: calculating a plurality of identifier strings from the contents of at least some of the plurality of pre-existing sets of parameter values. 24. The method of claim 22, wherein determining the first identifier comprises a concatenation of one or more of the following: the identifier of the general-purpose computer program, name-value pairs of the particular set of parameter values, a function prototype of the general-purpose computer program, and the project scope of an executable instance of the general-purpose computer program. 25. The method of claim 22, wherein determining the first identifier comprises applying a data mapping function to one or more of: the identifier of the general-purpose computer program, name-value pairs of the particular set of parameter values, a function prototype of the general-purpose computer program, and the project scope of an executable instance of the general-purpose computer program. 26. The method of claim 25, wherein the data mapping function comprises a hash function. 27. The method of claim 22, further comprising: analyzing the general-purpose computer program to classify each of one or more parameters associated with the general-purpose computer program as a member of a first class of parameters or a second class of parameters. 28. The method of claim 27, wherein processing the log entries to form a specific set of parameter values further comprises: The specific set includes any extracted parameter values that appear in the log entries and are classified as members of the first type of parameter, as well as Exclude any extracted parameter values from the specific set that appear in the log entries and are classified as members of the second type of parameter. 29. The method of claim 27, wherein the first type of parameters includes parameters that affect the logical operation of the general-purpose computer program, and the second type of parameters includes parameters that do not affect the logical operation of the general-purpose computer program. 30. A computer-readable medium comprising instructions for managing a set of parameter values, the instructions being configured to cause a computing system to: Receive general-purpose computer programs; Receive the set of first parameter values; An executable instance of the general-purpose computer program is generated by instantiating the general-purpose computer program according to the first set of parameter values. Receive data from one or more datasets; An executable instance of the general-purpose computer program is executed to process at least some of the received data; Log entries for generating executable instances of the general-purpose computer program are generated, the log entries including at least some parameter values from the first set of parameter values; Store the log entries; Receive the log entry; Processing the log entries to form a specific set of parameter values, wherein the processing includes: extracting at least some parameter values from the log entries of the first set of parameter values, and forming the specific set of parameter values from the extracted parameter values; and Whether to add the specific set of parameter values to the plurality of pre-existing parameter value sets is determined by comparing a first identifier of the specific set of parameter values with the identifiers of at least some of the pre-existing parameter value sets. The comparison between the identifier of the specific set of parameter values and the identifiers of at least some of the multiple pre-existing sets of parameter values includes: The first identifier is determined based on the specific set of parameter values and the identifier of the general computer program; Determine a plurality of second identifiers, each second identifier corresponding to one of the at least some pre-existing sets of parameter values; and The first identifier is compared with each of the plurality of second identifiers to determine whether the first identifier matches any one of the plurality of second identifiers. Determining whether to add the specific set of parameter values to the plurality of pre-existing sets of parameter values includes: if no second identifier matches the first identifier, then determining to add the specific set of parameter values to the plurality of pre-existing sets of parameter values. 31. A system for managing a set of parameter values, the system comprising: The first computing system includes: A first input device or port is used to receive a general-purpose computer program, a first set of parameter values, and data from one or more datasets; The first group of one or more processors is configured as follows: An executable instance of the general-purpose computer program is generated by instantiating the general-purpose computer program according to the first set of parameter values. An executable instance of the general-purpose computer program is executed to process at least some of the received data; Log entries for generating executable instances of the general-purpose computer program, the log entries including at least some parameter values from the first set of parameter values; a first output device or port for storing the log entries in a storage device; The second computing system includes: A second input device or port is used to receive the log entries; a second group of one or more processors are configured to: Processing the log entries to form a specific set of parameter values, wherein the processing includes: extracting at least some parameter values from the log entries of the first set of parameter values, and forming the specific set of parameter values from the extracted parameter values; and Whether to add the specific set of parameter values to the plurality of pre-existing parameter value sets is determined by comparing a first identifier of the specific set of parameter values with the identifiers of at least some of the pre-existing parameter value sets. The comparison between the identifier of the specific set of parameter values and the identifiers of at least some of the multiple pre-existing sets of parameter values includes: The first identifier is determined based on the specific set of parameter values and the identifier of the general computer program; Determine a plurality of second identifiers, each second identifier corresponding to one of the at least some pre-existing sets of parameter values; and The first identifier is compared with each of the plurality of second identifiers to determine whether the first identifier matches any one of the plurality of second identifiers. Determining whether to add the specific set of parameter values to the plurality of pre-existing sets of parameter values includes: if no second identifier matches the first identifier, then determining to add the specific set of parameter values to the plurality of pre-existing sets of parameter values.