EP3217334A1

EP3217334A1 - System and method for visual bayesian data fusion

Info

Publication number: EP3217334A1
Application number: EP17160100.8A
Authority: EP
Inventors: Geetika Sharma; Karamjit Singh; Garima GUPTA; Gautam Shroff; Puneet Agarwal; Aditeya Pandey; Kaushal Ashokbhai Paneri; Gunjan Sehgal
Original assignee: Tata Consultancy Services Ltd
Current assignee: Tata Consultancy Services Ltd
Priority date: 2016-03-10
Filing date: 2017-03-09
Publication date: 2017-09-13
Anticipated expiration: 2037-03-09
Also published as: JP6431945B2; BR102017004835A2; MX390124B; AU2017201653B2; JP2017199358A; CA2960505C; CA2960505A1; MX2017003188A; US10430417B2; AU2017201653A1; US20170262506A1; EP3217334B1

Abstract

System and method for visual Bayesian data fusion are disclosed. In an example, a plurality of datasets associated with a topic are obtained from a data lake. Each of the plurality of datasets include information corresponding to various attributes of the topic. Further, the plurality of datasets are joined to obtain a joined dataset. Furthermore, distribution associated with a target attribute is predicted using Bayesian modeling by selecting a plurality of attributes (k) based on mutual information with the target attribute in the joined dataset, learning a minimum spanning tree based Bayesian structure using the selected attributes and the target attribute, learning conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure; and predicting the distribution associated with the target attribute by querying the conditional probabilistic tables, thereby facilitating visual Bayesian data fusion.

Description

CROSS REFERENCE TO RELATED APPLICATIONS AND PRIORITY

The present invention claims priority to Indian Provisional specification (Title: System and method for visual Bayesian data fusion) No. 201621008419 , filed in India on March 10, 2016.

TECHNICAL FIELD

The embodiments herein generally relate to data fusion and, more particularly, to system and method for visual Bayesian data fusion.

BACKGROUND

The evolution from traditional business intelligence to big data analytics has witnessed the emergence of data lakes in which data is ingested in raw form rather than into traditional data warehouses. With the increasing availability of many more pieces of information or data about each entity of interest, e.g., a customer, often from diverse sources (social media, mobility, internet-of-things), fusing, visualizing and deriving insights from such data may pose a number of challenges. The challenges are due to disparate datasets which often lack a natural join key. Also, the datasets may describe measures at different levels of granularity, e.g., individual versus aggregate data, different datasets may be derived from physically distinct populations. Moreover, once data has been fused, queries are often an inefficient and inaccurate mechanism to derive insight from high-dimensional data.

SUMMARY

The following presents a simplified summary of some embodiments of the disclosure in order to provide a basic understanding of the embodiments. This summary is not an extensive overview of the embodiments. It is not intended to identify key/critical elements of the embodiments or to delineate the scope of the embodiments. Its sole purpose is to present some embodiments in a simplified form as a prelude to the more detailed description that is presented below.
In view of the foregoing, an embodiment herein provides methods and systems for visual Bayesian data fusion are disclosed. In one aspect, a processor-implemented method includes steps of: obtaining a plurality of datasets associated with a topic from a data lake, wherein each of the plurality of datasets comprise information corresponding to various attributes of the topic; joining the plurality of datasets to obtain a joined dataset; predicting distribution associated with a target attribute using Bayesian modeling by selecting a plurality of attributes (k) based on mutual information with the target attribute in the joined dataset; learning a minimum spanning tree based Bayesian structure using the selected attributes and the target attribute; learning conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure; and predicting the distribution associated with the target attribute by querying the conditional probabilistic tables, thereby facilitating visual Bayesian data fusion.
In another aspect, a system for visual Bayesian data fusion is disclosed. The system includes one or more memories; and one or more hardware processors, the one or more memories coupled to the one or more hardware processors, wherein the one or more hardware processors are capable of executing programmed instructions stored in the one or more memories to: obtain a plurality of datasets associated with a topic from a data lake, wherein each of the plurality of datasets comprise information corresponding to various attributes of the topic; join the plurality of datasets to obtain a joined dataset; predict distribution associated with a target attribute using Bayesian modeling by select a plurality of attributes (k) based on mutual information with the target attribute in the joined dataset; learn a minimum spanning tree based Bayesian structure using the selected attributes and the target attribute; learn conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure; and predict the distribution associated with the target attribute by querying the conditional probabilistic tables, thereby facilitating visual Bayesian data fusion.
In yet another aspect, a non-transitory computer-readable medium having embodied thereon a computer program for executing a method for visual Bayesian data fusion is disclosed. The method includes steps of: obtaining a plurality of datasets associated with a topic from a data lake, wherein each of the plurality of datasets comprise information corresponding to various attributes of the topic; joining the plurality of datasets to obtain a joined dataset; predicting distribution associated with a target attribute using Bayesian modeling by selecting a plurality of attributes (k) based on mutual information with the target attribute in the joined dataset; learning a minimum spanning tree based Bayesian structure using the selected attributes and the target attribute; learning conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure; and predicting the distribution associated with the target attribute by querying the conditional probabilistic tables, thereby facilitating visual Bayesian data fusion.
It should be appreciated by those skilled in the art that any block diagram herein represents conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it is appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computing device or processor, whether or not such computing device or processor is explicitly shown.

BRIEF DESCRIPTION OF THE FIGURES

The detailed description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the drawings to reference like features and modules.

FIG. 1 illustrates a system for visual Bayesian data fusion, according to an embodiment of a present subject matter;
FIG. 2 illustrates a minimum spanning tree network learned on a feature subset, according to an embodiment of a present subject matter;
FIGS. 3A and 3B illustrate salary imputations for a dataset, according to an embodiment of a present subject matter; and
FIG. 4 is a flowchart illustrating a method for visual Bayesian data fusion, according to an embodiment of a present subject matter.

It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative systems and devices embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.

DETAILED DESCRIPTION

The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.
The present technique describes about visual Bayesian data fusion. In this technique, attributes which are most relevant to predict a target attribute are learnt. Next, an efficiently executable Bayesian network is learnt on this feature subset (via an minimum spanning tree embedded in a graph derived from pair-wise mutual information values). The technique then uses exact inference to predict the target attribute.
The methods and systems are not limited to the specific embodiments described herein. In addition, the method and system can be practiced independently and separately from other modules and methods described herein. Each device element/module and method can be used in combination with other elements/modules and other methods.
The manner, in which the system and method for cheque image data masking, has been explained in details with respect to the FIGS. 1 through 4. While aspects of described methods and systems for visual Bayesian data fusion can be implemented in any number of different systems, utility environments, and/or configurations, the embodiments are described in the context of the following exemplary system(s).
FIG. 1 illustrates a block diagram of a system 100 for visual Bayesian data fusion, in accordance with an example embodiment. In an example embodiment, the system 100 may be embodied in, or is in direct communication with a computing device. The system 100 includes or is otherwise in communication with one or more hardware processors such as processor(s) 102, one or more memories such as a memory 104, and a network interface unit such as a network interface unit 106. In an embodiment, the processor 102, memory 104, and the network interface unit 106 may be coupled by a system bus such as a system bus or a similar mechanism. Although FIG. 1 shows example components of the system 100, in other implementations, the system 100 may contain fewer components, additional components, different components, or differently arranged components than depicted in FIG. 1.
The processor 102 may include circuitry implementing, among others, audio and logic functions associated with the communication, and imaging, displaying, decoding and rendering functions. For example, the processor 102 may include, but are not limited to, one or more digital signal processors (DSPs), one or more microprocessor, one or more special-purpose computer chips, one or more field-programmable gate arrays (FPGAs), one or more application-specific integrated circuits (ASICs), one or more computer(s), various analog to digital converters, digital to analog converters, and/or other support circuits. The processor 102 thus may also include the functionality to encode messages and/or data or information. The processor 102 may include, among other things, a clock, an arithmetic logic unit (ALU) and logic gates configured to support operation of the processor 102. Further, the processor 102 may include functionality to execute one or more software programs, which may be stored in the memory 104 or otherwise accessible to the processor 102.
The functions of the various elements shown in the figure, including any functional blocks labeled as "processor(s)", may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term "processor" should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation DSP hardware, network processor, application specific integrated circuit (ASIC), FPGA, read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional, and/or custom, may also be included.
The interface(s) 106 may include a variety of software and hardware interfaces, for example, interfaces for peripheral device(s), such as a keyboard, a mouse, an external memory, and a printer. The interface(s) 106 can facilitate multiple communications within a wide variety of networks and protocol types, including wired networks, for example, local area network (LAN), cable, etc., and wireless networks, such as Wireless LAN (WLAN), cellular, or satellite.
The one or more memories such as a memory 104, may store any number of pieces of information, and data, used by the system to implement the functions of the system. The memory 104 may include for example, volatile memory and/or non-volatile memory. Examples of volatile memory may include, but are not limited to volatile random access memory. The non-volatile memory may additionally or alternatively comprise an electrically erasable programmable read only memory (EEPROM), flash memory, hard drive, or the like. Some examples of the volatile memory includes, but are not limited to, random access memory, dynamic random access memory, static random access memory, and the like. Some example of the non-volatile memory includes, but are not limited to, hard disks, magnetic tapes, optical disks, programmable read only memory, erasable programmable read only memory, electrically erasable programmable read only memory, flash memory, and the like. The memory 104 may be configured to store information, data, applications, instructions or the like for enabling the system 100 to carry out various functions in accordance with various example embodiments. Additionally or alternatively, the memory 104 may be configured to store instructions which when executed by the processor 102 causes the system to behave in a manner as described in various embodiments. The memory 104 includes a data fusion module 108 and/or other modules. The module 108 include routines, programs, objects, components, data structures, etc., which perform particular tasks or implement particular abstract data types. The other modules may include programs or coded instructions that supplement applications and functions of the system 100.
In operation, the data fusion module 108 obtains a plurality of datasets associated with a topic from a data lake. For example, each of the plurality of datasets include information (i.e., text or retailer description) corresponding to various attributes of the topic. Further, the data fusion module 108 joins the plurality of datasets to obtain a joined dataset. In an example, the plurality of datasets are joined based on a type of join. The concept of data join is to merge relevant attributes from different datasets and create a candidate file for learning correlations among these datasets. In this example, the type of join includes an inner join, an outer join, a left join, and a right join. In the inner join, common data from the datasets is kept. In the outer join, all the data from the datasets is kept. In the left join, common data from both datasets and data from a first dataset is kept. In the right join, common data from both datasets and data from a second dataset is kept.
Furthermore, the data fusion module 108 provides keyword-based search over the datasets. When datasets are added, the data fusion module 108 automatically generates tags based on column headers and used for indexing files. In an example, users may be enabled to add tags and enter the tags as search keywords for retrieval later. In addition, the data fusion module 108 performs exploratory data analysis to get a better understanding data before Bayesian modelling can be done. The data fusion module 108 provides many visualizations for exploratory analysis as described below.

Motion Charts: A multi-dimensional visualization which can visually represent up to 4 temporal data attributes using position, size and color of circles. Motion animation of the circles is used to depict changes in data over time.
Parallel Coordinates: A multi-dimensional data visualization which allows a larger number of data attributes to be visualized together. Attributes are represented as multiple parallel vertical or horizontal axes and a data point is represented as a polyline connecting points on each attribute axis (last two axes visualize error between actual and predicted salary). The order of axes may be changed by dragging and attributes can be deleted or added to the plot.
Bubble Map Charts: Plot bubbles or circles at geographical locations on a map with data attributes mapped to properties of the bubble such as size and color.
Cartograms: Use maps to visualize data about regions such as countries and states. Color and rubber sheet distortions in area proportional to data values allow easy comparison of spatial data. The visualization designs further includes data tiles display icons for visualizations associated with them. In case multiple visualizations can be drawn for a single dataset, an icon for each is displayed and clicking the icon opens the selected visualization. All visualizations open in the 'compare view' page. A list of thumbnails is displayed on the left side of the page using which users can re-order visualizations vertically, close or open the thumbnails in a full screen mode.

Furthermore in operation, the data fusion module 108 performs model learning on the datasets. In an example implementation, the data fusion module 108 predicts distribution associated with a target attribute using Bayesian modeling. For example, the data fusion module 108 enables users to choose multiple attributes from different datasets, and add them to an attribute-cart. After they select one variable as the target variable or attribute, they can request for automatic creation of the network. In this example, dependencies between attributes from different datasets may be discovered using Bayesian networks. A Bayesian network (BN) is a graphical structure that allows representation and reasoning about an uncertain domain. It is a representation of joint probability distribution (JPD) which consist of two components. The first component G is directed acyclic graph whose vertices correspond to random variables. The second component, the conditional probability table (CPT), describes the conditional distribution for each variable given its parent nodes. A CPT of a node indicates the probability that the each value of a node can take given all combinations of values of its parent-nodes. Considering a BN consisting of N random variables X = (X₁, X₂, ..., X_N), the general form of joint probability distribution of the BN can be represented as in Equation 1, which encodes the BN property that each node is independent of other nodes, given its parents, where Pa(X_i) is the set of parents of X_i. $P (X_{1}, X_{2}, \dots, X_{n}) = \prod_{i = 1}^{n} P (X_{i} | Pa (X_{i}))$
Further, the data fusion module 108 selects a plurality of attributes (k) based on mutual information with the target attribute in the joined dataset. The plurality of attributes and the target attribute may include discrete and continuous variables. In an embodiment, the data fusion module 108 selects top-K features from the dataset based on the mutual information of all features with target variable. Mutual information between continuous-continuous, and continuous-discrete variables may be calculated using non-parametric entropy estimation toolbox (NPEET). This tool implements an approach to find mutual information estimators, which is based on entropy estimates from k-nearest neighbor distances. For example, mutual information between x and y variables is calculated using the following equation: $I (X; Y) = \sum_{y \in Y} \sum_{x \in X} p (x, y) \log (\frac{p (x, y)}{p (x) p (y)}),$
Furthermore, the data fusion module 108 learns a minimum spanning tree (MST) based Bayesian structure (e.g., a MST 200 shown in FIG. 2) using the selected attributes and the target attribute (e.g., salary in FIG. 2). In an example embodiment, the data fusion module 108 learns the minimum spanning tree (MST) on the plurality of attributes and the target attribute using pairwise mutual information as a threshold. For example, the data fusion module 108 learns a minimum spanning tree on feature graph, created by calculating pairwise mutual information between various features (i.e., variables or attributes), and dropping the edges that have this mutual information smaller than the threshold. In this example, the minimum spanning tree ensures learning sparse structures which helps in better estimation of parameters (especially in case of data having high cardinality variables) and faster inference.
In addition, the data fusion module 108 initializes each edge in the minimum spanning tree to random direction. Moreover, the data fusion module 108 flips each edge direction to compute 2^(k) directed graphs. As 'k' is a number of edges in the MST based structure, when the data fusion module 108 flips edge direction of each edge, 2^(k) different trees or graphs are computed. Further, the data fusion module 108 calculates the cross entropy of each graph. Also, the data fusion module 108 selects a graph with least cross entropy as the minimum spanning tree based Bayesian structure.
Furthermore, the data fusion module 108 learns conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure (i.e., parameters of each node). In an example, the data fusion module 108 discretizes the continuous variables by fixed size binning. The data fusion module 108 learns the conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure upon discretizing the continuous variables. Moreover, the data fusion module 108 predicts the distribution associated with the target attribute by querying the conditional probabilistic tables, thereby facilitating visual Bayesian data fusion. The data fusion module 108 uses SQL for querying the CPTs considering them as set of relational tables.
In an example embodiment, the data fusion module 108 uses the Bayesian networks for prediction. Often attributes may need to be predicted for new datasets in which they do not exist at all. The data fusion module 108 may use the saved Bayesian network to predict the target variable for new datasets in which it may be missing using imputation: the expected value of the target variable is computed from the posterior distribution obtained after Bayesian inference and used as the predicted.
In another example embodiment, the data fusion module 108 uses Bayesian networks to determine the missing data. Very often data values may be missing for certain attributes of constituent datasets in the data lake, reducing the utility of such datasets. The data fusion module 108 provides data completion feature using Bayesian imputation. Attributes that are part of a Bayesian network may be imputed in datasets where they are partially missing. For this, users are enabled to select a network based tile to be used for imputation as well as the dataset with missing values. The data fusion module 108 then generates a new dataset with complete data, included in the inverted-index, and made available for further visual analysis.
For example, consider a challenge to predict salary for a set of candidates for whom it was not provided. The data fusion module 108 uses the MST, learned on the relevant feature subset, to predict the salary of each candidate in the test dataset using rest of the features in a network as evidence. Further, the data fusion module 108 uses exact inference accelerated by an SQL engine which internally performs query optimization which is analogous to poly-tree based exact inference to predict the salary.
In some embodiments, the system 100 provides a visual interface for model-based prediction using parallel coordinates. The user is enabled to select a network to be used for prediction via imputation and a dataset with the target variable missing. In this embodiment, a horizontal parallel coordinates plot is used to differentiate it from the exploratory parallel co-ordinates visualization as well as to indicate a network structure which is usually drawn in a top-down order even though the edges have no directionality in this case. The value of the attribute to be imputed is 0 for all data points initially as shown in graph 300A (graph before imputation) of FIG. 3A. Clicking the 'Impute' button fires the data fusion module 108 at the backend and lines for the imputed values are moved to their position along the axis, as shown in graph 300B (graph after imputation) of FIG. 3B. A visualization of the predicted salary on test data created from of the training data in 70-30(%) ratio is shown in FIGs 3A and 3B. The error in prediction can be visualized in 300B on the last two axes - actual salary and imputed salary.
In another example embodiment, the data fusion module 108 uses the MST based Bayesian structure for probabilistic join. For example, data from disparate sources may not have a common join key however their fusion may be warranted for analytical reasoning. This is achieved by predicting global-product id for every local-product-id using an ensemble of Bayesian prediction and textual similarity.
Consider, a data lake consisting of real-life data of consumer products from a global 'information and measurement' company. Available datasets in the data lake are as follows:

1) Local dataset: It contains set of items I = {Ir : r =1, 2, ..., m}. For each item Ir ∈ I, local attributes, retailer descriptions, as well as measures such as sales figures are available.
2) Global dataset: It contains the global market share of each item, where items are described by global attributes.

In this example, local attributes may be different from those used in the global dataset (i.e., these may include attributes not used at the global level). Even for similar attributes, local attributes use geography specific nomenclature, differing from global naming conventions. For example, in case of carbonated drinks, both global and local datasets may contains brand, flavour etc., of a drink, however, the actual values used may differ: e.g., 'Coke' vs 'Coca-cola', or 'Sweetened', vs 'Contains sugar'. Additionally, local attributes include 'retailer descriptions', which are fields with free form text describing the product. Such text often points clues towards the global attributes for the item.
Consider a scenario where country wise sales vs global market share of each brand needs to be compared. The datasets cannot be compared as there is no natural join key between these two datasets. However, if value of global attribute g ∀ I_r∈ I can be predicted using local attributes (Bayesian model) and retailer description, the datasets can be compared. Thus, the data fusion module 108 make probabilistic prediction of g for items in local dataset. Further, the data fusion module 108 join the local dataset with the global dataset using g as a join key and call it as probabilistic join. Let g be the global attribute with n possible states say g_i, where i = 1, 2, ..., n. In order to predict value of g ∀I_r ∈I, the data fusion module 108 uses two different models, a) Bayesian Model and b) Text Information Retrieval Model which uses local attributes and retailer description, respectively. The data fusion module 108 also calculate confidence of prediction in both models. Further, the data fusion module 108 do ensemble of these two models to get the better prediction accuracy.
Bayesian Model (BM): In BM, the data fusion module 108 use only local attributes to predict global attributes for each item. For a global attribute g, the data fusion module 108 selects top K local attributes based on mutual information with g. Further, the data fusion module 108 learns minimum spanning tree based structure with global attribute g and selected top K local attributes. Furthermore, the data fusion module 108 calculates probability {p_i : i= 1, ..., n} of each state gi of global attribute g, with local attributes as evidence in the MSTN. Finally, for every item, the data fusion module 108 choose the state gi which have maximum probability p_i.
Confidence in BM Model (ConfBM): For each item, the data fusion module 108 also calculates the confidence of prediction of BM model. Given a probability distribution {p_i: i =1 ,2,..., n} for a global attribute g. Consider an ideal distribution $q_{i} = {\begin{matrix} 1 & p_{i} = \max \{p_{i} : i = 1, 2, \dots, n\} \\ 0 & otherwise \end{matrix}$
and the confidence is given by: ${Conf}_{BM} = 1 - \sqrt{\sum_{i = 1}^{n} {(p_{i} - q_{i})}^{2}}$
For each item I_r, the data fusion module 108 predicted two values (say g_B and g_T) of g using local attributes (along with its confidence c_B) and retailer description, respectively. For an item I_r, the data fusion module 108 chooses g_B as the prediction of g if c_B > t, else chooses g_T as a prediction of g, where t is the threshold learned on validation set. Once global attribute of every item in a set I of local dataset is predicted, the data fusion module 108 joins local dataset and global dataset after group by using g_i as a key in local datasets.
In an example embodiment, the data fusion module 108 uses Bayesian networks to make inferences and for 'what-if' analysis. The data fusion module 108 provides a 'Linked Query View' for the same. A 'Linked View' in data visualization parlance refers to a view containing multiple charts such that interaction with any chart synchronously updates all. The data fusion module 108 uses a linked view to visualize conditional queries on attributes in the network. Once a network has been saved, it can be used to perform visual model inferencing using a 'Linked Query View'. The data fusion module 108 enables user to selects n attributes from the network to query and these are visualized in an n x n chart grid with attributes repeated horizontally and vertically. Charts along the diagonal, show the probability distributions of the corresponding attribute as bar charts. In the cells above the diagonal, scatter plots of the data with row and column attributes on the x and y axis of the plot, respectively, are shown. These provide a view of the data used to build the network and can be used to analyze pair-wise correlations between attributes.
In order to query the network, users can select ranges for multiple attributes by clicking on appropriate bars in the bar charts. This puts a condition on the attribute to be in the range selected by the user. On hitting the query button, the data fusion module 108 executes a conditional query on the network using Bayesian inference. The conditional distributions of the other attributes are computed and the bar charts are updated accordingly. For example, a 'Bayesian Student Advisor' is designed to provide recommendations using the Linked Query View. The original probability distributions are shown on the diagonal using bar charts. Salary is plotted on log10 scale. Consider the case when a candidate is interested in getting a very high salary and wants to know the ideal profile for the same. It may be observed that there is a significant rise in the probability of English score, indicating that a candidate must have high English score to get a high salary. Additionally, it is found that although the distributions of logical test score and CGPA do not change much, probability of 12th percentage increases significantly for the higher range bins. Thus, for a very high salary English score and 12th percentage must be high. In this manner, a candidate may impose conditions on any number of the variables in the network and get answers to how his/her profile should change in order to meet the salary goal.
FIG. 4 is a flowchart illustrating a method for visual Bayesian data fusion, according to an embodiment of a present subject matter. The processor-implemented method 400 may be described in the general context of computer executable instructions. Generally, computer executable instructions can include routines, programs, objects, components, data structures, procedures, modules, functions, etc., that perform particular functions or implement particular abstract data types. The method 400 may also be practiced in a distributed computing environment where functions are performed by remote processing devices that are linked through a communication network. The order in which the method 400 is described is not intended to be construed as a limitation, and any number of the described method blocks can be combined in any order to implement the method 400, or an alternative method. Furthermore, the method 400 can be implemented in any suitable hardware, software, firmware, or combination thereof. In an embodiment, the method 400 depicted in the flow chart may be executed by a system, for example, the system 100 of FIG. 1.
At block 402, a plurality of datasets associated with a topic are obtained from a data lake. For example, each of the plurality of datasets include information (i.e., text or retailer description) corresponding to various attributes of the topic. At block 404, the plurality of datasets are joined to obtain a joined dataset. In an example, the plurality of datasets are joined based on a type of join. In this example, the type of join includes an inner join, an outer join, a left join, and a right join. At block 406, distribution associated with a target attribute is predicted using Bayesian modeling. In an embodiment, a plurality of attributes (k) is selected based on mutual information with the target attribute in the joined dataset. The plurality of attributes and the target attribute may include discrete and continuous variables.
Further, a minimum spanning tree based Bayesian structure is learnt using the selected attributes and the target attribute. In an example embodiment, the minimum spanning tree is learnt on the plurality of attributes and the target attribute using pairwise mutual information as a threshold. Further in this embodiment, each edge in the minimum spanning tree is initialized to random direction. Furthermore, each edge direction is flipped to compute 2^(k) directed graphs. In addition, the cross entropy of each graph is calculated. Also, a graph with least cross entropy is selected as the minimum spanning tree based Bayesian structure.
Furthermore, conditional probabilistic tables are learnt at each node of the minimum spanning tree based Bayesian structure. In an example, the continuous variables are discretized by fixed size binning. The conditional probabilistic tables are then learnt at each node of the minimum spanning tree based Bayesian structure upon discretizing the continuous variables. Moreover, the distribution associated with the target attribute is predicted by querying the conditional probabilistic tables, thereby facilitating visual Bayesian data fusion.
In some embodiments, a confidence score for the prediction of the distribution associated with the target attribute is computed by querying the conditional probabilistic tables using ideal distribution and probabilistic distribution. Further, distribution associated with the target attribute is predicted using textual similarity. Furthermore, the distribution associated with the target attribute predicted using the textual similarity or the distribution associated with the target attribute predicted by querying the conditional probabilistic tables is selected based on the computed confidence score.
The written description describes the subject matter herein to enable any person skilled in the art to make and use the embodiments. The scope of the subject matter embodiments is defined by the claims and may include other modifications that occur to those skilled in the art. Such other modifications are intended to be within the scope of the claims if they have similar elements that do not differ from the literal language of the claims or if they include equivalent elements with insubstantial differences from the literal language of the claims.
It is, however to be understood that the scope of the protection is extended to such a program and in addition to a computer-readable means having a message therein; such non-transitory computer-readable storage means contain program-code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof. The device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. Thus, the means can include both hardware means and software means. The method embodiments described herein could be implemented in hardware and software. The device may also include software means. Alternatively, the embodiments may be implemented on different hardware devices, e.g. using a plurality of CPUs.
The embodiments herein can comprise hardware and software elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc. The functions performed by various modules described herein may be implemented in other modules or combinations of other modules. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
The foregoing description of the specific implementations and embodiments will so fully reveal the general nature of the implementations and embodiments herein that others can, by applying current knowledge, readily modify and/or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the embodiments as described herein.
The preceding description has been presented with reference to various embodiments. Persons having ordinary skill in the art and technology to which this application pertains will appreciate that alterations and changes in the described structures and methods of operation can be practiced without meaningfully departing from the principle, spirit and scope.

Claims

A processor-implemented method comprising:
obtaining a plurality of datasets associated with a topic from a data lake, wherein each of the plurality of datasets comprise information corresponding to various attributes of the topic;

joining the plurality of datasets to obtain a joined dataset;

predicting distribution associated with a target attribute using Bayesian modeling by
selecting a plurality of attributes (k) based on mutual information with the target attribute in the joined dataset;

learning a minimum spanning tree based Bayesian structure using the selected attributes and the target attribute;

learning conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure; and

predicting the distribution associated with the target attribute by querying the conditional probabilistic tables, thereby facilitating visual Bayesian data fusion.
The method according to claim 1, wherein the plurality of datasets are joined based on a type of join and wherein the type of join comprises an inner join, an outer join, a left join, and a right join.
The method according to claim 1, wherein learning a minimum spanning tree based Bayesian structure using the selected attributes and the target attribute comprises:
learning the minimum spanning tree on the plurality of attributes and the target attribute using pairwise mutual information as a threshold;

initializing each edge in the minimum spanning tree to random direction;

fliping each edge direction to compute 2^(k) directed graphs;

calculating the cross entropy of each graph; and

selecting a graph with least cross entropy as the minimum spanning tree based Bayesian structure.
The method according to claim 1, wherein the plurality of attributes and the target attribute comprise discrete and continuous variables.
The method according to claim 4, wherein learning the conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure comprises:
discretizing the continuous variables by fixed size binning; and

learning the conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure upon discretizing the continuous variables.
The method according to claim 1, further comprising:
computing a confidence score for the distribution associated with the target attribute predicted by querying the conditional probabilistic tables using ideal distribution and probabilistic distribution;

predicting distribution associated with the target attribute using textual similarity; and

selecting one of a) the distribution associated with the target attribute predicted using the textual similarity and b) the distribution associated with the target attribute predicted by querying the conditional probabilistic tables based on the computed confidence score.
A system (100) comprising:
one or more memories (104); and

one or more hardware processors (102), the one or more memories (104) coupled to the one or more hardware processors (102), wherein the one or more hardware processors (102) are capable of executing programmed instructions stored in the one or more memories (104) to:
obtain a plurality of datasets associated with a topic from a data lake, wherein each of the plurality of datasets comprise information corresponding to various attributes of the topic;

join the plurality of datasets to obtain a joined dataset;

predict distribution associated with a target attribute using Bayesian modeling by
select a plurality of attributes (k) based on mutual information with the target attribute in the joined dataset;

learn a minimum spanning tree based Bayesian structure using the selected attributes and the target attribute;

learn conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure; and

predict the distribution associated with the target attribute by querying the conditional probabilistic tables, thereby facilitating visual Bayesian data fusion.
The system according to claim 7, wherein the plurality of datasets are joined based on a type of join and wherein the type of join comprises an inner join, an outer join, a left join, and a right join.
The system according to claim 7, wherein one or more hardware processors are capable of executing programmed instructions to:
learn the minimum spanning tree on the plurality of attributes and the target attribute using pairwise mutual information as a threshold;

initialize each edge in the minimum spanning tree to random direction;

flip each edge direction to compute 2^(k) directed graphs;

calculate the cross entropy of each graph; and

select a graph with least cross entropy as the minimum spanning tree based Bayesian structure.
The system according to claim 7, wherein the plurality of attributes and the target attribute comprise discrete and continuous variables.
The system according to claim 10, wherein one or more hardware processors are capable of executing programmed instructions to:
discretize the continuous variables by fixed size binning; and

learn the conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure upon discretizing the continuous variables.
The system according to claim 7, wherein one or more hardware processors are further capable of executing programmed instructions to:
compute a confidence score for the distribution associated with the target attribute predicted by querying the conditional probabilistic tables using ideal distribution and probabilistic distribution;

predict distribution associated with the target attribute using textual similarity; and

select one of a) the distribution associated with the target attribute predicted using the textual similarity and b) the distribution associated with the target attribute predicted by querying the conditional probabilistic tables based on the computed confidence score.
A non-transitory computer readable medium embodying a program executable in a computing device, said program comprising:
a program code for obtaining a plurality of datasets associated with a topic from a data lake, wherein each of the plurality of datasets comprise information corresponding to various attributes of the topic;

a program code for joining the plurality of datasets to obtain a joined dataset;

a program code for predicting distribution associated with a target attribute using Bayesian modeling by
selecting a plurality of attributes (k) based on mutual information with the target attribute in the joined dataset;

learning a minimum spanning tree based Bayesian structure using the selected attributes and the target attribute;

learning conditional probabilistic tables at each node of the minimum spanning tree based Bayesian structure; and

predicting the distribution associated with the target attribute by querying the conditional probabilistic tables, thereby facilitating visual Bayesian data fusion.