DE112021006196T5 - METHOD AND APPARATUS FOR VISUAL INFERENCE - Google Patents

Abstract

The present disclosure provides a method for visual reasoning. The method includes: providing a network having sets of inputs and sets of outputs, each set of inputs from the sets of inputs being mapped to one of a set of outputs corresponding to the set of inputs based on visual information about the set of inputs , and wherein the network includes a probabilistic generative model (PGM) and a set of modules; determining by the PGM a posterior distribution over combinations of one or more modules of the set of modules based on the sets of inputs and sets of outputs; and applying domain knowledge as one or more posterior regularization constraints to the particular posterior distribution.

Description

AREA

Aspects of the present disclosure relate generally to artificial intelligence, and more particularly to a method and network for visual reasoning.

BACKGROUND

Artificial intelligence (AI) is used in a variety of areas such as image classification, object recognition, scene understanding, machine translation and the like. There is increasing interest in visual reasoning with an increasing growth of applications such as visual question answering (VQA), embodied question answering, visual navigation, autopilot and the like, where AI models may generally be required to predict high-level cognitive processes over low-level perceptual outcomes level, for example, to perform high-level abstract reasoning about simple visual concepts such as lines, shapes, and the like.

Deep neural networks have been widely applied in the field of visual inference, where deep neural networks can be trained to model the correlation between task input and output and can succeed in various visual inference tasks with deep and rich representation learning, in particular in perceptual tasks. Additionally, in recent years, modularized networks have attracted more and more attention to visual inference, allowing deep learning and symbolic inference to be combined, with a focus on building neural-symbolic models, aiming to get the best of representation learning and to combine symbolic conclusions. The basic idea is to manually design neural modules, each representing a primitive step in the reasoning process, and solve reasoning problems by assembling these modules into respective symbolic networks corresponding to the solved reasoning problems.

This modularized network using neural-symbolic methodology can generally correctly solve a traditional visual question answering (VQA) problem, where the questions are generally in the form of texts. In addition to VQA, abstract visual inference is recently proposed to extract abstract concepts or questions directly from a visual input without natural language questioning, such as from an image, and perform inference processes accordingly. Since inference about abstract concepts has long been a challenge in the field of machine learning, current methods or AI models as described above may have unsatisfactory performance in such abstract visual inference.

It may be desirable to provide even better methods or AI models to process abstract visual reasoning tasks.

SHORT PRESENTATION

The following presents a simplified summary of one or more aspects in accordance with the present disclosure to provide a basic understanding of such aspects. This brief is not a comprehensive overview of all considerations and is not intended to identify key or critical elements of all considerations nor to delineate the scope of any or all considerations. Its sole purpose is to present in a simplified form some concepts of one or more points of view in anticipation of the more detailed description presented below.

In one aspect of the disclosure, a method for visual reasoning includes: providing a network having sets of inputs and sets of outputs, each set of inputs of the sets of inputs being based on one of a set of outputs corresponding to the set of inputs information about the set of inputs, and wherein the network comprises a probabilistic generative model (PGM) and a set of modules; determining by the PGM a posterior distribution over combinations of one or more modules of the set of modules based on the sets of inputs and sets of outputs; and applying domain knowledge sen as one or more posterior regularization constraints on the particular posterior distribution.

In another aspect of the disclosure, there is provided a method for visual reasoning with a network comprising a probabilistic generative model (PGM) and a set of modules, the method comprising: providing the network with a set of input images and a set of candidate images ; Generating a combination of one or more modules of the set of modules based on a posterior distribution over combinations of one or more modules of the set of modules and the set of input images, where the posterior distribution of the PGM trained under domain knowledge is one or more posterior regularization constraints are formulated; processing the set of input images and the set of candidate images through the generated combination of one or more modules; and selecting a candidate image from the set of candidate images based on a score of each candidate image in the set of candidate images estimated by the processing.

In another aspect of the disclosure, a visual inference network includes: a set of modules, each of the set of modules being implemented as a neural network and having at least one trainable parameter for focusing that module on one or more variable image properties; and a probabilistic generative model (PGM) coupled to the set of modules, the PGM configured to output a posterior distribution over combinations of one or more modules of the set of modules.

In another aspect of the disclosure, the visual reasoning device includes a memory; and at least one processor coupled to the memory. The at least one processor is configured to provide a network with sets of inputs and sets of outputs, each set of inputs from the sets of inputs being mapped to one of a set of outputs corresponding to the set of inputs based on visual information about the set of inputs, and wherein the network comprises a probabilistic generative model (PGM) and a set of modules; determining by the PGM a posterior distribution over combinations of one or more modules of the set of modules based on the sets of inputs and sets of outputs; and applying domain knowledge as one or more posterior regularization constraints to the particular posterior distribution.

In another aspect of the disclosure, a visual reasoning computer program product includes processor-executable computer code for providing a network having sets of inputs and sets of outputs, each set of inputs of the sets of inputs being responsive to one of a set of outputs corresponding to the set of inputs is mapped based on visual information about the set of inputs, and wherein the network comprises a probabilistic generative model (PGM) and a set of modules; determining by the PGM a posterior distribution over combinations of one or more modules of the set of modules based on the sets of inputs and sets of outputs; and applying domain knowledge as one or more posterior regularization constraints to the particular posterior distribution.

In another aspect of the disclosure, a computer-readable medium stores computer code for visual reasoning. The computer code, when executed by a processor, causes the processor to provide a network of sets of inputs and sets of outputs, each set of inputs of the sets of inputs being mapped to one of a set of outputs corresponding to the set of inputs based on visual information about the set of inputs, and wherein the network comprises a probabilistic generative model (PGM) and a set of modules; determining by the PGM a posterior distribution over combinations of one or more modules of the set of modules based on the sets of inputs and sets of outputs; and applying domain knowledge as one or more posterior regularization constraints to the particular posterior distribution.

With the support of domain knowledge, the generated modularized networks can provide structures that accurately represent a human-interpretable reasoning process, which can lead to improved performance.

Other aspects or variations of the disclosure, as well as other advantages, will become apparent upon consideration of the following detailed description and accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

• 1 zeigt ein Beispiel für abstrakte visuelle Schlussfolgerung.
• 2 veranschaulicht ein beispielhaftes Netzwerk, in dem Gesichtspunkte der vorliegenden Offenbarung durchgeführt werden können.
• 3A und 38 veranschaulichen beispielhafte modularisierte Netzwerke mit unterschiedlichen Strukturen.
• 4 zeigt ein beispielhaftes Flussdiagramm, das ein Verfahren zum Durchführen einer abstrakten visuellen Schlussfolgerungsaufgabe mit einem probabilistischen neuronal-symbolischen Modell veranschaulicht, das gemäß einem oder mehreren Gesichtspunkten der vorliegenden Offenbarung mit Domänenwissen regularisiert wird.
• 5 stellt ein beispielhaftes Flussdiagramm dar, das einen Optimierungsprozess für eine abstrakte visuelle Schlussfolgerungsaufgabe gemäß einem oder mehreren Gesichtspunkten der vorliegenden Offenbarung veranschaulicht.
• 6 zeigt ein beispielhaftes Flussdiagramm, das ein Verfahren zum Durchführen einer abstrakten visuellen Schlussfolgerungsaufgabe mit einem probabilistischen neuronal-symbolischen Modell veranschaulicht, das gemäß einem oder mehreren Gesichtspunkten der vorliegenden Offenbarung mit Domänenwissen regularisiert wird.
• 7 veranschaulicht ein weiteres beispielhaftes Netzwerk, in dem Gesichtspunkte der vorliegenden Offenbarung durchgeführt werden können.
• 8 stellt ein beispielhaftes Flussdiagramm dar, das einen Optimierungsprozess für eine abstrakte visuelle Schlussfolgerungsaufgabe gemäß einem oder mehreren Gesichtspunkten der vorliegenden Offenbarung veranschaulicht.
• 9 veranschaulicht ein Beispiel einer Hardware-Implementierung für eine Vorrichtung gemäß einer Ausführungsform der vorliegenden Offenbarung.

The aspects disclosed are described below in connection with the accompanying drawings, which are provided to illustrate and not limit the aspects disclosed.

• 1 shows an example of abstract visual inference.
• 2 illustrates an example network in which aspects of the present disclosure may be implemented.
• 3A and 38 illustrate exemplary modularized networks with different structures.
• 4 shows an example flowchart illustrating a method for performing an abstract visual reasoning task with a probabilistic neural-symbolic model regularized with domain knowledge in accordance with one or more aspects of the present disclosure.
• 5 depicts an example flowchart illustrating an optimization process for an abstract visual reasoning task in accordance with one or more aspects of the present disclosure.
• 6 shows an example flowchart illustrating a method for performing an abstract visual reasoning task with a probabilistic neural-symbolic model regularized with domain knowledge in accordance with one or more aspects of the present disclosure.
• 7 illustrates another example network in which aspects of the present disclosure may be implemented.
• 8th depicts an example flowchart illustrating an optimization process for an abstract visual reasoning task in accordance with one or more aspects of the present disclosure.
• 9 illustrates an example of a hardware implementation for a device according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure will now be discussed with reference to several example implementations. It is understood that these implementations are discussed only to enable those skilled in the art to better understand and thus implement the embodiments of the present disclosure, and not to suggest limitations on the scope of the present disclosure.

Compared to traditional computer vision tasks such as image classification and object detection, visual inference goes a step further and requires not only a comprehensive understanding of the visual content but also the ability to reason about the extracted concepts to make inferences. 1 shows an example of abstract visual reasoning in which the eight panels in the left dashed box represent a set of inputs and the six panels in the right dashed box represent a set of outputs. There may be one or more common rules between the set of inputs and the correct set of outputs. In order to select the correct one from among several candidate output fields, the common rules are extracted and mapped to the correct output field using these rules. In the example of 1 For example, the common rule for the eight input image fields can be an increasing number of shapes per line, and the correct output field D can be selected based on the rule. For example, extracting the rule of increasing number of shapes per row can be a high-level abstract inference task based on one or more low-level visual concepts, such as different shapes in each of the input image fields. The present disclosure proposes a method for performing an abstract visual reasoning task with a probabilistic neural-symbolic model regularized with domain knowledge. A neural-symbolic model can provide a powerful tool that combines symbolic program execution for reasoning and deep representation learning for visual recognition. For example, a neural-symbolic model may form a particular modularized network that includes, for each input, one or more modules, each selected from a set of modules like a stock of reusable modules. A probabilistic formulation for training models with stochastic latent variables can obtain an interpretable and readable inference system with fewer supervisions.

Domain knowledge can provide guidance in generating an appropriate modularized network, as it is generally an optimization problem with a mix of continuous and discrete variables. With the support of domain knowledge, the generated modularized networks can provide structures that accurately represent a human-interpretable reasoning process, which can lead to improved performance.

2 illustrates an example network 200 in which aspects of the present disclosure may be performed. For example, the network 200 may include a probabilistic generative model (PGM) 210 and a set of modules 220, such as a reusable module inventory. In one aspect of the present disclosure, a plurality of combinations of one or more modules from the set of modules 220 may be selected to solve respective sets of inputs, and the plurality of combinations of the set of modules 220 may be considered a latent variable , for which a posterior distribution can be formulated by the PGM 210 by learning a data set. For example, one or more modules may be selected from the inventory of reusable modules to compose a modularized network with a structure that specifies the assembled modules and the connections between them. For example, the structure of the assembled modularized network can be represented as a directed acyclic graph (DAG). The PGM 210 can be used to formulate a distribution across structures of modularized networks, where the set of modules 220 can be a collection of reusable modules for composing modularized networks. For example, the PGM 210 may formulate a posterior distribution over structures of modularized networks by learning a data set. The formulated posterior distribution over structures of modularized networks can be regularized with domain knowledge.

For example, the PGM 210 may include a variational autoencoder (VAE), where an encoder of a VAE can formulate a varying posterior distribution of structures of modularized networks, and a decoder of the VAE can formulate a generative distribution. The encoder's formulated varying posterior distribution of modularized network structures may be an estimated posterior distribution of modularized network structures based on the observed data set. The formulated generative distribution by the decoder can be used for reconstruction (as via Route 4 of 8th illustrated). In some aspects of the present disclosure, a decoder may be omitted from the PGM 210. In other aspects of the present disclosure, both an encoder and a decoder may be included in the PGM 210.

For example, the set of modules 220 may include one or more prebuilt neural modules, each representing a primitive step in a reasoning process. For example, each module of the set of modules 220 can be implemented as a multilayer neural network with one or more trainable parameters. In one aspect of the present disclosure, each module of the set of modules 220 may be dynamically interconnected to form a particular modularized network that may be used to map a given set of inputs to the correct output. In one aspect of the present disclosure, the PGM 210 can be used to create modularized networks with structures corresponding to the individual inputs to predict the respective fundamental rules within the individual inputs.

3A and 38 illustrate exemplary modularized networks with different structures. For example, the structure of the modularized network can be represented as a DAG, denoted by G = (v, A), where v GM ^d , v each node (i.e. each module) of the structure, M the set of modules 220, d the size of the structure and A ∈ {0,1} ^d×d represents the adjacency matrix, which can represent the connections between the modules of the structure. For example, the number of vertices of the graph may be specified to be less than or equal to a threshold (e.g., d ≤ 4 or 6 or the like), and each vertex may be filled with a particular module from the set of modules 220 . For example, the set of modules M 220 may include ten modules numbered 0 to 9, represented _{as v0} , _v1 , _v2 , _v3 , _v4 , v5, _v5 , _v7 , _v5 , _v9 can.

As an example, the in 3A The structure shown has the modules v ₁ , v ₂ , v ₃ , v ₄ , which were respectively filled into the vertices 310-1, 310-2, 310-4 and 310-3, as well as an adjacency matrix $$A=\left\{\begin{array}{cccc}0& 1& 1& 0\\ 0& 0& 0& 1\\ 0& 0& 0& 1\\ 0& 0& 0& 0\end{array}\right\}.$$

As another example, the in 3B The structure shown has the modules v ₁ , v ₂ , v ₃ , v ₄ , which were respectively filled into the vertices 310-1, 310-4, 310-3 and 310-2, as well as an adjacency matrix $$A=\left\{\begin{array}{cccc}0& 1& 0& 1\\ 0& 0& 1& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right\}.$$

In some aspects of the present disclosure, the modularized networks with the respective in 3A and 3B The structures shown may be suitable for extracting different rules contained in different sets of inputs. In one aspect of the present disclosure, the network 200 or 700 may learn associations between the sets of inputs and corresponding structures used therefor by training a data set including sets of inputs and sets of outputs associated with the respective sets of inputs to display the correct outputs. For example, a posterior distribution of modularized network structures may be learned by the PGM 210 and used to derive a modularized network structure for any set of inputs. In another aspect of the present disclosure, domain knowledge may be applied in creating structures. For example, domain knowledge may be applied to the posterior distribution of structures of modularized networks learned by the PGM 210 from the dataset as one or more posterior regularization constraints. Using domain knowledge, the regularized distribution of structures of modularized networks can be used to produce a precise and interpretable structure for a set of inputs, potentially representing hidden rules within the set of inputs.

One skilled in the art will understand that other structures and other representations for at least a portion of the set of modules 220 are also possible.

4 shows an example flowchart illustrating a method 400 for performing an abstract visual reasoning task with a probabilistic neural-symbolic model regularized with domain knowledge in accordance with one or more aspects of the present disclosure. For example, method 400 may be performed by network 200 and network 700, which are described in detail below. For example, the method 400 can also be carried out by other networks, systems or models.

In block 410, sets of inputs and sets of outputs may be provided to a network 200 or 700, where each set of inputs of the sets of inputs may be mapped to a set of outputs corresponding to the set of inputs based on visual information about the Set of inputs. The sets of inputs and the sets of outputs may include, for example, a training data set, such as the Procedurally Generated Matrice (PGM) data set or the relational and analog visual rEasoNing data set (RAVEN), or the like. The network 200, 700 may include a probabilistic generative model (PGM) 210, 710 and a set of modules 220, 720.

At block 420, a posterior distribution may be determined by the PGM 210, 710 based on the sets of inputs and sets of outputs provided with respect to the set of modules 220, 720. In one aspect of the present disclosure, a posterior distribution over combinations of one or more modules of the set of modules 220, 720 may be determined by the PGM 210, 710 based on the sets of inputs and sets of outputs provided. In one example, the combinations of one or more modules of the set of modules 220, 720 may include modularized networks consisting of one or more modules of the set of modules 220, 720 are composed, whereby the modularized networks can have structures that can be represented as G = (v, A). In another example, the combinations of one or more modules of the set of modules 220 may include any permutations of one or more modules of the set of modules 220. For example, the PGM 210 may include a VAE. An estimated posterior distribution over structures of modularized networks can be formulated by an encoder of the VAE based on the observed data set.

In block 430, the domain knowledge may be applied to the particular posterior distribution of the set of modules 220 as one or more posterior regularization constraints. For example, a regularized Bayesian framework (RegBayes) can be used to integrate human domain knowledge into Bayesian methods by directly applying constraints on the posterior distribution. The flexibility of RegBayes can enable the explicit consideration of domain knowledge by incorporating knowledge into arbitrary Bayesian models as soft constraints.

Using domain knowledge, method 400 can be used to generate precise and interpretable structures for different sets of inputs because the generated structures can capture hidden rules between sets of inputs.

One skilled in the art will understand that other probabilistic generative models are also possible and other distributions may be possible with respect to the set of modules 220.

In one aspect of the present disclosure, one or more posterior regularization constraints may include one or more first-order logic (FOL) constraints that may include domain knowledge. For example, a constraint function may consist of first-order logic calculations over each of the structures and each of the sets of inputs. Specifically, each constraint function takes each of the structures and each of the sets of inputs as input and calculates the designed first-order logic expression as output. The output of the constraint function may take a value in a range of [0, 1] indicating the degree to which the input of each of the structures and each of the sets of inputs conforms to a specific requirement, where a lower value may indicate a stronger match . Therefore, by minimizing values of such constraint functions while optimizing the posterior distribution of structures, the network 200 can learn to generate structures that can correspond to the applied domain knowledge.

In another aspect of the present disclosure, it may be advantageous to consider internal connections between constraints. Constraints that take into account different aspects of domain knowledge can be independent of each other. On the other hand, constraints applied to different nodes of a structure but sharing the same domain knowledge viewpoint can be correlated with each other. Accordingly, the constraints that share the same domain knowledge viewpoint can be grouped into a group of constraints. For example, a total of L groups of constraints may be proposed, each group corresponding to a particular type of inference, including Boolean logical inference, temporal inference, spatial inference, arithmetic inference, and the like.

$$

die aus den folgenden primitiven Sätzen gesammelt wird:

• • Beziehungstypen: (
  
  mit Elementen r): Progression, XOR, OR, AND, konsistente Vereinigung
• • Objekttypen: (
  
  mit Elementen o): Form, Linie
• • Attributtypen: (
  
  mit Elementen a): Größe, Typ, Farbe, Position, Nummer

In another aspect of the present disclosure, the one or more FOL constraints may be generated based on one or more characteristics of each set of inputs. For example, in a Procedurally Generated Matrices (PGM) data set, each pair of a set of inputs and the corresponding set of outputs may have one or more rules, where each rule may be represented as a triple, $T=\left\{\left[r,O,a\right]:r\in R,O\in \mathcal{O},a\in \mathcal{A}\right\},$

$$

which is collected from the following primitive sentences:

• • Relationship types: (
  
  with elements r): progression, XOR, OR, AND, consistent union
• • Object types: (
  
  with elements o): shape, line
• • Attribute types: (
  
  with elements a): size, type, color, position, number

das Tripel [Progression, Form, Farbe] enthält, kann der Satz von Eingaben und die entsprechende korrekte Ausgabe eine progressive Beziehung aufweisen, die sich auf die Farbe (z. B. die Graustufenintensität) von Formen bezieht. Beispielsweise kann jeder Attributtyp $a\in \mathcal{A}$

(z. B. Farbe) einen von einer endlichen Anzahl diskreter Werte z ∈ Z annehmen (z. B. 10 Ganzzahlen zwischen [0, 255] für die Graustufenintensität). Daher kann eine gegebene Regel

eine Vielzahl von Realisierungen abhängig von den Werten für die Attributtypen aufweisen, aber alle diese Realisierungen können derselben grundlegenden abstrakten Regel unterliegen. Auswahl von r kann die zu realisierenden Werte von z einschränken. Wenn beispielsweise r eine Progression ist, können die Werte von z entlang der Zeilen oder Spalten in der Matrix von Eingabebildfeldern zunehmen und nach dieser Regel mit unterschiedlichen Werten variieren.These triples can determine abstract inference rules by a given set of inputs and the corresponding correct output. For example, if

contains the triple [progression, shape, color], the set of inputs and the corresponding correct output may have a progressive relationship related to the color (e.g. grayscale intensity) of shapes. For example, any attribute type $a\in \mathcal{A}$

(e.g. color) take one of a finite number of discrete values z ∈ Z (e.g. 10 integers between [0, 255] for grayscale intensity). Therefore, a given rule can

have a variety of realizations depending on the values for the attribute types, but all of these realizations can be governed by the same basic abstract rule. Choosing r can limit the values of z that can be realized. For example, if r is a progression, the values of z can increase along the rows or columns in the matrix of input image patches and vary with different values according to this rule.

In one aspect of the present disclosure, the one or more FOL constraints may be generated based on at least one of relationship types, object types, or attribute types of the sets of inputs. For example, an exemplary formation of a FOL constraint may be given by: $${\Phi }_j\left(G,x\right):=1-1\left[v_j\in S\left(x\right)\right]$$

des Satzes von Eingaben x extrahiert werden können. Wobei der j-te Knoten in der Struktur G bezeichnet wird durch v_j.Where 1 [•] is the indicator function and v _j ∈ s(x) is true if the semantic representation of v _j is found in S(x). Where S(x) are semantic attributes of a set of inputs x composed of one or more triples $T\left\{\left[r,O,a\right]\right\}$

of the set of inputs x can be extracted. Where the jth node in the structure G is denoted by v _j .

des Satzes von Eingaben x, gemäß einem bestimmten Gesichtspunkt des Domänenwissens, wie logische Schlussfolgerung, zeitliche Schlussfolgerung, räumliche Schlussfolgerung oder arithmetische Schlussfolgerung und dergleichen. Beispielsweise kann die logische Schlussfolgerung logische UND, ODER, XOR oder dergleichen umfassen. Beispielsweise kann die arithmetische Schlussfolgerung arithmetische ADD, SUB, MUL und dergleichen umfassen. Beispielsweise kann die räumliche Schlussfolgerung STRUC (Struktur) umfassen, z. B. zum Ändern der Berechnungsregeln von Eingabemodulen und dergleichen. Beispielsweise kann die zeitliche Schlussfolgerung PROG (Fortschritt), ID (Identisch) und dergleichen umfassen.In one or more aspects of the present disclosure, a set of FOL constraints may be generated based on one or more triples $T\left\{\left[r,O,a\right]\right\}$

of the set of inputs x, according to a certain aspect of domain knowledge, such as logical reasoning, temporal reasoning, spatial reasoning or arithmetic reasoning and the like. For example, the logical conclusion may include logical AND, OR, XOR, or the like. For example, the arithmetic conclusion may include arithmetic ADD, SUB, MUL, and the like. For example, spatial inference may include STRUC (structure), e.g. B. for changing the calculation rules of input modules and the like. For example, the temporal conclusion may include PROG (Progress), ID (Identical), and the like.

In one or more aspects of the present disclosure, a set of FOL constraints generated according to a particular aspect of domain knowledge may be applied to each of the nodes of a structure. For example, constraints in the group can perform a FOL rule on all nodes of the tree, which can check a specific aspect of domain knowledge.

One skilled in the art will understand that one or more of the aspects described above may be performed by network 200, 700 or other networks, systems or models.

In one example, in the example flowchart of method 400, inference tasks may be performed by optimizing trainable parameters of PGM 210, 710 and modules of the set of modules 220, 720 to minimize prediction loss over observed samples, as formulated by the following objective : $${\text{min}}_{\phi }mi{n}_{\theta }{\mathcal{l}}_{err}\left(\phi ,\theta \right):={\displaystyle {\sum }_D{\displaystyle {\sum }_{G\sim q_{\phi }}\left[-lOG{p}_{net}\left(y_n|x_n,G,\theta \right)\right]}}$$

Where φ denotes trainable parameters in the PGM 210,710, ϑ denotes trainable parameters of modules of the set of modules 220,720 and D = {(x _n , y _n )} _n=1:N comprises a data set containing the nth input x _n , assigned to the output y _n , denoted.

In one aspect of the present disclosure, the network 200, 700 may utilize a PGM 210, 710 to represent a generative distribution p _φ (x|G) and a variational distribution q _φ (G|x). For example, an encoder of a VAE can represent the variation distribution q _φ (G|x), and a decoder of VAE can represent the generative distribution p _φ (x|G). In particular, by optimizing formulation (2), an estimated posterior distribution of the structures p̃ _{φ 0} (G|x) and the corresponding module parameter ϑ ₀ are obtained.

In another aspect of the present disclosure, one or more FOL regularization constraints may be applied to represent the new posterior distribution of the structures (l). Formally, the overall goal can be formulated as: $$\begin{array}{c}{\text{min}}_{\phi ,\xi ,\eta }mi{n}_{\theta }{\mathcal{l}}_{err}\left(\phi ,\theta \right)+{\mathrm{<figure-callout\; id="1"\; label="C"\; filenames="DE112021006196T5\_0029.png,DE112021006196T5\_0031.png"\; state="\{\{state\}\}">C</figure-callout>}}_1{\displaystyle \sum _{i=1}^L{\xi }_i+{\mathrm{<figure-callout\; id="2"\; label="C"\; filenames="DE112021006196T5\_0031.png,DE112021006196T5\_0036.png"\; state="\{\{state\}\}">C</figure-callout>}}_2\eta },\\ \begin{array}{cc}\text{s}\text{.t}\text{.}\forall i,& E_{x_n\in D}\left|E_{G\sim q_{\phi }}\left[{\displaystyle {\sum }_{j=1}^{T_i}{\Phi }_{ij}\left(G,x_n\right)}\right]\right|\le {\xi }_i+\epsilon ,\end{array}\\ KL\left[q_{\phi }\left(G|x\right)\Vert {\tilde{p}}_{{\phi }_0}\left(G|x\right)\right]\le \eta +\epsilon ,\\ \text{Where}{\mathrm{<figure-callout\; id="0"\; label="\phi "\; filenames="DE112021006196T5\_0036.png"\; state="\{\{state\}\}">\phi </figure-callout>}}_0=arGmi{n}_{\mathrm{\phi }}{l}_{err}\left(\phi ;\theta \right)\end{array}$$

Where q _φ (G|x) is the regularized posterior distribution of the structures, p̃ _{φ 0} (G|x) is the estimated posterior distribution of the structures, given by optimizing formulation (2), ξi=1:L ≥ 0 and η ≥ 0 are slack variables with corresponding regularization parameters C ₁ and C ₂ , and e is a small positive precision parameter.

The Φc _ij (G, x _n ) functions in formulation (3), whose values can be bounded by the slack variables, are FOL constraints. In an example, each constraint function may take a value in the range [0,1], where a smaller value may denote a better match between the structure G and the input x _n according to the domain knowledge. Note that constraint functions can form L groups instead of being independent of each other. The i-th group may include T _i correlated constraints, which may correspond to a common slack variable (i.

While the main goal of formulation (3) may be to minimize the task loss, ℓ _err , the slack variables ξi=1:L in the formulation can take into account the FOL constraints. The process of structure generation can be regularized with the applied domain knowledge. To achieve the minimum of the overall goal, the network 200, 700 can learn to produce structures that satisfy the applied FOL constraints. Furthermore, the KL divergence between q _φ (G|x) and p̃ _{φ 0} (G|x) can be considered an additional constraint that can prevent the network 200 or 700 from being overly responsive to the domain knowledge.

Additionally, one or more additional restrictions may be added and one or more of the example restrictions described above may be omitted.

5 illustrates an example flowchart depicting an optimization process 500 for formulation (3) according to one or more aspects of the present disclosure. For example, process 500 may be performed by network 200, network 700, described in detail below, or other networks, systems, models, or the like.

In block 510, parameters of the PGM 210, 710 and parameters of modules of the set of modules 220, 720 may alternatively be updated by maximizing evidence of the sets of inputs and the sets of outputs to obtain an estimated posterior distribution over the combinations of one or more multiple sets of modules of the set of modules 220, 720 and optimized parameters of the modules of the set of modules 220, 720.

In block 520, one or more weights of one or more posterior regularization constraints applied to the estimated posterior distribution over the combinations of one or more modules of the set of modules 220, 720 may be updated to provide one or more optimal solutions of the to obtain one or more weights.

In block 530, the estimated posterior distribution over the combinations of one or more modules of the set of modules 220, 720 may be determined by applying the one or more optimal solutions of the one or more weights and the one or more values of the one or more several constraints on the estimated posterior distribution.

In block 540, the optimized parameters of the modules of the set of modules 220, 720 may be updated based on the adjusted estimated posterior distribution across the combinations of one or more modules of the set of modules 220, 720 to fit the updated structural distribution.

In an example, assuming ϑ is fixed, the goal of the probabilistic generative model may be given by maximizing the evidence of the observed data samples, which can be written as: $$\begin{array}{c}{\text{min}}_{\phi }{\mathcal{l}}_{prOb}\left(\phi ,\theta \right):={\displaystyle \sum _n-lOGp\left(x_n,y_n\right)}\\ ={\displaystyle \sum _n-\left[lOGp\left(x_n\right)+lOGp\left(y_n|x_n\right)\right]}\\ \approx {\displaystyle \sum _n-E_{G\sim q_{\phi }}\left[lOG{p}_{\phi }\left(x_n|G\right)-\beta lOG{p}_{\phi }\left(G|x_n\right)+\beta lOGp\left(G\right)+\gamma lOG{p}_{net}\left(y_n|x_n,G,\theta \right)\right]},\end{array}$$

Where the scaling hyperparameter is the prediction probability and is a constant parameter that satisfies β > 1. Since ℓ _prob (φ, θ) for the expected value E _G ~ _{q φ} , may not be differentiable, the REINFORCE algorithm can be applied to obtain an estimated gradient for the updates. Updates of can be calculated directly using gradients.

Assuming that the PGM 210, 710 parameters have reached the optimum, optimizing the process via ϑ can become optimizing the execution performance of the network, which can be written as: $$mi{n}_{\theta }{\mathcal{l}}_{err}\left(\phi ,\theta \right)={\displaystyle {\sum }_D{\displaystyle {\sum }_{G\sim q_{\phi }}\left[-lOG{p}_{net}\left(y_n|x_n,G,\theta \right)\right]}}$$

The gradient ∇ _θ ℓ _err (φ, θ) can be estimated using stochastic gradient descent (SGD), where the structure G is captured during training.

Suppose that the results of the above optimization procedure with respect to formulation (2) are denoted by φ ₀ and θ ₀ , and the estimated posterior distribution of the structures can be denoted by p̃ _{φ 0} (G|x). be referred to. To obtain an approximate solution to formulation (3), φ0 can be considered fixed, and the target can be transformed into a RegBayes formation, which can be written as: $$\begin{array}{c}mi{n}_{\phi ,\xi ,\eta }KL\left[q_{\phi }\left(G|x\right)\Vert {\tilde{p}}_{{\phi }_0}\left(G|x\right)\right]+C{\displaystyle \sum _{i=1}^L{\xi }_i},\\ \text{s}\text{.t}\text{.}{E}_{x_n\in D}\left|E_{G\sim q_{\phi }}\left[{\displaystyle {\sum }_{j=1}^{T_i}{\Phi }_{ij}\left(G,x_n\right)}\right]\right|\le {\xi }_i+\epsilon ,\end{array}$$

In one aspect of the present disclosure, a dual problem introduced by convex analysis may be applied to solve formulation (6). Therefore, by introducing variables of the dual problem, µ, an optimal distribution of the RegBayes objective can be obtained by the following formulation: $$q_{\phi }\left(G|x;\mu *\right)=\frac{{\tilde{p}}_{{\phi }_0}\left(G|x\right)}{Z\left(\mu *\right)}exp\left({\displaystyle {\sum }_{i=1}^L\mu *|{\Phi }_{\left[i\right]}^{\left(D\right)}\left(G,x\right)}\right)$$

die gruppierte Summierung der FOL-Beschränkungen in der i-ten Gruppe ist, $${\Phi }_{\left[i\right]}^{\left(D\right)}\left(G,x\right):={\displaystyle {\sum }_{j=1}^{T_i}{\Phi }_{ij}^{\left(D\right)}}\left(G,x\right)$$

Where ${\Phi }_{\left[i\right]}^{\left(D\right)}\left(G,x\right)$

is the grouped summation of the FOL constraints in the i-th group, $${\Phi }_{\left[i\right]}^{\left(D\right)}\left(G,x\right):={\displaystyle {\sum }_{j=1}^{T_i}{\Phi }_{ij}^{\left(D\right)}}\left(G,x\right)$$

ein Erwartungswert über beobachtete Proben für die entsprechende Beschränkungsfunktion ist, $${\Phi }_{ij}^{\left(D\right)}\left(G,x\right):=E_{x_n\in D}\left[{\Phi }_{ij}\left(G,x_n\right)\right]$$

Whereby everyone ${\Phi }_{ij}^{\left(D\right)}\left(G,x\right)$

is an expected value over observed samples for the corresponding constraint function, $${\Phi }_{ij}^{\left(D\right)}\left(G,x\right):=E_{x_n\in D}\left[{\Phi }_{ij}\left(G,x_n\right)\right]$$

wobei C und E Hyperparameter in Formulierung (3) sind.Z (µ*) is the normalization factor for q _φ , where µ* is the optimal solution to the dual problem: $$\begin{array}{l}ma{x}_{\mu }L\left(\mu \right)=-lOGZ\left(\mu \right)-\epsilon {\displaystyle \sum _{i=1}^L{\mu }_i},\\ \text{s}.\text{t}.\left|{\mu }_i\right|\le C,\forall i=\mathrm{1.2,}\dots \mathrm{.,}L\end{array}$$

where C and E are hyperparameters in formulation (3).

The optimization of the dual problem (10) can be processed using an approximate stochastic gradient descent (SGD) method. In particular, the gradient can be approximated as: $${\partial }_{{\mu }_i}\text{log}Z\left(\mu \right)={\displaystyle {\sum }_G{q}_{\phi }}\left(G|x\right){\Phi }_{\left[i\right]}^{\left(D\right)}\left(G,x\right)\approx {\widehat{\Phi }}_{\left[i\right]}\left(G,x\right),\forall i=\mathrm{1.2,}\dots ,L$$

Where the first equation is due to duality and the approach is to estimate the expected value, Φ̂ _[i] (G,x), which can be given by uniformly sampling the observed samples and calculating the constraint function values. In particular, the updates µ _i can be given by the SGD rule: $${\mu }_i^{\left(t+1\right)}=PrO{j}_{\left[-C,C\right]}\left({\mu }_i^{\left(t\right)}+r_t\left(-{\partial }_{{\mu }_i}lOGZ\left(\mu \right)+\epsilon \right)\right)$$

Where Proj _[-C,C] denotes the Euclidean projection of the input onto [-C, C] and r _t is the step length. After solving µ*, the regularized posterior distribution of the structures q _φ (G|x) can be given by the formulation (7). The module parameters ϑ can be further optimized to fit the updated structure distribution.

In one example, the overall pipeline of the example optimization process 500 may be illustrated in Algorithm 1.

• ♦ Zufälliges Initialisieren von ϑ, φ und µ
• ♦ Bei Konvergenz mit
  1. 1) Satz ϑ ist fest, Gradient ∇ℓ_prob(, ϑ) wird berechnet, um φ gemäß Formulierung (4) zu aktualisieren;
  2. 2) Satz q ist fest, Gradient ∇_ϑ ℓ_err(, ϑ) wird berechnet, um ϑ gemäß Formulierung (5) zu aktualisieren;
• ♦ Ende
• ♦ kann φ₀ das Ergebnis des vorstehenden Verfahrens bezeichnen;
• ♦ Bei Konvergenz mit
  • 3) Aktualisieren von µ gemäß dem dualen Problem (10), wobei die Aktualisierungen in der Formulierung (12) gegeben sind;
• ♦ Ende
• ♦ 4) Berechnen von q (G|x) in Formulierung (7) mit φ₀ und µ*;
• ♦ Bei Konvergenz mit
  • 5) Berechnen des Gradienten ∇_ϑ ℓ_err(, ϑ) um ϑ gemäß Formulierung (5) zu aktualisieren;
• ♦ Ende

Algorithm 1:

• ♦ Random initialization of ϑ, φ and µ
• ♦ When convergent with
  1. 1) Set ϑ is fixed, gradient ∇ℓ _prob (, ϑ) is calculated to update φ according to formulation (4);
  2. 2) Set q is fixed, gradient ∇ _ϑ ℓ _err (, ϑ) is calculated to update ϑ according to formulation (5);
• ♦ End
• ♦ φ ₀ can denote the result of the above procedure;
• ♦ When convergent with
  • 3) updating µ according to the dual problem (10), where the updates are given in the formulation (12);
• ♦ End
• ♦ 4) Calculate q (G|x) in formulation (7) with φ ₀ and µ*;
• ♦ When convergent with
  • 5) Calculate the gradient ∇ _ϑ ℓ _err (, ϑ) to update ϑ according to formulation (5);
• ♦ End

Where µ can be viewed as the weight of the FOL constraints. In one aspect of the present disclosure, one or more FOL constraints may be grouped into one or more groups of FOL constraints, and the grouped FOL constraints may together correspond to only one weight. As illustrated in step 3) of Algorithm 1, the optimization process 500 may need to perform multiple iteration calculations to update each of the weights until it converges. The grouped FOL constraints can reduce the number of weights, which can accordingly save computing resources.

In another aspect of the present disclosure, a value of a FOL constraint may be determined based on a correlation between a set of inputs and a module in a combination of one or more modules of the set of modules according to the estimated posterior distribution given the set generated from inputs. For example, the correlation may refer to whether the semantic representation of a module in a structure illustrated according to the estimated posterior distribution (e.g. at x _n , φ ₀ ) can be found in S(x _n ), as illustrated by formulation (1).

6 shows an example flowchart illustrating a method 600 for performing an abstract visual reasoning task with a probabilistic neural-symbolic model regularized with domain knowledge in accordance with one or more aspects of the present disclosure. For example, method 600 may be performed by network 200 or network 700, which are described in detail below. For example, the method 600 can also be carried out by other networks, systems or models.

In block 610, the network 200, 700 may be provided with a set of input images and a set of candidate images.

In block 620, a combination of one or more modules of the set of modules 220, 720 may be generated based on a posterior distribution over combinations of one or more modules of the set of modules 220, 720 and the set of input images, where the posterior Distribution is formulated by the PGM 210, 710 trained under domain knowledge as one or more posterior regularization constraints. In an example, the training process may be according to method 400 with reference to 4 , as illustrated above, can be carried out.

In block 630, the set of input images and the set of candidate images may be processed by the generated combination of one or more modules of the set of modules 220, 720.

At block 640, a candidate image may be selected from the set of candidate images based on a score of each candidate image in the set of candidate images estimated by the processing.

7 illustrates another example network 700 in which aspects of the present disclosure may be performed. Network 700 may be an example of network 200, as shown in 2 illustrated. For example, the network 700 may include a probabilistic generative model (PGM) 710 and a set of modules 720, such as a reusable module inventory. The PGM 710 and the set of modules 720 may be an example of the PGM 210 and the set of modules 220, respectively. Each module of the set of modules 720 may include a type of processing that may be predetermined to evaluate whether the fields satisfy a specific relationship. The processing types may include the operators logical AND, logical OR, logical XOR, arithmetic ADD, arithmetic SUB, arithmetic MUL and the like. Additionally, each module of the set of modules 720 may include one or more trainable parameters for focusing that module on one or more variable image properties. For example, a module may have a logical AND type and focus on different image properties via the trainable parameters trained by a data set. For example, the logical AND type module may perform a logical AND between line colors, and it may also perform a logical AND between shape positions depending on different trained values of the trainable parameters.

In one aspect of the present disclosure, each module of the set of modules 720 may be configured to perform a pre-developed process on one or more variable image features, and the one or more variable image features may result from processing an input image feature map through at least one trainable parameter . For example, a module with a type of logical AND can be represented as follows: $${ƒ}_{UND}\left(d,e\right)=\left(W_d\cdot d\right)\Lambda \left(W_e\cdot e\right)$$

Where d and e are input field features, W _d and W _e are trainable parameters for focusing on a specific field feature.

basieren, von denen Beschränkungen abhängig sein können.In one aspect of the present disclosure, an image field property may include any property that may be present on an image. In another aspect of the present disclosure, one or more variable image properties may include shape, line, size, type, color, position or number or the like based at least in part on triples using domain knowledge $T\left[r,O,a\right]$

based on which restrictions may depend.

In one aspect of the present disclosure, PGM 710 may be configured to output a posterior distribution across structures of modularized networks 730 composed of the set of modules 720, where the structures 730 may identify the types of composed modules and the connections therebetween. The one or more variable image properties of each module 740 may be determined by training the at least one trainable parameter. Separate generation of structures 730 (e.g., generated by the PGM 710) and variable image properties 740 (e.g., generated based on the trainable parameters) may provide the network 700 more flexibility in abstracting high-level and representative concepts Providing learning.

8th shows an example diagram illustrating an example of performing the method 400, the optimization process 500, or the method 600 through a network 800 in accordance with one or more aspects of the present disclosure. For example, network 800 may be an example of network 200 or network 700. For example, a VAE that includes an encoder 810-1 and a decoder 810-2 may be an example of the PGM 210 or 710. The set of modules 820 may be an example of the set of modules 220, 720 and may form structures G = (v, A). The subnetwork 860 can be used to calculate a score for each candidate image field, which accordingly indicates a degree of correlation between each candidate image field and a result of processing a set of inputs according to a generated modularized network with a structure G = (v, A). For example, the score can be calculated based on various metrics, such as an energy function, where higher energy may indicate better correlation. The posterior distribution unit 850 may store parameters of a posterior distribution that is output by the encoder 810-1 and based on which a structure can be generated, e.g. B. by sampling according to the parameters of the posterior distribution.

In one example, the method 400 may begin by providing the network 800 with sets of inputs and sets of outputs (e.g., via Route 1), where each set of inputs (e.g., X ₁ of 3 × 3 fields of 8th ) of sets of inputs to a set of outputs (e.g. the first field in the first line of Y ₁ of 8th ) corresponding to the set of inputs based on visual information about the set of inputs, and wherein the network 800 includes a probabilistic generative model (PGM) (e.g., an encoder 810-1 and a decoder 810-2) and a set of modules 820 includes. The encoder _810-1 can map or encode the set of inputs B. a total of 20 variables for a summation of 4x4 adjacency matrix entries and 4 vertices of the examples of 3A and 3B) , based on which a structure G = (v, A) can be generated. The sets of inputs Xi and/or outputs Y ₁ can be provided and processed via route 2 to the generated modularized network with the generated structure G = (v, A). The subnetwork 860 may use the processed inputs Xi and outputs Y ₁ to evaluate the correct output (e.g., the first field in the first row of Y ₁ of 8th ) via routes 3 and 5.

The method 400 may repeat the method described with reference to the inputs Xi and outputs Y1, e.g. B. with X ₂ , Y ₂ , X ₃ , Y ₃ , ..., X _n , Y _n . The parameters φ, ϑ of the encoder 810-1, the decoder 810-2 and the modules of the set of modules 820 can be set according to the above with reference to 5 The optimization process 500 described can be updated to obtain the estimated posterior distribution of structures with p̃ _{φ 0} (G|x). be referred to. Furthermore, optimal solutions of the weights µ* can be obtained and used to calculate the regularized posterior distribution of structures according to the above with reference to 5 optimization process 500 described can be used, e.g. B. via Route 6.

Preferably, the parameters ϑ of the modules of the set of modules 820 may be further updated to fit the updated regularized posterior distribution of structures.

In one aspect of the present disclosure, the decoder 810-2 may be used for backward propagation, e.g. via Route 4. In another aspect of the present disclosure, the decoder 810-2 may be omitted.

In one example, the method 600 may be performed for an inference process after the network 800 has been trained according to the method 400 and/or the optimization process 500.

One skilled in the art will understand that the posterior distribution 850 and/or the subnetwork 860 may be integrated into one or more parts of the network 800, rather than as a separate part 8th to be illustrated depending on a design preference and/or a specific implementation without departing from the present disclosure.

9 illustrates an example of a hardware implementation for a device 900 according to an embodiment of the present disclosure. The visual reasoning device 900 may include a memory 910 and at least one processor 920.

The processor 920 may be coupled to the memory 910 and configured to perform the method 400, the optimization process 500, and the method 600, as described above with reference to 4 , 5 and 6 described. The processor 920 can be a general purpose computer or can be implemented as a combination of computing devices, e.g. B. a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. The memory 910 may store the input data, output data, data generated by a processor 920, and/or instructions executed by a processor 920.

The various processes, models, and networks described herein in connection with the disclosure may be implemented in hardware, processor-executed software, firmware, or any combination thereof. According to one embodiment of the disclosure, a visual inference computer program product may include processor-executable computer code for performing the method 400, the optimization process 500, and the method 600 described above with reference to 4 , 5 and 6 are described. According to another embodiment of the disclosure, a computer-readable medium may store computer code for visual reasoning, the computer code, when executed by a processor, may cause the processor to perform the method 400, the optimization process 500, and the method 600, referenced above on 4 , 5 and 6 are described. Computer-readable media includes both non-transitory, computer-readable storage media and communications media, including any media that supports the transmission of a computer program from one location to another. Any compound can be referred to as a computer-readable medium. Other embodiments and implementations are within the scope of the disclosure.

The foregoing description of the disclosed embodiments is provided to enable one skilled in the art to make or use the various embodiments. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the various embodiments. Thus, the claims are not intended to be limited to the embodiments shown herein, but are to be accorded the broadest scope consistent with the following claims and the principles and novel features disclosed herein.

Claims (18)

A method for visual reasoning, comprising: providing a network having sets of inputs and sets of outputs, each set of inputs from the sets of inputs being mapped to one of a set of outputs corresponding to the set of inputs based on visual information about the set of inputs, and wherein the network comprises a probabilistic generative model (PGM) and a set of modules; determining by the PGM a posterior distribution over combinations of one or more modules of the set of modules based on the sets of inputs and sets of outputs; and Applying domain knowledge as one or more posterior regularization constraints to the given posterior distribution. Procedure according to Claim 1 , wherein the one or more posterior regularization constraints are grouped into one or more groups of constraints according to one or more aspects of domain knowledge. Procedure according to Claim 2 , wherein the one or more aspects of domain knowledge include one or more of logical reasoning, temporal reasoning, spatial reasoning or arithmetic reasoning. Procedure according to Claim 1 , where the one or more posterior regularization constraints are one or more first-order logic (FOL) constraints. Procedure according to Claim 4 , wherein the one or more FOL constraints are generated based on at least one of relationship types, object types, or attribute types of the sets of inputs. Procedure according to Claim 1 , wherein each of the combinations of one or more modules of the set of modules comprises a modularized network, the modularized network being composed of one or more modules of the set of modules with a structure indicating the assembled one or more modules and connections therebetween. Procedure according to Claim 6 , further comprising: determining a posterior distribution over structures of modularized networks by the PGM based on the provided sets of inputs and the sets of outputs. Procedure according to Claim 6 , wherein each module of the set of modules includes at least one trainable parameter for focusing that module on one or more variable image properties and is configured to apply a pre-built process type to the one or more variable image properties; and wherein the method further comprises: determining, by the PGM, a posterior distribution over structures of modularized networks indicating the types of the composite one or more modules and the connections therebetween based on the provided sets of inputs and sets of outputs. Procedure according to Claim 1 , the method further comprising optimizing the network by: updating parameters of the PGM and parameters of modules of the set of modules, alternatively by maximizing evidence of the sets of inputs and the sets of outputs to obtain an estimated posterior distribution over the to obtain combinations of one or more modules of the set of modules and optimized parameters of the modules of the set of modules; Updating one or more weights of the one or more posterior regularization constraints applied to the estimated posterior distribution over the combinations of one or more modules of the set of modules to obtain one or more optimal solutions for the one or more weights ; adjusting the estimated posterior distribution over the combinations of one or more modules of the set of modules by applying the one or more optimal solutions of the one or more weights and one or more values of the one or more constraints to the estimated posterior distribution; and updating the optimized parameters of the modules based on the adjusted estimated posterior distribution over the combinations of one or more modules from the set of modules. Procedure according to Claim 9 , where the one or more posterior regularization constraints are grouped into one or more groups of constraints and a group of constraints corresponds to a weight. Procedure according to Claim 9 , wherein a value of a constraint is determined based on a correlation between a set of inputs and a module in a combination of one or more modules of the set of modules generated according to the estimated posterior distribution given the set of inputs. A method for visual reasoning with a network, the network comprising a probabilistic generative model (PGM) and a set of modules, the method comprising: providing the network with a set of input images and a set of candidate images; Generating a combination of one or more modules of the set of modules based on a posterior distribution over combinations of one or more modules of the set of modules and the set of input images, where the posterior distribution of the PGM trained under domain knowledge is one or more posterior regularization constraints are formulated; processing the set of input images and the set of candidate images through the generated combination of one or more modules; and selecting a candidate image from the set of candidate images based on a score of each candidate image in the set of candidate images estimated by the processing. A visual reasoning device comprising: a memory; and at least one processor coupled to the memory and configured to implement the method according to one of Claims 1 until 12 to carry out. A computer program product for visual reasoning, comprising: computer code executable by a processor for performing the method according to one of the Claims 1 until 12 . A computer-readable medium storing computer code for visual reasoning, the computer code, when executed by a processor, causing the processor to execute the method of one of the Claims 1 until 12 to carry out. Visual reasoning network comprising: a set of modules, each of the set of modules being implemented as a neural network and having at least one trainable parameter for focusing that module on one or more variable image properties; and a probabilistic generative model (PGM) coupled to the set of modules, the PGM configured to output a posterior distribution over combinations of one or more modules of the set of modules. network Claim 16 , wherein each of the set of modules is configured to perform a predefined type of processing on the one or more variable image features, and the one or more variable image features result from processing an image feature map through the at least one trainable parameter. network Claim 17 , wherein the one or more variable image properties include one or more of shape, line, size, type, color, position or number and the prefabricated processing type is a logical AND, logical OR, logical XOR, arithmetic ADD, arithmetic SUB, arithmetic MUL, spatial STRUC, temporal PROG or temporal ID.

Images