JP2000172842A - Unknown target and method for estimating unknown target from observation record of training data - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a general kind of problem of low-level vision by sending a local evidence to an adjacent node in an inference stage and determining the maximum post-probability of scene estimation. SOLUTION: Training data 2 is an observation record of a known target. A display of discontinuity 11 or continuity 12 modeling the training data 2 is selected. The statistical relationship of the training data 2 is learnt by using the display of discontinuity 11 or continuity 12. Their relationship is represented as a mix of a vector, a matrix, or a Gaussian distribution. After the learning stage, inference is carried out as to the unknown target. A probability function Pd21 or Pc22 is used to infer one which is possibly a target 31 from the observation record 32 of the unknown target. This inference is carried out by locally transmitting the reliability through the Markov network.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates generally to computer vision, and more particularly, to estimating the characteristics of a scene represented by an image. That is,
The present invention relates to a method for estimating a target by using statistical characteristics of observation records of a known target.

[0002]

BACKGROUND OF THE INVENTION One of the common problems in computer vision is how to determine the characteristics of an underlying scene from an image representing the scene. Some specific issues are listed below. For motion estimation, the input is typically a temporally ordered sequence of images, eg, “video”. The question is how to estimate the estimated speed of various things-humans, cars, balls, moving background in the video. Other problems include recovering real-world three-dimensional (3D) structures from 2D images, eg,
It deals with how to recover the shape of an object from a line drawing, a photograph, or a pair of stereo photographs. Yet another problem is how to recover high-resolution scene details from low-resolution images.

[0003] Humans always make this type of estimation, often semi-involuntarily. There are also many applications that allow machines to do this. These problems have been studied by many researchers for many years with different approaches, with varying success. The problem with the best-known approach is that it lacks machine learning methods that can utilize the power of current processors within a general framework.

[0004]

In the prior art, methods have been developed for interpreting an image of the world of blocks. Other prior art work with hand-labeled scenes has analyzed local features of aerial images based on vector codes, and has developed rules that convey scene interpretation. However, these solutions are not
It is for the category of steps and therefore cannot be used to solve the general class of low-level vision problems. Although methods of communicating probabilities have been used, they are not within the general framework for solving vision problems.

[0005] Alternatively, by using a set of four trees, the optical flow can be estimated from the image to convey motion information at various rates. In that case, using the assumption of constant brightness, the reliability of the speed of the light flow is expressed as a Gaussian probability distribution.

[0006]

SUMMARY OF THE INVENTION The present invention analyzes the statistical properties of a labeled visual world to estimate a visual scene from corresponding image data. The image data may have a single frame or a large number of frames. The estimated scene characteristics may be a projected object velocity, surface shape, reflectance pattern, or color. The present invention uses statistical properties gathered from labeled training data to form a "best guess" estimate, or optimal interpretation, of the underlying scene.

The present invention operates in the learning and inference stages. During the learning phase, the statistical properties for the training data are modeled as a mix of probability density functions, eg, Gaussian distributions. A Markov network is established. During the inference phase, the confidence and density functions extracted from a particular image are passed around the network to make an estimate about the particular scene corresponding to that particular image.

Thus, during the learning phase, training data for normal images and scenes is synthesized and generated. Parameter symbol tables are generated for both images and scenes. Like the probability of a scene parameter conditioned on adjacent scene parameters, the probability of an image parameter conditioned on a scene parameter (likelihood function) is modeled. These relationships are modeled in a Markov network, where local evidence is passed to neighboring nodes during the inference phase to determine the maximum posterior probability of the scene estimation.

The way in which humans interpret scenes is largely unknown, but certainly not mathematically explicit. We describe a visual system that interprets visual scenes by determining the probabilities of each possible scene interpretation for all local images and determining the probabilities of any two local scenes adjacent to each other. . The first probability allows the visual system to make scene estimates from local image data, and the second probability conveys these local estimates. 1
One embodiment uses a Bayesian method constrained by the Markov assumption.

The method according to the invention can be applied to various low-level vision problems, such as estimating high-resolution scene details from low-resolution image versions, estimating the shape of objects from line drawings. . In these applications, even without domain knowledge, spatially localized statistical information is sufficient to reach a reasonable overall scene interpretation.

In particular, the present invention provides a method for estimating a scene from an image. A plurality of scenes are generated, and an image is rendered for each scene. By these,
Training data is formed. These scenes and corresponding images are divided into patches. Each patch is quantified as vectors, and these vectors are modeled as a mix of probability densities, eg, Gaussian. Statistical relationships between patches are modeled as Markov networks. Local probability information is repeatedly transmitted to adjacent nodes of the network, and the resulting probability density at each node,
The “reliability” is read to estimate the scene.

In one application of the present invention, it is possible to estimate high-resolution details from blurry, ie, low-resolution, images. The low resolution image is the input "image" data, and the "scene" data is
High resolution detail image strength. The invention can also be used to estimate scene motion from a sequence of images. In this application, the image data is the image intensity from two consecutive images in the series, and the scene data is a continuous speed map showing the projection speed of the visible object at each pixel location. Another application of the invention is shading and reflectivity unification.

The present invention also provides other estimation problems that allow training data and target data to be modeled with a probability density function, such as speech recognition, seismic research, and medical diagnostic signals such as EEG and EIKG. It can also be applied. Further, the probability indication may be discontinuous or continuous, at either the learning stage or the inference stage.

[0014]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 (Introduction) Describes how to use the statistical properties of the labeled visual world to estimate the properties of a scene using either a single image or multiple images. The estimated scene characteristics may include a projected object velocity in the scene, a surface shape of the object, a reflectance pattern, or a color. This general approach can be applied to many low-level vision problems. The method can also be used to model the statistical properties of other complex digital signals, such as, for example, human speech, seismographs, and the like.

As shown in FIG. 1, the general method 1 starts with training data 2. The training data is an observation record of a known target. The training data 2 may be randomly generated. In step 10, a display of discontinuities 11 or series 12 for modeling the training data is selected. In step 20, a statistical relationship for the training data is learned using either a discontinuous or continuous representation. These relationships can be expressed as either discrete or continuous probability functions (P _d 21, or P _c 22), for example, vectors and matrices, or a mix of Gaussian distributions.

After the learning phase, inferences about unknown targets can be made. In step 30, P
Either _d 21 or P _c 22 is used to infer what is likely to be the target 31 from the observation record 32 of the unknown target. This inference is performed in a Markov network by locally transmitting the reliability. In a Markov network, nodes in the network represent observational records, and the confidence is a statistical indication of the discontinuity or continuity of the confidence. Step 30 is
It may be repeated for other targets similar to the training data.

Generating Random Scenes and Images for Training Data As shown in more detail in FIG. 2, during the training phase, the general method 100 includes, in step 110, a random scene x _i (known target) ) And the corresponding image y _i (observation record) are generated as training data 111. Random scenes and rendered images can be generated synthetically using computer graphics. The composite image shows some of the features of the unknown image that the system processes.

(Division of Scene into Patches) Step 120
, The scene and the corresponding image are in local patch 1
It is divided into 21. The segmentation may be a square patchwork covering the scene and the image. The size of the patch may be many, and the patch may be redundantly mounted. For example, patches may be formed in multiple Gaussian pyramids. The pyramid has, for example, five levels of resolution-
It may have-from dense to coarse. Further, the patches may represent image information viewed through differently oriented filters.

All patches of a given set of references, such as resolution, orientation, etc., but spatially different, are said to be of the same partition and are assumed to be drawn from the same statistical distribution. . The size of the patch is small enough to allow modeling, but large enough to convey meaningful information about the entire scene.

(Quantification of Patch as Vector) In step 130, the principal component
The display for each patch is determined using analysis (PCA). Each patch is represented as a linear combination of the base functions. Patch 12
1 is represented as a low-dimensional vector 131. For example, each scene patch may be represented as a five-dimensional vector, and each image patch may be represented as a seven-dimensional vector.
In other words, random training data, scenes,
And each patch of the image is represented, for example, as a point in five-dimensional and seven-dimensional space.

(Modeling of probability density of training data) In step 140, the probability densities of all training data in these low-dimensional spaces are modeled by a mix of Gaussian distribution. The training data is used to estimate the local patch probabilities in a very general form:

P (scene), P (image | sc
ene) and P (neighboring sc
ene | scene)

More specifically, the following three probability densities 141 are modeled.

(1) There is a priori probability of each scene element x, and a different a priori probability exists for each segment of the scene element.

(2) The conditional probability of the scene element x when the related image element y is given, that is, P (y |
x), and

(3) The conditional probability of the scene element x ₁ and the adjacent scene element x ₂ , that is, P (x ₁ | x ₂ ).

The adjacent elements may be close in spatial position, or may be close in any one of the segmentation attributes such as scale and orientation.

It may be useful to modify the training data so that it has a probability distribution that is easier to fit into a mix of Gaussian distributions. For real images, many distributions of interest have very steep spikes at the origin. This peak matches the Gaussian distribution mix, and it is difficult to manipulate the Gaussian mix. A priori probabilities of the scene data can be determined from the statistical analysis of the labeled visual data. Then, it is possible to pass the training data a second time and randomly delete each training sample with a probability that is inversely proportional to the a priori probability of the scene data.
This gives a biased set of data with a probability distribution that is easier to model.

(Establishment of Markov Network) In the last step 150 of the learning phase, the patches and their associated probability densities are organized into a Markov network 200 representing the statistical relationship between the scene and the image. In a Markov network, each node represents a low-dimensional vector, node x _i represents a scene, and node y _i represents an image. The edges connecting nodes represent the statistical dependence between those nodes.

Also, when a Gaussian pyramid is used, nodes of a given resolution level can be connected to nodes of the same level spatially adjacent and nodes of the same resolution level and at the same spatial location.
In addition, different scene elements can be connected in some other dimension, such as the orientation of an oriented filter.

These connections facilitate removing spatial artifacts while estimating the scene. The connected Markov network 200 allows each scene node to update its confidence based on accumulated local evidence gathered from other nodes during the following inference phase. Confidence is the combined probability density that forms the final best estimate.

During the inference phase (repeating confidence and reading out the best estimate), an unknown observation record or image 17
2, an unknown target scene 171 is estimated. Based on the rules described below, step 160 determines the Bayesian "reliability" at each node in message 16
1 is repeatedly transmitted to adjacent nodes. The Bayesian or regular approach has been used in low-level vision issues. However, in contrast to the prior art, it uses labeled training data (scenes and corresponding images) and uses strong Markov assumptions.

At step 170, the best estimate 171 at each node for the corresponding scene given the observed image information is read. This can be done by examining the probability distribution for reliability at each node and taking either the mean or the maximum of that mix of Gaussian distributions. This tells us what scene value is the best estimate for the target scene directly below at that location given the observed image data.

(Example of 3 × 3 Markov Network) FIG. 3 shows a simple 3 × 3 Markov network. For simplicity, we have made all data one-dimensional so that we can plot the data. The “scene data” to be estimated is a 1D x at each node.
201. Using 1D image data y202 coming to each node, what x is is estimated.

In a typical use of the present invention, randomly generated computer graphic scenes (known targets) and their corresponding rendered images (observation records) to produce a set of images and scenes for training. Generate They are used to generate vectors representing image and scene training patches from which to gather desired a priori and conditional statistics.

However, for this simple example, composite data corresponding to vectors representing training patches for images and scenes is formed. Form the underlying joint probability relationship that governs the image and the scene.

FIG. 4 shows the joint probability relationship 300 of the variables x and y for this simple example. 4, the variable x is along the horizontal axis 301 and the variable y is the vertical axis 30.
Along 2. If y is zero, the variable x is
It can have one of many possible values, as shown by the wide distribution of the central blurred horizontal line 303 in FIG. If the observation record y is 2, x is somewhat closer to 3.

Further, in this simple example, the relationship between the values x of adjacent scene patches is as follows.
When lowering the "row" 203 of the network 200, the scene data x is always multiplied by 2; when going to the right one column 204, the scene data x is multiplied by 1.5.

For this simple example, image data y coming to a node is formed. Again, for simplicity, all nodes except node 5 are set to y = 0.

Therefore, all nodes have a wide range of uncertainties regarding their values. Node 5 determines the observed value y =
2 In this case, the observed value of the central node 5 should almost certainly be 3. Propagating the Bayesian trust then conveys that knowledge to all other nodes in the network 200. The final estimate is x = 3 at node 5;
The x value of the other node is 1.5 or 2 each time one goes horizontally to the right or down in the direction away from node 5.
(And going in the opposite direction, 1 /
1.5 and 1/2).

The example network 200 is a nine-node dendrogram, starting at 1 at the root of the dendrogram, with consecutive numbers assigned to each node. The local scene state of the i-th node is x _i , and the image evidence at the i-th node is y _i .

Following each step of the general method 100 outlined above, proceeds as follows. Gather training data from computer graphic simulations of the problem. For the problem in this example, simulated data is generated by drawing from a known joint distribution of y and x, and x ₁ and its neighboring nodes x ₂ .

For simple 1D problems, it is not necessary to perform Principal Component Analysis (PCA) to reduce the dimensions of the data collected at each node. Next, a desired joint probability is estimated using a mix of Gaussian probability models. Bishop “Neural n
See etworks for pattern recognition, "Oxford, 1995.

FIG. 5 shows a histogram of the observed values of x, FIG. 6 shows a mix of Gaussian distributions that fits a priori probability density, and FIG. 7 shows a simplified mix of the Gaussian distributions. It is. After each multiplication or probability match, it is deleted for the reasons described below.

FIG. 8 shows a mix of Gaussian distributions that fits some of the required conditional probabilities 141.
Using simultaneous data about what happens (a) and (b) at the same time, P (a, b) / P (b) = P (a |
Given b), fit the Gaussian mix to the model's conditional probability P (a | b) by weighting each point by 1 / P (b). FIG. 8 (a)
Shows the mix of fits of the Gaussian distribution to the probability density y given x, and FIG. 8 (b) shows a straight line with a slope of 1 / 1.5 given the value of x. Shown is a mix of Gaussian fits to the probability density of the neighbors to the right of x. FIG. 8 (c) shows a mix of fits of the Gaussian distribution to the probability density of the neighbors below x for a straight line of 勾 配 slope, given the value of x.

According to the rules described later, the reliability is repeatedly calculated at each node. The first step is to determine what message to pass from each node to each of its neighbors.

FIGS. 9 (a)-(d) graphically illustrate each of the probabilities of multiplying together to generate a message that node 5 conveys to node 4 above it in the first iteration. FIG. 9A shows the probability from the image.
9 (b) is from node 2, FIG. 9 (c) is from node 6, and FIG. 9 (d) is from node 8.

FIG. 9E is the product of the probabilities shown in FIGS. 9A to 9D. Next, by increasing the dimension of the distribution shown in FIG. 9 (e) and keeping the distribution constant in the dimensions included in FIG. 9 (f) but not included in FIG. 9 (e). , The dimensions of the distribution shown in FIG. Next, the raised distribution is multiplied by the conditional density shown in FIG. 9F to form a margin along the dimension of the distribution included in FIG. 9E. As a result, the message 16 shown in FIG.
1 is sent from node 5 to node 4.

FIG. 10 shows a message that node 5 sends to node 4 in the order of a priori probabilities multiplied together, from local image data, a message from adjacent nodes 4, 2, 6, and 8. , And the probability of calculating the final confidence (estimate) from the image at node 5 at the end of the first iteration.

FIGS. 11-13 show the "reliability" at each node in the network during the first three iterations of the method. As shown in FIG. 11, information has not yet been transmitted between the nodes, and each node depends only on its own local image information y,
Estimate its own x value. Since y = 0 at all nodes except node 5, they have received very little information about their x-values and their confidence in their x-values is very widely distributed. Node 5 knows that its x value is close to three. This is because it is implied by y = 2. The reliability shown at each node is P (y | x) for an appropriate value of y at each node.
P (x).

In the second transmission, as shown in FIG. 12, each node shares its own information with its adjacent nodes. Nodes 2, 4, 6, and 8 have received informational messages from node 5, which is the only node that probably knows what value of x they have, The node adjusts its confidence in its value of x accordingly. The distribution shown at each node is P
(Y | x) P (x) multiplied by a message from each of the nodes adjacent to the node.

According to the third transmission, each node is informed by all two nodes behind it, and thus each node has received knowledge from node 5. After the third propagation, the average or maximum of the confidence of each node is about the same as it should be. That is, x of the node 5 has a value of approximately 3, and the other x values are 1.5 times to the right and 2 to the bottom.
Double.

(Simplification of Mix) When the probability mix of N Gaussian distributions is multiplied by the probability mix of M Gaussian distributions, a mix of NM Gaussian distributions is generated. Therefore, when the Gaussian distribution mix is multiplied, the number of Gaussian distributions increases rapidly, so that the Gaussian distribution must be simplified. The Gaussian distribution can be easily sieved by the threshold with very little weight from the mix, but this can result in inaccurate mix fitting.

(Factorization of Joint Probability) The factorization of joint probability used for transmitting local evidence to an adjacent node will be described in detail with reference to FIG. The network shown in FIG. 14 has the following four scene nodes and image nodes, respectively.

X ₁ ,. . . x _4, and y _1,. . . y ₄

Find the factorization of joint probabilities that results in rules that convey local evidence. In this factorization, the following three probability operation rules are used repeatedly.

Rule [1] The basic probability P (a, b) = P (a | b) P (b).

Rule [2] If node b is between node a and node c,
P (a, c | b) = P (a | b) P (c | b).
This is a conditionally independent statement of a and c when b is given.

Rule [3] If node b is between node a and node c,
P (c | a, b) = P (c | b). This is a Markov property that allows knowledge of the rest of the chain to be summarized by knowledge of the closest node.

It should be noted that none of these three rules need send the edges connecting the nodes. This eliminates the need for arbitrarily selecting causal relationships in the network 200.

To estimate the maximum posterior (MAP) probabilities of the parameters x ₁ , x ₂ , x ₃ , x ₄ , argmax
_{x1, x2, x3, x4 P} (x 1, x 2, x 3, x 4 | y 1, y 2, y 3,
y ₄₎ I want to determine. This conditional probability is the joint probability P
(X ₁ , x ₂ , x ₃ , x ₄ , y ₁ , y ₂ , y ₃ , y ₄ ) by a factor that is constant over the changing independent variables.
_{Therefore, argmax x1, x2, x3,} x4 P (x 1, x 2, x 3,
x ₄ , y ₁ , y ₂ , y ₃ , y ₄ ) can be equally selected to be determined, which is easier to determine.

Another useful estimate of each parameter x _i is the mean of the marginal distribution, P (x _i | y ₁ , y ₂ , y ₃ , y
₄ ). This average value is represented by the joint distribution P (x ₁ , x ₂ ,
x ₃ , x ₄ , y ₁ , y ₂ , y ₃ , y ₄ ) by peripheralizing (integrating) all x parameters other than x _i . By this peripheralization, P (x
_{i, y 1, y 2,} y 3, y 4) is produced. This is due to the distribution P (x _i | y ₁ , y ₂ , y ₃ , y ₄ )
Therefore, the average values of the two distributions are the same. The next factorization step for MAP estimation also applies to the mean of the marginal distribution, with the following changes. Instead of the operation argmax _xj , a variable x _j (I
x _j ). Instead of the final argmax operation on the reliability at a node, take the average of that reliability.

For the calculation at each node, the joint probabilities are factored in different ways. Each node j has P (x
_j ), and does not transmit the quantity to the adjacent node. This allows an algorithm to convey invariant local evidence, and the output is always optimal given the number of nodes being reported.

Continuing with an example, the network 200
, Four different cases will be described for each of the four nodes. First, argma at node j
Factorization performed at each node so that x _j has the same value as the following equation will be described.

Argmax _{x1, x2, x3, x4} P (x ₁ , x ₂ ,
_{x 3, x 4 | y 1} , y 2, y 3, y 4)

After these four cases, the rules that convey general local evidence are presented. These perform the calculation of each factorization.

(Calculation at Node 1) By applying rule 1 and then applying rule 2, the following equation is obtained.

P (x ₁ , x ₂ , x ₃ , x ₄ , y ₁ , y ₂ , y ₃ , y ₄ ) = P (x ₂ , x ₃ , x ₄ , y ₁ , y ₂ , y ₃ , y ₄ | x ₁ ) P (x ₁ ) = P (y ₁ , x ₁ ) P (x ₂ , x ₃ , x ₄ , y ₂ , y ₃ , y ₄ | x ₁ ) P (x ₁ )

Applying rule 1 and then applying rule 3, the factorization continues as follows:

P (x ₂ , x ₃ , x ₄ , y ₂ , y ₃ , y ₄ | x ₁ ) = P (x ₃ , x ₄ , y ₂ , y ₃ , y ₄ | x ₁ , x ₂ ) P (x ₂ | x ₁ ) = P (x ₃ , x ₄ , y ₂ , y ₃ , y ₄ | x ₂ ) P (x ₂ | x ₁ )

Applying rule 2 twice,

P (x ₃ , x ₄ , y ₂ , y ₃ , y ₄ | x ₂ ) = P (y ₂ | x ₂ ) P (x ₃ , y ₃ | x ₂ ) P (x ₄ , y ₄ | X ₂ )

Applying rule 1, then applying rule 3,

P (x ₃ , y ₃ | x ₂ ) = P (y ₃ | x ₂ , x ₃ ) P (x ₃ | x ₂ ) = P (y ₃ | x ₃ ) P (x ₃ | x ₂ ) And P (x ₄ , y ₄ | x ₂ ) = P (y ₄ | x ₂ , x ₄ ) P (x ₄ | x ₂ ) = P (y ₄ | x ₄ ) P (x ₄ | x ₂ )

By applying all these substitutions, the following equation is obtained.

P (x ₁ , x ₂ , x ₃ , x ₄ , y ₁ , y ₂ , y ₃ , y ₄ ) = P (x ₁ ) P (y ₁ | x ₁ ) P (x ₂ | x ₁ ) P (y ₂ | x ₂ ) P (x ₃ | x ₂ ) P (y ₃ | x ₃ ) P (x ₄ | x ₂ ) P (y ₄ | x ₄ )

Passing the argmax gradient through a variable with constant substitution yields:

[0078] _{argmax x1, x2, x3, x4} P (x 1, x 2, x 3, x 4, y 1, y 2, y 3, y 4) = argmax x1 P (x 1) P (y 1 | _{x 1) argmax x2 P (x} 2 | x 1) P (y 2 | x 2) argmax x3 P (x 3 | x 2) P (y 3 | x 3)

The above result is for obtaining the MAP estimation of the joint posterior probability. As described above, instead of this, to obtain the average value of the marginal distribution, x _{1 of the} following expression
Take an average about.

P (x ₁ , y ₁ , y ₂ , y ₃ , y ₄ ) = P (x ₁ ) P (y ₁ | x ₁ ) I _x2 P (x ₂ | x ₁ ) P (y ₂ | x ₂ ) I _x3 P (x ₃ | x ₂ ) P (y ₃ | x ₃ )

(Generalization) Using rule 1, P (x _a ) is made to appear at node a. According to rule 2, node a
Each edge gives a coefficient in the form of P (the other variable | x _a ). Each of these "other variable" strings is decomposed again using rules 1 and 2 and
Simplifies any additional conditioning variables.

Thus, the joint probability is factorized in a manner that reflects the topology of the network from the viewpoint of node a. For a chain of three nodes where nodes b and c branch off from node a, the following equation is obtained.

P (x _a , x _b , x _c ) = P (x _a ) P (x _b
│x _a ) P (x _c │x _a )

Image y branching from each node
Is included, the following equation is obtained.

[0085] _{P (x a, x b,} x c, y a, y b, y c) = P (x a) P (y a | x a) P (x b | x a) P (y b | _{x b) P (x c |} x a) P (y c | x c)

(Calculation at Node 2) The different factorizations used at Node 2 are written using the three operating rules. Now, the only a priori probability for a single variable is
P (x ₂ ).

[0087] _{argmax x1, x2, x3, x4} P (x 1, x 2, x 3, x 4, y 1, y 2, y 3, y 4) = argmax x2 P (x 2) P (y 2 | _{x 2) argmax x1 P (x} 1 | x 2) P (y 1 | x 1) argmax x3 P (x 3 | x 2) P (y 3 | x 3) argmax x4 P (x 4 | x 2) P (Y ₄ | x ₄ )

(Calculation at Node 3) P (x ₁ , x ₂ ,
x ₃ , x ₄ , y ₁ , y ₂ , y ₃ , y ₄ ) are factorized, and a factor of the following equation is taken out.

P (x_Three), Argmax_{x1, x2, x3, x4}P (x₁, X_Two, X_Three, X_Four, Y₁, Y_Two, Y _Three , Y_Four) = Argmax_x3P (x_Three) P (y_Three| X_Three) Argmax_x2P (x_Two| X_Three) P (y_Two| X_Two) Argmax_x1P (x₁| X_Two) P (y₁| X₁) Argmax_x4P (x_Four| X_Two) P (y_Four| X_Four)

(Calculation at Node 4) P (x ₁ , x ₂ ,
x ₃ , x ₄ , y ₁ , y ₂ , y ₃ , y ₄ ) are factorized, and a factor of the following equation is taken out.

P (x_Four), Argmax_{x1, x2, x3, x4}P (x₁, X_Two, X_Three, X_Four, Y₁, Y_Two, Y _Three , Y_Four) = Argmax_x4P (x_Four) P (y_Four| X_Four) Argmax_x2P (x_Two| X_Four) P (y_Two| X_Two) Argmax_x1P (x₁| X_Two) P (y₁| X₁) Argmax_x3P (x_Three| X_Two) P (y_Three| X_Three)

(Locally Transmitted Rules) With a single set of transmitted rules, each of the above four calculations arrives at four different nodes.

During each iteration, each node
Code x_jGathers evidence, then each connection node x_kTo
Give an appropriate message. The evidence from node k is
Here is the latest message you will receive. Image y_j
Evidence from P (y_j| X _j).

(1) A message sent from the node j to the node k starts with a product Q (j; k) of evidence at the node j from a node other than the node k. Node k is the node receiving the message. This includes
Includes local node evidence P (y _j | x _j ).

(2) Then, the message sent to node k is argmax _xj P (x _j | x _k ) Q (j; k)
It is. Using a different calculation, the optimal x _j from node _j
Is read.

(3) P (x ₁ , x ₂ , x ₃ , x ₄ , y ₁ ,
argma for the product of P (x _j ) with all the evidence at node _j to find x _j that maximizes y ₂ )
Take x _xj .

(Conveying Rule, In the Case of Discontinuity) This transmitting operation may be more easily expressed in the case of displaying the probability of discontinuity. In this embodiment,
A discontinuous probability representation is used during both the learning phase and the inference phase. During training, for node j next to node k, the concurrent histogram H (y _j , x _j )
And H (x _j , x _k ) are measured. From these histograms, P (y _j | x _j ) and P (x _j | x _k ) can be estimated. The histograms H (a,
If b) is stored as a matrix of rows denoted by a and columns denoted by b, P (a | b) after adding a small constant to each count for Poisson arrival statistics
Are standardized rows of the matrix. Each row sums to one.

Node j receives a column vector message from each node. To send a message from node j to node k, node j

(1) Each incoming message (node k
) Are multiplied one by one and multiplied by a column vector P (y _j | x _j ), and then

(2) The resulting vector and P
(Maximum matrix multiplication) with (x _j | x _j ).

The resulting column vector is the message to node k.

The term "maximum matrix multiplication" means the product of the column vector multiplied by one term with each row of the matrix, and the output for the index of the output column vector is multiplied by the maximum value of the multiplied product. Set to equal. For minimum mean squared error (MMSE) estimation, the prior art vector-matrix product is used instead of the maximum matrix multiplication step.

In the discrete probability representation, to read out the best estimate of x at node j, the most recent message from each connected node is multiplied by one term to form a column vector P (y _j | x _j ). And multiply by the column vector P (x _j ). The index that maximizes the resulting column vector is the best estimate of x, which is in the target scene.

(Super Resolution Problem) In one application of the present invention, high resolution details are estimated from a blurry, ie, low resolution, image. In this application, the image data is the image intensity of the low-resolution image, and the "scene" data is the image intensity of the high-resolution detail.

The training image begins with a randomly shaped blob covered with random surface markings rendered by computer graphics techniques. First, in order to obtain a bandpass image, an oriented bandpass filter is applied. A spatially varying local multiplicative gain control coefficient is applied to this bandpass image. The gain control factor is calculated as the square root of the squared blurred value of the bandpass image. This constant gain control standardizes the edge strength of the image and reduces the burden on the next modeling step. The resulting image represents "observed" information.

The oriented high-pass filter also acts on the rendered image, and then applies the spatially varying local gain control coefficients calculated from the bandpass image. The result is the corresponding target or "scene"
Represents information.

Many such pairs of images and scenes were generated to establish training data. Each image-scene pair was divided into patches in the same grid structure at a single spatial rate. Separate PC for image patch and scene patch
A was applied to obtain a low dimensional representation for each patch.

The required conditional and a priori probabilities were determined from the training data and a Gaussian mix was fitted to the data. The local information was conveyed to obtain an estimated high-resolution image. In this embodiment, a continuous probability expression is used during both the learning stage and the inference stage.

(Hybrid Probability Density Display) The above method of transmitting the probability density can be improved in terms of processing speed. The continuous display of the probability densities allows the input image data to better match during the learning phase. Discontinuous indications can travel faster during the inference phase. Next, a hybrid method is described that allows both good fit and fast propagation.

In the case of this hybrid, FIG.
The a priori and conditional distributions 141 of each node 20 in the Markov network 200 of FIG.
1 is evaluated only in a set of different discontinuous scene values. The scene value is a sampling of the scene to render in the image at that node. This concentrates the computation on locally viable scene interpretation.
The conditional probabilities P (x _j | x _k ) are evaluated in the scene samples at nodes j and k, respectively.
Represents the ratio between the Gaussian distribution mix P ( _xj , _xk ) and P ( _xk ). The conditional probability P (y _k | x _k ) is the probability P (y _k , x _k ) / P (x _k ) evaluated in the scene sample at node k for the observed image value y _k there. .

Thus, instead of multiplying the Gaussian mix by each other in order to convey information between nodes as described above, the probability samples are multiplied by a set of discrete points in the scene domain. Match. The set of scene samples is different at each node of the network 200 and depends on the image information coming to that node. The confidence at one node is a set of probability weights on each of the scene samples at that node. The transfer of reliability from node j to node k includes P (x _j |, which is the product of the vector of discontinuous samples from Q (j: k) and each row of the link matrix, one point at a time. x _k ) (from Rule 1 of the Tradition).
In accordance with Tradition Rule 2, the value of the resulting vector is the maximum of the product of each row. For example, 1
When using samples from 0 to 15, it is possible to fully describe the underlying scene while reducing processing time. By using discontinuous expressions during inference,
Processing speed is improved by more than two orders of magnitude from 24 hours to about 10 minutes.

To select a scene sample, the image element y observed at each node is mixed P
Given the condition of (y, x), the scene element x can be sampled from the resulting Gaussian mix. Better results can be obtained simply by using these scenes from the training set whose image best matches the image observed at the node.
This avoids one modeling step of a Gaussian mix.

With this scene estimation method in a supersampling application, low resolution images are zoomed with high quality.

(Other Applications) The present invention also provides
It can also be used to estimate scene motion from a series of images. In this application, the image data is the image intensity from two consecutive images in the series, and the scene data is a continuous speed map showing the projection speed of the visible object at each pixel location.

Another application of the present invention is unifying shading and reflectivity. Images can result from shading effects on the surface as well as changes in the reflectivity of the surface itself. For example, an image of a shaded surface can originate from the shaded surface itself, as well as from a flat surface (eg, a flat picture thereof) painted to look like the shaded surface. The image data for the application would be the image itself. The underlying estimated scene data would be the underlying surface shape and reflectivity pattern. The method can be used to best estimate the 3D scene represented by the image and the pattern to be drawn.

The present invention can also be used to make estimates for other complex digital signals. For example, the present invention can be used with any signal that can be statistically sampled and represented as a probability density function, such as speech, seismic data, medical diagnostic data, and the like.

[0117]

The method according to the invention applies to various low-level vision problems, for example estimating high-resolution scene details from low-resolution image versions, estimating the shape of objects from line drawings. Can be. In these applications, even without domain knowledge, spatially localized statistical information is sufficient to reach a reasonable overall scene interpretation.

In this description of the present invention, certain terms and examples have been used. It is to be understood that various other adaptations and modifications may be made within the spirit and scope of the invention. It is therefore the object of the appended claims to cover all such changes and modifications that fall within the true spirit and scope of the invention.

[Brief description of the drawings]

FIG. 1 is a flowchart of a method for estimating a target.

FIG. 2 is a detailed flowchart of a method for estimating a scene from an image according to the present invention;

FIG. 3 is a graph of a network conveying the reliability of the method.

FIG. 4 is a graph of the true underlying joint probability relating a scene variable x to an image variable y.

FIG. 5 is a histogram of scene values observed in training data.

6 is an initial mix of a Gaussian distribution that fits the distribution shown in the histogram of FIG.

FIG. 7 is a simplification of the adaptation of FIG. 6;

FIG. 8 shows a mix of fitting a Gaussian distribution to the conditional probabilities observed in the training data.

FIG. 9 is a graph of probabilities at various nodes of the network (ad), a product of probabilities shown in ad (e), a graph of conditional density (f), and a graph of probabilities of propagating in a message (g) ).

FIG. 10 is a graph of the probability of combining to form node reliability.

FIG. 11 is a graph of initial probabilities.

FIG. 12 is a graph of the probability after the first iteration.

FIG. 13 is a graph of the probability after the second iteration.

FIG. 14 is a graph of a Markov network having four scene nodes and image nodes.

[Explanation of symbols]

1 general method, 2 training data, 11 discontinuous, 12 continuous, 31 targets, 32 observation records, 1
00 General method, 200 Markov network.

 ──────────────────────────────────────────────────続 き Continuation of the front page (71) Applicant 597067574 201 BROADWAY, CAMBRIDGE, MASSACHUSETS 02139, S. A. (72) Inventor William T. Freeman Half Moon Hill, Acton, Massachusetts, United States of America 16 (72) Inventor Egon Sea Pasteur Jamaica Plain, Warren Square, Massachusetts, United States of America 6

Claims (9)

[Claims] 1. A method for estimating an unknown target from an unknown target and an observation record of training data, comprising: generating a plurality of known targets and an observation record of the known target to form training data; Dividing the training data into corresponding subsets; quantifying each subset as a vector and modeling the probabilities for each vector; observing an unknown target and the probabilities of the training data. Repeatedly transmitting local probability information to adjacent nodes of the network; reading the probabilities at each node to estimate the unknown target from the unknown target and the observation record of the training data. Method of estimating the unknown target from observations of an unknown target and the training data including and. 2. The method of claim 1, wherein said dividing and said quantifying are performed during a learning phase, and said transmitting and reading are performed during an inference phase. A method for estimating unknown targets from observation records of targets and training data. 3. Estimating an unknown target from an unknown target and an observation record of training data according to claim 2, wherein the probability is represented as a function of a discontinuity between the learning phase and the inference phase. Method. 4. The method of claim 2, wherein the probabilities are expressed as a continuous function during the learning and inference stages. . 5. The unknown target and training data of claim 2, wherein the probabilities are represented as a discontinuous function during the learning phase and as a discontinuous function during the inference phase. A method of estimating unknown targets from observation records. 6. The unknown target according to claim 5, wherein the discontinuous function comprises a vector and a matrix, and the continuous function is a mix of Gaussian distributions. how to. 7. The observation of unknown target and training data according to claim 1, wherein the unknown target is a scene to be estimated, and the training data includes a random target and a corresponding image of the random target. A method of estimating unknown targets from records. 8. The unknown target and the unknown target from the training data observation record of claim 1, wherein the network is a Markov network and the nodes in the network represent the observation record of the unknown target. How to estimate. 9. The method of claim 1, further comprising the step of repeating the inference step for different observation records of an unknown target.
The method for estimating an unknown target from the unknown target and the observation record of the training data described in 1.