JP6504590B2 - System and computer implemented method for semantic segmentation of images and non-transitory computer readable medium - Google Patents

Description

  The present invention relates generally to computer vision and machine learning, and more particularly to semantically labeling images.

  Semantic segmentation aimed at predicting the category labels of every pixel in an image is an important task for understanding the scene. Semantic segmentation is a difficult problem due to the large changes in the visual appearance of semantic classes and the complex interactions between the various classes in the visual world. In recent years convolutional neural networks (CNNs) have been shown to be effective for this difficult task. However, convolutional neural networks may not be optimal for structural prediction tasks such as semantic segmentation. This is because the structural prediction task does not directly model the interaction between output variables.

  Various semantic segmentation methods use discrete conditional random fields (CRFs) on CNN. By combining CNN and CRF, these methods provide the ability of CNN to model complex input-output relationships and the ability of CRF to directly model interactions between output variables. Most of these methods use CRF as a separate post-processing step. Typically, the CNN processes the image to generate unary energy, which is then processed by the CRF to label the image. However, CRF has an operating principle different from CNN. That decouples the CNN from the CRF and prevents their joint training. In general, CRF is either manually adjusted or trained separately from CNN.

  One alternative to using CRF as a post-processing step is to train CNN with discrete CRF by transforming the discrete CRF estimation procedure into a recurrent neural network. However, in general, estimation in discrete CRF is cumbersome due to the discrete and non-differentiable nature of CRF formulation. To that end, the method uses an approximate estimation procedure that does not have a global optimum guarantee and can lead to poor training results.

  Some embodiments of the present invention are based on the recognition that it is advantageous to provide semantic segmentation of images using a combination of convolutional neural networks (CNN) and discrete conditional random fields (CRF). On the other hand, some embodiments are based on yet another recognition that it is advantageous to replace CRF with a neural network (NN) in this combination. Such permutations can connect the various sub-networks participating in semantic segmentation into a common neural network that can be jointly trained. However, emulating the operation of CRF with NN is difficult due to the discrete and non-differentiable nature of the CRF formulation.

  Some embodiments are based initially on the recognition that CRF can be replaced by a Gaussian random field (GRF), which is a subclass of CRF. The operations of GRF estimation are continuous and differentiable, and can be solved optimally. Although image segmentation is a discrete task, GRF is still suitable for semantic segmentation.

  Some embodiments are based on the recognition that neural networks can be used to emulate the operation of GRF estimation. Since both neuron operations and GRF operations are continuous and differentiable, the continuity of the operations of GRF makes it possible to replace each algebraic operation in GRF with several neuron operations. These neuron operations are applied sequentially as their algebraic operations applied during GRF estimation.

  To that end, the embodiment creates a first sub-network for unary energy, a second sub-network for pairwise energy, and a third sub-network emulating GRF estimation. And jointly train all three sub-networks.

  Thus, one embodiment of the present invention discloses a computer-implemented method for semantic segmentation of images. The method uses a first subnetwork to determine the unary energy of each pixel in the image and a second subnetwork to determine the pairwise energy of at least some pairs of pixels of the image. And the meaning of each pixel in the image using a third subnetwork to obtain an estimation result on a Gaussian random field (GRF) minimizing an energy function including a combination of the unary energy and the pairwise energy Generating a GRF estimation result defining the probability of the label, and having pixels in the semantic segmented image with the highest probability of the corresponding pixel in the image among the probabilities determined by the third subnetwork By assigning a semantic label, It comprises, converting the image into the semantic segmented image, the first sub-network, the second sub-network, and the third sub-network is part of a neural network. The steps of the method are performed by a processor.

  Yet another embodiment is a system for semantic segmentation of an image, using at least one non-temporary computer readable memory for storing the image and semantically segmented image, and a Gaussian random field (GRF) network. A processor for performing semantic segmentation of the image to generate the semantically segmented image, the GRF network comprising: a first sub-network for determining the unary energy of each pixel in the image; A second sub-network for determining pairwise energy of at least some pairs of pixels, and estimation results on Gaussian random field (GRF) minimizing an energy function including a combination of the unary energy and the pairwise energy And a third sub-network for generating a GRF estimation result defining the probability of the semantic label of each pixel in the image, and the processor is further configured to: A system is disclosed that converts the image into the semantic segmented image by assigning a semantic label having the highest probability of the corresponding pixel in the image among the probabilities determined by the third subnetwork.

  Yet another embodiment is a non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by the processor, use a first subnetwork to generate a unique energy for each pixel in the image. Determining the pairwise energy of at least some pairs of pixels of the image using a second subnetwork, and using the third subnetwork to calculate the unary energy and the pairwise energy. Determining an estimation result on a Gaussian random field (GRF) minimizing energy functions including combinations, generating a GRF estimation result defining the probability of the semantic label of each pixel in the image, and in the semantic segmented image Determined by the third sub-network And D. converting the image into the semantic segmented image by assigning the semantic label having the highest probability of the corresponding pixel in the image among the probabilities. The subnetwork, the second subnetwork, and the third subnetwork disclose a non-transitory computer readable medium jointly trained as part of a neural network.

FIG. 7 is a block diagram of a computer system for semantic segmentation of images according to some embodiments of the present invention. FIG. 5 is a schematic diagram of semantic segmentation via image labeling using a Gaussian Random Field (GRF) neural network according to some embodiments of the present invention. FIG. 6 is a block diagram of a computer-implemented method for semantic labeling of images according to one embodiment of the present invention. FIG. 1 is a block diagram of a GRF network according to one embodiment of the present invention. FIG. 5 is a schematic view of the minimization of the energy function according to some embodiments of the invention. FIG. 1 is a block diagram of a GRF network according to one embodiment of the present invention. 7 is pseudo code of an implementation of a GRF network according to one embodiment of the invention. FIG. 5 is a block diagram of a method of forming a pair of pixels for pairwise energy, according to one embodiment of the present invention. FIG. 4B is a block diagram of a network utilizing the bipartite graph structure of FIG. 4A according to some embodiments of the present invention. FIG. 7 is a schematic view of a training method used by some embodiments of the present invention. FIG. 6 is a block diagram of a training method used by some embodiments of the present invention. FIG. 1 is a block diagram of a training system according to one embodiment of the present invention.

  FIG. 1A shows a block diagram of a computer system 100 for semantic segmentation of images according to some embodiments of the present invention. Computer system 100 comprises a processor 102 configured to execute stored instructions, and a memory 104 that stores instructions executable by the processor. Processor 102 may be a single core processor, a multi-core processor, a computing cluster, or any number of other configurations. Memory 104 may include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. Processor 102 is connected to one or more input and output devices through bus 106.

  FIG. 1B shows a schematic of semantic segmentation via image labeling using a Gaussian random field (GRF) neural network according to some embodiments of the present invention. Semantic segmentation may be performed by processor 102 executing instructions stored in memory 104. The GRF network 114 performs semantic labeling of the image 160 to produce a segmented image 170 having pixels labeled with semantic classes, eg, semantic labels 171, 172, and 173. GRF network 114 is a neural network, and at least some operations of GRF network 114 emulate operations of GRF estimation.

  GRF is a random field with a Gaussian distribution of variables and / or Gaussian probability density functions. One-dimensional GRF is also called Gaussian process. For example, the GRF network 114 models the probability density of possible semantic labels 171, 172, and 173 conditional on the value of each pixel of the image 160 as a Gaussian distribution of energy functions including unary energy and pairwise energy, the energy Perform Gaussian estimation on the function to determine the probability of each semantic label of each pixel of the image.

  In general, Gaussian estimation refers to determining properties (eg, mean or covariance) of the underlying Gaussian distribution. In this case, this Gaussian distribution is formed by statistical variables which define the probability that the pixels of the image belong to different semantic classes. Thus, the unary energy and the pairwise energy are functions of the probability of the semantic label of the pixel. For example, in some embodiments, Gaussian estimation determines the mean of Gaussian distributions defined using unary energy and pairwise energy.

  Some embodiments are based initially on the recognition that CRF can be replaced by GRF, which is a subclass of CRF. The operations of GRF estimation are continuous and differentiable, and can be solved optimally. Although semantic segmentation of images is a discrete task, GRF is still suitable for semantic segmentation.

  Computer system 100 may also include a storage device 108 adapted to store the original image 110, and a filter 112 that filters the original image to produce an image 160 suitable for segmentation. For example, the filter may resize the original image to align with the training data image. Storage device 108 may also store the structure and parameters of GRF network 114. In various embodiments, the GRF network 114 is trained on a set of training images and a set of corresponding training semantic labels.

  Storage device 108 may include a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. A human machine interface 116 within computer system 100 can connect the system to keyboard 118 and pointing device 120, which, among other things, is a mouse, trackball, touch pad, joystick, pointing stick, stylus, or touch It can contain a screen. Computer system 100 may be linked through bus 106 to display interface 122 adapted to connect system 100 to display device 124, which may be, inter alia, a computer monitor, a camera, a television, a projector, or It can include mobile devices.

  Computer system 100 may also be connected to imaging interface 126 adapted to connect the system to imaging device 128. In one embodiment, an image for semantic segmentation is received from the imaging device. Imaging device 128 may include a camera, a computer, a scanner, a mobile device, a webcam, or any combination thereof. The printer interface 130 may also be connected to the computer system 100 through the bus 106 and may be adapted to connect the computer system 100 to the printing device 132, which may be, inter alia, a liquid ink jet printer, a solid ink printer , Large scale commercial printers, thermal printers, UV printers, or sublimation printers. Network interface controller 134 is adapted to connect computer system 100 to network 136 through bus 106. Through the network 136, the image 138, including one or a combination of electronic text and imaged input documents, may be downloaded and stored in the computer storage system 108 for storage and / or further processing.

及び

は、行列

の転置行列及び逆行列を示す。表記

は、ベクトル

の二乗

ノルムを示す。

は、

が対称半正定値行列（symmetric and positive semidefinite matrix）であることを意味する。 For ease of explanation, the disclosure uses boldface lowercase letters to indicate vectors and boldface capital letters to indicate matrices.

as well as

Is the matrix

Shows the transposed matrix and the inverse matrix of. Notation

Is the vector

Square of

Indicates the norm.

Is

It means that is a symmetric and positive semidefinite matrix.

  A neural network is a family of models inspired by biological neural networks and is used to estimate or approximate functions that may depend on multiple inputs and are generally unknown. Neural networks are generally provided as a system of interconnected nodes or "neurons" that exchange messages between one another. Each node is associated with a function that translates messages. This function is usually non-linear to form the non-linear part of message conversion. Each connection between nodes is associated with a numerical weight that scales the message to form a linear part of the message transformation. Usually, these functions are fixed for all nodes and are predetermined, for example, selected by the designer of the neural network. Examples of functions that are usually selected for nodes include sigmoid functions and rectification functions. In contrast, the numerical weights are different and are adjusted based on the training data, making the neural network adaptive and learnable to the input.

  Some embodiments are based on the recognition that neural networks can be used to emulate the operation of GRF estimation. Since both neuron operations and GRF operations are continuous and differentiable, the continuity of the operations of GRF makes it possible to replace each algebraic operation in GRF with several neuron operations. These neuron operations are applied sequentially as their algebraic operations applied during GRF estimation.

１６０における各ピクセルを、画像１７０におけるＫ個の可能なクラスのうちの１つに割り当てる。そのような割り当ては、本明細書では、意味的ラベル付けと呼ばれる。意味的ラベル付けが行われた後、ピクセルの意味的ラベル付けの結果は、画像のセマンティックセグメンテーションを生成する。幾つかの実施形態は、Ｋ個の変数（各クラスにつき１つ）を用いて、各ピクセルにおける出力をモデル化し、最終ラベル割り当ては、これらのＫ個の変数のうちのいずれが最大値、例えば、確率の値を有するのかに基づいて行われる。第ｉのピクセルに関連付けられたＫ個の出力変数のベクトルを

とし、全ての出力変数のベクトルを

とする。例えば、条件付き確率密度

は、以下の式によって与えられるガウス分布としてモデル化することができる。

Semantic segmentation is an image

Each pixel at 160 is assigned to one of K possible classes in image 170. Such assignment is referred to herein as semantic labeling. After semantic labeling has been performed, the result of semantic labeling of the pixels produces a semantic segmentation of the image. Some embodiments model the output at each pixel using K variables (one for each class), and the final label assignment is such that any of these K variables is a maximum, eg , Is performed based on whether it has a probability value. A vector of K output variables associated with the i th pixel

And the vector of all output variables

I assume. For example, conditional probability density

Can be modeled as a Gaussian distribution given by:

及び第１のピクセルｉと第２のピクセルｊとの間のペアワイズエネルギーパラメーター

の双方は、θ_ｕ及びθ_ｐがそれぞれの関数パラメーターである入力画像

の関数を用いて計算される。ピクセルの全てのペアについて

を有する実施形態では、ユーナリ項及びペアワイズ項を互いに組み合わせて、単一の半正定値二次形式にすることができる。 The first term in the energy function E is a unary term representing unary energy, and the second term is a pairwise term representing pairwise energy. Where the unary energy parameter of each pixel i

And pairwise energy parameters between the first pixel i and the second pixel j

Both have an input image where θ _u and θ _p are the respective function parameters

Calculated using the function of For every pair of pixels

In an embodiment having a Eunary term and a Pairwise term can be combined with one another into a single semidefinite quadratic form.

  FIG. 1C shows a block diagram of a method of semantic labeling of an image according to one embodiment of the invention. The method may be performed by the GRF network 114 executed by the processor 102. The method determines (180) the unary energy 185 of each pixel in the image and determines (190) the pairwise energy 195 of at least some pairs of pixels of the image. Next, the method determines 175 a GRF estimate 176 of the image by processing the unary energy 185 and the pairwise energy 195. For example, in some embodiments, GRF estimation is determined by minimizing an energy function that includes a combination of unary energy and pairwise energy.

  In various embodiments, the unary energy 185 is determined using the first subnetwork 180 and the pairwise energy 195 is determined using the second subnetwork 190 and the GRF estimate 176 is It is determined 175 using the third subnetwork. The first subnetwork, the second subnetwork, and the third subnetwork are parts of a neural network. In such a way, all the parameters of the neural network can be trained jointly.

  GRF estimation defines the probability of semantic labels for each pixel in the image. For example, in some embodiments of the present invention, the unary energy 185 is a first function of the probability of the semantic label of the pixel determined using the first subnetwork, and the pairwise energy 195 is a second It is a second function of the probability of the semantic label of the pixel determined using the subnetwork. To that end, the method assigns 196 the pixels in the semantic segmented image 170 the semantic label with the highest probability of the corresponding pixel in the image 160 among the probabilities determined by the third subnetwork (196). , Image 160 is transformed into semantically segmented image 170. Here, the first subnetwork, the second subnetwork.

は、閉形式で取得することができる。なぜならば、Ｅの最小化は制約なし２次計画法であるからである。しかしながら、この閉形式解は、クラスの数にピクセルの数を乗算したものに等しい数の変数を有する線形システムを解くことを必要とする。幾つかの実施形態は、そのような大規模な線形システムを解くことは計算上法外であり得るという認識に基づいている。それらの実施形態では、第３のサブネットワークは、ガウス平均場（ＧＭＩ(Gaussian mean field)）推定の演算をエミュレートすることによってＧＲＦ推定を求める。 Optimal semantic label to minimize energy function E

Can be obtained in closed form. This is because the minimization of E is an unconstrained quadratic programming. However, this closed-form solution requires solving a linear system with a number of variables equal to the number of classes multiplied by the number of pixels. Some embodiments are based on the recognition that solving such large linear systems can be computationally prohibitive. In those embodiments, the third subnetwork determines the GRF estimate by emulating the operation of a Gaussian mean field (GMI) estimate.

及び

を生成する一方、ＧＭＩネットワークは、ユーナリネットワーク及びペアワイズネットワークの出力を用いてガウス平均場推定を実行する。 FIG. 2A shows a block diagram of a GRF network according to one embodiment of the present invention. In this embodiment, the GRF network is trained as three sub-networks: a first sub-network trained as the unary network 201 seeking unary energy 185 and a pair-wise network 202 seeking the pairwise energy 195 2 sub-network and the third sub-network which is the GMI network 203 which seeks the mean field estimation update which minimizes the energy function. The unary network and the pairwise network are parameters used in the unary term and the pairwise term of the energy function equation (1), respectively.

as well as

The GMI network performs Gaussian mean field estimation using the output of the unary network and the pairwise network.

を計算する平均場更新は、以下の式によって与えられる。

In one embodiment, the average

The mean field update to calculate is given by:

（又は

）が固定されているときの最適な

（又は

）を見つける。そのために、逐次的な更新を実行して最大事後確率（ＭＡＰ(maximum a posteriori)）解を与えることが保証される。 Here, these updates are performed sequentially for each pixel i. The energy function is convex quadratic in the case of GRF, and the update of equation (2) optimally solves each sub-problem. All other

(Or

Optimal when) is fixed

(Or

Find out). To that end, it is guaranteed that successive updates are performed to give a maximum a posteriori (MAP) solution.

  FIG. 2B shows a schematic diagram of the minimization of the energy function containing the NN according to some embodiments of the present invention. The energy function 210 comprises a combination of unary energy 185 and pairwise energy 195. An example of the energy function is the function of equation (1). Each layer 231, 232, 233, 234, 235, and 236 of the third subnetwork 203 recursively determines the mean field estimate update that minimizes the energy function 210. An example of recursive minimization is provided in equation (2). The number of layers in subnetwork 203 can be selected based on the desired number of iterations of the update.

を有するユーナリＣＮＮ３０５と本明細書では呼ばれる畳み込みＮＮ（ＣＮＮ）である。ユーナリＣＮＮは、画像１６０の各ピクセルについて、そのピクセルの近傍にあり、かつ、そのピクセルが各可能な意味ラベルに属する確率を生成するピクセルのサブセットを入力として受信する。例えば、このサブセットのピクセルは、そのピクセルを中心とする矩形パッチのピクセルとすることができる。 FIG. 3A shows a block diagram of a GRF network according to one embodiment of the present invention. In this embodiment, the first subnetwork 201 has parameters

Is a convolutional NN (CNN), referred to herein as the Unary CNN 305. For each pixel of the image 160, the unary CNN receives as input a subset of pixels that are close to that pixel and that generate the probability that the pixel belongs to each possible semantic label. For example, the subset of pixels may be pixels of a rectangular patch centered on that pixel.

３０６は、ピクセルの近傍にあるピクセルのサブセットの関数を用いて計算され、式（１）のエネルギー関数のユーナリ項において用いられる。例えば、ユーナリエネルギー関数は、二次関数

である。ここで、

は、ユーナリＣＮＮを通じて計算されるユーナリエネルギーパラメーターであり、θ_ｕは、線形フィルターのパラメーターであり、

は、意味ラベルの確率であり、ｉは、ピクセルのインデックスである。ユーナリＣＮＮは、畳み込み演算を実行する一連の線形フィルターを各層への入力に適用し、少なくとも幾つかの層において、各線形フィルターの出力の非線形関数を適用する。 In this embodiment, the unary energy parameters

306 is calculated using a function of the subset of pixels in the vicinity of the pixel and is used in the unary term of the energy function of equation (1). For example, the unary energy function is a quadratic function

It is. here,

Is the unary energy parameter calculated through the unary CNN, and θ _u is the parameter of the linear filter,

Is the probability of the semantic label, i is the index of the pixel. Unary CNN applies a series of linear filters that perform convolution operations to the input to each layer, and applies a non-linear function of the output of each linear filter in at least some layers.

  For example, in one embodiment, Unary CNN 305 is a modified version of the Oxford Visual Geometry Group (VGG-16) network. Changes compared to VGG-16 include converting the fully connected layer to the convolutional layer, skipping the downsampling layer, and, for example, modifying the convolutional layer after the fourth pooling layer to downsample Compensation of the loss of visual field due to skipping and using multi-scale features.

３１０を求めるパラメーター

を有するペアワイズＣＮＮ３０１を備える。例えば、ペアワイズネットワーク２０２は、ペアワイズＣＮＮ３０１を用いてペアのピクセル間の類似度を求め、この類似度に基づいて共分散行列を求め、この共分散行列の関数としてペアワイズエネルギーを求める。 The second subnetwork (ie, pairwise network) 202 is a matrix used in the pairwise terms of the energy function of equation (1)

Parameter for finding 310

And a pairwise CNN 301. For example, the pairwise network 202 determines the similarity between the pixels of the pair using the pairwise CNN 301, determines the covariance matrix based on the similarity, and determines the pairwise energy as a function of the covariance matrix.

を生成する（３０２）とともに、ペアの第２のピクセルｊの近傍の第２のピクセルのサブセットを処理して、第２のピクセルの特徴量

を生成する（３０２）。ペアワイズネットワーク２０２は、第１の特徴量と第２の特徴量との間の差の関数を求めて類似度ｓ_ｉｊを生成し（３０３）、ペアワイズエネルギーを共分散行列

として以下の式に従って求める（３０４）。

ここで、ｓ_ｉｊ∈［０，１］は、ピクセルｉとピクセルｊとの間の類似度であり、学習された行列

は、クラス適合性情報（class compatibility information）を符号化する。類似度ｓ_ｉｊは、以下の式に従って求めることができる（３０３）。

ここで、

（３０２）は、ペアワイズＣＮＮ３０１を用いて第ｉのピクセルにおいて抽出された特徴量ベクトルであり、学習された行列

は、距離関数、例えばマハラノビス（Mahalanobis）距離関数を規定する。 For example, the pairwise network 202 processes the subset of the first pixels in the vicinity of the first pixel i of the pair, and the feature value of the first pixel

Processing the second subset of pixels in the vicinity of the second pixel j of the pair together with generating 302

Are generated (302). The pairwise network 202 obtains a function of the difference between the first feature amount and the second feature amount to generate the similarity s _ij (303), and the pairwise energy is covariance matrix

It calculates | requires according to the following formula as (304).

Where s _ij ∈ [0, 1] is the similarity between pixel i and pixel j, and the learned matrix

Encodes class compatibility information. The similarity s _ij can be determined according to the following equation (303).

here,

(302) is a feature quantity vector extracted at the ith pixel using the pairwise CNN 301, and the learned matrix

Defines a distance function, for example a Mahalanobis distance function.

ここで、

である。この実施形態では、マハラノビス距離計算は、

とフィルター

との畳み込み及びその後に続くユークリッド距離計算として実施される。 The structure of the pairwise CNN can be the same as the unary CNN. In some embodiments, the index of s _ij is

here,

It is. In this embodiment, the Mahalanobis distance calculation

And filters

And the Euclidean distance calculation that follows.

を生成するペアワイズＣＮＮと、接続されたピクセルのあらゆるペアのｓ_ｉｊを式（４）及び／又は式（５）を用いて計算する類似層３０３と、行列

を式（３）を用いて計算する行列生成層３０４とを備える。この実施形態では、

は、類似層３０３のパラメーターであり、

は、行列生成層３０４のパラメーターである。 In one embodiment, the pairwise network 202 comprises pixel features

A pairwise CNN that generates H, and a similar layer 303 that calculates s _ij of every pair of connected pixels using equation (4) and / or equation (5), and a matrix

And a matrix generation layer 304 for calculating In this embodiment

Is a parameter of the similar layer 303,

Is a parameter of the matrix generation layer 304.

とする。その場合、第ｉのピクセルの意味ラベルは、

３０８によって与えられる。 The GMI 203 repeatedly finds the probability of the semantic label of each pixel such that the energy function including the combination of the unary energy and the pairwise energy is minimized. The final output at each pixel is the K-dimensional class prediction score vector 307. Here, K is the number of classes. The final output at the ith pixel

I assume. In that case, the meaning label of the ith pixel is

Given by 308.

  FIG. 3B is pseudo code of an implementation of a GRF network according to one embodiment of the present invention.

  FIG. 4A shows a block diagram of a method of forming pairs of pixels of image 160 for determining pairwise energy according to one embodiment of the invention. This embodiment is based on the understanding that determining the pairwise energy of all possible pairs of pixels in the image 160 slows the computation due to a large number of variables. Using parallel updates of all pixels simultaneously seems to be a reasonable alternative, but the convergence of parallel updates is only guaranteed under limited conditions.

  To address this problem, embodiments use a bipartite graph structure. This bipartite graph structure allows half of the variables to be updated in parallel at each step, while still ensuring convergence without diagonal dominance constraints. For example, the embodiment divides 420 pixels in the image 160 into odd pixels or even pixels based on the parity of the index of the column or row of pixels (420), and in each pair of pixels, the first pixel is an odd pixel , 430 forming a pair of pixels such that the second pixel is an even pixel. For example, pixel 410 is only paired with pixels in the 7x7 spatial neighborhood shown with larger black circles such as pixels 411, 412, 413 and 414.

を有する全てのノード）を固定した状態にしておく。画像を奇数列及び偶数列（又は奇数行及び偶数行）に分割するとともに、分割した部分内のエッジを回避することによって、偶数列（又は偶数行）を固定した状態のままで全ての奇数列（又は奇数行）を、式（２）を用いて並列に更新することが可能になり、また、その逆も可能になる。この交互の最小化を最適に解いて、大域的最適に収束することができる。 In some implementations, the graphical model has nodes for each pixel, and each node represents a vector of K variables. In order to update the i-th node using equation (2), the embodiment includes all other nodes connected to the i-th node (ie non-zero)

All nodes that have) are fixed. By dividing the image into odd and even columns (or odd and even rows) and avoiding the edges in the divided part, all odd columns remain with the even columns (or even rows) fixed. (Or odd rows) can be updated in parallel using equation (2), and vice versa. This alternating minimization can be solved optimally to converge on a global optimum.

を用いて初期化される。 FIG. 4B shows a block diagram of a GMI network 440 that utilizes the bipartite graph structure of FIG. 4A according to some embodiments of the present invention. GMI network 440 performs a fixed number of Gaussian mean field updates using the outputs of the unary and pairwise networks. The input to this network is a unary output

Initialized using.

  The GMI network 440 comprises several GMI layers 401 combined in series. Each layer has two sublayers, ie, an even update sublayer 402 and an odd update sublayer 403 following or preceding it. The even update sublayer 402 takes the output of the previous layer as input and updates the even pixel nodes using equation (2) with the odd pixel nodes fixed. Similarly, the odd update sublayer takes the output of the even update sublayer as an input and updates the odd pixel nodes using equation (2) while keeping the even pixel nodes fixed. The order of the odd update sublayer and the even update sublayer can be reversed.

  Due to the bipartite graph structure, the updates performed by each of the sublayers may be optimal updates. Thus, each layer of our GMI network is guaranteed to produce an output that is closer to the MAP solution compared to its input (if the input itself is not a MAP solution and the input itself is a MAP solution) In which case the output is equal to the input).

及びペアワイズエネルギーパラメーター

を生成するように学習する。 Training Since the GRF network 114 comprises interconnected sub-networks, these various sub-networks of the GRF network 114 can be trained jointly. For example, the combination of the unary network, the pairwise network, and the GMI network of FIG. 3A can be trained in an end-to-end fashion. One embodiment uses a fixed number of layers in the GMI network. Because of the finite number of layers, the output of the GRF network can potentially be suboptimal. On the other hand, since the embodiment differentially trains the entire GRF network in an end-to-end manner, the unary network and the pairwise network are close to each other so that the output after the fixed number of mean field updating approaches the optimum solution. Nari energy parameter

And pairwise energy parameters

Learn to generate

と、ペアワイズネットワークパラメーター

とを含むことができる。 FIG. 5 shows a schematic view of the training used by some embodiments of the present invention. The training 510 uses the training set 501 of the image pair and the corresponding semantic segmented image 502 to generate the parameters 520 of the GRF network. In general, training an artificial neural network involves applying to the artificial neural network a training algorithm, sometimes referred to as a "learning" algorithm, taking into account the training set. The training set may include one or more sets of inputs and one or more sets of outputs, each set of inputs corresponding to one set of outputs. The set of outputs in the training set includes the set of outputs that the artificial neural network is desired to generate when the corresponding set of inputs is input to the artificial neural network and the artificial neural network is subsequently operated in a feedforward manner. . Training a neural network involves calculating parameters, eg, weight values associated with connections in the artificial neural network. For example, GRF network parameters are unique network parameters

And pairwise network parameters

And can be included.

ここで、ｌ_ｉは、距離６４０としての真のクラスラベルである。この損失関数は、基本的に、真のクラスに関連付けられた出力をマージンＴによって他の全てのクラスに関連付けられた出力よりも大きくなるように促進する。 FIG. 6 shows a block diagram of a training method 510 used by some embodiments of the present invention. The method processes the image 610 from the set 501 using the GRF network 114 to generate a semantically segmented image 630, which is a corresponding semantically segmented image from the set 502. The distance between these two semantically segmented images is generated (640) as compared to 620. For example, one embodiment determines the following loss function at each pixel:

Where l _i is the true class label as distance 640. This loss function basically encourages the output associated with the true class to be greater by margin T than the output associated with all other classes.

に対する対称半正定値性制約に起因した制約付き最適化を含むことができる。１つの実施形態は、

を

としてパラメーター化することによってこの制約付き最適化を制約なし最適化に変換し、確率的勾配降下法を最適化に用いる。ここで、

は、下三角行列である。 To that end, embodiments differentially train the GRF network 114 by minimizing the loss function. For example, training is performed using backpropagation to calculate the slope of the network parameters. Training is a parameter

Can include constrained optimization due to symmetric semidefinite constraints on. One embodiment is

The

Convert this constrained optimization into an unconstrained optimization by parameterizing as and use stochastic gradient descent for optimization. here,

Is a lower triangular matrix.

  FIG. 7 shows a block diagram of a training system according to one embodiment of the present invention. The training system comprises a processor connected by a bus 22 to a read only memory (ROM) 24 and a memory 38. The training system may also include a display 28 for presenting information to the user, and a plurality of input devices including a keyboard 26, a mouse 34 and other devices that may be attached via the input / output port 30. Other pointing devices or other input devices such as voice or image sensors may also be attached. Other pointing devices include tablets, numeric keypads, touch screens, touch screen overlays, trackballs, joysticks, light pens, thumbwheels and the like. The I / O 30 can be connected to communication lines, disk storage, input devices, output devices or other I / O devices. The memory 38 comprises a display buffer 72 containing pixel intensity values of the display screen. The display 28 periodically reads the pixel values from the display buffer 72 and displays these values on the display screen. The pixel intensity values can represent gray levels or colors.

  The memory 38 includes a database 90, a trainer 82, a GRF 114, and a pre-processor 84. The database 90 can include historical data 105, training data, and test data 92. The database can also include results from an operating mode using a neural network, a training mode or a holding mode. These elements are described in detail above.

  Also shown in memory 38 is operating system 74. Examples of operating systems include AIX, OS / 2, and DOS. Other elements shown in memory 38 include a device driver 76 that interprets the electrical signals generated by devices such as a keyboard and mouse. A working memory area 78 is also shown in the memory 38. The working memory area 78 can be used by any of the elements shown in the memory 38. The working memory area may be utilized by neural network 101, trainer 82, operating system 74 and other functions. The working memory area 78 can be divided between elements or can be divided within an element. The working memory area 78 can be used for communication, buffering, temporary storage, or storage of data while the program is being executed.

  The above-described embodiments of the present invention can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software or a combination thereof. When implemented in software, the software codes may be executed on any suitable processor or collection of processors, whether provided on a single computer or distributed among multiple computers. Such processors can be implemented as integrated circuits having one or more processors in integrated circuit components. However, a processor may be implemented using circuitry of any suitable format.

  Also, embodiments of the present invention can be implemented as a method provided an example. The operations performed as part of this method can be ordered in any suitable manner. Thus, embodiments can be constructed in which the operations are performed in a different order than that shown, and although this is illustrated as a series of operations in the illustrated embodiment, several operations may be performed simultaneously. It can also include doing.

  The use of ordinal numbers such as “first”, “second” and the like in a claim to modify elements of the claims themselves is also a priority to elements of another claim of one claim In order to distinguish between the elements of the claims, it is merely understood that the advantages, the ordering, the ordering and the implicit order of execution, and neither the temporal ordering in which the acts of the method are performed nor implied. The elements of one claim are merely used as labels to distinguish them from other elements having the same (except for the use of ordinal terms) names.

Claims (18)

A computer-implemented method for semantic segmentation of an image, comprising
Determining the unary energy of each pixel in the image using the first subnetwork;
Determining the pairwise energy of at least some pairs of pixels of the image using a second subnetwork;
The probability of the semantic label of each pixel in the image is determined using the third subnetwork to obtain an estimation result on a Gaussian random field (GRF) minimizing an energy function including a combination of the unary energy and the pairwise energy. Generating a GRF estimation result defining
The semantic segmentation of the image by assigning to the pixels in the semantic segmentation image the semantic label having the highest probability of the corresponding pixel in the image among the probabilities determined by the third subnetwork. Converting into an image, the first subnetwork, the second subnetwork, and the third subnetwork being part of a neural network.
The step of determining the pairwise energy of pixel pairs of the image is:
Determining the similarity between the pixels of the pair in the image;
Determining a covariance matrix based on the degree of similarity;
Determining the pairwise energy as a function of the covariance matrix
Including
Each said step of the method is performed by a processor. Rendering said semantically segmented image in non-transitory computer readable memory;
The method of claim 1, further comprising   The third subnetwork is a Gaussian mean field such that each layer of the third subnetwork recursively finds a mean field estimate update that minimizes an energy function including a combination of the unary energy and the pairwise energy The method according to claim 1, wherein the GRF estimation result is obtained by emulating an operation of (GMI) estimation.   For each pixel in the image, the first sub-network receives as input a subset of pixels near the pixel in the image and generates a unique energy parameter of the pixel, the unique energy being The method according to claim 1, which is a function of the unary energy parameter of each pixel in the image and the probability of each pixel in the image belonging to each possible semantic label. Applying a series of linear filters to perform convolution operations on the inputs to each layer of the first subnetwork;
5. The method according to claim 4, further comprising: applying a non-linear function for the output of each linear filter in several layers of the first subnetwork. The unary energy function is a quadratic function

And here

Is the unary energy parameter calculated through the first subnetwork, θ _u is the parameter of the linear filter,

6. The method of claim 5, wherein is the probability of the semantic label and i is an index of the pixel.   5. The method of claim 4, wherein the subset of pixels is a rectangular patch centered on the pixels in the image. The determination of the similarity is
The second sub-network is used to process a subset of first pixels in the vicinity of the first pixel i of the pair to characterize the first pixels

To generate
The second sub-network is used to process a subset of second pixels in the vicinity of the second pixel j of the pair to characterize the second pixels.

To generate
The method according to claim 1 , comprising determining the function of the difference between the first feature and the second feature to generate the similarity s _ij . Dividing the pixels in the image into odd pixels or even pixels based on the parity of the index of the column or row of pixels in the image;
In each pair of the pixels, the first pixel is said odd pixels, as the second pixel is in the even pixels, further including forming a pair of said pixel to claim 8 Method described.   The method according to claim 1, wherein the first subnetwork, the second subnetwork, and the third subnetwork are jointly trained.   The first subnetwork, the second subnetwork, and the third subnetwork are jointly trained to minimize the loss function of the set of training images and the corresponding set of training semantic labels. The method of claim 1. A system for semantic segmentation of images,
At least one non-transitory computer readable memory for storing the image and semantically segmented image;
A processor that performs semantic segmentation of the image using a Gaussian random field (GRF) network to generate the semantic segmented image;
Equipped with
The GRF network is
A first subnetwork for determining the unary energy of each pixel in the image;
A second sub-network for determining pairwise energy of at least some pairs of pixels of the image;
An estimation result on a Gaussian random field (GRF) minimizing an energy function including a combination of the unary energy and the pairwise energy is obtained, and a GRF estimation result defining the probability of the semantic label of each pixel in the image is generated. The third subnetwork,
A neural network comprising
The processor assigns the image to the pixels in the semantic segmented image by assigning a semantic label having the highest probability of the corresponding pixel in the image among the probabilities determined by the third subnetwork. Convert to semantically segmented images,
The second subnetwork is
Determine the similarity between the pixels of the pair in the image;
Determining a covariance matrix based on the degree of similarity;
Determine the pairwise energy as a function of the covariance matrix,
system. The third subnetwork is a Gaussian mean field such that each layer of the third subnetwork recursively finds a mean field estimate update that minimizes an energy function including a combination of the unary energy and the pairwise energy Determine the GRF estimation result by emulating the operation of the (GMI) estimation,
A system according to claim 12 . For each pixel in the image, the first sub-network receives as input a subset of pixels near the pixel in the image and generates a unique energy parameter of the pixel, the unique energy being The system according to claim 12 , wherein the system is a function of the unary energy parameter of each pixel in the image and the probability of each pixel in the image belonging to each possible semantic label. The second subnetwork is
Processing a subset of the first pixels in the vicinity of the first pixel i of the pair to obtain the feature quantities of the first pixels

To generate
Processing a second subset of pixels in the vicinity of the second pixel j of the pair, and processing the feature quantity of the second pixel

To generate
The system according to claim 12 , wherein the similarity is determined by determining a function of a difference between the first feature and the second feature to generate the similarity s _ij . The processor is
Dividing the pixels in the image into odd pixels or even pixels based on the parity of the index of the column or row of pixels in the image;
The system of claim 12 , wherein in each pair of pixels, the pair of pixels is formed such that the first pixel is the odd pixel and the second pixel is the even pixel. The system according to claim 12 , wherein the first subnetwork, the second subnetwork, and the third subnetwork are jointly trained. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions are executed by a processor
Determining the unary energy of each pixel in the image using the first subnetwork;
Determining the pairwise energy of at least some pairs of pixels of the image using a second subnetwork;
The probability of the semantic label of each pixel in the image is determined using the third subnetwork to obtain an estimation result on a Gaussian random field (GRF) minimizing an energy function including a combination of the unary energy and the pairwise energy. Generating a GRF estimation result defining
The semantic segmentation of the image by assigning to the pixels in the semantic segmentation image the semantic label having the highest probability of the corresponding pixel in the image among the probabilities determined by the third subnetwork. Perform the steps of converting to an image,
The first subnetwork, the second subnetwork, and the third subnetwork are jointly trained as part of a neural network ,
The step of determining the pairwise energy of pixel pairs of the image is:
Determining the similarity between the pixels of the pair in the image;
Determining a covariance matrix based on the degree of similarity;
Determining the pairwise energy as a function of the covariance matrix
including,
Non-transitory computer readable medium.

Images