KR102235745B1 - Method for training a convolutional recurrent neural network and for semantic segmentation of inputted video using the trained convolutional recurrent neural network - Google Patents

Abstract

)의 각 쌍에 대해, 연속하는 프레임들의 쌍의 프레임들 간의 추정된 광학 흐름(optical flow)에 따라 순환 레이어의 내부 상태를, 상기 내부 상태가 상기 쌍의 프레임들 간의 픽셀들의 모션에 적응하도록(adapt), 와핑하는 단계 및 적어도 상기 순환 모듈의 파라미터들을 학습하는 단계를 포함한다. A method of training a convolutional recurrent neural network for semantic segmentation of videos includes: (a) training a first convolutional neural network using a set of semantically segmented training images; And (b) training a convolutional recurrent neural network, corresponding to the first convolutional neural network, using a set of semantically segmented training videos.- The convolutional layer has a hidden state. Replaced by module-includes. Training the convolutional recurrent neural network includes consecutive frames of one video of the set of semantically segmented training videos (

For each pair of ), the internal state of the cyclic layer is adapted according to the estimated optical flow between the frames of the pair of consecutive frames, so that the internal state adapts to the motion of the pixels between the pair of frames ( adapt), warping and learning at least the parameters of the cyclic module.

Description

METHOD FOR TRAINING A CONVOLUTIONAL RECURRENT NEURAL NETWORK AND FOR SEMANTIC SEGMENTATION OF INPUTTED VIDEO USING THE TRAINED CONVOLUTIONAL RECURRENT NEURAL NETWORK }

It relates to a method of training a convolutional recurrent neural network and a method of semantic segmentation of an input video using the trained convolutional recurrent neural network.

By 35 USC§ 119 (a), the claims herein claim the benefit of the earlier filing date priority of European patent application EP 18306104.3 filed on August 10, 2018, thereby The entire contents of European patent application EP 18306104.3 are incorporated by reference.

Computer vision is a recent field of effort aimed at providing computers with a high-level understanding of digital videos. It seeks to automate the tasks that the human visual system can do.

One of these tasks is so-called "semantic" segmentation. In general, segmentation is the process of partitioning an image into sets of pixels. When each of these sets corresponds to an entity (car, person, building, etc.) whose type can be identified, segmentation is considered semantic. Indeed, semantic segmentation consists of assigning a class label to each pixel among a given set of classes.

This is of great interest in many applications, including robotics and autonomous driving (to understand scenes and identify where the robot can navigate). In a similar sense, semantic segmentation is useful in the context of augmented reality for understanding the scene and discovering a range of areas to which objects and virtual objects can be added.

Semantic segmentation includes many (such as buildings) that can be large (such as a traffic light) or thin (such as sky, grass, etc.) and well-defined objects (such as cars, bottles, etc.). This is a challenging task for computer vision due to the kinds of classes. Semantic segmentation outputs are spatial (neighboring pixels that tend to belong to the same class, excluding object boundaries) and time (real-world points have a constant label in time. These projections) It also means that it is both to be smooth.

For this, it has been proposed to use a convolutional neural network. A convolutional neural network is a kind of neural network in which the pattern of connections between neurons is inspired by the visual cortex of animals. Convolutional neural networks are therefore particularly suitable for video processing as they allow efficient recognition of objects within images.

Thus, the convolutional neural network is pre-segmented; That is, after a supervised learning step where the labels of each pixel in the frames of the videos are trained by providing thereto a training set of videos provided; The convolutional neural network will be able to segment any of its own input videos (unlabeled, especially "fresh" videos from live CCTV).

Most of the existing methods for semantic segmentation are performed at the image level, that is, the algorithm is applied independently to every frame of the video.

One conventional approach is to transform the image classification convolutional neural network architecture into a fully convolutional version to output a dense prediction. Note that by the stride of standard computer vision convolutional neural network architectures, feature maps generally have a lower resolution than input images. The first deep segmentation approaches were thus to refine the output using an up-sampling strategy, i.e., graphical models such as conditional random fields using an encoder-decoder architecture.

More recently, dilated convolutions (also termed atrous convolutions) have been introduced, which allow the extraction of denser feature maps from existing classification convolutional neural network architectures. .

For better accuracy, a recent trend consists in modeling contextual information at multiple scales of the last layers. For example, this can be accomplished by concatenating the output of the expanded convolutions into a plurality of factors and globally pooled image features, or spatially with several grid scales of the PSP-Net. It is done by performing pooling.

Typically, this leads to an unsatisfactory flickering effect, where some areas of the real world undergo many changes in semantic labels between consecutive frames. This is illustrated in FIG. 2 showing examples of per-frame estimation on three consecutive frames (FIG. 1).

Note that noise artifacts are observed in labels (boxes) predicted over time, even in the region containing flickering between more than two labels (box on the right).

In order to improve consistency over time, semantic segmentation has to be performed at the video level, ie for frames together.

There have been several recent attempts to deal with semantic video segmentation. The first approaches were based on Markov Random Fields, Perturb-and-MAP random fields or conditional random fields in time and space. More recently, it has been proposed to leverage optical flow to model motion in pairwise potentials between frames. Another way to refine the semantic segmentation of videos consists of using a filtering strategy. All these approaches, however, do not produce consistent video segmentation outputs.

More recently, it has been proposed to introduce a NetWarp module to integrate some temporal consistency into semantic segmented convolutional neural network architectures. The idea is to combine the features of the current frame with features from the previous frame that were warped according to the optical flow. Indeed, features from successive frames, after warping, are aggregated according to the optical flow and used to generate the final estimate, but the features remain limited to a predefined and fixed number of frames. have.

Alternatively, a clockwork convolutional neural network architecture has been proposed for semantic video segmentation. Clockwork convolutional neural network architectures consist of reusing intermediate features from previous frames with the aim of reducing the runtime of a video segment at the cost of degrading accuracy.

As a result, there is a need for improvement in segmentation methods using neural networks that allow direct leverage of time-series information.

The following drawings are for the purpose of describing various embodiments and are not to be construed as limiting:
1 shows an example of three consecutive frames;
Figure 2 shows the semantic segmentation of the example frames of Figure 1 using a conventional per-frame technique;
Fig. 3 shows the semantic segmentation of the example frames of Fig. 1 using the convolutional recurrent neural network architecture of Fig. 7;
4 describes an example of a system for semantic segmentation;
5 illustrates how the optical flow displaces pixels for the example of two consecutive frames;
6 shows an example of a convolutional neural network architecture for use in semantic segmentation;
7 shows an example of a convolutional recurrent neural network architecture for use in semantic segmentation.

As described below, two complementary aspects of semantic video segmentation are described: (a) a method of training a convolutional recurrent neural network for semantic segmentation of videos; And (b) a method of semantic segmentation of the input video, advantageously using a convolutional recurrent neural network, trained according to the first method.

인, 프레임 t로서 참조될 것이다. 각 프레임은 이미지, 즉, 주어진 크기의 픽셀들의 매트릭스, 예컨대, 321 x 321 픽셀들이 된다.Any video (depending on its length) is a sequence of T frames, numbered from 1 to T. In other words, a typical frame of video is,

In, it will be referred to as frame t. Each frame is an image, i.e. a matrix of pixels of a given size, e.g. 321 x 321 pixels.

Semantic segmentation of video is the classification of each pixel of each frame of the video; That is, it corresponds to predicting the label of each pixel for each frame, which defines the type of entity depicted by each pixel among the labels of a given list of labels. The frame is thus divided into multiple sets of pixels with the same label, and each set of "connected" pixels with the same label defines an object, ie a "real world" object. For example, all pixels depicting one automobile should be labeled as "automobile" type pixels.

The two types of methods described above are implemented in a system such as that described in Fig. 4, using a first and/or second server 1a, 1b. The first server 1a is a learning server (implementing the first method), and the second server 1b is a segmentation server (implementing the second method). It is completely possible for these two servers to be integrated into a single server.

Each of these servers 1a, 1b is typically a remote computer equipment connected to an extended network 2 such as the Internet for data exchange. Each one of the processor-type data processing means (11a, 11b) (in particular, the learning is long and complex compared to the general use of the trained convolutional recurrent neural network, so the data processing means (11a) of the first server has a strong computing power. ) And optionally, computer memory; And storage means 12 such as, for example, a hard disk.

The first server 1a includes one or more training databases; That is, you have or have access to a set of data that has already been classified (as opposed to the so-called input data that is intended to be classified correctly). As will be explained, the data here consists of images and/or videos and at least includes training videos that have already been semantically segmented. In other words, the pixels of the training videos are already labeled.

The architecture advantageously comprises one or more items of the client equipment 10, which may be at any workstation (also connected to the network 2), preferably from servers 1a, 1b. It can be separate and can be integrated with one and/or the other. It has one or more data items. Operators of equipment are typically "clients" in the commercial sense of the term, of a service provider operating the first and/or second servers 1a, 1b.

Recurrent neural networks are a type of neural networks that utilize sequential information, sequences of frames in the current case. In other words, the output in a given frame is not only dependent on features from this frame, but also on estimates from previous frames, thanks to the "internal state", also termed hidden state or memory. Will be. Recurrent neural networks are well suited for tasks such as handwriting or speech recognition.

Recurrent neural networks have proven to be effective for modeling sequences within neural networks: the idea is to learn internal states that accumulate relevant information over time, and estimates are based on current inputs and these internal states. Recurrent neural networks are often difficult to train due to vanishing gradient issues.

A recurrent neural network includes a “recurrent neural network module”, which is a block of one or more layers representing a “cyclic” behavior. Recurrent neural network modules are well known to those skilled in the art. The recurrent neural network may include a gated recurrent unit and/or a long short-term memory. The gate cycle unit and long-term memory contain learnable gates for selectively updating the internal state, thus making it possible to propagating gradients over longer sequences during training.

Typically, x _t , h _t And o _t a circular neural network from each frame t; the features at the output o _t is the current frame (t) of the input, defining a general configuration showing the hidden state and output, cycling the neural network modules (Recurrent Neural Network RNN) It is a function of not only _{x t} but also the hidden state h _t-1 in the previous frame t -1. At the same time the hidden state h _t is updated based on the same inputs, so the recurrent neural network module is defined by the following equation:

h _o is initialized to a tensor that is all zero ( h _o ). In this zero initialization the training step is preferably performed in longer sequences than pairs.

The recurrent neural network module can be adapted to the task of semantic segmentation to take into account the inherent motion of the videos.

In fact, x _t And o _t are presently feature maps for the task of semantic segmentation, and thus convolution operators (in which input, internal state, and output are 3D tenders) are recursive neural network modules, e.g., It can be used inside the convolutional gate circulation unit.

These convolution operators have been used, for example, for frame prediction and video captioning. Convolutional recurrent neural networks have also been applied for other dense video estimation tasks, such as video object segmentation, whose purpose is to output a binary segmentation of an object for a given video in the first frame.

For example, it has been proposed to apply a convolutional gate circular unit to semantic video segmentation. However, these models lack motion modeling: the estimate at a given pixel is based on the history of the estimate at that pixel, but these pixels can represent different points in the real world if there is motion.

In fact, the output o _t is the size of the neighborhood according to the kernel size of the known convolution operators, and will depend on the local spatial neighborhood of x _t and h _t-1.

Standard convolutional neural network cycle are, however, (meaning that the "same pixel", which is exactly the same pixel coordinates) in frame t _t a given pixel p and the same pixel in the previous frame t -1 p _t-1 is Only in cases where there are projections of the same real-world point, that is, if this point remains static in the real world (or if it moves along a projection ray), it can be considered. In other words, this assumes that the video is static, i.e. there is no motion. However, for many applications, such as autonomous driving, this assumption is not valid and pixels can move significantly between frames.

More precisely, the estimation of a pixel p in frame _t of t p _t are those pixels in the local neighborhood N (p _t) in the local neighborhood from a feature in _t x and concealed state in the previous frame t h -1 _t-1 It will depend on N(p _t-1 ). Conversely, the estimation at p _t will actually have to depend on the features x _t of N(p _t ) and the local neighbor N(p' _t-1 ) from the hidden state h _t-1 . p _'t-1 is the pixel corresponding to the same point in the real world of _t p.

5, showing examples of consecutive frames t- 1 (left) and t (right), on the previous frame t- 1 at exactly the same coordinates of two pixels p _t and q _{t on the current frame t} The pixels p _t-1 and q _t-1 are shown. _{t p-1} and q _t-1 is, as opposed to p _'t-1, and q' _t-1, does not belong to the same area as the p and _t q _t, respectively.

의 쌍, 즉, 이전의 프레임 t-1 및 현재의 프레임 t)로의 모든 픽셀들의 변위를 정의하는 벡터 필드이다. The so-called "optical flow" is another one of an image (especially successive frames).

A pair of, i.e. the previous frame t -1 And a vector field defining the displacement of all pixels to the current frame t ).

은 이전의 프레임 t-1으로부터 현재의 프레임 t으로의 모든 픽셀들의 변위를 정의하고, 광학 흐름

("역방향 흐름")은 현재의 프레임 t으로부터 이전의 프레임 t-1으로의 모든 픽셀들의 변위를 정의한다.E.g. optical flow

Defines the displacement of all pixels from the previous frame t -1 to the current frame t, and the optical flow

("Reverse flow") defines the displacement of all pixels from the current frame t to the previous frame t -1.

에 의해 주어진다: 다시 말해, 광학 흐름은 벡터 필드에 따라 각 픽셀을 "시프트"한다.Pixels p _'t-1 is formula

Given by: In other words, the optical flow "shifts" each pixel according to the vector field.

가 이전 프레임 t-1에서의 피쳐 맵이고

가 현재의 프레임 t에서의 피쳐 맵이면(피쳐 맵은 벡터 필드들임; 즉, 각 픽셀에 벡터를 연관시킴),

및

의 여하한 픽셀 피쳐들은 결합된다. 이러한 연산은 효율적으로 구현하기가 어렵기 때문에(관련된 픽셀들이 상이한 좌표들을 가지기 때문에), 소위 피쳐들의 와핑이 먼저 계산된다; 즉,

로서 정의되는 벡터 필드

를 구축하도록 계산된다. 따라서,

및

의 피쳐들은 픽셀 단위(pixel by pixel)로 직접적으로 결합될 수 있다. In order to adapt the feature maps to the motion of all pixels along the flow, the use of the cycle module is combined with the use of a "warping" function. Actually,

Is the feature map from the previous frame t -1

If is the feature map in the current frame t (the feature map is vector fields; i.e., associating a vector to each pixel),

And

Any pixel features of are combined. Since this operation is difficult to implement efficiently (since the pixels involved have different coordinates), the so-called warping of features is calculated first; In other words,

Vector field defined as

Is calculated to build. therefore,

And

The features of can be directly combined in pixel by pixel.

)을 알면, 와핑은 여하한 주어진 피쳐 맵

으로부터의 주어진 피쳐 맵

에 대응하는 와핑된 피쳐 맵

의 계산하는 것이고, 각 픽셀은 광학 흐름에 따른 변위를 겪은 것이 된다.In other words, optical flow(

), warping is any given feature map

A given feature map from

Feature maps corresponding to

Is to be calculated, and each pixel is subjected to a displacement according to the optical flow.

에 따른 다른 픽셀

에 대한 와핑될 피쳐 맵의 값(이전의 이미지 t-1에 연관된 내부 상태 h _t-1 )을 연관시킨다.Warping does not change the values of the feature map (i.e. vectors), it just "spatially rearranges" them: for warping from the previous frame t -1 to frame t in this case, the warping function is For each pixel p _t of image t , the reverse optical flow

Other pixels according to

Associate the value of the feature map to be warped for (the internal state h _t-1 associated with the previous image t -1 ).

Indeed, optical flow includes not only integers but also float values. The warping function can be based on bilinear interpolation (e.g., between four values), which can be differentiated, except for the exceptional case where the flow values are integers, where the gradient is set to zero. .

A typical warped recurrent neural network module is named "FlowingRNN", and thus, can be written as a recurrent neural network module in which the internal state is warped between frames according to the optical flow, that is, as follows:

In a preferred embodiment, "FlowingGRU", that is, a Convolutional Gate Cyclic Unit Module based FlowingRNN is used. The following equation can thus be given:

는 요소별(element-wise) 곱(multiplication)을,

는 컨볼루션 연산자를,

는 시그모이드 함수를, ReLU는 정류된 선형 유닛(rectified linear unit) 비선형성을(아래 참조), W와 b는 학습 가능한 파라미터들(각각 가중치들 및 바이어스들)을 나타낸다.

Is the element-wise multiplication,

Is the convolution operator,

Is a sigmoid function, ReLU is a rectified linear unit nonlinearity (see below), and W and b are learnable parameters (weights and biases, respectively).

Intuitively, the reset gate r _t learns how to combine the input x _t with the previous hidden state h _t-1, and the update gate z _t learns how much previous memory should be kept.

The main difference between this FlowingGRU and the standard convolutional gate circulation unit is that the hidden state is warped according to the optical flow, and therefore, even in the case of pixel motion, the estimate at a given pixel and time step is the history of this particular point in the real world. It is to be based on (history). One small difference is also that the more commonly ReLU is used in computer vision convolutional neural network architectures, the ReLU non-linearity is preferably used instead of the standard tanh.

One of skill in the art would be able to transpose the described architecture into any other type of circular module. In particular, instead of "FlowingGRU", "FlowingLSTM"; That is, FlowingRNN based on convolutional short-term memory module may be used.

In general, FlowingRNN can be plugged into any fully convolutional image segmentation approaches to perform improved semantic segmentation of videos, in terms of increased performance, especially consistency over time.

3 shows examples of semantic segmentation using FlowingRNN for three consecutive frames of FIG. 1. Referring to FIG. 2, the absence (boxes) of noise artifacts in labels predicted over time can be observed in FIG. 3.

The convolutional recurrent neural network, flow estimation and warping modules are all differentiable and make full network end-to-end training possible.

In the first aspect, the training method is implemented by the data processing means 11a of the first server 1a. The method trains a convolutional recurrent neural network for semantic segmentation of videos.

In the first step (a), the standard convolutional neural network (acyclic) will be referred to as the “first convolutional neural network” and a base of pre-semantically segmented training images (e.g., MS-Coco data set). ) From

In other words, the first convolutional neural network is a classic "per frame" baseline that does not take into account local dependence over time. Any known architecture can be used for the first convolutional neural network.

Convolutional neural networks generally contain four types of layers that process information:

(a) a convolution layer that processes blocks of an image one by one;

(b) a non-linear layer (also called a correction layer) that allows the pertinence of the result to be improved by applying the "activation function";

(c) a pulling layer that allows several neurons to be grouped together within a single neuron;

(d) A fully connected layer connecting all neurons in the layer with all neurons in the preceding layer.

Note that in the case of a fully connected convolutional network, the fully connected layer(s) no longer exist and are converted to a convolutional layer of kernel size 1. This allows an output map of probabilities instead of just one probability distribution over the whole image.

As the activation function of the non-linear layer, the ReLU function (rectified linear unit) is most often used, which is equivalent to f(x) = max(0, x), and the pooling layer (POOL) is square (square). The MaxPool2Х2 function, which corresponds to the maximum value among the four values of) (4 values are pooled into one), is the most used, but other pooling functions such as AveragePool or SumPool can be used.

The convolutional layer (CONV) and the fully connected layer (FC) generally correspond to a scalar product between the neurons of the previous layer and the weights of the convolutional neural network.

RELU 의 쌍들을 적층하고, 그 다음으로 레이어 POOL을 추가하고, 이러한 스킴[(CONV

RELU)p

POOL]을 충분히 작은 출력 벡터가 획득될 때까지 반복하고, 하나 또는 두 개의 완전히 연결된 레이어들로 종료한다.Typical convolutional neural network architectures have several layers of CONV

Stacking pairs of RELUs, then adding a layer POOL, this scheme[(CONV

RELU)p

POOL] is repeated until a sufficiently small output vector is obtained, and ends with one or two fully connected layers.

In the example of FIG. 6, an architecture for a first convolutional neural network is described.

As illustrated in Figure 6, the first convolution layer and blocks 1 to 4 are from ResNet-101, and blocks 3 and 4 are expanded with a factor of 2 and 4, respectively (also referred to as rate). We are using the convolutions and reduce the stride of the network from 32 to 8.

Next is the Atrous Spatial Pyramid Pooling (ASPP) module in which results from different convolution kernels with various expansion factors are stacked. The Atrus spatial pyramid pooling module is fed into a 1x1 convolution layer (with batch normalization and ReUL) with 256 filters, before the final 1x1 convolution layer that outputs the class scores. These scores are bilinearly upsampled to the original image resolution, e.g., by applying a softmax loss per pixel during training time, or by taking argmax per pixel during testing time to estimate the predicted class. .

만큼 곱해지는 것; 이 적용될 수 있다.To train the first convolutional neural network, a stochastic gradient descent (SGD) method having a batch size of 16 may be used. In particular, the so-called'poly' learning rate schedule; That is, at the iteration i for the total number of N iterations at the initial learning rate

Multiplied by; Can be applied.

Training patches are generated from random crops of training images having a size of, for example, 321x321 pixels, and a convolutional neural network is tested over the entire images. In training, before taking the random crop, data augmentation preferably randomly flips the images left-right and ranges [0:5; 1:5] by applying a random scaling factor.

Additionally (step (a)), additional training of the second convolutional neural network from the base of pairs of consecutive frames for which the optical flow is known is performed.

을 추정하는 것을 목표로 한다. As will be explained, the second convolutional neural network is the optical flow of pairs of frames, in particular, the reverse optical flow from the current frame t to the previous frame t -1.

It aims to estimate.

The second convolutional neural network may be such as FlowNetSimple or FlowNetCorrelation. The architecture FlowNetSimple (generally just named FlowNetS) is preferred and is used in the example of FIG. 6. In this architecture, both input images are stacked together and fed through a network made only of convolutional layers.

ReLU non-linearity and'upconvolution' can also be implemented by convolutions.

Next (step (b)), the convolutional recurrent neural network is trained. The idea is to build a convolutional recurrent neural network from a first convolutional neural network (and a second convolutional neural network if present).

The convolutional recurrent neural network corresponds to the first convolutional neural network, and the convolutional layer is replaced by a recursive module (FlowingRNN module such as FlowingGRU) having a hidden state as described above.

Preferably, as represented by Fig. 6, the convolutional layer of the first convolutional neural network, which is replaced by the recursive module, is the penultimate convolutional layer. This location provides excellent results because it is immediately after the Atrus spatial pyramid pooling module in the example based on ResNet-101, but the recursive module can be replaced anywhere in the first convolutional neural network, e.g., the last convolution. Layers can be replaced. In addition, three convolutional layers may exist after the Atrus spatial pyramid pooling module, and the recursive module will replace the first one (that is, the third (antepenultimate) convolutional layer behind the first convolutional neural network). I can.

Training of the first convolutional neural network (step (a)) can be viewed as "initialization" of the parameters of the convolutional recurrent neural network, and additional training from the base of already semantically segmented training videos (as described below). By, the training of the convolutional recurrent neural network (step (b)) can be viewed as a "fine-tuning" of these parameters.

Some of the parameters obtained thanks to the training of the first convolutional neural network may be fixed, i.e., may not be further learned in the training of the convolutional recurrent neural network, preferably before the recursive module (i.e., Atrus Until the spatial pyramid pooling is included), the parameters of each layer of the convolutional recurrent neural network may be. The further learned parameters are then preferably the recursive module, the last convolutional layer of the convolutional neural network (and generally each non-fixed convolutional layer) and possibly the parameters of the second convolutional neural network (if any).

The most popular benchmarks for semantic segmentation are limited to images and cannot be used in the training of convolutional recurrent neural networks. This method can be trained by ignoring pixels that are not annotated lossy, even if only a few pixels are annotated in the videos, and training using dense ground-truth is better for viewing. This will lead to thermal consistency.

Most real-world data sets have only one frame annotated per sequence, but some synthetic benchmarks are limited either by the realism of the data or by a low number of sequences, and thus already semantically segmented. The base of the customized training videos is preferably a recent Viper data set consisting of about 250k frames from nearly 200 video sequences captured from a realistic Grand Theft Auto (GTA) V video game.

Synthetic rendering allows for obtaining ground verification data for multiple tasks, including semantic segmentation. Lighting and weather conditions change from day to night, from sunny to rain, when snow or when there is fog, making the data set challenging. The semantic segmentation task has 23 classes, including elements (such as sky, terrain), vehicles (such as cars, trucks), and small objects (such as traffic signs, traffic lights).

의 각 쌍에 대해, 해당 쌍의 프레임들 간의 광학 흐름(특히, 현재의 프레임 t로부터 이전의 프레임 t-1으로의 역방향 광학 흐름

)을 추정하는 것(서브-단계(b0))으로 시작한다.Training of the convolutional recurrent neural network (step (b)) advantageously consists of consecutive frames of one video among the bases of training videos that have already been semantically segmented.

For each pair of, the optical flow between the frames of that pair (in particular, the reverse optical flow from the current frame t to the previous frame t -1)

We start with estimating (sub-step (b0)).

This step in training (step (b)) is preferably carried out using a second training of a convolutional neural network (FlowNetS in Fig. 6) using two frames in a pair as input, but the method is optical. It is not limited to any technique for estimating the flow.

Note that the optical flow may have already been estimated between the frames of the training base, and thus this step remains optional in training (sub-step (b0)).

Additionally (sub-step (b1)), as already explained, the inner state of the cyclic layer is warped according to the estimated optical flow so that the inner state adapts to the motion of the pixels between the paired frames.

Then (sub-step (b2)), at least the parameters of the cyclic module are learned.

In addition to the recursive module, the output of this layer will be different from one of the baselines per frame, so subsequent layers are also retrained. As already explained, the additionally learned parameters are then preferably the recursive module, the last convolutional layer of the convolutional neural network (and generally each non-fixed convolutional layer) and (if any) maybe the second convolutional neural network. Becomes the parameters of.

To this end, we train sequences of 12 consecutive frames with a batch size of 4, and use backpropagation through a temporal algorithm with a poly learning rate schedule similar to that for the initial training of the first CNN, Stochastic gradient descent can be used. Similar data augmentation strategies can also be used. At test time, the hidden state from the previous frame can be used, without any restrictions on the length of the sequence.

In the second aspect, a semantic segmentation of the input video implemented by the data processing means 11b of the second server 1b is proposed. In other words, the method according to the second aspect performs semantic segmentation of the input video, that is, labels the pixels of the frames of the input video.

The input video that is segmented may be received from the client equipment 10.

In the first step (a), the training of the convolutional recurrent neural network (at least on the basis of pre-semantically segmented training videos and advantageously at least pre-semantically segmented training images), as illustrated in FIG. Is performed by the first server 1a). Preferably, the training is consistent with the method according to the first aspect; That is, the first convolutional neural network is trained, and thus, a convolutional recurrent neural network based on the first convolutional neural network is constructed, and the second convolutional layer at the end is replaced by a FlowingRNN-type recursive module.

Alternatively or according to a combination, step (a') comprises training a convolutional neural network (i.e., the second convolutional neural network of the training method) from the base of training pairs of successive frames for which the optical flow is known. It may contain more.

The first and second servers 1a, 1b are the same equipment, or step (a') is the first server 1a of trained convolutional recurrent neural network parameters and weights for storage in the memory 12. ) To the second server 1b.

In the second step (b'), the processing means 11b of the second server 1b uses a convolutional recurrent neural network for segmentation.

)의 각 쌍에 대해, 다음을 포함한다: To this end, step (b') includes consecutive frames of the input video (

For each pair of ), include:

(b'0) estimating an optical flow between frames forming a pair;

(b'1) warping the internal state of the cyclic layer according to the estimated optical flow to adapt the internal state to the motion of pixels between the paired frames; And

(b'2) Semantic segmentation of each frame with a convolutional recurrent neural network.

Sub-steps (b'0) and (b'1) are advantageously equivalent to sub-steps (b0) and (b1), as described above, and may include any purification thereof.

Finally, the obtained classification result may be transmitted back to the client equipment 10.

The effect of the present invention is a standard mean Intersection-over-Union metric (mIoU ^P ) per pixel: for each class, the ground verification data or estimation of the corresponding class is calculated within the corresponding class. It can be evaluated using the ratio of the number of pixels correctly estimated for that class to the number of total pixels, and the average for all classes is reported.

However, this does not measure the time-series consistency of semantic segmentation estimation. To this end, the min-intersection-over-union indicator can be further computed at the trajectory level, where the correct dense trajectories for all videos are extracted at half resolution and density of 8 pixels. The trajectory must be consistently labeled across all frames with the ground verification data class in order to be considered as being able to be extracted with the flow.

Only trajectories with consistent ground verification data labeling are maintained to filter out noise trajectories due to errors in flow estimation. Indeed, this filtering step removes about 8% of the extracted trajectories.

This makes it possible to report a "mean Intersection-over-Union metric (mIoU ^P ) per trajectory, and trajectories where pixels have different estimated labels are counted as wrong. . The consistency of each class can also be calculated according to the following: For each predicted class, the proportion of trajectories for which the prediction is consistent among trajectories corresponding to a given class with consistent ground verification data is calculated. "Mean consistency" (mCons.) can be reported by averaging over all classes.

First, the effect of the length T of training sequences on the performance of the preferred FlowingGRU module is studied. At test time, independent of the length of the sequences used in training, the hidden state of the previous frame is used in all cases. As a sanity check, the model is first used for all cases and sequences of one frame in training with all-zeros internal state to verify that the performance is similar to the baseline per frame. . T is then increased from 2 to 12. The consistency gain in performance is observed as T increases. When using training sequences of 12 frames compared to baseline per frame, mIoU ^P was higher by 1.2%, mIoU ^T was 2.1% and mCons was 3.5% higher.

Next, the influence of the number of output channels (C) of the convolutions of the preferred FlowingGRU module (and therefore also of the internal state) and the kernel size (K) (of all convolutions in the FlowingGRU) is studied. Note that optimal performance is observed at C = 256 and K = 3, which can be explained by the following facts:

The lower number of channels is not sufficient to convey adequate information spatially from the previous layer and temporally from previous frames for accurate semantic segmentation;

A kernel with a size greater than 1 incorporates information about the local neighbor, thus allowing recovery from small errors in optical flow calculations;

Kernels with a size larger than 3 have too many parameters and cannot effectively learn local time-series consistency.

The impact of the FlowingGRU module (with preferred values of T = 12, K = 3 and C = 256) is compared with the tasks associated with several variants in Table 1 (left). Note that none of the tasks have been tested with the Viper data set and these tasks use different backbone architectures. For fair comparison, the same baseline is used, and each of the proposed modules is connected at the same location as FlowingRNN and follows the same training scheme.

[Table 1]

Compared to baseline per frame (with ReLU non-linearity), the FlowingGRU module shows a clear improvement of 1.22% ^{for mIoU P , 2.14% for mIoU T} and 3.44% for mCons. This emphasizes that FlowingGRU effectively utilizes time-series information for dense video prediction tasks while considering pixel motion.

The FlowingGRU module with ReLU non-linearity could be compared with two variants. The first one consists of using the standard tanh nonlinearity (instead of ReLU), which refers to the'FlowingGRU (tanh)' row in Table 1. A drop of 0.8% was observed for mIoU ^P , indicating that ReLU is more appropriate for semantic segmentation.

In the second variant, the parameters of FlowNetS are fixed, but not fine-tuned during training (i.e., the second convolutional neural network is not further trained in step (b)). This refers to the'FlowingGRU (fixed flow)' row of Table 1. In this case, there ^{is a 0.9% drop in mIoU P} , which shows the importance of training the flow estimation with the FlowingGRU module.

The FlowingGRU module is further compared to other circulation models. As use in the context of semantic video segmentation, without warping of the internal state, for example, the standard convolutional gate circular unit (ConvGRU) is first tried. Results are reported in the'ConvGRU' row of Table 1. The three indicators increase slightly compared to the baseline per frame, showing that some of the time-series consistency is actually learned. However, ConvGRU exhibits significantly inferior performance to the preferred FlowingGRU, e.g. 0.8% lower mIoU ^P. In fact, by maintaining the internal state between successive frames, ConvGRU assumes that pixels at the same location in successive frames are projections of the same real-world point, which is not correct in most cases. Similarly, mIoU ^T and mCons. are also significantly reduced to 1.63% and 2.55%, respectively.

Next, the gate circulation unit is replaced by a long-term memory where ReLU non-linearity is also used instead of tanh, which refers to the'ConvLSTM'and'FlowingLSTM' rows of Table 1. The performance is considerably lower than with the gate circulation unit. One explanation is that short and long-term memories are more difficult to train than gate cycle units for vision tasks. FlowingLSTM performed significantly better than convolutional LSTM ( ^{+4.06% for mIoU P} ) with one interesting result, which again emphasizes the importance of warping the internal state.

Finally, by replacing the last convolutional layer instead of the second to last layer, the location of the FlowingGRU module could be studied. Note that in this case, the output goes into softmax (nonlinearity was not used). Note that this modification leads to a reduction in ^{mIoU P} of 0.6% for the Viper data set. When using ConvGRU, the performance is also lower when replacing the module in the last layer. By replacing the second layer at the end, the history is embedded in the latent space where the final estimate is made, which is more powerful than estimating the final segmentation directly from the hidden state and current features.

Some experiments could also be run on a real-world Cityscapes dataset using only fine annotation, ie 2975 sequences for training and 500 sequences for verification. Each sequence has 30 frames with annotations only on one frame (12th frame).

Training and testing procedures can be kept similar to that of the Viper data set. The only difference is that the ground verification data is annotated for sparse frames. Thus, during training, the loss for unannotated frames is neglected, which makes the video information less suitably utilized. Noise trajectories cannot be cleaned with only one annotated frame per sequence, so ^{only mIoU P} can be reported.

The results for the different variants are reported in the right column of Table 1. First, FlowingGRU exceeds the baseline per frame with a gain of 1.14% ^{in mIoU P.} In all cases, the'Flowing' correspondence represents significantly better performance than a static convolution module.

In short, a method of training a convolutional recurrent neural network for semantic segmentation of videos includes: training a first convolutional neural network from a base of already semantically segmented training images; And training a convolutional recurrent neural network corresponding to the first convolutional neural network from the base of the training videos that have already been semantically segmented, wherein the convolutional layer has a hidden state. Was replaced by In the training, for each pair of consecutive frames of one video among the bases of the training videos that are already semantically segmented, the cyclic layer is formed according to an estimated optical flow between the frames of the pair. Warping the internal state so that the internal state adapts to the motion of the pixels between the frames of the pair.

Advantageously, providing a standard convolutional cyclic layer (which can be estimated using other convolutional neural networks) where the internal state is warped between frames according to the optical flow allows for semantic segmentation of videos, in particular. , Improve, in terms of consistency over time.

The pair of successive frames preferably includes a previous frame and a current frame, the estimated optical flow is a backward optical flow from the current frame to the previous frame, and the previous frame A warping function is applied to the internal state associated with, to obtain a warped internal state associated with the previous frame corresponding to the internal state, and each pixel has undergone a displacement according to the reverse optical flow;

이고, p _t 는 상기 현재의 프레임(t)의 픽셀이고,

는 상기 역방향 광학 흐름이고, h _t-1 는 상기 이전의 프레임(t-1)과 연관된 내부 상태이고,

는 상기 이전의 프레임(t-1)과 연관된 와핑된 내부 상태이다;The warping function is

And p _t is a pixel of the current frame ( t ),

Is the reverse optical flow, h _t-1 is the internal state associated with the previous frame ( t -1),

Is the warped internal state associated with the previous frame ( t -1);

Estimation of the optical flow of the paired frames may be performed using the second convolutional neural network by training a second convolutional neural network from the base of training pairs of consecutive frames whose optical flow is known. . The second convolutional neural network may be a FlowNetSimple network or a FlowNetCorrelation network;

Parameters of each layer of the convolutional recurrent neural network before the recursive module may be fixed.

The convolutional layer of the first convolutional neural network that is replaced by a recursive module may be a penultimate convolutional layer.

The learned parameters of the convolutional recurrent neural network may be parameters of the recurrent module and the last convolutional layer of the first convolutional neural network.

The first convolutional neural network may include an atrous spatial pyramid pooling module before the second convolutional layer from the end.

The circulation module may include a convolutional gated recurrent unit or a convolutional long short-term memory.

)의 각 쌍에 대해, 연속하는 프레임들의 쌍의 프레임들 간의 추정된 광학 흐름(optical flow)에 따라 순환 레이어의 내부 상태를, 상기 내부 상태가 상기 쌍의 프레임들 간의 픽셀들의 모션에 적응하도록(adapt), 와핑하는 단계 및 적어도 상기 순환 모듈의 파라미터들을 학습하는 단계를 포함함 -에 의해 훈련된 컨볼루션 순환 신경망을 사용하는, 입력된 비디오의 의미적 세그먼트화 방법은, (a) 상기 입력된 비디오의 연속하는 프레임들(

)의 각 쌍에 대해, 상기 쌍의 프레임들 간의 광학 흐름을 추정하는 단계; (b) 상기 입력된 비디오의 연속하는 프레임들(

)의 각 쌍에 대해, 상기 추정된 광학 흐름에 따라 상기 순환 레이어의 내부 상태를, 상기 내부 상태가 상기 쌍의 프레임들 간의 픽셀들의 모션에 적응하도록(adapt), 와핑하는 단계; 및 (c) 상기 입력된 비디오의 연속하는 프레임들(

)의 각 쌍에 대해, 상기 컨볼루션 순환 신경망으로 각 프레임을 의미적으로 세그먼트화하는 단계를 포함한다. Training a first convolutional neural network using a set of semantically segmented training images; And training a convolutional recurrent neural network, corresponding to the first convolutional neural network, using a set of semantically segmented training videos, the convolutional layer by a recursive module having a hidden state. Replaced; Training the convolutional recurrent neural network includes consecutive frames of one video of the set of semantically segmented training videos (

For each pair of ), the internal state of the cyclic layer is adapted according to the estimated optical flow between the frames of the pair of consecutive frames, so that the internal state adapts to the motion of the pixels between the pair of frames ( adapt), warping, and learning at least the parameters of the recursive module. A method for semantic segmentation of an input video using a convolutional recurrent neural network trained by: (a) the input Successive frames of video (

For each pair of ), estimating the optical flow between the frames of the pair; (b) successive frames of the input video (

For each pair of ), warping the internal state of the cyclic layer according to the estimated optical flow, such that the internal state adapts to the motion of pixels between the frames of the pair; And (c) consecutive frames of the input video (

For each pair of ), semantically segmenting each frame with the convolutional recurrent neural network.

Training the convolutional recurrent neural network may include training a second convolutional neural network using a set of training pairs of consecutive frames whose optical flow is known, wherein (a) Is performed using the second convolutional neural network.

The second convolutional neural network may be a FlowNetSimple network or a FlowNetCorrelation network.

)의 각 쌍에 대해, 상기 쌍의 프레임들 간의 광학 흐름을 추정하는 단계; (b) 상기 입력된 비디오의 연속하는 프레임들(

)의 각 쌍에 대해, 상기 추정된 광학 흐름에 따라 순환 레이어의 내부 상태를, 상기 내부 상태가 상기 쌍의 프레임들 간의 픽셀들의 모션에 적응하도록(adapt), 와핑하는 단계; (c) 상기 입력된 비디오의 연속하는 프레임들(

)의 각 쌍에 대해, 상기 컨볼루션 순환 신경망으로 각 프레임을 의미적으로 세그먼트화하는 단계를 포함하고, 상기 컨볼루션 신경망은, 의미적으로 세그먼트화된 훈련 이미지들의 세트를 사용하여, 제1 컨볼루션 신경망을 훈련시키는 단계; 및 의미적으로 세그먼트화된 훈련 비디오들의 세트를 사용하여, 상기 제1 컨볼루션 신경망에 대응하는, 컨볼루션 순환 신경망을 훈련시키는 단계에 의해 훈련되고, 컨볼루션 레이어는 은닉 상태(hidden state)를 갖는 순환 모듈에 의해 대체되었고; 상기 컨볼루션 순환 신경망을 훈련시키는 단계는, 상기 의미적으로 세그먼트화된 훈련 비디오들의 세트 중 하나의 비디오의 연속하는 프레임들(

)의 각 쌍에 대해, 연속하는 프레임들의 쌍의 프레임들 간의 추정된 광학 흐름(optical flow)에 따라 순환 레이어의 내부 상태를, 상기 내부 상태가 상기 쌍의 프레임들 간의 픽셀들의 모션에 적응하도록(adapt), 와핑하는 단계 및 적어도 상기 순환 모듈의 파라미터들을 학습하는 단계를 포함한다. A method for semantic segmentation of an input video using a convolutional recurrent neural network includes: (a) successive frames of the input video (

For each pair of ), estimating the optical flow between the frames of the pair; (b) successive frames of the input video (

For each pair of ), warping the internal state of the cyclic layer according to the estimated optical flow, such that the internal state adapts to the motion of pixels between the frames of the pair; (c) successive frames of the input video (

), for each pair of ), semantically segmenting each frame with the convolutional recurrent neural network, wherein the convolutional neural network uses a set of semantically segmented training images, Training a lusion neural network; And training a convolutional recurrent neural network, corresponding to the first convolutional neural network, using a set of semantically segmented training videos, wherein the convolutional layer has a hidden state. Replaced by a circulation module; Training the convolutional recurrent neural network includes consecutive frames of one video of the set of semantically segmented training videos (

For each pair of ), the internal state of the cyclic layer is adapted according to the estimated optical flow between the frames of the pair of consecutive frames, so that the internal state adapts to the motion of the pixels between the pair of frames ( adapt), warping and learning at least the parameters of the cyclic module.

The training of the convolutional recurrent neural network includes training a second convolutional neural network using a set of training pairs of consecutive frames whose optical flow is known, wherein (a) is the It is performed using a second convolutional neural network.

The second convolutional neural network may be a FlowNetSimple network or a FlowNetCorrelation network.

)의 각 쌍에 대해, (b1) 연속하는 프레임들의 쌍의 프레임들 간의 추정된 광학 흐름(optical flow)에 따라 순환 레이어의 내부 상태를, 상기 내부 상태가 상기 쌍의 프레임들 간의 픽셀들의 모션에 적응하도록(adapt), 와핑하고, (b1) 적어도 상기 순환 모듈의 파라미터들을 학습하는 것을 포함할 수 있다. In another aspect, a system for training a convolutional recurrent neural network for semantic segmentation of videos is provided. The system is configured to: (a) train a first convolutional neural network using a set of semantically segmented training images, and (b) use a set of semantically segmented training videos, wherein the first It is possible to train a convolutional recurrent neural network, corresponding to a convolutional neural network. The convolutional layer has been replaced by a circular module with a hidden state. Training the convolutional recurrent neural network includes successive frames of one video of the set of semantically segmented training videos (

) For each pair of, (b1) the internal state of the cyclic layer according to the estimated optical flow between the frames of the pair of consecutive frames, and the internal state is the motion of the pixels between the pair of frames. Adapting, warping, and (b1) learning at least the parameters of the recursive module.

)의 각 쌍에 대해, (b1) 연속하는 프레임들의 쌍의 프레임들 간의 추정된 광학 흐름(optical flow)에 따라 순환 레이어의 내부 상태를, 상기 내부 상태가 상기 쌍의 프레임들 간의 픽셀들의 모션에 적응하도록(adapt), 와핑하는 단계 및 (b1) 적어도 상기 순환 모듈의 파라미터들을 학습하는 단계를 포함할 수 있다. In another aspect, a program stored in a computer-readable recording medium is provided to execute a method of training a convolutional recurrent neural network for semantic segmentation of videos on a computer. The method comprises the steps of: (a) training a first convolutional neural network using a set of semantically segmented training images, and (b) using a set of semantically segmented training videos, 1 Training a convolutional recurrent neural network, corresponding to a convolutional neural network, may include-the convolutional layer has been replaced by a recursive module having a hidden state. Training the convolutional recurrent neural network includes consecutive frames of one video of the set of semantically segmented training videos (

) For each pair of, (b1) the internal state of the cyclic layer according to the estimated optical flow between the frames of the pair of consecutive frames, and the internal state is the motion of the pixels between the pair of frames. It may include adapting, warping, and (b1) learning at least the parameters of the cyclic module.

It will be appreciated that variations of the previously disclosed embodiments and other features and functions or their substitutes may be advantageously combined with many other different systems or applications. In addition, various substitutes, changes, modifications or improvements that are not currently predicted or predicted herein may be made later by those of ordinary skill in the art, which are described above and claimed below. It is also intended to be encompassed by range.

Claims (23)

In a method of training a convolutional recurrent neural network for semantic segmentation of videos, performed by a computer,
(a) training a first convolutional neural network using a set of semantically segmented training images; And
(b) training a convolutional recurrent neural network constructed based on the first convolutional neural network using a set of semantically segmented training videos-a second (penultimate) from the end of the first convolutional neural network ) The convolution layer is replaced by a circular module with a hidden state-
Including,
Training the convolutional recurrent neural network includes consecutive frames of one video of the set of semantically segmented training videos (

For each pair of ),
(b1) warping the internal state of the cyclic layer according to the estimated optical flow between frames of a pair of consecutive frames, so that the internal state adapts to the motion of pixels between the pair of frames. The step of doing; And
(b2) learning at least the parameters of the recursive module
Including,
The consecutive frames (

The pair of) includes a previous frame ( t -1) and a current frame ( t ), and the estimated optical flow is in the reverse direction from the current frame (t ) to the previous frame ( t -1) ( backward) optical flow (

)ego;
Wherein (b1) is the previous frame (t -1) corresponding to the warping function, the internal state (h _t-1), the internal state (h _t-1) in association with the previous frame (t -1) The warped internal state associated with (

In order to obtain ), it is a step of applying, and each pixel has the reverse optical flow (

A method of training a convolutional recurrent neural network that has undergone displacement according to ). delete The method of claim 1,
The warping function is

And p _t is a pixel of the current frame ( t ),

Is the reverse optical flow, h _t-1 is the internal state associated with the previous frame ( t -1),

Is a warped internal state associated with the previous frame ( t -1). The method of claim 1,
The (b1) includes the step of estimating an optical flow between frames of a pair of consecutive frames, which is performed using a second convolutional neural network, and the second convolutional neural network has a known optical flow ( known) A method of training a convolutional recurrent neural network, which is trained using a set of training pairs of consecutive frames. The method of claim 4,
The second convolutional neural network is a FlowNetSimple network, a method for training a convolutional recurrent neural network. The method of claim 4,
The second convolutional neural network is a FlowNetCorrelation network, a method for training a convolutional recurrent neural network. The method of claim 1,
The method of training a convolutional recurrent neural network, wherein parameters of each layer of the convolutional recurrent neural network before the recursive module are fixed during (b2). delete The method of claim 7,
The parameters of the convolutional recurrent neural network learned in (b2) are parameters of the recurrent module and the last convolutional layer of the first convolutional neural network. The method of claim 4,
(B2) is the step of learning parameters of the second convolutional neural network
A method of training a convolutional recurrent neural network further comprising. The method of claim 7,
The first convolutional recurrent neural network, before the second convolutional layer from the end, includes an atrous spatial pyramid pooling module. The method of claim 1,
The method of training a convolutional recurrent neural network, wherein the recursive module comprises a convolutional gated recurrent unit. The method of claim 1,
The method of training a convolutional recurrent neural network, wherein the recursive module comprises a convolutional long short-term memory. Training a first convolutional neural network using a set of semantically segmented training images, performed by a computer; And training a convolutional recurrent neural network constructed based on the first convolutional neural network using a set of semantically segmented training videos-a penultimate convolution at the end of the first convolutional neural network. The root layer is replaced by a circular module with a hidden state; Training the convolutional recurrent neural network includes consecutive frames of one video of the set of semantically segmented training videos (

For each pair of ), the internal state of the cyclic layer is adapted according to the estimated optical flow between the frames of the pair of consecutive frames, so that the internal state adapts to the motion of the pixels between the pair of frames ( adapt), warping, and learning at least the parameters of the recursive module. A method for semantic segmentation of an input video using a convolutional recurrent neural network trained by-
The consecutive frames (

The pair of) includes a previous frame ( t -1) and a current frame ( t ), and the estimated optical flow is in the reverse direction from the current frame (t ) to the previous frame ( t -1) ( backward) optical flow (

)ego;
The step of warping includes a warping function in an internal state ( h _t-1 ) associated with the previous frame (t -1), and the previous frame ( t -1) corresponding to the internal state ( h _t-1). The warped internal state associated with (

In order to obtain ), it is a step of applying, and each pixel has the reverse optical flow (

) That has undergone displacement,
(a) successive frames of the input video (

For each pair of ), estimating the optical flow between the frames of the pair;
(b) successive frames of the input video (

For each pair of ), warping the internal state of the cyclic layer according to the optical flow estimated in (a) so that the internal state adapts to the motion of pixels between the frames of the pair; And
(c) successive frames of the input video (

For each pair of ), semantically segmenting each frame with the convolutional recurrent neural network
Including a method of semantic segmentation of the input video. The method of claim 14,
The training of the convolutional recurrent neural network includes training a second convolutional neural network using a set of training pairs of consecutive frames whose optical flow is known, wherein (a) is the Semantic segmentation method of input video, performed using a second convolutional neural network. The method of claim 15,
The second convolutional neural network is a FlowNetSimple network, a method for semantic segmentation of input video. The method of claim 15,
The second convolutional neural network is a FlowNetCorrelation network, a method for semantic segmentation of input video. In a method for semantic segmentation of input video using a convolutional recurrent neural network, performed by a computer,
(a) successive frames of the input video (

For each pair of ), estimating the optical flow between the frames of the pair;
(b) successive frames of the input video (

For each pair of ), warping the internal state of the cyclic layer according to the estimated optical flow, such that the internal state adapts to the motion of pixels between the frames of the pair; And
(c) successive frames of the input video (

For each pair of ), semantically segmenting each frame with the convolutional recurrent neural network
Including,
The convolutional recurrent neural network is trained according to the following training method,
The training method,
(a') training a first convolutional neural network using a set of semantically segmented training images; And
(b') training a convolutional recurrent neural network built based on the first convolutional neural network using a set of semantically segmented training videos-a second from the end of the first convolutional neural network ( penultimate) convolutional layer is replaced by a circular module with hidden state-
Including,
Training the convolutional recurrent neural network includes consecutive frames of one video of the set of semantically segmented training videos (

For each pair of ),
(b1') adapt the internal state of the cyclic layer according to the estimated optical flow between frames of a pair of consecutive frames, and the internal state adapts to the motion of pixels between the pair of frames, Warping; And
(b2') learning at least the parameters of the cyclic module
Including,
In the training method, the consecutive frames (

The pair of) includes a previous frame ( t -1) and a current frame ( t ), and the estimated optical flow is in the reverse direction from the current frame (t ) to the previous frame ( t -1) ( backward) optical flow (

)ego;
The (b1 ') is the previous frame corresponding to the warping function in the previous frame (t -1) internal state (h _t-1), the internal state (h _t-1) and the associated (t -1 ) And associated warped internal state (

In order to obtain ), it is a step of applying, and each pixel has the reverse optical flow (

A method of semantic segmentation of the input video, which is to have undergone displacement according to ). The method of claim 18,
The training of the convolutional recurrent neural network includes training a second convolutional neural network using a set of training pairs of consecutive frames whose optical flow is known, wherein (a) is the Semantic segmentation method of input video, performed using a second convolutional neural network. The method of claim 19,
The second convolutional neural network is a FlowNetSimple network, a method for semantic segmentation of input video. The method of claim 19,
The second convolutional neural network is a FlowNetCorrelation network, a method for semantic segmentation of input video. In a system for training a convolutional recurrent neural network for semantic segmentation of videos,
The system,
(a) training a first convolutional neural network using a set of semantically segmented training images,
(b) Using a set of semantically segmented training videos, train a convolutional recurrent neural network built on the basis of the first convolutional neural network-and a penultimate at the end of the first convolutional neural network. The convolutional layer is replaced by a circular module with a hidden state -,
Training the convolutional recurrent neural network includes successive frames of one video of the set of semantically segmented training videos (

For each pair of ),
(b1) warping the internal state of the cyclic layer according to the estimated optical flow between frames of a pair of consecutive frames, so that the internal state adapts to the motion of pixels between the pair of frames. and,
(b2) learning at least the parameters of the recursive module
Including,
The consecutive frames (

The pair of) includes a previous frame ( t -1) and a current frame ( t ), and the estimated optical flow is in the reverse direction from the current frame (t ) to the previous frame ( t -1) ( backward) optical flow (

)ego;
Wherein (b1) is the previous frame (t -1) corresponding to the warping function, the internal state (h _t-1), the internal state (h _t-1) in association with the previous frame (t -1) The warped internal state associated with (

In order to obtain ), it is to apply, and each pixel has the reverse optical flow (

A method of training a convolutional recurrent neural network that has undergone displacement according to ). In a program stored in a computer-readable recording medium to execute a method of training a convolutional recurrent neural network for semantic segmentation of videos in a computer,
The above method,
(a) training a first convolutional neural network using a set of semantically segmented training images; And
(b) training a convolutional recurrent neural network constructed based on the first convolutional neural network using a set of semantically segmented training videos-a second (penultimate) from the end of the first convolutional neural network ) The convolution layer is replaced by a circular module with a hidden state-
Including,
Training the convolutional recurrent neural network includes consecutive frames of one video of the set of semantically segmented training videos (

For each pair of ),
(b1) warping the internal state of the cyclic layer according to the estimated optical flow between frames of a pair of consecutive frames, so that the internal state adapts to the motion of pixels between the pair of frames. The step of doing; And
(b2) learning at least the parameters of the recursive module
Including,
The consecutive frames (

The pair of) includes a previous frame ( t -1) and a current frame ( t ), and the estimated optical flow is in the reverse direction from the current frame (t ) to the previous frame ( t -1) ( backward) optical flow (

)ego;
Wherein (b1) is the previous frame (t -1) corresponding to the warping function, the internal state (h _t-1), the internal state (h _t-1) in association with the previous frame (t -1) The warped internal state associated with (

In order to obtain ), it is a step of applying, and each pixel has the reverse optical flow (

A program stored on a computer-readable recording medium that has undergone displacement according to ).

Images