KR101563569B1 - Learnable Dynamic Visual Image Pattern Recognition System and Method - Google Patents

Abstract

The present invention relates to a system and a method for recognizing a pattern of a dynamic visual image for learning. The system according to an embodiment of the present invention uses a multiple spatio-temporal scales neural network (MSTNN) to recognize a pattern of a dynamic visual image through preliminary training on a set of exemplary patterns. In the system, a pixel pattern or a pattern of a dynamic visual image for sequence of visual characteristics is inputted into the MSTNN and a recognition result of the inputted pattern of a dynamic visual image is obtained in an output unit by a delayed response method.

Description

[0001] The present invention relates to a dynamic visual image pattern recognition system,

The present invention relates to an information processing method for visual image pattern recognition, and more particularly, to a visual image pattern recognition method in a video camera stream using a neural network.

In recent years, learning models have begun to be applied to many vision applications, including human behavior awareness. The existing vision method uses handcrafted features such as HOG, SIFT, SURF, etc., while the learning model can automatically learn features from the data.

One of the most popular learning models is CNN (Convolutional Neural Network). However, CNN can handle only static vision, and CNN itself can not handle dynamic vision. Some models that extend CNN like 3D CNN can handle dynamic vision for a short time but still have difficulty handling dynamic vision for a long time.

Therefore, it is required to search for a technique that can overcome the limitation of CNN.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a dynamic visual recognition method and system with superior performance that complements the limitations of CNN.

According to another aspect of the present invention, there is provided a method for recognizing a dynamic image pattern, the method comprising: receiving an image; At least one second subnetwork sequentially convoluting the input image; And the third subnetwork outputting an image recognition result via convolution, the at least one second subnetwork having different time-scale dynamics. Here, each neural unit of a sub-network of each level is connected top-down or bottom-up one level with local neural units that are retino-close to one another in another sub-network , There is a side connection between local neural units within the same subnetwork. Also, the at least one third sub-network may be composed of a single layer as well as a plurality of layers.

The time scale of the second subnetwork adjacent to the first subnetwork may be fast, and the time scale of the second subnetwork adjacent to the third subnetwork may be slow.

Also, the second subnetwork may comprise a number of feature maps.

The neural units in the subnetwork may be arranged in a two-dimensional retinotopic scheme.

In addition, the subnetworks have a wider neighbors' receptive field (i.e., a wider viewing range), while the number of neural units decreases and resolution decreases as the level increases.

And, in a sublevel of a certain level, each neural unit may be connected top-down or bottom-up with local neural units that are retino-close to one another in a subnetwork .

Also, within the same sub-network, the local neural units can be bilaterally connected.

The training and recognition by the subnetworks may be performed in a delayed response manner.

In addition, the training by the sub-networks may use an error back-propagation scheme scheme.

According to another aspect of the present invention, there is provided a dynamic image pattern recognition system including: an input unit for receiving an image; A recognition unit for sequentially convoluting images input through the input unit; And an output unit for outputting image recognition results through convolution. The recognition unit sequentially convolutes images input to the at least one sub-network having different time-scale dynamics.

As described above, according to embodiments of the present invention, information processing using a self-organizing spatiotemporal hierarchy of MSTNN enables a robust, robust and efficient recognition of complex dynamic image image patterns.

1 is a diagram illustrating an MSTNN architecture,
Figure 2 is a diagram provided for a description of the forward dynamics of the MSTNN,
FIG. 3 is a diagram illustrating a concept of a learning dynamic visual image pattern recognition process using an MSTNN,
FIG. 4 is a diagram illustrating a learning process using an MSTNN,
FIG. 5 is a flowchart showing a plurality of video sequence pattern training processes,
6 is a flowchart showing a video sequence recognition process,
7 is a block diagram of a learning dynamic visual image pattern recognition system.

Hereinafter, the present invention will be described in detail with reference to the drawings.

One. MSTNN  architecture

The Multiple Spatio-Temporal Scales Neural Network (MSTNN) architecture is shown in FIG. As shown in FIG. 1, the MSTNN architecture is composed of one input layer (Layer 1), three convolution layers (Layer 2-4), and one all-connection layer (Layer 5) .

On the other hand, the total-connection layer (Layer 5) can be implemented in a plurality of, but not limited to, one.

The standard Convolutional Neural Network (CNN) has a subsampling layer between the convolution layers that extracts small distortion and invariant features, significantly reducing computational complexity. However, the subsampling operation causes the discontinuity of the neuron values in the sub-sampling process over time.

In MSTNN, the convolution and subsampling operations are combined into one operation performed in the convolution layer to remove the subsampling operation.

Layer 1 is an input layer that has one 48x54 feature map that contains the input image.

Layer 2 is a convolution layer that has a step size (or stride) of 2 (which indicates how much the kernel for the next convolution is to be shifted) and six 22x22 feature maps with a time constant of two. The time constant is a parameter indicating a time scale characteristic. Small time constants represent short time scales, and large time constants represent long time scales. In the model according to an embodiment of the present invention, the convolution layer and the all-connection layer have a certain time constant value. That is, in the embodiment of the present invention, only the convolution layer has a time scale, and when it is generalized, the convolution layer as well as the all-connection layer may have a time scale. Each feature map in layer 2 is linked to a feature map in layer 1 with a 6x12 kernel.

Layer 3 is a convolution layer that has 50 8x8 feature maps with a step size of 2 and a time constant of 5. Each feature map of layer 3 is linked to a feature map of layer 2 with an 8x8 kernel.

Layer 4 is a convolution layer that has 100 1x1 feature maps with a step size of 1 and a time constant of 100. Each feature map of layer 4 is linked to a feature map of layer 3 with an 8x8 kernel.

Layer 5 is a fully-connected layer that uses softmax as an activation function for classification. On the other hand, it is also possible to use another function (for example, tanh) other than the soft max function. The number of neurons in layer 5 is equal to the number of classes in the data set. Each neuron in layer 5 is connected to all the neurons in layer 4's 100 feature maps. Activated neurons in layer 5 are the result of classification.

2. Forward dynamic ( Forward Dynamics )

로 표기하며, 다음과 같이 계산된다.At time t, the internal state of the neuron at the location (x, y) of the mth feature map of the lth layer is

And is calculated as follows.

은 (l-1) 번째 레이어의 n 번째 특징 맵으로부터 현재 특징 맵까지 연결된 커널의 (p, q)에서의 값이고,

는 시간 단계 t에서 (l-1) 번째 레이어의 n 번째 특징 맵의 (x,y)에서 뉴런의 활성화 값이며,

은 현재 특징 맵에 대한 바이어스이다.Where I _C is a set of convolutional layer indexes, I _F is a set of all-connected layer indices, τ _l is the time constant of the i th layer, N _(l-1) P _l and Q _l are the height and width of the kernel in the lth layer, S _l is the step size of the lth layer,

Is a value at (p, q) of the kernel connected from the n-th feature map of the (l-1) -th layer to the current feature map,

Is the activation value of the neuron at (x, y) of the nth feature map of the (l-1) th layer at time step t,

Is the bias for the current feature map.

The kernel is shifted by S _l pixels in both horizontal and vertical directions prior to convolution. The mathematical expressions for both the convolution and the all-connection layer can be expressed in the same way by defining neurons and weights of the all-connection layer as 1x1 feature maps and 1x1 kernel, respectively, and setting S _l to 1.

The feature maps of the top-down, bottom-up, or bilateral layers, and the neural network of the neurons, rather than the convolution firing rate model, where activation of each neuron is determined only by the current input, An integrate-and-fire model is used in which the activation of each neuron is calculated by propagating the convoluted and reduced previous internal state.

The time constant τ represents the degree to which the history of the past input affects the current internal state. If τ is large, the activation of the neurons changes slowly, because the internal state is strongly influenced by the history of the past input versus the current input.

On the other hand, if τ is small, the activation of the neurons changes rapidly, as shown in FIG. 2, because the current input more strongly affects the internal state.

은 다음과 같이 계산된다.Activation value of neurons

Is calculated as follows.

Here, L is the number of layers in the model.

3. Training method ( Training Method )

The error function E is determined using Kullback-Leibler divergence as follows:

은 표기법을 단순화하기 위해 시간 단계 t에서 시퀀스 주어진 경우 클래스 m 의 신뢰도를 나타내는

로부터 재정의한 것이며,

는 라벨 값이다.Where T is the length of the sequence, d is the length of the label sequence,

Denotes the reliability of a class m given a sequence at time t to simplify the notation.

And,

Is the label value.

는 1로 설정되고,

인 나머지는 0으로 설정된다. 모델이 지연된 교육을 따르기 때문에, 시퀀스의 마지막 d 시간 단계에서만 에러가 발생한다. 시간 단계 t에서 생성된 에러 때문에 E_t가 이전 시간 단계로 전파되지 않으므로, CNN과 같은 피드 포워드 뉴럴 네트워크 모델은 지연된 교육을 사용할 수 없다.If the input sequence belongs to class c,

Is set to 1,

Is set to zero. Because the model follows a delayed training, errors occur only in the last d time step of the sequence. Feed-forward neural network models such as CNN can not use delayed training because E _t is not propagated to previous time steps due to errors generated at time step t.

로 전파된다.The iterative neural network model can propagate error to the previous time step, but the error is quickly attenuated through this time. However, the proposed model has a large time constant at the upper layer that reduces the attenuation of the propagated error, allowing the correct amount of errors generated at the end of the sequence to be propagated back to the initial time step. For example, assume that the sequence has 100 frames and the time constant of the upper layer is set to 100. [ In this case, the error E ₁₀₀ occurring in the time step ₁₀₀ is

.

In the training phase, all learning parameters for the model at step n are updated with the following equation:

은 다음과 같이 주어진다.Here,? Is the learning rate. Partial differentials of learning parameters solved by existing back-propagation

Is given as follows.

이고,

이며, X_l 및 Y_l은 l 번째 레이어에서 특징 맵의 높이와 폭이다.here,

ego,

And X _l and Y _l are the height and width of the feature map in the lth layer.

를 만족하는 파라미터

에 의해 조정된다. 델타 에러 스케일은 출력 뉴런의 개수 N_L 및 라벨 시퀀스의 길이 d에 의존 하기 때문이다.In the course of training,

≪ / RTI >

. Since the delta error scale depends on the number N _L of output neurons and the length d of the label sequence.

은 0.1로 설정된다. 훈련을 가속화하기 위해, 단계 n에서 평균 제곱 오차가 단계 n-1에서 평균 제곱 오차 보다 작은 경우,

에 1.05를 곱하고, 그렇지 않으면

를 2로 나눈다.parameter

Is set to 0.1. To speed up the training, if the mean squared error at step n is less than the mean squared error at step n-1,

Multiplied by 1.05, otherwise

Is divided by 2.

는 0으로 설정된다.All kernel weights and biases are initialized to randomly selected values from the Gaussian distribution with a standard deviation of 0.05. The initial state of neurons in the convolution layer

Is set to zero.

The remaining parameters, including the number of layers, the number of feature maps, the feature map size, the kernel size, the step size, and the time constant have already been defined in the Model Architecture section.

4. Learning dynamic visual image pattern recognition Learnable Dynamic Visual Image  Pattern Recognition )

The MSTNN processes information that changes rapidly in time at a lower level and spatially in a higher level but uses information in a narrower receptive field at a lower level by using a hierarchical structure, Spatially processes information with a low resolution but a wide receptive field. Therefore, the spatial information of various scales inherent in the data can be successfully extracted or processed while the information is processed from the lower level to the higher level.

As shown in FIG. 3, the MSTNN consists of several levels of subnetworks in which neural activity is controlled by the dynamics of different time scales. The time scale of the lth level is determined by the time constant parameter? _l set for the neural units at the level. The activation dynamics of each neural unit follows the next differential equation using the time constant parameter τ _l .

Here, u _i is the potential of the ith unit, w _ij is the connection weight from the j th unit to the ith unit, a _i is the activation value of the i th unit, I _k is the k th external input value, f () Is a nonlinear function such as a sigmoid function.

The lowest level receives the video frame input sequence.

After the lowest level, several levels of subnetworks are deployed, from the fastest dynamic subnetwork with the smallest time constant to the slowest dynamic subnetwork with the largest time constant.

Each subnetwork consists of several feature maps. At the top of the slowest dynamic subnetwork, the output layer is located and outputs the recognition result for the video frame input sequence. The output layer (output portion) may be a multilayer including a plurality of layers as well as a single layer.

In addition to the constraints of the various time scales described above, there are constraints on various spatial scales that apply to all networks, with specific connections for each level and allocation of retinotopic resolution.

In each level of subnetworks, neural units are arranged in a two-dimensional retinotopic scheme. Each neural unit in a particular level of subnetwork is connected top-down or bottom-up with local neural units that are retino-close to one another in the other subnetwork. There is also a side connection between local neural units within the same subnetwork.

That is, in addition to the top-down connection, it can be expanded to a bottom-up connection and a bilateral connection within the same layer.

As the number of neuronal units decreases as the level increases, the receptive field of the neural unit becomes wider and the resolution decreases accordingly.

The model architecture is constructed by applying the idea of several space-time scales, allowing slow dynamics to maintain a higher level of global connectivity and fast dynamics maintaining local peripheral connections at low level acceptance fields.

This configuration enables complex dynamic visual pattern recognition by processing through multiple levels of abstraction.

5. Learning and recognition processing ( Learning and recognition processes )

As shown in FIG. 4, the MSTNN is trained for a video sequence set associated with the training output sequence. An end-point display (e.g., a blank image) is added to the training row image sequence.

The training output sequence is provided in the form of a delayed response. It means that the training output can have an intermediate value for all the output units before the end indication is received. After the end display, a training output unit indicating an accurate recognition result is activated.

This is because, after successful training, it is possible to recognize the same category of visual image sequences and expect the same output unit to be active after the end mark.

5 is a flowchart illustrating a plurality of video sequence pattern training processes. 5, a training sequence for the input sequence is acquired (S110), an end mark is added to the end of the training sequence (S120), and Back-Propagation Through Time (BPTT) training is performed (S130) . Thereafter, the next sequence is added (S140), and the process is resumed from the step S110.

After network training, video sequence recognition is performed as shown in Fig. 6 is a flowchart illustrating a video sequence recognition process. As shown in FIG. 6, a target sequence is obtained (S210), an end mark is added to the end of the target sequence (S220), and a dynamic operation for acquiring an output sequence is performed for target sequence recognition (S230) .

7 is a block diagram of a learning dynamic visual image pattern recognition system. 7, the system according to the embodiment of the present invention includes an input unit 310, a learning unit 320, a recognition unit 330, and an output unit 340.

The input unit 310 receives the video sequence, and the learning unit 320 performs learning according to the algorithm shown in FIG. The recognition unit 330 performs recognition according to the algorithm shown in FIG. 6, and the output unit 340 outputs a learning result and a recognition result.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

Layer 1: Input Layer
Layer 2-4: Convolution Layer
Layer 5: Full-Link Layer

Claims (9)

The first sub-network receiving the image;
At least one second subnetwork comprises convoluting the input image with characteristic maps of top-down, bottom-up and bilateral layers; And
The third sub-network outputting the image recognition result through the convolution,
Wherein the at least one second sub-
With different timescale dynamics,
In the subnetworks,
Dimensional retinotopic technique. ≪ Desc / Clms Page number 19 >
The method according to claim 1,
The time scale of the second subnetwork adjacent to the first subnetwork is fast,
And the time scale of the second subnetwork adjacent to the third subnetwork is slow.
The method according to claim 1,
Wherein the second sub-network comprises a plurality of feature maps.
delete The method according to claim 1,
The sub-
Wherein the number of neural units is reduced and the resolution is reduced as the level is increased, but the receptive field is widened.
6. The method of claim 5,
Each neural unit in a particular level of subnetwork is characterized in that one level is top-down or bottom-up connected to local neural units that are retino-close in other subnetworks A dynamic image pattern recognition method.
The method according to claim 6,
Wherein in the same sub-network, the local neural units are connected laterally.
8. The method of claim 7,
Wherein the training and recognition by the subnetworks is performed in a delayed response manner.
An input unit for receiving an image;
A recognition unit for convoluting the image input through the input unit with characteristic maps of top-down, bottom-up and bilateral layers; And
And an output unit for outputting an image recognition result through convolution,
Wherein,
Sequentially convolving images input into at least one sub-network having different time-scale dynamics,
In the subnetworks,
Dimensional dynamic pattern recognition system is arranged as a two-dimensional retinotopic technique.

Images