WO2021255514A1

WO2021255514A1 - Padding method for convolutional neural network layers adapted to perform multivariate time series analysis

Info

Publication number: WO2021255514A1
Application number: PCT/IB2020/061237
Authority: WO
Inventors: Rui Jorge PEREIRA GONÇALVES; Fernando Manuel FERREIRA LOBO PEREIRA
Original assignee: Universidade Do Porto
Priority date: 2020-06-15
Filing date: 2020-11-27
Publication date: 2021-12-23

Abstract

The present invention is enclosed in the field of Convolutional Neural Networks configured to perform Multivariate Time Series analysis, and relates to a padding method specially conceived for configuring convolutional neural network layers adapted to perform Multivariate Time Series analysis on a bidimensional input map (1). The padding method involves mapping the input map (1) onto a cylinder (2) by adding a predefined number of wrapped rows, depending on the vertical size of the convolutional kernel (3) to be used in the subsequent convolutional operation.

Description

DESCRIPTION

PADDING METHOD FOR CONVOLUTIONAL NEURAL NETWORK LAYERS ADAPTED TO PERFORM MULTIVARIATE TIME SERIES ANALYSIS

FIELD OF THE INVENTION

The present invention is enclosed in the area of Convolutional Neural Networks. In particular, the present invention relates to the field of Convolutional Neural Networks configured to perform Multivariate Time Series analysis .

PRIOR ART

A time series is a continuous sequence of observations taken repeatedly, normally at equal intervals, over time. The relationship between past and future observations can be stochastic or non-deterministic. In some Multivariate Time Series (MTS) studies, it is normal for the training and test datasets to be composed by observations/examples of independent time series segments with all the available time distributed information such that the context changes from example to example. Although the Long Short-Term Memory, which is a particular type of Recurrent Neural Network, is more suitable for MTS analysis from a theoretical point of view, the Convolutional Neural Networks (CNN) has gained popularity to analyse this particular type of problems (i.e. segmented MTS). Some studies have applied CNNs in this domain, and a good example is WaveNet from Google DeepMind [1] applied to audio signal generation .

When the convolution kernel is larger than (1,1), then the convolution result is smaller than the original output. Usually, this is not a concern for inputs with large dimensions, i.e. images, and small filters. However, it can constitute a major problem with small input dimensions or when considering a high number of staked convolutional layers. As such, the practical effect of large filter sizes and/or very deep CNNs on the size of the resulting feature map entails loss of information such that the model can simply run out of data upon which it operates.

An important mechanism used in convolution operations to tackle this issue is padding, which is a process used for the border treatment of data before the convolution operation in a CNN. Through this mechanism, the border treatment by the convolution kernel in the input space is pre-processed in order to retain as much original meaningful information as possible at the output level. Currently, the standard procedure to avoid the border effect problem consists of including of zeros outside of the input map, in a number of rows above the top row and below the bottom row, and in a number of columns on the left of the first column and on the right of the last column of the input map. In this way, the convolution output size will have the same spatial extent as the input.

Note, however, that if the goal is to analyse a MTS problem, the input feature map has a relatively small size in the variables dimension. Therefore, the inclusion of zeros through the same padding approach implies a weaker learning capability since the learned kernel is affected by the dot product operation with the included zero values, thus, potentially promoting an erroneous generalization. In this regard, other known padding methods are commonly provided in image processing environments, that make use of the information in the input map to fill in the borders. Examples of padding mechanisms are Valid padding - or no padding -, Same padding, Casual padding, Constant n' padding, Reflection padding or Tile 2 padding.

Solutions exist in the art, such as the document US2020285963, which describes a padding method for a convolutional neural network. The method includes receiving concentrically configured data of an object, the concentrically configured data correlating with an image, which was recorded concentrically to the object, deconvolving the concentrically configured data to form a data array, including real-coherent data on opposite sides of the data array, carrying out a convolution operation by using ring padding. In the case of ring padding, the real- coherent data of one side of the data array is utilized for padding the real-coherent data of a side of the data array opposite thereto, and/or vice versa. However, the method described in the referred document is applied only to concentrically configured data and not to MTS analysis which is non-concentrically information by nature.

Document CN106447030A describes a method for optimizing a computational resource of a convolutional neural network, wherein a padding method is used to repeat information of the bidimensional input.

Document CN104794548A describes a parallelization calculation method based on ARIMA (Autoregressive integrated moving average) model load prediction, applicable to data correlation of time series data. It is proposed a multi threading technology, able to analyse the characteristics of the power load data, and to use the ARIMA model for parallelization processing.

However, all the existing solutions are silent regarding the applicability of the respective padding methods to MTS analyses, and how the accuracy of the CNN architecture can be improved in the subsequent convolution operation.

The present solution intended to innovatively overcome such issues.

SUMMARY OF THE INVENTION

The present application describes a new padding mechanism, which is specially conceived for MTS analysis, wherein the bidimensional input map is defined by a variables dimension and a time-steps dimension. Such padding mechanism when included in a standard CNN architecture plays an important role in the overall performance of a CNN based model for MTS classification, improving its accuracy.

It is therefore an object of the present invention a padding method, for configuring convolutional neural network layers adapted to perform MTS analysis on a bidimensional input map defined by a variables dimension and a time-steps dimension.

It is also an object of the present invention a processing system operable to execute the padding method developed.

A method for operating such processing system is also provided.

DESCRIPTION OF FIGURES

Figure 1 - representation of a MTS bidimensional input map (1), defined by a variables dimension and a time-steps dimension.

Figure 2 - representation of the roll padding scheme of the invention applied to MTS analysis, wherein the reference signs represent: 1 - MTS bidimensional input map;

2 - mapped bidimensional input map onto a cylinder;

3 - convolutional kernel matrix with a horizontal size K_H and a vertical size K_w.

Figure 3 - representation of the roll padding method of the invention, wherein the time-steps dimension remains with no padding (i.e. valid padding).

Figure 4 - representation of the roll padding scheme of the invention method of the invention, wherein causal padding is applied to the time-steps dimension.

Figure 5 - representation of the application of the roll padding method of the invention to a MTS bidimensional input map.

DETAILED DESCRIPTION

The more general and advantageous configurations of the present invention are described in the Summary of the invention. Such configurations are detailed below in accordance with other advantageous and/or preferred embodiments of implementation of the present invention.

The present invention relates to a padding method, designated for the purpose of the present description as roll padding, specially conceived for configuring convolutional neural network layers adapted to perform MTS analysis on a MTS bidimensional input map (1) defined by a variables dimension and a time-steps dimension, as can be seen in figure 1.

In particular, the characteristics defining the roll padding method now developed, creates a significant difference between the known padding methods from the state of the art, by allowing the MTS bidimensional input map (1) to be mapped onto a cylinder (2). The configuration of CNN layers with this padding method allows a maximization of the levels of intercorrelation between input variables during the convolution operation, providing an improvement in the accuracy of standard CNN architectures in performing MTS analysis .

Figures 2 and 3 help to understand how the roll padding method is applicable to the MTS bidimensional input map (1). The roll padding method copies information from the opposite sides of the bidimensional input map (1) but only in one dimension, which is the variables dimension in the MTS bidimensional input map (1), effectively mapping the MTS bidimensional input map (1) onto a cylinder (2). The wrapped rows above the top of the MTS bidimensional input map (1) are a copy of the bottom rows MTS bidimensional input map (1), respectively and vice-versa. On the right and left columns of the MTS bidimensional input map (1), no padding, i.e. valid padding, is applied. The reduction of time-steps dimension after several convolutions is not problematic due to the frequent presence of a high number of time-steps in this type of MTS problems. Nevertheless, for the time-steps dimension, roll padding method can be combined with other types of padding methods, such as causal padding, as can be seen in figure 4.

The use of the roll padding method of the invention avoids the variable dimension reduction throughout a Deep neural network with convolution layers, if no padding is applied, and correctly addresses the issue of convolution kernel correlations on the borders of the MTS bidimensional input map (1). Therefore, by applying the roll padding method in the variable dimension before the convolution operation will enable the convolution kernel (3) to search patterns that correlate the first time series variables with the last variables in the MTS bidimensional input map (1). This roll padding method not only provides a way to apply deep convolution-based layers neural networks architectures to MTS problems, as also ensures an accurate performance in the analysis .

In figure 5 it is sketched an example of the application of the roll padding method of the invention. In this example the MTS bidimensional input map has a size of 72 time-steps x 8 variables. By applying the roll padding method, it is possible to increase the variables dimension from 8 to 12, by copying information contained in the two rows from bottom of the map (1) in to two wrapped rows to be added above the top row of the map (1), and vice-versa, since the kernel (3) size in the next convolutional layer to be processed is of size (5,5). Note that the final output, after the convolution, has been reduced to size 8 in the variables dimension, the same as original input.

EMBODIMENTS

The roll padding method of the present invention comprises the steps of mapping a MTS bidimensional input map (1) onto a cylinder (2), by generating at least one wrapped row above the top row of the MTS bidimensional input map (1) and at least one wrapped row below the bottom row of the MTS bidimensional input map (1). More particularly, the wrapped rows added above the top row of the MTS bidimensional input map (1) are padded with copies of information contained in the variables dimension of bottom rows of the MTS bidimensional input map (1). The wrapped rows added below the bottom row of the MTS bidimensional input map (1) are padded with copies of information contained in the variables dimension of top rows of the MTS bidimensional input map (1).

In a particular embodiment of the padding method developed, the wrapped rows added above the top row of the MTS bidimensional input map (1) are padded with a copy of the bottom rows of the MTS bidimensional input map (1). The wrapped rows added below the bottom row of the MTS bidimensional input map (1) are padded with a copy of the top rows of the MTS bidimensional input map (1). Alternatively, in a particular embodiment of the padding method developed, the wrapped rows added above the top row of the MTS bidimensional input map (1) are a copy in the inverse order of the bottom rows of the MTS bidimensional input map (1). The wrapped rows added below the bottom row of the MTS bidimensional input map (1) are padded with a copy in the inverse order of the top rows of the MTS bidimensional input map (1).

In another particular embodiment of the padding method developed, the number of wrapped rows added above the top row and below the bottom row of the MTS bidimensional input map (1) is dependent of the vertical size, K_w, of a convolutional kernel (3) to be applied to the neural convolutional network layer. More particularly, and in another embodiment of the padding method, the number of wrapped rows is equal to

. In another particular embodiment of the padding method developed, no columns are added on the left of the first column of the MTS bidimensional input map (1), and no columns are added on the right of the last column of the MTS bidimensional input map (1). Alternatively, and in another embodiment of the padding method developed, a padding mechanism is applied to fill a number of columns added on the left of the first column of the MTS bidimensional input map (1) and a number of columns added on the right of the last column of the MTS bidimensional input map (1), using information contained in the time-steps dimension. The padding mechanism that can be used is one of the following: Same padding, Casual padding, Constant 'h' padding, Reflection padding or Tile 2 padding.

The present invention also relates to a processing system comprising processing means programmed to execute the padding method develop, in order to configure convolution neural network layers adapted to perform MTS analysis on a MTS bidimensional input map (1).

In one embodiment of the processing system, it further comprises processing means adapted to implement a neural network architecture configured to execute a convolution operation based on convolutional network layers pre-processed using the padding method developed. The convolution operation is executed according to the following parameters : convolutional kernel (3) size ( K_H,K_W); and stride. The present invention also relates to a method for operating the processing system described above. The referred method comprising the steps of: ii. Configuring convolutional neural network layers adapted to perform MTS analysis on a MTS bidimensional input map (1), by executing the roll padding method developed; and iii. Implementing a convolution operation to each layer.

In one embodiment of the method the convolution operation is performed according to the following parameters: convolutional kernel (3) size (K_H,K_W); and stride.

In another embodiment of the method the roll padding method is dependent of the vertical size, K_w, of the convolutional kernel.

As will be clear to one skilled in the art, the present invention should not be limited to the embodiments described herein, and a number of changes are possible which remain within the scope of the present invention.

Of course, the preferred embodiments shown above are combinable, in the different possible forms, being herein avoided the repetition all such combinations.

REFERENCES

[1] - Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen

Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio, 2016.

Claims

1. Padding method for convolutional neural network layers adapted to perform Multivariate Time Series analysis on a Multivariate Time Series bidimensional input map (1) defined by a variables dimension and a time-steps dimension; the padding method characterised by mapping the Multivariate Time Series bidimensional input map (1) onto a cylinder (2), through the following step: i. generating at least one wrapped row above the top row of the Multivariate Time Series bidimensional input map (1) and at least one wrapped row below the bottom row of the Multivariate Time Series bidimensional input map (1); wherein, wrapped rows added above the top row of the Multivariate Time Series bidimensional input map (1) are padded with copies of information contained in the variables dimension of bottom rows of the Multivariate Time Series bidimensional input map (1); and wrapped rows added below the bottom row of the Multivariate Time Series bidimensional input map (1) are padded with copies of information contained in the variables dimension of top rows of the Multivariate Time Series bidimensional input map (1).

2 . Padding method according to claim 1, wherein the wrapped rows added above the top row of the

Multivariate Time Series bidimensional input map (1) are padded with a copy of the bottom rows of the Multivariate Time Series bidimensional input map (1); and the wrapped rows added below the bottom row of the Multivariate Time Series bidimensional input map (1) are padded with a copy of the top rows of the Multivariate Time Series bidimensional input map (1).

3. Padding method according to claim 1, wherein the wrapped rows added above the top row of the

Multivariate Time Series bidimensional input map (1) are a copy in the inverse order of the bottom rows of the Multivariate Time Series bidimensional input map (1); and the wrapped rows added below the bottom row of the Multivariate Time Series bidimensional input map (1) are padded with a copy in the inverse order of the top rows of the Multivariate Time Series bidimensional input map (1).

4. Padding method according to any of the previous claims, wherein the number of wrapped rows added above the top row and below the bottom row of the Multivariate Time Series bidimensional input map (1) is dependent of the vertical size, K_w, of a convolutional kernel (3) to be applied to the neural convolutional network layer.

5. Padding method according to claim 4, wherein the number of wrapped rows is equal to

.

6. Padding method according to any of the previous claims, wherein, no columns are added on the left of the first column of the Multivariate Time Series bidimensional input map (1); and no columns are added on the right of the last column of the Multivariate Time Series bidimensional input map (1).

7 . Padding method according to any of the previous claims 1 to 5, wherein a padding mechanism is applied to fill, a number of columns added on the left of the first column of the Multivariate Time Series bidimensional input map (1); and a number of columns added on the right of the last column of the Multivariate Time Series bidimensional input map (1), using information contained in the time-steps dimension.

8. Padding method according to claim 7, wherein the padding mechanism is one of the following: Same padding, Casual padding, Constant n' padding, Reflection padding or Tile 2 padding.

9. Processing system comprising processing means programmed to execute the padding method of claims 1 to 8 to configure convolution neural network layers adapted to perform Multivariate Time Series analysis on a Multivariate Time Series bidimensional input map (1) defined by a variables dimension and a time-steps dimension.

10 . Processing system according to claim 9 further comprising processing means adapted to implement a neural network architecture configured to execute a convolution operation based on convolutional network layers pre-processed by the padding method; said convolution operation being executed according to the following parameters: convolutional kernel (3) size ( K_H,K_W); and stride.

11. Method for operating the processing system of claims 9 and 10, comprising the following steps: i. Configure a convolutional neural network layer adapted to perform Multivariate Time Series analysis on a Multivariate Time Series bidimensional input map (1) defined by a variables dimension and a time-steps dimension, by executing the padding method of claims 1 to 8; ii. Implement a convolution operation to that layer.

12. Method according to claim 11, wherein the convolution operation is performed according to the following parameters: convolutional kernel (3) size ( K_H,K_W); and stride.

13. Method according to claims 11 and 12, wherein the padding method is dependent of the vertical size, K_w, of the convolutional kernel.