JP7041427B2 - Series convolutional neural network - Google Patents

Description

The techniques disclosed generally relate to neural networks, in particular to methods and systems for series convolutional neural networks.

Convolutional neural networks (CNNs) are known in the art. Such networks are typically employed for object detection and classification in images. A convolutional neural network (CNN) is typically constructed from one of many more layers. Operations are performed on each layer. Typically, this operation is one of a convolution operation and a multiplication by an activation function, and this operation may further include pooling, also referred to as downsampling.

For each layer, each set of meta-parameters is defined. These meta-parameters include the number of filters adopted, the size of the filter, the stride of the convolution, the downsampling ratio, the size of the downsampling size, its stride, and the activation function adopted. Here, reference is made to FIG. 1, which schematically represents a CNN known in the art and generally labeled with a reference numeral 10. The CNN 10 is employed to detect features in an image such as the image 16. The neural network 10 includes a plurality of layers such as layer 12 ₁ (FIG. 1). The CNN 10 includes a plurality of layers 12 ₁ , 12 ₂ , ..., 12 _N , and a classifier 14. The input image 16 is supplied to the layer ₁₂₁ . Layer 12 ₁ convolves the image 16 with at least its respective filters and multiplies each of the outputs of the filters with an activation function. Layer 12 ₁ provides its output to layer _{12 2} , which performs its respective computations with its respective filters. This process is repeated until the output of layer _12N is provided to the classifier 14. The output of layer 12N is a map of the features corresponding to the filters adopted in _CNN10 . This feature map relates to the probability that the feature is present in the input image 16 in each image window associated with the feature map. The feature map at the output of layer _12N may be embodied as multiple matrices, each corresponding to a feature, where the value of the entry in each matrix is the position of the entry in the matrix (ie, the index of the entry). Represents the probability that the input image 16 will contain features associated with that matrix within a particular associated image window (ie, the bounding box). The size of the image window is determined according to the number of layers in CNN10, the size of the kernel, and the stride of the kernel during the convolution operation.

The classifier 14 may be any type of classifier known in the art (eg, random forest classifier, support vector machine-SVM classifier, convolution classifier, etc.). The classifier 14 classifies objects trained to be detected by the CNN 10. The classifier 14 can provide classification information for each image window, along with the respective detection reliability level at which the object is located within the image window. In general, the output of the classifier 14 is a vector (s) of values related to the detection and classification of objects in the corresponding image window. This vector of values (s) is referred to herein as a "classification vector".

Here, reference is made to FIG. 2, which schematically represents an exemplary CNN known in the art and generally labeled 50. The CNN comprises two layers, a first layer 51 ₁ and a _second layer 512. The first layer 51 ₁ receives the image 52 as an input to it. In the _first layer 511 the convolution operation is performed and in the _second layer 512 the activation function is applied to the result of the convolution. The image 52 includes a matrix of pixels, where each pixel is associated with a value (eg, a gray level value) or a value (s) (eg, a color value). Image 52 can represent a scene containing objects (eg, a person walking in the street, a dog playing in a park, a vehicle in the street, and the like).

In the _first layer 51 ₁ , the image 52 is convoluted by _one of each of the filters 541 and 542. Filters ₅₄₁ and 542 _are also referred to as convolution kernels or simply kernels. Therefore, _each of the filters 541 and ₅₄₂ is shifted on the selected position in the image. At each selected position, the pixel values duplicated by the filter are multiplied by their respective weights of the filter and the results of this multiplication are summed (ie, multiplication and sum operation). Overall, the selected position is defined by shifting the filter on the image by a predetermined step size called a "stride". _Each of the filters 541 and ₅₄₂ corresponds to a feature that will be identified in the image. Stride, along with filter size, is a design parameter selected by the CNN designer. Folding the image 52 from _each of the filters 541 and 542 creates a feature map containing the _two feature images or a matrix, the feature images 56 ₁ and the feature images ₅₆₂ of the filters ₅₄₁ and 542, _respectively (ie). , Each image is created for each filter). Each pixel or entry in the feature image corresponds to the result of one multiplication and sum operation. Thus, each of the matrices 56 ₁ and 56 ₂ is associated with the respective image feature corresponding to each of the filters ₅₄₁ and ₅₄₂ . Also, each entry is associated with an image window for the input image 52. Therefore, the value of each entry in _each of the matrices 56 ₁ and 562 represents the feature intensity of the feature associated with it in the image window associated with it. It should be noted that the size of the feature images 56 ₁ and 56 ₂ (ie, the number of pixels) may be smaller than the size of the image 52. The output of the first layer 51 ₁ is provided to the _second layer 512. In the _second layer 521, each value in _each of the feature images 56 ₁ and 562 is then applied as an input to the activation function 58 (eg, sigmoid, gauss, and hyperbolic tanh, etc.). _The output of layer 521 is then provided to the classifier 60, which detects and classifies the objects in the image 52 and creates a classification vector for each entry in the feature map.

Before detecting and classifying objects in an image, the weights and parameters of various filters of the function adopted by the CNN, such as CNN10 (FIG. 1) or CNN50 (FIG. 2), need to be determined. Their weights and parameters are determined during the training process. The initial weights and parameters of the CNN (ie, before training is started) are determined as appropriate (eg, randomly). During training, the training image or image (s) in which the object is detected and classified is provided to the CNN as input. In other words, an image having each predetermined classification vector for each image window is provided to the CNN as an input. The layers of the CNN network are applied to each training image and the classification vector, each of each training image is determined (ie, the objects in it are detected and classified). Those classification vectors are compared with a predetermined classification vector. An error between the CNN classification vector and a predetermined classification vector (eg, sum of squares of differences, log loss, softmaxlog loss) is determined. This error is then employed to update the weights and parameters of the CNN in the backpropagation step, which may include one or more iterations.

The publication "A convolutional Neural Network Cascade for Face Detection", Li et al., relates to a CNN containing three pairs of networks. Each pair includes a classification (detection) network and a boundary box regression network. During detection, the image plumid is generated to allow multiscale scanning of the image. A first classification network (DET12) is then employed to scan all windows in the image and filter them for low reliability. A first boundary box regression network (CLB12) is employed to correct the position of all remaining windows. Non-maximal suppression is then applied to remove windows with high overlap. In the next step, a second classification network (DET24), followed by a second boundary box regression network (CLB24) that performs the boundary box regression, is employed to filter the remaining windows. Finally, a third classification network (DET48) is adopted, followed by a third boundary box regression network (CLB48).

The purpose of the disclosed technique is to provide new convolutional neural network methods and systems. Thus, according to the disclosed technique, a convolutional neural network system is provided that detects at least one object in at least one image. The system includes a plurality of object detectors corresponding to a predetermined image window size in at least one image. Each object detector is associated with its own downsampling ratio for at least one image. Each object detector includes a convolutional neural network and an object classifier coupled with the convolutional neural network. Each convolutional neural network contains multiple convolutional layers. The object classifier classifies objects according to the results from the convolutional neural network. The object detector associated with each identical downsampling ratio defines at least one group of object detectors. Object detectors in the group of object detectors are associated with a common convolution layer.

Thus, according to another aspect of the disclosed technique, a convolutional neural network method is provided that includes a procedure for downsampling an image according to a plurality of downsampling ratios in order to create a plurality of downsampled images. Each downsampled image is associated with a respective downsampling ratio. The method further comprises the procedure of detecting objects in a predetermined image window size for at least one image for each downsampled image by a corresponding convolutional neural network and classifying the objects in the image. A convolutional neural network that detects objects in each downsampled image associated with the same respective downsampling ratio defines at least one group of convolutional neural networks. A convolutional neural network in a group of convolutional neural networks is associated with a common convolutional layer.

The disclosed technology will be more fully understood and recognized from the following detailed description combined with the drawings.

CNNs known in the art are schematically represented. Schematic representation of exemplary CNNs known in the art. Schematic representation of a CNN system that detects objects in an input image according to embodiments of the disclosed technique. Schematic representation of a CNN system that detects objects in an input image according to embodiments of the disclosed technique. Schematic representation of an exemplary CNN system for detecting objects in an input image that is constructed and operational according to another embodiment of the disclosed technique. An image with an object in it, which is employed to determine a training set according to a further embodiment of the disclosed technique, is schematically represented. An image with an object in it, which is employed to determine a training set according to a further embodiment of the disclosed technique, is schematically represented. An image with an object in it, which is employed to determine a training set according to a further embodiment of the disclosed technique, is schematically represented. An image with an object in it, which is employed to determine a training set according to a further embodiment of the disclosed technique, is schematically represented. An image with an object in it, which is employed to determine a training set according to a further embodiment of the disclosed technique, is schematically represented. An image with an object in it, which is employed to determine a training set according to a further embodiment of the disclosed technique, is schematically represented. An image with an object in it, which is employed to determine a training set according to a further embodiment of the disclosed technique, is schematically represented. An image with an object in it, which is employed to determine a training set according to a further embodiment of the disclosed technique, is schematically represented. Schematic representation of a method of determining a training set for a neural network that is operable according to another embodiment of the disclosed technique. Schematic representation of a method for a CNN that is operable according to a further embodiment of the disclosed technique.

The disclosed technique eliminates the shortcomings of prior art by providing a CNN network system for detecting objects in an image. A CNN network according to the disclosed technology includes multiple object detectors. Each object detector is associated with a predetermined image window size in the image. Each object detector is associated with its own downsampling ratio at its input to the image. Each object detector contains at least each CNN containing multiple convolution layers. Each convolution layer convolves the input to it with multiple filters, and the result of this convolution is processed by the activation function. Each object detector further includes an object classifier coupled with a convolutional neural network that classifies the objects in the image according to the results from the convolutional neural network. The object detector associated with each identical downsampling ratio defines at least one group of object detectors. The object detectors in the group of object detectors share a common convolution layer. Thus, those common convolution layers may be calculated once for all object detectors in the group of object detectors.

Also, according to the disclosed technique, the object detector associated with the same respective image window size for the CNN input image defines a scale detector. Each scale detector is associated with each scale of the CNN input image. When the scale detector shows the same configuration of the object detector and the downsampler, and when the CNN in the object detector shows a group of layers with similar properties, the object detector will then be described further in common below. Trained to have layers. Once the CNN weights and parameters of the training scale detector have been determined, a copy of this training scale detector is arranged to define the CNN system of the disclosed technology.

Also, the number of samples employed to train the CNN is by aligning each sample with a feature reference position and randomly perturbing the samples, as further described with FIGS. 5A-5H and 6. It can be increased beyond the initial number.

Here, with reference to FIGS. 3A and 3B, which schematically represent a CNN system, generally labeled 100, which detects an object in the input image 106, according to the disclosed technical embodiments. The CNN system 100 includes a plurality of scale detectors 102 ₁ , 102 ₂ , ..., 102 _N , and a plurality of downsamplers 104 1-104 _{N-1 .} Each of the down samplers 104 ₁ to 104 _N-1 is associated with their respective downsampling ratio. Each of the scale detectors 102 ₂ , ..., 102 _N is coupled with the respective downsamplers 104 1-1 04 _{N-1 at} the input to it. Thus, each scale detector is associated with a respective downsampling ratio (ie, scale) to the input image 106. At its input, the scale detector 102 ₁ receives the input image 106 (ie, each of the scale detectors 102 ₁ has one downsampling ratio). The system 100 may be viewed as multiple scale object detectors, each of which scale detectors 102 ₂ , ..., 102 _N receives a downsampled version of the input image 106 at its input. In other words, each of the scale detectors 102 ₁ , 102 ₂ , ..., 102 _N is associated with the respective scale of the input image 106. In FIG. 3A, the downsamplers 104 _1-1 to 104 _N-1 are arranged in series downsamplers, and each downsampler receives the output of the previous downsampler at its input (ie, the input image 106 at its input). (Except for the down sampler 104 ₁ ). However, the downsamplers 104 ₁ to 104 _N-1 may be arranged in parallel, and each downsampler receives the input image 106 at its input and the scale detectors 102 ₂ , ..., 102 _N , respectively. The input image 106 is downsampled by the corresponding downsampling ratio associated with one.

With reference to FIG. 3B, represented there is a scale detector 102 _i , which is one of the scale detectors 102 ₁ , 102 ₂ , ..., 102 _N. The object detector 102 _i includes a plurality of object detectors 108 ₁ , 108 ₂ , ..., 108 _L , and a plurality of L-1 down samplers, and from the plurality of L-1 down samplers, the down sampler 110 _L-1 and 110 _L-2 is represented in FIG. 3B. Each of the object detectors 108 ₁ , 108 ₂ , ..., 108 _L includes its own CNN and classifier. Each CNN contains a plurality of convolutional layers. The object detector 108 ₁ includes the M1 layer, the object detector 108 ₂ includes the M2 layer, the object detector 108 ₃ includes the M3 layer, and M1, M2, and M3 are integers. M3> = M2> = M1 without loss of generality.

Each of the object detectors is also associated with the respective image window size for the image at the input to it. In the example shown in FIG. 3B, the object detector 108 ₁ is associated with an image window size of I1 × I2 for the downsampled image at the input to it, and the object detector 108 ₂ is down at the input to it. Associated with the image window size of K1xK2 for the sampled image, the object detector 108 _L is associated with the image window size of J1xJ2 for the image at the input to it (ie, it is the downsampled image). , Or the original input image 106 when the object detector 108 _L is located at the scale detector 102 ₁ ). I1, K1, and J1 correspond to the width of the image window size, and I2, K2, and J2 correspond to the height of the image window size. As such, each of the object detectors 108 ₁ , 108 ₂ , ..., 108 _L is associated with the same image window size for the input image 106. Each of these image window sizes (ie, the receiving area) is the downsampling ratio associated with each 108 ₁ , 108 ₂ , ..., 108 _L at the input to it, each object detector during the convolution operation. It is associated with the number of convolution layers in, the size of the kernel, and the stride of the kernel.

The output of each CNN is combined with each classifier. Each one input of the detectors 108 ₁ , 108 ₂ , ..., 108 _L-1 is coupled to the respective downsampler. Each downsampler and the object detector 108 _L may receive an image 105, which may be a downsampled version of the input image 106 at its input. Each of the downsamplers downsamples the input image to it by their respective downsampling ratio and provides the downsampled image to each one of the object detectors 108 ₁ , 108 ₂ , ..., 108 _L-1 . do. As a result, each of 108 ₁ , 108 ₂ , ..., 108 _L is associated with their respective downsampling ratio to the input image 106. Each of these downsampling ratios is determined by the downsampling ratio of the downsamplers 104 ₁ to 104 _N-1 and the downsampling ratio of the downsampling combined with each of the object detectors 108 ₁ , 108 ₂ , ..., 108 _L. To.

Each layer of each CNN in each of the object detectors 108 ₁ , 108 ₂ , ..., 108 _L convolves the image provided to it by the corresponding filter. The output of each CNN is a map of the features corresponding to the filters adopted by the CNN. The feature map contains a value entry. Each value of each entry in the feature map represents the feature intensity of the feature associated with the various filters in the image window associated with the entry. This feature map is provided for each classifier. Each classifier classifies objects trained to be detected by the CNN system 100 and provides a classification vector for each image window. This classification vector contains values related to the detection confidence level at which the object is located within its image window, and may further include an image window correction factor (ie, boundary box regression, as described in more detail below). ).

As further illustrated below with FIG. 4, each of the object detectors 108 ₁ , 108 ₂ , ..., 108 _L is associated with their respective downsampling ratio to the input image 104. Object detectors with the same respective downsampling ratio define a group of object detectors. According to the technique disclosed, object detectors in a group of object detectors are associated with a common convolution layer (ie, because the input images to those object detectors are the same). As such, those common convolution layers need to be calculated once for each group of object detectors.

As mentioned above, each output of the object detectors 108 ₁ , 108 ₂ , ..., 108 _L is associated with the respective image window size for the input image 106. As such, when multiple scale detectors are employed, there can be two or more object detectors associated with the same image window size. Therefore, only one of those object detectors may be employed to detect and classify the objects in the input image 104 (ie, in the image window associated with each image window size). .. However, if the detection confidence level is not sufficient, another object detector with a larger number of layers may be employed, thus reducing the complexity of the calculation (ie, on average) (eg, performing). In terms of the number of operations performed). Other object detectors process only the image window with the probability that the object will be located there above a predetermined value. In other words, before adopting another object detector, the image window associated with the background is removed according to the probability determined by the first object detector.

Here, schematically an exemplary CNN system, constructed and operational according to another embodiment of the disclosed technique, which detects an object in an input image, is generally labeled with 200 and is referenced. Refer to FIG. 4, which is represented. An exemplary CNN system 200 includes two scale detectors, a first scale detector 202 ₁ and a second scale detector 202 ₂ , and a down sampler 218. Each of the first scale detector 202 ₁ and the second scale detector 202 ₂ includes a plurality of object detectors and a plurality of downsamplers. The down sampler is abbreviated as "DS" in FIG. The first scale detector 202 ₁ is an object detector 204 ₁ , 204 ₂ , and 204 ₃ . .. .. Also included are down samplers 210 and 212. The second scale detector 202 ₂ includes object detectors 206 ₁ , 206 ₂ , and 206 ₃ , as well as downsamplers 214 and 216. Each of the object detectors 204 ₁ , 204 ₂ , 204 ₃ , 206 ₁ , 206 ₂ , and 206 ₃ is a CNN and a respective classifier (abbreviated as "CLASS" in FIG. 4) 205 ₁ , 205 ₂ . , 205 ₃ , 207 ₁ , 207 ₂ , and 207 ₃ . Each CNN contains a plurality of convolutional layers (abbreviated as "L" in FIG. 4). The CNNs of the object detectors 204 ₁ and 206 ₁ include the M1 layer, the CNNs of the object detectors 204 ₂ and 206 ₂ include the M2 layer, and the CNNs of the object detectors 204 ₃ and 206 ₃ include the M3 layer. , M1, M2, and M3 are integers. M3> = M2> = M1 without loss of generality.

Each of the object detectors is also associated with the respective image window size for the image at the input to it. In the example shown in FIG. 4, the object detectors 204 ₁ and 206 ₁ are associated with an image window size of I1 × I2 for the downsampled image at the input to it, and the object detectors 204 ₂ and 206 ₂ are , The object detectors 204 3 and 206 3 are associated with the image window size of K1 × K2 for the downsampled image at the input to it, and the object detectors 204 ₃ and 206 ₃ are associated with the image window size of J1 × J 2 for the image at the input to it. (That is _, only the input image to the object detector 2063 is downsampled). I1, K1, and J1 correspond to the width of the image window size, and I2, K2, and J2 correspond to the height of the image window size. As such, each of the object detectors 204 ₁ , 204 ₂ , 204 ₃ , 206 ₁ , 206 ₂ , and 206 ₃ is associated with their respective image window size for the input image 208. Each of these image window sizes is the downsampling ratio associated with each of the object detectors 204 ₁ , 204 ₂ , 204 ₃ , 206 ₁ , 206 ₂ , and 206 ₃ at the input to it during the convolution operation. Determined according to the number of convolution layers in each object detector, the size of the kernel, and the stride of the kernel. The respective image window sizes of the object detectors 204 ₁ , 204 ₂ , 204 ₃ , 206 ₁ , 206 ₂ , and 206 ₃ for the input image 208 are the object detectors 204 ₁ , 204 ₂ , 204 ₃ , 206 ₁ , respectively. The downsampling ratio associated with each of 206 ₂ and 206 ₃ is associated with the size of each image window at the input to it. For example, the _respective image window size of the detector 2041 for the input image 208 is R2 * I1 × R2 * I2. Similarly, the respective image window size of the detector 204 ₂ for the input image 208 is R1 * K1 × R1 * K2.

The output of each convolutional network is combined with the inputs of the respective classifiers 205 ₁ , 205 ₂ , 205 ₃ , 207 ₁ , 207 ₂ , and 207 ₃ . In the arrangement shown in FIG. 4, each of the object detectors 204 ₁ , 204 ₂ , 204 ₃ , 206 ₁ , 206 ₂ , and 206 ₃ is coupled to their respective downsamplers. The input of the object detector 204 ₁ is combined with the output of the downsampler 210. The input of the object detector 204 ₂ is combined with the output of the downsampler 212. The input of the object detector 206 ₁ is combined with the output of the downsampler 214. The input of the object detector 206 ₂ is coupled with the output of the downsampler 216 and the input of the object detector 206 ₃ is coupled with the output downsampler 218. The inputs of the downsamplers 214 and 216 are also coupled with the outputs of the downsamplers 218.

The object detector 204 ₃ , the down sampler 210, the down sampler 212, and the down sampler 218 receive the input image 208 at their input. Each of the down sampler 210, the down sampler 212, and the down sampler 218 downsamples the input image 208 according to their respective downsampling ratios. _The down sampler 210 provides the downsampled image to the object detector 2041. The down sampler 212 provides the downsampled image to the object detector 204 ₂ , and the down sampler 218 provides the down sampled image to the object detector 206 ₃ , the down sampler 214, and the down sampler 216. _The down sampler 214 further downsamples the image provided to it and provides the image downsampled twice to the object detector 2061. The downsampler 216 also further downsamples the image provided to it and provides the image twice downsampled to the object detector 206 ₂ .

Each layer of each CNN in each one of the object detectors 204 ₁ , 204 ₂ , 204 ₃ , 206 ₁ , 206 ₂ , and 206 ₃ convolves the image provided to it by the corresponding filter. The output of each CNN is a map of the features corresponding to the filters employed in the CNN. As described above, the feature map contains values, and each value of each entry in the feature map represents the feature intensity of the feature associated with the various filters within the image window associated with the entry. Each of the feature maps is provided to each of the classifiers 205 ₁ , 205 ₂ , 205 ₃ , 207 ₁ , 207 ₂ , and 207 ₃ .

Each of the classifiers 205 ₁ , 205 ₂ , 205 ₃ , 207 ₁ , 207 ₂ , and 207 ₃ receives their respective matrix as input to it. Each of the classifiers 205 ₁ , 205 ₂ , 205 ₃ , 207 ₁ , 207 ₂ , and 207 ₃ determines the classification vector. This classification vector determines the value associated with the probability that an object (single or plural) (ie, trained to be detected by a CNN) will be located in each of the image windows associated with the feature map provided to it. include. In addition, the classification vector determined by each one of the classifiers 205 ₁ , 205 ₂ , 205 ₃ , 207 ₁ , 207 ₂ , and 207 ₃ is an image per image window associated with the feature map provided to it. Contains values related to window correction factors. These image window correction coefficients include, for example, corrections to the width and height of the image window. Those image window correction coefficients may further include correction to the orientation of the image window as well as the position of the image window. Those image window correction coefficients are part of the classification vector trained to provide by CNN, as described in more detail below. The classification vector contains, for example, a binary that specifies that the sample belongs to a particular class. For example, the vector [1,0] indicates that the sample belongs to the "FACE" class and not the "NOT-FACE" class. The classification vector may include three or more classes. In addition, this vector may include a numerical representation of additional information such as 3D posture, attributes (age, gender on face, color and type in the car), and boundary box regression target values.

Each of the classifiers 205 ₁ , 205 ₂ , 205 ₃ , 207 ₁ , 207 ₂ , and 207 ₃ convolves a classification filter or filters (s) according to a feature map (eg, 1 × 1 × Q × N filters. Q is the number of matrices in the feature map, and N is the number of classification filters related to the classification information to be determined), which may be embodied as a convolutional classifier (single). The output of one or more) is the probability and correction factor mentioned above. The parameters of such a convolution classifier are determined during CNN training, as further described below.

As mentioned above, each of the object detectors 204 ₁ , 204 ₂ , 204 ₃ , 206 ₁ , 206 ₂ , and 206 ₃ is for the input image 208 to the CNN 200 (ie, the object is detected in it). Image), associated with each downsampling ratio at the input to it. Further, as mentioned above, an object detector having the same respective downsampling ratio at the input to it defines a group of object detectors. In the CNN system 200, the downsamplers 212 and 218 downsample the input image 208 by the same first downsampling ratio, R1. The downsampler 216 further downsamples the input image 208 by the downsampling ratio R1. Therefore, the sampling ratio associated with the object detector 206 ₂ is R1 * R1. The downsampler 210 downsamples the input image 208 with a second downsampling ratio, R2, which is different from R1. When R2 = R1 * R1, the object detectors 204 ₁ and 206 ₂ are then associated with the same respective downsampling ratio (ie, R2), defining a group of object detectors (ie, in FIG. 4). As indicated by the shaded left diagonal line). Similarly, the object detectors 204 ₂ and 206 ₃ are associated with the same downsampling ratio (ie, R1) and define another group of object detectors (ie, by shaded vertical lines in FIG. 4). As shown). The downsampler 214 downsamples the output from the downsampler 218 with a downsampling ratio R2. Note that the downsampling arrangement represented in FIG. 4 is only shown herein as an example. As a further example, in FIG. 4, since three downsampling ratios are adopted (ie, R1, R2, and R1 * R2), then the three downsamplers are sufficient and the output of each downsampler is , Provided to the object detector associated with the downsampling ratio of the downsampler. Such three downsamplers may be arranged in parallel or in series downsamplers.

According to the technique disclosed, the object detector associated with the same respective downsampling ratio at the input there defines a group of object detectors. Object detectors in the same group of object detectors are associated with a common convolution layer (ie, because the size of the input image to those object detectors is the same). Those common convolution layers share the same convolution kernel (ie, filter) and operate for the same image size at the input to it. As such, those common convolution layers need to be calculated only once per group of object detectors. In FIG. 4, the object detectors 204 ₁ and 206 ₂ are associated with the same respective downsampling ratio at the input to it, defining a group of object detectors. As such, layers 1-M1 in the object detectors 204 ₁ and 206 ₂ are common layers. Therefore, layer 1-M1 in the object detectors 204 ₁ and 206 ₂ may be calculated once during object detection. Object detector 206 ₂ adopts the results from layer M1 to continue and calculate layers M1 + 1-M2. Similarly, the object detectors 204 ₂ and 206 ₃ are associated with the same downsampling ratio, respectively, and define a group of object detectors. As such, layers 1-M2 in the object detectors 204 ₂ and 206 ₃ are common layers and may be calculated once. The object detector 206 ₃ adopts the results from layer M2 to continue and calculate layers M2 + 1-M3. In general, object detectors in a group may be associated with different scale detectors. Thus, the object detector CNNs in the group of object detectors may be considered to create feature maps at different scales of the image plume, features created by one object detector CNN at one scale. The map is adopted by the CNN of another object detector at another scale.

Also, as mentioned above, the outputs of the object detectors 204 ₁ , 204 ₂ , 204 ₃ 206 ₁ , 206 ₂ , and 206 ₃ are associated with their respective image window sizes for the input image 208. In particular, the output from the object detectors 204 ₁ , 204 ₂ and 204 ₃ is associated with the same first image window size in image 208. Similarly, the output from the object detectors 206 ₁ , 206 ₂ , and 206 ₃ is associated with the same second image window size in image 208. As such, for the first image window size, only one of the object detectors 204 ₁ , 204 ₂ and 204 ₃ may be employed to detect and classify the objects in the input image 208. .. Similarly, for the second image window size, only one of the object detectors 206 ₁ , 206 ₂ , and 206 ₃ may be employed to detect and classify the objects in the input image 208. Typically, an object detector with a CNN showing a smaller number of layers is detected. However, if the detection confidence level is not sufficient, different detectors with a larger number of images may be employed, thus reducing the complexity of the calculation (ie, on average). For example, if the detection reliability created by the object detector 204 ₁ is not sufficient, then the object detector 204 ₂ will be adopted. Nevertheless, the object detector 204 ₂ may process only the image window with the probability that the object will be located there above a predetermined value. In other words, prior to adopting the object detector 204 ₂ , the image window associated with the background is removed according to the probability determined by the object detector 204 ₁ .

Training As explained above, CNNs according to the disclosed techniques include multiple scale detectors. Each scale detector contains a plurality of object detectors. Each object detector contains its own CNN. When each of the scale detectors shows the same configuration of the object detector and the downsampler, and when the CNN in the object detector shows a group of layers with similar properties (ie, the same filter size, stride, and activity). The CNN of the object detector is then trained to have a common layer (showing the activation function and being ordered in the same order).

The terms "groups of layers with similar properties" above and below are associated with groups of layers, where the layers in each group exhibit the same filter size, stride, and activation function, groups. The layers in are ordered in the same order. The term "common layer" in the above and below is associated with a group of layers having similar properties (ie, in different object detectors) and the corresponding layer in the group (ie, the first in each group). Layers, a second layer in each group, etc.) have similar weights and parameters. For example, with reference to FIG. 4, scale detectors 202 ₁ and 202 ₂ show the same configuration of an object detector and a downsampler. Furthermore, layers 1-M1 in the CNN of the object detectors 204 ₁ , 204 ₂ , 204 ₃ , 206 ₁ , 206 ₂ , and 206 ₃ are a group of layers with similar properties, respectively. CNNs are trained to have a common layer. Also, the layers M1 + 1-M2 in the _CNNs of the object detectors 204 ₂ , 204 ₃ , 206 ₂ and 2063 are also a group of layers with similar properties, and the respective CNNs of those object detectors are common. Trained to have layers. Similarly, the CNN layers M2 + ₁ -M3 of the object detectors 204 ₃ and 2063 are a group of layers with similar properties, and each CNN of those object detectors is trained to have a common layer. The object.

According to one alternative, the object detectors in the CNN system are provided with the same training sample or sample, each with its own predetermined classification vector. The size of the sample or sample corresponds to the image window size associated with each object detector for the image at its input (eg, I1xI2, K1xK2, and J1 in FIGS. 3B and 4). × J2). Each object detector employs its own CNN and classifier to detect and classify the objects in the training sample provided to it and creates its own classification vector for each sample. Those classification vectors are compared with a predetermined classification vector. The error between the classification vector created by each CNN of the object detector and each predetermined classification vector (eg, sum of squares of differences, log loss, softmaxlog loss) is determined. The correction factors for the weights and parameters that minimize this error are then determined for the weights and parameters for each CNN in each respective object detector. The CNN weights and parameters are then updated accordingly. All weights and parameters of groups of layers with similar properties in their respective CNNs of all object detectors are then averaged to each group of layers with similar properties to create a common layer. Applies. For example, the weights and parameters for the first M1 layer of all CNNs in all object detectors are averaged. Similarly, the weights and parameters for the M1 + 1-M2 layers of all CNNs in all object detectors are averaged, and so on. Averaging the updated weights and parameters, averaging the correction factors, and updating the weights and parameters according to those averaged correction factors is equivalent when the CNN is initialized with the same weights and parameters. be.

According to another alternative, when each of the scale detectors shows the same configuration of the object detector and the downsampler, and when the CNN in the object detector shows a group of layers with similar properties, the scale detector alone. One instance may then be trained. This single instance of the scale detector is referred to herein as the "training scale detector". To train a CNN according to the techniques disclosed by the training scale detector, the training scale detector is provided with a training sample, each having its own predetermined classification vector. According to one alternative, an image containing a list of objects and a bounding box are provided to the training scale detector. According to another alternative, the sample has a size similar to (ie, not necessarily) the maximum image window size (eg, J1 × J2 in FIGS. 3B and 4) for input to the object detector in the scale detector. show. Those samples are then downsampled to create training samples showing the respective sizes of the other object detectors (eg, I1xI2, K1xK2 in FIGS. 3B and 4) (ie, object detectors). 102 _i -By downsampling a training scale detector similar to the downsamples 110 _L-1 and 110 _L-2 in Figure 3B). Each object detector employs its own CNN and classifier to detect and classify the objects in the training sample provided to it and creates its own classification vector for each sample. Those classification vectors are compared with a predetermined classification vector. The error between the CNN classification vector and the predetermined classification vector is determined.

In order to employ multiple scale detectors in configurations such as those described above with FIGS. 3A, 3B, and 4, correction factors for the weights and parameters that minimize the errors mentioned above are then in the training scale detector. Determined for each CNN weight and parameter in each respective object detector. The CNN weights and parameters are then updated accordingly. The weights and parameters of all groups of layers with similar properties in their respective CNNs of all object detectors in the training scale detector are then averaged and layers with similar properties to create a common layer. Applies to each group of. Once the CNN weights and parameters of the training scale detector have been determined, a replica of this training scale detector is arranged to mount each _one of the scale detectors 108 1-108 _N (FIG. 3A). Define a CNN system for the disclosed technology.

During training, the CNN weights and parameters are updated, thus minimizing this error. Such optimization may be implemented by adopting a gradient descent process such as, for example, Stochastic Gradient Descent (SGD). According to the gradient descent process, the corrections for the weights and parameters (or new weights and parameters) determined layer by layer and sample by sample in the CNN are averaged for all samples. Corrections for weights and parameters are determined according to the partial derivative of the error with respect to the weights and parameters of the CNN (ie, because the CNN may be considered as a synthetic function). This step is repeated in multiple iterations, either by a determined number of iterations or until the error falls below a predetermined value. According to the SGD, at each iteration, only a portion of the sample is adopted at each iteration. In addition, chain rules, inputs to layers, outputs of layers, and derivatives of outputs to errors are needed to determine the derivatives of the layers' weights and parameters.

As mentioned above, the classification vector provided by CNN in accordance with the disclosed technology comprises an image window correction factor. To train the CNN to provide image window correction factors during training, the difference between the position and orientation of the image window corresponding to each classification vector (ie defined by the index of this vector in the feature map). As such), and the difference between the actual position and orientation of the sample is determined. This difference is minimized, for example, using the stochastic gradient descent method.

Training Data Generally, the CNN is trained to define an object in the input image and creates information related to the probability that the object trained to be detected by the CNN will be in various positions in the input image. CNNs are trained using a training set containing samples (images or any other data), each associated with a predetermined classification vector. The sample used for training is typically an image window clipped from the image. Each sample is categorized according to the overlap of the image window with the objects in the image (ie, the class is determined for the sample). When a sample does not overlap with any object in the image, the sample is classified as a background. As mentioned above, the training process is designed to reduce the error between the output value of the CNN and the value associated with the sample adopted (eg, sum of squares of differences, log loss, softmaxlog loss). Modify the parameters.

Assuming an initial training set with an initial number of training samples and with detected and classified objects according to the disclosed technique, the number of training samples in the training set can be increased above the initial number. In other words, a training set with a larger number of training samples is created from the initial training set. Here, a diagram schematically illustrating images 250, 280, and 310 having an object (eg, a face) therein, employed to determine a training set, according to a further embodiment of the disclosed technique. See 5A-5H. First, objects 253, 255, 283, 285, 313, and 315 in images 250, 280, and 310 are detected. A rectangular boundary indicating a predetermined size is then defined around the detected object. Rectangle boundaries and the like are referred to as "boundary boxes" of objects in the above and below.

Each of the objects 253, 255, 283, 285, 313, and 315 in images 250, 280, and 310 is bounded by their respective bounding boxes. Within the image 250, the object 253 is bounded by the bounding box 252 and the object 255 is bounded by the bounding box 254. Within the image 280, the object 283 is bounded by the bounding box 282 and the object 285 is bounded by the bounding box 284. Within image 310, the object 313 is bounded by the bounding box 312 and the object 315 is bounded by the bounding box 314. Each border box indicates its size. Within images 250, 280, and 310, two different bounding box sizes are illustrated. Boundary boxes 252, 282, and 314 indicate the first size, and boundary boxes 254, 284, and 312 indicate the second size. Each bounding box is associated with its own relative coordinate system. The boundary box 252 is associated with the coordinate system 256, the boundary box 254 is associated with the coordinate system 258, the boundary box 282 is associated with the coordinate system 286, and the boundary box 284 is associated with the coordinate system 288. 312 is associated with the coordinate system 316 and the boundary box 214 is associated with the coordinate system 318.

For each key point of the object, each feature position is determined in the coordinate system associated with the bounding box. In the example shown in FIGS. 5A-5H, the feature types of the object are facial eyes, nose, and mouth (ie, the object is a face). With reference to FIG. 5B, in the coordinate system 256 of the boundary box 252, points 260 ₁ and 260 ₂ represent the position of the eyes of face 253, point 262 represents the position of the nose of face 253, and points 264 ₁ ,. 264 ₂ and 264 ₃ represent the position of the mouth of the face 253. Similarly, in the coordinate system 258 of the boundary box 254, points 266 ₁ and 266 ₂ represent the position of the eyes of object 255, point 268 represents the position of the nose of object 255, points 270 ₁ , 270 ₂ , and. 270 ₃ represents the position of the mouth of the object 255. With reference to FIG. 5D, in the coordinate system 286 of the boundary box 282, points 290 ₁ and 290 ₂ represent the position of the eyes of object 283, point 292 represents the position of the nose of object 283, and points 294 ₁ , 294 ₂ and 294 ₃ represent the position of the mouth of the object 283. Similarly, in the coordinate system 288 of the boundary box 284, points 296 ₁ and 296 ₂ represent the position of the eyes of object 285, point 298 represents the position of the nose of object 285, points 300 ₁ , 300 ₂ , and. 300 ₃ represents the position of the mouth of the object 285. With reference to FIG. 5F, in the coordinate system 316 of the boundary box 312, points 320 ₁ and 320 ₂ represent the position of the eyes of object 313, point 322 represents the position of the nose of object 313, points 322 ₁ , 322 ₂ and 322 ₃ represent the position of the mouth of the object 313. Similarly, in the coordinate system 218 of the boundary box 314, points 326 ₁ and 326 ₂ represent the position of the eyes of object 315, point 328 represents the position of the nose of object 315, points 329 ₁ , 329 ₂ , and. 329 ₃ represents the position of the mouth of the object 315. Typically, the position of the object's keypoints in each coordinate system is normalized to be, for example, 0 to 1 (ie, the corners of the bounding box are the coordinates [0,0], [0,0, 1], [1,1], [1,0]). In other words, the coordinate systems 256, 258, 286, 288, 316, and 316 of the respective boundary boxes 252, 254, 282, 284, 312, and 314 are normalized to the position and size of the boundary boxes, respectively. .. Thus, the location of various features can be related independently of the size of the bounding box.

With reference to FIG. 5G, the normalized positions of the key points of the various objects are superposed in any bounding box 330. Since the coordinate system of the bounding box is normalized (ie, the position in one coordinate system corresponds to the same position in another coordinate system), the keypoint type of the same object in different bounding boxes (eg, eyes). The positions associated with may be averaged.

After that, the feature reference position is determined for each key point type of the object (eg, eyes, nose, mouth). With reference to FIG. 5H, for the bounding box, any size, point 336 ₁ represents the _average position of the positions of points 260 ₁ , 290 ₁ , 326 1266 ₁ , 296 ₁ , and 320 ₁ , where point 336 ₂ is. , Points 260 ₂ , 290 ₂ , 326 ₂ , 266 ₂ , 296 ₂ , and 320 ₂ represent the average position of the positions. Point 338 represents the average position of the positions of points 262, 292, 328, 268, 298, and 322. Point 340 ₁ represents the average position of the positions of points 264 ₁ , 294 ₁ , 329 ₁ , 270 ₁ , 300 ₁ , and 324 ₁ . Point 340 ₂ represents the _average position of the positions of points 264 ₂ , 294 ₂ , 3292 ₂ , 264 ₂ , 300 ₂ , and 3242, and point 340 ₃ is point 264 ₃ , 294 ₃ , 329 ₃ , 270 ₃ , Represents the average position of the positions of 300 ₃ and 324 ₃ .

Their average position defines the feature reference position. Points 336 ₁ and 336 ₂ define the feature reference position of the eye, point 338 defines the reference position of the nose, and points 340 ₁ , 340 ₁ and 340 ₃ define the reference position of the mouth.

Once the reference positions for those key points were determined, each object in each of the initial training samples was aligned with the reference position for those key points, so that the key points for each object were selected. Aligns with the reference position of each keypoint to the extent determined by optimizing the alignment cost function (eg, the squared error of the difference between the keypoint of the object and the reference position of the keypoint). Each of the training samples is then perturbed from this reference position, thus creating a new training sample. Perturbations include at least one of an object's horizontal, vertical, and directional shifts. The perturbations of each sample are randomly determined according to a selected probability distribution (eg Gauss). By adopting those perturbations, the number of training samples in the training set can be increased beyond their initial size. This process is also referred to as "training sample augmentation" and the training samples produced thereby are referred to as "enhanced training samples". In the exemplary training set shown in FIGS. 5A-5H, each image contains a training sample showing two boundary box sizes. However, in general, a training sample showing a bounding box of one size may be scaled to produce a training sample showing a bounding box of a different size. This scaled bounding box may then be adopted as a training sample.

Reference is now made to FIG. 6, schematically illustrating a method of determining a training set for a neural network that is operational according to another embodiment of the disclosed technique. In step 350, objects in multiple images of the training set and key points for each object are detected, and a bounding box indicating a predetermined size is defined around each detected object. The object may be determined, for example, by a human observer (ie, manually). Objects may also be determined by adopting an automated detector or in a semi-automated manner (eg, an object is detected by an automated detector and verified by a human observer).

In step 352, the position of the key point of each object in the bounding box is determined. In step 354, the reference position of each key point is determined for the key point type of the object. The reference position of each key point is determined according to the average position of the key points of the same type of object, and the average value is determined according to the key point position of the object of all objects in the initial training set.

In step 356, all training samples in the initial training set with their respective reference positions are registered.

In step 358, each of the aligned samples from the reference position is randomly perturbed.

Here, reference is made to FIG. 7, which schematically illustrates a method for a CNN that is operational according to a further embodiment of the disclosed technique. In step 400, the augmented training sample is created from the initial training set. It is described above with FIGS. 5A-5H and 6.

In step 402, the object detector CNN is trained to have a common layer. According to one alternative, the weights and parameters (or correction factors for them) of all groups of layers with similar properties of the object detector are averaged to create a common layer. According to another alternative, a single training scale detector is trained, a replica of the training scale detector is deployed, and a CNN system is defined. Each replica is associated with a scaled version of the input image, and a copy of the training scale detector defines a CNN system.

In step 404, at least one object in at least one image is detected and classified by adopting a defined CNN system. Detecting and classifying at least one object in at least one image involves the following sub-procedures:
To create multiple downsampled images, images are downsampled according to multiple downsampling ratios, and each downsampled image is associated with its own downsampling ratio.
For each downsampled image, the corresponding CNN detects the object at a predetermined image window size for the image.
Classify objects in the image. The CNNs that detect objects in each downsampled image and are associated with the same each downsampling ratio define at least one group of CNNs. CNNs in a group of convolutional networks are associated with a common convolutional layer.

With reference to FIG. 4, each CNN of the object detectors 204 ₁ and 206 ₂ defines a group of CNNs, as described above. Similarly, each CNN of the object detectors 204 ₂ and 206 ₃ defines a group of CNNs.

Those skilled in the art will recognize that the techniques disclosed are not limited to those specifically indicated and described above. The scope of the disclosed technology is defined solely by the following claims.

Claims (14)

A convolutional neural network system that detects at least one object in at least one image.
A plurality of object detectors, each of which corresponds to a predetermined image window size within the at least one image, and each of the object detectors for the at least one image. Associated with the downsampling ratio, each said object detector
A convolutional neural network containing multiple convolutional layers,
An object classifier coupled with the convolutional neural network for classifying objects in the at least one image according to the results from the convolutional neural network .
Including the object detector
Equipped with
The object detector associated with the same downsampling ratio is a convolutional neural network system that defines at least one group of object detectors in a group of object detectors associated with a common convolutional layer . Each is a plurality of downsamplers associated with their respective downsampling ratios, the downsampler being configured to produce a scaled version of the at least one image, said scaled version. The convolutional neural network system of claim 1, wherein each of the convolutional neural networks further comprises a downsampler associated with the respective downsampling ratio. The downsampler, and the object detector associated with the same respective image window size for the at least one image, define a scale detector, and each said scale detector is each of the at least one image. The convolutional neural network system according to claim 2, which is associated with a scaled version of. The convolutional neural network system according to claim 1, wherein the object classifier is a convolutional classifier and convolves at least one classification filter with a feature map provided by each of the convolutional neural networks. Each of the convolutional neural networks creates a feature map containing multiple features, each entry represents feature intensity within the image window associated with the entry, and the image window represents the respective image window size. The convolutional neural network system according to claim 4, as shown. The convolutional neural network system of claim 5, wherein the object classifier provides the probability that the object will be located in each of the image windows associated with the feature. The object classifier provides a classification vector for each image window that includes an image window correction coefficient for each image window associated with the feature map, the image window correction coefficient being the width and height of each image window. The convolutional neural network system of claim 6, comprising corrections to the edges, corrections to the position of each image window, and corrections to the orientation of each image window. 3. A single training scale detector is trained when the scale detector shows the same configuration of the object detector and when the CNN in the object detector shows a layer of groups with similar characteristics. Convolutional neural network system described in. Before training the single training scale detector, the number of training samples in the training set
Determine the location of each object's keypoint within the bounding box of each training sample and
For the keypoint type of an object, each feature reference position is determined according to the average position of the keypoints of the same type of object, and the average value is determined according to the keypoint position of the object of all objects in the initial training set.
All training samples in the initial training set are registered in the feature reference position and
The convolutional neural network system of claim 8, wherein each of the aligned training samples from this reference position is randomly perturbed to increase above the initial number of training samples. It is a convolutional neural network method, and the above method is
A procedure for downsampling an image according to a plurality of downsampling ratios in order to create a plurality of downsampled images, wherein each downsampled image is associated with a respective downsampling ratio. And the procedure to do
For each downsampled image, the procedure for detecting an object with a predetermined image window size for at least one image by the corresponding convolutional neural network.
The procedure for classifying objects in the image and
Including
A convolutional neural network that detects objects in each downsampled image associated with the same respective downsampling ratio defines at least one group of convolutional neural networks and is a convolutional neural network in a group of convolutional neural networks. Is a convolutional neural network method associated with a common convolutional layer. Prior to the procedure of downsampling the image,
Procedures for creating augmented training samples from the initial training set,
The procedure for training the convolutional neural network to have a common layer ,
The convolutional neural network method according to claim 10, further comprising. 22. The convolutional neural according to claim 11, wherein training the convolutional neural network to have a common layer comprises averaging the weights and parameters of all groups of layers having similar properties of the object detector. Network method. The convolutional neural network for having a common layer comprises training the single training scale detector by adopting the enhanced training sample and arranging a replica of the single training scale detector. 11. A replica of the single training scale detector defines a convolutional neural network system, each replica being associated with a scaled version of the at least one image. Convolutional neural network method. The above procedure for creating an enhanced training sample is
Sub-procedures to determine the location of key points for each object within the bounding box of each training sample,
For the keypoint type of an object, it is a sub-procedure to determine the reference position of each keypoint according to the average position of the keypoints of the object of the same type, and the average value is the average value of all the objects in the initial training set. The determination sub-procedure, which is determined according to the position of the key point of the object,
A sub-procedure for registering all training samples in the initial training set at the feature reference position,
A sub-procedure that randomly perturbs each of the training samples aligned from this reference position ,
11. The convolutional neural network method according to claim 11.

Images