JP2018165948A - Image recognition device, image recognition method, computer program, and product monitoring system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image recognition device, an image recognition method, a computer program, and a product monitoring system improving accuracy of image recognition by a hierarchical neural network.SOLUTION: An image recognition device 10 comprises an arithmetic processing unit 1 having a data generation unit 3 performing predetermined data processing on an original image to generate input data, and an image processing unit 2 having a hierarchical neural network recognizing a type of object included in the generated input data. The image processing unit performs process of learning a parameter of the network based on a recognition result of the network if the original image is a sample image, and performs process of outputting the recognition result of the network if the original image is a target image to be recognized. The data processing performed by the data generation unit is process of imparting at least one invariance of rotation and inversion to the original image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image recognition apparatus, an image recognition method, a computer program, and a product monitoring system. Specifically, the present invention relates to an image processing technique for improving the accuracy of image recognition using a hierarchical convolution neural network.

  In recent years, the performance of image recognition by deep learning has improved dramatically. Deep learning is a general term for machine learning using a multilayer hierarchical neural network. For example, a convolutional neural network (hereinafter also referred to as “CNN”) is used as the multilayer hierarchical neural network.

The CNN has a multilayer structure in which a convolution layer and a pooling layer in a local region are repeated, and the image recognition performance is improved by such a structure.
As shown in Non-Patent Documents 1 and 2, an object class has already been recognized by deep learning using a convolutional neural network.

"ImageNet Classification with Deep Convolutional Neural Networks" A. krizhevsky et al. In: Proc. Adv. Neural Inf. Proc. Syst. (NIPS), 2012, PP.1097-1105 "Very Deep Convolutional Networks for Large-Scale Image Recognition" K.Symonyan et al. ArXiv: 1409.1556v6 [cs.CV] 10 Apr.2015

In image recognition using a convolutional neural network, the original image pixel values (RGB values) are input to the network as they are without preprocessing, or principal component analysis (Principle Component Analysis) is performed on the pixel values. Is called.
As described above, conventionally, since the pixel value (raw data) of the original image is used as it is or only a preprocessing for extracting a single feature factor from the original image is performed, in order to improve the recognition accuracy, Multiple sample images and the same class of polymorphic sample images need to be collected.

In particular, since sample images including rotated or inverted objects are rare, there is a problem that the recognition accuracy of rotated or inverted objects cannot be improved much even if CNN learning is performed using sample images of normal orientation. .
The present invention has been made in view of such conventional problems, and an object thereof is to improve the accuracy of image recognition by a hierarchical neural network.

  (1) An image recognition apparatus according to the present invention includes a data generation unit that performs predetermined data processing on an original image to generate input data, and a hierarchical neural network that recognizes the type of object included in the generated input data An image recognition device comprising: an image processing unit comprising: an image processing unit that, when the original image is a sample image, performs processing for learning parameters of the network based on a recognition result of the network If the original image is a recognition target image, a process of outputting a recognition result of the network is performed, and the data processing performed by the data generation unit includes rotation and inversion of the original image. It is a process for imparting at least one invariance.

According to the image recognition apparatus of the present invention, the data processing performed by the data generation unit includes processing for imparting at least one invariance of rotation and inversion to the original image.
For this reason, when learning with the same number of sample images, the accuracy of image recognition by the hierarchical neural network can be improved as compared with the case where the original image is used as input data without performing the above data processing. Yes (see FIG. 10). Further, even a rotated or inverted object can be accurately recognized.

(2) In the image recognition apparatus of the present invention, specifically, the data processing performed by the data generation unit includes a first process and a second process defined below.
1st process: The process which produces | generates the image filter which has at least 1 invariance of rotation and inversion with respect to an original image 2nd process: The process which convolves the image filter produced | generated by the 1st process with an original image

(3) More specifically, the image filter is configured to open a color vector of an arbitrary pixel point defined by polar coordinates with the predetermined point of the original image as an origin at a predetermined angle starting from the pixel point. It consists of elements contained in multiple color vectors divided in the direction.
The reason is that the color vector obtained by dividing the color vector of the pixel point of the polar coordinate display as described above remains equivalent even when rotated at an arbitrary angle around the origin, and is rotated and inverted with respect to the original image. This is because it has at least one of the invariance.

(4) More specifically, the image filter uses a color vector of an arbitrary pixel point defined by polar coordinates with a predetermined point of the original image as an origin in a radial direction and a tangential direction starting from the pixel point. It is preferable that the divided elements are included in two color vectors.
The reason is that if the color vector is decomposed in two directions, the radial direction and the tangential direction, the number of calculation parameters can be minimized, and the processing load on the data generation unit can be reduced.

(5) In the image recognition apparatus of the present invention, specifically, the hierarchical neural network is a convolutional neural network.
The reason is that the convolutional neural network can realize high performance for image recognition even in the hierarchical neural network.

(6) In the image recognition apparatus of the present invention, the object whose type is recognized may be at least one object of handwritten characters, humans, animals, plants, and products.
The reason for this is that data processing that imparts at least one invariance of rotation and inversion to the original image, which is a feature of the present invention, can be applied to various objects regardless of the attributes of the objects included in the original image. This is because it is considered applicable. Therefore, the application range of the image recognition apparatus of the present invention is not limited to the recognition of a specific object.

(7) The computer program of this invention is related with the computer program for functioning a computer as an image recognition apparatus in any one of said (1)-(6).
Therefore, the computer program of the present invention has the same operational effects as those of the image recognition device described in any one of (1) to (6) above.

(8) The image recognition method of this invention is related with the image recognition method which the image recognition apparatus in any one of the above-mentioned (1)-(6) performs.
Therefore, the image recognition method of the present invention has the same effects as the image recognition apparatus in any of the above (1) to (6).

  (9) The product monitoring system of the present invention includes a photographing device that photographs a plurality of products, a robot device that takes out one of the plurality of photographed products to the outside, and the product that should be removed to the robot device. A product monitoring system comprising: a control device for instructing, wherein the control device includes the image recognition device according to any one of (1) to (6) described above, and the image recognition device is a defective product. The robot apparatus is instructed to take out the recognized product.

  According to the product monitoring system of the present invention, since the image recognition device instructs the robot device to pick up a product recognized as a defective product, the defective product is picked up by learning the image recognition device from an appropriate number of sample images. Can be done automatically and accurately.

The present invention can be realized not only as a system and apparatus having the above-described characteristic configuration, but also as a computer program for causing a computer to execute such characteristic configuration.
Further, the present invention described above can be realized as one or a plurality of semiconductor integrated circuits that realize part or all of the system and apparatus.

  According to the present invention, it is possible to improve the accuracy of image recognition by a hierarchical neural network.

1 is a block diagram of an image identification device according to an embodiment of the present invention. It is a schematic block diagram of CNN contained in a CNN process part. It is a conceptual diagram of the processing content of a convolution layer. It is a conceptual diagram of the structure of a receptive field. It is explanatory drawing of the 1st process by a data generation part. It is explanatory drawing of the 2nd process by a data generation part. It is explanatory drawing of the rotation point etc. which rotated arbitrary pixel points only the predetermined angle. It is a structural diagram of the deep CNN constructed in the CNN processing unit. It is a figure which shows an example of the handwritten character used for simulation experiment. It is a graph showing the test result of the recognition accuracy for every character class. 1 is an overall configuration diagram of a product monitoring system according to an embodiment of the present invention.

  Hereinafter, details of embodiments of the present invention will be described with reference to the drawings. In addition, you may combine arbitrarily at least one part of embodiment described below.

[Overall configuration of image processing apparatus]
FIG. 1 is a block diagram of an image recognition apparatus 10 according to an embodiment of the present invention.
As shown in FIG. 1, the image recognition apparatus 10 of this embodiment includes, for example, an arithmetic processing unit 1 and an image processing unit 2 mounted on a PC (Personal Computer) (not shown).

The arithmetic processing unit 1 includes a CPU (Central Processing Unit). The number of CPUs in the arithmetic processing unit 1 may be one or plural, and may include an integrated circuit such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).
The arithmetic processing unit 1 includes a RAM (Random Access Memory). The RAM is composed of a memory element such as SRAM (Static RAM) or DRAM (Dynamic RAM), and temporarily stores a computer program executed by the CPU and data necessary for the execution.

The image processing unit 2 includes a GPU (Graphics Processing Unit). The number of GPUs in the image processing unit 2 may be one or plural, and may include an integrated circuit such as an FPGA or an ASIC.
The image processing unit 2 includes a RAM. The RAM is configured by a memory element such as SRAM or DRAM, and temporarily stores a computer program executed by the GPU and data necessary for the execution.

The arithmetic processing unit 1 includes a data generation unit 3 that generates an input image to the CNN processing unit 4 as a functional unit realized by the CPU executing a computer program for arithmetic processing recorded in a memory.
The data generation unit 3 performs the following first and second processing on the labeled sample image 7 or the identification target captured image (hereinafter also referred to as “target image”) 8, thereby providing a CNN processing unit. 4 is generated (hereinafter also referred to as “input data”).

First processing: processing for generating an image filter 9 having invariance of rotation and inversion with respect to the images 7 and 8 from the sample image 7 or the target image 8 (see FIG. 5).
Second process: a process of convolving the image filter 9 generated in the first process with the sample image 7 or the target image 8 (see FIG. 6)

Hereinafter, a generic term of “sample image” and “target image” input to the data generation unit 3 is also referred to as “original image”. The original image refers only to the object area image to be processed (training or recognition).
The data generation unit 3 inputs input data obtained by performing the first and second processing on the original images 7 and 8 to the CNN processing unit 4 in the subsequent image processing unit 2.

The image processing unit 2 includes a CNN processing unit 4, a learning unit 5, and a recognition unit 6 as functional units realized by the GPU executing an image processing computer program recorded in a memory.
The CNN processing unit 4 performs recognition of the type of object included in the input data (for example, recognition of the type of character included in the input image), and the recognition result (specifically, the probability for each classification class) ) Is input to the learning unit 5 or the recognition unit 6.

Specifically, when CNN is trained using the labeled sample image 7, the CNN processing unit 4 identifies the classification class of the sample image 7 and inputs the identified classification class to the learning unit 5.
On the other hand, when the learned CNN processing unit 4 specifies the classification class of the target image 8, that is, when the image processing unit 2 operates as a discriminator, the CNN processing unit 4 recognizes the specified classification class. 6

The learning unit 5 updates parameters (weights and biases) held by the CNN processing unit 4 based on the input classification class, and stores the updated parameters in the CNN processing unit 4.
The recognition unit 6 outputs a recognition result based on the input classification class. Specifically, the classification class having the highest probability input from the CNN processing unit 4 is output as the classification class of the target image 8. The recognition result output from the recognition unit 6 is displayed on a PC display or the like, thereby notifying the PC operator.

[Processing content of CNN processing unit]
(Configuration example of CNN)
FIG. 2 is a schematic configuration diagram of the CNN included in the CNN processing unit 4.
As shown in FIG. 2, the CNN constructed in the CNN processing unit 4 includes three arithmetic processing layers including convolution layers (also referred to as “down-sampling layers”) C 1 and C 2, pooling layers P 1 and P 2, and a fully coupled layer F. And a final layer E which is an output layer of CNN.

Pooling layers P1 and P2 are arranged after the convolution layers C1 and C2, and all coupling layers F are arranged after the last pooling layer P2. The final layer E of the CNN includes the same number (10 in FIG. 2) of final nodes as the number of classification classes set in advance.
FIG. 2 illustrates the case where the convolution layers C1 and C2 and the corresponding pooling layers P1 and P2 are two. However, the convolution layer and the pooling layer may be three or more. Further, at least one total coupling layer F is disposed.

The j-th node in a certain layer C1, P1, C2, P2 receives inputs x _i (i = 1, 2,... M) from m nodes in the immediately preceding layer, respectively, and biases these weighted sums. An intermediate variable u _j obtained by adding is calculated. That is, the intermediate variable u _j is calculated by the following equation. In the following equation, w _ij is a weight and b _j is a bias.

The response y _j obtained by applying the intermediate variable u _j to the activation function a (•) that is a nonlinear function, that is, y _j = a (u _j ) becomes the output of the node of this layer, and this output is input to the next layer. Is done.
As the activation function a, a “sigmoid function” or a (x _j ) = max (x _j , 0) is used. In particular, the latter activation function is called “ReLU (Rectified Linear Unit)”. ReLU is often used in recent years because it contributes to good convergence and improved learning speed.

In the vicinity of the output layer of the CNN, one or more total coupling layers F in which all nodes between adjacent layers are coupled are arranged. The final layer E giving the output of CNN is designed in the same way as a normal neural network.
For the purpose of classifying the input image, the same number of nodes as the number of classification classes are arranged in the final layer E, and the “softmax function” is used as the activation function a of the final layer E.

Specifically, the following equation is calculated based on inputs u _j (j = 1, 2,..., N) to n nodes. At the time of recognition, the index j = argmax _j p _{j of the} node where p _j has the maximum value is selected as the estimation class.

(Processing content of convolution layer)
FIG. 3 is a conceptual diagram of processing contents of the convolution layers C1 and C2.
As shown in FIG. 3, the inputs of the convolution layers C1 and C2 are in the form of N sheets (N channels) having a vertically long size of S × S pixels.
Hereinafter, this type of image is referred to as S × S × N. An input of S × S × N is described as x _ijk (where (i, j, k) ∈ [0, S-1], [0, S-1], [1, N]).

In CNN, the number of channels in the first input layer (convolution layer C1) is N = 1 if the input image is grayscale, and N = 3 (three RGB channels) if the input image is color.
In the convolution layers C1 and C2, a calculation for convolving a filter (also referred to as “kernel”) to the input x _ijk is executed.

This calculation is performed by convolution of a filter in general image processing, for example, processing of blurring an image by two-dimensionally convolution of a small size image (Gaussian filter), processing of enhancing an edge (sharpening filter) ) Is basically the same process.
Specifically, a two-dimensional filter of L × L size is convoluted with a pixel of size S × S of input x _ijk of each channel k (k = 1 to N), and the result is obtained for all channels k = 1 to N. Add over. This calculation result is in the form of a one-channel image u _ij .

If the filter is defined as w _ijk (where (i, j, k) ∈ [1, L-1], [1, L-1], [1, N]), u _ij is calculated by the following equation. .

Here, P _ij is a square area of size L × L pixels with the pixel (i, j) in the image as a vertex. That is, P _ij is defined by the following equation.

b _k is a bias. In this embodiment, the bias is common among all output nodes for each channel. That is, b _ijk = b _k .
The filter may be applied at intervals of a plurality of pixels instead of all pixels. That is, for a predetermined number of pixels s, P _ij is defined as follows, and w _ij is calculated by replacing w _{p-i, q-j, k} with w _{p-si, q-sj, k.} Also good. This pixel interval s is called “slide”.

The u _ij calculated as described above becomes an output y _ij of the convolution layers C1 and C2 through the activation function a (•). That is, y _ij = a (u _ij ).
As a result, for one filter w _ijk , an output y _ij for one S × S channel having the same vertical and horizontal sizes as the input x _ijk is obtained.

If N ′ similar filters are prepared and the above calculation is performed independently, an output of S × S for N ′ channels, that is, an output y _{ijk of} S × S × N ′ size (where, (i, j, k) ∈ [1, S-1], [1, S-1], [1, N ′]) is obtained.
The output y _{ijk for the} N ′ channel becomes the input x _ijk to the next layer. FIG. 3 shows the calculation content for one of the N ′ filters.

The above calculation can be expressed as a single layer network in which interlayer nodes are connected in a special manner as shown in FIG. FIG. 4 is a conceptual diagram of the structure of the receptive field. In the figure on the left, the receptive field is represented by a rectangle, and in the figure on the right, the receptive field is represented by a node.
Specifically, each node in the upper layer is coupled to a part of each node in the lower layer (this is referred to as “local receptive field”). Also, the connection weight is common among the nodes (this is referred to as “weight sharing”).

(Processing contents of the pooling layer)
As shown in FIG. 2, the pooling layers P1 and P2 exist in pairs with the convolution layers C1 and C2. Accordingly, the outputs of the convolution layers C1 and C2 are inputs to the pooling layers P1 and P2, and the inputs of the pooling layers P1 and P2 are in the form of S × S × N.
The purpose of the pooling layers P1 and P2 is to discard part of the information about the position of the filter where the response of the filter was strong, and to realize the invariance of the response to a slight change in the feature.

The nodes (i, j) of the pooling layers P1, P2 have local receptive fields P _{i, j} in the input-side layers, like the convolution layers C1, C2. The nodes (i, j) of the pooling layers P1, P2 consolidate the outputs y _{p, q} of the nodes (p, q) εP _{i, j} inside the local receptive fields P _{i, j} into one.
The size of the local receptive fields P _{i, j} of the pooling layers P1, P2 is set independently of that of the convolution layers C1, C2 (filter size).

When the input is a plurality of channels, the above processing is performed for each channel. That is, the number of output channels of the convolution layers C1 and C2 and the pooling layers P1 and P2 are the same.
Pooling is performed by thinning out the image in the vertical and horizontal (i, j) directions. That is, two or more strides s are set. For example, when s = 2, the vertical and horizontal size of the output is half of the vertical and horizontal size of the input, and the number of output nodes of the pooling layer is 1 / s ² times the number of input nodes.

There are “average pooling”, “maximum pooling”, and the like as a method of collecting the inputs from the nodes inside the receptive field P _{i, j} into one.
Average pooling is a method of outputting an average value of inputs x _pqk from nodes belonging to P _{i, j as shown} in the following equation.

The maximum pooling is a method of outputting the maximum value of the input x _pqk from the nodes belonging to P _{i, j} as follows: Although average pooling was the mainstream in early CNN research, now maximum pooling is generally employed.

Note that, unlike the convolution layers C1 and C2, in the pooling layers P1 and P2, there is no weight that changes due to learning, and no activation function is applied.
Either the average pooling or the maximum pooling may be employed in the CNN of the present embodiment, but the maximum pooling is employed in the CNN implementation example shown in FIG.

[Processing content of the learning unit]
In CNN training, “supervised learning” is fundamental. Also in this embodiment, the learning unit 5 performs supervised learning.
Specifically, the learning unit 5 is executed by minimizing the classification error of each sample image for a set including a large number of labeled sample images as learning data. Hereinafter, this process will be described.

Each node in the final layer E of the CNN processing unit 4 outputs the probability p _j (j = 1, 2,... N) for the corresponding class by normalization with the softmax function (the above-mentioned [Expression 2]). . This probability p _j is input to the learning unit 5.
The learning unit 5 updates parameters such as weights set in the CNN processing unit 4 so as to minimize the classification error calculated from the input probability p _j .

Specifically, the learning unit 5 outputs ideal outputs d1, d2,... Dn (labels) for the input samples and outputs p1. p2. ...... The pn deviation is calculated by the following cross entropy C. This cross entropy C is a classification error.

The target outputs d1, d2,... Dn are set so that d _j = 1 only in the correct class j, and d _k = 0 in all other k (≠ j).
The learning unit 5 uses the filter coefficients w _ijk of the convolution layers C1 and C2, the bias b _{k of} each node, and all coupling layers arranged on the output layer side of the CNN so that the cross entropy C is reduced. Adjust the weight and bias of F.

A stochastic gradient descent method is used to minimize the classification error C. The learning unit 5 calculates an error gradient (∂C / ∂w _ij ) related to weights and biases by an error back propagation method (BP method). The calculation method by the BP method is the same as that of a normal neural network.
Of course, in the back propagation when the CNN processing unit 4 adopts the maximum pooling, it stores which node value of the pooling region is selected in the forward propagation for the learning sample, and is combined with only that node at the time of back propagation ( Combined with weight 1).

The evaluation of the classification error C by the learning unit 5 and the update of parameters (weights and the like) based on the evaluation may be performed for all learning samples. However, from the viewpoint of convergence and calculation speed, it is preferable to execute for each set (mini-batch) of several to several hundred samples. In this case, the update amount Δw _ij of the weight w _ij is determined by the following equation.

In the above equation, Δw _ij ^(t) is the current weight update amount, and Δw _ij ^(t−1) is the previous weight update amount. The first term of the above equation is a term representing the correction amount of w _ij for reducing the error by the gradient descent method, and ε is the learning rate.
The second term in the above formula is the momentum. The momentum suppresses the weight bias due to the selection of the mini-patch by adding α (˜0.9) times the previous update amount. The third term is weight decay. The weight attenuation is a parameter that prevents the weight from becoming excessive. The same applies to the update of the bias b _k .

[Processing content of image generator]
FIG. 5 is an explanatory diagram of the first process performed by the data generation unit 3.
As described above, the “first process” is a process for generating the image filter 9 having invariance of rotation and inversion with respect to the original images 7 and 8 from the sample image 7 or the target image 8.
In FIG. 5, the original image at the learning stage is the sample image 7, and the original image when the image processing apparatus 2 is used as a discriminator is the target image 8.

The original images 7 and 8 include RGB values (0 to 255) at the respective pixel points p (x, y) in the orthogonal coordinates. Here, as shown in FIG. 5A, the data string (R, G, B) having the RGB value at the pixel point p as an element is referred to as a color vector “g”.
The data generation unit 3 first extracts the center point c of the original images 7 and 8 and sets the extracted center point c as the origin of coordinates. Next, as shown in FIG. 5B, the data generation unit 3 converts the pixel point p of the orthogonal coordinates (x, y) into polar coordinates having the center point c as the origin (polarization process).

  Note that the origin of the polar coordinates does not necessarily have to be the center point c of the original images 7 and 8, but may be a predetermined point that is slightly shifted from the center point c.

Next, the data generation unit 3 uses the color vector g = (R, G, B) of an arbitrary pixel point p included in polar coordinates with the center point c as the origin as the radial color vector gr at the pixel point p. And tangential color vector gt. The decomposition of the color vector g is executed by the following equation.

Here, “g ^T ” is a transposed vector of the color vector g = (R, G, B). “R” is a unit vector in the radial direction at the pixel point p defined by the following equation. “T” is a unit vector in the tangential direction at the pixel point p defined by the following equation.

In the above equation, “R _θ ” is a rotation vector that rotates the unit vector r by an angle θ. In the present embodiment, since the direction of the unit vector t is a tangential direction (the angle from the unit vector r is 90 degrees), the value of the angle θ of the rotation matrix R _θ is θ = π / 2.
By performing the above calculation for all pixel points p included in the polar coordinates of the original images 7 and 8, each pixel point p includes a total of six types of elements (Rr, Rt, Gr, Gt, Br, Bt). A single channel image filter 9 is generated.

FIG. 6 is an explanatory diagram of the second process by the data generation unit 3.
As described above, the “second process” is a process for convolving the image filters 9 generated in the first process with the original images 7 and 8.
In FIG. 6, the original image at the learning stage is the sample image 7, and the original image when the image processing apparatus 2 is used as a discriminator is the target image 8.

[Invariance of image filter rotation and reversal]
FIG. 7 is an explanatory diagram of a conversion point p _{θ obtained} by rotating the pixel point p by a predetermined angle θ, an inverted inversion point p ′, and a plurality of change points p ′ _{θ obtained} by rotating and reversing (or reversing and rotating).
As shown in FIG. 7, a rotation point that is a point advanced by a predetermined angle θ counterclockwise with the same radius with respect to an arbitrary pixel point p is defined as “p _θ ”.
As shown in FIG. 7, an inversion point obtained by inverting an arbitrary pixel point p is “p ′”.
As shown in FIG. 7, a rotation point that is a point advanced by a predetermined angle θ counterclockwise with the same radius with respect to an inversion point p ′ of an arbitrary pixel point p is defined as “p ′ _θ ”. However, when there are a plurality of changes, the conversion procedure may be reversed rotation or rotation reversed.
Further, the color vectors at the rotation point p _θ , the inversion point p ′, and the rotation inversion (or inversion rotation) point p ′ _θ are respectively “g _θ ”, “g ′”, and “g ′ _θ ”, and the rotation point p ′ _θ. The unit vectors in the radial direction and the tangential direction in FIG. 4 are respectively “r _θ ”, “r ′”, “r ′ _θ ”, “t _θ ”, “t ′”, and “t ′ _θ ”.

In this case, as shown in the following equation, the color vectors in the radial direction and the tangential direction at the pixel point p, the rotation point p _θ , the inversion point p ′, and the plurality of conversion points p ′ _θ are (g ^T r, g ^T t) ^{_{, ((R θ g) T}} r, (R θ g) T t), ((Mg) T r, (Mg) T t), ((MR θ g) T r, (MR θ g) T t) Matches.
In the following equation, M is an inversion matrix. Since M is a diagonal matrix whose diagonal component is 1 or −1, M ^T M = I.

(For rotation)

(In the case of inversion)

(In case of rotation reversal (or reversal rotation))

The above equation holds regardless of the value of the angle θ around the center point c. That is, the color vector gr in the radial direction and the color vector gt in the tangential direction at the pixel point p are unchanged regardless of the angle θ around the center point c. Similarly, inversion and multiple changes have invariance. Note that the inversion includes both left-right inversion (y-axis symmetry) and up-down inversion (x-axis symmetry).
Therefore, if the process of convolving the image filter 9 having the color vectors gr and gt in each direction as elements is performed on the color vectors g of the original images 7 and 8 configured as discrete information, the input data after processing is The input data has invariance with respect to rotation and inversion around the center point c.

[Recommended CNN structure example]
FIG. 8 is a structural diagram of the deep CNN constructed in the CNN processing unit 4.
As shown in FIG. 8, the CNN architecture for image recognition recommended by the inventors of the present application is based on a stack of convolutional layers C1 to C4 for converting an input volume into an output volume, and all coupling layers A1 to A3. It is configured.

Each layer C1-C4, A1-A3 of the CNN has neurons that are arranged three-dimensionally in width, height and depth.
The size of the width, height, and depth of the first input layer C1 is preferably 56 × 56 × 3. Neurons inside the convolution layers C2 to C4 and the total connection layer A1 are connected only to a small area node called a receptive field of the previous layer.

The spatial size of the output volume can be calculated by the following equation.
W2 = 1 + (W1-K + 2P) / S
In the above equation, W1 is the size of the input volume. K is the field size of the nucleus (node) of the neuron in the convolution layer. S means the stride, that is, the distance between the centers of the receptive fields of adjacent neurons in the kernel map. P means the amount of zero padding used on the border.

In the CNN of FIG. 8, W1 = 56, K = 5, S = 2, and P = 2 in the first convolution layer C1. Therefore, the spatial size of the output volume of the second convolution layer C2 is W2 = 1 + (56-5 + 2 × 2) /2=28.5→28.
The network of FIG. 8 includes seven layers with weights. The first four are convolutional layers C1-C4, and the remaining three are all connected layers A1-A3 that are fully connected. All coupling layers A1 to A3 include dropouts.

The output of the last fully coupled layer A3 is fed to 7-way SOFTMAX, which generates a distribution of 7 class labels, which is the final layer fully connected to this layer A3.
The neurons of the convolutional layers C2 to C4 and the fully connected layer A1 are connected to the receptive field of the previous layer, and the neurons of the fully connected layers A2 to A3 are connected to all the neurons of the previous layer.

The convolution layers C1, C2 are followed by a batch normalization layer. Each batch normalization layer is followed by a pooling layer that performs the aforementioned maximum pooling.
The non-linear mapping function for the convolution layers C1 to C4 and all the coupling layers A1 to A3 is composed of a rectifying linear unit (ReLU).

The first convolution layer C1 filters an input image (AGE image) of 56 × 56 × 3 with a stride of 2 pixels by 64 kernels having a size of 5 × 5 × 3.
The stride is the distance between the centers of the receptive fields of adjacent neurons in the kernel map. The stride is set to 1 pixel in all convolution layers.

The input of the second convolution layer C2 is the output of the first convolution layer C1 which is batch normalized and maximum pooled. The second convolution layer C2 filters the input with 128 kernels that are 3 × 3 × 64 in size.
The third convolutional layer C3 has 128 kernels of size 3x3x64, which are connected to the output of the second layer C2 (batch normalization and MAX pooling).

  The fourth convolution layer C4 includes 128 kernels having a size of 3 × 3 × 128. All the fully connected layers A1 to A3 are each provided with 1024 neurons.

[Recommended learning examples]
The inventors of the present application actually trained (learned) the deep CNN having the structure of FIG. The training was performed using “MATLAB”, an open source numerical analysis software, on a PC that implements a 4GB GPU of NVIDIA GTX745.
In the learning step of CNN, there are important settings such as weight attenuation, momentum, batch size, learning rate and parameters including learning cycle. Hereinafter, this point will be described.

In the training by the present inventors, an asynchronous stochastic gradient descent method with a momentum of 0.9 and a weight decay of 0.0005 was adopted. The following equation is the weight w update rule adopted this time.

In the above equation, i is the number of iterations and m is a momentum variable. ε means the learning rate. The third term on the right side is an average value for the i-th batch Di of the correction amount of the weight w for reducing the error L in wi.
Increasing the batch size results in a more reliable gradient estimate and can reduce the learning time, but it does not provide the largest stable increase in learning rate ε. Therefore, it is necessary to select a batch size suitable for the CNN model.

Here, the results of training (learning) employing batch sizes of 64, 128, 256, and 512 were compared for the convolution layers C1 to C4, respectively. As a result, it was found that the batch size of 256 is optimal for the CNN of FIG.
In addition, the same learning rate was used for all strata and adjusted manually throughout the training. The learning rate was initialized to 0.1, and when the error rate stopped improving at the current learning rate, the learning rate was divided by 10. In the training, the network was trained in about 20 cycles.

[Experimental example: Effect of identifying handwritten characters]
The inventors of the present application use the handwritten character image included in the “Chairman's memorandum 1” of the “Kanebo Material Database” in the “Corporate Research Center of the University of Kobe” A comparative experiment was conducted to test the significance.

  The types of objects (handwritten characters) to be identified are “support”, “allocation”, “person”, “engineer”, “place”, “long”, “meeting (會)”, “ "Company", "Ming" and "Oji". The number of samples of each character used for learning was 400 (56 × 56 pixels). FIG. 9 is a diagram illustrating an example of a handwritten character (original image) used in the comparison experiment.

FIG. 10 is a graph showing a recognition accuracy test result for each character class. In FIG. 10, a circle (ours) indicates the recognition accuracy of the present invention. The square (alexnet) indicates the recognition accuracy in the case of Non-Patent Document 1 (however, the input data is RGB values).
The graph (vgg-vd-16) of ▽ indicates the recognition accuracy in the case of Non-Patent Document 2 (input data is RGB values and the number of layers is 16). The graph of * (vgg-vd-19) shows the recognition accuracy in the case of Non-Patent Document 2 (input data is RGB values and the number of layers is 19).

As shown in FIG. 10, in the case of Non-Patent Documents 1 and 2 using conventional RGB values as an input image, recognition accuracy exceeding 90% cannot be achieved for all ten character classes.
On the other hand, in the case of the present invention in which the original image of handwritten characters is invariant to rotation and reversal, recognition accuracy of 90% or more was obtained for all ten character classes.

  As is apparent from the graph of FIG. 10, in the image recognition using the deep CNN (character recognition in the example of FIG. 10), if processing for giving rotation and inversion invariance to the original image is performed as the input data of the CNN, Compared with the case where conventional raw data (RGB image) is used as input data, the recognition accuracy is significantly improved.

[Effect of image recognition device]
As described above, according to the image recognition apparatus 10 of the present embodiment, the input data to the CNN processing unit 4 is generated by convolving the original image 7 and 8 with the image filter 9 having invariance of rotation and inversion. To do. For this reason, even if it does not collect so many sample images 7, it can exhibit high recognition accuracy compared with other deep learning techniques.
For example, as shown in FIG. 10, when the number of sample images 7 is “400”, recognition accuracy higher than that of the conventional deep learning technique can be obtained.

In the present embodiment, the single-channel image filter 9 including six types of elements (Rr, Rt, Gr, Gt, Br, Bt) is convoluted with the sample image 7, so that the amount of data included in one sample image 7 is at least The expansion will be 16 times.
For this reason, when 100 sample images 7 are required in the conventional deep learning, if approximately 6 (100/16 = 6.25) sample images 7 are collected, the same as in the conventional deep learning. A certain degree of identification accuracy is obtained.

According to the image recognition apparatus 10 of the present embodiment, an object that has been rotated or reversed by the convolution of the image filter 9 can be accurately recognized.
Therefore, for example, it can be used as an apparatus for accurately recognizing characters included in a photographed image such as an old document and converting it into text or word processor document data. It can also be used as a device that reads characters described in a document such as a form and converts them into text or word processor document data, or a device that recognizes characters handwritten on the touch panel in real time.

In addition, the image recognition apparatus 10 according to the present embodiment can be used for recognition of human facial expressions, age recognition from face images, and recognition of all types of objects such as animals, plants, and products.
Thus, in the image recognition device 10 of the present embodiment, the object that can be recognized by the CNN processing unit 4 may be at least one object of handwritten characters, humans, animals, plants, and products. It can be used to recognize any object type.

[Other application examples of image recognition device]
FIG. 11 is an overall configuration diagram of the product monitoring system 20 according to the present embodiment.
The product monitoring system 20 is a system that uses the image recognition apparatus 10 (see FIG. 1) for recognizing the type of the product 25 included in the photographed image for sorting and taking out defective products.
As shown in FIG. 11, the product monitoring system 20 includes an imaging device 21, a drive control device 22, and a movable arm type robot device 23.

The imaging device 21 is composed of, for example, a digital camera that generates a digital image using a CCD (charge coupled device). The imaging device 21 is suspended above the belt conveyor 24, and images a plurality of products 25 on the belt conveyor 24 traveling downstream (right side in FIG. 11) from above.
The imaging device 21 transmits a captured image formed of a digital image including a plurality of products 25 to the drive control device 22. The captured image may be either a still image or a moving image.

The drive control device 22 includes a computer device that controls the operation of the robot device 23. The drive control device 22 includes a first communication unit 26, a second communication unit 27, a control unit 28, and a storage unit 29.
The first communication unit 26 includes a communication device that communicates with the imaging device 21 according to a predetermined I / O interface standard. Communication between the first communication unit 26 and the photographing apparatus 21 may be either wired communication or wireless communication.

  The second communication unit 27 includes a communication device that communicates with the robot device 23 according to a predetermined I / O interface standard. Communication between the second communication unit 27 and the robot apparatus 23 may be either wired communication or wireless communication.

The control unit 28 is an information processing apparatus having one or a plurality of CPUs, and includes a control apparatus including the above-described image identification apparatus (see FIG. 1) of the present embodiment.
The memory | storage part 29 consists of memory | storage devices containing memories, such as 1 or several RAM and ROM. The storage unit 29 functions as a temporary or non-temporary recording medium for various computer programs to be executed by the control unit 28 and image data received from the imaging device 21 or the like.

As described above, the drive control device 22 includes a computer. Therefore, each function of the drive control device 22 is exhibited when the computer program stored in the storage device of the computer is executed by the CPU and GPU of the computer.
Such a computer program can be stored in a temporary or non-temporary recording medium such as a CD-ROM or a USB memory.

The control unit 28 can realize communication control for the first and second communication units 26 and 27 and operation control of the robot device 23 by reading and executing the computer program stored in the storage unit 29.
For example, the control unit 28 extracts the image portion of the product 25 (hereinafter referred to as “product image”) from the captured image received by the first communication unit 26, and classifies the extracted product image (here, “normal” Or “bad”).

When the control unit 28 detects a product image with a bad classification class, the control unit 28 calculates the time and position at which the product 25 passes under the robot device 23 from the photographing time and the traveling speed of the belt conveyor 24, and the calculated passing time. The time and the passing position are transmitted to the second communication unit 27.
The robot apparatus 23 includes an articulated robot arm 30 and an actuator 31 that drives the robot arm 30. The robot arm 30 has a hand portion (not shown) that can grip the product 25 on the conveyor 24.

The passage time and passage position of the defective product calculated by the control unit 28 are transmitted to the actuator 31. The actuator 31 drives each joint of the robot arm 30 so that the hand part moves to the passing position at the passing time of the defective product, and the product 25 is grasped and taken out to the outside.
The storage unit 29 stores a CNN (for example, FIG. 8) having a predetermined structure capable of executing the quality determination of the product 25, a learned weight and bias for the CNN, and the like. The control unit 28 determines the quality of the product image extracted from the captured image based on the learned CNN.

It is as follows when the work process which should be performed by a factory manager required for implement | achieving the above-mentioned product monitoring system 20 is enumerated.
Step 1) An operator 32 on the downstream side of the robot arm 30 determines a defective product from the products 25 flowing on the conveyor 24, and photographs the defective product with a digital camera (not shown).
Step 2) The captured image data is input to the storage unit 29 of the drive control device 22 as a defective sample image 7.

Step 3) The above steps 1 and 2 are repeated until a desired identification accuracy (for example, 99% or more) is obtained.
If the representative shape of the defective product is known in advance, the sample image of the defective product may be image data obtained by photographing a product other than the product 25 flowing on the conveyor 24.

In the learning stage where Steps 1 and 2 are repeated, since there are few defective product sample images 7 at the beginning of operation, the recognition accuracy of the defective product by the drive control device 22 is low.
However, as the number of defective sample images 7 increases, the recognition accuracy of the drive control device 22 improves, and defective products can be eliminated in a state close to 100%. For this reason, when the drive control apparatus 22 learns with the predetermined number of sample images 7, visual monitoring by the worker 32 becomes unnecessary.

In particular, in the present embodiment, a discriminator with high recognition accuracy is used even for a rotated or inverted object, so that, for example, as shown in FIG. Defective products can be identified in a form close to 100%.
In addition, as an example of the product 25 put on the conveyor 24 in various directions, food products such as sweets and kneaded products, molded products produced by a molding machine, and the like are conceivable. In addition, when the product 25 is currently conveyed on the conveyor 24 and visually checked by the worker 32 and defective products are removed from the conveyor 24, the product monitoring system 20 of the present embodiment is employed. Therefore, it can be expected that the number of the workers 32 performing the visual inspection can be greatly reduced.

[First Modification]
In the embodiment described above, the separation direction of the color vector g is not limited to the “radial direction” (the r direction in FIG. 5) and the “tangential direction” (the t direction in FIG. 5). In other words, the separation direction of the color vector g may be any two directions that open at a predetermined angle with the pixel point p as a base point.
However, if the decomposition is performed in two directions other than the radial direction and the tangential direction, the calculation expression of the color vector in each direction becomes complicated, and the processing load on the data generation unit 3 increases. Therefore, as described above, it is preferable that the color vector g is decomposed in the radial direction and the tangential direction.

[Second Modification]
In the above-described embodiment, a three-dimensional polar coordinate having the center point c as the origin is defined, and the color vector g has elements that are decomposed into a radial direction and two tangential directions (three directions in the case of three dimensions). The original images 7 and 8 may be convolved with a three-dimensional image filter.
In this way, not only two-dimensional rotation or inversion with the center point c as the origin, but also the image recognition accuracy is improved for the target image 8 inclined in the depth direction with the center point c as the origin. be able to.

[Third Modification]
In the above-described embodiment, the image filter 9 having rotation and inversion invariance with respect to the original images 7 and 8 is employed. However, the image filter 9 has only rotation invariance with respect to the original images 7 and 8. The image filter may be an image filter or an image filter having invariance of only inversion with respect to the original images 7 and 8. That is, the image filter 9 of the present invention may be an image filter having at least one invariance of rotation and inversion with respect to the original images 7 and 8.

[Other variations]
The embodiments (including modifications) disclosed herein are illustrative and non-restrictive in every respect. The scope of rights of the present invention is not limited to the above-described embodiments, but includes all modifications within the scope equivalent to the configurations described in the claims.
For example, in the above-described embodiment, the neural network is a convolutional neural network (CNN), but may be a hierarchical neural network having another structure that does not have a convolutional layer.

DESCRIPTION OF SYMBOLS 1 Computation processing part 2 Image processing part 3 Data generation part 4 CNN processing part 5 Learning part 6 Recognition part 7 Sample image (original image)
8 Symmetric image (original image)
9 Image Filter 10 Image Recognition Device 20 Product Monitoring System 21 Imaging Device 22 Drive Control Device (Control Device)
23 Robot Device 24 Belt Conveyor 25 Product 26 First Communication Unit 27 Second Communication Unit 28 Control Unit 29 Storage Unit 30 Robot Arm 31 Actuator 32 Worker

Claims (9)

A data generation unit that performs predetermined data processing on the original image to generate input data;
An image processing unit comprising a hierarchical neural network that recognizes the type of object included in the generated input data,
When the original image is a sample image, the image processing unit performs processing for learning parameters of the network based on the recognition result of the network, and when the original image is a recognition target image, Perform processing to output network recognition results,
The data processing performed by the data generation unit is an image recognition apparatus that is a process of imparting at least one invariance of rotation and inversion to the original image. The image recognition apparatus according to claim 1, wherein the data processing performed by the data generation unit includes a first process and a second process defined below.
1st process: The process which produces | generates the image filter which has at least 1 invariance of rotation and inversion with respect to an original image 2nd process: The process which convolves the image filter produced | generated by the 1st process with an original image   The image filter includes a plurality of color vectors obtained by dividing a color vector of an arbitrary pixel point defined by polar coordinates having a predetermined point of the original image as an origin into an arbitrary direction opening from the pixel point as a starting point at a predetermined angle. The image recognition apparatus according to claim 2, wherein the image recognition apparatus comprises elements included in   The image filter includes two color vectors obtained by dividing a color vector of an arbitrary pixel point defined by polar coordinates with a predetermined point of the original image as an origin into a radial direction and a tangential direction starting from the pixel point. The image recognition apparatus according to claim 2, comprising the above elements.   The image recognition apparatus according to claim 1, wherein the hierarchical neural network is a convolutional neural network.   The image recognition apparatus according to any one of claims 1 to 5, wherein the object whose type is recognized includes at least one object of handwritten characters, humans, animals, plants, and products. A data generation unit that performs predetermined data processing on the original image to generate input data;
A computer program for causing a computer to function as an image recognition device comprising an image processing unit having a hierarchical neural network that recognizes an object included in the generated input data,
The data generation unit performing, as the data processing, a process of imparting at least one invariance of rotation and inversion to the original image;
The image processing unit, when the original image is a sample image, performing a process of learning parameters of the network based on the recognition result of the network;
And a step of outputting a recognition result of the network when the original image is a recognition target image. A data generation unit that performs predetermined data processing on the original image to generate input data;
An image recognition method executed by an image recognition apparatus comprising: an image processing unit having a hierarchical neural network that recognizes an object included in the generated input data,
The data generation unit performing, as the data processing, a process of imparting at least one invariance of rotation and inversion to the original image;
The image processing unit, when the original image is a sample image, performing a process of learning parameters of the network based on the recognition result of the network;
And a step of outputting a recognition result of the network when the original image is a recognition target image. A photographing device for photographing a plurality of products;
A robot apparatus for taking out one of the plurality of photographed products to the outside;
A product monitoring system comprising: a control device that instructs the robot device to take out the product to be extracted;
The control device includes the image recognition device according to any one of claims 1 to 6,
The product recognition system, wherein the image recognition device instructs the robot device to take out the product recognized as a defective product.