JP2020119154A - Information processing device, information processing method, and program - Google Patents

Abstract

To enhance the data recognition accuracy of a recognition process, and shorten the time of the recognition process.SOLUTION: An information processing device includes an input unit 21 that acquires recognition target data, a feature extraction unit 221 that maps the recognition target data in a feature space, a representative feature amount generation unit 222 that generates a plurality of points in a data space and/or the feature space of the recognition target data, a distance calculation unit 223 that compares the recognition target data with the plurality of points in the feature space and/or the data space, and an abnormality determination unit 31 that outputs a recognition process result based on the comparison result.SELECTED DRAWING: Figure 1

Description

The present invention relates to a technique for recognizing a recognition target from data such as video and images.

In order to recognize an object as a recognition target and its state from data such as video and images, the feature amount is extracted using the data of the recognition target, and the distance from the feature amount to the feature amount of the nearest learning data. There is a technique for discriminating the state or class of the recognition target based on the. For example, as a method for detecting abnormality, Non-Patent Document 1 discloses a method of detecting abnormality based on a distance from normal data existing in the vicinity of recognition target data or a distance from a cluster of normal data. ing.

Nearest-Neighbor and Clustering based Anomaly Detection Algorithms for RapidMiner. M. Amer and M. Goldstein. 2012

Here, in order to recognize the input data at a higher speed, a recognition system with reduced recognition processing time is desired. Further, in order to recognize the input data with higher accuracy, a recognition system having further improved recognition ability is desired.
When calculating the distance between the recognition target data and the neighboring data as in Non-Patent Document 1, a processing time depending on the number of learning data is required for the neighboring data search processing. Further, when calculating the distance between the recognition target data and the center of the cluster as in Non-Patent Document 1, similarly, processing time depending on the number of clusters is required. On the other hand, when the learning data and the amount thereof are diverse and large, it is necessary to approximate the learning data by using a large number of cluster types and the number of classes in order to express various learning data. It may not be desirable for the number dependent processing to occur. However, if the number of clusters is reduced in order to reduce the processing time that depends on the number of clusters, the degree of deviation between the cluster center and the data that belongs to the clusters increases, so there is an The percentage may increase.

Therefore, it is an object of the present invention to improve the accuracy of data recognition in recognition processing and reduce the recognition processing time.

The information processing apparatus of the present invention includes a plurality of acquisition means for acquiring recognition target data, mapping means for mapping the recognition target data into a feature space, and at least one of the feature space and the data space of the recognition target data. Generating means for generating points, comparing means for comparing the recognition target data and the plurality of points in at least one of the feature space and the data space, and outputting a recognition processing result based on the result of the comparison. And an output unit that operates.

According to the present invention, it is possible to improve the accuracy of data recognition in the recognition processing and reduce the recognition processing time.

It is a figure which shows the structural example of an abnormality detection system. It is a flowchart which shows operation|movement (at the time of learning) of an abnormality detection system. It is a flow chart which shows operation of a selection part. It is a figure which shows the example of an input image and a human body region. It is a flowchart which shows operation|movement of a learning part. It is a figure which shows the structural example (at the time of learning) of CNN. It is a flowchart which shows operation|movement (at the time of detection) of an abnormality detection system. It is a figure which shows the structural example (at the time of detection) of CNN. It is a figure which shows the structural example (at the time of learning) of Siamese CNN. It is a figure which shows the structural example (at the time of learning) of CNN. It is a figure which shows the structural example (at the time of learning) of Autoencoder. It is a figure which shows the structural example (at the time of learning) at the time of generating a representative point by using auxiliary information. FIG. 6 is a diagram showing an example of representative point generation and hierarchical cluster loss in a plurality of layers. It is a figure which shows the example of the representative point generator for every cluster. It is a figure which shows the example of a multistep sequence length.

Embodiments of the present invention will be described below with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.
<First Embodiment>
In the following description, the data regarding the recognition target for the object and its state will be referred to as recognition target data, and the feature amount extracted from the recognition target data will be referred to as the target feature amount. Note that even when the recognition target data is mapped onto the feature space, it is generally considered that it is often referred to as data on the feature space. Therefore, when simply referred to as recognition target data in the following description, for example, the original image Not only the data on the space but also the data on the feature space are included. The recognition target may be specified by a human, may be specified by a machine as a processing target, and may be singular or plural. In each of the following cases, the nature of the recognition target may be described as a specific example, but the present invention is not intended to be limited to such a case in principle. In the present embodiment, firstly, an example of performing recognition processing of a recognition target in a constant time that does not depend on the number of learning data will be shown. In addition, since an example of the abnormality detection system is given in the present embodiment, an example of an algorithm for learning for the purpose of abnormality detection is given. Examples of the learning algorithm include a Two-Class learning algorithm based on a normal class/abnormal class, and a One-Class learning using only normal data for learning. Of course, the present invention can be used for other purposes, and various configuration examples will be described below in order.

Although details will be described later, in the present embodiment, a point in the vicinity of the recognition target data (hereinafter, referred to as a recognition target vicinity) is predicted as a representative point to be compared with the recognition target data, and the representative point and the recognition target data are compared with each other. An example of a configuration and an operation of calculating a distance between the two and performing abnormality detection based on the calculated distance will be described. Here, it is assumed that the representative points predicted based on the recognition target data have been learned in advance so as to represent normal data in the vicinity of the recognition target data. It is assumed that there are two types of learning described here, that is, learning related to processing for generating a representative point and learning related to processing for performing feature extraction. As will be described later, these two types of learning processing may be regarded as one learning processing system. In addition, the representative points predicted based on the recognition target data represent learning data in the vicinity of the recognition target data. Therefore, by using the representative points, a smaller number than in the case of comparing all the learning data with the recognition target data, or in the case of comparing all the cluster centers and the recognition target after clustering all the learning data. It becomes possible to recognize the recognition target by comparing Originally, the distance between the recognition target data and the neighborhood determined based on the recognition target neighborhood data (or the cluster center thereof) is predicted to be "here" based on the recognition target data. It is calculated by the distance from the "point to be compared with the recognition target data". Therefore, in this embodiment, the expression "predicting" the representative point may be used, which in other words can be said to "generate" the point predicted to be the representative point. Further, in the present embodiment, first, the description will be made assuming that the number of representative points is fixed M. Then, in the present embodiment, the feature extractor and the representative point generator are learned so that M representative points are sufficient. Although the term “representative point” is used here, the configuration is not limited to a point as described later. In addition, it will be described later that it can be used for purposes other than the pattern "the representative point is considered to represent normal data in the vicinity of the recognition target" depending on the way of learning the representative point.

In the following, for example, learning is performed by using, for example, Convolutional Neural Network (CNN) for feature extraction and representative point generation. The representative point will be referred to as a representative feature amount hereinafter, as it is generated in the CNN feature space. As will be described later, the CNN is not an indispensable configuration example, but an example of the feature extractor.

In the present embodiment, the configuration of the abnormality detection system is taken as a specific example, and as the abnormality detection system, a system having a function of detecting an abnormality from an image captured by a surveillance camera is illustrated.
In the abnormality detection system according to the present embodiment, the monitoring target is photographed by an imaging device such as a camera, and it is determined whether or not the monitoring target is abnormal based on the photographed video data. When it is determined that there is an abnormality, a warning indicating that there is an abnormality, for example, a warning is displayed or a warning sound is output, is issued to a monitor resident in a monitoring center such as a security room. The monitoring target includes, for example, indoors and outdoors of general households, or public facilities such as hospitals and stations.

The operation of the abnormality detection system in the present embodiment includes two stages, "at the time of learning" and "at the time of detection". Of these two stages, the first first stage shows the operation when learning is performed, and the second second stage shows the operation when actual detection processing is performed based on the learned result. ing.
FIG. 1 is a block diagram showing a schematic configuration example of an abnormality detection system as an example of the information processing apparatus according to the first embodiment. The abnormality detection system of this embodiment includes a learning device 10, a learning data storage unit D1, a recognition device 20, a recognition target data storage unit D2, a determination device 30, and a terminal device 40.

First, the configuration related to the operation during learning in the abnormality detection system of this embodiment will be described.
The abnormality detection system in FIG. 1 includes a learning device 10 as a configuration related to learning. It should be noted that these devices and devices may be connected via an electronic circuit, an external storage device, or a network. A mobile phone network or the Internet can be applied to this network, for example. The devices described below may be similarly connected by various methods. This also applies to the recognition device 20 and the like described later.
The learning device 10 includes a selection unit 11 and a learning unit 12.
The learning unit 12 includes a feature extraction unit 121, a representative feature amount generation unit 122, and a loss calculation unit 123.

Next, the operation during learning in the abnormality detection system will be described with reference to FIG. FIG. 2 is a flowchart showing the operation at the time of learning in the abnormality detection system. In the following description, steps S200 to S204 in the flowchart of FIG. 2 will be abbreviated as S200 to S204. This also applies to other flowcharts described later.

First, in S200, the learning device 10 performs an initialization process. For example, the learning device 10 initializes the CNN used during learning. A specific example will be described later.
Next, in S201, the selection unit 11 reads the learning target data that is the target of the learning process from the learning data storage unit D1, and sends the minibatch data to the learning unit 12. The feature extraction unit 121 receives the learning data from the selection unit 11, extracts a feature amount by a feature extractor based on the received data, and generates a recognition target feature amount (hereinafter, referred to as a target feature amount) as a representative feature amount. To the loss calculation unit 123. The representative feature amount generation unit 122 generates a representative feature amount from the received target feature amount by the representative feature amount generator and sends it to the loss calculation unit 123. The loss calculation unit 123 calculates a loss value from the received minibat feature amount and representative feature amount based on the loss function. Hereinafter, the loss value and the gradient information obtained by differentiating the loss function will be referred to as error-related information or simply error. Details of these will be described later.

Next, in S202, the loss calculation unit 123 sends the error information to the representative feature amount generation unit 122 and the feature extraction unit 121. The representative feature amount generation unit 122 further calculates the error of the representative feature amount generator based on the received error information and sends it to the feature extraction unit 121. The feature extraction unit 121 uses the error (which is the loss of the upper layer of CNN) received from the loss calculation unit 123 and the representative feature amount generation unit 122 to perform the learning process of the feature extractor. Details will be described later.

In step S203, the representative feature amount generation unit 122 uses the error information received from the loss calculation unit 123 in step S202 to perform the learning process of the representative feature amount generator.
After that, in S204, when the learning is completed, the learning device 10 stores the feature extractor and the representative feature amount generation unit that have been learned so far in the learning data storage unit D1.

Next, the operation of the selection unit 11 will be described in detail with reference to FIG. FIG. 3 is a flowchart showing the learning data selecting operation in the selecting unit 11.
First, in S301, the selection unit 11 receives the learning data from the learning data storage unit D1. The learning data here is assumed to be a person image created from a surveillance video obtained from a certain surveillance camera.

Here, an example of an image obtained by extracting one frame from a certain monitoring video is shown in FIG. An image 401 shown in FIG. 4 shows an example of a surveillance video image at a certain intersection, and it is assumed that objects 402 to 405 such as the imaged subject are present in this image 401. Areas 406 to 409 indicated by dotted lines in the image 401 represent Bounding Boxes (bounding boxes) extracted by the human body region extractor included in the selection unit 11. Each of the partial images surrounded by these Bounding Boxes corresponds to a human body image. It should be noted that the Bounding Box shown here is only one of the specific examples when the human body region is extracted, and even if a small region along the contour of the object imaged by the background subtraction method described later is extracted, for example. Good.

There are a plurality of methods for extracting the human body region as described above, and there are three methods, for example, the background subtraction method, the object detection/tracking method, and the area division method. When the object to be monitored is a known object such as a human body in advance, it is considered that the object detection/tracking method narrowed down to the purpose of detecting/tracking only the target object is relatively suitable. The object detection/tracking method includes, for example, the method disclosed in Reference Document 1, and this method may be used.

Reference 1: Real-Time Tracking via On-line Boosting. H. Grabner, M. Grabner and H. Bischof. Proceedings of the British Machine Conference, pages 6.1-6. 10. BMVA Press, September 2006.

Further, the selection unit 11 uses the teacher data previously given to the learning data (the human body image cut out as a human body region in a rectangular region or the like from the monitoring video) to give the teacher data to the human body image. For example, when a rectangular human body area is extracted, a bounding box of the human body area can be defined and teacher data can be added to the bounding box. Note that when a human body region is extracted along a contour where a human body exists, instead of a rectangle, a mask of the human body region for an image can be defined and a human body image can be created based on the mask.

Here, the teacher data is a label indicating how to classify the target. Since the type of label and what kind of label is given to what target depends on the problem, in the present embodiment, the user who uses or introduces the abnormality detection system determines in advance and sets the learning data in the learning data. On the other hand, teacher data should be added. For example, human body images obtained from normal images may be regarded as normal, and those human body images may be labeled as normal. In addition, hereinafter, a label "abnormality" is given to the human body image that the observer has determined to be abnormal. That is, the problem setting here is a normal/abnormal two-class problem related to (a human body image included in) an image obtained from a specific surveillance camera.

The teacher data that is previously added to the learning data can be added by the user to, for example, the region on the image of the captured subject. More specifically, for example, the area of the pedestrian object 403 can be manually designated by the user, and a label indicating normality can be given to the area. At this time, when the extracted human body region and the region of the imparted teacher data overlap each other, the teacher data superimposed on the human body region in the largest area ratio may be imparted. Of course, it is not always necessary to attach the teacher data in the above-described manner, and for example, the human body region may be extracted in advance and the user may attach the teacher data to each human body region. In the present embodiment, an example in which the human body region is extracted has been shown, but naturally other object regions may be extracted, or the entire image may be recognized without extracting the region. It is necessary to set which region is extracted and which is to be the target of the recognition process because it depends on the problem. Further, here, for simplification of description, it is assumed that learning data (human body image) is created in advance from all the monitoring videos that are targets of the learning process.

Next, when proceeding to S302, the selection unit 11 performs preprocessing of the obtained learning data. In the present embodiment, it is assumed that each image is transformed into an image size of 224×224 pixels, and then the process of subtracting the average image is performed. Here, the average image indicates the average image of the learning data. In the present embodiment, the learning image is inverted in the horizontal direction to be padded, and the image before padding and the image after padding are all treated as the learning image.

Next, when proceeding to S303, the selection unit 11 determines the learning data to be a recognition target, and performs a neighborhood search process for each recognition target. As for the learning data to be recognized, all learning data may be treated as recognition targets, or important learning data may be selected in advance and selected as recognition targets. The recognition target selected here is one piece of learning data. That is, when a certain recognition target is selected, a plurality of neighborhood data similar to the recognition target is selected. It is also possible to configure the recognition target by a plurality of data, and such an example will be described later. Also, although it is possible to select only one, if there may be different types of neighborhood data distributions in the neighborhood of the recognition target, it is considered that a plurality of neighborhood data are good.

It is assumed that the space for performing the neighborhood search is the image space as the first example described here. That is, the selection unit 11 considers that the learning data, which is cut out from the monitoring video and preprocessed, is distributed in the image space, and performs a neighborhood search of the recognition target data. Besides the image space, for example, a feature space may be used, which will be described later. Any known means may be selected as the method using the neighborhood search, but here, the k-Nearest Neighbor (k-NN) described in Reference Document 2 is used. Note that k-NN may be executed only once at the beginning, may be executed at the start of each epoch, or may be executed each time a recognition target for learning is selected (a so-called minibatch is created). I say good.

Reference 2: C. Huang et al. Learning Deep Representation for Imbalanced Classification. CVPR2016.

Next, in S304, the selection unit 11 sets Minibatch as the learning data subset. In the first example of the present embodiment, the size of the Minibatch is one recognition target and 99 recognition target neighborhood data are grouped together. It is assumed that this recognition target neighborhood is obtained by the above-mentioned neighborhood search processing result. As this Minibatch, all Minibatches may be created at once for use in learning, may be recreated each time the learning epoch progresses, or may be created at any other timing of learning. I shall. In this example, it is assumed that all Minibatches are first created at one time to obtain a Minibatch set. In addition, you may use the method of the reference document 3 mentioned later as an example of a learning process.

In step S<b>305, the selection unit 11 selects one Minibatch from the Minibatch set and sends it to the learning unit 12. As the Minibatch to be selected at this time, for example, the Minibatch is selected in a random order. Note that the epoch represents the number of times all Minibatches of the Minibatch set have been learned (counting one epoch after learning once). In the present embodiment, as one example, it is assumed that the learning completion is determined based on a predetermined number of epochs.
Then, if it progresses to S306, the selection part 11 will end a learning process, when the predetermined number of epochs has been reached.

Next, the operation of the learning unit 12 will be described in detail with reference to FIG. FIG. 5 is a flowchart showing the learning operation in the learning unit 12.
First, in step S501, the learning unit 12 initializes the CNN parameters (coupling weights, bias terms, learning parameters, etc.). Upon initialization, it is necessary to determine the network structure of CNN in advance. The network structure and the initial parameters used here may be the same as those in Reference Document 3, or may be a network structure defined uniquely.

Reference 3: A. Krizhevsky et al. ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems 25 (NIPS), 2012.

FIG. 6 is a diagram showing an example of a schematic diagram of CNN. An example of the configuration and operation of the CNN will be briefly described with reference to FIG. The feature extractor 620 and the representative point generator 621 of FIG. 6 show an example of the network structure of the CNN used in this embodiment. In FIG. 6, the feature extractor 620 is shown to be composed of an input layer 601, a convolution 1 layer 602, a pooling 1 layer 603, a convolution 2 layer 604, and a pooling 2 layer 1305. Further, FIG. 6 shows that a convolution process 610, a pooling process 611, and a convolution process 612 are set as the processing method between the two layers. Since the specific contents of each process are the same as those in Reference Document 3, the description thereof is omitted here. In the convolution process 610, data processing using a convolution filter is executed, and in the pooling process 611, for example, if max pooling, a local maximum value is output.

Further, FIG. 6 shows that a plurality of feature maps exist in the convolution layer and the pooling layer, and a plurality of neurons exist at positions corresponding to pixels on the image of the input layer. For example, if the learning image is an image in the RGB format of three primary colors, there are three neurons corresponding to the RGB channels. Further, if the learning image is an optical flow image having motion information of the picked-up image, there are two types of neurons expressing the horizontal axis direction and the vertical axis direction of the image.

Further, when a plurality of images are used as inputs at the same time, it is possible to deal with them by increasing the number of neurons in the input layer by an amount corresponding to the number of those input images. In this embodiment, an example in which a standard RGB image is targeted is shown. FIG. 6 illustrates sizes 630 to 635 of the feature maps and the like in each layer at the time of learning. In these sizes 630 to 635, in the description (a, b, c, d), a is a dimension corresponding to the number of data, for example, N is the size of minibatch and M is the number of representative points described later. .. It can be considered that b is the number of feature dimensions corresponding to the channel of the image, c is the number of feature dimensions corresponding to the y-axis of the image, and d is the number of feature dimensions corresponding to the x-axis of the image. Although the above part is described as a feature extractor here, it is also possible to regard the entire CNN as a feature extractor as a property of CNN, and if necessary, even if it is interpreted as such. Good.

In addition, FIG. 6 shows that the representative point generator 621 includes a target feature amount layer 605 that is a recognition target and a representative feature amount layer 606. The target feature amount layer 605 is a layer that holds the feature amount of the recognition target. In the recognition target extraction processing 613, based on the rule that the first data of the minibat is the recognition target, the first (1,512) is calculated from the intermediate feature amount of the (N,512,32,32) size of the Convolution 2 layer. , 32, 32) size feature quantity is extracted. Then, the extracted feature amount is set as the target feature amount. As a matter of course, if it is possible to determine which data is the recognition target, it is not necessary to put the recognition target in the first position of the minibatch, and simply use the mark indicating that it is the recognition target. It may be replaced with a process of extracting the target feature amount.

Next, the learning unit 12 performs a representative point generation process 614 to generate M representative feature amounts of the representative feature amount layer 606 by using the feature amounts of the target feature amount layer 606. A specific example of the representative point generation processing 614 will be described later.
Finally, the learning unit 12 obtains the loss value 607 by the loss calculation process 615. A specific example of the loss calculation processing 615 will be described later.
The above is an example of the configuration and operation of the CNN during learning. As the initialization method, the CNN parameters can be initialized by a known method. Parameters such as the learning rate can be arbitrarily determined as needed.

Next, in S502, the learning unit 12 initializes the variable Iteration to 0. Here, the variable Iteration has a meaning of how many times the minibatch is read to update the CNN. If necessary, parameters necessary for learning such as the learning rate may be changed based on the variable Iteration or the number of epochs. For example, for stable learning, a process of multiplying the learning rate by 1/10 when the number of epochs exceeds 10 can be introduced.

Next, in S<b>503, the learning unit 12 receives the Minibat for learning from the selection unit 11.
Further, when proceeding to S504, the learning unit 12 executes CNN learning recognition processing. Specifically, the learning unit 12 performs the hierarchical feature extraction processing in S5041, the representative point generation processing in S5042, and the loss calculation processing in S5043. The outline of these processes was described above when the contents of FIG. 4 were described, but the detailed contents of the processes will be described later. Here, basically, the loss is calculated based on the N feature amounts of the Convolution 2 layer and the generated M representative points (representative feature amount). Although the representative point generation process is performed after the feature extraction process and the loss calculation process is performed after that, other various configurations such as a case of repeatedly executing the process as described later are also possible. It is also possible to Further, for example, learning may be performed exclusively by learning only the feature extractor and then learning only the representative point generator. The loss calculation process when learning only the feature extractor will be described later.

Next, in S505, the learning unit 12 back propagates the learning error to the CNN based on the loss calculated in S5043. In the present embodiment, a method combining the error back propagation method and SGD is used as the method for learning CNN. The method that combines the error backpropagation method and the SGD is described in detail in Reference Document 3, and thus will not be described in detail here. Basically, Minibatch is selected and the CNN parameters are sequentially updated. It is characterized by repeating the procedure. Note that the error information is generally propagated in the opposite direction to the data flow of the recognition processing, and here also the error information is propagated to the feature extractor, the representative point generator, and the like.

Next, in S506, the learning unit 12 updates the CNN parameters using the error information propagated in S505. For example, the learning unit 12 adds 1 to the variable Iteration.
Next, in S507, the learning unit 12 determines whether all Minibatches have been used for learning. If the learning unit 12 determines that all Minibatches have been used for learning, the learning unit 12 proceeds to the process of S508, and otherwise returns to the process of S503.

When the process proceeds to S508, the learning unit 12 adds 1 to the epoch.
Next, in S509, the learning unit 12 determines whether or not the epoch has reached the preset upper limit. If the learning unit 12 determines that the upper limit is reached, the learning of the CNN is terminated and the process proceeds to S510, and if not, the process proceeds to S503. In many cases, the learning stop condition of the NN employs either of whether or not the predetermined upper limit value of the epoch number is reached, automatic determination using the slope of the learning curve, or the like. In the present embodiment, the learning stop condition is whether or not the upper limit value of the number of epochs has been reached. For example, in this embodiment, the upper limit value of the number of epochs is set to 20000 times as an example. A learning process based on the number of epochs may be introduced, such as a method of decreasing the learning rate of the NN by increasing the number of epochs.

When the process proceeds to S510, the learning unit 12 stores the learned CNN model in the learning data storage unit D1. Here, in consideration of the use of the recognition device in the operation of the recognition process in the latter stage, it is assumed that the recognition device is used to store it in the memory of the computer used for learning, but for example, another storage unit is prepared and stored. Alternatively, other storage methods may be used.

In the above-described CNN feature extraction processing, learning is performed through feature learning so as to reduce the loss value based on the loss function. For example, learning is performed so that the representative point and the recognition target data are brought close to each other. As a result, the recognition process can be performed accurately.

Next, a specific processing content of the representative point generation processing will be described with an example. In the first example of this embodiment, as described above, the M representative points are generated based on the recognition target data. In the example of the representative point generation process 614 illustrated in FIG. 6, the process of converting the feature amount of (1,512,32,32) size to the feature amount of (M,512,32,32) size is the representative point. This is a generation process (the size of the feature amount is an example).

The representative point generation process may be a process that generates M representative points based on the recognition target, and thus the conversion process described above can be applied. Note that any representative point generation processing can be adopted as necessary. Here, the following representative point generation processing will be described as an example.
First, the learning unit 12 copies M feature amounts of (1,512,32,32) size to create feature amounts of (M,512,32,32) size, and then (M,512,32). The feature quantity of 32, 32) size is transformed into (1, 512·M, 32, 32) size. Note that "." represents multiplication of scalar values. For the transformation, M feature amounts of (1, 512, 32, 32) size may be concatenate-processed in the second dimension of the feature amount, or other known means may be used. Here, the concatenate process is a process of combining two pieces of data and feature amounts in a specified dimension.

Next, the learning unit 12 performs a conversion process from the feature amount of (1,512,32,32) size to the feature amount of (1,512·M,32,32) size. At this time, there are only M identical feature amounts before the conversion, but it is aimed that different M feature amounts appear by the conversion processing. Since a known method can be used for this processing, detailed description thereof will be omitted. For example, if it is not necessary to perform a large non-linear transformation on the feature quantity, a margin of one size may be created around the feature quantity before the transformation and the transformation process may be performed using a 3×3 convolution function. Further, in the above-mentioned example, an example in which M target feature amounts are copied and then conversion processing is performed, but a known method using, for example, Deconvolution (deconvolution function) may be used. That is, the target feature amount of (1,512,32,32) size may be converted into the feature amount of (1,512·M,32,32) size by the deconvolution process. Moreover, you may use methods other than another well-known deconvolution function.

Finally, the learning unit 12 performs a process of transforming the converted (1,512·M, 32, 32) size feature amount into a (M, 512, 32, 32) size feature amount. The transformation process used here may be performed, for example, by the method described above. Note that these examples are the configuration examples of the representative point generator shown for reference, and may be used as (a part of) the representative point generator by using other known methods.

Hereinafter, an example of configuring the representative point generator by a method other than this time will be described later. In a second embodiment to be described later, an example of using a generative model system method that has been developed in recent years for representative point generation will be described. Although M may be 1, it is considered better to use a plurality of representative points in order to improve the recognition accuracy. Moreover, when N is the number of the entire learning data, an objective function such as clustering can be obtained.

Next, an example of specific processing contents of the loss calculation processing will be described. In the first example of the present embodiment, as described above, the M representative points are compared with the Minibatch consisting of N pieces of data including the recognition target data, and the loss is calculated. At this time, there is a variation in how to perform the loss calculation process, but when solving as a normal-abnormal two-class problem as described above, the loss function shown in the following equation (1) is used. be able to.

Here, X in the equation (1) is a feature amount of Minibatch data, and A is a representative feature amount. In particular, X ₀ is the recognition target data. In addition, at least Minibatch data other than the recognition target is normal. ||·|| is a function for finding the distance between two data (features), and for example, the squared Frobenius norm is used. Formula (1) has a smaller value as the representative feature value A more accurately approximates normal Minibatch data (normal except for the recognition target). Further, when the label of the recognition target is abnormal, it can be seen that the value of the loss function becomes smaller as A is farther from the recognition target data. That is, it can be interpreted that A represents a normal representative feature amount.

In Expression (1), the distance between the feature amount of the Minibatch data and the representative feature amount is calculated, but when considering an example in which the data in the vicinity of the recognition target has multiple peaks, for each representative feature amount, You may use only the distance between the feature-values of the Minibatch data used as the nearest neighbor. As a result, it is possible to perform matching considering multimodality. In that case, the nearest neighbor can be found using k-NN on the feature space, for example. Note that only the distance between the representative feature amount that is the closest to the feature amount of each Minibatch data may be used. If necessary, the distance to k may be calculated as an arbitrary number instead of k=1. For example, when k=M, the loss function becomes the same as the equation (1).

The expression (1) can be differentiated for each case, and can be used as the objective function of the above-described back propagation method. Further, for example, in order to maintain the diversity of the representative points, if there is Minibatch data already assigned to a certain representative point, the data in other Minibatches may not be assigned to the same representative point. That is, in the case of N>M, only M pieces of data contribute to learning in the Minibatch. On the other hand, when N<M, only N representative points are used for learning. That is, it becomes a dynamic representative score. Furthermore, it is also possible to judge whether or not each data in the Minibatch is noise data (disturbance) by the multitask learning described later, and select the data in the Minibatch based on the noise degree. In this case, N will fluctuate, which can also be a dynamic representative score. If it is considered that the constraint is too tight as in the case where only one piece is assigned, only a small number of data in the Minibatch may be assigned to the representative point.
The above is the description of the operation of the learning device 10. A more detailed description of each unit will be given later.

Next, the configuration related to the operation at the time of detection of the abnormality detection system will be described.
The recognition device 20 of the abnormality detection system in FIG. 1 includes an input unit 21 and a recognition unit 22.
The recognition unit 22 includes a feature extraction unit 221, a representative feature amount generation unit 222, and a distance calculation unit 223.
The determination device 30 includes an abnormality determination unit 31.
The terminal device 40 includes a display unit 41. Note that, as the terminal device 40, for example, a display of a PC (Personal Computer), a tablet PC, a smartphone, a future phone, or the like can be applied.

Hereinafter, the operation of the abnormality detection system at the time of detection will be described with reference to FIG. 7. FIG. 7 is a flowchart of the abnormality detection operation in the abnormality detection system.
First, in step S<b>701, the input unit 21 receives recognition target data (detection target data) from the recognition target data storage unit D<b>2 and sends the recognition target data to the recognition unit 22. At this time, it is assumed that the recognition target data is subjected to the same preprocessing as during learning. In the pre-processing, processing such as subtracting an average image may be performed, and image inversion processing may not be performed. The recognition unit 22 performs CNN recognition processing using the received recognition target data. As the CNN model used at this time, the model obtained at the time of learning is used. If necessary, other models may be used instead of using the model obtained during learning of the abnormality detection system. In this embodiment, the model learned as described above is used.

To explain with FIG. 6 as an example, one recognition target is given at the time of detection as shown in FIG. 6 when N=1. A specific example is shown in FIG. Note that, in FIG. 8, the same processing contents as those in FIG. 6 are denoted by the same reference numerals as those in FIG. 6, and redundant description will be omitted.
In FIG. 8, sizes 830 to 833 represent that only one recognition target is input at the time of detection in the present embodiment, and the recognition target is subjected to recognition processing. Further, in the case of FIG. 8, there is no loss calculation process, and the distance 807 is obtained using the distance calculation process 815 that calculates the distance between the target feature amount and the representative feature amount.

Next, in step S<b>702, the determination device 30 performs abnormality determination processing on the CNN recognition processing result for the recognition target data. In this embodiment, the CNN recognition processing result that appears at this time is the distance between the recognition target and the generated representative point.

Here, at the time of learning of the present embodiment, since the representative points are learned so as to represent normal data in the vicinity of the recognition target, the representative points generated based on the recognition target when the recognition target is given. Is a representative point predicted to exist near the recognition target. That is, the distance between the recognition target and the representative point can be regarded as the distance between the recognition target and a normal point that may exist near the recognition target. Since it is considered that the larger the distance is, the more abnormal it is, the abnormality can be detected based on the distance obtained here. At this time, it is assumed that a threshold value used for the abnormality determination is a predetermined value here. The determination device 30 determines normal when the distance is within the threshold, and detects abnormal when the distance exceeds the threshold. The threshold may be determined by the operator, may be determined mechanically, or may be determined by another learning method or the like.

Next, when proceeding to S703, the terminal device 40 receives the abnormality determination result from the determination device 30 and performs display processing based on the determination result. For example, when the abnormality determination result indicating that the detection target is abnormal is sent, the display unit 41 may receive the abnormality determination result and perform a warning process. At this time, warning processing may be performed according to the function of the terminal device 40. For example, if a lamp and a siren are provided, you may sound a warning sound with the blinking of the lamp, etc.If a display for checking the image is provided, highlight an abnormal place in the normal surveillance image. Good. However, at this time, in order to highlight the abnormal place, it is necessary to identify the abnormal place on the image. To that end, for example, screen coordinate data may be added to the input detection target data, and the coordinate data may be used according to the abnormality determination result.

Next, proceeding to S704, the recognition device 20 returns to S701 if data to be detected remains in the recognition target data storage unit D2, and otherwise ends the operation at the time of detection.

In the above description, the formula (1) is shown as an example in the case of solving as a two-class problem, but the present embodiment can be applied to problems other than the two-class problem. For example, in a one-class problem (also called normal learning in the context of anomaly detection), the loss can be calculated by the loss function shown in the following equation (2) and used for learning.

In the equation (2), X is (minimum feature amount) Minibatch data including only normal data. As in the case of the above-mentioned formula (1), only the distances to the neighboring representative points may be obtained without comparing with all the representative points. The normal learning is particularly effective when only normal data is used at the time of learning, and can also be used when analyzing the behavior of a person imaged by a surveillance camera, or in the appearance inspection of a product in a factory or the like. In such a case, a feature extractor other than the feature extractor that requires a lot of processing capability such as CNN may be desired depending on the use case of application. In such a case, other known feature extractors or the like may be used instead of CNN. When the feature learning is difficult, only the representative point generator may be learned and the feature learning may not be performed.

Further, in the abnormal behavior detection for monitoring purposes, abnormal data that did not exist until now may be obtained by continuously collecting images. At that time, additional learning may be performed and used as teacher data. That is, initially, learning may be performed based on the equation (2), and additionally learning may be performed based on the equation (1). The additional learning may be performed each time new learning data is acquired, other than the problem setting described above. In that case, the entire CNN may be relearned, or only a part of the configuration (for example, the feature extractor or the representative point generator) may be relearned. For example, in a use case using surveillance cameras, performing feature learning for each surveillance camera requires a large calculation cost and learning cost. However, if the learning cost of the representative point generator is relatively low, the feature extractor can be commonly used by a plurality of cameras, and only the representative point generator can be learned for each camera. The feature extractor used here may be one already learned in another domain.

Although an example of solving as a one-class problem or a two-class problem has been described above, it is possible to solve as a problem with a larger number of classes. For example, the present invention can be applied to classification problems such as image recognition. Specifically, for example, a maximum of M types of exclusive labels can be given to M representative points, and a loss function as shown in the following Expression (3) can be used.

In Expression (3), the lower right term of ||•|| indicates the condition for calculating the distance, and C represents the class of the recognition target data. In addition, iεC _j indicates the condition “when the label of the i-th representative point corresponds to the j-th data class included in Minibatch”, and i¬εC _j indicates the opposite. |・| represents the number when enumerating the elements of the set, and has the role of normalizing the summed value by giving it as the denominator. For example, | _i ∈ C _j | is "the number when the label of the i-th keypoint corresponds to the j-th data class included in Minibatch", and is the number of times the condition is met. Equivalent to. It is necessary to determine the above correspondence relationship in advance. For example, when j={0, 1,..., 30}, C _j is the 0th label, and when j={31, 32,..., 60}, C _j is the 1st label. There is. That is, each representative feature amount generated based on X ₀ is labeled in advance, and learning is performed so as to be generated closer to the data of the recognition target class that is preliminarily linked to the label. Can be considered. Then, at the time of detection, the distance between the representative feature amount generated based on the recognition target data and the target feature amount is calculated, and the recognition target class corresponding to the label of the representative feature amount closest to the target feature amount is , Can be considered as the class of the target feature quantity. In addition, although the case where the label of the representative feature point is exclusive was described in the above-mentioned case, it is not exclusive and may be configured as a multi-label, for example.

In the above example, the case where the loss is calculated based on the representative point and the learning processing is performed, and the case where the detection processing is performed based on the representative point are shown, but other tasks are simultaneously solved as multi-task learning. Good. For example, when a value such as a suspiciousness degree is given to the recognition target as teacher data, the suspiciousness degree can be estimated (regression) in parallel with the generation of the representative points. For the learning of regression, for example, a squared error can be used. When the representative point is generated and the regression is performed by the CNN, for example, a CNN that performs the regression process based on the Convolution 2 layer of FIG. 6 may be separately prepared, and the end-to-end learning process may be performed. Then, the result may be used at the time of detection. Alternatively, processing such as regression may be performed based on the generated representative points. In the example of FIG. 6, the representative point of the (M, 512, 32, 32) size and the (1, 512, 32, 32) size of the Minibatch of the (N, 512, 32, 32) size of the Convolution 2 layer are selected. Concatenate the recognition target data. Then, the feature quantity of size (M+1, 512, 32, 32) by the concatenate processing is transformed into the feature quantity of size (1, 512·(M+1), 32, 32) and used for recognition processing. As a result, the recognition process can be performed in consideration of the data in the Minibatch. In a normal neural network, other data in the Minibatch does not participate in the feature extraction processing of certain data. On the other hand, by performing the process of moving the data in the Minibatch onto the feature dimension by this extraction process or the like, it becomes possible to perform the feature extraction in consideration of the data in the Minibatch. In multitask learning, learning of each task may be performed in any order, or learning may be performed simultaneously.

Further, the above-described expression of the size of the feature quantity (for example, (N, 512, 32, 32)) is merely an expression used to exemplify the size of the feature quantity, and how The amount may be retained. For example, the feature amount having the size of (N, 512, 32, 32) may be the size of the array of (1, 512·N, 32, 32) internally. In the above process, the feature amount transformation process may be repeatedly performed, and therefore efficient data representation is desired. Further, even if it is internally held as a feature amount having a size of (N, 512, 32, 32), for example, it is handled without being transformed like (1, 512·N, 32, 32). Therefore, for example, the convolution processing may be extended to the first dimension (dimension of the number of data). That is, as an example of the normal convolution processing, convolution filters are used for the second dimension (the dimension of the number of channels). In this case, in the example of (N, 512, 32, 32), 512 convolution filters are used, but the convolution processing extended to the first dimension is (N, 512, 32, 32). In the example, 512·N convolution filters are used. This is an expression equivalent to the case where the convolution processing is performed after the transformation processing such as (1,512·N,32,32), and thus the calculation cost can be reduced by the amount of the transformation processing.

In the above example, the representative point is mainly a point on the feature space, but it may be a point on the same space as the space of the input data. Specifically, when the input data is an image, representative points may be generated and used in the image space. In such a case, as in Reference Document 4, for example, a reconstructed image of the recognition target data is generated in the image space using the Autoencoder, and can be used in a form of comparison with the representative point. There are various ways to generate the representative point, but the representative point can be generated using a method similar to the method using CNN as an example. Here, an example in which the representative point is generated in the image space using the Autoencoder has been described, but the representative point may be generated in the middle layer of the Autoencoder. In that case, optimization may be performed with respect to the reconstruction error and the distance between the representative points in the form of the above-described multitask learning. In addition, for example, when the representative point is generated in the original space (for example, an image space), not only the data in the Minibatch is reconstructed in the image space but also the input learning data is used as the representative point. It can be used for learning the generator. For example, a loss function term for generating a representative point may be added so as to approach the input learning data, or other known methods may be applied to input the learning data and the representative point. You may design a loss function for the relationship If the input learning data is provided with teacher information such as abnormality and normality, the learning may be performed so that the distance between the abnormal learning data and the generated representative point becomes large. Specifically, for example, the above-mentioned object can be achieved by adding a term of a loss function such that the loss increases as the square error between the abnormal learning data and the generated representative point increases.

Reference 4: C. Zhou et al. Anomaly Detection with Robust Deep Autoencoders. KDD, 2017.

In addition, when the input data is image data and text data, for example, in the case of a multi-source configuration, representative points may be generated only in the space of some input data, or all input data may be generated. The representative points may be generated in the space. Of course, the representative point may be generated only in the space of a part of the input data, without using the multi-source configuration.

Further, in the present embodiment, an example in which the neighborhood search is mainly performed in the space of the input data and the neighborhood image or the like is selected has been described, but the neighborhood search may be performed in an arbitrary feature space. Still further, learning-based methods may be used to select neighborhoods. For example, three methods of feature extraction, clustering, and representative point generation can be learned by using a method of simultaneously performing feature learning and clustering learning. For example, 100 pieces of randomly selected data including recognition target data are first subjected to feature extraction/clustering as Minibatches. Then, 50 pieces of data assigned to the same cluster as the recognition target may be used as the recognition target neighborhood for feature extraction/representative point generation. Feature extraction/clustering/representative point generation can be performed by a common CNN, and learning processing can be performed efficiently. For example, the CNN can be commonly used up to the Convolution 2 layer in FIG. 6, and the CNN can be learned by dividing the subsequent processing into different systems by clustering and representative point generation. Note that the above-described example using clustering is merely an example, and other known techniques capable of selecting and searching for neighborhoods may be used.

Further, in the present embodiment, as a first example, an example of a configuration and an operation for performing a neighborhood search and selecting neighborhood data from a close order to create a Minibatch has been described. However, as another method, for example, a recognition target neighborhood Random selection may be performed by collecting data existing within a certain distance of. This makes it possible to configure different Minibatches for the same recognition target. Further, in that case, the selection range in the vicinity of the recognition target may be narrowed as the learning epoch or iteration progresses. As a result, it is possible to gradually obtain fine-grained feature expressions and representative point generators. Note that these neighborhood selections may be performed in the feature space as described above.

Further, when selecting the data in the vicinity of the recognition target, not only the proximity but also other criteria may be used. By defining this criterion, it is possible to learn representative point generators of various properties. For example, as the data in the vicinity of the recognition target, data having the same attribute as the recognition target may be selected. Specifically, for example, in a task such as person matching, by treating the data of the same person as the recognition target as the data in the vicinity of the recognition target, it can be expected to generate representative points that seem to be the same person. This attribute may be, for example, an attribute related to the object type.

In addition, for example, in behavior recognition using the monitoring video, the persons considered to be the same group captured in the monitoring video may be treated as the same attribute. In addition, learning can be expected to proceed faster by selecting more neighboring data in which recognition processing is likely to be erroneous. If the recognition process is likely to be erroneous, it may be, for example, data with a large number of misclassifications, data with a large distance from a desired representative point, or any definition may be used if necessary. In order to make more selection easier, it is possible to always include error-prone data within a certain neighborhood range in the Minibatch, or to make it probabilistically selectable. The error-prone data may be reviewed for each learning epoch, for each iteration, or may be selected based on feedback from the user using the terminal device 40 or the like.

Weighting learning may be introduced if the importance of each data is known as well as the fact that it is easy to make a mistake. Here, the weighted learning means, for example, in the loss function, when the distance between the representative point and the data is obtained, the size is changed based on the weight of each data. For example, when the degree of importance is in the range of 1 to 10, the loss may be calculated by performing a process of multiplying the degree of importance determined in the range by, for example, a value obtained by squaring the difference between the representative point and the data. .. Further, as an example in which other criteria are used when selecting the neighborhood data, the data captured under various illumination conditions or the data captured by different viewpoints/cameras may be included.

By doing so, it becomes possible to learn a representative point generator or the like that is robust against variations. At this time, for example, in the person image search, the same person as the recognition target may be selected as an image captured in various environments as the recognition target vicinity, and in that case, the robustness of the recognition process is further increased. Is expected to improve.

It should be noted that, if necessary, the Minibatch may be created using other different criteria instead of selecting the recognition target neighborhood data. In that case, it is possible to learn representative point generators having different properties, instead of representative point generation considering neighborhood.

The present invention can also be used for tasks related to one-to-one matching such as matching of person images. First, as an example, it is considered that two person images to be collated are given and whether or not these two images are images of the same person is confirmed by a recognition process. Therefore, for example, a CNN having a Siamese structure as shown in Reference 5 is used.

Reference 5: E. Ahmed et al. An Improved Deep Learning Architecture for Person Re-Identification. CVPR, 2015.

A specific example is shown in FIG. The configuration shown here is for learning. For simplification of description, elements common to FIG. 6 in FIG. 9 are given the same reference numerals as those in FIG. 6, and description thereof will be omitted. The Minibatch used here is composed of the recognition target data and the data in the vicinity of the recognition target given the recognition target data, as in the above-mentioned example. The recognition target data here is one person image among the matching targets. Here, the neighborhood data is an image in which a person to be recognized is captured and is similar to the recognition target data. One Minibatch configured in this way is given to each recognizer. In addition, in order to efficiently perform learning, exclusivity may be taken into consideration so that data existing in each Minibatch can be obtained without being covered. FIG. 9 includes a recognizer A 901 and a recognizer B 902, which represent a recognizer that processes each of the two human images given in the above example. The loss value can be obtained by processing the outputs of the two recognizers by the loss calculation processing 903. The output of each recognizer is obtained by extracting data in Minibatch for each recognizer as a feature amount and the generated representative point. That is, in the Siamese configuration, two extracted feature points of Minibatch data and two generated representative points are obtained. In the learning for performing the one-to-one matching, the loss calculation process in the case of using only the image of the positive example can use the loss function shown in the following Expression (4), for example.

In Expression (4), using only the images of the positive examples means that the pair of images used for learning is always the same person. Here, X ^A, the feature amount of Minibatch data X ^B each recognizer A, recognizer B (may be data in the image space), ^A A, A ^B Each recognizer A, the representative point of the recognizer B Is. In particular, X ₀ ^A and X ₀ ^B are recognition target data of the recognizer A and the recognizer B, respectively. a ₁ , a ₂ , a ₃ , and a ₄ are weights for the respective terms, and are to be determined in advance here. Expression (4) is a loss function for the purpose of One-class learning as in Expression (2). However, the difference is that the distance between the representative point and the feature amount of the Minibatch data is considered across different recognizers. Basically, the closer the respective distances are, the smaller the output value of the loss function is, and the balance thereof can be determined by a ₁ , a ₂ , a ₃ , and a _{4 in the} equation (4), and the purpose is for one-class learning. It is a formula. In addition, when it is desired to perform learning based on other objective functions instead of One-class learning, a function suitable for each objective may be used. For example, in the multi-class learning, the method described above in the present embodiment may be used to separate the representative points of different labels from the feature amount of the Minibatch data, and a loss function for bringing the same labels closer to each other may be used. .. Specifically, when two-class learning is performed on whether the same person is a positive example or a negative example, learning can be performed using a loss function shown in the following Expression (5).

In Expression (5), the expression on the upper right side matches Expression (4), and the lower expression on the right side shows the loss function in the case of a negative example (in the case of different image pairs). In the case of the negative example, some of the signs are inverted so that the target feature amount and the representative point obtained from different recognizers are learned so as to be further apart. Also, in this example, it was assumed that the data to be recognized and the other data in the data in the Minibatch are always the same person, but this is not the case. You can also do it. In that case, representative points with different labels may be generated for representative points representing different persons and representative points representing the same person. In the above, the case of learning is exemplified, but it is possible to perform the distance-based recognition processing even at the time of detection. Specifically, for example, the distance can be calculated in consideration of the representative points generated by the following formula (6).

It is possible to determine whether or not they are the same person based on the distance obtained by the equation (6) and a predetermined threshold value. It should be noted that it is not a value that comprehensively considers the target feature amount and the representative point as in Expression (6), but when the target feature amount and the representative point output by different recognizers are compared, May be adopted. For example, the distance between the target feature amount output by the recognizer A and the representative point output by the recognizer B is calculated, and the smallest distance value is used.

Note that the example of the case of performing one-to-one matching with respect to matching of person images has been described above, but when a plurality of recognition candidates are given and the most matched data is selected from a plurality of search candidates, Possible as a use case. For example, in monitoring in a spectator seat of a stadium, it is desired to identify a person who behaves differently from other persons. As a specific example, for example, in Reference 6, “Figure 3 Shooting Example in Spectator Seat of Sports Stadium: Among people watching the competition, only the people indicated by red circles are different places such as watches and smartphones. , And is classified as suspicious behavior in this study. The size of the face in this video is about 25 to 30 pixels in the vertical direction.” Thus, acting differently from other persons is considered to be important evidence for the observer in finding suspicious persons.

Reference 6: Kenji Kurosawa et al. Construction of evaluation video database for development of safety and security technology using video analysis. 23rd Image Sensing Symposium, 2017.

Therefore, as an example, it is considered to determine whether or not a certain person is acting differently from other persons on the screen. In such a case, it is possible to regard it as a problem setting in which the most matched data is selected from a plurality of search candidates, given a plurality of recognition target candidates. In the following, for simplification of description, a case will be described in which the recognition target candidate and the search candidate overlap, and the correspondence between the plurality of search candidates is checked. The case where there is no overlap and the case where there is only a partial overlap will be described later.

Here, FIG. 10 shows an example of the configuration in the case of performing matching with a plurality of search candidates using a plurality of recognition targets (during learning). In the example of FIG. 10, a part of the configuration has a function similar to the function of FIG. 6, and in that case, the same reference numerals as those in FIG. 6 are given and duplicate description is omitted. The transformation processing 1001 of FIG. 10 is the transformation processing already described in the present embodiment, and processing for bringing all Minibatch data to the second dimension (feature amount dimension) is performed to create a feature amount of size 1002. .. Here, N may be variable. In order to convert this feature amount into the feature amount of size 635, the normalized variable representative point generation processing 1003 is used.

An example of the normalized variable representative point generation processing 1003 is shown in the lower left of FIG. Here, it is assumed that there are N feature quantities of (1,512,32,32) size, and the representative point generation processing for the feature quantity of (1,512,32,32) size is repeated N times (N parallel processing is performed). You may go). As the representative point generation processing, the representative point generation processing described in this embodiment may be used. Then, the representative point generation processing repeated N times is added and input to the representative feature amount layer 606. At this time, since N times are added, it is assumed that the normalization process is performed by dividing by N before inputting to the representative feature amount layer 606. As a result, the arbitrary number N disappears in the representative feature amount layer 606, and the feature amounts of M representative points are obtained as the output of the representative point generator 1004. Finally, the loss is calculated using N target feature values (feature values of the Convolution 2 layer) and M representative points. At this time, if all the learning data are normal, 1 class learning can be performed, and the equation (2) can be used for the loss function. By performing learning using the normalized variable representative point generation processing 1003 and Expression (2), it is possible to learn an average representative point of N pieces of data. If an abnormal label is given, it is possible to learn by giving a loss function such that the loss decreases as the abnormal recognition target data moves away from the representative point.

Further, at the time of detection, a distance calculation process is introduced in place of the loss calculation process 615 in the configuration of FIG. 10, and the distance between the N pieces of recognition target data obtained at the time of detection and the M representative points generated from them. The abnormality determination may be performed based on the result of calculating. Normally, when there are N recognition targets, it is expected to increase the calculation time in the order of N squared in order to calculate the distance between them. According to the method of FIG. 10, it can be compressed in the calculation time of the order of N·M. When M is larger than N, for example, N pieces of recognition target data may be compared with each other one-to-one, or the representative points generated based on the respective recognition target data may be converted by the method described in the present embodiment. The distance may be calculated by comparing. As described above, when the recognition target and the search candidate do not overlap, or when there is only a partial overlap, only the distance between the search candidate and the representative point is calculated when calculating the distance. The calculation process may be shortened by performing the calculation.

As a further development, a plurality of search candidates may be gradually narrowed down, for example. In the above-described example of a plurality of recognition targets, the number N of recognition targets may be variable, and thus a method of gradually narrowing down search candidates may be used at the time of learning or detection, and other methods may be used. A process for narrowing down search candidates may be performed using a known method. When using an example of a plurality of recognition targets, it is possible to perform processing for narrowing down the search candidates, for example, by excluding search candidates that are close to the generated representative point. Then, based on the narrowed-down search candidates, the process of generating a representative point is performed again.
In order to perform more detailed recognition processing, not only the distance between the representative point and the data is calculated as in Expression (2), but also the distance between the data and the representative points is calculated, and It may be used for learning processing or may be used for detection.

In the above example, the representative point is generated based on the recognition target data, but this is an example of using CNN as a specific example, and the representative point is used by using another known method. Good. For example, the representative points already generated may be used for the recurrent and reused for the recognition processing.

Further, in the above example, the example of the recognition process using the image is mainly described, but the recognition process for other data may be performed. For example, image data, sensor data, reflected light spectrum data, material composition data, chemical data, and the like. For example, in order to discover a new type of compound, only normal data of the compound is learned, and when a new type of compound appears as a recognition target, a new type can be determined by distance-based detection.

Further, in the above example, an example in which learning processing is performed using actual data has been described, but data may be created using CG (computer graphics) and used for learning processing or the like. For example, CG data similar to the recognition target data is created, CG data in which variations that can occur due to changes in the light source, etc. are added to the CG data, and these CG data and the characteristic data extracted from the CG data are It is used as recognition target neighborhood data and recognition target data. This makes it possible to deal with various variations on the image.

In the above example, in order to cope with changes in the light source and the like (for example, to correctly perform recognition processing even when images of different light sources are given), examples in which the learning image itself and the selection method of the neighborhood are added Indicated. On the other hand, regarding the representative point, more robust recognition processing can be performed by devising the recognition processing corresponding to the change of the light source and the like. Here, a specific example in the visual inspection is shown below. For example, it is assumed that the Autoencoder shown in this embodiment is used. A convolutional autoencoder in which a convolution filter is introduced in the autoencoder or a model incorporating other recursive structure may be used.

First, the recognition target data of the product to be inspected is selected. For simplification of explanation, the number of recognition target data used here is one, but there may be a plurality of recognition target data.
Next, the data of the same product as the recognition target data selected here, which has a variation such as a light source different from the recognition target data, is selected. If the Autoencoder is simply applied, the Autoencoder is used for learning to reconstruct each data independently, but consider generating representative points based on the middle layer feature amount of the Autoencoder as described above. ..

A specific example at the time of learning is shown in FIG. In the example of FIG. 11, a part of the configuration has the same function as that of FIG. 6, and in that case, the same reference numerals as those in FIG. 6 are given and the description thereof is omitted.
It is assumed that the auxiliary data selector 1141 selects auxiliary data based on the recognition target data 1101. Here, the auxiliary data corresponds to the data described as the recognition target neighborhood data in the present embodiment. This time, "select the data of the same product as the selected recognition target data, which has a variation such as a light source different from the recognition target data". Since these are not necessarily in the vicinity of the recognition target on the feature space or the like, they are called auxiliary data here.

When the Minibatch size is N, N-1 pieces of auxiliary data (auxiliary data 1102 to 110N) are selected. Although it is possible to select the auxiliary data based on various criteria, here, data that is labeled with the same product label as the selected recognition target data and that was captured with a different exposure from the recognition target data is used. Select randomly". Although only the data captured by different exposures is used as the auxiliary data, it is possible to try to stabilize the learning process by mixing the data captured by the same exposure as the recognition target data as the auxiliary data. Further, the auxiliary data may be selected by a human.

The reconstruction function 1120 is an Autoencoder, and the reconstruction recognition target data 1111 and reconstruction auxiliary data 1112 to 111N are obtained by the reconstruction function 1120. Here, it is considered that the reconstruction error is minimized and learning of representative point generation is advanced as in the case of the ordinary Autoencoder. In the middle layer of the Autoencoder from the recognition target data 1101 to the reconstructed recognition target data 1111, the representative point generation processing 614 is performed using the middle layer feature amount of the Autoencoder to obtain the representative point 1131.

Here, the middle layer of the Autoencoder used in the representative point generation processing 614 may be any layer, and the feature amounts of a plurality of layers may be used. The feature amount extraction process 1121 is performed using the feature amount of the same layer as the Autoencoder intermediate layer used in the representative point generation process 614, and the feature amounts 1132 to 113N are obtained. Here, the feature amount extraction process 1121 is a process for extracting a feature amount of auxiliary data used for comparison with a representative point, and the intermediate layer feature amount of Autoencoder may be used as it is, or other conversion process may be performed. You can go. A loss calculation process 615 is performed using the feature amount and the representative point obtained here, and a loss value 607 is obtained. As the loss function used at this time, any one exemplified in the present embodiment may be used, and another known loss function may be applied. For example, Unsupervised learning can be performed by the equation (2). Note that there is a teacher in the sense that it is assumed that all learning data is normal.

At this time, since the auxiliary data is data with different exposures, the positional relationship between the representative point generated from the recognition target data and the feature amount of the auxiliary data may exist at a distant place due to the influence of the exposure. However, the learning based on the equation (2) advances the learning of the feature extractor and the representative point generator, and the positional relationship between the generated representative point and the feature amount of the auxiliary data becomes closer. Can be. This can be regarded as an effect of reducing the influence of exposure.

Furthermore, in addition to the loss function related to the representative point such as the formula (2), a reproduction error function (such as a square error between the input data and the reproduction data) unique to the Autoencoder is introduced, and the learning is efficiently performed by solving it as the above-mentioned multitask learning. You can proceed. Further, when the feature space of the intermediate layer with Autoencoder is used as it is as the feature space for generating the representative point, the generated representative point can be used to compose an image by a part of the function of Autoencoder. That is, an image represented by the generated representative point can be generated. This can be interpreted as generating a representative point in the Autoencoder intermediate layer and further generating a representative point in the image space based on the representative point. That is, it can be interpreted as stepwise representative point generation or hierarchical representative point generation processing.

If, for example, when the labeled representative points are generated in a plurality of layers and the result is that the classes to which the recognition target data belong are different in each layer, for example, the results of the plurality of layers are taken into consideration. You may decide which class you belong to. For example, the distances normalized by the average distances obtained in the respective layers may be added together, and the class having the smallest distance value may be set as the belonging class. Further, even when not labeled, the detection process such as abnormality detection may be performed using the maximum or minimum distance, for example. Further, different thresholds may be used in each layer, and detection processing such as abnormality detection may be independently performed based on the representative point generation result obtained in each layer.

Further, in the example of the Autoencoder described above, the representative points generated in the image space may be visualized by the terminal device 40, thereby facilitating the understanding of the detection result for the surveillance staff and the like. Further, since these are representative points, a loss function is further applied by Equation (2) or the like to promote learning, and abnormality detection is performed based on the distance between the representative point in the image space and the input data. It can also be used. Although the explanation is given here based on the assumption that all the learning data are normal, if there is a label such as abnormal data, it is obtained by learning using a loss function such as equation (1). The detection process may be performed based on the learning result. Note that the Autoencoder is an example of the configuration of this embodiment when reconstructing data, and other known methods may be used.

Although exposure has been described in the above example, other camera parameters, product-related data, and the like may be used as auxiliary data. For example, in the case of the camera parameter A, the camera parameter B, the camera parameter C,..., The product A, the product B, the product C,... It may be treated as information and any input process may be performed.

In the above-described example of this embodiment, the same representative point is generated when the same recognition target is input at the time of detection. For example, when the purpose is to monitor surveillance video, the definition of normal may change depending on the situation.Therefore, the change of the representative point generated depending on the situation can express the change of definition of normal. It is desirable to be able to. For example, the terminal device 40 is provided with the auxiliary information input device 1201, and external information such as the weather and the state of the installed store (for example, whether it is open or not) is automatically input or given by the user. The auxiliary information is information that gives the state of the situation. The representative point generator may be switched based on this, or the representative point generator using the auxiliary information may be used.

Hereinafter, an example of learning the representative point generator using the auxiliary information will be described with reference to FIG. Note that the Minibat during learning is configured for each piece of auxiliary information in order to express that the definition of normal changes based on the auxiliary information. For example, it is assumed that a Minibat that collects normal data when the auxiliary information is 0 and a Minibat that collects normal data when the auxiliary information is 1 are prepared. The method described in the present embodiment may be used to select the data in these Minibatches. In the example of FIG. 12, a part of the configuration has the same function as that of FIG. 6, and in that case, the same reference numerals as those in FIG. 6 are given and the description thereof is omitted.

The auxiliary information input device 1201 of FIG. 12 inputs the above-mentioned auxiliary information. The auxiliary information 1202 indicates the input auxiliary information and may be a discrete value, a continuous value, or the number of dimensions, but here, in order to simplify the explanation, it is assumed that one-dimensional {0, 1} values can be taken. To do. The given auxiliary information 1202 and the feature amount of the target feature amount layer 605 are converted by the representative point addition process with auxiliary information 1203 to obtain the representative feature amount of the representative feature amount layer 606. At this time, the representative feature amount is determined depending on the auxiliary information. These are the roles of the representative point generator 1204.

A specific example regarding the operation of the representative point with auxiliary information generation process 1203 is shown in the lower left frame of FIG. 12 as an example 1210 of situation change correspondence based on input of auxiliary information. The upper part of the frame in FIG. 12 is a diagram for explaining the operation when the auxiliary information is 0, and the lower part is a diagram for explaining the operation when the auxiliary information is 1. The Convolution 2 layer and the representative feature amount layer 606 are feature spaces of the same size, and a schematic diagram of representative points and the like in this feature space is shown at 1220. The black dot in the figure is the target feature amount 1213 and exists at the same position regardless of the value of the auxiliary information. The generated points of the representative point group 1211 and the representative point group 1212 vary depending on the value of the auxiliary information. This operation represents that the normality changes according to the change of the situation. At this time, the distance between each representative point group and the target feature amount may change in the above two cases, and thus the abnormality detection result may change. The loss at this time may be calculated by using the loss function exemplified in this embodiment or other known methods and used for learning.

The representative information added representative point generation processing 1203 may be configured as follows, for example. Specifically, auxiliary information can be concatenated to the feature amount of the target feature amount layer 605, and the feature amount can be input to the representative point generation processing 614 illustrated in FIG. 6 to create a representative point. Here, the auxiliary information to be concatenated is assumed to be obtained by expanding the one-hot vector to a map having the same size as the feature map to be concatenated and then concatenating. Specifically, if the auxiliary information has a scalar value of 1, the map in which the first dimension is filled with 0 and the second dimension is filled with 1 is concatenated. In this way, the representative points can be generated in consideration of the auxiliary information. Note that other known methods may be used to generate the representative points in consideration of the auxiliary information.

After performing these learnings, by generating representative points in consideration of auxiliary information at the time of detection, it can be used for abnormality detection or the like according to changes in the situation. By extending these methods to the multi-class learning described in the present embodiment, it is possible to apply them to a multi-class classification problem or the like corresponding to a change in situation.

In the learning process described in the present embodiment, the constraint of the optimization technique is not described, but a constraint or regularization may be added if necessary. For example, the representative point and the feature amount may be constrained to be generated on the hypersphere. By doing so, the learning process can be stabilized. Moreover, you may add restrictions and regularization using another well-known method. Further, for example, by adding the representative point similarity regularization term shown in the following Expression (7) to the loss function, it is possible to perform learning so that the representative points generated by the same representative point generator are not similar to each other.

なお、代表点同士が似すぎると、幅広い認識対象近傍を表現しづらくなる虞があるため、このような正則化項を追加してもよい。なお、式（７）のｂは任意のパラメータである。
また、上述の例では、代表点それぞれを点として扱って、認識対象データ等との距離を測ったが、距離ではなく、例えば代表点群とＭｉｎｉｂａｔｃｈ中のデータとの分布としての差異を測ってもよい。同様に、距離以外の公知の尺度を用いて、代表点等の差異を測ってもよい。

If the representative points are too similar to each other, it may be difficult to represent a wide recognition target neighborhood, and thus such a regularization term may be added. In addition, b of Formula (7) is an arbitrary parameter.
Further, in the above example, each representative point is treated as a point and the distance to the recognition target data or the like is measured. However, not the distance but the difference as the distribution between the representative point group and the data in Minibatch is measured. Good. Similarly, a difference such as a representative point may be measured using a known scale other than the distance.

The generation of representative points has been described above, but other points may be generated instead of points. For example, a hypersphere having a mass may be generated. A hypersphere having a mass is generated so as to generate the representative points described above, and it can be considered that the feature amount is filled in the hypersphere within a certain radius range. Here, the radius is to be determined in advance, and may be 1, for example. The distance between the supersphere having a mass (representative point) and the recognition target data may be defined by the shortest straight line that intersects the recognition target data and the mass supersphere.

Further, for visualization, the generated representative points and the recognition target data may be mapped in a low dimension and displayed as a map. Any known technique may be used as the method of mapping in a low dimension, and for example, the method described in Reference Document 6 may be used.

Reference 6: S. Wold et al. Principal component analysis. 1987.

Also, for example, if one of the representative points and recognition target data placed on the map is designated using a graphical interface, etc., the representative points and recognition target data existing in the vicinity are further displayed, and the graphical interface is used. It is possible to perform a neighborhood search using the above-mentioned potato-based formula. The terminal device 40 may include such an interface.
Besides, the optimum M may be selected by cross validation. Further, the representative point generator may be configured by, for example, linear regression or the like without using CNN. The example of using CNN is an example used when high nonlinearity is required, and the present invention is not limited to the case of using CNN.

As described above, in the first embodiment, by using the recognition target data, the representative points of the recognition target such as the recognition target person in the data and the state thereof are predicted (generated) in the feature space or the original data space. It has a representative point generator that performs detection processing based on comparison of recognition target data and representative points. Thus, according to the first embodiment, high-speed recognition processing and recognition processing with high recognition accuracy can be realized.

<Second Embodiment>
In the first embodiment, the recognition target data is used to predict (generate) a representative point of the recognition target such as a recognition target person or the like in the feature space or the original data space, and the recognition target data and the representative point are calculated. An example in which the detection processing is performed based on the comparison with In the first example of the configuration at that time, there is a feature extractor for extracting a feature amount of data and a representative point generator for generating a representative point, and an example of configuring them by a convolutional neural network (CNN) is described. (The example when CNN is not used is also described). In the second embodiment, a configuration and an operation different from those in the first embodiment will be described regarding discriminatively learning a representative point generator.

Most of the configuration shown in the second embodiment is the same as the configuration illustrated in the first embodiment, and some configurations and operations are different. The present embodiment differs from the first embodiment in the configuration and operation as described above in the configuration and operation for learning the representative point generator. The expressions used in this embodiment are the same as those used in the first embodiment.

In the present embodiment, as one configuration example, in addition to the configuration of the first embodiment, the representative point generator is learned by applying a part of an adversarial generation network such as a General Adversarial Network (GAN). Here is an example: GAN is a method disclosed in Reference Document 7, and has a generation model for generating fake data in the original data space and an identification model for identifying the authenticity of the generated fake data. By performing the processing, the advanced generator can be learned. In the present embodiment, using this discrimination model, learning for discriminating whether the generated representative point is recognition target data or is the generated representative point (hereinafter, authenticity is identified). And the representative point generator learns by tricking the discriminator. It should be noted that these are examples of the case where the accuracy of the representative point generator is improved by using the discriminator, and the present invention is not limited to the configuration and operation of the GAN as described above. It should be noted that the representative point used in the present embodiment is different from the normal GAN and is supposed to be compared not only in the original data space but also in the feature space.

Reference 7: I. J. Goodfellow et al. Generative Adversarial Networks. Arxiv, 2014.

For example, the following expression (8) can be obtained by applying the GAN objective function to representative point generation data.

In Expression (8), D is a discriminant function (discriminator) for discriminating authenticity. The first term represents the error between (the feature amount of) the data in Minibatch that matches Equation (2) and the generated representative point. The second term represents an error in which the discriminator erroneously discriminates (feature amount) of the data in the Minibatch as false data, and the third term indicates that the discriminator recognizes the representative point as false data. It represents the loss of whether or not it was correctly identified. c _{1 to} c ₃ are weights of the respective terms, and an arbitrary number may be set. At this time, the representative point A is a point generated by the generation model G that is the representative point generator, and is generated using the recognition target data X ₀ , so it can be written as A=G(X ₀ ). .. Here, A={A ₁ , A ₂ ,..., A _M }.

The generative model G may be configured by using the method described in the first embodiment, or may be configured by using another known method. At this time, considering performing adversarial learning, the objective function can be written as, for example, min _D max _G 1class GAN Loss by using Expression (8). Since the expression (8) is differentiable, when the CNN is used, the learning can be performed by using the error back-propagation method as in the case of the normal CNN learning. Further, Expression (8) is an example in which the error of the discriminator for identifying authenticity and Expression (2) are combined in the case of performing 1 class learning, and instead of Expression (2), as shown in the first embodiment. You may use the loss function like this.

The representative point generator learned in this way compares the representative point generated by the representative point generator capable of generating a representative point closer to actual data with the recognition target data at the time of detection. The detection processing will be carried out. As a result, in the second embodiment, the recognition accuracy can be improved.

<Third Embodiment>
In the first embodiment, an example in which a representative point is generated in each of the middle layer and the output layer (layer after reconstruction) by using a configuration for reconstructing input data such as Autoencoder Indicated. This is an example showing that representative points can be generated in a plurality of layers when feature extraction of a plurality of layers is performed using CNN or the like. In the third embodiment, regarding the learning of the representative point generator under the hierarchical loss, a configuration and an operation different from those in the first embodiment will be described. In particular, a case where a plurality of recognition targets exist and the input data are all recognition targets will be described.

Hereinafter, the third embodiment will be described with reference to the drawings. Most of the configuration of the third embodiment is the same as the configuration illustrated in the first embodiment, and a part of the configuration and operation are different. The configuration and operation of the third embodiment differ from those of the first embodiment in the configuration and operation for hierarchically learning the representative point generator as described above. The expressions in the third embodiment are the same as those in the first embodiment.

FIG. 13 is a schematic diagram (during learning) regarding representative point generation in a plurality of layers and hierarchical cluster loss. In FIG. 13, the same processing contents as those in FIG. 6 are given the same reference numerals as those in FIG. In FIG. 13, an example of the feature extractor and the representative point generator is shown on the left side, and an operation of hierarchical loss calculation on the space where the representative point is generated is shown on the right side. The representative point generators 1321 to 1323 acquire recognition target data from a specific layer of the feature extractor 620 based on the recognition target extraction processing 613, and perform processing of generating representative points. Then, the representative point generators 1321 to 1323 perform loss calculation processing 1301 to 1303 for comparing the generated representative point with the feature amount of the data in the Minibatch.

At this time, in the loss calculation processing 1301, a neighborhood search is performed between the representative point generated by the representative point generator 1321 and the feature amount of the data in the Minibatch, and each data is assigned to the nearest representative point, for example. This process can be interpreted as clustering. For the neighborhood search processing, the method described in the first embodiment can be used. Then, the allocation information obtained here is sent to the representative point generator 1322 and the loss calculation processing 1302 as a cluster 1351. Further, the loss value 1311 is obtained by the method described in the first embodiment.

Next, in the loss calculation processing 1302, clustering is performed by performing a neighborhood search between the representative point generated by the representative point generator 1322 and the feature amount of the data in the Minibatch. At this time, the representative point generator 1322 generates a representative point for each cluster using the information of the cluster obtained by the clustering performed in the loss calculation process 1301. This processing will be described later. Further, in the clustering performed here, hierarchical clustering for further dividing the cluster of the cluster 1351 is performed. That is, with respect to the data assigned to each cluster of the cluster 1351, representative points are generated and each clustered to divide the second cluster. By doing so, the cluster becomes finer in stages, and a more detailed expression can be obtained. In addition, since it is unnecessary to generate a large number of representative points at once by generating representative points in multiple stages, it may be possible to perform learning more efficiently. Then, the loss value 1312 is obtained by the loss calculation processing 1302. By performing such an operation with the representative point generator 1323 and the loss calculation processing 1303, it becomes possible to perform the loss calculation processing in multiple stages.

In the above example, the representative points are generated in all the layers of the feature extractor, but a configuration in which an arbitrary layer is selected to generate the representative points may be used. On the right side of FIG. 13, a schematic diagram in the feature space regarding the above-described operation is shown. The upper right diagram in FIG. 13 is a schematic diagram of clusters used in the loss calculation processing 1301, and the target feature amount 1331, representative points 1332, and clusters 1340 are in the feature space of the same size as the Convolution 1 layer 602. Here, the white circles in the figure represent the representative points, the black circles represent the target feature amount, and the broken line represents the schematic diagram of the cluster range. It should be noted that, as described in the first embodiment, the space where the representative points are generated does not have to be the same as the space of the original feature amount. In other words, this is because the feature quantity of the data in the Minibatch may be mapped to the space in which the representative point is generated.

Here, as an example, two representative points are generated, and a plurality of target feature amounts are assigned to the representative points by neighborhood search. This cluster is the first-stage cluster in the hierarchical clustering. Then, also in the third embodiment, the loss may be calculated using this cluster allocation, as in the first embodiment. For example, the loss can be calculated using the following equation (9).

In Expression (9), C′ _i represents the i-th cluster, and i corresponds to the representative point. jεC′ _i represents that the j-th data is assigned to the i-th cluster, and |C′ _i | represents the number of data assigned to the i-th cluster. Further, weighting may be performed on the loss in each layer.

The lower right diagram in FIG. 13 is a schematic diagram of clusters used in the loss calculation processing 1302, and it can be seen that the clusters in the lower layer are further divided.
Note that the plurality of loss values obtained as described above may be used for all learning as described in the first embodiment, or may be used for only part of learning.

Next, a method of generating a representative point for each cluster will be illustrated with reference to FIG. In FIG. 14, the same processing contents as those in FIGS. 6 and 10 are given the same reference numerals, and the duplicated description will be omitted. A cluster 1440 in FIG. 14 shows a clustering result obtained by performing clustering in the previous stage. The cluster 1440 is used for the processing of the division processing 1442 by the cluster information input 1441 and divides the data of the target feature amount layer 605 into each cluster. Assuming that the number of clusters is L, the data divided into L pieces is input to the target feature amount layer 1405.

The sizes 1401 to 140L indicate the sizes of the feature quantities in consideration of the number of divided data, and the data numbers N _{1 to NL} represent the number of data of each cluster. Since the number of data of clusters is variable, M representative points can be generated for each cluster of data using the normalized variable representative point generation processing 1003 exemplified in the first embodiment. An arbitrary number can be set for M, and it is preferable to set a number of 2 or more in order to further divide the cluster. The representative point generator 1410 described above generates ML representative points. Finally, these representative points can be concatenate-processed to have a size of (M·L, 512, 32, 32) and used for the loss calculation process.

In addition, although the above-mentioned example gives the configuration and operation at the time of learning, it can be used at the time of detection using the same representative point generation processing. In that case, as illustrated in the first embodiment, the recognition target data may be compared with the representative point, the distance or the like may be calculated, and used in the detection process. At that time, a representative point generation result in a plurality of layers may be used.

Further, although the above-described example is, so to speak, a division-and-conquer type clustering, agglomerative type clustering can also be performed. In that case, after dividing the cluster, it is sufficient to perform representative point generation and loss calculation while hierarchically integrating the clusters, and this can be achieved by performing a process similar to the above-mentioned divide-and-conquer type. , Details are omitted.

Further, the above example is an example in the case where a plurality of recognition target data exist. The above-described method can be used when the recognition target data is singular and the data in the vicinity of the recognition target is used for learning. When a representative point is generated using a single piece of recognition target data, and when performing division-and-conquer type clustering, only the cluster to which the recognition target data belongs is hierarchically clustered. At that time, the subsequent recognition processing can be omitted for the data in the vicinity of the recognition target assigned to a cluster other than the cluster to which the recognition target data belongs.

As described above, according to the third embodiment, the representative point generator learned hierarchically in multiple stages can perform the learning process with higher accuracy, and thus the recognition accuracy at the time of detection can be improved. ..

<Fourth Embodiment>
In the fourth embodiment, what kind of device can be introduced regarding representative point generation, feature extraction, and the like in consideration of a series such as time will be described. Most of the configuration shown in the fourth embodiment is the same as the configuration illustrated in the first embodiment, and part of the configuration and operation are different. The configuration and operation of the fourth embodiment are different from those of the first embodiment in that, as described above, the time series property is taken into consideration.

Although described in advance, any of the examples described in the first to third embodiments can be configured/operated with respect to time-series data. For example, in the second embodiment, it is possible to generate a video, a motion vector, etc., instead of a single image. Further, when an image of a certain person is set as the recognition target, it is possible to use images of the same person at different times as images near the recognition target. Described in the fourth embodiment is an example in which time series is further used.

In the first embodiment, an example in which a person image is used in action recognition or suspicious action detection has been described. As another example, not only the RGB image but also the optical flow image may be extracted using the method disclosed in Reference Document 8 and used for the recognition process.

Reference 8: K. Simonyan et al. Two-Stream Convolutional Networks for Action Recognition in Videos. NIPS, 2014.

In that case, a representative point may be independently generated in each stream and used for recognition processing. When used for abnormality detection or the like, for example, the following equation (10) may be used as the loss function.

In Expression (10), X ^T and X ^S represent the feature amount of the temporal stream (Temporal Stream) and the feature amount of the spatial stream (Spatial Stream) corresponding to the recognition target data, respectively. A ^T and A ^S respectively represent a representative point on the temporal stream and a representative point on the spatial stream corresponding to the recognition target data. d ¹ and d ² are weighting parameters and may be set arbitrarily. Equation (10) measures how close the representative point generated from the target feature amount of the spatial stream is to the target feature amount of the temporal stream, and the representative point generated from the target feature amount of the temporal stream is It is a formula to measure how close the target feature amount of the spatial stream is. Then, through normal learning based on Expression (10), a feature extractor/representative point generator for predicting one normal feature amount from one stream normal feature amount can be learned.

At the time of detection, the distances to these characteristic quantities and representative points are measured as in the case of Expression (10), and if the distances are large, it is considered that the normal relationship at the time of normal learning is broken, and it is abnormal. Can be determined. Note that although Expression (10) considers a plurality of combinations of the feature amount and the representative point, it may be modified to consider only the nearest point, as in the examples shown in the other embodiments. Further, although the expression (10) is an example in which N pieces of input data of the temporal stream and N pieces of the input data of the spatial stream are used, these data may be data extracted in a continuous section of the video. Further, it may be data extracted from a certain image randomly or by some standard.

When it is desired to perform matching between a certain series and a certain series such as the one-to-one matching in the first embodiment, a representative point is generated by considering a plurality of patterns of the sequence length hierarchically and in multiple stages. , May be used for matching. For example, in the third embodiment, an example in which representative points are hierarchically generated and a loss function is hierarchically calculated has been described, but in this case, the sequence length can be considered in multiple stages.

An example thereof will be described in FIG. The time direction 1501 indicates that the time advances toward the right. It is assumed that the CNN feature amount is extracted at a plurality of timings like the CNN feature amount 1502. A range 1503 shows a range in which a plurality of CNN feature quantities are collected, and an operation 1504 shows an operation of generating a representative point. That is, an example in which a representative point is generated using a plurality of CNN feature amounts as recognition target data is shown. In addition, the CNN feature quantities are summarized in the upper row, and representative points are generated as recognition target data. In this way, the representative points can be generated in multiple stages. FIG. 15 shows an example of one-to-one matching of series 1 and series 2 in the upper and lower rows, and, for example, the operation 1504 in the upper series 1 and the operation 1505 in the lower series 2 correspond to each other. In the fourth embodiment, it can be considered to compare the representative point and the recognition target data at each of the corresponding places. As the method of comparison, any method described in the other embodiments described above may be used. Further, since the sequence shown in FIG. 15 can be sustained in time, it is possible to newly perform one-to-one matching by moving the range in which the CNN feature amounts are collected to the right, for example. .. By using the result of learning in this way and the method shown in the other embodiment described above, it can be used for action recognition, similar action search, detection process of abnormal action, and the like.

As described above, according to the fourth embodiment, the representative point generator that has learned the sequence hierarchically and in multiple stages can improve the recognition performance for the sequence data.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Each of the above-described embodiments is merely an example of an embodiment for carrying out the present invention, and the technical scope of the present invention should not be limitedly interpreted by these. That is, the present invention can be implemented in various forms without departing from its technical idea or its main features.

10: learning device, 11: selection unit, 12: learning unit, 20: recognition device, 21: input unit, 22: recognition unit, 30: determination device, 31: abnormality determination unit 31, 40: terminal device, 41: display Department

Claims (14)

Acquisition means for acquiring recognition target data,
Mapping means for mapping the recognition target data to a feature space,
Generating means for generating a plurality of points in at least one of the feature space and the data space of the recognition target data;
Comparing means for comparing the recognition target data and the plurality of points in at least one of the feature space and the data space,
Output means for outputting a recognition processing result based on the result of the comparison;
An information processing device comprising: Holding means having a plurality of learning data,
Selecting means for selecting auxiliary data corresponding to recognition target data from the plurality of learning data;
Representative point generating means for generating a representative point from the learning data,
Calculating means for calculating a loss based on the representative point and the auxiliary data,
Learning means for performing a learning process based on the loss;
The information processing apparatus according to claim 1, further comprising: Holding means having a plurality of learning data,
Selecting means for selecting auxiliary data corresponding to recognition target data from the plurality of learning data;
First generation means for generating a representative point from the learning data,
Calculating means for calculating a loss based on the representative point and the auxiliary data,
Learning means for performing a learning process based on the loss;
Acquisition means for acquiring recognition target data,
Second generation means for generating a representative point of the recognition target data based on the result of the learning;
Comparing means for comparing the recognition target data and the representative points of the recognition target data,
Output means for outputting a recognition processing result based on the result of the comparison;
An information processing device comprising: The information processing apparatus according to claim 2 or 3, wherein the selection unit selects, as the auxiliary data, data existing in the vicinity of the recognition target data. The calculation means calculates a loss based on a distance between the representative point and the auxiliary data,
The information processing apparatus according to claim 2, wherein the learning unit performs a learning process of reducing the distance. The generating means generates a plurality of representative points from the learning data,
The calculation means determines whether normal based on the distance between the plurality of representative points and the auxiliary data,
The learning means performs a learning process so as to reduce the distance when it is determined to be normal, and performs a learning process so as to increase the distance when it is determined to be not normal. 5. The information processing device according to item 5. The generating means generates a plurality of representative points from the learning data,
The information processing apparatus according to claim 2, wherein the calculation unit calculates the loss based on a distribution of the plurality of representative points and the auxiliary data. The information processing apparatus according to claim 2, wherein the calculation unit calculates the loss based on a weight given to the representative point and the auxiliary data. The information processing apparatus according to claim 2, wherein the representative point generation unit generates a representative point based on a model of a hostile generation network. The information processing apparatus according to claim 2, wherein the representative point generation unit generates a representative point for each layer of a plurality of layers. The information processing apparatus according to claim 2, wherein the representative point generation unit generates a representative point from data having time series. An information processing method executed by an information processing device, comprising:
An acquisition process for acquiring recognition target data,
A mapping step of mapping the recognition target data to a feature space,
A generation step of generating a plurality of points in at least one of the feature space and the data space of the recognition target data;
A comparison step of comparing the recognition target data and the plurality of points in at least one of the feature space and the data space;
An output step of outputting a recognition processing result based on the result of the comparison,
An information processing method comprising: An information processing method executed by an information processing device, comprising:
A selection step of selecting auxiliary data corresponding to the recognition target data from the plurality of held learning data,
A first generation step of generating a representative point from the learning data,
A calculation step of calculating a loss based on the representative point and the auxiliary data,
A learning step of performing a learning process based on the loss;
An acquisition process for acquiring recognition target data,
A second generation step of generating a representative point of the recognition target data based on the result of the learning;
A comparing step of comparing the recognition target data and a representative point of the recognition target data,
An output step of outputting a recognition processing result based on the result of the comparison,
An information processing method comprising: A program for causing a computer to function as each unit of the information processing apparatus according to claim 1.

Images