JP4641450B2 - Unsteady image detection method, unsteady image detection device, and unsteady image detection program - Google Patents

Description

  The present invention relates to a technique for detecting an unsteady image from an input image.

  A technique is disclosed in which a sensor such as a camera is installed in a place where inspection or maintenance is necessary, and whether or not an abnormality has occurred is determined from sensor information such as an image.

For example, a technique is disclosed in which normal data is collected in advance from sensor data of a hydroelectric power plant, and abnormalities in sensor information are discovered by clustering normal data using a one-class SVM (for example, Non-Patent Document 1). reference.). In this technique, it is necessary to collect normal data in advance.
Onoda et al., “Prediction of Abnormal Vibration of Hydroelectric Power Station Bearing Based on One-Class SVM (1)”, The 18th Annual Conference of the Japan Society for Artificial Intelligence, 2004.

  When an unsteady state occurs in a daily image, new images are sequentially learned to detect this, and if a similar sequence appears repeatedly, this is automatically It is necessary to have a mechanism to add to the video.

  In the conventional abnormal data detection technique, since it is difficult to define the distribution of normal data, it has been dealt with by using unsupervised learning.

  However, it was necessary to collect data that was determined to be normal in advance.

  In addition, when similar data is added, there is no mechanism for re-learning while adding this to normal data.

  An object of the present invention is to provide a technique for detecting images having different characteristics from a plurality of images in a captured video.

In order to solve the above-mentioned problem, the invention described in claim 1 creates a cluster from a plurality of inputted images, and whether an image inputted after the image used for creating the cluster belongs to the cluster. image the input is a non-stationary image detection method in unsteady image detection apparatus for determining whether a non-stationary image based on an image input step of inputting an image, a feature quantity from the input image a feature extraction step of extracting, perform unsupervised learning based on the distribution of the extracted feature quantity, and the cluster configuration steps to create a cluster of image input order is the input of the threshold value or less, the input of the input image Position is determined whether or exceeds the threshold value, the input image of the threshold is greater than the input rank, whether the input image belongs to the cluster A similarity judgment step of cross, and outputs the feature quantity of the input image and outputs the effect when the input image of the threshold is greater than the input rank is determined to unsteady image, the threshold value is greater than the input rank When it is determined that the input image is not a non-stationary image, a determination result output step for outputting the feature amount of the input image, and when an input image having an input rank greater than the threshold is determined to be a non-stationary image Even when it is determined that the image is not a non-stationary image, the teacher is based on the distribution of the feature amount of the image when the cluster is created in the cluster configuration step and the feature amount distribution of the input image having the input order higher than the threshold value. A cluster updating step of performing absence learning and reconfiguring the cluster.

In the invention according to claim 2, the image input step inputs a continuously shot moving image, and the feature extraction step extracts a sequence based on a standard defined from the continuously shot moving image, The feature amount is extracted for each sequence.

According to a third aspect of the present invention, in the sequence extraction, sensor information that is temporally correlated with the continuously captured moving images is input, and a prescribed number of images are obtained from images corresponding to the sensor information. One sequence is a feature.

According to a fourth aspect of the present invention, in the sequence extraction, a feature amount is extracted for each frame of the continuously shot moving image, and a frame in which the value of the extracted feature amount exceeds a threshold is set as a start frame. The number of frames defined from the start frame is one sequence.

The invention described in claim 5 is characterized by an unsteady image detection program for causing a computer to execute each step described in any one of claims 1 to 4 .

According to the sixth aspect of the present invention, a cluster is created from a plurality of inputted images, and the inputted image is determined based on whether an image inputted after the image used for creating the cluster belongs to the cluster. A non-stationary image detection apparatus that determines whether a image is a stationary image, an image input unit that inputs an image, a feature vector extraction unit that extracts a feature amount from the input image, and a distribution of the extracted feature amount The unsupervised learning based on the above is performed to create a cluster of the input images whose input order is less than or equal to the threshold value, and when the input image having the input order greater than the threshold value is determined as a non-steady image by the determination result output unit Even if it is determined that the image is not a non-stationary image, the distribution of the feature amount of the image when the cluster is created and the feature amount of the input image having an input rank higher than the threshold value Therefore, unsupervised learning is performed to determine whether or not the input order of the images input to the image input unit exceeds the threshold by reconfiguring the cluster, and input an input order higher than the threshold For an image, a similarity determination unit that determines whether the input image belongs to the cluster, and when an input image having an input order higher than the threshold is determined to be a non-steady image, the fact is output and the input image The determination result of transmitting the feature amount of the input image to the cluster configuration unit when it is determined that an input image having an input order higher than the threshold is not a non-stationary image. And an output unit .

In the inventions according to claims 1 and 6 and the subordinate inventions, since the cluster forming unit creates a cluster of the input image based on the feature amount of the input image, the data determined to be normal in advance is collected. There is no need to keep it. In addition, since the cluster configuration unit reconstructs the cluster based on the feature amount of the image belonging to the cluster transmitted from the determination result output unit, when similar data is added, it is re-learned while adding this to normal data The mechanism to do is realized.

In addition, since the cluster configuration unit reconstructs the cluster based on the feature amount of the image that does not belong to the cluster transmitted from the determination result output unit, even if it is determined as an unsteady image in the initial stage of the cluster configuration, When an image similar to a non-stationary image appears frequently, an image similar to the non-stationary image can be determined as a stationary image.

Furthermore, in the invention described in claim 2 and the invention subordinate thereto, it is possible to extract a non-stationary image from continuously captured moving images.
When the input image is determined to be a non-steady image, a message to that effect is output. This indicates that the non-steady image is displayed on the user interface, and features different from those of other steady images are highlighted. , Displaying the time when this image was input, notifying as sound, and the like.

  The sensor information includes camera sensor information, infrared sensor information, microphone sensor information, vibration sensor information, pressure sensor information, and the like.

According to the invention described in claims 1 to 6, in the creation of the normal data distribution in the detection technology of abnormal data, not necessary to collect is determined in advance normal data. Further, when similar data is added, a mechanism for re-learning while adding this to normal data is realized.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 is a configuration diagram of an unsteady image detection apparatus. As shown in FIG. 1, the unsteady image detection apparatus includes an image input unit 11, a feature vector extraction unit 12, a similarity determination unit 13, a determination result output unit 14, a cluster configuration unit 15, and a cluster information storage unit 16. ing.

  The image input unit 11 inputs a plurality of images.

  The feature vector extraction unit 12 extracts a feature amount from the input image.

  The cluster configuration unit 15 configures a cluster of similar images from a plurality of images input based on the extracted feature amount.

  The similarity determination unit 13 determines whether or not a newly input image belongs to a cluster of similar images configured in the cluster configuration unit 15, and sends the result to the determination result output unit 14.

  The discrimination result output unit 14 outputs the fact that the discrimination result in the similarity discrimination unit 13 of the newly input image does not belong to the cluster of similar images. On the other hand, if the image belongs to a cluster of similar images, the image is input to the cluster configuration unit 15.

  When there is a new input, the cluster configuration unit 15 reconstructs a cluster from all the images input so far and the input similar images.

  Next, operation examples of the cluster configuration unit 15 and the similarity determination unit 13 will be described with reference to FIG.

  Here, a case will be described in which the input is a video that is constantly shot, and notification is given when a video that differs from a daily video is obtained.

  The feature vector extraction unit 12 extracts a feature amount for each video sequence. Here, it is assumed that up to the feature amount of the k-th sequence is input.

  The cluster configuration unit 15 learns feature value data input one after another to form a cluster, but the characteristics of the daily feature values are not defined in advance. For this reason, unsupervised learning based on the distribution of input data is performed. As a result, a similar data group is defined as one cluster, and the data belonging to this cluster among the input data can be interpreted as everyday data, and the data not belonging to this can be interpreted as non-stationary data.

  In FIG. 2, a cluster obtained by learning daily data from the k-th feature amount is defined as Ck. Since a certain amount of data is required to configure a cluster for the first time, the number of data n for the initial cluster configuration is set, and the data is stored in the cluster configuration unit 15 while k <n. Thereafter, the following sequential processing is performed.

  The similarity determination unit 13 outputs a determination that non-stationary data has been found when the k + 1st input data deviates from the daily data cluster Ck learned from the kth feature quantity. To do.

  Next, up to the kth is the initial stage of learning, but it is assumed that the Ith (I> k) sequence has been input. If a large amount of data having a similar feature amount to the data input to the (k + 1) th is input up to the Ith, a similar cluster constituted by the data up to the Ith is represented by the k + 1th data in the feature space. The feature quantity that extends to the existing area and is similar to the (k + 1) th is no longer determined as non-stationary data.

  Examples of unsupervised learning methods in the cluster configuration unit 15 include a k-means method, a self-organizing map, a One-Class SVM, and an EM algorithm. The k-means method is a method of updating the average value of all input data as the cluster center while adding the input data. The self-organizing map is a method of creating a prototype that represents a cluster, searching for a prototype closest to the input data, and changing the neighborhood to update it. One-Class SVM is a method that utilizes the property that isolated points in the input space are mapped to the vicinity of the origin of the feature space by the Gaussian kernel. The EM algorithm is a method of estimating a distribution parameter assuming a Gaussian distribution as a data distribution.

  Next, processing in the cluster information storage unit 16, the similarity determination unit 13, and the determination result output unit 14 will be described.

  As a result of learning in the cluster configuration unit 15, the type of classifier used for learning and the parameters of the classifier are stored in the cluster information storage unit 16.

  The similarity determination unit 13 converts the newly added feature vector using this parameter, determines whether it belongs to a similar class, and sends the feature vector to the determination result output unit 14.

  The discrimination result output unit 14 inputs the feature vector to the cluster configuration unit 15 in the case of the discrimination result of belonging.

  On the other hand, in the case of the determination result that it does not belong, an unsteady image detection notification is output. Also in this case, a feature vector can be input to the cluster configuration unit 15, a class different from the similar class can be learned by the cluster configuration unit 15, and stored in the cluster information storage unit 16. Thereby, even if it is determined to be non-stationary at the initial stage of learning, a new stationary class can be formed when it frequently appears thereafter.

  Next, a configuration example of the feature vector extraction unit 12 will be described with reference to FIG.

  The feature vector extraction unit 12 includes a sequence extraction unit 21 and a feature amount calculation unit 22, and the sequence extraction unit 21 includes a sensor information input unit 23.

When the input from the image input unit 11 is a still image, the sequence extraction unit 21 regards each still image as a sequence and sends it to the feature amount calculation unit 22 . If the input is a moving image, it is divided into sequences and sent to the feature quantity calculation unit 22 .

  In order to perform the operation of dividing into a sequence most easily, it is conceivable to use a predetermined number (one or more) of frames from the current frame. However, when this non-stationary image detection apparatus is applied to a long-shot video, if the sequence start and end frames are not constant, even if it is part of a sequence learned in the past, There is a possibility of being discriminated as a sequence.

  Therefore, it is conceivable to set a sequence extraction criterion in advance. Examples of extraction criteria include using sensor information from the sensor information input unit 23 and using a change in the feature amount calculated by the feature amount calculation unit 22.

  When using sensor information, the sensor information input unit 23 inputs sensor information other than an image to the feature amount calculation unit 22 as a trigger.

  Sensors used in the sensor information input unit 23 are, for example, a camera, an infrared sensor, a microphone, a vibration sensor, a pressure sensor, and the like, and a sensor that can detect information generated along with information to be extracted by the feature vector extraction unit 12 is effective. is there.

  For example, in the feature vector extraction unit 12, a microphone can be used when it is desired to cut out an operation sequence accompanied by sound.

  When using a change in the feature amount calculated by the feature amount calculation unit 22, a method may be considered in which a feature amount is obtained for each frame, and an area having a feature amount equal to or greater than a certain value is cut out as one sequence.

  For example, when the input is a monitoring video of a fixed camera, the feature amount calculation unit 22 creates a difference image from the background image acquired in advance, calculates the area of the change area as the feature amount, and calculates the value as a sequence extraction unit Send to 21.

  The sequence extraction unit 21 can set a predetermined number of frames as one sequence with a frame having a certain value or more as a start frame.

  In addition to this, the feature amount obtained by the feature amount calculation unit 22 includes a luminance value of a pixel, a color, an optical flow, and their temporal change.

  Next, a processing example in the unsteady image detection apparatus will be described with reference to FIG.

  Here, it is assumed that an image of tropical fish shown in (a) is input from the image input unit 11.

  The feature vector extraction unit 12 regards a video change as one sequence every two frames, extracts an optical flow for each block, extracts a feature vector, and inputs it to the cluster configuration unit 15.

  The cluster configuration unit 15 uses One-Class SVM using a second-order polynomial kernel as a discriminator. (B) is an example of an image considered as one class as a result of learning of the initial cluster, and includes an image having a similar composition and direction of motion.

  (C) is an example of an image that is far from the cluster. However, when a number of videos similar to (c) are learned in the cluster configuration unit 15, all of them can be configured as one class.

  In the above embodiment, the non-stationary image detection device is realized by, for example, a CPU included in a computer device constituting the non-stationary image detection device, and features vector extraction processing, similarity determination processing, determination result output processing, cluster configuration Processing and the like can be installed as an application program.

  Also, the extracted feature vectors, the processing results of the calculated clusters, the calculation results, and the information necessary for the calculation and calculation performed in the feature vector extraction process, the similarity determination process, the determination result output process, the cluster configuration process, etc. Such data may be written to and read from an internal memory or an external storage device.

  In addition, a recording medium in which a program code of software for realizing the function of the face detection apparatus is recorded is supplied to the system or apparatus, and the CPU (MPU) of the system or apparatus reads the program code stored in the storage medium. It is also possible to execute. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and examples of the storage medium storing the program code include CD-ROM, DVD-ROM, and CD-R. , CD-RW, MO, and HDD.

The block diagram of a non-stationary image detection apparatus. Explanatory drawing of the operation example of the cluster structure part 15 and the similarity determination part 13. FIG. The block diagram of the feature vector extraction part 12. FIG. Explanatory drawing of the process example in an unsteady image detection apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 Image input part 12 Feature vector extraction part 13 Similarity determination part 14 Determination result output part 15 Cluster structure part 16 Cluster information storage part 21 Sequence extraction part 22 Feature-value calculation part 23 Sensor information input part

Claims (6)

A non-stationary image that creates a cluster from a plurality of input images and determines whether the input image is a non-stationary image based on whether an image input after the image used for the cluster generation belongs to the cluster A non-stationary image detection method in a detection device,
An image input step of inputting an image,
A feature extraction step of extracting a feature quantity from the input image,
Perform unsupervised learning based on the distribution of the extracted feature quantity, and the cluster configuration steps to create the input rank of following the input image threshold clusters,
Determining whether the input order of the input image exceeds the threshold value, the input image of the threshold is greater than the input rank, and similarity determination step of the input image to determine whether belonging to the cluster,
When an input image with an input order higher than the threshold is determined to be a non-stationary image, a message to that effect is output and a feature amount of the input image is output, and an input image with an input order higher than the threshold is not a non-stationary image A determination result output step for outputting the feature amount of the input image when it is determined,
Even when an input image having an input order higher than the threshold is determined to be a non-stationary image or not to be a non-stationary image, the image of the image when the cluster is created in the cluster configuration step is determined. A non-stationary image detection method comprising: a cluster update step of performing unsupervised learning based on a feature amount and a feature amount distribution of an input image having an input order higher than the threshold value, and reconstructing the cluster. In the image input step , a continuously captured moving image is input,
The non-stationary image detection method according to claim 1 , wherein in the feature extraction step, a sequence is extracted based on a standard defined from the continuously captured moving images, and the feature amount is extracted for each sequence. . Claim wherein the sequence extraction, characterized in that said type continuous shooting moving images and temporal sensor information corresponding is taken, the image number 1 sequence as defined by the corresponding image to the sensor information 3. The non-stationary image detection method according to 2. In the sequence extraction, a feature amount is extracted for each frame of the continuously shot moving images, and a frame whose value of the extracted feature amount exceeds a threshold is set as a start frame, and the number of frames defined from the start frame is 1 3. The non-stationary image detection method according to claim 2 , wherein the sequence is a sequence. The non-stationary image detection program which makes a computer perform each step in any one of Claims 1-4 .
A non-stationary image that creates a cluster from a plurality of input images and determines whether the input image is a non-stationary image based on whether an image input after the image used for the cluster generation belongs to the cluster A detection device,
An image input unit for inputting an image;
A feature vector extraction unit that extracts a feature amount from the input image;
An unsupervised learning based on the extracted feature quantity distribution is performed to create a cluster of the input images whose input order is less than or equal to a threshold, and an input image having an input order greater than the threshold is unsteady by the determination result output unit Even when it is determined that the image is not a non-steady image, the distribution of the feature amount of the image when the cluster is created and the feature amount of the input image having an input order higher than the threshold value Based on the unsupervised learning and reconfiguring the cluster,
Similarity determination that determines whether or not the input order of the images input to the image input unit has exceeded the threshold, and determines whether or not the input image belongs to the cluster for input images that are higher than the threshold And
When an input image with an input rank greater than the threshold is determined to be a non-steady image, the fact is output, and the feature quantity of the input image is transmitted to the cluster configuration unit. A non-stationary image detection apparatus comprising: a determination result output unit configured to transmit a feature amount of the input image to the cluster configuration unit when it is determined that the image is not a non-stationary image.