JP2012146040A - Detecting system of abnormal situation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a detecting method of abnormal situations capable of calculating conspicuity as an index of a possibility of the abnormal situation without limiting to a case where occurrence of the abnormal situation is predicted previously, or capable of detecting time and place candidates having a high possibility of the occurrence of abnormal situation.SOLUTION: The detecting system of abnormal situations includes: an input unit in which space-time data having values corresponding to respective space-time points is inputted; a feature extracting unit that extracts a vector representing features of space-time data at predetermined multiple space-time points; a feature analyzing unit that calculates a statistic quantity of the features of the extracted vector; and a conspicuity calculating unit that calculates the conspicuity at the predetermined multiple space-time points based on the result of the feature extracting unit and the result of the feature analyzing unit.

Description

  The present invention relates to an abnormal situation detection system that narrows down abnormal situation candidates based on a model simulating a biological attention function in a system that detects abnormal situations based on sensor information such as a camera.

  In order to create a city where people can live safely and securely, it is desired to realize a monitoring system for preventing crimes and a monitoring system for children and the elderly. At present, surveillance cameras are mostly manually monitored, and are physically and mentally burdensome. Therefore, automation is strongly desired.

  As a technique for automating monitoring of a surveillance camera, a system has been proposed in which an image photographed by a surveillance camera is processed to automatically select an image on which a human should determine whether or not there is an abnormality (for example, patent document). 1). In addition, an apparatus and a method for detecting an abnormal operation from a moving image based on inter-frame difference data have been proposed (see, for example, Patent Document 2).

  Furthermore, as a model that mimics the attention function of a living organism, a method of calculating saliency from an image (for example, see Non-Patent Document 1), or a method of calculating saliency from an image by incorporating the effects of visual experience (for example, (See Patent Document 3 and Non-Patent Document 2).

JP 2008-278128 A JP 2006-79272 A JP 2007-287071 A

L. Itti, C.I. Koch, and E.M. Niebur, “AModel of Salientity-Based Visual Attention for Rapid Scene Analysis,” IEEE Trans. PAMI, Vol. 20, no. 11, pp. 1254-1259, 1998. Takashi Toriu and Shigeyoshi Nakajima, "AMethod of Calculating Image Saliency and Optimizing Efficient Distribution of Image Window", Proceedings of the first International Conference on Innovative Computing Information and Control (ICICIC-06), Beijing, Aug. , Vol. 1, pp. 290-293, 2006

  In the invention disclosed in Patent Document 1, an image taken by a surveillance camera is processed to determine which class it is defined in advance. Seeking judgment. This method classifies situations that can occur in advance into several classes and defines them in advance, but this is possible only if you can anticipate situations that can occur in advance, such as ATMs and parking lots. There is a problem that is.

  In the invention disclosed in Patent Document 2, it is determined that an abnormality occurs when the characteristics of the inter-frame difference data deviate from the characteristics of the normal state. Has not judged. Therefore, there is a problem that when an abnormality occurs only in a part of the image, the degree of the abnormality is relatively small and the possibility of overlooking is high.

  The visual system of living organisms including humans has a function called attention. Instead of processing all the information input to the eyes uniformly, attention is focused on a specific object, and that part is processed with priority. As a result, it is possible to quickly detect danger and take appropriate measures.

  If the surveillance system has a similar attention function by imitating the attention function of the biological visual system, it may be possible to appropriately extract candidates for abnormal situations. For this purpose, it is necessary to engineer the attention function of the visual system of living organisms. As an example of such an attempt, Patent Literature 3, Non-Patent Literature 1, and Non-Patent Literature 2 disclose a method for calculating the saliency of each part of an image. The saliency expresses the degree of conspicuousness, such as the color being different from the surroundings, and the portion with high saliency is likely to attract attention. If this model is used, a highly remarkable part can be detected as a candidate for an abnormal situation. However, since this method detects the saliency at that moment at each moment, there is a problem in that a candidate for an abnormal situation that cannot be determined for the first time after the passage of time can be detected.

  An object of the present invention is to provide an abnormal situation detection system that narrows down candidates for an abnormal situation based on a saliency calculation model simulating a biological attention function, and has a value corresponding to each space-time point. An input unit for inputting spatio-temporal data; a feature extraction unit for extracting a vector representing features of the spatio-temporal data at a predetermined number of spatio-temporal points; a feature analysis unit for calculating a statistic of the extracted vector features; By providing a saliency calculating unit that calculates saliency at a predetermined number of space-time points based on the result of the feature extracting unit and the result of the feature analyzing unit, the above problem is solved.

  The present invention may further include an abnormal situation candidate detection unit that detects space-time point candidates that are highly likely to have an abnormal situation based on the result of the saliency calculation section.

  According to the present invention, the saliency can be calculated as an index of the possibility of an abnormal situation without being limited to the case where a situation that can occur in advance such as an ATM or a parking lot can be predicted. Can detect high time and place candidates. Since the place where the abnormal situation has occurred can be specified, there is an effect that the possibility of overlooking is small even if the abnormal situation is limited to a narrow area. In addition, since the saliency is calculated based on the characteristics of a certain area in the time space rather than the saliency at each moment, there is an effect that it is possible to detect a candidate for an abnormal situation that can be determined only after the passage of time. .

Schematic configuration diagram of the detection system in the first embodiment Schematic configuration diagram of the detection system in the second embodiment The figure which shows an example of the algorithm of the abnormal condition candidate detection part in 2nd Example. Schematic configuration diagram of the detection system in the third embodiment Schematic configuration diagram of a detection system in the fourth embodiment Schematic configuration diagram of a detection system in the fifth embodiment Schematic configuration diagram of a detection system in the sixth embodiment

Example 1
The present invention is an abnormal situation detection system that narrows down abnormal situation candidates based on a model simulating a biological attention function. Here, the abnormal situation in this specification refers to “not normal operation”. Normal operation refers to an operation that can be learned from a statistical distribution in which the distribution is concentrated when a statistical distribution of operation characteristics is considered. An operation that deviates greatly from the distribution is defined as an abnormal operation. Hereinafter, an abnormal situation candidate detection system according to the present invention will be described with reference to the drawings. The structure of the first embodiment of the present invention is shown in FIG. The detection system of the present embodiment includes an input unit (1), a feature extraction unit (2), a feature analysis unit (3), and a saliency calculation unit (4).

  When an image captured by the camera is input to the input unit, the image is digitized and a time-series image I (x, y, t) is output. The time-series image I (x, y, t) is spatio-temporal data having a value corresponding to each space-time point (x, y, t). This time-series image I (x, y, t) is a digital image, and x, y, and t each have an integer value between 0 and W-1, H-1, and T-1. I (x, y, t) is R (x, y, t) representing the red intensity at each space-time point (x, y, t), and G (x, y, t) representing the green intensity. t), composed of pixel values of B (x, y, t) representing the intensity of blue, each having an integer value from 0 to 255.

  In the feature extraction unit, each of the N space-time points (x, y, t) satisfying Δt ≦ t ≦ T−Δt, Δx ≦ x ≦ X−Δx, Δy ≦ y ≦ Y−Δx, The pixel values R (x, y, t), G (x, x, y) of the pixels included in the space-time regions t−Δt ≦ t + Δt, x−Δx ≦ x + Δx, and y−Δy ≦ y + Δy centered on y, t) Based on y, t) and B (x, y, t), a K-dimensional feature v (x, y, t) representing the feature of the region is extracted. Various methods are known as to what features are extracted. The present invention is not particularly limited as to what kind of features are extracted. For example, a method that characterizes the direction of the outline of an object other than a color, or a method that characterizes the direction of movement of an object is considered. It is done. Here, a specific configuration of the present invention will be described by taking, as an example, characteristics that are commonly used.

  The example feature here relates to color. As described above, I (x, y, t) is R (x, y, t) representing the intensity of red at each space-time point (x, y, t), and G representing the intensity of green. (X, y, t) is composed of pixel values of B (x, y, t) representing the intensity of blue, each having an integer value from 0 to 255. In each of R (x, y, t), G (x, y, t), and B (x, y, t), an integer from 0 to 255 is set to 63 or less, from 64 to 127, from 128 to 191, It is divided into four stages of 192 or more. In this way, the color is divided into 4 × 4 × 4 = 64 sections. For example, in a certain section, 64 ≦ R ≦ 127, 0 ≦ G ≦ 63, and 128 ≦ B ≦ 191.

  Pixel value R (x, y, t) of each pixel included in a space-time region t-Δt ≦ t + Δt, x-Δx ≦ x + Δx, y-Δy ≦ y + Δy centering on the space-time point (x, y, t) , G (x, y, t) and B (x, y, t) are determined to which of the 64 types of partitions belong, and the number of pixels included in each partition is counted. A 64-dimensional vector v (x, y, t) is collected. The kth element of this vector represents the number of pixels included in the space-time areas t−Δt ≦ t + Δt, x−Δx ≦ x + Δx, and y−Δy ≦ y + Δy in the kth color section. In this case, the feature dimension is K = 64.

  The feature analysis unit calculates a statistic of the vector K-dimensional feature extracted by the feature extraction unit. Specifically, N extracted at each of N space-time points (x, y, t) satisfying Δt ≦ t ≦ T−Δt, Δx ≦ x ≦ X−Δx, and Δy ≦ y ≦ Y−Δx. The average <v> of the K-dimensional feature v (x, y, t) and the standard deviation vector σ are calculated according to the following equations (Equations 1 and 2).

  The saliency calculation unit calculates saliency at a predetermined number of space-time points (x, y, t) based on the vector K-dimensional feature extracted by the feature extraction unit and the calculation result calculated by the feature analysis unit. To do. Hereinafter, a calculation method by the saliency calculation unit will be specifically described. First, features at each of N space-time points (x, y, t) satisfying Δt ≦ t ≦ T−Δt, Δx ≦ x ≦ X−Δx, and Δy ≦ y ≦ Y−Δx calculated by the feature extraction unit. The difference between the average <v> of the vector K-dimensional feature v (x, y, t) representing N and the N K-dimensional features v (x, y, t) calculated by the feature analysis unit is Let it be w (x, y, t).

  The saliency S (x, y, t) is calculated by the following equation. Here, saliency is a measure that generally indicates the degree to which a person's eyes are more likely to pay attention when observing an image. For example, an index that increases when there is a different object in the image data. It is.

  This equation is a measure of how far the vector K-dimensional feature v (x, y, t) representing the feature at space-time point (x, y, t) is from their mean v (x, y, t) (ie Saliency). According to this scale, the saliency increases when the features are different from the surroundings.

  According to the present embodiment, the saliency is high where the features are different from the surroundings in terms of time and space, and this can be used as an index of occurrence of an abnormal situation. Accordingly, the saliency can be calculated as an index of the possibility of an abnormal situation without being limited to the case where a situation that can occur in advance such as a TM or a parking lot can be predicted. In addition, since the conspicuousness is high where the features are different from the surroundings in terms of time and space, it is possible to specify not only the time when the abnormal situation occurred but also the location. Therefore, even if the abnormal situation is limited to a narrow area, there is an effect that the possibility of overlooking is small. Furthermore, since the saliency is calculated not based on the saliency at each moment but based on the characteristics of a certain area of time space, there is an effect that it is possible to calculate an index of an abnormal situation that can be judged only after the passage of time. .

(Example 2)
The configuration of the second embodiment of the present invention is shown in FIG. This embodiment includes an input unit (1), a feature extraction unit (2), a feature analysis unit (3), a saliency calculation unit (4), and an abnormal situation candidate detection unit (5). The input unit (1), feature extraction unit (2), feature analysis unit (3), and saliency calculation unit (4) are the same as those in the first embodiment. The second embodiment has a configuration in which an abnormal situation candidate detection unit (5) is added to the first embodiment.

  In the abnormal situation candidate detecting unit, N space-time points (x, y, t) satisfying Δt ≦ t ≦ T−Δt, Δx ≦ x ≦ X−Δx, Δy ≦ y ≦ Y−Δx calculated by the saliency calculating unit. ) To detect a space-time region in which there is a possibility or high possibility that an abnormal situation has occurred. FIG. 3 shows an example of the algorithm.

  The area size setting unit determines the size of the area to be detected as a candidate. The initial value is, for example, one half of all space-time areas 0 ≦ t ≦ T, 0 ≦ x ≦ X, and 0 ≦ y ≦ Y. The abnormal area candidate extraction unit calculates the sum of the saliency in the area while shifting the position of the area whose size is set by the area size setting unit, and abnormally identifies the area where the saliency sum is maximum. Temporarily extracted as a situation area candidate. The verification unit determines whether the area is appropriate as an abnormal situation area candidate based on the sum of the saliency and the size of the area, and outputs the area as an abnormal area candidate when it is determined as appropriate. Otherwise, the same process is repeated with the area size set smaller. If an abnormal situation area candidate that is determined to be appropriate is not output even if it is repeated a predetermined number of times, the process is terminated because there is no abnormal situation candidate.

  In the second embodiment, an abnormal situation candidate detection unit is added to the first embodiment. However, since a space-time area in which an abnormal situation occurs can be specified, it is possible to connect to a process for analyzing the area in more detail. There is an effect of becoming. It is also possible to present the image of the specified space-time region to a human and make him / her judge.

(Example 3)
The configuration of the third embodiment of the present invention is shown in FIG. In this embodiment, a base vector storage unit (6) for storing a plurality of base vectors is added to the configuration of the second embodiment. Accordingly, the processing contents in the feature extraction unit (2) are different from those in the first and second embodiments. In the feature extraction unit (2) of this embodiment, the spatio-temporal data is stored at a predetermined number of space-time points. A vector representing a feature is calculated by calculating an inner product of a vector having a pixel value of a small area of the space-time including a space-time point as an element and a plurality of base vectors stored in the base vector storage unit. Details will be described below.

  In the feature extraction unit, each of the N space-time points (x, y, t) satisfying Δt ≦ t ≦ T−Δt, Δx ≦ x ≦ X−Δx, Δy ≦ y ≦ Y−Δx, The pixel values R (x, y, t), G (x, x, y) of the pixels included in the space-time regions t−Δt ≦ t + Δt, x−Δx ≦ x + Δx, and y−Δy ≦ y + Δy centered on y, t) Based on y, t) and B (x, y, t), a K-dimensional feature v (x, y, t) representing the feature of the region is extracted. This is the same as in the first and second embodiments. The present embodiment is different from the first and second embodiments in what features are extracted. Pixel values R (x, y, t), G (included in space-time regions t−Δt ≦ t + Δt, x−Δx ≦ x + Δx, y−Δy ≦ y + Δy centered on the space-time point (x, y, t) The sum of x, y, t) and B (x, y, t) is as follows.

A vector having these n elements is U (x, y, z). This vector is n-dimensional, the base vector of the same dimension suitable number, for example, the K preparation, Q _1, Q _{2, ...,} and Q _K. These vectors are assumed to satisfy the orthonormal relationship represented by the following equation.

The above equation is a Kronecker delta, which has a value of 1 when i and j are equal, and has a value of 0 otherwise. These vectors Q _1, Q _2,..., Q _K are stored in the basis vector storage unit. At this time, a K-dimensional feature v (x, y, t) having v ₁ (x, y, t), v ₂ (x, y, t), ..., v _K (x, y, t) as elements. Is obtained as follows.

Although the present embodiment is obtained adding a base vector storage unit (6) to the configuration of the second embodiment, whereby the base vectors Q _1, Q _{2, ...,} the features extracted if Torikaere the Q _K Can be changed. The features are fixed in the first and second embodiments, but if the features can be changed as in the present embodiment, the extracted features are changed to appropriate ones according to the characteristics of the scene. This has the effect of improving the ability to detect candidates for abnormal situations.

Example 4
The configuration of the fourth embodiment of the present invention is shown in FIG. In this embodiment, a learning unit (7) is added to the third embodiment. In the learning unit (7), a plurality of learning time-series images J ₁ (x, y, t), J ₂ (x, y, t), ..., J _L (x, y, t) prepared in advance. Based on t), a base vector to be stored in the base vector storage unit is calculated.

First, in the learning unit, Δt ≦ t ≦ T in each of the L time-series images J ₁ (x, y, t), J ₂ (x, y, t),..., J _L (x, y, t). P space-time points (x, y, t) that satisfy −Δt, Δx ≦ x ≦ X−Δx, and Δy ≦ y ≦ Y−Δx are randomly selected. Pixels of the respective pixels included in the spatio-temporal regions t-Δt ≦ t + Δt, x-Δx ≦ x + Δx, y-Δy ≦ y + Δy centering on each of the P space-time points (x, y, t) thus selected. the value _{R i (x, y, t} ), G i (x, y, t), B i (x, y, t) the pixel value vector s and the element _i (x, y, t) to extract. s _i (x, y, t) is a vector at the space-time point (x, y, t) of the i-th time-series image.

Next, principal vectors are subjected to principal component analysis for a large number of vectors s _i (x, y, t) to calculate basis vectors. The specific procedure is described below. First, an average <s> and a covariance matrix M are calculated according to the following equation. In the following equation, N _L is the number of vectors s _i (x, y, t). S _i (x, y, t) ^T represents a transposed vector of s _i (x, y, t).

Next, eigenvectors of the covariance matrix M are calculated, K are arranged from the largest eigenvalue, and Q _1, Q _2,..., Q _K and these vectors and these vectors are stored in the basis vector storage unit.

In this embodiment, a learning unit (7) is added to the third embodiment. With this, the first and second types of features to be extracted from the time-series images that are candidates for detection of abnormal situations are detected. In the third embodiment, the learning time series images J ₁ (x, y, t) and J ₂ (J ₂ (x, y, t) are not fixed in advance as in the third embodiment. x, y, t), ... , J L (x, y, basis vectors Q ₁ be given to _{t), Q 2, ...,} Q K is automatically determined.

According to the present embodiment, if a learning time series image J ₁ (x, y, t), J ₂ (x, y, t),..., J _L (x, y, t) is given, the basis vector Q _1, Q _2,..., Q _K are automatically determined, so that a time-series image of the same type as the time-series image that is a candidate for detecting an abnormal situation is used as a time-series image for learning. There is an effect that it is possible to automatically select base vectors Q _1, Q _2,..., Q _K that are optimal for time series images of the same type as the time series images that are not necessary.

(Example 5)
The configuration of the fifth embodiment of the present invention is shown in FIG. The present embodiment differs from the fourth embodiment in the contents of the feature analysis unit (3) and the saliency calculation unit (4).

  In the feature analysis unit of the present embodiment, each of N space-time points (x, y, t) satisfying Δt ≦ t ≦ T−Δt, Δx ≦ x ≦ X−Δx, and Δy ≦ y ≦ Y−Δx is extracted. The average <v> and covariance C of the N K-dimensional features v (x, y, t) are calculated according to the following equations (Equations 11 and 12).

Next, eigenvectors of the covariance matrix C are calculated, M are arranged in order from the largest eigenvalue, u ₁ , u ₂ ,..., U _M are stored.

  The saliency calculation unit includes an approximation unit and an error calculation unit.

In the approximation unit, each of the N space-time points (x, y, t) satisfying Δt ≦ t ≦ T−Δt, Δx ≦ x ≦ X−Δx, and Δy ≦ y ≦ Y−Δx calculated by the feature extraction unit. The difference between the average <v> of the N K-dimensional features v (x, y, t) representing the features in N and the N K-dimensional features v (x, y, t) calculated by the feature analysis unit And let it be w (x, y, t) as in the following equation.

Next, as shown in the following equation, w (x, y, t) is approximated by a linear combination of _M eigenvectors u ₁ , u ₂ ,.

  In the error calculation unit, an error when w (x, y, t) is approximated by the above-described equation 13 is calculated by the following equation and output as saliency S (x, y, t).

  In this embodiment, the principal component analysis is performed by the feature analysis unit on the feature extracted by the feature extraction unit, and the error when approximating the feature at each space-time point with the eigenvector obtained by the principal component analysis is output as saliency. ing. In the first to fourth embodiments, the difference from the average of the features is merely noticeable, and even when various features are slightly different from the surroundings, the feature is high when the sum is large. there were. In contrast to this, according to the present embodiment, there is a property that the saliency becomes high when a small number of features are greatly changed. Humans have the property that they can easily focus on certain features such as the direction of the line when they are different from the surroundings, and this embodiment has a more similar property to the function of attention in the human visual system. In addition, there is an effect that it can be expected to be closer to human judgment in determining whether there is an abnormal situation.

(Example 6)
The configuration of the sixth embodiment of the present invention is shown in FIG. In this embodiment, the feature analysis section (3) and the saliency calculation section (4) are realized by a method different from that of the fifth embodiment.

  In the feature analysis unit, as in the fifth embodiment, N space-time points (x, y, t) that satisfy Δt ≦ t ≦ T−Δt, Δx ≦ x ≦ X−Δx, and Δy ≦ y ≦ Y−Δx. The average <v> and the covariance C of the N K-dimensional features v (x, y, t) extracted in each of the above are calculated according to the above equations (11) and (12). In the present embodiment, this is the only process of the feature analysis unit.

  In the saliency calculating unit, first, N space-time points (x, y, x) satisfying Δt ≦ t ≦ T−Δt, Δx ≦ x ≦ X−Δx, and Δy ≦ y ≦ Y−Δx calculated by the feature extracting unit. Assuming that N vector K-dimensional features v (x, y, t) representing features in each of t) follow a multidimensional normal distribution, the mean <v> and covariance C calculated by the feature analysis unit are The occurrence probability of the feature v (x, y, t) is calculated using the following equation.

  Next, the amount of information obtained when observing this feature is calculated by the following equation and output as saliency.

  In this embodiment, since the physical meaning that the saliency is the amount of information obtained by observing the feature is determined, there is an effect that an appropriate determination can be made based on this result.

DESCRIPTION OF SYMBOLS 1 Input part 2 Feature extraction part 3 Feature analysis part 4 Saliency calculation part 5 Abnormal situation candidate detection part 6 Base vector storage part 7 Learning part

Claims (6)

An input unit for inputting spatio-temporal data having a value corresponding to each spatio-temporal point, a feature extracting unit for extracting a vector representing features of spatio-temporal data at a predetermined number of spatio-temporal points, and a feature of the extracted vector An abnormal situation detection system comprising a feature analysis unit that calculates statistics, and a saliency calculation unit that calculates saliency at a predetermined number of space-time points based on the results of the feature extraction unit and the feature analysis unit The abnormal situation detection system according to claim 1, further comprising an abnormal situation candidate detection unit that detects a candidate of a space-time point that is highly likely to have an abnormal situation based on a result of the saliency calculation section. Furthermore, a basis vector storage unit for storing a plurality of basis vectors is provided, wherein the feature extraction unit is a vector having, as elements, pixel values of a space-time small region including space-time points of space-time data at a predetermined number of space-time points. The abnormal situation detection system according to claim 1 or 2, wherein a vector representing a feature is calculated by calculating an inner product with a plurality of base vectors stored in the base vector storage unit. Further, a vector having a pixel value of a small space-time region including the space-time point as an element is extracted from a large number of space-time points of a large number of space-time data, and stored in the base vector storage unit based on the vector. The abnormal situation detection system according to claim 3, further comprising a learning unit that determines a basis vector. The feature analysis unit performs principal component analysis on a large number of features extracted by the feature extraction unit to calculate a plurality of eigenvectors, and the saliency calculation unit extracts a spatiotemporal space extracted at a predetermined number of space-time points. The error of the abnormal situation according to any one of claims 1 to 4, wherein an error is calculated when a vector having a pixel value of a small space-time region including a space-time point of data as an element is approximated by a linear combination of the plurality of eigenvectors. Detection system The feature analysis unit calculates an average and a covariance of a number of features extracted by the feature extraction unit, and the saliency calculation unit calculates a predetermined number of spacetimes based on the average and the covariance calculated by the feature analysis unit. 5. The probability of occurrence of a vector whose element is a pixel value of a spatio-temporal small region including spatiotemporal data extracted at a point is calculated, and saliency is calculated based on the evaluated occurrence probability. The abnormal situation detection system described in Crab.

Images