JP4654347B2 - Abnormal operation monitoring device - Google Patents

Description

  The present invention relates to an abnormal operation monitoring apparatus using a three-dimensional higher-order local autocorrelation (CHLAC) technique.

  Currently, many surveillance systems using cameras have been introduced in parking lots and stores. However, in the conventional monitoring system, it is necessary to constantly monitor the camera video, and when a problem occurs, a human being visually finds the corresponding scene from the video that has been recorded and collected in advance. For this reason, in the conventional system, the labor required for monitoring is increased, and human errors such as being overlooked tend to occur. In addition, it is difficult to respond quickly when a problem actually occurs, and it has been desired to save labor by a computer. As a technique for solving this problem, Patent Publication 2006-79272 (Patent Document 1) has been proposed.

The method proposed in Patent Document 1 is roughly divided into two. On the other hand, a CHALC feature disclosed in Patent Document 2 is extracted from frame data in which a normal operation having no problem is recorded, and a space spanned by eigenvectors obtained by principal component analysis (hereinafter, space spanned by eigenvectors). This is an offline learning type abnormality monitoring method that calculates an abnormal value as a degree of deviation from the normal operation partial space at the time of monitoring. The other is an on-line learning type abnormality monitoring method that updates a partial space while monitoring. For updating the subspace, approximation of principal component analysis as described in Non-Patent Document 1 can be used. The online learning-type abnormality monitoring method can automatically follow changes in the monitoring environment such as the day's sunshine conditions, thereby further saving labor.
Patent Publication 2006-79272 Patent Publication 2005-92346 Juyang Weng, Yilu Zhang and Wey-Shiuan Hwang, "Candid Covariance-Free Incremental Principal Component Analysis", IEEE Transactions on Pattern Analysis and MachineIntelligence, Vol.25, No.8, pp.1034-1040, 2003 Noriyuki Otsu, “Automatic threshold selection method based on discriminant and least-squares criteria” IEICE Transactions D, J63-D-4, P349-356, 1980.

  By solving online learning type abnormality monitoring using the method described in Non-Patent Document 1 (hereinafter referred to as CCIPCA) described in Patent Document 1, various problems of offline learning type abnormality monitoring can be solved. However, since CCIPCA seeks an approximate solution without directly solving the eigenvalue problem, there is a new problem that the calculation accuracy is lowered.

  Furthermore, in online learning-type abnormality monitoring using CCIPCA, the speed of environmental changes (seasonal changes, changes in daily sunshine conditions, sudden weather changes such as sudden torrential rain, etc.) that you want to follow online are explicitly indicated. It was difficult to specify. For example, when it is desired to cope with a sudden change in weather, it is necessary to immediately respond to environmental changes and learn a new normal operation, but as a reaction, it becomes easy to forget past normal operations. Conversely, when it is desired to cope with a gradual seasonal change, it is necessary not to forget the past normal operation for a long period of time, but as a reaction, it becomes impossible to cope with a rapid environmental change. As described above, it is necessary to explicitly specify the learning condition according to the speed of the environmental fluctuation to be followed. However, since CCIPCA assumes that the sample distribution is in a steady state, this is difficult.

  It is also conceivable to solve the exact solution without using CCIPCA during online learning. However, the update of the subspace of the normal operation by the exact solution takes a calculation time, and in the calculation by the sequential processing of a single central processing unit, the monitoring operation may be interrupted during the update, There was also a problem that impaired real-time performance.

The abnormal operation monitoring device of the present invention includes means for storing moving image frame data input in real time in a memory,
Means for generating inter-frame difference data from the moving image frame data;
Means for extracting feature data from the inter-frame difference data by three-dimensional higher order local autocorrelation;
Means for generating an update matrix using the feature data;
Means for adding the update matrix and updating the autocorrelation matrix on the shared memory;
Means for reading eigenvectors used for anomaly detection from shared memory;
Means for obtaining an abnormal value from the characteristic data and the eigenvector read from the shared memory, determining whether the abnormal value is larger or smaller than a threshold,
Means for outputting a determination result when the abnormality determination means determines an abnormality;
A first central processing unit for executing
Means for generating eigenvectors from the autocorrelation matrix on the shared memory by principal component analysis;
Means for storing the generated eigenvector in the shared memory and performing an update process;
A second central processing unit for executing
Is provided.

  The update matrix update unit generates the update matrix using only the feature data determined to be abnormal by the abnormality determination unit.

  The autocorrelation matrix update means calculates a weighted average of the update matrix to be added and the autocorrelation matrix, and the weight value of the update matrix to be added does not fall below a certain value.

  The autocorrelation matrix update means updates the learning data using the eigenvector.

  The first central processing unit does not read from the shared memory when the eigenvector on the shared memory is not updated by the second central processing unit, and uses the old eigenvector that has already been read. It is characterized by performing a calculation.

  The feature data extracting means includes means for calculating frame CHLAC data, adding the latest frame CHLAC data to the current feature data, and subtracting the frame CHLAC data older than a predetermined period.

  The abnormal value calculation is characterized in that a vertical distance or an angle between the feature data and a partial space of a normal motion spanned by the eigenvector is obtained.

  The first central processing unit and the second central processing unit are integrally configured.

  In the present invention, existing data and newly added data are explicitly weighted in an autocorrelation matrix used when principal component analysis is performed in an online learning type. When the weight for newly added data is increased, the importance of the newly learned operation is increased, and automatic tracking can be performed even for a sudden environmental change. Also, if the weighting for newly added data is reduced, the importance of existing data increases, and the follow-up to environmental changes becomes gradual.

  The present invention also proposes a method for updating a subspace of normal operation using only CHLAC features having high abnormal values. Since the CHLAC feature for the operation sufficiently learned by this processing is excluded from the learning target, the subspace can be updated efficiently. Furthermore, by originally performing weighting based on the probability of occurrence, weighting proposed in the present invention can be quickly followed by environmental fluctuations.

  In recent years, PCs have become multiprocessors, and there are many cases where a plurality of central processing units (hereinafter, the central processing units are referred to as CPUs) are mounted on one PC. Therefore, in the present invention, in the update of the subspace of normal operation, an exact solution is used instead of using an approximate expression. In addition, we propose a mechanism that enables real-time abnormality monitoring without interrupting the monitoring operation even if it takes a long time to calculate the exact solution by performing parallel and distributed processing using a plurality of CPUs. Thereby, the deterioration of the monitoring accuracy due to the approximate expression during online learning can be solved, and monitoring can be performed with the same accuracy as during offline learning.

  FIG. 1 is a comparison diagram between the conventional method and the present invention in which a method for periodically updating a partial space for normal operation is performed. Since the update of the subspace for normal operation requires a minimum of nearly 100 milliseconds even for a high-spec PC, in the case of a frame interval such as a TV image, at least 3 frames to 5 as shown in the left figure of FIG. Frame loss is caused by the conventional method. In the present invention, the update of the partial space for normal operation is performed by a CPU different from the CPU that determines whether it is abnormal or normal, so that no frame is lost as shown in the right diagram of FIG. While the update process of the partial space for normal operation is being performed by another CPU, abnormality monitoring is performed using the partial space for normal operation that has been used until immediately before the update process starts. For this reason, it is possible to update the partial space of the normal operation using the principal component analysis without interrupting the monitoring operation without using a high-spec PC.

  In the present invention, the calculation of the subspace of the normal operation is executed on a CPU different from the CPU that performs abnormality monitoring. Further, while the sub-space of normal operation is being calculated by another CPU, the abnormal value is calculated using the sub-space of normal operation that has already been calculated using the autocorrelation matrix before update.

FIG. 2 is an example of a system configuration diagram of the abnormality monitoring system according to the present invention.
The camera 10 outputs moving image frame data of a target person or device in real time. The camera 10 may be monochrome or a color camera. Further, the connection method between the camera and the PC 11 may be any connection method such as USB, IEEE 1394, network connection or the like. PC11 is a normal PC or the like that is commercially available. The latest PC is equipped with a multi-core CPU, and a plurality of CPUs are mounted on one LSI like the CPUs 12 and 13.

  FIG. 3 shows an outline of the operation of the CPU 12 and the CPU 13. The CPU 12 reads the camera video as process 1. Next, as a process 2, a CHLAC feature is calculated. Further, the autocorrelation matrix is updated in the process 3, and the abnormality is determined in the process 4 as normal. In the CPU 13, a partial space for normal operation necessary for determining whether the process is abnormal or normal is generated from the autocorrelation matrix obtained by the CPU 12. Until the CPU 13 completes the calculation of the subspace for normal operation, the CPU 12 performs abnormality monitoring using the subspace for normal operation calculated in the past.

  The shared memory 14 indicates a memory mounted on the PC. When data is transferred between the CPU 12 and the CPU 13, the shared memory 14 is used. The keyboard 15 and mouse 16 are used to control the abnormality monitoring system according to the present invention. The monitor 17 is used to display output information of the abnormality monitoring system according to the present invention. This is realized by installing and starting the created program on the PC 11.

  FIG. 4 describes details of the processing of FIG. In step 1-0, the CPU 12 waits until the input of frame data from the camera connected to the PC is completed. Here, the frame data is an image for one frame of a moving image. Further, the frame data is not directly captured from the camera, but it is also possible to record and read the camera video in advance.

  In step 1-1, the frame data is read into the memory. The frame data at this time is, for example, 256 gray scale data or RGB color data.

In step 1-2, information focusing on the movement is extracted from the frame data and used as inter-frame difference data. Information extraction is performed by generating a difference image from two consecutive frame data acquired in 1-1. As an example of inter-frame difference data, a differential binary image that binarizes pixel values after inter-frame differences is raised. A specific method for extracting the difference binary image will be described.
As a method of taking the difference, an inter-frame difference method that extracts a change in luminance of a pixel at the same position between adjacent frames is adopted, but an edge difference that extracts a luminance change portion in a frame or both are adopted. May be. If each pixel has RGB color data, the distance between the two RGB color vectors may be calculated as difference data between the two pixels.

Further, in order to remove extraneous information such as color information and noise unrelated to “movement”, the difference image is binarized to obtain a difference binarized image. As a binarization method, for example, a fixed threshold value, a discrimination least square automatic threshold value method disclosed in Non-Patent Document 2, a threshold value 0 and a noise processing method (with a difference other than 0 in a grayscale image, all having motion = 1) It is possible to employ a method for removing noise by a known noise removing method. By the above preprocessing, the input moving image data becomes a column of frame data (binary image) having logical values of “moved (1)” and “not moved (0)” in the pixel values.
In the present embodiment, the difference binary image is used as the inter-frame difference data. However, a gray scale difference image obtained by performing gray scale (taking the absolute value of the difference value) instead of binarization may be used. Absent.

  In step 2-1, a correlation pattern related to stereoscopic pixel data for each frame is calculated to calculate a CHLAC feature. There are 251 types of correlation patterns related to the stereoscopic pixel data of the CHLAC binary image in consideration of the correlation up to the second order. In the present embodiment, accordingly, the CHLAC data is a 251-dimensional feature amount. A specific calculation method of the CHLAC data is performed using the processing of Patent Document 2 shown in the conventional invention.

  When performing abnormality monitoring in Step 2-2, the average of CHLAC data for N frames is used as the feature amount, not the CHLAC data calculated in Step 2-1. This is called a CHLAC feature. N is preferably equal to or longer than one cycle of the operation to be recognized. By this processing, the CHLAC feature (vector) is smoothed to convert the noise into a more robust value.

As shown in FIG. 5, a time window is set in a column in which CHLAC data is arranged in time series as shown in FIG. 5, and the sum of the CHLAC data is obtained while shifting the window one data at a time. A sum of the obtained CHLAC data divided by N is called a CHLAC feature.
As an efficient calculation method, there is a method of subtracting CHLAC data removed by time window shift from the sum of CHLAC data and adding CHLAC data added by time window shift to the sum of CHLAC data. 2.

  In step 4-3, a subspace of normal operation is used to determine whether it is abnormal or normal. this is,

  It is determined by the eigenvalue Λ and eigenvector U obtained by solving the eigenvalue problem of Equation 1 (principal component analysis). In step 3-1, calculation for obtaining R in an equation called an autocorrelation matrix is performed. Here, the approximation of Non-Patent Document 1 is not performed, and the eigenvector is obtained directly from the autocorrelation matrix R by Equation 1, so that the calculation accuracy does not deteriorate. The calculation for obtaining the autocorrelation matrix R is roughly divided into two stages. The first stage generates a matrix r (update matrix) for updating the autocorrelation matrix R while sampling the CHLAC features for a certain period. In the second stage, the autocorrelation matrix R is updated using the update matrix r obtained in the first stage.

In step 3-1, the first step is executed to update r. Usually, the update matrix r from the CHLAC feature is

の式にて計算される。ここでnは更新行列rを構成するＣＨＬＡＣ特徴のサンプル数を指し、xiはＣＨＬＡＣ特徴を指す。L次元のＣＨＬＡＣ特徴を以下のように表す。

It is calculated by the following formula. Here, n indicates the number of CHLAC feature samples constituting the update matrix r, and xi indicates the CHLAC feature. The L-dimensional CHLAC features are expressed as follows:

なお、本実施例ではL＝２５１であるため、更新行列rは251×251の正方行列となる。式２をそのまま解くと計算量がサンプル数に比例して増大していく。

Note that since L = 251 in the present embodiment, the update matrix r is a 251 × 251 square matrix. If Equation 2 is solved as it is, the amount of calculation increases in proportion to the number of samples.

  Therefore, in the present invention, a calculation method is used in which the update matrix r is obtained with a constant calculation amount regardless of the number of samples. Specifically, for the update matrix r composed of n CHLAC feature samples, an update in which a new CHLAC feature sample x is added is performed by the following equation.

これにより１ステップごとに計算量を割り振れるため、更新行列rの生成をより少ない計算量で行うことができる。

As a result, a calculation amount can be allocated for each step, so that the update matrix r can be generated with a smaller calculation amount.

  In Step 3-2, it is determined whether or not the condition for updating the partial space for normal operation is satisfied. If the condition is satisfied, the second stage process for updating the autocorrelation matrix R is performed. As a condition for determination, for example, a case where n in the above formula exceeds a certain value, that is, a case where the CHLAC feature constituting the update matrix exceeds a certain amount is raised.

  In step 3-3, a second stage process for updating the autocorrelation matrix R is performed. Specifically, an update matrix r is added to the autocorrelation matrix R. The update matrix r can be added using the following formula.

Here, N is the number of samples of the CHLAC feature constituting R, and M is the number of samples of the CHLAC feature constituting the update matrix r. Thereby, as described in 3-1, it is possible to update R with a constant calculation amount regardless of the number of samples. In addition, since it is not necessary to always store an enormous amount of CHLAC features in the memory, it is possible to perform abnormality monitoring with a small amount of memory.
In the above equation 5, when the value of N becomes sufficiently large, the influence of the newly added update matrix r on the autocorrelation matrix R becomes small, and it becomes difficult to follow a rapid environmental change.

In the present invention, in order to solve these problems, a mechanism has been prepared in which the effect of the newly added update matrix r on the autocorrelation matrix R does not become below a certain level. When this mechanism is used, the autocorrelation matrix R is updated by the following equation.

式5.0中のk0は[0 1]の実数である。

K0 in Equation 5.0 is a real number of [0 1].

FIG. 6 illustrates the ratio of the update matrix r newly added under the control of k0 to the autocorrelation matrix R. It can be seen that the lower limit of the ratio of the newly added update matrix r to the autocorrelation matrix R is kept constant by introducing k0.
If the value of k0 is increased, the influence of the newly added update matrix r becomes stronger, and automatic tracking can be performed even for sudden environmental changes. Conversely, if the value of k0 is reduced, the importance of the existing autocorrelation matrix R increases, and the follow-up to environmental changes becomes gradual.

  In step 3-4, the value of the autocorrelation matrix R on the shared memory is updated. Shared memory refers to memory that stores data shared with another CPU. After updating the shared memory, the update matrix r is initialized with a zero matrix.

In step 4-1, it is confirmed whether the eigenvector on the shared memory has been updated. The space spanned by this eigenvector becomes a partial space for normal operation used for abnormality monitoring. As an example of the update confirmation method, there is a method in which the CPU 13 that calculates the partial space for normal operation notifies the CPU 12 of the execution state (waiting, calculating, completion of calculation, etc.).
If the eigenvector on the shared memory has been updated, the process proceeds to step 4-2, and the normal operation partial space is updated. If the normal operation partial space on the shared memory has not been updated, the process proceeds to step 4-3 to calculate an abnormal value. The normal operation subspace used at that time continues the abnormality monitoring using the normal operation subspace already calculated using the old autocorrelation matrix as shown in FIG.
In addition, since the calculation method of the subspace of normal operation is performed by parallel processing on different processes, the method of deriving the subspace of normal operation will be described later.

  If the update interval of the subspace of normal operation is a period of about the frame interval, the CHLAC feature newly added to the autocorrelation matrix R is about 100 samples at most, so the error before and after the addition of the update matrix r is extremely small. A large performance difference does not occur in the partial space of normal operation before and after the update. Therefore, there is no problem even if abnormality monitoring is performed using the partial space for normal operation before the update for a short period during which the partial space for normal operation is updated. According to the present invention, it is possible to perform abnormality monitoring even during calculation of a partial space for normal operation.

  In step 4-2, the eigenvector calculated by another process is read from the shared memory as a partial space for normal operation.

  In step 4-3, an abnormal value serving as a guideline for abnormality or normality is calculated from the CHLAC feature obtained in step 2-2 and the normal operation partial space read from the shared memory in step 4-2. Specifically, the vertical distance d⊥ between the CHLAC feature and the normal operation subspace is obtained and used as an abnormal value.

  As a first step for calculating the vertical distance d⊥, a projector P to a subspace spanned by eigenvectors UK = [u1,..., UK] obtained by principal component analysis and its orthogonal complement space Projector P⊥ to is calculated from the following equation.

式中のＵ'は行列Ｕの転置行列であり、ＩMはＭ次の単位行列である。直交補空間での２乗距離、即ち、部分空間Ｕへの垂線の２乗距離ｄ2⊥は下記式で表現する。

U ′ in the equation is a transposed matrix of the matrix U, and IM is an M-th unit matrix. The square distance in the orthogonal complement space, that is, the square distance d2⊥ of the perpendicular to the subspace U is expressed by the following equation.

  In the present embodiment, this vertical distance d と す る is an abnormal value that is an indicator of whether or not it is abnormal. In addition to the above method, a method may be used in which an angle formed by the CHLAC feature and the normal operation subspace is an abnormal value.

  In Step 4-4, it is determined whether the abnormal value obtained in Step 4-3 is larger or smaller than the threshold value.

  Step 4-5 is processing when the abnormal value is smaller than the threshold value in Step 4-4. In this process, the corresponding frame is determined as a frame in which the normal operation is recorded.

  Step 4-6 is processing when the abnormal value is larger than the threshold value in Step 4-4. In this process, the corresponding frame is determined as a frame in which an abnormal operation is recorded.

  In Step 4-7, an abnormality or normality determination result is output to a display or the like.

In Step 4-8, it is determined whether or not abnormality monitoring is to be terminated. For example, when the user performs a process to end the abnormality monitoring, the abnormality monitoring is ended.
If abnormality monitoring is to be continued, return to step 1-0 and continue processing.

  The flowchart of the CPU 13 is a flowchart for updating the partial space for normal operation. This process is performed on a process different from the abnormality monitoring process described above. Another process refers to a state in which, for example, a multi-core CPU executes with another core. If the flowchart of the CPU 12 and the flowchart of the CPU 13 operate independently, there is no problem as a device. Therefore, a method of independently operating each flowchart on a single CPU that is not multi-core is also conceivable.

  In step 5-1, it is confirmed whether or not the autocorrelation matrix R has been updated by the process 3-4 on the CPU 12. As an example of the update confirmation method, there is a method in which the CPU 12 that calculates the partial space for normal operation notifies the CPU 13 of the execution state (standby, calculating, completion of calculation, etc.).

In step 5-2, a subspace of normal operation is obtained from the autocorrelation matrix acquired in step 5-1 using a principal component analysis method.
Principal component analysis as described above

の固有問題を解く事で求まる。式中の自己相関行列Rは本実施例に置いては２５１×２５１の行列で表現されている。
式中のUは主成分ベクトル（固有ベクトル）を列に並べた行列であり下記の様に表現する。

It is obtained by solving the eigenproblem of. In the present embodiment, the autocorrelation matrix R in the equation is represented by a 251 × 251 matrix.
U in the equation is a matrix in which principal component vectors (eigenvectors) are arranged in columns, and is expressed as follows.

uは主成分分析によって求まった固有ベクトルである。本実施例ではL=２５１となる。Λは固有値行列を指し下記式で表現する。

u is an eigenvector obtained by principal component analysis. In this embodiment, L = 251. Λ indicates an eigenvalue matrix and is expressed by the following equation.

ここで、λは主成分分析で求まった固有ベクトルに対応する固有値である。

Here, λ is an eigenvalue corresponding to the eigenvector obtained by principal component analysis.

  Further, the cumulative contribution rate ηK up to the Kth eigenvalue is expressed by the following equation.

ここでηKがある一定の値(例えばηK=0.99)になるまでK個の固有ベクトルを列にならべた下記行列を通常動作の部分空間とする。

Here, the following matrix in which K eigenvectors are arranged in columns until ηK reaches a certain value (for example, ηK = 0.99) is defined as a subspace of normal operation.

  Note that the optimum value of the cumulative contribution rate ηK is determined by an experiment or the like because it is considered to depend on the monitoring target and the detection accuracy. By performing the above calculation, a partial space for normal operation is generated.

  In step 5-3, the eigenvector obtained in step 5-2 is stored in the shared memory. The space spanned by this eigenvector becomes a subspace of normal operation.

  In step 5-4, the same end determination of abnormality monitoring as in step 4-8 is performed. However, if abnormality monitoring is continued, the process returns to step 5-1.

As another embodiment of the present invention, an embodiment will be described in which the normal operation subspace is updated using only the CHLAC features having high abnormal values. The system configuration example is the same as that of the first embodiment.
FIG. 7 shows an outline of the operation of the CPU 12 and the CPU 13. The CPU 12 reads the camera video as process 1. Next, as a process 2, a CHLAC feature is calculated. Next, as processing 3, it is determined whether it is abnormal or normal. Further, in the process 4, the autocorrelation matrix is updated. When updating the autocorrelation matrix, assuming that a CHLAC feature having a high abnormal value is data for an unlearned normal operation, only a CHLAC feature having a high abnormal value is used. In the CPU 13, a partial space for normal operation necessary for determining whether the process is abnormal or normal is generated from the autocorrelation matrix obtained by the CPU 12. Until the CPU 13 completes the calculation of the subspace for normal operation, the CPU 12 performs abnormality monitoring using the subspace for normal operation calculated in the past.

FIG. 8 describes details of the processing of FIG. In the CPU 12, the steps a-0, a-1, a-2, b-1, b-2 are respectively changed to steps 1-0, 1-1, 1-2, 2-1 and 2-2 in FIG. Correspondingly, from the reading of the frame data to the calculation of the CHLAC feature.
Steps c-1, c-2, c-3, c-4, c-5, c-6, c-7 are steps 4-1, 4-2, 4-3, 4-4 in FIG. Correspond to 4-5, 4-6, 4-7, and judge whether it is abnormal or normal.

d-1 determines whether to use the CHLAC feature determined in b-2 for updating the subspace of the normal operation based on the abnormal value calculated in c-3.
When the CHLAC feature is used to update the subspace for normal operation, the process proceeds to d-2. Otherwise, transition to e-1.
As an example of specific determination conditions, a method of sampling only the CHLAC feature having the maximum abnormal value from a series of operations determined to be abnormal can be considered.
As a method of defining a series of operations, for example, there is a method of defining a collection of abnormal values satisfying abnormal value> threshold as a series of operations as shown in FIG. As an example of the threshold value, a threshold value for determining whether abnormality is normal or normal may be set.
With this method, it is possible to efficiently sample the CHLAC features used for updating the subspace in normal operation.

  In step d-2, the CHLAC feature x satisfying the sampling condition in d-1 is added to the CHLAC feature group Fa represented by the following equation.

In this embodiment, the CHLAC feature group Fa is treated as an update matrix.
In this embodiment, the dimension number L of the CHLAC feature is 251.
In step c-3, the normal operation subspace is used to determine whether the operation is abnormal or normal. this is,

It is obtained from an eigenvalue matrix Λ and an eigenvector U obtained by a method called principal component analysis that solves the eigenvalue problem of Equation 14. In the first embodiment, the autocorrelation matrix calculated from the CHLAC feature group is adopted as R, and the eigenvalue matrix Λ and the eigenvector U are calculated.
In addition to such a method, a method of calculating the autocorrelation matrix R using a partial space of normal operation that has already been obtained, as shown in the present embodiment, may be used.
When the method shown in the present embodiment is adopted, since the matrix size of the autocorrelation matrix R is reduced, it is possible to reduce the time required to update the partial space for normal operation.

  In step d-3, it is determined whether or not a condition for updating the partial space for normal operation is satisfied. As a determination condition, for example, a case where the number of stored CHLAC feature samples of d-2 exceeds a predetermined value can be cited. Let this predetermined value be n.

  In step d-4, an autocorrelation matrix R necessary for updating the subspace of normal operation is obtained. In the first embodiment, the autocorrelation matrix R calculated from the CHLAC feature group is calculated. However, in this embodiment, the CHLAC feature group Fa sampled at d-2 and the eigenvector Uo that currently spans the subspace of normal operation, An autocorrelation matrix R is calculated from a matrix obtained by the following equation from the eigenvalue Λo obtained when Uo is derived.

Here, N is the total number of CHLAC features calculated until n samples of CHLAC features are collected.
The CHLAC feature group sampled in the past is represented using the eigenvector Uo. The number of dimensions of the subspace Uo of the normal operation becomes the representative sample number of the CHLAC feature group sampled in the past.

  In order to follow environmental fluctuations, the weighting ratio between the existing data Uo and the newly added data Fa is controlled as in the first embodiment. As a result, the influence of the newly added data Fa on the autocorrelation matrix R becomes a certain level or more. The derivation of X with the weighted ratio controlled is performed by the following equation, for example.

式4.0中のk0は[0 1]の実数である。

K0 in Equation 4.0 is a real number of [0 1].

FIG. 6 illustrates the ratio of the update matrix newly added under the control of k0 to the learning data X. It can be seen that the lower limit of the ratio of the update matrix newly added to the learning data X is kept constant by introducing k0.
When the value of k0 is increased, the influence of a newly added update matrix becomes stronger, and automatic tracking can be performed even for a sudden environmental change. Conversely, if the value of k0 is reduced, the importance of the existing data will increase, and the follow-up to environmental changes will be gradual.

  In step d-5, an autocorrelation matrix R used for calculation of principal component analysis is calculated from the learning data X generated in step d-4. In this embodiment, R is calculated by the following formula.

When R is calculated in this way when there are few CHLAC feature groups used for updating the normal subspace, the R matrix size is smaller than the autocorrelation matrix calculated from the CHLAC feature groups.
For example, when this method is used when the subspace of normal operation is 5 dimensions and the number of samples of CHLAC features used for updating is 100 samples, the autocorrelation matrix R is a 105 × 105 matrix, and the matrix size is The time required for executing the principal component analysis is reduced by the smaller amount.

  In step d-6, the autocorrelation matrix R on the shared memory is updated. Shared memory refers to memory that stores data shared with another CPU. After updating the shared memory, empty the update matrix.

In step e-1, it is determined whether or not to stop the abnormality monitoring. For example, when the user performs a process to end the abnormality monitoring, the abnormality monitoring is ended.
When the abnormality monitoring is continued, the process returns to step a-0 to continue the processing.

  The flowchart of the CPU 13 is a process for updating the partial space for normal operation. This process is performed on a process different from the abnormality monitoring process described above. Another process refers to a state in which, for example, a multi-core CPU executes with another core. If the flowchart of the CPU 12 and the flowchart of the CPU 13 operate independently, there is no problem as a device. Therefore, a method of independently operating each flowchart on a single CPU that is not multi-core is also conceivable.

  In step f-1, it is confirmed whether the autocorrelation matrix R has been updated by the process of d-4 on the CPU 12. As an example of the update confirmation method, there is a method in which the CPU 12 that calculates the partial space for normal operation notifies the CPU 13 of the execution state (standby, calculating, completion of calculation, etc.).

In step f-3, a subspace of normal operation is obtained from the autocorrelation matrix R calculated in step f-2 using the principal component analysis method.
Principal component analysis as described above

の固有問題を解く事で求まる。
式中のUAは主成分ベクトル（固有ベクトル）を列に並べた行列であり下記の様に表現する。

It is obtained by solving the eigenproblem of.
UA in the equation is a matrix in which principal component vectors (eigenvectors) are arranged in columns, and is expressed as follows.

uは主成分分析によって求まった固有ベクトルである。ｌは学習データＸのサンプル数である。ΛAは固有値行列を指し下記式で表現する。

u is an eigenvector obtained by principal component analysis. l is the number of samples of the learning data X. ΛA indicates an eigenvalue matrix and is expressed by the following equation.

ここで、λは主成分分析で求まった固有ベクトルに対応する固有値である。
さらに第Ｋ固有値までの累積寄与率ηK は、以下の様な式で表す。

Here, λ is an eigenvalue corresponding to the eigenvector obtained by principal component analysis.
Further, the cumulative contribution rate ηK up to the Kth eigenvalue is expressed by the following equation.

ここでηKがある一定の値(例えばηK=0.99)になるまでK個の固有ベクトルを列にならべた行列を

Here, a matrix in which K eigenvectors are arranged in columns until ηK reaches a certain value (for example, ηK = 0.99).

また、ηKがある一定の値(例えばηK=0.99)になるまでK個の固有値から求めた行列を、

Further, a matrix obtained from K eigenvalues until ηK becomes a certain value (for example, ηK = 0.99),

とした時、通常動作の部分空間を張る固有ベクトルＵnは下記の式で求まる。

Then, the eigenvector Un that spans the subspace of normal operation is obtained by the following equation.

なお、累積寄与率ηKの最適値は監視対象や検出精度にも依存するものと考えられるので実験等により決定する。以上の計算を行うことにより、通常動作の部分空間が生成される。

Note that the optimum value of the cumulative contribution rate ηK is determined by an experiment or the like because it is considered to depend on the monitoring target and the detection accuracy. By performing the above calculation, a partial space for normal operation is generated.

  In step f-4, the eigenvector Un obtained in step f-3 is stored on the shared memory as a partial space for normal operation.

  In step f-5, the same abnormality monitoring end determination as in step e-1 is performed. However, when abnormality monitoring is continued, the process returns to step f-1.

In the first and second embodiments, it is assumed that the CPU 12 and the CPU 13 exist on one PC. However, a configuration in which two PCs distributed on the network are used as the CPU 12 and the CPU 13, respectively. Conceivable. In such a configuration, when the CPU 12 and the CPU 13 transmit and receive the autocorrelation matrix and the normal partial space, there is a method of sharing the memory via the network.
Furthermore, in the first and second embodiments, the flowchart operating on the CPU 13 continues to loop until the abnormality monitoring is completed, but may be handled as a subroutine called from the CPU 12 when the normal operation is updated.

  In the first embodiment and the second embodiment, the lower limit value of the weight for the update matrix is given as a fixed value, but it may be obtained adaptively depending on the frequency of abnormality. For example, when the frequency of abnormalities in the past fixed period is obtained, if the frequency of abnormalities is large, the lower limit value of the weight for the update matrix is set large, and if the frequency of abnormalities is small, the lower limit value of the weight for the update matrix is set small To do.

As an advantage of using the present invention, it is easier to introduce compared to existing systems. Existing off-line monitoring systems need to learn normal operation in advance, and it is necessary to prepare normal operation images in consideration of environmental fluctuations, which requires enormous costs and time during installation and setup. There was a need. On the other hand, even with conventional online monitoring systems, it is difficult to follow up with explicit environmental fluctuations, so it is not possible to follow environmental fluctuations and learning must be performed from the beginning. There were cases where it increased dramatically. On the other hand, according to the present invention, it is possible to realize an on-line learning type abnormality monitoring system that can update normal operation with a short update interval and high calculation accuracy, and drastically reduce the cost at the time of product introduction and the time spent on the setup. It becomes possible to do.
As a concrete usage method of the present invention, there are many normal operations that should be defined as in a condominium entrance or factory premises, and introduction into a place where the environment fluctuates rapidly is conceivable.

It is a comparison figure of the update method of the partial space of normal operation of the present invention and the conventional method. It is a block diagram which shows the abnormality monitoring system by this invention. It is the schematic which shows operation | movement of the 1st abnormality monitoring apparatus by this invention. It is a flowchart figure which shows operation | movement of the 1st abnormality monitoring apparatus by this invention of FIG. It is the figure which set the time window to the row | line | column which arranged the CHLAC data of the abnormality monitoring apparatus by this invention in time series. It is a figure which shows the mode of the ratio which the update matrix added newly adds to an autocorrelation matrix. It is the schematic which shows operation | movement of the 2nd abnormality monitoring apparatus by this invention. It is a flowchart figure which shows operation | movement of the 2nd abnormality monitoring apparatus by this invention of FIG. It is a figure which shows the CHLAC feature which satisfy | fills the sampling condition of a CHLAC feature.

Explanation of symbols

10; Camera 11; PC
12; CPU1
13; CPU2
14; shared memory 15; keyboard 16; mouse 17; monitor

Claims (8)

Means for storing moving image frame data input in real time in a memory;
Means for generating inter-frame difference data from the moving image frame data;
Means for extracting feature data from the inter-frame difference data by three-dimensional higher order local autocorrelation;
Means for generating an update matrix using the feature data;
Means for adding the update matrix and updating the autocorrelation matrix on the shared memory;
Means for reading eigenvectors used for anomaly detection from shared memory;
Means for obtaining an abnormal value from the characteristic data and the eigenvector read from the shared memory, determining whether the abnormal value is larger or smaller than a threshold,
Means for outputting a determination result when the abnormality determination means determines an abnormality;
A first central processing unit for executing
Means for generating eigenvectors from the autocorrelation matrix on the shared memory by principal component analysis;
Means for storing the generated eigenvector in the shared memory and performing an update process;
A second central processing unit for executing
An abnormal operation monitoring device comprising:   The abnormal operation monitoring apparatus according to claim 1, wherein the update matrix update unit generates an update matrix using only feature data determined to be abnormal by the abnormality determination unit.   3. The abnormal operation according to claim 1, wherein the autocorrelation matrix update unit calculates a weighted average of the update matrix to be added and the autocorrelation matrix, and the weight value of the update matrix to be added does not become a predetermined value or less. Monitoring device.   The abnormal operation monitoring apparatus according to claim 1, wherein the autocorrelation matrix updating unit performs learning data update processing using the eigenvector.   The first central processing unit does not read from the shared memory when the eigenvector on the shared memory is not updated by the second central processing unit, and uses the old eigenvector that has already been read. 3. The abnormal operation monitoring apparatus according to claim 1, wherein calculation is performed.   2. The feature data extracting means comprises means for calculating frame CHLAC data, adding the latest frame CHLAC data to the current feature data, and subtracting the frame CHLAC data older than a predetermined period. Or the abnormal operation | movement monitoring apparatus of 2 description.   The abnormal operation monitoring apparatus according to claim 1, wherein the abnormal value calculation obtains a vertical distance or angle between the feature data and a partial space of a normal operation stretched by the eigenvector.   The abnormal operation monitoring apparatus according to claim 1 or 2, wherein the first central processing unit and the second central processing unit are integrally configured.

Images