KR20210114163A - Method for detecting abnomaly using segmenting video image frames, and apparatus for the same - Google Patents

Abstract

According to an embodiment, the present invention relates to a technology detecting an anomaly from a video image monitored using an artificial intelligence technology. An anomaly detecting method using segmenting of a video image frame comprises: a step of collecting a selected number of frames from video image input to generate segments; a step of learning a neural network based on a feature of an intermediate layer of the neural network with the segments as input; and a step of inputting the segments to the learned neural network to detect the anomaly in a unit of the segments.

Description

Anomaly detection method using segmenting of video image frame and apparatus therefor

The present invention relates to a technology for detecting an abnormal situation from a video image monitored using artificial intelligence technology.

A video image-based anomaly detection system detects anomalies based on video images in a system equipped with a fixed camera such as CCTV or a mobile camera such as a mobile robot. And recently, artificial intelligence technology is used to detect abnormal situations. In order to distinguish between normal and abnormal situations, normal and abnormal are labeled on input data, and the labeled input data is input into artificial intelligence to learn and normalize. to distinguish between abnormal and abnormal.

That is, in the video-based anomaly detection system using the existing artificial intelligence technology, a label (a flag indicating whether a given frame is normal or abnormal) is given to each frame of the input video image, and this label is used as a learning signal for AI learning. The supervised learning base used was the main method.

However, such a supervised learning-based method has a problem in that it is necessary to define what is an anomaly in advance, and accordingly, an anomaly that is not defined in advance cannot be detected. And in reality, a bigger problem than this is that a person has to label every frame of the video to see if there are any abnormalities, because such a labeling operation requires a lot of effort and cost.

Korean Patent Application Laid-Open No. 10-2019-0100085 discloses a robot capable of detecting a dangerous situation and an operation method thereof using artificial intelligence. The prior patent relates to a technology for detecting a dangerous situation using a voice recognition model and an image recognition model, and a cost function corresponding to the difference between the target feature vector outputted by the image recognition model and the current labeled (or labeled) situation. learn to minimize Accordingly, the prior patent also has a problem in that the current situation must be labeled on a frame-by-frame basis, as pointed out as a problem of the existing technology.

Therefore, there is a need for a technology that solves the problems of the prior art and reduces the labor and cost of the labeling task required for artificial intelligence learning of an anomaly detection system.

An object of the present invention is to provide a method for detecting anomalies using segmenting of a video image frame and an apparatus for the same, thereby detecting anomalies at the frame level with only labeling information at the video level, that is, information indicating whether the video contains abnormal conditions. An object of the present invention is to provide an anomaly detection system capable of detecting.

Another object of the present invention is to provide a method for learning an artificial intelligence neural network to be used for anomaly detection by using an abnormal video image or a normal video image.

An anomaly detection method using segmenting a video image frame according to an embodiment includes generating segments by combining a number of frames selected from a video image input; learning a neural network based on the characteristics of an intermediate layer of the neural network to which the segments are input; and inputting the segments into a learned neural network to detect anomalies in units of the segments.

Learning the neural network may include learning from an abnormal video image including at least one or more frames.

The step of learning from the abnormal video image may include: generating two clusters based on a characteristic of an intermediate layer of the neural network for the segment; selecting a cluster having a larger number of normal segments predicted to be normal by the neural network as a normal cluster among the two clusters; and selecting the remaining clusters that are not selected as the normal cluster among the two clusters as the abnormal cluster.

Learning from the abnormal video image may include: labeling labels of all segments belonging to the normal cluster as normal; labeling labels of all segments belonging to the abnormal cluster as abnormal; and setting the labels as a set of correct answers as a result of prediction of the neural network.

The step of learning from the abnormal video image may include calculating a loss function 1 by comparing final values of a neural network to which the segments are input with a set of prediction results of the neural network; calculating a loss function 2 by comparing the normal cluster with the abnormal cluster; and calculating an overall loss function by synthesizing the loss function 1 and the loss function 2 .

The loss function 2 may be inversely proportional to a distance between the center of the normal cluster and the center of the abnormal cluster.

Learning the neural network may include learning from a normal video image that does not include any abnormal frames.

The learning from the normal video image may include: generating two clusters based on a characteristic of an intermediate layer of the neural network for the segment; selecting one of the two clusters as a normal 1 cluster; and selecting the other one of the two clusters as the normal 2 cluster.

The learning with the normal video image may include calculating a loss function 1 by comparing final values of the neural network to which the segments are input with a set of correct answers indicating normal; calculating a loss function 2 by comparing the normal 1 cluster with the normal 2 cluster; and calculating an overall loss function by synthesizing the loss function 1 and the loss function 2 .

The loss function 2 may be proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster.

According to an embodiment, an apparatus for detecting anomalies using segmenting of video image frames includes: a segment generator configured to generate segments by combining a number of frames selected from a video image input; a neural network learning unit for learning the neural network based on the characteristics of the middle layer of the neural network to which the segments are input; and an anomaly detection unit configured to input the segments into a learned neural network and detect anomalies in units of the segments.

The neural network learning unit may include an abnormal video learning unit that learns from an abnormal video image including at least one or more frames.

The abnormal video learning unit includes an abnormal cluster generation unit that generates two clusters based on the characteristics of the intermediate layer of the neural network for the segment, and the abnormal cluster generation unit is a normal one of the two clusters predicted by the neural network as normal. A cluster having a larger number of segments may be selected as a normal cluster, and a remaining cluster not selected as the normal cluster among the two clusters may be selected as an abnormal cluster.

The abnormal video learning unit further includes a labeling generation unit generating labels for the segments, and the labeling generation unit generating normal labels of all segments belonging to the normal cluster, and The label may be generated as an ideal, and the labels may be set as a set of correct answers as a result of prediction of the neural network.

The abnormal video learning unit includes: an abnormal loss function 1 calculator for calculating a loss function 1 by comparing final values of the neural network to which the segments are input with a set of prediction results of the neural network; an abnormal loss function 2 calculator for calculating a loss function 2 by comparing the normal cluster with the abnormal cluster; and an ideal total loss function calculator for calculating an overall loss function by synthesizing the loss function 1 and the loss function 2 .

The loss function 2 may be inversely proportional to a distance between the center of the normal cluster and the center of the abnormal cluster.

The neural network learning unit may include a normal video learning unit that learns from a normal video image that does not include any abnormal frames.

The normal video learning unit includes a normal cluster generation unit that generates two clusters based on the characteristics of the intermediate layer of the neural network for the segment, and the normal cluster generation unit selects one of the two clusters as a normal 1 cluster, , the other one of the two clusters may be selected as the normal 2 cluster.

The normal video learning unit includes: a normal loss function 1 calculator for calculating a loss function 1 by comparing final values of the neural network to which the segments are input with a set of correct answers indicating normal; a normal loss function 2 calculator for calculating a loss function 2 by comparing the normal 1 cluster with the normal 2 cluster; and a stationary total loss function calculator for calculating a total loss function by synthesizing the loss function 1 and the loss function 2 .

The loss function 2 may be proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster.

According to the present invention, an anomaly detection method using video image frame segmenting and an apparatus therefor are provided, thereby detecting anomaly at the frame level only with labeling information at the video level, that is, information indicating whether the video includes an abnormal condition. An anomaly detection system that can do this can be provided.

In addition, it is possible to provide a method for learning an artificial intelligence neural network to be used for anomaly detection by using an abnormal video image or a normal video image.

1 is a block diagram illustrating an example of an anomaly detection apparatus using segmenting of a video image frame according to an embodiment.
FIG. 2 is a block diagram illustrating an example of the neural network learning unit 130 shown in FIG. 1 .
FIG. 3 is a diagram illustrating an example of the abnormal video learning unit 220 shown in FIG. 2 .
FIG. 4 is a diagram illustrating an example of the normal video learning unit 230 shown in FIG. 2 .
FIG. 5 is a diagram illustrating an example of application of the abnormal video learning unit 220 shown in FIG. 2 .
FIG. 6 is a diagram illustrating an example of application of the normal video learning unit 230 shown in FIG. 2 .
7 is an operation flowchart illustrating an example of an anomaly detection method using segmenting a video image frame according to an embodiment.
8 is an operation flowchart illustrating an example of neural network learning ( S730 ) shown in FIG. 7 according to an embodiment.
9 is an operation flowchart illustrating another example of neural network learning ( S730 ) shown in FIG. 7 according to an embodiment.
10 is a diagram showing the configuration of a computer system according to an embodiment.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

Although "first" or "second" is used to describe various elements, these elements are not limited by the above terms. Such terms may only be used to distinguish one component from another. Accordingly, the first component mentioned below may be the second component within the spirit of the present invention.

The terminology used herein is for the purpose of describing the embodiment and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” or “comprising” implies that the stated component or step does not exclude the presence or addition of one or more other components or steps.

Unless otherwise defined, all terms used herein may be interpreted with meanings commonly understood by those of ordinary skill in the art to which the present invention pertains. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, an apparatus for detecting anomaly using segmentation of a video image frame and an apparatus therefor according to an embodiment will be described in detail with reference to FIGS. 1 to 10 .

1 is a block diagram illustrating an example of an anomaly detection apparatus using segmenting of a video image frame according to an embodiment.

Referring to FIG. 1 , an anomaly detection apparatus 100 using segmenting of a video image frame includes a segment generator 120 , a neural network learning unit 130 , and an anomaly detection unit 140 . In addition, the device receives the video input image 110 as an input and outputs a segment-unit abnormality detection result 150 as an output.

The segment generator 120 receives the video input image 110 as an input and combines the selected number of frames to generate segments. When the selected number (the number of frames constituting one segment) becomes 1, the abnormality detection device becomes a system capable of determining abnormality and normality at the frame level, and when it is greater than 1, it is possible to determine abnormality and normality at the segment level. It can be a system that can

The neural network learning unit 130 receives the segments generated by the segment generating unit 120 as input and learns the neural network based on the characteristics of the intermediate layer of the neural network.

The neural network learning unit 130 may perform an operation of learning from an abnormal video image including at least one or more frames.

First, the neural network learning unit 130 may generate two groups, ie, two clusters, based on the characteristics of the intermediate layer of the neural network for the segment in order to perform the operation of learning from the abnormal video image.

In order for the anomaly detection device to learn the neural network, there must be an abnormality or normal labeling corresponding to the abnormality or normality determination result for each segment. However, in the present invention, since only labeling at the video level, ie, whether the input video image contains an abnormal situation, is known, which segment contains an abnormal situation and which segment is a normal segment for each segment is not known. In order to overcome this problem, the present invention proposes a method of clustering segments using the characteristics of the middle layer of the neural network. Here, clustering refers to grouping segments with similar characteristics into a group based on segment characteristics. Therefore, the neural network learning unit 130 divides the segments into two groups, ie, two clusters, labels each cluster and uses them as a set of correct answers.

The neural network learning unit 130 may select a cluster having a larger number of normal segments predicted to be normal by the neural network among the two clusters as a normal cluster, and select the remaining clusters that are not selected as the normal cluster as an abnormal cluster. . That is, among the two clusters, the cluster in which the segment predicted by the neural network is distributed more in each segment predicted by the neural network is selected as the normal cluster, and the remaining clusters are selected as the abnormal cluster.

In addition, the neural network learning unit 130 may label labels of all segments belonging to the normal cluster as normal, and may label labels of all segments belonging to the abnormal cluster as abnormal. The labels generated in this way can be set as a set of correct answers as a result of prediction of the neural network.

In the same manner as described above, the anomaly detection apparatus may generate a set of correct answers as a result of prediction of the neural network in units of segments, and may perform neural network learning using the set of correct answers.

The neural network learning unit 130 may calculate the loss function 1 by comparing final values of the neural network to which the segments are input with the set of prediction results of the neural network. In addition, the loss function 2 may be calculated by comparing the normal cluster and the abnormal cluster. And the total loss function can be calculated by synthesizing the loss function 1 and the loss function 2 . And the loss function 2 may be inversely proportional to a distance between the center of the normal cluster and the center of the abnormal cluster.

The loss function 1 can be obtained by comparing the final value (output or predicted value) of the neural network of each segment with a set of correct answers, and the equation can be defined as Equation 1 below.

[Equation 1]

는 손실함수1,

은 하나의 비디오 입력 영상에 있는 세그먼트의 개수,

는 i번째 세그먼트에 대한 신경망의 출력,

는 i번째 세그먼트가 속한 클러스터(또는 정답셋)를 의미한다. here,

is the loss function 1,

is the number of segments in one video input image,

is the output of the neural network for the ith segment,

denotes the cluster (or correct answer set) to which the i-th segment belongs.

The loss function 2 can be obtained by comparing two clusters. Since one of the two clusters indicates normal and the other indicates abnormality, it is preferable that the two clusters be separated as far apart as possible. Therefore, the loss function 2 can be calculated as in Equation 2 below using the distance between the two clusters.

[Equation 2]

는 손실함수2,

은 첫 번째 클러스터의 중심,

는 두번째 클러스터의 중심을 의미한다. here,

is the loss function 2,

is the center of the first cluster,

is the center of the second cluster.

As described above, since two elements of a segment and a cluster can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 below.

[Equation 3]

은 전체손실함수,

는 0과 1사이의 실수값,

는 손실함수1,

는 손실함수2이다. here

is the total loss function,

is a real value between 0 and 1,

is the loss function 1,

is the loss function 2.

는 교차 엔트로피 손실일 수도 있고,

는 클러스터에 속한 모든 원소들의 거리에 대한 평균일 수도 있다. The loss functions shown by the above equations are only examples, and may be implemented differently according to user's selection and application. E.g

may be a cross-entropy loss,

may be an average of the distances of all elements in the cluster.

Also, the neural network learning unit 130 may perform an operation of learning from a normal video image that does not include any abnormal frames.

First, the neural network learning unit 130 may generate two clusters based on the characteristics of the intermediate layer of the neural network for the segment in order to perform an operation of learning from a normal video image.

As described above, in order to learn the neural network, labeling of abnormality or normality corresponding to the determination result of abnormality or normality for each segment is required. Accordingly, the neural network learning unit 130 divides the segments into two groups, ie, two clusters, based on the characteristics of the intermediate layer of the neural network, labels each cluster, and uses the set as a correct answer set.

The neural network learning unit 130 may select one of the two clusters as the normal 1 cluster and select the other as the normal 2 cluster. In other words, since we know that all segments are normal, both clusters become normal clusters, and the prediction result of the segment-by-segment neural network can also be set to normal. Therefore, neural network learning can be performed with a set of correct answers for prediction of the neural network in the segment unit.

The neural network learning unit 130 may calculate the loss function 1 by comparing final values of the neural network to which the segments are input with a set of correct answers indicating normality. Also, the loss function 2 may be calculated by comparing the normal 1 cluster and the normal 2 cluster. And the total loss function can be calculated by synthesizing the loss function 1 and the loss function 2 . In addition, the loss function 2 may be proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster.

The loss function 1 can be obtained by comparing the final value (output or predicted value) of the neural network of each segment with the correct answer set, and the equation can be defined as Equation 1 above.

The loss function 2 can also be obtained by comparing two clusters. In addition, since both of the two clusters indicate normality, it is preferable that the distance between the two clusters be as close as possible. Therefore, the loss function 2 can be calculated as in Equation 4 below using the distance between the two clusters.

[Equation 4]

는 손실함수2,

은 첫 번째 클러스터의 중심,

는 두번째 클러스터의 중심을 의미한다. here,

is the loss function 2,

is the center of the first cluster,

is the center of the second cluster.

As described above, since two elements, a segment and a cluster, can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 above.

The neural network learning unit 130 may construct a plurality of abnormal video images and normal video images, and repeat the operation of learning from the abnormal video image and the operation of learning from the normal video image until the results converge. Clustering is performed whenever each video image is learned, and even if it is the same video image, the clustering result may change as learning proceeds. When learning from an abnormal video image, which cluster is normal or abnormal in the clustering result is not defined in advance, and a cluster having more segments determined as normal according to the output result of the neural network is selected as the normal cluster.

The abnormality detection unit 130 inputs the segments generated by the segment generation unit 120 to the neural network learned by the neural network learning unit, determines abnormality or normality with respect to each segment, and outputs the segment unit abnormality detection result 150 . .

FIG. 2 is a block diagram illustrating an example of the neural network learning unit 130 shown in FIG. 1 .

Referring to FIG. 2 , the neural network learning unit 130 may include a neural network 210 , and may include an abnormal video learning unit 220 or a normal video learning unit 230 .

The neural network 210 is an artificial intelligence neural network for receiving the segment generated by the segment generator as an input and determining abnormality or normality. The neural network 210 may be trained through the abnormal video learner 220 or the normal video learner 230 . The abnormal video learner 220 or the normal video learner 230 may learn based on the characteristics of the intermediate layer of the neural network. That is, the neural network performs learning through interaction with the abnormal video learner or the normal video learner.

The neural network learning unit 130 may include the abnormal video learning unit 220 performing an operation of learning from an abnormal video image including at least one or more frames.

The abnormal video learning unit 220 may generate two clusters based on the characteristics of the middle layer of the neural network for the segment in order to perform the operation of learning from the abnormal video image.

In order for the abnormal video learning unit 220 to learn the neural network, labeling of abnormality or normality corresponding to the determination result of abnormality or normality for each segment is required. However, in the present invention, since only labeling at the video level, ie, whether the input video image contains an abnormal situation, is known, which segment contains an abnormal situation and which segment is a normal segment for each segment is not known. In order to overcome this problem, the present invention proposes a method of clustering segments using the characteristics of the middle layer of the neural network. Here, clustering refers to grouping segments with similar characteristics into a group based on segment characteristics. Therefore, the abnormal video learning unit 220 divides the segments into two groups, ie, two clusters, labels each cluster, and uses them as a correct answer set.

The abnormal video learning unit 220 may select a cluster with a larger number of normal segments predicted as normal by the neural network among the two clusters as a normal cluster, and select the remaining clusters that are not selected as the normal cluster as an abnormal cluster. have. That is, among the two clusters, the cluster in which the segment predicted by the neural network is distributed more in each segment predicted by the neural network is selected as the normal cluster, and the remaining clusters are selected as the abnormal cluster.

The abnormal video learning unit 220 may label labels of all segments belonging to the normal cluster as normal, and label labels of all segments belonging to the abnormal cluster as abnormal. The labels generated in this way can be set as a set of correct answers as a result of prediction of the neural network.

In the same way as described above, the abnormal video learning unit 220 may generate a set of correct answers as a result of prediction of the neural network in units of segments, and may perform neural network learning with the set of correct answers.

The abnormal video learning unit 220 may calculate the loss function 1 by comparing final values of the neural network to which the segments are input with the set of prediction results of the neural network. In addition, the loss function 2 may be calculated by comparing the normal cluster and the abnormal cluster. And the total loss function can be calculated by synthesizing the loss function 1 and the loss function 2 . And the loss function 2 may be inversely proportional to a distance between the center of the normal cluster and the center of the abnormal cluster.

The loss function 1 can be obtained by comparing the final value (output or predicted value) of the neural network of each segment with the correct answer set, and the equation can be defined as Equation 1 above. The loss function 2 can be obtained by comparing two clusters. Since one of the two clusters indicates normal and the other indicates abnormality, it is preferable that the two clusters be separated as far apart as possible. Therefore, the loss function 2 can be calculated as in Equation 2 by using the distance between the two clusters. As described above, since two elements, a segment and a cluster, can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 above.

는 교차 엔트로피 손실일 수도 있고,

는 클러스터에 속한 모든 원소들의 거리에 대한 평균일 수도 있다. The loss functions shown by the above equations are merely examples, and may be implemented differently according to a user's selection and application. E.g

may be a cross-entropy loss,

may be an average of the distances of all elements in the cluster.

In addition, the neural network learning unit 130 may include a normal video learning unit 230 that performs an operation of learning from a normal video image that does not include any abnormal frames.

First, in order for the normal video learning unit 230 to learn from a normal video image, two clusters may be generated based on the characteristics of the intermediate layer of the neural network for the segment.

As described above, in order to learn the neural network, labeling of abnormality or normality corresponding to the determination result of abnormality or normality for each segment is required. Therefore, the normal video learning unit 230 divides the segments into two groups, ie, two clusters, based on the characteristics of the intermediate layer of the neural network, labels each cluster, and uses it as a correct answer set.

The normal video learning unit 230 may select one of the two clusters as the normal 1 cluster and select the other as the normal 2 cluster. In other words, since we know that all segments are normal, both clusters become normal clusters, and the prediction result of the segment-by-segment neural network can also be set to normal. Therefore, neural network learning can be performed with a set of correct answers for prediction of the neural network in the segment unit.

In addition, the normal video learning unit 230 may calculate the loss function 1 by comparing final values of the neural network to which the segments are input with a set of correct answers indicating normality. Also, the loss function 2 may be calculated by comparing the normal 1 cluster and the normal 2 cluster. And the total loss function can be calculated by synthesizing the loss function 1 and the loss function 2 . In addition, the loss function 2 may be proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster.

The loss function 1 can be obtained by comparing the final value (output or predicted value) of the neural network of each segment with the correct answer set, and the equation can be defined as Equation 1 above. The loss function 2 can also be obtained by comparing two clusters. In addition, since both of the two clusters indicate normality, it is preferable that the distance between the two clusters be as close as possible. Therefore, the loss function 2 can be calculated as in Equation 4 using the distance between the two clusters. As described above, since two elements, a segment and a cluster, can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 above.

The neural network learning unit 130 may construct a plurality of abnormal video images and normal video images, and repeat the operation of learning from the abnormal video image and the operation of learning from the normal video image until the results converge. At this time, clustering is performed every time each video image is learned, and even for the same video image, the clustering result may change as learning proceeds. When learning from an abnormal video image, which cluster is normal or abnormal in the clustering result is not defined in advance, and a cluster having more segments determined as normal according to the output result of the neural network is selected as the normal cluster.

FIG. 3 is a diagram illustrating an example of the abnormal video learning unit 220 shown in FIG. 2 .

Referring to FIG. 3 , the anomaly video learning unit 220 includes an anomaly cluster generation unit 310 , a label generation unit 320 , an abnormal loss function 1 calculation unit 330 , an abnormal loss function 2 calculation unit 340 , and an anomaly A total loss function calculation unit 350 may be included.

The abnormal cluster generating unit 310 may generate two clusters based on the characteristics of the middle layer of the neural network 210 for the segment.

In order for the abnormal video learning unit 220 to learn the neural network, labeling of abnormality or normality corresponding to the determination result of abnormality or normality for each segment is required. However, in the present invention, since only labeling at the video level, ie, whether the input video image contains an abnormal situation, is known, which segment contains an abnormal situation and which segment is a normal segment for each segment is not known. In order to overcome this problem, the present invention proposes a method of clustering segments using the characteristics of the middle layer of the neural network. Here, clustering refers to grouping segments with similar characteristics into a group based on segment characteristics. Therefore, the abnormal video learning unit 220 divides the segments into two groups, ie, two clusters, labels each cluster, and uses them as a correct answer set.

The abnormal cluster generation unit 310 may select a cluster having a larger number of normal segments predicted as normal by the neural network among the two clusters as a normal cluster, and select the remaining clusters that are not selected as the normal cluster as an abnormal cluster. have. That is, among the two clusters, the cluster in which the segment predicted by the neural network is distributed more in each segment predicted by the neural network is selected as the normal cluster, and the remaining clusters are selected as the abnormal cluster.

The label generator 320 may label labels of all segments included in the normal cluster as normal, and label labels of all segments included in the abnormal cluster as abnormal. The labels generated in this way can be set as a set of correct answers as a result of prediction of the neural network.

In the same way as described above, the abnormal video learning unit 220 may generate a set of correct answers as a result of prediction of the neural network in units of segments, and may perform neural network learning with the set of correct answers.

The ideal loss function 1 calculator 330 may calculate the loss function 1 by comparing the final values of the neural network to which the segments are input with the set of prediction results of the neural network generated by the label generator 320 . Also, the abnormal loss function 2 calculation unit 340 may calculate the loss function 2 by comparing the abnormal cluster generated by the abnormal cluster generating unit 310 with the abnormal cluster. And the ideal total loss function calculation unit 350 synthesizes the loss function 1 calculated by the ideal loss function 1 calculation unit 330 and the loss function 2 calculated by the ideal loss function 2 calculation unit 340 to obtain a total The loss function can be calculated. And the loss function 2 may be inversely proportional to a distance between the center of the normal cluster and the center of the abnormal cluster.

The loss function 1 can be obtained by comparing the final value (output or predicted value) of the neural network of each segment with the correct answer set, and the equation can be defined as Equation 1 above. The loss function 2 can be obtained by comparing two clusters. Since one of the two clusters indicates normal and the other indicates abnormality, it is preferable that the two clusters be separated as far apart as possible. Therefore, the loss function 2 can be calculated as in Equation 2 by using the distance between the two clusters. As described above, since two elements, a segment and a cluster, can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 above.

는 교차 엔트로피 손실일 수도 있고,

는 클러스터에 속한 모든 원소들의 거리에 대한 평균일 수도 있다. The loss functions shown by the above equations are merely examples, and may be implemented differently according to a user's selection and application. E.g

may be a cross-entropy loss,

may be an average of the distances of all elements in the cluster.

The abnormal video learning unit 220 may construct a plurality of abnormal video images and repeat the operation of learning from the abnormal video images until the results converge. In this case, clustering is performed whenever each abnormal video image is learned, and even if it is the same video image, the clustering result may change as learning proceeds. In this case, which cluster is normal or abnormal in the clustering result is not defined in advance, and a cluster having more segments determined to be normal according to the output result of the neural network is selected as the normal cluster.

FIG. 4 is a diagram illustrating an example of the normal video learning unit 230 shown in FIG. 2 .

Referring to FIG. 4 , the normal video learning unit 230 includes a normal cluster generation unit 410 , a stationary loss function 1 calculation unit 420 , a stationary loss function 2 calculation unit 430 , and a stationary total loss function calculation unit 440 . ) may be included.

The normal cluster generator 410 may generate two clusters based on the characteristics of the middle layer of the neural network 210 for the segment.

In order to learn the neural network, there should be labeling of abnormality or normality corresponding to the determination result of abnormality or normality for each segment. Therefore, the normal video learning unit 230 divides the segments into two groups, ie, two clusters, based on the characteristics of the intermediate layer of the neural network, labels each cluster, and uses it as a correct answer set.

The normal cluster generator 410 may select one of the two clusters as the normal 1 cluster and select the other as the normal 2 cluster. In other words, since we know that all segments are normal, both clusters become normal clusters, and the prediction result of the segment-by-segment neural network can also be set to normal. Therefore, neural network learning can be performed with a set of correct answers for prediction of the neural network in the segment unit.

The normal loss function 1 calculator 420 may calculate the loss function 1 by comparing the final values of the neural network to which the segments are input with a set of correct answers indicating normality. Also, the normal loss function 2 calculator 430 may calculate the loss function 2 by comparing the normal 1 cluster and the normal 2 cluster. And the stationary total loss function calculation unit 440 synthesizes the loss function 1 calculated by the normal loss function 1 calculation unit 420 and the loss function 2 calculated by the stationary loss function 2 calculation unit 430 to obtain a total The loss function can be calculated. In addition, the loss function 2 may be proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster.

The loss function 1 can be obtained by comparing the final value (output or predicted value) of the neural network of each segment with the correct answer set, and the equation can be defined as Equation 1 above. The loss function 2 can also be obtained by comparing two clusters. In addition, since both of the two clusters indicate normality, it is preferable that the distance between the two clusters be as close as possible. Therefore, the loss function 2 can be calculated as in Equation 4 using the distance between the two clusters. As described above, since two elements, a segment and a cluster, can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 above.

The normal video learning unit 230 may repeat the operation of constructing a plurality of normal video images and learning from the normal video images until the results converge. At this time, clustering is performed every time each video image is learned, and even for the same video image, the clustering result may change as learning proceeds.

FIG. 5 is a diagram illustrating an example of application of the abnormal video learning unit 220 shown in FIG. 2 .

Referring to FIG. 5 , an abnormal video image 510 including an abnormal frame is input to the segment generator. In the abnormal video image 510, each rectangle represents a frame of a video image, and a dark color among the rectangles indicates an abnormal frame.

Thereafter, the segment generating unit generates segments 520 by bundling each frame of the abnormal video image by a selected number.

Then, the segments are input to the neural network 530 , and the abnormal cluster generator generates two clusters 550 based on the characteristics of the middle layer of the neural network. In addition, a cluster having a larger number of normal segments predicted by the neural network as a normal among the two clusters may be selected as the normal cluster 560 , and the remaining clusters may be selected as the abnormal cluster 570 .

The label generator may label labels of segments belonging to the normal cluster as normal, and label labels of segments belonging to the abnormal cluster as abnormal. The labels generated in this way can be set as a set of correct answers as a result of prediction of the neural network.

In this way, the abnormal video learning unit may generate a set of correct answers as a result of prediction of the neural network in units of segments, and may perform neural network learning using the set of correct answers.

The abnormal loss function 1 calculator may calculate the loss function 1 580 by comparing the final values 540 of the neural network to which the segments are input with the prediction result set 550 of the neural network generated by the label generator. . In addition, the abnormal loss function 2 calculation unit may calculate the loss function 2 (590) by comparing the normal cluster 560 and the abnormal cluster 570 generated by the abnormal cluster generation unit. In this case, the loss function 2 may be inversely proportional to a distance between the center of the normal cluster and the center of the abnormal cluster. Thereafter, the ideal total loss function calculation unit may calculate the total loss function by synthesizing the loss function 1 and the loss function 2 .

The loss function 1 can be obtained by comparing the final value (output or predicted value) of the neural network of each segment with the correct answer set, and the equation can be defined as Equation 1 above. The loss function 2 can be obtained by comparing two clusters. Since one of the two clusters indicates normal and the other indicates abnormality, it is preferable that the two clusters be separated as far apart as possible. Therefore, the loss function 2 can be calculated as in Equation 2 by using the distance between the two clusters. As described above, since two elements, a segment and a cluster, can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 above.

는 교차 엔트로피 손실일 수도 있고,

는 클러스터에 속한 모든 원소들의 거리에 대한 평균일 수도 있다. The loss functions shown by the above equations are merely examples, and may be implemented differently according to a user's selection and application. E.g

may be a cross-entropy loss,

may be an average of the distances of all elements in the cluster.

FIG. 6 is a diagram illustrating an example of application of the normal video learning unit 230 shown in FIG. 2 .

Referring to FIG. 6 , a normal video image 610 including no abnormal frames is input to the segment generator. In the normal video image 610, each rectangle is a frame of a video image, and all of them represent normal frames.

Thereafter, the segment generator generates segments 620 by grouping each frame of the abnormal video image by a selected number.

Then, the segments are input to the neural network 630 , and the normal cluster generator generates two clusters 660 based on the characteristics of the middle layer of the neural network. Then, one of the two clusters may be selected as the normal 1 cluster 670 , and the other may be selected as the normal 2 cluster 675 . That is, since it is known that all segments are normal, both clusters become normal clusters, and the prediction result set 650 of the neural network for each segment is also set to normal. Therefore, neural network learning can be performed with a set of correct answers for prediction of the neural network in the segment unit.

The normal loss function 1 calculator may calculate the loss function 1 680 by comparing the final values 640 of the neural network to which the segments are input with the correct answer set 650 indicating normality. Also, the stationary loss function 2 calculator may calculate the loss function 2 690 by comparing the normal 1 cluster 670 and the normal 2 cluster 675 . In this case, the loss function 2 may be proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster. Thereafter, the stationary total loss function calculation unit may calculate the total loss function by synthesizing the loss function 1 and the loss function 2 .

The loss function 1 can be obtained by comparing the final value (output or predicted value) of the neural network of each segment with the correct answer set, and the equation can be defined as Equation 1 above. The loss function 2 can also be obtained by comparing two clusters. In addition, since both of the two clusters indicate normality, it is preferable that the distance between the two clusters be as close as possible. Therefore, the loss function 2 can be calculated as in Equation 4 using the distance between the two clusters. As described above, since two elements, a segment and a cluster, can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 above.

7 is an operation flowchart illustrating an example of an anomaly detection method using segmenting a video image frame according to an embodiment.

Referring to FIG. 7 , a video image is input to a segment generator of an anomaly detection apparatus using segmenting of a video image frame ( S710 ).

The segment generator generates segments by combining a selected number of frames from the input video image (S720). When the selected number (the number of frames constituting one segment) becomes 1, the anomaly detection device becomes a system capable of determining abnormality and normality at the frame level, and when it is greater than 1, it is possible to determine abnormality and normality at the segment level. It can be a system that can

The neural network learning unit learns the neural network based on the characteristics of the middle layer of the neural network to which the segments are input (S730). If learning of the neural network is completed, the neural network learning step S730 may be omitted.

First, the neural network learning unit may perform an operation of learning from an abnormal video image including at least one or more frames.

In order to perform the operation of the neural network learning unit learning from the abnormal video image, two clusters may be generated based on the characteristics of the intermediate layer of the neural network for the segment.

The neural network learning unit may select a cluster having a larger number of normal segments predicted as normal by the neural network from among the two clusters as a normal cluster, and select the remaining clusters that are not selected as the normal cluster as an abnormal cluster. That is, among the two clusters, the cluster in which the segment predicted by the neural network is distributed more in each segment predicted by the neural network is selected as the normal cluster, and the remaining clusters are selected as the abnormal cluster.

The neural network learning unit may label labels of all segments included in the normal cluster as normal, and label labels of all segments included in the abnormal cluster as abnormal. The labels generated in this way can be set as a set of correct answers as a result of prediction of the neural network.

The neural network learning unit may calculate the loss function 1 by comparing final values of the neural network to which the segments are input with the set of prediction results of the neural network. In addition, the loss function 2 may be calculated by comparing the normal cluster and the abnormal cluster. And the total loss function can be calculated by synthesizing the loss function 1 and the loss function 2 . And the loss function 2 may be inversely proportional to a distance between the center of the normal cluster and the center of the abnormal cluster.

The loss function 1 can be obtained by comparing the final value (output or predicted value) of the neural network of each segment with the correct answer set, and the equation can be defined as Equation 1 above. The loss function 2 can be obtained by comparing two clusters. Since one of the two clusters indicates normal and the other indicates abnormality, it is preferable that the two clusters be separated as far apart as possible. Therefore, the loss function 2 can be calculated as in Equation 2 by using the distance between the two clusters. As described above, since two elements, a segment and a cluster, can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 above.

Also, the neural network learning unit may perform an operation of learning from a normal video image that does not include any abnormal frames.

First, in order for the neural network learning unit to learn from a normal video image, two clusters may be generated based on a characteristic of an intermediate layer of the neural network for the segment.

The neural network learning unit may select one of the two clusters as the normal 1 cluster and select the other as the normal 2 cluster. In other words, since we know that all segments are normal, both clusters become normal clusters, and the prediction result of the segment-by-segment neural network can also be set to normal. Therefore, neural network learning can be performed with a set of correct answers for prediction of the neural network in the segment unit.

In addition, the neural network learning unit may calculate the loss function 1 as in Equation 1 by comparing the final values of the neural network to which the segments are input with a set of correct answers indicating normality. Also, the loss function 2 may be calculated by comparing the normal 1 cluster and the normal 2 cluster. And the total loss function can be calculated by synthesizing the loss function 1 and the loss function 2 . In addition, the loss function 2 may be proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster.

The loss function 1 can be obtained by comparing the predicted value of the neural network of each segment with the correct answer set, and the equation can be defined as Equation 1 above. The loss function 2 can be obtained by comparing two clusters. In addition, since both of the two clusters indicate normality, it is preferable that the distance between the two clusters be as close as possible. Therefore, the loss function 2 can be calculated as in Equation 4 using the distance between the two clusters. As described above, since two elements of a segment and a cluster can be considered in the loss function, the overall loss function in the case of learning with an abnormal video can be defined as in Equation 3 above.

The abnormality detection unit inputs the segments generated by the segment generation unit to the neural network learned by the neural network learning unit to detect the abnormality in the segment unit (S740), and an abnormality detection result in the segment unit is generated (S750).

8 is an operation flowchart illustrating an example of neural network learning ( S730 ) shown in FIG. 7 according to an embodiment.

A method for the neural network learning unit to learn a neural network when an abnormal video image is received as an input will be described with reference to FIG. 8 .

First, the segment generated by the segment generator is input to the neural network (S810).

The abnormal cluster generating unit of the abnormal video learning unit generates two clusters based on the characteristics of the middle layer of the neural network for the segment ( S820 ). In this case, a cluster having a larger number of normal segments predicted by the neural network as a normal among the two clusters may be selected as a normal cluster, and the remaining clusters not selected as the normal cluster may be selected as an abnormal cluster. That is, among the two clusters, the cluster in which the segment predicted by the neural network is distributed more in each segment predicted by the neural network is selected as the normal cluster, and the remaining cluster is selected as the abnormal cluster.

The label generating unit of the abnormal video learning unit labels labels of all segments belonging to the normal cluster as normal, and labels labels of all segments belonging to the abnormal cluster as abnormal (S830). The labels generated in this way can be set as a set of correct answers as a result of prediction of the neural network.

The abnormal loss function 1 calculation unit of the abnormal video learning unit calculates the loss function 1 as in Equation 1 by comparing the final values of the neural network to which the segments are input with the prediction result set of the neural network (S840). In addition, the abnormal loss function 2 calculation unit of the abnormal video learning unit calculates the loss function 2 as in Equation 2 by comparing the normal cluster with the abnormal cluster (S850). And the ideal total loss function calculation unit of the ideal video learning unit calculates the total loss function as in Equation 3 by combining the loss function 1 and the loss function 2 (S860). And the loss function 2 may be inversely proportional to a distance between the center of the normal cluster and the center of the abnormal cluster.

9 is an operation flowchart illustrating another example of neural network learning ( S730 ) shown in FIG. 7 according to an embodiment.

A method for the neural network learning unit to learn a neural network when a normal video image is received as an input will be described with reference to FIG. 9 .

First, the segment generated by the segment generator is input to the neural network (S910).

The normal cluster generating unit of the normal video learning unit generates two clusters based on the characteristics of the middle layer of the neural network for the segment (S920). In this case, one of the two clusters may be selected as the normal 1 cluster and the other may be selected as the normal 2 cluster. In other words, since all segments are known to be normal, both clusters become normal clusters, and the set of correct answers for the prediction of the segment unit neural network is also set to normal. Therefore, neural network learning can be performed with a set of correct answers for prediction of the neural network in the segment unit.

The normal loss function 1 calculator of the normal video learning unit calculates the loss function 1 as in Equation 1 by comparing the final values of the neural network to which the segments are input with the set of correct answers indicating normal (S930). Also, the normal loss function 2 calculator of the normal video learning unit calculates the loss function 2 as in Equation 4 by comparing the normal 1 cluster and the normal 2 cluster (S940). And the stationary total loss function calculation unit of the normal video learning unit calculates the total loss function as in Equation 3 by synthesizing the loss function 1 and the loss function 2 (S950). In addition, the loss function 2 may be proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster.

10 is a diagram showing the configuration of a computer system according to an embodiment.

The apparatus for detecting anomalies using segmenting of a video image frame according to an embodiment may be implemented in the computer system 1000 such as a computer-readable recording medium.

Computer system 1000 may include one or more processors 1010 , memory 1030 , user interface input device 1040 , user interface output device 1050 , and storage 1060 that communicate with each other via bus 1020 . can Additionally, the computer system 1000 may further include a network interface 1070 coupled to the network 1080 . The processor 1010 may be a central processing unit or a semiconductor device that executes programs or processing instructions stored in the memory 1030 or storage 1060 . The memory 1030 and the storage 1060 may be storage media including at least one of a volatile medium, a non-volatile medium, a removable medium, a non-removable medium, a communication medium, and an information delivery medium. For example, the memory 1030 may include a ROM 1031 or a RAM 1032 .

According to the embodiment described above, by providing a method for detecting anomalies using segmenting of video image frames and an apparatus for the same, labeling information at the video level, that is, information indicating whether the video contains anomalies, is provided at the frame level. It is possible to provide an anomaly detection system capable of detecting anomalies in

In addition, it is possible to provide a method for learning an artificial intelligence neural network to be used for anomaly detection by using an abnormal video image or a normal video image.

Although embodiments of the present invention have been described above with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can practice the present invention in other specific forms without changing its technical spirit or essential features. You will understand that there is Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: anomaly detection device using segmentation of video image frames
110: video input image
120: segment generator
130: neural network learning unit
140: abnormality detection unit
150: Segment unit abnormality detection result

Claims (20)

generating segments by combining a selected number of frames from the video image input;
learning a neural network based on the characteristics of an intermediate layer of the neural network to which the segments are input; and
and inputting the segments into a learned neural network to detect anomalies in units of the segments. According to claim 1,
The step of learning the neural network is
An anomaly detection method using segmenting of a video image frame, comprising learning from an abnormal video image including at least one or more frames. 3. The method of claim 2,
The step of learning from the above video image is
generating two clusters based on a characteristic of an intermediate layer of the neural network for the segment;
selecting a cluster having a larger number of normal segments predicted to be normal by the neural network as a normal cluster among the two clusters; and
and selecting the remaining clusters that are not selected as the normal cluster among the two clusters as the abnormal cluster. 4. The method of claim 3,
The step of learning from the above video image is
labeling labels of all segments belonging to the normal cluster as normal;
labeling labels of all segments belonging to the abnormal cluster as abnormal; and
The method of detecting anomalies using segmenting of video image frames, further comprising the step of setting the labels as a set of prediction results of the neural network. 5. The method of claim 4,
The step of learning from the above video image is
calculating a loss function 1 by comparing final values of the neural network to which the segments are input with the set of prediction results of the neural network;
calculating a loss function 2 by comparing the normal cluster with the abnormal cluster; and
The method of detecting anomalies using segmenting of video image frames, further comprising calculating a total loss function by synthesizing the loss function 1 and the loss function 2. 6. The method of claim 5,
The loss function 2 is inversely proportional to a distance between the center of the normal cluster and the center of the anomaly cluster. According to claim 1,
The step of learning the neural network is
An anomaly detection method using segmenting of video image frames, comprising learning a normal video image that does not include any abnormal frames. 8. The method of claim 7,
The step of learning with the normal video image is
generating two clusters based on a characteristic of an intermediate layer of the neural network for the segment;
selecting one of the two clusters as a normal 1 cluster; and
and selecting the other one of the two clusters as the normal 2 cluster. 9. The method of claim 8,
The step of learning with the normal video image is
calculating a loss function 1 by comparing final values of the neural network to which the segments are input with a set of correct answers indicating normality;
calculating a loss function 2 by comparing the normal 1 cluster with the normal 2 cluster; and
An anomaly detection method using segmenting of a video image frame, further comprising calculating a total loss function by synthesizing the loss function 1 and the loss function 2. 10. The method of claim 9,
Wherein the loss function 2 is proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster. a segment generating unit generating segments by combining a selected number of frames from a video image input;
a neural network learning unit for learning the neural network based on the characteristics of the middle layer of the neural network to which the segments are input; and
and an anomaly detection unit configured to input the segments into a learned neural network and detect anomalies in units of the segments. 12. The method of claim 11,
The neural network learning unit
An anomaly detection apparatus using segmenting of video image frames, comprising: an anomaly video learning unit that learns from an abnormal video image including at least one or more frames. 13. The method of claim 12,
The above video learning unit
An abnormal cluster generating unit for generating two clusters based on the characteristics of the intermediate layer of the neural network for the segment,
The abnormal cluster generating unit
Among the two clusters, a cluster with a larger number of normal segments predicted as normal by the neural network is selected as a normal cluster,
An anomaly detection apparatus using segmenting of a video image frame, wherein the remaining clusters that are not selected as the normal cluster among the two clusters are selected as the abnormal cluster. 14. The method of claim 13,
The above video learning unit
Further comprising a labeling generator to generate a label for the segments,
The labeling generation unit
generating labels of all segments belonging to the normal cluster as normal;
Generates the labels of all segments belonging to the abnormal cluster as an abnormality,
An anomaly detection apparatus using segmenting of a video image frame, which sets the labels as a set of correct answers as a result of prediction of the neural network. 15. The method of claim 14,
The above video learning unit
an abnormal loss function 1 calculator for calculating a loss function 1 by comparing the final values of the neural network to which the segments are input with the prediction result set of the neural network;
an abnormal loss function 2 calculator for calculating a loss function 2 by comparing the normal cluster with the abnormal cluster; and
Anomaly detection apparatus using segmentation of a video image frame, further comprising an abnormal total loss function calculation unit for calculating a total loss function by synthesizing the loss function 1 and the loss function 2. 16. The method of claim 15,
The loss function 2 is inversely proportional to a distance between the center of the normal cluster and the center of the abnormal cluster. 12. The method of claim 11,
The neural network learning unit
An anomaly detection apparatus using segmenting of video image frames, comprising a normal video learning unit learning from a normal video image that does not include any abnormal frames. 18. The method of claim 17,
The normal video learning unit
And a normal cluster generator for generating two clusters based on the characteristics of the middle layer of the neural network for the segment,
The normal cluster generating unit
Selecting one of the two clusters as the normal 1 cluster,
An anomaly detection apparatus using segmenting of a video image frame, wherein the other one of the two clusters is selected as a normal 2 cluster. 19. The method of claim 18,
The normal video learning unit
a normal loss function 1 calculator for calculating a loss function 1 by comparing final values of the neural network to which the segments are input with a set of correct answers indicating normality;
a normal loss function 2 calculator for calculating a loss function 2 by comparing the normal 1 cluster with the normal 2 cluster; and
The apparatus for detecting anomalies using segmenting of video image frames, further comprising a stationary total loss function calculator for calculating a total loss function by synthesizing the loss function 1 and the loss function 2. 20. The method of claim 19,
Wherein the loss function 2 is proportional to a distance between the center of the normal 1 cluster and the center of the normal 2 cluster.

Images