KR102570126B1 - Method and System for Generating Video Synopsis Based on Abnormal Object Detection - Google Patents

Abstract

It can be applied to a multi-view video surveillance system composed of multiple cameras, and provides a method and device that can quickly detect dangerous situations even in various congested situations and generate video synopses in real time. The video synopsis generation method of the present invention receives a plurality of videos and extracts abnormal frames including a predetermined object from the plurality of videos to generate a video synopsis. extracting frames; generating a summary image by combining the abnormal frames; and combining summary images of each of the plurality of videos to generate the video synopsis. The extracting of the abnormal frames includes distinguishing blurry frames from the input image frame according to a predetermined criterion. Accordingly, the blurry frame is not included in the summary video and the video synopsis.

Description

Method and apparatus for generating video synopsis based on abnormal object detection {Method and System for Generating Video Synopsis Based on Abnormal Object Detection}

The present invention relates to a video surveillance system, and more particularly, to a method and apparatus for generating a synopsis from an image acquired by a video surveillance camera.

With the increase in video surveillance cameras, the amount of video data has increased excessively. Accordingly, it is difficult to analyze, extract, and store image data, and more data storage space and analysis personnel are required. A video synopsis is a short summary of a long video. It reduces the amount of video data and allows hours of video to be reviewed in just a few minutes, eliminating data storage problems and reducing the cost and cost of video analysis. It can greatly reduce the labor force.

However, since the existing video synopsis generation method deals with the synopsis of an image acquired by a single camera, it is not suitable for a video surveillance system composed of multiple cameras. In addition, since the existing video synopsis generation method is generated by processing not only the entire video frame but also includes all moving objects, the performance for real-time scenarios is low due to the heavy preprocessing burden, especially in congested situations due to the burden of processing meaningless object data. There is a problem that performance is significantly degraded in .

The present invention is to solve this problem, and it can be applied to a multi-view video surveillance system composed of several cameras, and a method and apparatus capable of generating a video synopsis in real time by quickly detecting a dangerous situation even in various congested situations. It is a technical task to provide

According to an aspect of the present invention for achieving the above technical problem, a method of generating a video synopsis by receiving a plurality of videos and extracting abnormal frames including a predetermined object from the plurality of videos is provided. The video synopsis generating method may include extracting the abnormal frames from input video frames for each of the plurality of videos; generating a summary image by combining the abnormal frames; and combining summary images of each of the plurality of videos to generate the video synopsis. The extracting of the abnormal frames includes distinguishing blurry frames from the input image frame according to a predetermined criterion. Accordingly, the blurry frame is not included in the summary video and the video synopsis.

The distinguishing of the blurry frames may include calculating a Laplacian variance by performing a convolutional operation on the input image frame with a predetermined kernel. In this case, the blurry frame may be identified by comparing the Laplacian variance with a predetermined threshold.

In the step of extracting the abnormal frames, the abnormal frames may be extracted by a predetermined artificial neural network model.

The artificial neural network model may be a YOLO model.

The extracting of the abnormal frames may include converting a color system of the input image frame into a CIELUV color space. In this case, the artificial neural network may extract the abnormal frame based on the color space converted image frame.

The extracting of the abnormal frames may include extracting the predetermined object from the input image frame and calculating an abnormal score indicating a probability that the predetermined object is actually an abnormal object; and determining a frame abnormality score for the input image frame based on the abnormality score for the object.

The summary image may be generated by excluding a frame determined as a normal frame based on the frame abnormality score and a frame divided into the blurry frame from among the input image frames. In this case, the generating of the summary image may include determining cross entropy of the input image frames: and determining frames to be included in the summary image according to the cross entropy.

The predetermined object may include at least one of a gun and a knife.

According to another aspect of the present invention, an apparatus for generating a video synopsis is provided that receives a plurality of videos and extracts abnormal frames including a predetermined object from the plurality of videos to generate a video synopsis. An image synopsis generating device includes a memory for storing program instructions; and a processor communicatively connected to the memory and executing the program instructions stored in the memory. The program instructions, when executed by the processor, cause the processor to: extract the abnormal frames from an input video frame for each of the plurality of videos; generating a summary image by combining the abnormal frames; and generating the video synopsis by combining summary images for each of the plurality of videos. The instructions for performing the operation of extracting the abnormal frames may include instructions for performing an operation of distinguishing blurry frames from the input image frame according to a predetermined criterion. Accordingly, the blurry frame is not included in the summary video and the video synopsis.

Instructions for performing the operation of distinguishing the blurry frames include instructions for performing an operation of calculating a Laplacian variance by performing a convolution operation on a predetermined kernel with respect to the input image frame; and comparing the Laplacian variance with a predetermined threshold. By doing so, the blurry frame can be distinguished.

Instructions for extracting the abnormal frames may extract the abnormal frames according to a predetermined artificial neural network model. The artificial neural network model may be a YOLO model.

Commands for extracting the abnormal frames may include commands for performing an operation of converting a color system of the input image frame into a CIELUV color space. In this case, the artificial neural network may extract the abnormal frame based on the color space converted image frame.

The instructions for performing the operation of extracting the abnormal frames may include extracting the predetermined object from the input image frame and calculating an abnormality score indicating a probability that the predetermined object is actually an abnormal object; and determining a frame abnormality score for the input image frame based on the abnormality score for the object.

The summary image may be generated by excluding a frame determined as a normal frame based on the frame abnormality score and a frame divided into the blurry frame from among the input image frames. In this case, the instructions for generating the summary image may include instructions for performing: determining cross entropy for the input image frames; and determining frames to be included in the summary image according to the cross entropy. .

The predetermined object may include at least one of a gun and a knife.

According to an embodiment of the present invention, it is possible to quickly detect a dangerous situation even in various congestion situations and generate an image synopsis in real time. In particular, the synopsis generation method and apparatus according to the present invention can be applied to a multi-view video surveillance system composed of multiple cameras.

1 is a schematic block diagram showing the overall configuration of a video surveillance system according to an embodiment of the present invention.
2 is a flowchart illustrating a process of generating an image synopsis according to an embodiment of the present invention.
3 is a block diagram of a frame selector module according to an embodiment of the present invention.
4 shows an example of colors displayed in the CIELUV color space.
5 shows that frame abnormality scores are visualized in color with respect to image frames input from a plurality of cameras.
6 shows a difference in frame sharpness according to Laplacian dispersion.
7 shows an example of pseudo-code for a process of generating an abnormal summary image.
8 shows an example of generating a video synopsis for a stored multi-view video.
9 shows an example of generating a video synopsis for a live camera image.
10 is a block diagram of an apparatus for generating a video synopsis according to an embodiment of the present invention.
11 is a table summarizing information on test images used for object detection model training in the experiment of the present invention.
12 is a table summarizing the average accuracy of the object detection model according to the present invention and other up-to-date detection models in an experiment for detecting guns and knives.
12 is a table summarizing quantitative evaluation indexes for detection accuracy of the object detection model according to the present invention and other models.
FIG. 13 is a graph showing the quantitative evaluation index summarized in FIG. 12 .
FIG. 14 is a graph showing a comparison result of accuracy in generating a summary image of the method for generating an image synopsis according to the present invention and other algorithms.
15 is a table arranged in comparison with the video synopsis generation method according to the present invention and the video synopsis generation results of other algorithms.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term “and/or” includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate overall understanding in the description of the present invention, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

1 is a schematic block diagram showing the overall configuration of a video surveillance system according to an embodiment of the present invention. The illustrated video surveillance system includes a plurality of surveillance cameras 10a to 10n that are distributed and installed in a surveillance target area to obtain surveillance images, and a monitoring and control server 20 to which the plurality of surveillance cameras 10a to 10n can access. provide In one embodiment, the monitoring cameras 10a to 10n are connected to the monitoring and control server 20 through an IP network based on a wired network and/or a wireless network. The monitoring cameras 10a to 10n may be connected to the monitoring and control server 20 through a USB cable as an example of a wired network.

Each of the monitoring cameras 10a to 10n may photograph the surrounding area at an appropriate zoom magnification while horizontally rotating (panning) and vertically rotating (tilting) in response to a control signal from the monitoring and control server 20 . The monitoring and control server 20 may detect a moving object or an abnormal object in the images from each monitoring camera 10a to 10n, and control pan, tilt, and zoom of the monitoring cameras 10a to 10n as necessary. In addition, the monitoring and control server 20 may construct an image synopsis by extracting image frames including abnormal objects from images obtained from the monitoring cameras 10a to 10n. In this specification, including the claims, an “abnormal object” refers to an object that may harm others or cause a dangerous situation, such as a gun, a knife, or a person holding a gun or knife. However, the type of abnormal object is not limited thereto. For example, in another embodiment of the present invention, an abnormal object is...

The monitoring and control server 20 may display the video synopsis to the operator through a display device, and may store it in a storage device. As will be described later, the monitoring and control server 20 includes at least one processor, and functions such as detecting an abnormal object, generating an image synopsis, displaying an image, and storing an image through the execution of program codes by the at least one processor. Do it.

The monitoring and control server 20 may be implemented, for example, in a cloud server. In most camera networks, the video feeds generated by each camera are stored on cloud servers, wasting resources and bandwidth. In order to solve this problem, in the present invention, the monitoring and control server 20 can be directly implemented in the edge/cloud computing device, and as a result, the entire original video obtained by each monitoring camera 10a to 10n is monitored. Instead of being stored in the control server 20, only the synopsis video may be stored.

2 is a flowchart illustrating a process of generating an image synopsis according to an embodiment of the present invention. The process of generating an image synopsis according to the present embodiment includes training an object detection model for detecting an abnormal situation (step 100), checking the connection of a plurality of cameras (step 110), and using the learned object detection model. and detecting and extracting an abnormal situation (step 120), generating a summary of abnormal frames (step 130), and generating an abnormal situation video synopsis for a plurality of cameras (step 140). .

As mentioned above, the present invention detects whether there is an abnormal object such as a gun or a knife in an image frame and determines whether the corresponding frame is in an abnormal situation. In an exemplary embodiment of the present invention, YOLOv3 is used as an object detection model for detecting abnormal objects, and is implemented based on Keras, an open source neural network library written in Python. That is, the object detection model according to the present embodiment can be referred to as 'Keras based-abnormal frame-extraction-Yolo model (KB-AF-E-YM) implemented based on Keras and extracting abnormal frame objects'. there is. However, the object detection model in the present invention is not limited to the YOLOv3 model, and other YOLO models such as YOLOv2 or YOLOv5 may be used, or models other than the YOLO model may be used.

Step 100 of FIG. 2 is an off-line pre-processing step, in which object detection model training is performed. Training of the object detection model may be performed to detect an abnormal object within a frame when a frame is input and output an abnormal score indicating a probability that the object is an actual abnormal object as accurately as possible.

Training a model requires a lot of computation on huge data sets. According to one embodiment, a larger custom data set was created using the ImageNet data set, the Open-Image data set, the Internet Movie Firearms Database (IMFD), and images provided by Gettyimages. In addition, a total of 432427 training image data was constructed by applying data transformation techniques such as scaling, flipping, and shearing to these images to enlarge the data. After adding annotations to each image, we obtained updated weights to reconstruct the object detection model by pre-training YOLOv3 weights using a Keras-based framework.

In step 110, the server generating the video synopsis, for example, the monitoring and control server 20 may check the connection state of the plurality of cameras 10a to 10n. At this time, the confirmation of the connection state may mean a new connection to the plurality of cameras 10a to 10n, or an operation to check the list of connected cameras, or the previously connected cameras 10a to 10n. It may also mean selecting some cameras from which to accept the original video to be used for generating the video synopsis. The cameras to be connected may be connected to the monitoring and control server 20 using a network such as a wired LAN or a wireless LAN, or may be connected through a USB cable. A serial number to be applied when generating an image synopsis may be assigned to each camera.

In step 120, an abnormal situation is detected and extracted using the object detection model learned in step 100. 3 is a block diagram of a frame selector module used for detecting and extracting anomalies. The frame selector module 122 includes a frame extractor 124 , a learned object detection model 124 (KB-AF-E-YM), and a frame selector 128 .

The frame extractor 124 may convert a color system of image frames input from the cameras 10a to 10n into a CIELUV color space. The frame extraction unit 124 extracts frames having a local luminance level different from the previous frame by more than a predetermined level by utilizing an absolute difference value of the CIELUV color space. That is, as a pre-processing task, the frame extractor 124 calculates an absolute difference value in the CIELUV color space from the previous frame in units of pixels or predetermined image blocks for each received input image frame, and calculates the local difference value of the absolute difference value. Alternatively, a frame having a large overall sum value is extracted.

4 shows an example of colors displayed in the CIELUV color space. In the CIELUV color space, luminance (L) can have values ranging from 0 to 100, and the u, v coordinates allow us to estimate the orientation of the green/red and blue/yellow axes, with values ranging from -100 to 100. have The CIELUV color space is a color system that can express luminance in more detail in consideration of human visual perception characteristics, and the color system conversion improves the feature quality of images to better detect abnormal objects. It is efficient to measure the minimum color contrast, which has many advantages when detecting small objects.

According to this embodiment, only frames whose variation is greater than a predetermined standard are input to the object detection model 126, so that the object detection model 126 detects an abnormal object only for frames in which the variation is large. However, in the modified embodiment, the frame extractor 124 may be omitted and all image frames may be supplied to the object detection model 126 . In addition, among the functions of the frame extraction unit 124, only one of color space conversion and frame extraction may be omitted.

The object detection model 126 receives an image frame from the frame extractor 124 and detects an abnormal object in the image frame. As mentioned above, detection of an abnormal object may be performed by a YOLO model named KB-AF-E-YM. The YOLO model extracts a feature map through a plurality of convolutional layers, determines a bounding box through a fully connected layer, and predicts a class score. That is, the YOLO model divides each image frame into SxS grids, and calculates object class scores to calculate whether abnormal objects are included in each grid. By merging neighboring grids and adjusting the position of the bounding box to increase the reliability of each grid, it is possible to increase the accuracy of object recognition and increase the reliability of class scores.

The object detection model 126 may output abnormal objects and object class scores. In one embodiment, information of an abnormal object may be represented by the location and size of a bounding box. The object class score may be an abnormal score indicating a probability that the corresponding object is an actual abnormal object. The abnormal score may be set to a value between 0 and 1 and added to the image frame or bounding box in the form of annotations. The abnormality score may mean that the closer to 1, the higher the probability of abnormality.

An anomaly score for a frame may be calculated based on an anomaly score for an object. For example, the highest abnormal score among abnormal scores of objects detected in an image frame may be determined as the abnormal score of the frame. Each frame can be divided into a normal frame and an abnormal frame according to the frame abnormality score α. That is, if α=0, there is no desired object in the frame currently being processed, and the corresponding frame is displayed as a normal frame (NF). If α=1, it means that the desired object exists in the frame being processed, and the corresponding frame is displayed as an abnormal frame (ANF). In this way, KB-AF-E-YM will select objects in the frame and annotate the frame. Annotated frame information is passed to the next step in the form of a matrix (MA) sequence. In this case, the annotation includes information about the abnormal object, frame abnormality score, and/or information about whether the frame is normal/abnormal.

5 shows that frame abnormality scores are visualized in color with respect to image frames input from a plurality of cameras. 5(a) shows a sequential frame series and an attribute, that is, a frame abnormality score, for each of 6 frames of images input from 8 cameras. In the figure, the purple part represents an abnormal frame and all other colors represent a normal frame. A downward arrow indicates an abnormal frame, and an upward arrow indicates a normal frame.

The frame selection unit 128 discards blurry frames from frames received from the object detection model 126 and selects only clear frames, thereby enabling a smooth synopsis to be generated only with frames of excellent quality. In one embodiment, the frame selection unit 128 calculates a Laplacian variance by applying a Laplacian kernel to an image frame, thereby distinguishing a blur frame from a sharp frame based on the Laplacian variance. . Equation 1 shows an example of a Laplacian kernel in a 3x3 matrix. The frame selector 128 may calculate the Laplacian variance by performing a convolution operation while relatively moving an image frame by sliding it horizontally and vertically in units of a predetermined movement interval (stride).

If the Laplacian variance is less than a certain threshold value, the corresponding frame is regarded as a blurry frame and may be discarded or marked as a comment. On the other hand, if the Laplacian variance is greater than a certain threshold value, the corresponding frame may be regarded as a clear frame. 6 shows a difference in frame sharpness according to Laplacian dispersion. According to the experiment of the inventors, if the Laplacian variance is in the range of (0, 159), the image quality of the frame image is the worst, and if the Laplacian variance is in the range of (160, 349), the image quality can be said to be in the average range. , when the Laplacian variance was 350 or more, the image quality of the frame image was excellent. Therefore, it is important to set a threshold value to select a frame image with not poor image quality while preventing the discarding of an abnormal frame, which is important information. Depending on the application field, in this embodiment, the threshold value of the Laplacian variance was set to 160 and used.

Referring back to FIG. 2 , in step 130, abnormal frames are collected from among image frames output from the frame selector module 122 to generate an abnormal summary video. Step 130 is separately performed for each of the plurality of input images input from the plurality of surveillance cameras 10a to 10n, and a summary of all camera images is generated.

In an embodiment, an abnormal summary image may be generated by selecting an abnormal frame based on a frame abnormality score, excluding frames whose image quality does not meet a standard, and then concatenating remaining frames. For example, an abnormal summary image may be configured only with frames having a frame abnormality score of 0.5 or more, excluding frames having a frame abnormality score of less than 0.5.

7 shows an example of pseudo-code for a process of generating an abnormal summary image. The object detection model 126, that is, KB-AF-E-YM, determines the abnormal score of the frame based on the abnormal scores of objects in the frame and the abnormal scores of the objects, and the attention model determines the background simultaneously, real-time synopsis generation is possible.

In step 140 of FIG. 2 , abnormal summaries of each camera image are stitched and connected in the order of sequence numbers assigned to each camera in advance. That is, as shown in FIGS. 8 and 9 , abnormal frames summarized for each camera image are sequentially connected. 8 shows an example of creating a video synopsis for a stored multi-view video, and FIG. 9 shows an example of creating a video synopsis for a live camera image.

While generating a summary image for each camera image in step 130 or generating a synopsis by stitching summary images of multiple camera images in step 140, jitter, ghosting, or shaking may occur in the image. there is. Considering this, it is preferable that stitching starts with a stitching pipeline, calculates feature points of pixels, and then goes through a process of registering and blending images.

10 is a block diagram of an apparatus for generating a video synopsis according to an embodiment of the present invention. An apparatus for generating a video synopsis according to an embodiment of the present invention may include at least one processor 220 , a memory 240 , and a storage device 260 . As mentioned above, the video synopsis generating device may be implemented by the monitoring and control server 20 in the video surveillance system shown in FIG. 1 . In this case, the monitoring and control server 20 receives monitoring images acquired by each monitoring camera 10a to 10n from the corresponding camera, detects an abnormal frame including an abnormal object in the received monitoring image, and generates an image synopsis.

Processor 220 may execute program instructions stored in memory 240 and/or storage device 260 . The processor 220 may be implemented by at least one central processing unit (CPU) or graphics processing unit (GPU), and other processors capable of performing the method according to the present invention. can be

The memory 240 may include, for example, volatile memory such as RAM (Random Access Memory) and non-volatile memory such as ROM (Read Only Memory). The memory 240 may load a program command stored in the storage device 260 and provide it to the processor 220 so that the processor 220 can execute it.

The storage device 260 is a recording medium suitable for storing program commands and data, for example, magnetic media such as hard disks, floppy disks and magnetic tapes, CD-ROMs (Compact Disk Read Only Memory), DVDs ( Optical media such as digital video disks, magneto-optical media such as floptical disks, flash memory or EPROM (Erasable Programmable ROM), or SSDs based on them It may include a semiconductor memory such as.

The storage device 260 stores the program instructions. In particular, the program command may include a video synopsis generating program for implementing the video synopsis generating method according to the present invention. The video synopsis generating program extracts the abnormal frames from input video frames for each of the plurality of videos; generating a summary image by combining the abnormal frames; and generating the video synopsis by combining summary videos for each of the plurality of videos. In particular, the instructions for performing the operation of extracting the abnormal frames include instructions for performing an operation of distinguishing blurry frames from the input image frames according to a predetermined criterion, so that the blurry frames are included in the summary image and the video synopsis. It may contain instructions that prevent it from being included. Such program commands may be executed by the processor 220 in a state in which they are loaded into the memory 240 under the control of the processor 220 to implement the method according to the present invention.

In order to evaluate the performance of the image synopsis generation method according to the present invention, an experiment was conducted, and the experimental results were compared with other recent studies. All ratings are based on an AMD Ryzen 5 3500X 6-core processor clocked at 3.59GHz, 16GB RAM, Nvidia Gigabyte Geforce RTX 2060 graphics card, Logitech C920 Pro H.D. Evaluated using a webcam.

TensorFlow was used to train the object detection model 124, that is, KB-AF-E-YM, and additional transformation was performed using Keras API. To train the KB-AF-E-YM model for object detection, images from ImageNet, IMFD, and OpenImage datasets were used.

To test the video synopsis performance, we used videos extracted from YouTube, IMFD, and live cameras. Among the six images, images 1, 5, and 6 were obtained from YouTube, and these images contained 2, 1, and 2 abnormal objects, respectively. Image No. 2 was acquired from IMFD, and there were 3 abnormal objects in this image. Images 3 and 4 were input from the live camera, and there was one abnormal object in each of these images. Detailed information on each image is summarized in the table of FIG. 11 .

12 summarizes the average accuracy in the experiment of detecting guns and knives of the object detection model according to the present invention and other latest detection models. In an experiment to detect two abnormal objects, a gun and a knife, the KB-AF-E-YM model according to the present invention reached 96.54% of average precision (mAP), which is superior to other state-of-the-art detection models. showed

In addition, the KB-AF-E-YM model according to the present invention is applied to three existing algorithms, that is, Gaussian Mixture Model (GMM) [R1], LOB-STERBGS [R2], and background subtraction method The [R3] algorithm was compared with Precision, Recall, F1, Similarity, and PCC. The comparison results are summarized in the table of FIG. 13 and shown in the graph of FIG. 14 . In all scales, it was confirmed that the KB-AF-E-YM model according to the present invention was overall excellent.

In FIGS. 13 and 14 , precision refers to the ratio of things that are actually true among things classified as true by the model. Recall is the ratio of what the model predicts to be true out of what is actually true. F1 is the weighted average of Precision and Recall. Similarity refers to the ratio of actual abnormal objects among the sum of all objects and predicted abnormal objects (i.e., TruePositive/(TruePositive+FalseNegative+FalsePositive)).

On the other hand, the accuracy of the summary image was also compared with other techniques. In general, video summarization results can vary depending on various environmental properties such as changes in lighting conditions and congestion scenarios. In this comparison, accuracy was compared based on the ability to discard unuseful information. And, the same evaluation scale was used for evaluation. A comprehensive comparison with other techniques is shown in FIG. 14 . Although the performance of the synopsis generation method of the present invention was not inferior to other techniques, there was no significant difference in precision because it only generated summaries for abnormal frames.

Finally, the method of the present invention and other algorithms were compared for the finally generated synopsis.

Considering real indoor and outdoor threat scenarios, a total of 6 online and offline videos, each with a length of more than 3500 frames, were selected. The method of the present invention was compared with algorithms such as Time Equidistant Algorithm (TEA) [T1], Constant Equidistant Algorithm (CEA) [T2], and SeDiM [T3]. 15 is a table arranged in comparison with the video synopsis generation method according to the present invention and the video synopsis generation results of other algorithms. As a result of the experiment, it was confirmed that CEA performed better than the TEA algorithm, and distortion was found in the resulting image synopsis in both of these algorithms compared to SeDiM. In addition, the method according to the present invention performed relatively better than the T1, T2 and T3 methods.

Although the existing algorithm used for contrast generates a synopsis for a single camera view, the method of the present invention generates a combined image synopsis for a multi-view camera, compared to the latest synopsis generation technique, the method of the present invention recalls , F1, and Similarity, it was confirmed that it had excellent performance, and the video summary time was also shorter than other methods, making it possible to analyze the video more quickly.

As mentioned above, devices and methods according to embodiments of the present invention can be implemented as program instructions or codes that can be read and executed by a computer while stored in a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. In addition, computer-readable recording media may be distributed to computer systems connected through a network to store and execute computer-readable programs or codes in a distributed manner.

The computer-readable recording medium may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. The program command may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine code generated by a compiler.

Although some aspects of the present invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, where a block or apparatus corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method may also be represented by a corresponding block or item or a corresponding feature of a device. Some or all of the method steps may be performed by or using a hardware device such as, for example, a microprocessor, programmable computer or electronic circuitry. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In embodiments, a field programmable gate array may operate in conjunction with a microprocessor to perform one of the methods described herein. Generally, methods are preferably performed by some hardware device.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

Claims (18)

A method of receiving a plurality of videos obtained through each of a plurality of cameras, extracting abnormal frames including a predetermined abnormal object from the plurality of videos, and generating an image synopsis including the abnormal frames, the method comprising:
extracting the abnormal frames from an input image frame for each of the plurality of videos, and generating a summary image including the extracted abnormal frames; and
generating the video synopsis in the form of one frame sequence including the abnormal frames included in the summary video for each of the plurality of videos;
Extracting the abnormal frames and generating the summary image
extracting the predetermined object from the input image frame and calculating an abnormality score representing a probability that the extracted object is actually the abnormal object;
determining a frame abnormality score for the input image frame based on the abnormality score for the extracted object; and
distinguishing blurry frames from the input image frame based on the frame abnormality score; thereby preventing frames determined as normal frames and frames classified as blurry frames from being included in the summary image and the image synopsis. How to create a synopsis. The method according to claim 1, wherein the step of classifying the blurry frame
and calculating a Laplacian variance by performing a convolution with a predetermined kernel on the input image frame; wherein the image synopsis generation method distinguishes the blurry frame by comparing the Laplacian variance with a predetermined threshold. The method of claim 1 , wherein in the step of extracting the abnormal frames, the abnormal frames are extracted according to a predetermined artificial neural network model. The method of claim 3 , wherein the artificial neural network model is a YOLO model. The method according to claim 3, wherein the step of extracting the abnormal frames
and converting the color system of the input image frame into a CIELUV color space, wherein the artificial neural network extracts the abnormal frame based on the color space-converted image frame. delete delete The method according to claim 1, wherein generating the summary image
Determining cross entropy for the input image frame; and
determining frames to be included in the summary image according to the cross entropy;
Image synopsis generation method further comprising a. The method of claim 1 , wherein the predetermined object includes at least one of a gun and a knife. An apparatus for receiving a plurality of videos acquired through each of a plurality of cameras, extracting abnormal frames including a predetermined abnormal object from the plurality of videos, and generating an image synopsis including the abnormal frames, the apparatus comprising:
a memory for storing program instructions; a processor communicatively connected to the memory and executing the program instructions stored in the memory;
The program instructions, when executed by the processor, cause the processor to:
extracting the abnormal frames from an input image frame for each of the plurality of videos, and generating a summary image including the extracted abnormal frames; and
An operation of generating the video synopsis in the form of one frame sequence including the abnormal frames included in the summary video for each of the plurality of videos;
Commands for extracting the abnormal frames and generating the summary image are
extracting the predetermined object from the input image frame and calculating an abnormality score representing a probability that the extracted object is actually the abnormal object;
determining a frame abnormality score for the input image frame based on the abnormality score for the extracted object; and
By including commands for performing an operation of classifying blurry frames based on the frame abnormality score in the input image frame, the frame determined as a normal frame and the frame divided into the blurry frame are included in the summary image and the image synopsis. A video synopsis generating device that prevents The method according to claim 10, wherein the instructions for performing the operation of distinguishing the blurry frame are
An image synopsis generating device that distinguishes the blurry frame by comparing the Laplacian variance with a predetermined threshold, including instructions for performing an operation of calculating a Laplacian variance by performing a convolutional operation on the input image frame with a predetermined kernel. The apparatus of claim 10 , wherein the commands for extracting the abnormal frames extract the abnormal frames according to a predetermined artificial neural network model. The apparatus of claim 12 , wherein the artificial neural network model is a YOLO model. The method according to claim 12, wherein the instructions for extracting the abnormal frames are
and converting the color system of the input image frame into a CIELUV color space; wherein the artificial neural network extracts the abnormal frame based on the color space-converted image frame. delete delete The method according to claim 10, the instructions for generating the summary image
Determining cross entropy for the input image frame; and
determining frames to be included in the summary image according to the cross entropy;
Image synopsis generating device further comprising instructions for performing. The apparatus of claim 10 , wherein the predetermined object includes at least one of a gun and a knife.

Images