KR102127276B1 - The System and Method for Panoramic Video Surveillance with Multiple High-Resolution Video Cameras - Google Patents

Abstract

According to one embodiment of the present invention, a panoramic video surveillance method executed in a video surveillance apparatus connected to multiple high-resolution video cameras comprises: a step of receiving high-resolution video camera videos from the multiple high-resolution video cameras; a step of generating and displaying a low-resolution panoramic video by using the high-resolution video camera videos based on a low-resolution panoramic video generation table; and a step of, when a specific position or a specific moving object is designated in the low-resolution panoramic video, generating and displaying a high-resolution digital PTZ video falling under a designated region by using an ultrahigh-resolution panoramic video generation table. Accordingly, the present invention is able to generate and use a low-resolution panoramic video instead of generating an ultrahigh-resolution panoramic video for displaying and analyzing panoramic videos, to generate and use only a video falling under the region of a subject of interest (a high-resolution digital PTZ video) instead of generating a whole ultrahigh-resolution panoramic video for displaying a high-resolution video of the subject of interest, and to build an efficient high-resolution panoramic video surveillance system.

Description

Panoramic Video Surveillance with Multiple High-Resolution Video Cameras

The present invention relates to a system for monitoring a panoramic image using a plurality of high-resolution cameras and a method thereof, and more specifically, to detect/track objects of interest from an image received from a plurality of high-resolution cameras, and to designate an event based on this. As a video surveillance device that detects and gives an alarm, it relates to a panoramic image surveillance system and method using a plurality of high-resolution cameras that perform robust and efficient image analysis using object image recognition Deep Convolutional Neural Network (DCNN).

Generally, a video surveillance system refers to a system that installs a camera at a specific location and monitors a specific location by displaying an image captured by a camera installed at a specific location in another location.

Among the video surveillance systems, the panoramic video-based surveillance system is a system that provides a wide-angle image of 180 degrees or 360 degrees, and has the advantage of being able to monitor a large area at once with a single image. In addition, in the case of automatically tracking moving objects from a video image for automating video surveillance, the panoramic image has an advantage of continuously tracking moving objects in a large area.

In general, as a method of acquiring a panoramic video image, a fisheye lens or a reflector is attached to one high resolution camera to obtain a wide angle image of 180 degrees/360 degrees, and images obtained from a plurality of general cameras arranged radially are pasted together. It can be divided into the way to obtain a panoramic image of.

Recently, a "single camera + fisheye lens" method that can acquire a panoramic image with a relatively simple configuration is mainly used in an image surveillance system. However, this method has a problem in that the overall resolution of a panoramic image is determined by the resolution of a single camera, and the image distortion increases as it approaches the edge of the lens. Therefore, this method is mainly used for short-range video surveillance.

On the other hand, the method of acquiring a panoramic image using a plurality of general cameras is more complicated than the above "single camera + fisheye lens" method, and the method of generating a panoramic image is complicated, but the "single camera + fisheye lens" method has a problem. Because it can be overcome, it can be used not only for short distance, but also for remote video surveillance.

In general, the higher the resolution of the camera image when the angle of view of the camera is fixed, the more detailed information can be obtained on a distant subject. Therefore, high-resolution cameras are mainly used in wide area/remote surveillance systems.

In a plurality of general camera-based panoramic image surveillance systems, if the image resolutions of the individual cameras are high, the panoramic images generated by pasting the images of the individual cameras have a very high image resolution. For example, if 5 general cameras are horizontally placed and the video resolution of each camera is Full-HD (1920x1080), the maximum resolution of the panoramic image obtained through this is about 8448x1080 (between neighboring camera images. Assuming the degree of overlap is 15%). In order to generate such ultra-high-resolution panoramic images in real-time (ie, at least 15 FPS or higher) on a general PC without a separate dedicated hardware device, real-time processing is very difficult in reality. Therefore, in the case of a panoramic video surveillance system based on a plurality of high-resolution cameras, it is difficult to operate in a conventional manner.

Meanwhile, in recent years, the number of CCTV cameras installed for security/safety in government offices and enterprises has exploded. However, compared to the number of installed CCTV cameras, the number of agents monitoring CCTV camera images is far from sufficient. In order to solve this problem, the introduction of an intelligent CCTV video surveillance system has been actively conducted. The CCTV video analysis device, which forms the core of the intelligent CCTV video surveillance system, receives video images from CCTV cameras, detects and tracks moving objects, and automatically detects and detects abnormal conditions such as “intrusion in the forbidden area” based on this. Causes The monitoring agent can effectively monitor multiple CCTV camera images by checking only the CCTV image where the alarm has occurred without having to constantly watch for multiple (insignificant) CCTV images.

However, since most of the existing CCTV image analysis devices use motion-based object detection algorithms, various causes (eg, branches swaying in the wind, sloshing) in addition to detection of actual objects of interest (eg, people and vehicles) The object detection is frequently caused by waves, moving shadows, sudden lighting changes, twinkling lights, snow/rain, etc. Through this, false alarms also occur frequently, making efficient monitoring impossible.

In addition to motion-based object detection technology, computer vision researchers have developed object detection technology based on object shape learning since the mid-2000s. In the above technique, an object shape feature is extracted and learned from various learning images of a specific type of object (for example, a pedestrian), and an object is detected by finding a region having a shape characteristic similar to that of the learned object in the image. To perform. Typically, there are object detection technologies such as Viola-Jones, HOG, ICF, ACF, and DPM. However, due to limitations in detection performance and processing load of these object detection technologies, it was difficult to apply them to commercial CCTV image analysis devices.

In the midst of this, as Professor G. Hinton of the University of Toronto, Canada, in 2012, used a DCNN called AlexNet to win the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) with an overwhelming performance difference from existing image recognition algorithms. In the field of vision, deep learning technology has started to receive attention, and since then, attempts have been made to solve various problems of computer vision using deep learning technology.

From 2014, DCNN-based object detection technologies began to be announced. These DCNN-based object detection technologies provide detection capabilities far exceeding those of existing object detection technologies. Typically, there are object detection technologies such as Fast/Faster R-CNN, RFCN, SSD, and YOLO.

However, there are still several limitations in applying DCNN-based object detection technology to commercial CCTV image analysis devices. A typical limitation is that the hardware cost for real-time video processing using a DCNN-based object detector is very high. Since a typical DCNN-based object detector requires a considerable amount of computation to detect an object from a single video frame, the general CPU processes the video in real time using a DCNN-based object detector (eg, typically 7 per second). It is very difficult to perform object detection of frame anomalies). Therefore, in order to process video in real time using a DCNN-based object detector, a GPU capable of large-scale parallel computation is essential. In addition, even if a GPU is used, it is difficult to simultaneously process multiple video streams simultaneously in a single image analysis device unless an expensive GPU with high performance is used.

Korean Patent Publication No. 10-2006-0044129 (2000.05.16.) Korean Patent Publication No. 10-2005-0045845 (2005.05.17.) Korean Patent Publication No. 10-2005-0046822 (2005.05.18.) Korean Patent Publication No. 10-2009-0121102 (2009.10.26.) Korean Patent Publication No. 10-2018-0072561 (2018.06.29.) Korean Patent Publication No. 10-2018-0107930 (2018.10.04.) Korean Registered Patent Publication No. 10-1850286 (2018.04.13.) Korean Registered Patent Publication No. 10-1040049 (2011.06.02.) Korean Registered Patent Publication No. 10-1173853 (2012.08.08.) Korean Registered Patent Publication No. 10-1178539 (2012.08.24.) Korean Registered Patent Publication No. 10-1748121 (2017.06.12.) Korean Registered Patent Publication No. 10-1789690 (October 18, 2017) Korean Registered Patent Publication No. 10-1808587 (2017.12.07.)

“Pedestrian Detection: An Evaluation of the State of the Art” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 34, Issue 4, April 2012 “Efficient Processing of Deep Neural Networks: A Tutorial and Survey”, Proceedings of the IEEE, Vol. 105, Issue 12, Dec. 2017 “Object Detection with Deep Learning: A Review” arXiv.org, arXiv: 1807.05511 [cs.CV], Jul. 2018

In an embodiment of the present invention, in a panoramic image surveillance system using a plurality of high-resolution cameras, instead of generating an ultra-high resolution panoramic image for generating and analyzing a panoramic image, a low-resolution panoramic image is generated and used. In order to display the image, instead of generating the entire ultra-high resolution panoramic image, only the image (high resolution digital PTZ image) corresponding to the region of interest is generated and used, and an image surveillance method to implement an efficient high-resolution panoramic image surveillance system and execution thereof An object of the present invention is to provide a device.

In addition, an embodiment of the present invention can detect/track objects of interest from images received from a plurality of high-resolution cameras, and use DCNN for object image recognition as an image surveillance device that detects and alerts designated events based on this. The objective is to provide a panoramic image surveillance system and method using a plurality of high-resolution cameras that perform robust and efficient image analysis.

The problem to be solved by the present invention is not limited to the problem(s) mentioned above, and another problem(s) not mentioned will be clearly understood by those skilled in the art from the following description.

Among embodiments, an image surveillance method executed in an image surveillance apparatus connected to a plurality of high resolution cameras arranged in a radial form and partially overlapping each other includes receiving a high resolution camera image from the plurality of high resolution cameras, and a low resolution panoramic image Generating and displaying a low-resolution panoramic image using the high-resolution camera image based on a generation table. When a specific location or a specific moving object is specified in the low-resolution panoramic image, a high-resolution panoramic image generation table is used to designate a high-resolution image corresponding to a designated area. And generating and displaying a digital PTZ image.

Among embodiments, an image surveillance device arranged in a radial form and connected to a plurality of high-resolution cameras in which a part of the images are overlapped with each other is based on an image input unit that receives high-resolution camera images from the plurality of high-resolution cameras and a low-resolution panoramic image generation table. A low-resolution panoramic image generator that generates a low-resolution panoramic image using the high-resolution camera image, and when a specific location or a specific moving object is specified in the low-resolution panoramic image, a high-resolution digital image corresponding to a designated area using an ultra-high-resolution panoramic image generation table It includes a high-resolution digital PTZ image generation unit for generating a PTZ image.

In addition, the panoramic image surveillance system using a plurality of high-resolution cameras according to an embodiment of the present invention includes a plurality of high-resolution cameras performing image analysis using a deep convolutional neural network (DCNN) that provides an object recognition result in an image. A panoramic video surveillance system, the first video data consisting of a plurality of first video frames having a first pixel format, a first resolution, and a first frame rate are received from the high-resolution camera, and the received The first resolution, the first frame rate, and the first pixel format of the first image data are respectively a second resolution lower than the first resolution, a second frame rate lower than the first frame rate, and the first pixel format, respectively. An image conversion unit for converting second image data in a second pixel format having a different format from (pixel format), and a plurality of moving objects based on motion in a video frame of the converted second image data Performs detection and tracking, extracts images of the plurality of objects being tracked, distinguishes the plurality of tracking objects by using the recognition result obtained by inputting the extracted object images to the DCNN, and the plurality of the separated objects Motion that detects an event based on tracking information and a specified rule of a user-interest object obtained by removing the identified user non-interest object from the target of the plurality of tracking objects by checking a user non-interest object that is out of a specified criterion from the tracking object It includes a base image analysis unit.

The image converting unit, a video frame acquiring unit receiving the plurality of video frames input according to the first frame rate, sampling a smaller number of video frames than the input plurality of first video frames, and a plurality of second videos A video frame subsampling unit for generating a frame, a video frame scaling unit for converting the generated plurality of second video frames to the second resolution, and the first of the plurality of second video frames converted to the second resolution. And a pixel format converter configured to provide the second image data generated by converting one pixel format to the second pixel format to the motion-based image analyzer.

The motion-based image analysis unit generates a difference image by using the second image data and the learned background image, and removes noise from the generated difference image to generate a motion region for the plurality of moving objects. A motion area detection unit to detect, an object tracking unit to detect and track the plurality of moving objects using the detected motion area and the second image data, and the plurality of extracted video frames from one of the plurality of second video frames The first recognition result obtained by applying the object image of the tracking object to the DCNN and the plurality of extracted from at least one other video frame input at a predetermined time interval from the one video frame among the plurality of second video frames Based on the second recognition result obtained by applying the object image of the tracking object to the DCNN, a tracking object classification unit for classifying the plurality of tracking objects, and checking a user's uninteresting object that deviates from a specified criterion among the divided tracking objects The uninterested tracking object removal unit for removing the identified user uninterested object from the object of the plurality of tracking objects in the object tracking unit, and an event in which tracking information of the user interest object obtained according to the removal satisfies a specified rule. It may include an event detection unit based on a tracking object of interest to detect.

The tracking object classification unit may calculate and accumulate the first recognition result and the second recognition result as scores, respectively, and determine a class having the highest accumulated score as a type of the tracking object.

The non-interest tracking object removal unit may identify a user non-interest tracking object that exceeds the designated criterion in the determined type of object and remove it from the target of the plurality of tracking objects.

The image converting unit may include the first pixel format having a pixel format of a luminance signal (Y), a difference (U) of a red signal, and a difference (V) of a luminance signal and a blue component (Red-Green-Blue) or gray. The second pixel format may be converted to a gray-scale pixel format.

In addition, the method for monitoring a panoramic image using a plurality of high-resolution cameras according to an embodiment of the present invention includes a panoramic image monitoring system using a plurality of high-resolution cameras that perform image analysis using DCNN, which provides an object recognition result in an image. As an image monitoring method, the image conversion unit constituting the panoramic image monitoring system receives first image data composed of a plurality of first video frames having a first pixel format, a first resolution, and a first frame rate from the high-resolution camera. And a second resolution lower than the first resolution and a second frame rate lower than the first frame rate, respectively, for receiving the first resolution, the first frame rate, and the first pixel format of the received first image data. And converting second image data of a second type of heterogeneous pixel having a format different from that of the first pixel format, and a motion-based image analysis unit constituting the panoramic image surveillance system. 2 Performs detection and tracking of a plurality of moving objects based on motion in a video frame of video data, extracts images of the plurality of tracking objects, and uses the recognition results obtained by inputting the extracted object images to the DCNN The user interest obtained by classifying the plurality of tracking objects and checking the user non-interesting objects that deviate from the specified criteria from the plurality of tracking objects to remove the checked user non-interesting objects from the target of the plurality of tracking objects And detecting an event based on object tracking information and a specified rule.

The image conversion unit includes a video frame acquisition unit, a video frame subsampling unit, a video frame scaling unit, and a pixel format conversion unit, and converting the second image data into the video frame acquisition unit includes: the first frame Receiving the plurality of video frames input according to a rate, and generating a plurality of second video frames by sampling fewer video frames than the plurality of input first video frames in the video frame subsampling unit Step, in the video frame scaling unit, converting the generated plurality of second video frames to the second resolution, and the pixel format conversion unit of the plurality of second video frames converted to the second resolution The method may include providing the second image data generated by converting the first pixel format to the second pixel format to the motion-based image analysis unit.

The motion-based image analysis unit includes a motion area detection unit, an object tracking unit, a tracking object classification unit, an uninterested tracking object removal unit, and an interest tracking object-based event detection unit, and detecting the event comprises: Generating a difference image using the second image data and the learned background image, and removing noise from the generated difference image to detect motion regions for the plurality of moving objects, in the object tracking unit, Detecting and tracking the plurality of moving objects using the detected motion region and the second image data, and in the tracking object classification unit, the plurality of tracking extracted from one video frame among the plurality of second video frames The first recognition result obtained by applying the object image of the object to the DCNN and the plurality of tracking objects extracted from at least one other video frame input at a predetermined time interval from the one video frame among the plurality of second video frames Classifying the plurality of tracking objects based on the second recognition result obtained by applying the object image of the DCNN, and in the non-interest tracking object removing unit, a user ratio that deviates from a specified criterion among the divided tracking objects. Checking the object of interest and removing the identified user uninterested object from the object of the plurality of tracking objects in the object tracking unit, and tracking information of the user interest object obtained by the removal in the interest tracking object-based event detection unit It may include the step of detecting an event that satisfies the specified rule.

The step of distinguishing the plurality of tracking objects may calculate and accumulate the first recognition result and the second recognition result as scores, respectively, to determine the type having the highest accumulated score as the type of the tracking object.

In the step of removing from the targets of the plurality of tracking objects, a user uninterested tracking object that exceeds the specified criterion in the determined type of object may be identified and removed from the targets of the plurality of tracking objects.

In the converting of the second image data, the first pixel format having a pixel format of a luminance signal (Y), a difference (U) of a red signal, and a difference (V) of a luminance signal and a blue component is RGB or grayscale. The pixel format may be converted to the second pixel format.

Details of other embodiments are included in the detailed description and accompanying drawings.

Advantages and/or features of the present invention and methods for achieving them will become apparent by referring to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains. It is provided to completely inform the person having the scope of the invention, and the present invention is only defined by the scope of the claims. The same reference numerals refer to the same components throughout the specification.

According to an embodiment of the present invention, a low-resolution panoramic image is generated from a high-resolution image received from a plurality of high-resolution image photographing devices arranged in a radial form to display a panoramic image, and when an event occurs through real-time image analysis, input to a corresponding location By providing a digital PTZ image using the old high resolution image, there is an effect of constructing a high resolution panoramic image surveillance system using less CPU resources.

In addition, according to an embodiment of the present invention, the problem of the CCTV image analysis apparatus using the existing motion-based object detection algorithm is improved by using DCNN-based image recognition technology, but DCNN-based image analysis is performed even in a device with a general CPU. By making this possible, it would be possible to save the cost of purchasing expensive equipment.

Furthermore, by using DCNN-based image recognition technology to which deep learning is applied, the accuracy of object detection is increased, so that an alarm is sent according to event detection.

1 is an overall configuration diagram for explaining a panoramic video surveillance system according to an embodiment of the present invention,
2 is a block diagram for explaining a panoramic image sensor value according to an embodiment of the present invention,
3 is a flowchart for explaining an embodiment of a method for monitoring a panoramic image according to the present invention,
4 is a flowchart for explaining another embodiment of a method for monitoring a panoramic image according to the present invention,
5 is a reference diagram for explaining a process of generating a low-resolution panoramic image by the panoramic image surveillance apparatus according to an embodiment of the present invention,
6 is a reference diagram for explaining a process of generating a high-resolution digital PTZ image by the panoramic image surveillance apparatus according to an embodiment of the present invention,
7 is a block diagram of an object image recognition DCNN based CCTV image analysis apparatus according to another embodiment of the present invention,
8 is an exemplary view showing a processing process and results of a motion region detection unit according to an embodiment of the present invention,
9 is an exemplary view showing the processing process and results of the object tracking unit according to an embodiment of the present invention,
10 is a view for explaining a processing process of a tracking object classification unit according to an embodiment of the present invention;
11 is an exemplary diagram of sample images for learning an object image recognition DCNN classifying a given image into one of people, vehicles, and unidentified classes;
12 is an exemplary view of classifying tracking objects through a tracking object classification unit according to an embodiment of the present invention;
13 is an exemplary view showing a processing result of a non-interest tracking object removal unit according to an embodiment of the present invention,
14 is an exemplary view showing a result of processing an interest tracking object-based event detection unit according to an embodiment of the present invention;
15 is a block diagram of an object image recognition DCNN-based CCTV image analysis device for simultaneously real-time analysis of N CCTV camera images according to an embodiment of the present invention;
16 is a block diagram of an object image recognition DCNN-based CCTV video analysis system for performing video analysis on a large scale in a CCTV control center, according to an embodiment of the present invention;
17 is a block diagram showing the structure of an image analysis apparatus according to another embodiment of the present invention, and
18 is a flowchart illustrating a driving process of an image analysis apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is an overall configuration diagram for explaining a panoramic image surveillance system according to an embodiment of the present invention.

Referring to FIG. 1, a high-resolution panoramic image surveillance system includes a plurality of image photographing devices (or a plurality of high-resolution cameras) 100 and an image surveillance device 200 arranged in a radial form.

The plurality of image photographing devices 100 arranged in a radial form are installed so that a part of the photographing area overlaps with each other and photographs surrounding images to provide a high-resolution image to the image monitoring device 200.

Here, the plurality of image photographing devices 100 are fixed in the form of a radiation and mean a device capable of acquiring a high-resolution image by photographing a corresponding area, for example, a high-resolution megapixel camera.

When a high-resolution image is received from a plurality of image photographing devices 100, the image surveillance apparatus 200 generates and displays a low-resolution panoramic image based on the low-resolution panoramic image generation table.

The image surveillance device 200 generates and displays a low-resolution panoramic image, performs object detection/tracking and event detection through real-time image analysis, and when a specified event is detected (for example, “intrusion” event is a designated panoramic image) This means that a moving object appears in the area), and generates and displays a high-resolution digital PTZ image corresponding to the object or event occurrence location.

As described above, the present invention generates and displays a low-resolution panoramic image by using a high-resolution image received from a plurality of imaging devices arranged in a radial form, and when an event occurs through real-time image analysis, displays a high-resolution digital PTZ image for the corresponding location. By providing it, there is an effect that an ultra-high resolution panoramic video surveillance system can be efficiently constructed and operated using less CPU resources. Hereinafter, the image sensing value will be described in more detail with reference to FIG. 2.

2 is a block diagram illustrating an image sensing value according to an embodiment of the present invention.

Referring to FIG. 2, the image surveillance device 200 includes an image input unit 210, an image decoding unit 211, an image storage unit 212, an image search/play unit 213, and a low-resolution panoramic image generation unit 214. , Low-resolution panoramic image generation table 215, object tracking unit 216, manual digital PTZ control unit 217, automatic digital PTZ control unit 218, high-resolution digital PTZ image generation unit 219, ultra-high resolution panoramic image generation table ( 220) and an image display unit 221.

The image input unit 210 is arranged in a radial form and receives a compressed high-resolution image from a plurality of image photographing devices 100 in which a part of the photographing area overlaps with each other, so that the image decoding unit 211 and the image storage unit 212 are respectively To provide.

The image decoding unit 211 receives a high resolution image from the image input unit 210 to perform decoding, and provides the decoded image to the low-resolution panoramic image generation unit 214 or the high-resolution digital PTZ image generation unit 219.

The image storage unit 212 receives the image stream from the image input unit 210 and stores it as it is.

When a search for an image corresponding to a specific condition is requested, the image search/playback unit 213 reads the image data corresponding to the specific condition from the image storage unit 212 and provides it to the image decoding unit 211. Here, a specific condition is based on event detection information (for example, event detection such as roaming, virtual fence, reverse movement, etc.) and object detection information (object information such as a specific color, size, etc.) to present or search a search condition. It includes at least one of the conditions based on the desired date.

The low-resolution panoramic image generation unit 214 generates a low-resolution panoramic image using the low-resolution panoramic image generation table 215 having the low-resolution panoramic image generation information. The low-resolution panoramic image generation unit 214 provides the low-resolution panoramic image to the object tracking unit 216 and the manual digital PTZ control unit 217, respectively, and to the image display unit 221 for image display.

More specifically, the low-resolution panoramic image generation unit 214 generates a low-resolution panoramic image using image warping and blending information stored in the low-resolution panoramic image generation table 215.

The object tracking unit 216 receives the low-resolution panoramic image from the low-resolution panoramic image generating unit 214 and performs object detection and event detection through tracking.

The object tracking unit 216 provides the location and size information of each frame to the automatic digital PTZ control unit 218. Here, the method of detecting and tracking the object is omitted because it is outside the scope of the present invention. However, the related contents of FIGS. 7 to 18 described later may be applied. The manual digital PTZ control unit 217 converts location coordinate information designated by the user from the low-resolution panoramic image into location coordinates of the high-resolution panoramic image and provides it to the high-resolution digital PTZ image generation unit 219.

The automatic digital PTZ control unit 218 converts the sensed location coordinate information from the low-resolution panoramic image detected by the object tracking unit 216 to the location coordinate of the high-resolution panoramic image and provides it to the high-resolution digital PTZ image generation unit 219.

The high-resolution digital PTZ image generation unit 219 generates a high-resolution digital PTZ image corresponding to the location coordinates of the received high-resolution panoramic image and provides it to the image display unit 221. At this time, the high-resolution digital PTZ image generation unit 219 generates a high-resolution digital PTZ image corresponding to the corresponding location coordinates by using only a part of the ultra-high resolution panoramic image generation table 220 without generating a full high-resolution panoramic image.

The image display unit 221 displays an image received from the low-resolution panoramic image generation unit 214 or the high-resolution digital PTZ image generation unit 219.

3 is a flowchart illustrating an embodiment of an image monitoring method according to the present invention.

Referring to FIG. 3, the image surveillance apparatus 200 receives a high-resolution image through the image input unit 210 from a plurality of high-resolution cameras that are disposed in a radial form and partially overlapped with each other in a photographing area (step S310). The received high-resolution image is usually encoded (compressed) through a codec such as H.264.

The image surveillance device 200 stores the high resolution image received for each camera through the image storage unit 212 (step S320). At this time, the type of image data stored is the type of encoded image data received from the camera.

The video surveillance device 200 decodes the video data received for each camera (step S330).

The video surveillance device 200 generates a low-resolution panoramic image through the low-resolution panoramic image generation unit 214 from the decoded image data for each camera, and displays the panoramic image through the image display unit 221. At this time, the low-resolution panoramic image generation unit 214 generates low-resolution panoramic images from high-resolution camera images using image warping and blending information stored in the low-resolution panoramic image generation table 215.

When a specific location or a specific moving object is designated in the low-resolution panoramic image (step S350), the image monitoring device 200 may specify a specified location or a location of the specified moving object through the manual digital PTZ control unit 217 or the automatic digital PTZ control unit 218. Area information of the high-resolution digital PTZ image corresponding to is obtained (step S360).

As an embodiment of step S350, the user may designate a location corresponding to a region of interest in the low-resolution panoramic image being displayed.

In another embodiment of step S350, the user may designate an object of interest among the moving objects being tracked through the object tracking unit 216.

As another embodiment of step S350, the location of occurrence of the detected event or the object that generated the detected event through the object tracking unit 216 may be automatically designated as the location of interest or the object of interest.

The image surveillance device 200 generates a high-resolution digital PTZ image through the high-resolution digital PTZ image generation unit 219 using the region information of the high-resolution digital PTZ image obtained in step S360, and displays the image through the image display unit 221. Express on. At this time, the high-resolution digital PTZ image generation unit 219 uses the image warping and blending information stored in the ultra-high resolution panoramic image generation table 220 to obtain high-resolution digital PTZ images from high-resolution camera images obtained from the image decoding unit 211. Produces

4 is a flowchart for explaining an embodiment of an image monitoring method according to the present invention.

Referring to FIG. 4, the user requests reproduction of an image recorded by the image monitoring device 200 (step S410). The playback start position of the recorded video may be specified in various ways. For example, when a user directly designates a time point corresponding to a playback start position or when a user selects a specific event among previously generated events, it may be designated as a start time point of the selected event.

The video surveillance device 200 obtains recorded video data for each camera recorded in the video storage unit 212 through the video search/playback unit 213 (step S420).

The video surveillance device 200 decodes the recorded video data for each camera through the video decoding unit 211 (step S430).

The video surveillance device 200 generates a low-resolution panoramic image from the decoded image data for each camera through the low-resolution panoramic image generation unit 214, and displays it on the screen through the image display unit 221 (step S440).

In order to view a high-resolution image of a region of interest or a moving object of interest, the user designates a corresponding location of the region of interest or a moving object of interest in a low-resolution panoramic image being displayed (step S450).

When the user designates a corresponding location of the region of interest, the image surveillance device 200 obtains region information of the corresponding high-resolution digital PTZ image through the manual digital PTZ control unit 217, and when the user designates a moving object of interest, automatic digital Through the PTZ control unit 218, area information of the corresponding high-resolution digital PTZ image is obtained (step S460).

The image surveillance device 200 generates a high-resolution digital PTZ image through the high-resolution digital PTZ image generation unit 219 from the decoded image data for each camera acquired in step S430 and the region information of the high-resolution digital PTZ image obtained in step S460. Then, it is displayed on the screen through the image display unit 221.

5 is a view for explaining a process of generating a low-resolution panoramic image by the image surveillance apparatus 200 according to an embodiment of the present invention.

2 and 5, from the high-resolution images of each camera acquired at the same time, the low-resolution panoramic image generation unit 214 of the image surveillance apparatus 200 stores images stored in the low-resolution panoramic image generation table 215 Using the warping and blending information, a panorama image of the same size (resolution) as the low resolution panorama image generation table is generated.

More specifically, the brightness value (R/G/B value in the case of a color image) of a specific pixel of a panoramic image is obtained as a weighted sum of pixel brightness values of camera images corresponding thereto. For example, suppose that among the N camera images obtained from N cameras, camera images having pixels corresponding to a specific pixel coordinate (x,y) of the panoramic image Ip are I1 and I2 (see FIG. 5). Then, the pixel brightness value Ip(x,y) at the pixel coordinates (x,y) of the panoramic image Ip is obtained by the following [Equation 1].

In [Equation 1], a(x,y) is a blending coefficient value in pixel coordinates (x,y) of a panoramic image, and has a value between 0 and 1. The pixel coordinates (x ₁ ,y ₁ ) and (x ₂ ,y ₂ ) refer to the pixel coordinate values in the camera images I ₁ and I ₂ corresponding to the pixel coordinates (x,y) of the panoramic image. That is, when generating the panoramic image Ip, the pixel coordinates (x ₁ ,y ₁ ) and (x ₂ ,y ₂ ) of the camera images I ₁ and I ₂ are warped to the pixel coordinates (x,y) of the panoramic image Ip. Means

The panoramic image generation table stores pixel coordinates and blending coefficient values of a camera image corresponding to each pixel of the panoramic image to be generated. Therefore, if the N camera images acquired at the same time from the N cameras are given as input, it is possible to easily calculate all or a part of the panoramic images using the panoramic image generation table. The panoramic image generation table can be obtained by a camera calibration technique, and a detailed method is out of the scope of the present patent, so the description is omitted.

The size of the low-resolution panoramic image generation table is determined by considering the resolution of the panoramic image for outputting to a screen or tracking an object. For example, in the case of using five Full-HD level (1920x1080) cameras horizontally arranged, the resolution of the panoramic image obtained through this is up to 8448x1080 (assuming that the overlap between the neighboring camera images is 15%). However, the resolution of a typical display device (monitor) is about 1920x1080, far below this. Therefore, when a panoramic image is displayed on the screen in one line, the resolution of the panoramic image for display is sufficient if it is 1920x246. Therefore, in the above example, a low-resolution panoramic image generation table having a size of 1920x246 may be provided.

6 is a view for explaining a process of generating a high-resolution digital PTZ image by the video surveillance apparatus 200 according to an embodiment of the present invention.

2 and 6, from a high-resolution image of each camera acquired at the same time, the high-resolution digital PTZ image generation unit 219 of the image surveillance device 200 is a manual digital PTZ control unit 217 or an automatic digital PTZ control unit. A high resolution digital PTZ image is generated by using the region warping and blending information stored in the ultra-high resolution panoramic image generation table 220 obtained from (218).

The structure of the ultra-high resolution panoramic image generation table 220 is the same as that of the low-resolution panoramic image generation table 215, and the size of the table follows the maximum resolution of the panoramic image that can be generated through the original camera images.

The high-resolution digital PTZ image is generated by performing image warping and blending only within a corresponding area of the ultra-high resolution panoramic image generation table, and the size (resolution) of the digital PTZ image can be arbitrarily designated by the user.

7 is a block diagram of an object image recognition DCNN-based CCTV image analysis apparatus according to another embodiment of the present invention.

The image analysis apparatus 690 of FIG. 7 may be applied to a panoramic image surveillance system using a plurality of high-resolution cameras as in FIG. 1, and thus may be used by replacing the image surveillance apparatus 200 of FIG. 1 or merging some components. Will be able to. As an example, the image analysis device 690 of FIG. 7 may operate in conjunction with a plurality of high-resolution cameras 100_1 to 100_N, as shown in FIG. 1, and for this purpose, a part of the configuration or operation of the corresponding video surveillance device 200 or You will be able to include everything. However, for convenience of description, hereinafter, it will be described in detail only on the assumption that the images of the high-resolution cameras 100_1 to 100_N can be used.

As shown in FIG. 7, the object image recognition DCNN-based CCTV image analysis apparatus (or image analysis apparatus, image surveillance apparatus) 690 according to an embodiment of the present invention includes an image conversion unit (or image acquisition unit) 721 ), a part or all of the motion-based image analysis unit 722 and the object image recognition DCNN (unit) 708.

Here, "including some or all" means that some components, such as object image recognition DCNN 708, are omitted, such that the CCTV image analysis apparatus 690 is configured, or some components such as a motion-based image analysis unit 722 A means that can be configured to be integrated with other components, such as the image conversion unit 721, it will be described as including all in order to fully understand the invention.

The image conversion unit 721 acquires image data to be used by the motion-based image analysis unit 722. For example, the video converter 721 may acquire video data provided by a CCTV or a plurality of high-resolution cameras or video data provided by a server, such as a government office, through a communication network of a communication company. The motion-based image analysis unit 722 receives image data provided by the image conversion unit 721 and performs motion and DCNN-based image analysis. Here, "image data" may mean an image signal, but may also mean video data. Typically, the video signal includes a video signal and an audio signal, and may further include additional information (eg, encoding information, subtitle information, etc.), wherein the video signal represents video data. However, in the image processing, since video signal processing is mainly dealt with, it is also likely that those skilled in the art mix video data with video data. Therefore, the embodiment of the present invention will not be particularly limited to the concept of the above terms. In other words, it is also possible to use video data and video data in almost the same concept.

However, the image data includes a series of unit frame images. In other words, it consists of a series of video frames. For example, if the frame rate is 60 frames per second (FPS), it means that 60 still images per second, that is, unit frame images are transmitted and received. Therefore, although a video frame, a frame video, or a unit frame is used in a nearly similar concept, the term used may differ slightly depending on whether the object to be referred to is a frame or data. Of course, since the unit frame image forms serial data in the communication process, the image data may be regarded as data in which pixel values of pixels constituting the unit frame image are serial. However, in the embodiment of the present invention, the above terms will not be particularly limited to the concept because they are used in various ways by those skilled in the art. However, if it is necessary to distinguish, use it clearly.

The image conversion unit 721 may include a part or all of the video frame acquisition unit 701, the video frame subsampling unit 702, the video frame scaling unit 703, and the pixel format conversion unit 704. The motion-based image analysis unit 722 includes a motion area detection unit 705, an object tracking unit 706, a tracking object classification unit 707, an uninterested tracking object removal unit 709, and an interest tracking object-based event detection unit 710 It may include some or all of. Here, "including some or all" has the same meaning as above.

The video frame acquisition unit 701 continuously acquires video frame data from a video providing device, such as a CCTV camera or video streaming server connected to the video analysis device 690. For example, when the video providing device connected to the video analysis device 690 is an IP camera, the video frame acquiring unit 701 receives and decodes the encoded video stream data from the IP camera, YUV pixel format (eg, YV12 pixel format) ) Continuously acquires video frame data (or frame video data). Here, YUV (for example, 16-bit) is a format representing a color with three pieces of information: the difference (U) between the luminance signal (Y) and the red signal, and the difference (V) between the luminance signal and the blue component. Since the Y component is error-sensitive, it codes more bits than the color components U and V. The ratio of Y:U:V is generally 4:2:2, so it is also called YUV422. In addition, since red and blue components are used, it is combined with chroma, which refers to color, and is also called YcrCb by using abbreviations of Chroma Red and Chroma Blue.

The video frame subsampling unit 702 subsamples frames to be used for image analysis from the (video) frames obtained by the video frame acquisition unit 701. For example, if the frame rate of the video input to the video analysis apparatus 690, that is, the frame rate is 30 FPS, and the video analysis frame rate is set to 6 FPS, the video frame subsampling unit 702 may check every input video. Video frames are acquired one frame for every 5 frames.

The reason for using subsampled frames (or a plurality of second video frames) instead of all the frames of the input video for video analysis is that the processing load of the video analysis device becomes very large when all the frames of the input video are used. This is because the effect (eg, improved object tracking performance) is usually small. 6~10FPS is enough to track pedestrians or vehicle objects in a typical CCTV image.

The video frame scaling unit 703 scales the size (eg, resolution) of the video frame obtained by the video frame subsampling unit 702 to a specified size. Typically, the video frame scaling unit 703 serves to reduce a high-resolution input image to a low resolution. For example, if the resolution of the video frame acquired by the video frame acquisition unit 701 is a high resolution of 1920x1080, and the resolution of the image to be used by the image analysis unit 722 is set to 640x480, the video frame scaling unit 703 is a 1920x1080 resolution. 640x480 resolution image is generated through pixel subsampling and linear interpolation from the input image.

The reason for using the reduced-resolution image that is reduced instead of using the original high-resolution image when analyzing the image is that when the high-resolution image is used as it is, the processing load of the image analysis device becomes very large while its effect (e.g., improved object detection performance) Because is usually small. In a remote monitoring environment, it may be necessary to process a high resolution input image without reducing it in order to detect a small object at a long distance. The processing load of the analytical device has a much reduced benefit.

The pixel format converting unit 704 is an RGB pixel format or gray-scale that is a pixel format that can be used by the image analysis unit 722 to scale YUV pixel-formatted images through the video frame scaling unit 703. Convert to pixel format.

The motion area detector 705 periodically obtains (or recognizes) a basic motion area through a difference (or difference) between the background image and the input image, and performs various noise motion pixel removal methods and morphology filtering. To detect the final motion area. 8 is an example showing the processing process and results of the motion region detection unit 705, the input image 801, the learned background image 802, the basic motion region detection result 803 due to the difference between the input image and the background image, This is an example of the final motion area detection result 804 by noise foreground pixel removal and morphological filtering.

The object tracking unit 706 performs multiple (or multiple) object tracking using the input image 801 of FIG. 8 and the final motion area detection result 804. Specifically, the object tracking unit 706 detects new objects, tracks objects between frames by matching, updates the template image and bounding box coordinates of the tracking objects, manages the tracking object list, and manages trajectories for each tracking object. . FIG. 9 is an example showing a process and results of the object tracking unit 706 of FIG. 7, and is an example of a template image 801 and object detection/tracking results 802 of several tracking objects. In the example of the object detection/tracking result 802 of FIG. 8, it can be seen that in addition to the human object, which is the object of interest, a number of meaningless objects (eg, objects detected by the motion of the branches swaying in the wind) are detected. .

The tracking object classification unit 707 performs object classification on the objects being tracked using the object image recognition DCNN 708. The details related to the tracking object classification unit 707 will be described later with reference to FIGS. 10 to 12.

Also, the non-interest tracking object removing unit 709 removes tracking objects classified as non-interesting classes (class, class, hierarchy) from the tracking object list of the object tracking unit 706.

The interest tracking object-based event detection unit 710 detects an event that satisfies a specified rule by using tracking information of objects of interest class.

10 is a view for explaining a processing process of the tracking object classification unit of FIG. 7.

For convenience of description, referring to FIG. 10 together with FIG. 7, as in the example of FIG. 10, the object image recognition DCNN 708 of FIG. 7 receives an object image with a normalized size as an input “person”, “vehicle”, “ It is a deep neural network recognized as one of the unidentified classes. For example, when an object image is input, the object image recognition DCNN 708 may compare the input object image with pre-stored images (eg, a sample image or image data) and provide a recognition result.

Accordingly, the tracking object classification unit 707 of FIG. 7 acquires an image patch (or object image) corresponding to the object bounding box area from the input image for each object being tracked, and normalizes the size of the image patch Then, it attempts to recognize through the object image recognition DCNN (708).

The tracking object classification unit 707 does not attempt to recognize DCNN only once, but periodically tries to perform more accurate object classification. This is because, as in the example of 1001 in FIG. 10, the DCNN recognition result obtained at a specific time point may be an unstable recognition result according to the detection state of the object bounding box.

Specifically, the tracking object classification unit 707 attempts DCNN recognition by acquiring an image (or an image patch) of an object from an input image every specified period P for a specific tracking object as shown in FIG. 10, and accumulates DCNN recognition results for each class Then, the class having the highest cumulative score in the current frame is determined as the class of the tracking object. For example, as shown in FIG. 10, when seven unit frames are input at regular time intervals, DCNNs are periodically obtained by obtaining object image patches for the same object in the first, third, fifth, and seventh unit frames. It can be applied at regular intervals.

The above period P value is usually 0.5 seconds to 1 second. The smaller the P value, the greater the processing load of the image analysis device 690, and the larger the P value, the lower the object classification performance. In addition, when acquiring an image patch of an object, it is preferable to acquire it from an image of the original resolution (ie, a video frame obtained from the video frame subsampling unit 702 of FIG. 7 ). This is because when an image patch of an object is obtained from an image after reduction, an image patch that is difficult to recognize due to deterioration in quality may be obtained.

The number of DCNN recognition attempts periodically performed may be limited to a total of N times. This is to prevent the DCNN recognition attempt to continue to be unnecessary for an object that is tracked for a long time.

11 is an example of sample images for learning an object image recognition DCNN classifying a given image into one of human (a), vehicle (b), and unidentified (c) classes.

Referring to FIG. 11 together with FIG. 7 for convenience of description, as in the example of FIG. 11, in particular, in the case of learning data of the “person” object, a partial image of a person or an image including two or more persons is also included in the learning data of the person. It is worth noting that it is included in the training sample. This is because, in the case of the human object region detected by the motion-based image analysis unit 722 of FIG. 7, not only the whole body of the person but also a part of the person or two or more persons is frequently included.

As the model of the object image recognition DCNN 708 of FIG. 7, various types of DCNN models proposed to date can be used. Examples include AlexNet, VGGNet, GoogLeNet, and ResNet models that have won the ILSVRC (ImageNet Large Scale Visual Recognition Challenge). When selecting a DCNN model to be actually used, there is a problem by simply considering DCNN recognition performance. This is because there is a trade-off relationship between DCNN processing time and capacity (eg, number of parameters) and DCNN recognition performance. Recently, DCNN models, which have similar recognition performance to existing DCNN models and have significantly reduced the processing time and the number of parameters, have been announced, for example, SqueezeNet or MobileNet.

FIG. 12 is an example in which tracking objects are classified through the tracking object classification unit of FIG. 7, FIG. 13 is an example of processing results of the uninterested tracking object removal unit of FIG. 7, and FIG. 14 is a tracking object-based event of interest in FIG. 7 This is an example showing the processing result of the detector.

As shown in FIG. 12, it can be seen that all other mis-detected objects are normally classified as “Unknown” in addition to the real human object.

In addition, FIG. 13 shows the processing result of the non-interest tracking object removal unit 709 from the object classification result of FIG. 12 when the “unconfirmed” class is designated as the non-interest class.

14 shows an example of detecting an event in which the “person” object invades a designated area.

15 is a configuration diagram of an object image recognition DCNN-based CCTV image analysis device for simultaneously real-time analysis of N CCTV camera images according to an embodiment of the present invention.

15, the CCTV image analysis apparatus 1490 according to an embodiment of the present invention includes N video channel processing units 1501, an object image recognition processing unit 1502 for processing N video streams simultaneously, and Includes some or all of the object image recognition DCNN 708, where "including some or all" has the same meaning as above.

The video channel processor 1501 may include part or all of the image converter 721 of FIG. 7 and the motion-based image analyzer 722. The N video channel processor 1501 and the object image recognition processor 1502 typically operate on the CPU of the image analysis device 1490. On the other hand, the object image recognition DCNN 708 can operate on a GPU capable of large-scale parallel computation for high-speed operation.

The N video channel processing units 1501 do not use N object image recognition DCNNs, respectively, but “share” one object image recognition DCNN 708. In order to drive one DCNN, a large amount of system memory is usually required. When N video channel processing units 1501 individually use DCNNs in memory, a memory shortage problem may occur as the value of N increases. to be.

Each motion-based image analysis unit 722 within the N video channel processing units 1501 performs normalized object image patching and recognition of the tracking object in order to perform classification of tracking objects in the same manner as described in FIG. The request message is periodically transmitted to the object image recognition processing unit 1502. The recognition request messages are sequentially stored in the request message queue of the object image recognition processing unit 1502. The object image recognition processing unit 1502 pulls the request message out of the queue and processes it as soon as it finds that the request message queue has a recognition request message. That is, the object image recognition processing unit 1502 inputs the normalized object image patch received with the recognition request message to the object image recognition DCNN 708 to obtain an object image recognition result, and analyzes the corresponding motion-based image that requested the recognition. The object image recognition result is transmitted to the unit 722. The motion-based image analysis unit 722 receives the object image recognition result and performs object classification as shown in FIG. 10.

16 is a configuration diagram of an object image recognition DCNN-based CCTV image analysis system for performing image analysis on a large scale in a CCTV control center, according to an embodiment of the present invention.

In order to construct a large-scale image analysis system, the system may be configured by simply using a plurality of image analysis devices 1490 shown in FIG. 15. However, the high-performance GPU available in the control center is usually very expensive, and thus, it is costly to install and use the GPU for each image analysis device.

In order to solve the above problem, in the embodiment of the present invention, as shown in FIG. 16, a typical video channel processing is performed by a plurality of image analysis servers 1601 not equipped with a GPU, and DCNN-based object image recognition is a high performance GPU. The image analysis system 1590 may be configured to be performed by a small number of object image recognition servers 1602 mounted thereon. The image analysis server 1601 performs high-speed network communication with the object image recognition server 1602.

The video channel processing unit of the image analysis server 1601 periodically transmits the normalized object image patch and the recognition request message of the tracking object to the object image recognition server 1602 in order to classify the tracking objects in the same manner as in FIG. 14. To deliver. At this time, the object image patch data is compressed and transmitted in a format such as JPEG. The recognition request messages are sequentially stored in the request message queue of the object image recognition processing unit 1603 of the object image recognition server 1602. The object image recognition processing unit 1603 pulls the request message out of the queue and processes it as soon as it finds that the request message queue has a recognition request message. That is, the object image recognition processing unit 1603 decodes the compressed object image patch data received with the recognition request message, and then inputs the object image recognition DCNN 708 to obtain the object image recognition result. The object image recognition processing unit 1603 transmits the object image recognition result to the video channel processing unit of the image analysis server 1601 that has requested the recognition. The video channel processing unit of the image analysis server 1601 receives the object image recognition result and performs object classification as shown in FIG. 10.

17 is a block diagram showing the structure of an image analysis apparatus according to another embodiment of the present invention.

As shown in FIG. 17, the image analysis apparatus 1700 according to another embodiment of the present invention includes a communication interface unit 1710, a control unit 1720, a DCNN-based image analysis unit 1730, and a storage unit 1740. Includes all or part of.

Here, "including some or all" means that some components such as the communication interface unit 1710 or the storage unit 1740 are omitted to configure the image analysis device 1700 or the DCNN-based image analysis unit 1730. It means that some of the same components may be integrated into other components such as the control unit 1720 and the like, and all of them are included to help the understanding of the invention.

The communication interface unit 1710 may communicate with a CCTV through a communication network of a communication company, and perform operations such as modulation/demodulation in the process.

The control unit 1720 is in charge of the overall control operation of the communication interface unit 1710, the DCNN-based image analysis unit 1730, and the storage unit 1740 constituting the image analysis device 1700 of FIG. For example, the control unit 1720 may provide the captured image of the CCTV provided through the communication interface unit 1710 to the DCNN-based image analysis unit 1730.

The DCNN-based image analysis unit 1730 may perform an image analysis operation related to an embodiment of the present invention as described with reference to FIGS. 7 to 16 above. Typically, a frame rate of a received frame image is converted or a unit frame image is randomly acquired. In the exemplary embodiment of the present invention, this is called subsampling. In addition, the resolution of the unit frame image acquired through subsampling is converted from high resolution to low resolution. Then, the format of the received frame image can be converted.

In addition, the DCNN-based image analysis unit 1730 extracts an object image patch from a subsampled frame image whose format has been converted, recognizes an object image based on DCNN, and accurately classifies the object. do. The DCNN-based image analysis unit 1730 detects an event only for a tracking object of interest through the removal operation. Then, an alarm can be output based on the event.

In addition to the above, the communication interface unit 1710, the control unit 1720, the DCNN-based image analysis unit 1730, and the storage unit 1740 of FIG. 17 have been sufficiently described above. I would like to replace it.

Meanwhile, as another embodiment of the present invention, the control unit 1720 of FIG. 17 may include a CPU and a memory. The CPU may include a control circuit, an operation unit (ALU), an instruction analysis unit, and a registry, and the memory may include RAM. Here, the CPU may mean the CPU in FIG. 15. The control circuit is in charge of the control operation, the operation unit performs a bit operation, and the instruction analysis unit may interpret the machine language to determine what command it is. The registry may be involved in the temporary storage of data. The control unit 1720 may be generally configured as a one-chip. Accordingly, the CPU may rapidly increase the computational processing speed of the CPU by copying the program stored in the DCNN-based image analysis unit 1730 of FIG. 17 and loading it into memory and executing it in the initial stage of operation of the image analysis apparatus 1700. .

18 is a flowchart illustrating a driving process of an image analysis apparatus according to an embodiment of the present invention.

Referring to FIG. 18 together with FIG. 17 for convenience of description, the image analysis device 1700 according to an embodiment of the present invention provides an image provided by a video providing device (eg, CCTV, a plurality of high-resolution cameras, etc.), that is, a plurality of videos. The frame image is received and converted into a format that reduces the load of image processing (S1800). Here, it is better to understand that the format is converted into a video image of a different form from the received video image, rather than the first format. Therefore, for this purpose, the image analysis apparatus 1700 preferably converts the resolution to a lower resolution than the input image, and fewer images to be processed, and preferably to a video image in RGB or G-scale pixel format. This can be analyzed to be possible through the video of CCTV even in a general CPU environment, and through this, it is intended to enable the control personnel to perform the control.

Subsequently, the image analysis apparatus 1700 extracts object images for a plurality of object objects for motion tracking from a plurality of format-converted video frame images, and applies the extracted object images to DCNN using an object image recognition method. Based on the application result, a plurality of tracking objects are classified, and an event is detected based on the tracking information of the user-interested object and a designation rule obtained by removing a user uninteresting object that exceeds a specified criterion from the classified plurality of tracking objects (S1810) ).

For example, DCNN may store a sample image and utilize it to provide a recognition result of an input object image. In addition, by accumulating application results by learning, a cumulative result is generated, and a class having a high cumulative result is finally determined as a tracking object. For example, if the cumulative result for a human object is 2 points and if it is 1 unconfirmed, it is determined as a human object in the corresponding frame section.

Accordingly, for example, a specific object classified as unidentified is excluded from the object of a plurality of tracking objects based on the object image of the object. Therefore, the bounding box formed on the tracking object on the screen can be removed.

In this way, only the object of interest of the user is tracked, and an event in which the tracking information satisfies the specified rule is detected.

In addition, various operations may be possible, but the details will be replaced by the contents described above.

On the other hand, that all components constituting the embodiments of the present invention are described as being combined or operated as one, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all of the constituent elements may be selectively combined and operated. In addition, although all of the components may be implemented as one independent hardware, a part or all of the components are selectively combined to perform a part or all of functions combined in one or a plurality of hardware. It may be implemented as a computer program having a. The codes and code segments constituting the computer program can be easily deduced by those skilled in the art of the present invention. Such a computer program is stored in a computer-readable non-transitory computer readable media and read and executed by a computer, thereby realizing an embodiment of the present invention.

Here, the non-transitory readable recording medium means a medium that stores data semi-permanently and that can be read by a device, rather than a medium that stores data for a short time, such as registers, caches, and memory. . Specifically, the above-described programs may be stored and provided on a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, although the preferred embodiment of the present invention has been illustrated and described, the present invention is not limited to the specific embodiments described above, and it is usually in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. It is of course possible to perform various modifications by a person having knowledge of, and these modifications should not be individually understood from the technical idea or prospect of the present invention.

100: video recording device 200: video surveillance device
210: video input unit 211: video decoding unit
212: image storage unit 213: image search / playback unit
214: low-resolution panoramic image generation unit 216, 706: object tracking unit
217: manual digital PTZ control unit 218: automatic digital PTZ control unit
221: image display unit 770, 1490, 1700: image analysis device
701: Video frame acquisition unit 702: Video frame subsampling unit
703: video frame scaling unit 704: pixel format conversion unit
705: motion area detection unit 707: tracking object classification unit
708: Object image recognition DCNN 709: Non-interest tracking object removal unit
710: interest tracking object-based event detection unit 721: image conversion unit
722: motion-based image analysis unit 1501: video channel processing unit
1502, 1603: object image recognition processing unit 1590: image analysis system
1601: image analysis server 1602: object image recognition server
1710: Communication interface unit 1720: Control unit
1730: DCNN-based image analysis unit 1740: storage unit

Claims (12)

A panoramic image surveillance system using a plurality of high-resolution cameras that perform image analysis using DCNN (Deep Convolutional Neural Network) that provides recognition results for objects in an image.
The panoramic video surveillance system using the plurality of high-resolution cameras,
When a high-resolution image is received from the plurality of high-resolution cameras and a low-resolution panoramic image is generated and displayed using the high-resolution image based on a low-resolution panoramic image generation table, an event occurs in the process of displaying the low-resolution panoramic image. Display a high-resolution digital PTZ image using the panoramic image generation table,
The first image data composed of a plurality of first video frames having a first pixel format, a first resolution, and a first frame rate is received from the high-resolution camera, and the first image data is received from the high-resolution camera. 1 resolution, the first frame rate and the first pixel format, respectively, a second resolution lower than the first resolution, a second frame rate lower than the first frame rate, and a format different from the first pixel format An image converter for converting second image data in a heterogeneous second pixel format having a; And
In the video frame of the converted second image data, motion and detection of a plurality of moving objects are performed based on motion, images of the plurality of objects being tracked are respectively extracted, and the extracted object images are stored in the DCNN. Using the recognition result obtained by inputting, each of the plurality of tracking objects through the DCNN is classified into one of a plurality of designated types, and a user uninterested object that deviates from a specified criterion in the classified plurality of tracking objects The motion-based image analysis unit detects an event based on tracking information and a designated rule of a user interest object obtained by removing the identified user uninterested object from the object of the plurality of tracking objects by checking the
The image conversion unit,
And a video frame subsampling unit that samples a smaller number of video frames than the input plurality of first video frames to generate a plurality of second video frames,
The motion-based image analysis unit,
Based on the accumulation result of the recognition result for the plurality of tracking objects in a designated frame section, an object of a specified type having a low accumulation result is removed from the object of the plurality of tracking objects as the user uninteresting object,
The first recognition result obtained by applying the object images of the plurality of tracking objects extracted from one video frame among the plurality of second video frames to the DCNN and a certain time from the one video frame among the plurality of second video frames A tracking object classification unit for classifying the plurality of tracking objects based on a second recognition result obtained by applying the object images of the plurality of tracking objects extracted from at least one other video frame input at intervals to the DCNN; And
It includes; a non-interest tracking object removal unit for removing the identified user non-interest object from the object of the plurality of tracking objects in the object tracking unit by identifying a user non-interest object that is out of a specified criterion among the divided plurality of tracking objects;
The tracking object classification unit,
A panoramic image surveillance system using a plurality of high-resolution cameras that calculates and accumulates the first recognition result and the second recognition result as scores, respectively, and determines a class having the highest accumulated score as a tracking object type. According to claim 1,
The image conversion unit,
A video frame acquiring unit receiving the plurality of video frames input according to the first frame rate;
A video frame scaling unit converting the generated second video frames into the second resolution; And
A pixel format converter that provides the second image data generated by converting the first pixel format of the plurality of second video frames converted to the second resolution into the second pixel format to the motion-based image analyzer; To
Panoramic image surveillance system using a plurality of high-resolution cameras further included. According to claim 1,
The motion-based image analysis unit,
A motion region detection unit generating a difference image using the second image data and the learned background image, and removing motion noise from the generated difference image to detect motion regions for the plurality of moving objects;
An object tracking unit for detecting and tracking the plurality of moving objects using the detected motion region and the second image data; And
An interest tracking object-based event detection unit that detects an event in which tracking information of the user interest object obtained according to the removal satisfies a specified rule;
Panoramic image surveillance system using a plurality of high-resolution cameras further included. delete According to claim 1,
The non-interest tracking object removal unit, a panoramic image surveillance system using a plurality of high-resolution cameras to identify and remove the user non-interest tracking object outside the specified criteria from the determined type of object. According to claim 1,
The image converting unit may include the first pixel format having a pixel format of a luminance signal (Y), a difference (U) of a red signal, and a difference (V) of a luminance signal and a blue component (Red-Green-Blue) or gray. A panoramic image surveillance system using a plurality of high-resolution cameras that convert to the second pixel format having a gray-scale pixel format. As a video surveillance method of a panoramic video surveillance system using a plurality of high-resolution cameras that perform video analysis using DCNN that provides recognition results for objects in the video,
The video surveillance method of the panoramic video surveillance system using the plurality of high-resolution cameras,
When a high-resolution image is received from the plurality of high-resolution cameras and a low-resolution panoramic image is generated and displayed using the high-resolution image based on a low-resolution panoramic image generation table, an event occurs in the process of displaying the low-resolution panoramic image. Display a high-resolution digital PTZ image using the panoramic image generation table,
The image conversion unit constituting the panoramic image surveillance system receives first image data composed of a plurality of first video frames having a first pixel format, a first resolution, and a first frame rate from the high-resolution camera, and receives the first image data. The first resolution of the first image data, the first frame rate, and the first pixel format are respectively a second resolution lower than the first resolution, a second frame rate lower than the first frame rate, and the first pixel. Converting the second image data in a heterogeneous second pixel format having a format different from the format; And
The motion-based image analysis unit constituting the panoramic image monitoring system detects and tracks a plurality of moving objects based on motion in video frames of the second image data converted by the image conversion unit, and the plurality of objects being tracked The images of each are extracted, and the plurality of tracking objects through the DCNN are classified into one of a plurality of designated types using the recognition result obtained by inputting the extracted object image to the DCNN, respectively. The event is based on the tracking information and the designation rule of the user interest object obtained by removing the identified user non-interest object from the object of the plurality of tracking objects by checking the user non-interest object that is out of the specified criteria from the plurality of classified tracking objects It comprises the steps of detecting;
The step of converting the second image data,
And generating, by the video frame subsampling unit of the image conversion unit, a plurality of second video frames by sampling fewer video frames than the input plurality of first video frames,
The detecting of the event may include:
The motion-based image analysis unit, based on the accumulation result of the recognition result with respect to the plurality of tracking objects in a designated frame section, uses the object of the plurality of tracking objects as an object of unspecified object as the user-not-interested object. Remove it,
In the tracking object classification unit of the motion-based image analysis unit, a first recognition result obtained by applying the object images of the plurality of tracking objects extracted from one video frame among the plurality of second video frames to the DCNN and the plurality of agents Based on the second recognition result obtained by applying the object images of the plurality of tracking objects extracted from at least one other video frame input at a predetermined time interval from the one video frame among the two video frames to the DCNN. Identifying a tracking object; And
In the non-interest tracking object removal unit of the motion-based image analysis unit, a user non-interest object that is out of a specified criterion among the plurality of divided tracking objects is identified, and the identified user non-interest object is the object of the plurality of tracking objects of the object tracking unit Removing from; includes,
The step of distinguishing the plurality of tracking objects,
A method of monitoring a panoramic image using a plurality of high-resolution cameras that calculates and accumulates the first recognition result and the second recognition result as a score, respectively, and determines the type having the highest accumulated score as a tracking object type. The method of claim 7,
The image conversion unit further includes a video frame acquisition unit, a video frame scaling unit, and a pixel format conversion unit,
The step of converting the second image data,
Receiving, by the video frame acquiring unit, the plurality of video frames input according to the first frame rate;
Converting the generated plurality of second video frames into the second resolution by the video frame scaling unit; And
The pixel format converter provides the second image data generated by converting the first pixel format of the plurality of second video frames converted to the second resolution into the second pixel format to the motion-based image analyzer To do;
Panoramic image monitoring method using a plurality of high-resolution cameras further included. The method of claim 7,
The motion-based image analysis unit further includes a motion area detection unit, an object tracking unit, and an interest tracking object-based event detection unit,
The detecting of the event may include:
Generating, by the motion area detection unit, a difference image using the second image data and the learned background image, and removing noise from the generated difference image to detect motion regions for the plurality of moving objects;
In the object tracking unit, detecting and tracking the plurality of moving objects using the detected motion region and the second image data; And
In the interest tracking object-based event detection unit, detecting an event in which tracking information of a user interest object obtained according to the removal satisfies a specified rule;
Panoramic image monitoring method using a plurality of high-resolution cameras further included. delete The method of claim 7,
The step of removing from the object of the plurality of tracking objects,
A method of monitoring a panoramic image using a plurality of high-resolution cameras that identifies a user's uninterested tracking object that exceeds the specified criteria from the determined type of object and removes it from the target of the plurality of tracking objects. The method of claim 7,
The step of converting the second image data,
Converting the first pixel format having the pixel format of the luminance signal (Y), the difference (U) of the red signal, and the difference (V) of the luminance signal and the blue component to the second pixel format having an RGB or grayscale pixel format Panorama video surveillance method using a plurality of high-resolution cameras.

Images