KR102139582B1 - Apparatus for CCTV Video Analytics Based on Multiple ROIs and an Object Detection DCNN and Driving Method Thereof - Google Patents

Abstract

The present invention relates to a CCTV image analysis device based on a multi-region of interest (ROI) and object detecting DCNN and to a driving method thereof. According to an embodiment of the present invention, the CCTV image analysis device based on a multi-ROI and object detecting DCN comprises: a storage unit for storing setting information on a first ROI and a second ROI separately set in appearance regions of objects of interest in a photographing area of a photographing device; and a control unit for separately extracting a first object image of the first ROI and a second object image of the second ROI on the basis of the (pre)stored setting information from a photographed image received from the photographing device to perform object detection based on DCNN with respect to the extracted first and second object images, and performing object tracking and event detection after merging objects of the first ROI and the second ROI detected by the object detection. Accordingly, a remote object can be more properly detected.

Description

{Apparatus for CCTV Video Analytics Based on Multiple ROIs and an Object Detection DCNN and Driving Method Thereof}

The present invention relates to a multi-ROI and object detection DCNN-based CCTV image analysis device and a method of driving the apparatus, and more specifically, for example, an object detection from a CCTV image centering on a multiple region of interest (ROI) Deep Convolutional Neural Network ) To detect the objects of interest, and based on this, it relates to a CCTV image analysis device for detecting an object tracking and an event of interest and a method of driving the device.

The CCTV video analysis device, which forms the core of the intelligent CCTV video surveillance system, detects and tracks objects of interest from the CCTV camera video, and automatically detects anomalies such as “intrusion in the forbidden area” based on this and generates an alarm. . One of the core technologies in implementing such a CCTV image analysis device is “interesting object detection” technology.

Of course, in the computer vision field, various methods for detecting an object of interest have been studied for a long time, and since deep-learning technology has been developed since the mid-2010s, DCNN-based object detection technologies have been announced. These DCNN-based object detection technologies have become popular by showing detection performance far exceeding that of existing object detection technologies. Representative object detection DCNN models include Faster R-CNN, Single Shot Multibox Detector (SSD), and You Only Look One (YOLO).

When actually using the object detection DCNN trained using the object detection DCNN model, the input resolution (input size) of the DCNN is fixed to a certain value. Therefore, in order to perform object detection from a given image, the input image must be scaled (usually reduced) to make it equal to the DCNN input resolution and then input to the DCNN. This causes a problem when trying to detect distant objects (objects appearing in a small size in an image) from an input image having a resolution higher than the input resolution of the object detection DCNN. This is because while the high-resolution input image is reduced to be equal to the input size of the object detection DCNN, the distant objects in the image often become small enough that the object detection DCNN cannot detect.

Most object detection DCNN models tend to increase the object detection performance (especially the detection performance of distant objects) as the DCNN input resolution increases. However, as the input resolution of the DCNN increases, the capacity and computation amount of the DCNN increases rapidly, so it is difficult to increase the input resolution of the blind object detection DCNN in order to improve the performance of the long-distance object detection.

Korean Registered Patent Publication No. 10-1942808 (2019.01.22) Korean Registered Patent Publication No. 10-1955919 (2019.03.04) Korean Patent Publication No. 10-2018-0107930 (2018.10.04) Korean Patent Publication No. 10-2018-0065856 (2018.06.18) Korean Patent Publication No. 10-2016-0096460 (2016.08.16)

According to an embodiment of the present invention, for example, a CCTV image analysis device for detecting an object of interest and detecting an object of interest and an object of interest based on the object detection DCNN based on the object detection DCNN from a CCTV image, and of the device. The object is to provide a driving method.

In addition, an embodiment of the present invention uses an object detection DCNN having a low input resolution to effectively detect objects of interest (especially long-distance objects) from high-resolution CCTV images, and analyze CCTV images based on this to perform object tracking and event detection. Another object is to provide a device and a method for driving the device.

The multi-ROI and object detection DCNN-based CCTV image analysis apparatus according to an embodiment of the present invention sets a first region of interest (ROI) and a second region of interest that are respectively set in regions where objects of interest appear in a region of the imaging apparatus. A first object image of the first region of interest and a second object image of the second region of interest are respectively extracted from the storage unit for storing information and the stored setting information from the captured image received from the imaging device, respectively. DCNN-based object detection is performed on the extracted first object image and the second object image, and after merging the objects of the first region of interest and the second region of interest detected by the object detection, object tracking and And a control unit that performs event detection.

When the first region of interest and the second region of interest overlap, the control unit may process the same object in the overlapped region as one object based on the unit frame of the captured image.

The control unit may generate a new bounding box by merging the first bounding box and the second bounding box respectively set for the same object in the overlapping region based on the unit frame.

The control unit may use an object detector with less loss on distant objects when performing DCNN-based object detection after resolution conversion of the first object image and the second object image.

The control unit includes a region of interest (ROI) image processing unit for the resolution conversion, and the converted resolution may match the resolution of the DCNN input image of the object detector.

The controller may determine a set size of the first region of interest and the second region of interest according to the resolution of the object detector to be used.

In addition, a method of driving a multi-ROI and object detection DCNN-based CCTV image analysis apparatus according to an embodiment of the present invention includes: a first region of interest (ROI) and a first region of interest (ROI) that are respectively set in a region where objects of interest appear in an imaging region of the imaging apparatus. 2 storing the setting information of the region of interest in the storage unit, and the control unit, based on the stored setting information in the captured image received from the photographing apparatus, the first object image and the second interest of the first region of interest DCNN-based object detection is performed on the extracted first object image and the second object image by respectively extracting a second object image of the region, and the first region of interest and the second region of interest detected by the object detection And performing object tracking and event detection after merging objects in the region.

In the step of performing, when the first region of interest and the second region of interest overlap, the same object in the overlapping region may be processed by merging into one object based on the unit frame of the captured image.

In the step of performing, a new bounding box may be generated by merging the first bounding box and the second bounding box respectively set for the same object in the overlapping region based on the unit frame.

In the step of performing, the DCNN-based object detection may be performed after resolution conversion of the first object image and the second object image, and an object detector with less loss on distant objects may be used.

The ROI image processing unit performs the resolution conversion, and the converted resolution may match the resolution of the DCNN input image of the object detector.

The driving method may further include determining a set size of the first region of interest and the second region of interest according to the resolution of the object detector to be used.

According to an embodiment of the present invention, when performing object detection from an input CCTV image through, for example, object detection DCNN, object detection is performed from a partial image (ROI image) corresponding to a set ROI, rather than from the entire input CCTV image. Therefore, when converting an ROI image to a DCNN input image, the loss on a far object is much smaller than when converting the entire image to a DCNN input image, so that far objects can be detected better than when detecting objects on the whole image. Will be.

1 is a view showing a DCNN-based video control system according to an embodiment of the present invention,
Figure 2 is a block diagram illustrating the detailed structure of the image analysis device of Figure 1,
3 is a block diagram illustrating another detailed structure of the image analysis apparatus of FIG. 1,
4 is a diagram showing an example of setting an ROI and an event detection area,
5 is a view showing ROI images corresponding to ROI-1 and ROI-2 of FIG. 4;
FIG. 6 is an exemplary view of detecting the bounding box of pedestrian objects in the ROI-1 by the DCNN-based object detection unit of the ROI-1 image of FIG. 5;
FIG. 7 is a view for explaining a correspondence between bounding box coordinates in an input video frame for the same object and bounding box coordinates in a normalized ROI image;
FIG. 8 is a diagram illustrating an example in which object bounding boxes detected in ROI-1 and ROI-2 of FIG. 4 are integrated based on an input video frame, and
9 is a flowchart of a driving process of an image analysis apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a view showing a DCNN-based image control system according to an embodiment of the present invention.

As shown in Figure 1, the DCNN-based video control system 90 according to an embodiment of the present invention is an imaging device 100, a communication network 110, an image analysis device 120, a control device 130 and a manager Includes some or all of device 140.

Here, "including some or all" means that some components, such as the image analysis apparatus 120, are omitted, so that the DCNN-based image control system 90 is configured, or components constituting the image analysis apparatus 120. It means that all or part of a network device (eg, a wireless switching device, etc.) constituting the communication network 110 may be integrated and configured.

The photographing apparatus 100 is a surveillance camera and includes a general CCTV (Closed Circuit Television) camera or an IP (Internet Protocol) camera. In addition, the photographing apparatus 100 may include a pan-tilt-zoom (PTZ) camera capable of a pan, tilt, and zoom operation as well as a fixed camera. The photographing apparatus 100 may be installed in various places for social safety, crime prevention, suicide issue, and public surveillance to provide a photographed image. For example, the photographing apparatus 100 may be installed in a public place such as a subway or a bus stop to monitor various situations such as incidents and accidents, and to monitor the railing of a bridge or violent activities occurring in a remote place. Furthermore, surveillance will be possible through CCTV installed in daycare centers.

The communication network 110 includes both wired and wireless communication networks. For example, a wired/wireless Internet network may be used or interlocked as the communication network 110. Here, the wired network includes an Internet network such as a cable network or a public telephone network (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, Evolved Packet Core (EPC), Long Term Evolution (LTE), Wibro network, etc. It is meant to include. Of course, the communication network 110 according to an embodiment of the present invention is not limited thereto, and may be used, for example, a cloud computing network under a cloud computing environment, a 5G network, and the like. For example, when the communication network 110 is a wired communication network, an access point in the communication network 110 may access a telephone exchange or the like, but in the case of a wireless communication network, data may be accessed by accessing a SGSN or a Gateway GPRS Support Node (GGSN) operated by a communication company. The data can be processed by accessing various repeaters such as BTS (Base Station Transmission), NodeB, and e-NodeB.

The communication network 110 may include an access point. The access point here includes small base stations, such as femto or pico base stations, which are often installed in buildings. The femto or pico base station is classified according to the maximum number of accesses to the photographing apparatus 100, etc., according to the classification of the small base station. Of course, the access point may include a short-range communication module for performing short-range communication, such as Zigbee and Wi-Fi, with the photographing apparatus 100. The access point can use TCP/IP or RTSP (Real-Time Streaming Protocol) for wireless communication. Here, the short-range communication may be performed in various specifications such as Bluetooth, ZigBee, infrared, UHF (Ultra High Frequency) and VHF (Very High Frequency), and Radio Frequency (RF) and Ultra Wideband Communication (UWB). Accordingly, the access point extracts the location of the data packet, designates the best communication path for the extracted location, and transmits the data packet along the designated communication path to the next device, such as the video analysis device 120 or the control device 130, etc. Can be delivered. An access point can share multiple lines in a typical network environment, including routers, repeaters and repeaters, for example.

According to an embodiment of the present invention, the communication network 110 may include an internal computer network, such as an intranet, through which the image analysis device 120, the control device 130, and the manager device 140 may interwork with each other. . However, in the embodiment of the present invention, since each component may be configured in various forms around the communication network 110, it will not be specifically limited to any one form.

The image analysis device 120 performs DCNN-based image analysis based on object detection using multiple (or multiple) regions of interest (ROI) on the captured image received from the imaging apparatus 100. Above all, the image analysis apparatus 120 according to an embodiment of the present invention effectively detects objects of interest (especially long-distance objects) from a high-resolution CCTV image by using an object detection DCNN having a low input resolution, and tracks objects based on this. Event detection is performed. Here, "low input resolution" may be understood to be less than the average when comparing the performance of the object detector currently distributed on the market, but the embodiment of the present invention will not be specifically limited to such a concept. When the location of occurrence of objects of interest in the input CCTV image is not limited to the entire image, but is limited to a specific area, the user agent, for example, sets a ROI in the area of appearance of objects of interest. When performing object detection from the input CCTV image through the object detection DCNN, the image analysis device 120 performs object detection from a partial image (ROI image) corresponding to the set ROI, rather than from the entire input CCTV image. . When converting an ROI image to a DCNN input image, since the loss on a far object is much less than when converting the entire image to a DCNN input image, it is possible to detect a far object better than when detecting an object on the whole image.

In addition, the image analysis apparatus 120 converts the bounding box coordinates of the object detected from the ROI image to the bounding box coordinates based on the original (original) input image. Object tracking and event detection are performed using object bounding box information based on the original input image. Coordinate value setting may be calculated based on pixels. In other words, (0. 0) may be the upper left pixel when viewed from the unit frame image. If this is a method using an absolute coordinate value, any number of relative coordinates having a center portion of the unit frame image as (0, 0) may be used. When two or more ROIs are set, an object bounding box is obtained by applying object detection DCNN to each ROI. When overlapping regions exist between the set ROIs, the bounding box of the corresponding object may be obtained from one or more ROIs for the same object. In this case, the overlapping bounding boxes are merged into one.

As described above, the image analysis apparatus 120 according to an embodiment of the present invention, for example, in the case where the control agent sets the first region of interest and the second region of interest for the surveillance region, in the received image based on the setting information The object image corresponding to the region of interest is extracted and the resolution of the extracted object image is converted to match the resolution of the input image of the object detector. By performing deep learning, for example, DCNN, through this method, it is possible to prevent loss of a distant object during resolution conversion. In other words, even if a low-resolution object detector is used in consideration of a cost aspect or a data operation processing speed, the object detector sets an area of interest corresponding to the size and sets the resolution of the object image related to the area of interest. Even if converted, the far object can be detected better. Of course, in this case, even when a zoom camera is used, the size of a region of interest is changed in a captured image in a state in which zoom-in and zoom-out are performed, so that a resolution conversion can be performed correspondingly.

Since the video analysis device 120 subdivides the surveillance area, which is the area of occurrence of the objects of interest, on the entire screen entering the lens of the imaging device 100 into the area of interest, object tracking may not be easy, for example, object tracking is a unit frame Since it has to be performed based on an image, that is, a picture, the object is detected by merging two object images based on a unit frame or merging the detected objects, and then tracking the object in the region of interest using the object. . In the merging process, the same duplicate object is treated as one object. In addition, when setting a bounding box for object tracking, the same object is processed as one bounding box in the merging process. In order to smoothly detect a long-distance object, after dividing the region of interest, DCNN-based object detection is performed, and then the divided region of interest is merged to perform object tracking and event detection. Objects that occur during the partitioning process can be corrected by processing overlapping objects as one object in the process.

For example, the image analysis apparatus 120 performs image analysis on the captured image of the photographing apparatus 100, that is, extracts an object, and tracks movements on the extracted object and events (eg, traffic accidents) When it occurs, etc.), it is possible to provide intelligent control by providing it to the control device 130. For example, after multi-ROI is set for a crossing, object detection is performed through this, and then merged again to track objects and provide event information to the control device when it is determined that a traffic accident has occurred at the crossing. It is possible.

The control device 130 includes a control server or a control monitor operated by a control center such as a local government. The captured image of the image pickup device 100 may be received through the communication network 110, and selection or intelligent control may be performed based on the analysis result of the image analysis apparatus 120. For example, the control monitor may include a plurality of screens showing each photographed image of the plurality of photographing devices 100. On the screen of this, the image analysis apparatus 120 may display only the captured image 100 that provides event occurrence information.

The manager device 140 may include various devices such as a desktop computer, a laptop computer, a tablet PC, and a smart phone of a control agent. The control agent may access the image analysis device 120 or the control device 130 through the manager device 140 to set a plurality of areas of interest in a surveillance area photographed by the imaging device 100. Of course, a plurality of regions of interest correspond to regions where objects of interest appear, such as crossings, in the entire surveillance region. For example, in the related art, if such regions of interest could be set in the entire region of occurrence, the embodiment of the present invention is based on the DCNN-based object detector. It is desirable to set a plurality of regions of interest in consideration of performance and the like. Therefore, the maximum size of the region of interest may be determined as a default. For example, when the performance of the object detector is changed, the maximum size of the region of interest may be correspondingly determined. Since this can be determined algorithmically in a program, and may vary as much as the intention of a system designer, the embodiment of the present invention will not be limited to any one form.

FIG. 2 is a block diagram illustrating a detailed structure of the image analysis device of FIG. 1.

As shown in FIG. 2, the image analysis apparatus 120 of FIG. 1 according to an embodiment of the present invention includes a communication interface unit 200, a control unit 210, a multi-ROI image processing unit 220 and a storage unit 230 Includes all or part of.

Here, "including some or all" is that some components, such as the storage unit 230, are omitted, so that the image analysis apparatus 120 is configured, or the multi-ROI image processing unit 220 is different from the control unit 210. It means that it can be integrated into a component and can be configured, and is described as including everything in order to help the understanding of the invention.

The communication interface 200 communicates with the imaging device 100 and the control device 130 via the communication network 110 of FIG. 1. Of course, in the process of performing communication, the communication interface 200 may perform operations such as modulation/demodulation, encoding/decoding, muxing/demuxing, and scaling to convert resolution. Since this is obvious to those skilled in the art, further description will be omitted.

When the photographing image is provided by the photographing apparatus 100, the communication interface 200 transmits it to the control unit 210, and also controls the event generation information as the image analysis result from the multi-ROI image processing unit 220. ) And transmits it to the control device 130.

The control unit 210 is responsible for the overall control operation of the communication interface unit 200, the multi-ROI image processing unit 220 and the storage unit 230 of FIG. When a photographed image of the photographing apparatus 100 is received through the communication interface 200, it is provided to the multi-ROI image processing unit 220. In the process of this, the image data of the captured image may be temporarily stored in the storage unit 230 and then provided to the multi-ROI image processing unit 220.

In addition, the control unit 210 controls the communication interface unit 200 to provide the analysis result of the multi-ROI image processing unit 220 to the control device 130. For example, when it is determined that an event has occurred as a result of analysis of a photographed image provided by the photographing apparatus 100 of a specific channel among a plurality of photographing apparatuses 100, it is provided in the form of event information from the multi-ROI image processing unit 220 to control The device 130 can be notified. As a result, the control agent can know that an event, such as an accident, has occurred in the surveillance area monitored by the specific photographing device 100 in the control device 130.

In addition, the controller 210 may perform an operation for setting multiple ROIs according to an embodiment of the present invention. For example, when the control agent through the manager device 140 wants to set multiple ROIs in the areas of interest among the surveillance areas of the specific imaging device 100, it may be involved in its operation. Here, the set multi-ROI may be variously set according to the performance of the DCNN-based object detector used in the multi-ROI image processing unit 220. Typically, the number of object detectors may be determined according to the number of multiple ROIs, and the set size of the region of interest may be determined according to the performance of the object detector.

When a control agent sets multiple ROIs for a specific monitoring area through, for example, the manager device 140, the control unit 210 stores the information in the storage unit 230 and requests the multi-ROI image processing unit 220. It can be provided, but it can be stored in a software registry or the like inside the program of the multi-ROI image processing unit 220 to be used for image analysis.

The multi-ROI image processor 220 analyzes an image of a captured image received from the photographing apparatus 100, and more precisely, detects an object of interest, that is, distant objects, when detecting a DCNN-based object and detecting an event of interest for the detected object It works so that effective detection can be achieved. To this end, the multi-ROI image processing unit 220 sets the multi-ROI by taking into account the resolution of the input image of the DCNN-based object detector. Of course, this can be regarded as the premise of extracting the ROI image for the region of interest from the captured image and converting the extracted ROI image in a specified range. Since the long distance object may not be properly detected when converting the resolution, it is easy to detect the object only when a fixed performance resolution conversion is applied, so applying multiple ROIs rather than resolution conversion factors effectively matches the resolution of the input object of the object detector. Will be detected.

Accordingly, the multi-ROI image processing unit 220 extracts an image of a region of interest corresponding to each ROI region from the input image, based on setting information of the multi-ROI set by, for example, a control agent, and then converts the extracted image through resolution. An object is detected by providing an ROI image with a converted resolution to the object detector. Of course, here, object detection is to determine whether various types of objects, such as people and objects, are true objects, and deep learning, or DCNN-based object detectors, can be used to increase accuracy. In addition, since the surveillance area of the control agent substantially corresponds to the entire area of the multi-ROI, the multi-ROI image processing unit 220 performs a merge operation on the object detected by each object detector. Merging is of course done on a per-frame basis. In the process of this, for example, the same object in a region overlapping each other is treated as one object. Of course, when setting a bounding box for object tracking, a new bounding box is generated and processed for the same object by merging the bounding box. Thereafter, object tracking may be performed in a picture based on the merged unit frame-based objects, and an event may be detected in the process. The occurrence of an event may be determined by comparing the object and the object, and the correlation between the object and the object based on a rule, or may be determined by analyzing based on deep learning, so it will not be specifically limited to any one type. . Rule-based big data is not necessary, but deep learning uses big data.

As described above, the multi-ROI image processing unit 220 detects an object using multiple ROIs and saves the cost of a designated resource or an object detector, or increases the data processing speed, and merges them again to determine whether an event occurs. By making judgments, it is possible to save data processing speed or cost for resources.

The storage unit 230 may store and output information or data processed under the control of the control unit 210. Here, if the information means a control signal such as a request or a response, the data means data such as image data of a captured image, but the two terms may be mixed with each other, and accordingly, in the embodiment of the present invention, the concept of the term It will not be particularly limited to. For example, the storage unit 230 may store corresponding setting information when a control agent or the like sets an ROI.

Meanwhile, the control unit 210 of FIG. 2 may include a CPU and a memory. The CPU and the memory may be formed by one chip. The CPU includes a control circuit, an operation unit (ALU), an instruction analysis unit, and a registry, and the memory may include RAM. The control circuit can perform the control operation, the operation unit performs binary bit information calculation, and the instruction analysis unit performs the operation of interpreting the high-level language as the machine language and the machine language as the high-level language, and the registry is involved in software data storage. Can. As a result of the above configuration, when the initial driving of the image analysis apparatus 120 or when necessary, the program or algorithm stored in the multi-ROI image processing unit 220 may be stored in a memory and then executed to rapidly increase the computational processing speed.

3 is a block diagram illustrating another detailed structure of the image analysis apparatus of FIG. 1 (or the multi-ROI image processing unit of FIG. 2), FIG. 4 is a diagram showing an example of setting an ROI and an event detection area, and FIG. 5 is a view of FIG. FIG. 6 is a view showing ROI images corresponding to ROI-1 and ROI-2, and FIG. 6 is an exemplary view of detecting a bounding box of pedestrian objects in ROI-1 by the DCNN-based object detection unit of the ROI-1 image of FIG. 5. Is a diagram for explaining the correspondence between the bounding box coordinates in the input video frame for the same object and the bounding box coordinates in the normalized ROI image, and FIG. 8 is an object bounding detected in ROI-1 and ROI-2 of FIG. 4. This is a diagram showing an example in which boxes are integrated based on an input video frame.

As shown in FIG. 3, the image analysis apparatus 120 ′ according to another embodiment of the present invention includes a video frame acquisition unit 300, an ROI image processing unit 310, an object bounding box merging unit 320, and object tracking Includes part or all of the unit 330 and the event detection unit 340, wherein the ROI image processing unit 310 includes an ROI image extraction unit 311, an ROI image pre-processing unit 312, a DCNN-based object detection unit 313, and It may include a part or all of the object bounding box coordinate conversion unit 314.

Here, "including some or all" means that the above components are configured by hardware (H/W), software (S/W), or a combination thereof, and some components are omitted or some components are different. It means things that can be integrated into components.

Suppose that the user sets at least one rectangular ROI to detect the object of interest when operating the CCTV image analysis device 120' of FIG. 3. In addition, it is assumed that the user sets at least one polygon-shaped event detection area to detect an event of interest. FIG. 4 shows an example in which two ROIs (a rectangular area indicated by a solid line) and one event detection area (a polygonal area indicated by a dotted line) are set in the vicinity of a pedestrian crossing to detect a "pedestrian entering a pedestrian crossing" event from a CCTV image. Show.

As in the example of FIG. 4, the operation flow of the CCTV image analysis device 120 ′ of FIG. 3 in the state where the ROI and event detection area are set is as follows.

In FIG. 3, the video frame acquisition unit 300 continuously acquires a video frame from a video providing device such as a CCTV camera connected to the CCTV image analysis device 120'. For example, when the video providing device connected to the CCTV video analysis device 120' is an IP camera, the video frame acquisition unit 300 continuously receives and decodes the encoded video stream from the IP camera to YUV pixel format or RGB pixel format. Video frames are continuously acquired.

The ROI image extraction unit 311 extracts an image (hereinafter referred to as an ROI image) corresponding to a ROI preset by a user, for example, a control agent, from an input video frame acquired by the video frame acquisition unit 300. 5A and 5B show ROI images corresponding to ROI#1 and ROI#2 of FIG. 4. The ROI image pre-processing unit 312 of FIG. 3 processes the ROI image obtained through the ROI image extraction unit 311 to generate normalized ROI image data used as input to the DCNN-based object detection unit 313. The normalized ROI image has the following characteristics. First, the normalized ROI image resolution matches the input resolution of the object detection DCNN included in the DCNN-based object detection unit 313. In addition, the data of the normalized ROI image is typically composed of three R/G/B channels.

The DCNN-based object detection unit 313 of FIG. 3 uses the object detection DCNN to obtain an object detection result from the normalized ROI image. The object detection result includes the detected object's bounding box coordinate value and object type (or object class) value. FIG. 6 shows an example of a pedestrian object detection result (FIG. 6( b)) obtained through the DCNN-based object detection unit 313 from the normalized ROI image (FIG. 6( a )).

Various object detection DCNN models have been released to date. Typical examples include Faster R-CNN, You Only Look Once (YOLO), and Single Shot Multibox Detector (SSD). The DCNN-based object detection unit 313 performs object detection using the object detection DCNN obtained by training the object detection DCNN model.

, 정규화된 ROI 이미지(720)의 해상도(즉, 객체 검출 DCNN의 입력 해상도)를

, 정규화된 ROI 이미지(720)를 기준으로 하는 특정 객체의 바운딩 박스 좌표 값(721)을

, 입력 비디오 프레임(700) 기준으로 한 동일 객체의 바운딩 박스 좌표 값(711)을

라고 하면,

와

의 관계는 <수학식 1>과 같이 표현된다.The object bounding box coordinate conversion unit 314 of FIG. 3 converts the object bounding box coordinate values (normalized ROI image reference) obtained through the DCNN-based object detection unit 313 into coordinate values of an input video frame reference. As shown in FIG. 7, the coordinate value 710 of the ROI based on the input video frame 700 is

, The resolution of the normalized ROI image 720 (ie, the input resolution of the object detection DCNN)

, The bounding box coordinate value 721 of a specific object based on the normalized ROI image 720

, The bounding box coordinate value 711 of the same object based on the input video frame 700

Speaking of,

Wow

The relationship of is expressed as <Equation 1>.

Meanwhile, the object bounding box integration unit 320 integrates object bounding boxes obtained from each ROI based on an input video frame. If an overlapping region exists between two or more set ROIs, two or more object bounding boxes for the same object may be simultaneously obtained from corresponding ROIs for an object existing in the overlapping region. The object bounding box integration unit 320 checks the overlap between object bounding boxes obtained from different ROIs, and merges the object bounding boxes in which the overlapping exists. Merging of bounding boxes with overlapping is accomplished by obtaining a new bounding box that encloses all of the bounding boxes. 8 is an object bounding box obtained from ROI-1 (FIG. 8(a)) and ROI-2 (FIG. 8(b)) of FIG. 4 by the object bounding box integration unit 320 of FIG. Shows an example of integrating them into an input video frame basis (Fig. 8 (c)).

The object tracking unit 330 performs multiple object tracking using the object bounding box data obtained through the object bounding box integration unit 320.

So far, various image-based multi-object tracking methods have been announced. The method of tracking multiple objects using the bounding box information of objects detected from the input video frame through a separate object detector is called a “Tracking-by-Detection” method. A representative multi-object tracking algorithm that follows the “Tracking-by-Detection” method is an algorithm called Simple Online and Realtime Tracking (SORT). The object tracking unit 330 according to an embodiment of the present invention performs object tracking by using the “tracking-by-detection” multi-object tracking algorithm.

The event detection unit 340 detects an event generated by tracking objects within a designated event detection area. Various types of events can be defined. The simplest event is an event in which an object of a specified type exists in a designated event detection area.

9 is a flowchart of a driving process of an image analysis apparatus according to an embodiment of the present invention.

Referring to FIG. 9 together with FIG. 1 for convenience of description, the image analysis apparatus 120 according to an embodiment of the present invention may respectively display a first region of interest and a second region of interest in an area where objects of interest appear in a region of a captured image. The set setting information is stored (S900). Here, the set ROI may be determined according to the number of object detectors. It is desirable to keep the number of ROIs the same as the object detector, and it is preferable not to exceed the maximum number. Also, the setting range of the region of interest, that is, the setting size, may be determined according to the performance of the object detector.

First of all, it is because DCNN-based object detection is related to data processing speed when setting one ROI as multiple ROI, and above all, the resolution of the object image from the object detector's input image is extracted. When it is inevitable to convert the resolution in order to match with, this can be considered as a solution to problems such as distant objects not being properly detected.

In addition, the image analysis device 120, the first object image of the first region of interest and the second object image of the second region of interest on the basis of the stored setting information in the captured image received from the imaging apparatus 100 DCNN-based object detection is performed on the extracted first object image and the second object image, respectively, and object tracking and events are performed after merging the objects of the first region of interest and the second region of interest detected by object detection. Detection is performed (S910).

As mentioned above, the image analysis apparatus 120 performs a merge operation on multiple ROIs, for example, integrates object bounding boxes obtained from each ROI based on an input video frame. If an overlapping region exists between two or more set ROIs, two or more object bounding boxes for the same object may be simultaneously obtained from corresponding ROIs for an object existing in the overlapping region. The overlap between object bounding boxes obtained from different ROIs is checked, and the object bounding boxes in which overlap exists are merged into one. Merging of bounding boxes with overlapping results in a new bounding box that wraps all of the bounding boxes. For example, when setting coordinates, since the device or system has the corresponding coordinate values, it may be possible to check whether they overlap by comparing two coordinate values.

In addition to the above, the image analysis device 120 of FIG. 1 may perform various operations, and other detailed contents have been sufficiently described above, and thus the contents thereof will be replaced.

On the other hand, that all components constituting the embodiments of the present invention are described as being combined or operated as one, the present invention is not necessarily limited to these embodiments. That is, if it is within the scope of the present invention, all of the components may be selectively combined and operated. In addition, although all of the components may be implemented as one independent hardware, a part or all of the components are selectively combined to perform a part or all of functions combined in one or a plurality of hardware. It may be implemented as a computer program having a. The codes and code segments constituting the computer program may be easily deduced by those skilled in the art of the present invention. Such a computer program is stored in a computer-readable non-transitory computer readable media, and read and executed by a computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium means a medium that stores data semi-permanently and that can be read by a device, rather than a medium that stores data for a short time, such as registers, caches, and memory. . Specifically, the above-described programs may be stored and provided on a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and it is usually in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Of course, various modifications can be implemented by a person having knowledge of these, and these modifications should not be individually understood from the technical idea or prospect of the present invention.

100: photographing device 110: communication network
120, 120': image analysis device 130: control device
140: manager device 200: communication interface
210: control unit 220: multi-ROI image processing unit
230: storage unit 300: video frame acquisition unit
310: ROI image processing unit 320: object bounding box integration unit
330: object tracking unit 340: event detection unit

Claims (12)

A storage unit for storing setting information of a first region of interest (ROI) and a second region of interest, which are respectively set in regions where objects of interest appear in a region of the imaging apparatus; And
The first object image of the first region of interest and the second object image of the second region of interest are extracted from the captured image received from the photographing apparatus, respectively, and the extracted first object image and the DCNN (Deep Convolutional Neural Network)-based object detection is performed on a second object image, and object tracking and event detection are performed after merging the objects of the first region of interest and the second region of interest detected by the object detection Includes a control unit for performing;
The control unit,
When the first region of interest and the second region of interest overlap, the same object in the overlapped region is processed by merging into one object based on the unit frame of the captured image,
The control unit,
A new bounding box is generated by merging the first bounding box and the second bounding box detected for the same object in the overlapping area based on the unit frame,
The control unit,
CCTV image analysis device based on multiple ROI and object detection DCNN that determines the set size of the first region of interest and the second region of interest according to the resolution of the object detector used. delete delete According to claim 1,
When the DCNN-based object detection is performed after the resolution conversion of the first object image and the second object image, the controller uses a multi-ROI and object detection DCNN based object detector that uses an object detector with less loss for distant objects. CCTV video analysis device. According to claim 4,
The control unit includes a region of interest (ROI) image processing unit for the resolution conversion, and the converted resolution is a multi-ROI and object detection DCNN-based CCTV image analysis device that matches the resolution of the input image of the DCNN of the object detector. delete Storing setting information of a first region of interest (ROI) and a second region of interest, respectively, which are set in regions where objects of interest appear in a region of the imaging apparatus;
The controller extracts the first object image of the first region of interest and the second object image of the second region of interest based on the stored setting information from the captured image received from the photographing apparatus, respectively, and extracts the first object DCNN-based object detection is performed on the image and the second object image, and object tracking and event detection are performed after merging the objects of the first region of interest and the second region of interest detected by the object detection. step; And
Including the step of determining the set size of the first region of interest and the second region of interest according to the resolution of the object detector used;
The step of performing,
When the first region of interest and the second region of interest overlap, the same object in the overlapping region is merged and processed into one object based on the unit frame of the captured image; And
Generating a new bounding box by merging the first bounding box and the second bounding box, respectively, detected for the same object in the overlapping region based on the unit frame;
A method of driving a multi-ROID and object detection DCNN-based CCTV image analysis device. delete delete The method of claim 7,
The step of performing,
Multi-ROI and object detection DCNN-based CCTV image analysis using an object detector with less loss for distant objects when performing DCNN-based object detection after resolution conversion for the first object image and the second object image How the device is driven. The method of claim 10,
A method of driving a CCTV image analysis apparatus based on a multi-ROI and object detection DCNN based on the resolution of the input image of the DCNN of the object detector, and the converted resolution performed by the ROI image processing unit. delete

Images