KR102201241B1 - Apaptive Object Recognizing Apparatus and Method for Processing Data Real Time In Multi Channel Video - Google Patents

Abstract

An adaptive object recognition device and method for processing data of a multichannel video stream in real-time efficiently performs real-time object recognition of a CCTV image by receiving a channel video frame, adaptively changes a channel selection list for selecting the order of input channels receiving image frames according to an object recognition result, and processes multi-channel CCTV video frames without increasing computing resources.

Description

[Apaptive Object Recognizing Apparatus and Method for Processing Data Real Time In Multi Channel Video}

The present invention relates to an entity recognition apparatus, in particular, without increasing computing resources by adaptively changing a channel selection list for receiving a multichannel video frame and selecting an order of an input channel to receive an image frame according to an entity recognition result. The present invention relates to an adaptive entity recognition apparatus and method for processing data in real time on a multi-channel video stream that efficiently performs real-time entity recognition of CCTV images by processing multi-channel CCTV image frames.

In general, CCTV (Closed Circuit Television) systems install cameras in various places that require indoor and outdoor security, and display video signals received from the cameras on multiple TV monitors located in a single place, so that many places with fewer surveillance personnel. Can be monitored at the same time. In addition, the video signal received from the camera is recorded in a storage medium such as a VCR, and can be used as important evidence as well as helpful in arresting the criminal in case of theft.

However, since the video in CCTV is recorded in an analog format, it is difficult to apply an image processing technique useful for detailed investigation, such as motion detection and object recognition. With the recent development of digital technology, techniques for digitizing and processing video signals have been introduced.

Digital Video Recorder (DVR) or Network Video Recorder (NVR) system converts analog video signals input through cameras into digital signals and compresses/compresses video with MPEG, an international moving picture compression method. Restore, save and play.

Such a DVR or NVR reproduces stored image information to confirm an event, and is connected to a plurality of CCTV cameras to store multi-channel image information.

Therefore, the conventional unmanned surveillance CCTV system uses image processing technology to monitor intrusion, roaming, etc. in a specific area in real time, and automatically generates an alarm of a suspected situation.

Such CCTV systems use machine learning-based entity recognition technology for real-time unmanned surveillance. In recent years, image processing technology based on Deep Learning object recognition algorithms have been developed, and representatively, SSD (Single Object Detector) and YOLO (You Only Look Once).

This deep learning-based algorithm performs deep learning analysis on one image file and outputs one detection result.In order to apply to multiple CCTV channels, the entity recognition module must operate in parallel, so a large amount of memory, GPU, It requires hardware resources such as MCU, and there is a problem that expensive cost is required for additional server construction.

However, most DVRs and NVRs currently use general PCs or build systems with PC-class performance. Therefore, most DVRs and NVRs have limitations in that they cannot apply or operate properly based on a deep learning-based entity recognition algorithm for real-time entity recognition from multi-channel CCTV.

Korean Patent Registration No. 10-1930940

In order to solve such a problem, the present invention adaptively changes a channel selection list that selects an order of input channels for receiving multi-channel video frames and receiving image frames according to an entity recognition result, without increasing computing resources. An object of the present invention is to provide an adaptive entity recognition apparatus and method for processing data in real time on a multi-channel video stream that efficiently performs real-time entity recognition of CCTV images by processing multi-channel CCTV image frames.

An adaptive entity recognition apparatus for processing data in real time on a multi-channel video stream according to a feature of the present invention for achieving the above object,

A mux to which output signal lines and input ports of a plurality of CCTV (Closed Circuit Television) cameras are connected as a video channel multiplexer;

An entity recognition module for receiving an image frame from one of the plurality of CCTV cameras from the mux, estimating the existence of an entity in the received image frame, and outputting an entity recognition result;

A detection result unit for adaptively changing a channel selection list for selecting an order of input channels for receiving an image frame based on the output object recognition result; And

And an adaptive channel selection unit generating a channel selection signal according to the changed channel selection list and transmitting it to the mux,

The mux is characterized in that it selectively receives an image frame through an input channel selected according to a channel selection signal of the adaptive channel selection unit.

An adaptive entity recognition apparatus for processing data in real time on a multi-channel video stream according to a feature of the present invention,

A mux to which output signal lines and input ports of a plurality of CCTV (Closed Circuit Television) cameras are connected as a video channel multiplexer;

Receives an image frame of one CCTV camera from among the plurality of CCTV cameras from the MUX, estimates the existence of an entity in the received image frame, outputs an entity recognition result, and inputs image information by accessing one input channel An entity recognition module that receives and measures a processing capability indicating a maximum frame recognition rate of the one input channel;

A detection result unit for adaptively changing a channel selection list for selecting an order of input channels for receiving an image frame based on the output object recognition result; And

And an adaptive channel selection unit generating a channel selection signal according to the changed channel selection list and transmitting it to the mux,

The channel selection list is characterized in that a priority order for selecting an order of input channels within the processing capability is dynamically allocated.

An adaptive entity recognition method for processing data in real time on a multi-channel video stream according to a feature of the present invention,

Sequentially receiving image frames from each of the CCTV cameras through a mux to which output signal lines and input ports of a plurality of CCTV (Closed Circuit Television) cameras are connected, respectively;

Estimating the existence of an object from the image frame received from each of the CCTV cameras using an object recognition module, and outputting object recognition results, respectively;

Adaptively changing a channel selection list for selecting an order of input channels for receiving image frames based on the respective object recognition results; And

And generating a channel selection signal according to the changed channel selection list and transmitting it to the MUX, and the MUX selectively receiving an image frame through an input channel selected according to the channel selection signal.

According to the above-described configuration, the present invention has the effect of efficiently performing real-time object recognition of CCTV images by processing multi-channel CCTV image frames without increasing computing resources.

The present invention has the effect of being compatible with various deep learning-based entity recognition modules through a modular software configuration of an adaptive entity recognition device.

The present invention has an effect that a sudden decrease in performance or an increase in hardware specifications according to an increase in the number of CCTVs is unnecessary.

1 is a diagram showing the configuration of an adaptive entity recognition apparatus for processing data in real time on a multi-channel video stream according to an embodiment of the present invention.
2 is a diagram illustrating an extended channel selection list according to an embodiment of the present invention.
FIG. 3 is a diagram illustrating a mode in which an image frame is selected and inputted according to a channel selection list according to an embodiment of the present invention.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless otherwise stated.

1 is a diagram showing a configuration of an adaptive entity recognition apparatus that processes data in real time on a multi-channel video stream according to an embodiment of the present invention, and FIG. 2 is a view of an expanded channel selection list according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a mode in which an image frame is selected and inputted according to a channel selection list according to an embodiment of the present invention.

The adaptive entity recognition apparatus 100 for processing data on a multi-channel video stream in real time according to an embodiment of the present invention includes a Mux 110, an entity recognition module 120, a detection result unit 130, and an event log. It includes a unit 140 and an adaptive channel selection unit 150.

The mux (110) is a video channel multiplexer (Video Channel Multiplexer), the output signal lines and input ports of a plurality of CCTV (Closed Circuit Television) cameras 160 are connected, respectively, to the output signal lines of the CCTV (160). Input signal lines of a digital video recorder (DVR) or a network video recorder (NVR) 170 are respectively connected.

As shown in FIG. 3, the mux 110 receives only the frames of the selected input channel and discards the frames of the remaining input channels.

The DVR/NVR 170 is connected to a plurality of CCTV cameras 160 to store multi-channel image frames.

When generating a selection signal for one input channel, the mux 110 receives and outputs an image frame from the CCTV camera 160 electrically connected to the corresponding input channel.

In other words, the mux 110 selects one input channel of the CCTV camera 160 from the plurality of CCTV cameras 160 and transmits the image frame to the entity recognition module 120.

The entity recognition module 120 includes Deep Neural Networks (DNN), Convolutional Deep Neural Networks (CNN), Reccurent Neural Networks (RNNs), and Deep Belief Networks (DBN). A feature map is extracted from the input image using any one of the neural networks.

The entity recognition module 120 may generate a feature map using a model that has already been learned by a learning unit based on deep learning. Deep learning is a set of machine learning algorithms that attempts high-level abstractions (abstractions, the task of summarizing key contents or functions in a large amount of data or complex data) through a combination of several nonlinear transducers. Is defined.

The entity recognition module 120 extracts a region in which an entity is estimated to exist from the image frame, and extracts a feature map representing a feature from the extracted region.

The entity recognition module 120 extracts at least one area from the image in which the existence of an entity is estimated based on the extracted feature map. A method of extracting the region may include, for example, faster RCNN, Single Shot MultiBox Detector (SSD), You Only Look Once (Yolo).

The entity recognition module 120 selects a feature map including the coordinates of each region class of the image among the feature maps, identifies the coordinates that distinguish the region from the selected feature map, and estimates the existence of the object based on the identified coordinates. It can be extracted to the area that can be used.

The entity recognition module 120 may set one or two or more of various entities such as objects, people, and animals.

In addition, the object recognition module 120 may display each of the extracted at least one area as a bounding box surrounding the outermost part of the object.

Each bounding box indicates that there is a possibility of an entity at the location of the bounding box in the image.

The object recognition module 120 receives a frame representing image information as an input and outputs result information obtained by using the position coordinates ((X1, Y1), (X2, Y2)) of the object within the frame as a bounding box.

When an image frame is received and played, the entity recognition module 120 processes 15 to 30 frames per second to output image information in which the bounding box is displayed.

The entity recognition module 120 receives the image frame of the input channel selected from the mux 110 by the adaptive channel selection unit 150, generates an entity recognition result through an entity recognition algorithm, and sends the image frame to the detection result unit 130. send.

In other words, the entity recognition module 120 receives an image frame of one CCTV camera 160 from among a plurality of CCTV cameras 160 from the MUX 110, and recognizes the entity by estimating the existence of the entity from the received image frame. Print the result.

Entity recognition module 120 is input from each CCTV camera 160, assuming that there are N input channels to perform monitoring, when accessing one input channel to receive video frames and play, Processing Capability ) Is measured as the result.

Here, the processing performance refers to the system performance of processing C frame/sec video.

For example, the object recognition module 120 is measured as a result of processing a video, such as 25 frames per second, 30 frames.

This processing performance may vary depending on the processing specifications of each system (PC, NVR, DVR).

The detection result unit 130 may receive processing performance (C frame/sec) from the entity recognition module 120 and set a deadline time.

Here, the deadline time Td represents the maximum allowable time according to the movement of the object in the image frame, and is set to, for example, 2 seconds or 3 seconds.

The detection result unit 130 calculates the maximum size of the channel selection list 152 by multiplying the processing performance by the deadline time (M = C × Td).

The detection result unit 130 sets the maximum supported channel to be equal to or smaller than the maximum size (M), and sets the event tick to 1/C sec. Here, the event tick represents an event interval time at which the adaptive channel selection unit 150 selects a channel from the mux 110.

For example, the adaptive channel selection unit 150 selects a channel from the mux 110 at every 1/C sec (event interval time) such as 1 channel, 2 channels, 4 channels, and 5 channels. Thus, the event tick follows the frame rate.

The adaptive channel selection unit 150 initially sets the size of the channel selection list 152 to N so that all channels are selected once in order.

The adaptive channel selection unit 150 generates a channel selection signal that is selected once in order for all channels and transmits it to the mux 110.

The event log unit 140 stores corresponding information in a logger when object recognition occurs. The event log unit 140 records input channel information, time, number of detected individuals, and bounding box information.

The entity recognition module 120 generates an entity recognition result through the entity recognition algorithm when the entity recognition for all input channels from the first input channel selected by the adaptive channel selection unit 150 to the nth input channel is completed. Then, it is transmitted to the detection result unit 130.

When entity recognition for all channels from the first channel to the nth channel is completed, it is expressed that one round is over.

The detection result unit 130 receives and stores the object recognition result for each individual image frame of one round. The entity recognition result may include a bounding box ((X1, Y1), (X2, Y2)), the number of recognized entities, and an entity recognition probability.

The detection result unit 130 may adaptively change the channel selection list 152 for the next round by analyzing the received entity recognition result.

The detection result unit 130 adaptively changes the channel selection list 152 for selecting the order of the input channels receiving the image frames based on the output object recognition result and transmits the adaptive channel selection unit 150 to the adaptive channel selection unit 150. .

The adaptive channel selection unit 150 generates a channel selection signal according to the changed channel selection list 152 and transmits it to the mux 110. The mux 110 selectively receives an image frame through an input channel selected according to a channel selection signal of the adaptive channel selection unit 150.

The detection result unit 130 stores the object recognition result generated during one round, analyzes the object recognition result, and when the object is not recognized in the image frames of all input channels, the channel selection list 152 of N channels again ) Is generated and transmitted to the adaptive channel selection unit 150.

When detecting one or more entities by analyzing the entity recognition result, the detection result unit 130 expands to M (C × Td), which is the maximum size of the channel selection list 152, and expands the expanded channel selection list 152. It is transmitted to the adaptive channel selection unit 150.

C is the maximum frame recognition rate of one input channel in the entity recognition module 120, and Td represents a preset maximum allowable time according to movement of an entity within an image frame.

Here, the expanded channel selection list 152 is divided into two areas: a private area (PA) 152a and a non-privileged area (NPA) 152b, and the PA 152a is a channel in which an entity is recognized in this round. It is a selection order list area for fields, and the NPA 152b indicates an area for channels in which an entity is not recognized in this round.

The PA 152a is a first list area indicating a selection order for input channels in which an entity is recognized, and the NPA 152b is a second list area indicating a selection order for input channels in which an entity is not recognized.

When one round is completed, the detection result unit 130 stores the recognized entities for each input channel, the entity recognition probability, and bounding box information for each entity.

The detection result unit 130 extracts, for each channel, a maximum value of an entity recognition probability of entities recognized for each input channel from among entity recognition results.

When an entity is recognized in at least one input channel, the detection result unit 130 expands the size of the channel selection list 152 to the maximum size (M = C × Td) and calculates the size of the PA section.

The detection result unit 130 is calculated as the size of the PA section 152a minus the PA = M-NPA (N-D) section. Here, D represents the number of recognized input channels.

The size of the PA section 152a increases as the entity is recognized, and decreases as the entity is not recognized.

The PA section 152a is to perform the object recognition process by inputting the image frame of the input channel in which the object is recognized to the object recognition module at least once more than the channel in which the object is not recognized.

The detection result unit 130 calculates a quotient obtained by dividing the PA section 152a by the number of input channels in which the entity is recognized. For example, assuming that the PA section 152a is 14 and the number of input channels D in which the entity is recognized is 3, the divided quotient is 14/3 = 4.

In the PA section 152a, input channels in which the entity is recognized are evenly and alternately allocated.

For example, as shown in FIG. 2, assuming that the PA section 152a is 13 and the number of input channels D for which channels are recognized are 3, 13/3 (PA/ D) = 4, that is, three channels in which the entity is recognized are evenly alternately allocated within the PA section 152a.

In the PA interval, the individual is assigned as much as multiplied by the quotient (number 4) obtained by dividing the recognized channel (ch[1], ch[2], ch[3]). That is, ch[1], ch[2], ch[3], ch[1], ch[2], ch[3], ch[1], ch[2], ch[3], ch[1 ], ch[2], ch[3] are assigned.

The detection result unit 130 calculates the remainder of the PA section 152a divided by the number of input channels in which the entity is recognized. For example, assuming that the PA section 152a is 13 and the number of input channels D in which the entity is recognized is 3, the remainder is 13% 3 = 1.

The remaining sections (one) of the PA section 152a are sections remaining after the input channel in which the entity is recognized is allocated, and the input channels are allocated in the order of the highest entity recognition probability among the input channels in which the entity is recognized.

In other words, for the remaining sections, input channels may be allocated in the order of the highest object recognition probability by comparing the maximum value of the object recognition probability among input channels in which the object is recognized.

For example, as shown in FIG. 2, ch[1] having a high probability of recognizing an individual is allocated to the remaining section (13% 3 = 1) in the PA.

The detection result unit 130 sequentially allocates ch[4] to ch[N] to the NPA section 152b once. In this way, the detection result unit 130 transmits the changed channel selection list 152 to the adaptive channel selection unit 150 when channel allocation is completed in the PA section 152a and the NPA section 152b.

The adaptive channel selection unit 150 receives the changed channel selection list 152 from the detection result unit 130 and controls the mux 110 based on the changed channel selection list 152 to perform the next round.

The channel selection list 152 may dynamically allocate a priority for selecting an order of input channels within the processing performance by the detection result unit 130. That is, it is divided into multiple input channels within the maximum frame.

In another embodiment, the DVR/NVR 170 includes a Mux 110, an entity recognition module 120, a detection result unit 130, an event log unit 140, and an adaptive channel selection unit 150. ) May be included, and these components may be implemented as hardware or software modules.

In the present invention, the adaptive entity recognition device can be compatible with various deep learning-based entity recognition modules through a modular software configuration, and there is an effect that a rapid deterioration in performance or an increase in hardware specifications according to an increase in the number of CCTVs is unnecessary.

In the above, embodiments of the present invention are not implemented only through an apparatus and/or method, but may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention, a recording medium in which the program is recorded, etc. In addition, this implementation can be easily implemented by an expert in the technical field to which the present invention belongs from the description of the above-described embodiment.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

100: adaptive object recognition device 110: mux
120: entity recognition module 130: detection result unit
140: event log unit 150: adaptive channel selection unit
160: CCTV 170: DVR/NVR

Claims (11)

A mux to which output signal lines and input ports of a plurality of CCTV (Closed Circuit Television) cameras are connected as a video channel multiplexer;
An entity recognition module for receiving an image frame from one of the plurality of CCTV cameras from the mux, estimating the existence of an entity in the received image frame, and outputting an entity recognition result;
A detection result unit for adaptively changing a channel selection list for selecting an order of input channels for receiving an image frame based on the output object recognition result; And
And an adaptive channel selection unit generating a channel selection signal according to the changed channel selection list and transmitting it to the mux,
The detection result unit expands or decreases the size of the channel selection list according to the entity recognition result,
Wherein the mux selectively receives an image frame through an input channel selected according to a channel selection signal of the adaptive channel selection unit. A mux to which output signal lines and input ports of a plurality of CCTV (Closed Circuit Television) cameras are connected as a video channel multiplexer;
Receives an image frame of one CCTV camera from among the plurality of CCTV cameras from the MUX, estimates the existence of an entity in the received image frame, outputs an entity recognition result, and inputs image information by accessing one input channel An entity recognition module that receives and measures a processing capability indicating a maximum frame recognition rate of the one input channel;
A detection result unit for adaptively changing a channel selection list for selecting an order of input channels for receiving an image frame based on the output object recognition result; And
And an adaptive channel selection unit generating a channel selection signal according to the changed channel selection list and transmitting it to the mux,
And the channel selection list dynamically allocates a priority for selecting an order of input channels within the processing capability. The method of claim 2,
The detection result unit adaptive entity recognition apparatus, wherein the size of the channel selection list is expanded or reduced according to the entity recognition result. The method according to claim 1 or 2,
The detection result unit expands the channel selection list to a maximum size when at least one entity is detected as the entity recognition result, and the maximum size is a maximum frame recognition rate of one input channel in the entity recognition module, and an image frame. Adaptive object recognition device, characterized in that multiplied by a predetermined maximum allowable time according to the movement of the object within. The method of claim 1 or 3,
The channel selection list includes a first list area indicating a selection order for input channels in which an entity is recognized, and a second list area indicating a selection order for input channels for which an entity is not recognized,
The detection result unit calculates a quotient obtained by dividing the size of the first list area by the number of input channels in which the entity is recognized, and alternately allocates the input channels in which the entity is recognized to the first list area, and the An adaptive entity recognition device, characterized in that sequentially allocating an input channel in which an entity is recognized by multiplying the calculated quotient. The method of claim 5,
The detection result unit calculates a remainder of the size of the first list area divided by the number of input channels in which the entity is recognized, and the remaining section calculated in the first list area is entity recognition among the input channels in which the entity is recognized. An adaptive entity recognition device, characterized in that the input channels are allocated in an order of high probability. Sequentially receiving an image frame from each of the CCTV cameras through a mux to which output signal lines of a plurality of CCTV (Closed Circuit Television) cameras and an input port are respectively connected;
Estimating the existence of an object from the image frame received from each of the CCTV cameras using an object recognition module, and outputting each object recognition result;
Adaptively changing a channel selection list for selecting an order of input channels for receiving an image frame based on the respective object recognition results; And
Generating a channel selection signal according to the changed channel selection list and transmitting it to the MUX, the MUX selectively receiving an image frame through an input channel selected according to the channel selection signal,
Outputting each of the object recognition results,
When the entity recognition module accesses one input channel to receive image information and plays it, the processing capability indicating the maximum frame recognition rate of one input channel is measured, and the maximum allowance according to the movement of the entity within the image frame An adaptive object recognition method comprising the step of setting a time. delete The method of claim 7,
Generating an entity recognition result by sequentially performing entity recognition for all input channels from the first input channel to the nth input channel;
Extracting a maximum value of an entity recognition probability of recognized entities for each of the input channels; And
And expanding the channel selection list to a maximum size by multiplying the processing performance and the maximum allowable time when at least one entity is detected in the generated entity recognition result. The method of claim 9,
The expanded channel selection list includes a first list area indicating a selection order for input channels in which an entity is recognized, and a second list area indicating a selection order for input channels for which an entity is not recognized,
A quotient is calculated by dividing the size of the first list area by the number of input channels in which the entity is recognized, and the input channels in which the entity is recognized are equally alternately allocated to the first list area, and the entity is recognized. And sequentially allocating an input channel by multiplying the calculated quotient. The method of claim 10,
The remainder of the size of the first list area divided by the number of input channels in which the entity is recognized is calculated, and the remaining section calculated in the first list area is the maximum of the entity recognition probability among the input channels in which the entity is recognized. And allocating input channels in an order of high probability of recognizing an entity by comparing values.

Images