KR101995107B1 - Method and system for artificial intelligence based video surveillance using deep learning - Google Patents

Abstract

An artificial intelligence based video surveillance method and system using deep learning is disclosed. The artificial intelligence based video surveillance system according to an embodiment of the present invention includes a detector for detecting at least one object preset for a surveillance camera image using a predefined sensing deep learning network; A recognition unit for recognizing information about the detected object based on a detection result for the object; And a tracking unit for tracking the detected object based on a predefined tracking deep learning network and a detection result for the detected object, wherein the detecting unit detects a heat map based on a convolution neural network (CNN) the object can be detected using a region proposal extraction including a heat map generation method.

Description

TECHNICAL FIELD [0001] The present invention relates to an artificial intelligence based video surveillance method and system using deep learning,

The present invention relates to artificial intelligence based video surveillance technology for smart control, and more specifically, to an artificial intelligence based video surveillance technology capable of detecting, recognizing, and tracking objects based on artificial intelligent deep learning technology for input video captured by CCTV ≪ / RTI >

CCTV monitoring (surveillance) is being activated to prevent various crimes and accidents, and surveillance systems linked to smart phones have been developed and are being applied to everyday living spaces. As the coverage of CCTV surveillance is expanded, a huge amount of video data is being produced in one surveillance system. Therefore, it is impossible to detect a danger and analyze situation by monitoring a large amount of video data by one manager. In the current market, smart monitoring function, which is the core of 4th generation CCTV, is being applied as the latest technology.

As a smart function, image recognition technology such as human detection and vehicle detection is applied to CCTV, but analysis of vast amount of image data is limited. Therefore, there is a need for a video monitoring method based on advanced image recognition technology capable of intelligently analyzing vast CCTV surveillance images with human judgment ability.

Conventional CCTV image recognition technology has a problem of practicality due to degradation of recognition performance due to change of weather condition, surrounding illumination condition, and shape of an object in an actual environment when applied in a real environment. In addition, the existing CCTV video recognition technology is a basic object detection technology and it helps to manage the surveillance system, but it has limitations. For a vast amount of video analysis, there is a demand for intelligent image recognition technology such as situation recognition and prediction, as well as high-performance object detection and recognition technology.

Therefore, there is a need for an artificial intelligent CCTV video surveillance technology for intelligent image recognition such as situation prediction together with high-performance object detection and recognition technology.

Embodiments of the present invention provide an artificial intelligence based video surveillance method capable of detecting, recognizing, and tracking an object, for example, a person, a face, a vehicle, a fire, etc., based on artificial intelligent deep learning technology, And a system.

The artificial intelligence based video surveillance system according to an embodiment of the present invention includes a detector for detecting at least one object preset for a surveillance camera image using a predefined sensing deep learning network; A recognition unit for recognizing information about the detected object based on a detection result for the object; And a tracking unit for tracking the detected object based on a predefined tracking deep learning network and a detection result for the detected object.

The detection unit may detect the object using region proposal extraction including a method of generating a heat map based on a convolution neural network (CNN).

The detecting unit may detect the object by extracting an object region and adjusting a bounding box based on a multi-tasking learning method using attributes of the object.

The detection unit may perform the region suggestion extraction on the surveillance camera image and extract the characteristics of each of the region suggestions through the convolutional neural network based feature extraction from the extracted region suggestions to detect the object .

The tracking unit measures the previous tracking result of the detected object and the reliability of the current tracking result, and updates the deep learning model on-line when the measured reliability is less than a predetermined value, thereby tracking the detected object.

Further, the artificial intelligence based video surveillance system according to an embodiment of the present invention may include an accident prediction and deep learning network that is interlocked with the detection unit and the recognition unit and is previously learned about an accident and a danger, And a prediction unit for predicting the risk factors based on the information.

According to an embodiment of the present invention, an artificial intelligence based video surveillance method includes: detecting at least one object preset for a surveillance camera image using a predefined sensing deep learning network; Recognizing information about the detected object based on a detection result for a predefined deep learning network and the object; And tracking the detected object based on a predefined trace deep learning network and a detection result for the detected object.

The detecting step may detect the object using a region proposal extraction including a method of generating a heat map based on a convolution neural network (CNN).

The detecting may detect the object by extracting an object region and adjusting a bounding box based on a multi-tasking learning method using attributes of the object.

The detecting step may include performing an area suggestion extraction on the surveillance camera image, extracting characteristics of each of the area suggestions through convolutional neural network-based feature extraction from the extracted area suggestions, .

The tracking step may track the detected object by measuring a previous tracking result for the detected object and a reliability of the current tracking result and updating the deep learning model online if the measured reliability is less than a predetermined value .

Further, the artificial intelligence based video surveillance method according to an embodiment of the present invention may include an accident prediction and deep learning network that is learned in advance about an accident and a risk, a step of predicting a risk element based on the detected object and the recognized object information Lt; RTI ID = 0.0 >

According to embodiments of the present invention, an object, for example, a person, a face, a vehicle, a fire, etc., can be detected, recognized, and tracked on the basis of an artificial intelligent deep learning technique for input video shot by CCTV.

According to the embodiments of the present invention, high-performance video surveillance is possible, and high-level vision technology is implemented through a predictive deep network, it is possible to acquire core technology for intelligent video surveillance in the surveillance system market, have.

In addition, it can enhance technological impact by implementing artificial intelligence prediction technology with deep learning technology that could not be done in the existing 4th generation smart CCTV.

According to embodiments of the present invention, the present invention can be applied to various surveillance camera environments such as CCTV, drone camera, and body camera.

1 is a conceptual diagram of an artificial intelligence based video surveillance system according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating an example of region proposal extraction through CNN-based heat map generation.
FIG. 3 shows an example of an artificial intelligence based human detection framework.
4 is a diagram illustrating an example of an artificial intelligence based face detection framework.
5 is a conceptual diagram of an embodiment of an artificial intelligence based surveillance system including an accident prediction.
FIG. 6 shows a configuration of an embodiment of an object recognition and accident prediction deep learning network.
FIG. 7 shows an example of an accident predicting deep learning network learning.
8 is a flowchart illustrating an artificial intelligence based video surveillance method according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.

Embodiments of the present invention relate to an artificial intelligence based video surveillance technology for smart control, and it is an object of the present invention to detect, recognize and track objects based on artificial intelligent deep learning technology for input video captured by CCTV do.

At this time, the present invention can constitute deep learning based deep networks such as video monitoring core technologies such as object (or object) detection, recognition, tracking and prediction.

1 is a conceptual diagram of an artificial intelligence based video surveillance system according to an embodiment of the present invention.

As shown in FIG. 1, an artificial intelligence based video surveillance system according to an embodiment of the present invention includes a detection unit, a recognition unit, and a tracking unit.

The detection unit is a constituent means for detecting an object based on a deep learning, and performs human detection, face detection, vehicle detection, and license plate detection using a deep network.

Here, the detection unit can perform automatic object detection at a predetermined time interval for the input video, apply real-time processing to the detected object using the deep learning network, and minimize the area in which the object is searched Can be applied to design and train deep networks.

The detection unit automatically detects the object of the surveillance camera image including the identifiable object using the detection deep network.

Here, the automatically detected object (or object) may include a person, a face, a car, a license plate, and the like.

Specifically, the detection unit detects a human region using a deep learning-based network, and detects a face region using a face detection deep network in a detected human region. Likewise, the detection unit detects automobiles existing in the image using the automobile detection deep network, and detects the license plate area within the detected automobile image.

At this time, the detection unit can output the position information, the size information, and the reliability value of detection of each detected object.

Furthermore, since the detection unit does not know the size of the object in advance, it is possible to check the presence or absence of the object while searching the object area on each of the scales preset in the image.

In addition, the detection unit can use a high-speed human detection network based on a region of interest to automatically detect people in the input image and quickly extract human-related information including various information such as a person's position, posture, .

Human detection in a real environment is very important as well as accuracy of detection performance. For example, in autonomous vehicles, pedestrian detection speed is an important factor that determines the occurrence of an accident. In an environment such as a surveillance system, it is necessary to provide not only detection but also various characteristics of the detected person.

To this end, the detection unit may use region proposal extraction to extract candidate regions at a high speed, as shown in FIG. As an example of the area suggestion extraction method, a heat map generation method based on a convolution neural network (CNN) can be used.

Here, as shown in FIG. 3, a CNN-based hit map generation method generates a heat map indicating a two-dimensional map representing a probability value for an object candidate region through a CNN-based network, Can be extracted.

The proposed area network in Fig. 2 is a region proposed network composed of a total of five convolution layers indicated by yellow. In the first and second convolution layers, a max pooling layer indicated in green and a normalization layer indicated in blue normalization layer can be connected.

At this time, a probability distribution map obtained from the fifth convolution layer is obtained, and regions having high probability values are extracted using the result, thereby finally obtaining the domain proposal as shown on the rightmost side. Here, the portion indicated by blue in the probability distribution map shown in FIG. 2 may represent a probability value of '0' and a higher probability value closer to '1' toward red.

In order to efficiently solve the problem of classifying the detected candidate region as a real person, the detection unit may perform a multitasking learning process using human attributes such as human attributes such as pedestrian size, pose, hat, multi-tasking learning method.

Here, the multitasking learning method is a step of extracting characteristics from the extracted domain proposals through a CNN-based feature extraction and performing final detection. The CNN-based network includes various methods related to human detection For example, pedestrian size, pedestrian pose, etc. can be studied in a direction that satisfies commonly, so that a more effective characteristic can be learned. That is, if the object function related to the existing person detection is as shown in Equation (1) below, the network can be learned so as to satisfy the object function expressed by Equation (2) by extending Equation (1) to T tasks.

[Equation 1]

&Quot; (2) "

는 i번째 학습데이터의 특성 벡터(feature vector)를 의미하며,

는 i번째 학습데이터의 t번째 태스크의 라벨(label) 값을 의미하고,

는 t번째 태스크의 로스 함수(loss function)를 의미할 수 있다.Here, N means the number of learning data,

Denotes a feature vector of the ith learning data,

Denotes the label value of the t-th task of the i-th learning data,

May be a loss function of the t-th task.

That is, as shown in FIG. 3, the artificial intelligence-based human detection framework performs region suggestion extraction through the network shown in FIG. 2 and extracts CNN-based characteristics from the extracted domain suggestions, By extracting the characteristics of each of the domain suggestions, the final detection is performed.

At this time, the CNN network for the characteristic extraction can improve the performance for the feature extraction by learning with the multitasking learning method as shown in FIG.

The face detection module of the detection unit detects a face region using a face detection deep network for an image input including a human region acquired by the human detection module, and detects position information, size information, and detection reliability Output the value.

In this case, since the face detection module performs face detection on the human area acquired by the human detection module, fast and efficient face detection can be performed due to the reduction of the search area.

As shown in FIG. 4, the face detection module can extract a face candidate region by extracting a heat map. As shown in FIG. 4, in order to perform high-performance face detection on faces displayed in various sizes, Extract face map with scale.

At this time, extracting the heat map at a single scale can be done via a network similar to that of FIG.

In a DIP network that determines whether a face candidate region is finally a face or not, a multi-task learning method is used so that a bounding box indicating a detected region is positioned at a more accurate position.

That is, as shown in FIG. 4, the face detection module extracts a multi-scale heat map based on the face component information for the human region extracted through the human detection module, and extracts a face candidate region through the extracted multi-scale heat map Extracting a final face area and adjusting a bounding box through a multitask running method on the extracted face candidate region, thereby providing a face detection result.

The vehicle detection module and the license plate detection module of the detection unit detect an automobile area in the input image through a deep learning network in a manner similar to the human detection module and the face detection module described above and detect the license plate area in the detected automobile area .

The recognition unit recognizes, for example, whether a person is present in the object detected through the deep learning network with respect to the object detection result detected by the detection unit, for example, through the identification information (ID) of the person, It automatically recognizes the number information on the license plate.

In this case, the recognition unit may perform image quality measurement to perform highly reliable recognition of an image having good image quality before performing recognition, and the image quality to be evaluated may include resolution, brightness, posture information, (blur) information.

The recognition unit performs recognition based on the measured image quality by the deep learning structure for the recognizable object, outputs the DB registration status, the human ID, and the recognition reliability value for the face, Print number information.

In this case, as in the detection network, the recognition unit performs automatic object recognition for a predetermined period of time for real-time processing, can interlock with the tracking network, and has multiple information that can be utilized in the image for high- Feature, color information, and the like can be used.

The tracking unit performs tracking on the detected object using the tracking network or performs tracking on the object initialized by the user in the image.

Here, the tracking unit can output the position information, the size information, and the motion information of the object through the tracking of the object, and the object is drifted by measuring the reliability of the tracking result for high reliability tracking Can be minimized.

In addition, the tracker can use a relatively shallow deep-run structure for real-time object tracking and a deep-run structure that enables on-line learning for high-reliability tracking.

Object Tracking Describing on-line learning in deep networks, tracking objects only with previously learned models can lead to tracking drift when lighting changes, attitude changes, and clipping occur, The reliability of object tracking can be measured through comparison between the results and the currently tracked results, and the on-line deep-learning model can be updated when the reliability falls below a certain level.

At this time, since the tracking unit requires a large amount of computation time when updating all the layers, it is possible to update the model by performing fine-tuning on CNN's fully-connected layer.

In order to update the model online, the tracking unit sets the sample corresponding to the position of the current object as positive, places the positive peripheral region as negative, performs learning, Positive and Negative sample information for the online learning is accumulated. In this way, the tracking unit can prevent the tracking result from drifting due to a phenomenon such as a clogging.

As described above, the artificial intelligence based video surveillance system according to an embodiment of the present invention configures the configuration means related to object detection, recognition, and tracking using a deep learning based network, thereby enabling high level vision technology And it can secure the core technology of the intelligent video surveillance system of the surveillance system.

An example of the operation scenario of the artificial intelligence based video surveillance system according to the present invention is to detect a person through DetectionNet which is a detection part which is an object detection network when a suspicious person appears in an image while shooting with a CCTV surveillance camera, As shown in FIG.

In this case, when the recognition network is continuously performed with respect to the detected person, the operation time is long and the performance deterioration due to the low quality image exists. Therefore, the system can operate in the following manner.

TrackingNet, a tracking network, tracks people who are detected in TrackingNet to follow moving lines continuously.

At this time, if the quality of the face being tracked is good, for example, when a situation in which there is no obstruction and the lighting condition is not bad is detected, RecognitionNet of Recognition Net, which is a recognition network, carries out recognition of the face and identifies the person .

If the person is judged to be a suspicious person, it can be reported to the surveillance center by judging whether the person is a safe person registered in advance or a suspicious person present in the criminal database.

In addition, the artificial intelligence based monitoring system using deep learning according to the present invention may be configured as an entire system with the same structure as the example shown in FIG. That is, the artificial intelligence based surveillance system using deep learning according to the present invention can detect a deep net for object recognition and a deep net for accident prediction ), It can be used for risk and accident prediction, crime prediction, and traffic accident prediction. Beyond detecting and recognizing objects appearing in images, risks and accidents can be predicted by the learned deep network based on safety and accident data and artificial intelligence technology, visualized and used as a surveillance system.

Accident Prediction As shown in FIG. 6, the deep learning network uses information obtained through object detection, object recognition, and situation recognition to predict risks / accidents and crimes that occur within a certain time range in the future or afterward. In the case of risks and accidents, CCTV observations can be used to predict risks from accidents (eg, fire, explosions, etc.), mass crowds, abnormal behavior, or information from criminals (eg criminals faces or criminal vehicles) .

The present invention predicts future crime incidents from current vehicle numbers and flows, crowd information, and behavioral awareness with deep learning technology.

At this time, the system according to the present invention can perform prediction through a deep learning network from correlation information extraction, multi-object and camera information data according to time and space change of surveillance video.

In the present invention, as shown in FIG. 7, by using data extracted from the CCTV, for example, criminal, crowd, vehicle, and situation information, and government department data related to safety and accident, You can learn the deep learning structure for crime, traffic accident, risk and accident prediction.

At this time, it is possible to use RNN, DAE, DNN-based data representation method and multi-modal deep learning network convergence structure as a predictive network structure.

FIG. 8 is a flowchart illustrating an operation of an artificial intelligence based video surveillance method according to an embodiment of the present invention, and shows an operation flow chart in the system of FIGS. 1 to 7 described above.

Referring to FIG. 8, an artificial intelligence based video surveillance method according to an embodiment of the present invention detects at least one object preset for a surveillance camera image using a predefined detection deep learning network in a detection unit (S810).

In step S810, a region proposal extraction for extracting a candidate region at a high speed, which can perform a human detection, a face detection, an automobile detection, a license plate detection, and the like using a DIP network, A method of generating a heat map based on a neural network (CNN) can be used to detect an object for a surveillance camera image.

At this time, step S810 can detect the object by extracting the object area and adjusting the bounding box based on the multi-tasking learning method using the attributes of the object. Specifically, in step S810, an area proposal extraction is performed on the surveillance camera image, and an object can be detected by extracting the characteristics of each of the area proposals through characteristic extraction based on the convolutional neural network from the extracted area proposals.

The recognition unit recognizes the information about the detected object based on the detection result of the predefined deep learning network and the objects (S820).

Here, in step S820, the image quality measurement may be performed to perform highly reliable recognition on an image having good image quality before performing recognition, and the image quality to be evaluated may include resolution, brightness, posture information, and blur information.

The tracking unit traces the detected object based on the detection results of the pre-defined tracking deep learning network and the detected object (S830).

Here, the step S830 may measure the reliability of the previous tracking result and the current tracking result for the detected object, and update the deep learning model on-line when the measured reliability is less than a predetermined value, thereby tracking the detected object, Reliable tracking can be performed.

Further, in step S830, in order to update the model on-line, the sample corresponding to the position of the current object is positively set, the positive surrounding area is negatively set, and the learning is performed. Positive and negative sample information for a certain number of frames is accumulated and used for on-line learning, thereby preventing a tracking result from drifting due to a phenomenon such as a clipping.

The predictor predicts the risk factors based on the predicted deep learning network and the detected objects and the recognized object information, which have been previously learned about the accident and the risk (S840).

That is, step S840 associates the deep net for object recognition and deep net for accident prediction with respect to the detected object to detect danger and accident prediction, , Traffic accident prediction, and so on.

Here, step S840 can be used to predict a risk factor from an accident (e.g., fire, explosion, etc.) through a CCTV observation, a number of crowds, an unusual activity, or information of an offender (for example, a criminal's face or a criminal vehicle) have.

Although not described in the artificial intelligence based video surveillance method of FIG. 8, it is to be understood that the artificial intelligence based video surveillance method of FIG. 8 may include all of the contents described in the system of FIGS. It will be obvious to those skilled in the art.

The system or apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the systems, devices, and components described in the embodiments may be implemented in various forms such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array ), A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to embodiments may be implemented in the form of a program instruction that may be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (12)

A detector for detecting at least one object preset for a surveillance camera image using a predefined detection deep learning network;
A recognition unit for recognizing information about the detected object based on a detection result for the object; And
And a tracking unit for tracking the detected object based on a detection result for the detected object,
Lt; / RTI >
The tracking unit
Tracking the detected object by measuring a previous tracking result for the detected object and a reliability of the current tracking result and updating the deep learning model on-line when the measured reliability is less than a predetermined value,
The tracking unit
Updating the deep learning model by performing a learning after placing a sample corresponding to a position of the detected object as positive and a positive surrounding region as a negative, Based video surveillance system that accumulates positive sample information and negative sample information for a predetermined number of frames for the next online update.
The method according to claim 1,
The detection unit
Wherein the object is detected using region proposal extraction including a method of generating a heat map based on a convolution neural network (CNN).
The method according to claim 1,
The detection unit
Wherein the object is detected by performing extraction of an object region and adjustment of a bounding box based on a multi-tasking learning method using attributes of the object, system.
The method according to claim 1,
The detection unit
Extracting characteristics of each of the region proposals by extracting region suggestion extraction for the surveillance camera image and extracting characteristics of the region proposal based on the convolutional neural network based on the extracted region proposals, Intelligent video surveillance system.
delete The method according to claim 1,
And a predictor for predicting a risk element based on the detected object and the recognized object information, wherein the predicted deep learning network includes an accident prediction and deep learning network which is interlocked with the detection unit and the recognition unit,
Further comprising: an artificial intelligence based video surveillance system.
Detecting at least one object preset for the surveillance camera image using a predefined detection deep learning network at the detection unit;
Recognizing information on the detected object based on a detection result for the object and a predefined deepening learning network in the recognition unit; And
Tracking the detected object based on a predefined tracking deep learning network and a detection result for the detected object in a tracking unit
Lt; / RTI >
The tracking step
Tracking the detected object by measuring a previous tracking result for the detected object and a reliability of the current tracking result and updating the deep learning model on-line when the measured reliability is less than a predetermined value,
The tracking step
Updating the deep learning model by performing a learning after placing a sample corresponding to a position of the detected object as positive and a positive surrounding region as a negative, And accumulates the positive sample information and the negative sample information with respect to the predetermined frame number for the next online update.
8. The method of claim 7,
The detecting step
Wherein the object is detected using region proposal extraction including a method of generating a heat map based on a convolution neural network (CNN).
8. The method of claim 7,
The detecting step
Wherein the object is detected by performing extraction of an object region and adjustment of a bounding box based on a multi-tasking learning method using attributes of the object, Way.
8. The method of claim 7,
The detecting step
Extracting characteristics of each of the region proposals by extracting region suggestion extraction for the surveillance camera image and extracting characteristics of the region proposal based on the convolutional neural network based on the extracted region proposals, Intelligent video monitoring method.
delete 8. The method of claim 7,
Predicting a risk factor based on the detected object and the recognized object information;
The method comprising the steps of: (a) providing an artificial intelligence-based video surveillance system comprising:

Images