KR20190088087A - method of providing categorized video processing for moving objects based on AI learning using moving information of objects - Google Patents

Abstract

The present invention relates to a technique which generates an object cluster in accordance with a similarity calculation result in motion information of a plurality of moving objects, then uses object clustering information to learn an object classification module based on artificial intelligence, and inputs motion information of an object into the object classification module when the object is identified in a CCTV video to quickly determine which object cluster the identified object belongs. A control agent inputs a tag into an object cluster based on a thumbnail image of a moving object video to classify a plurality of video clips associated with a moving object identified in a CCTV video by category. Generally, extraction and tracking of motion information of a video can be implemented by a light algorithm. Therefore, according to the present invention, large-scale CCTV videos can be analyzed in parallel, and a type of a moving object can be quickly classified. Also, according to the present invention, a CCTV video can be classified and searched based on a type of a moving object. Specifically, since the present invention classifies a moving object based on artificial intelligence learning, the present invention can adaptively respond to a specific photographing environment or a specific camera.

Description

[0001] The present invention relates to a moving object image classification method based on artificial intelligence learning using motion information,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention generally relates to a technique for classifying video clips of moving objects into categories according to CCTV shot images.

More specifically, the present invention generates an object cluster according to a result of calculating similarity between motion information of a plurality of moving objects, then learns an object classification module based on an artificial intelligence using the object clustering information, The present invention relates to a technique for quickly determining to which object cluster an object belongs by inputting motion information of the identified moving object to an object classification module. At this time, by allowing the controller to input a tag to the object cluster based on the thumbnail image of the moving object image, a plurality of image clips related to the moving object identified in the CCTV photographed image can be classified and categorized.

A number of cameras (eg CCTV) are installed for crime prevention and criminal investigation. Currently, about 5 million CCTVs are installed at intervals of 14.2 meters in the country, taking pictures of the surroundings once every 5.5 seconds. The images taken by these CCTVs are transmitted to the control system such as the police station or the integrated control center, and the crime prevention activities are performed by the control personnel through monitoring.

Compared to the explosive increase in the number of CCTV units, there are a lot of inspectors working in the Youngsan management system. According to the data of the police department in 2011, it monitors the average of 45 CCTVs per person in the Seoul area, making it difficult to prevent crime. Also, in case of watching images stored in CCTV for crime investigation, criminal detection, and loser search, a small number of control personnel are looking at a lot of images, and the efficiency of the processing is considerably lowered.

In order to solve these problems, it is practically limited to expand control personnel. Therefore, the control system which is equipped with the intelligent image analysis function that analyzes the CCTV shot image through the image processing technology and processes the image in a form convenient for the control personnel to perform the video control task, Interest in the 'control system' has been growing in recent years.

In view of the tendency to manage thousands of CCTVs for video surveillance and the tendency of CCTV cameras to be in high image quality, there is a need for a technique capable of performing high speed image processing required in an intelligent control system.

An object of the present invention is to provide a technique for classifying image clips of a moving object into categories according to a CCTV shot image.

In particular, an object of the present invention is to provide an object classification module based on an artificial intelligence based on the object clustering information after generating an object cluster according to a result of calculating similarity between motion information of a plurality of moving objects, The motion information of the identified moving object is input to the object classification module to quickly determine to which object cluster it belongs. At this time, the control agent inputs a tag to the object cluster based on the thumbnail image of the moving object image, thereby sorting a plurality of image clips related to the moving object identified in the CCTV shot image into categories.

In order to achieve the above object, an artificial intelligence learning based moving object image classification processing method using motion information according to the present invention is a method for classifying moving object images by tracking an object motion information for each of a plurality of first moving objects through image analysis, Stage 1; A second step of determining a plurality of object clusters for attributing one or more first moving objects to a plurality of first moving objects according to a result of calculating similarity based on object motion information; A third step of artificial intelligence learning of the object classification module using a plurality of combinations of object motion information and object cluster attribution information obtained from a plurality of first moving objects; A fourth step of extracting object motion information of each of the second moving objects from the plurality of second moving objects identified in the image and inputting the extracted object motion information into the object classification module to determine an object cluster to which each second moving object belongs; And a fifth step of managing and storing image clips for a plurality of second moving objects for each object cluster.

At this time, if the attribution change setting of the object cluster for the specific third moving object is identified from among the plurality of second moving objects, the combination of the object motion information of the third moving object and the changed object cluster belonging information And performing artificial intelligence learning on the object classification module.

According to another aspect of the present invention, there is provided a method of generating thumbnail images, the method comprising: generating thumbnail images for a plurality of first moving objects, Receiving at least one tag information for an object cluster corresponding to a thumbnail image; Designating a category according to tag information for each of a plurality of object clusters; Identifying a specific keyword in response to a user operation; Displaying one or more image clips designated by the category corresponding to the keyword on the display screen.

The second step of the present invention includes generating an object motion vector by vectorizing object motion information for each of a plurality of first moving objects; Calculating a degree of similarity between the plurality of first moving objects based on the Euclidean distance between the object motion vectors; And creating a plurality of object clusters by grouping a plurality of first moving objects whose similarities are equal to or greater than a predetermined threshold value.

Also, in the present invention, the object motion information is configured by grouping temporal dynamic exposure attributes identified in a series of image frames for the same first moving object, and the object motion vectors are grouped into a series of dynamic exposures grouped for the same first moving object And the artificial intelligence learning is performed by applying object motion information and object cluster attribution information of the first moving object or the third moving object to the input layer and the output layer of the object classification module, It can be done.

Meanwhile, a computer-readable nonvolatile recording medium according to the present invention records a program for executing a moving object image classification processing method based on artificial intelligence learning using motion information as described above in a computer.

In general, motion information extraction and tracking of a video image can be implemented with a lightweight algorithm. Therefore, according to the present invention, there is an advantage that a CCTV shot image can be analyzed in parallel and the types of moving objects can be quickly classified.

In addition, according to the present invention, it is possible to classify and search a CCTV photographed image based on the kind of a moving object. In particular, since the present invention classifies moving objects on the basis of artificial intelligence learning, it has an advantage that it can adaptively cope with a specific shooting environment or a specific camera.

1 is a flowchart showing an entire process of a moving object image classification processing method based on artificial intelligence learning according to the present invention.
FIG. 2 is a flowchart illustrating an object cluster creation process in the present invention. FIG.
FIG. 3 conceptually illustrates an initial learning process for an object classification module in the present invention; FIG.
FIG. 4 conceptually illustrates a category classification process for a moving object in the present invention. FIG.
FIG. 5 conceptually illustrates a continuous learning process for an object classification module in the present invention; FIG.
FIG. 6 is a diagram illustrating a concept in which object motion information is extracted for a moving object in the present invention; FIG.
7 is a diagram showing the concept of an object cluster in the present invention.
8 is a diagram illustrating a concept of generating an object motion vector and artificial intelligence learning of an object classification module in the present invention.
9 illustrates a concept of classifying a newly discovered moving object through an object classification module in the present invention.
FIG. 10 is a diagram summarizing features of a moving object image classification processing technique based on an artificial intelligence learning according to the present invention.

Hereinafter, the present invention will be described in detail with reference to the drawings.

1 is a flowchart illustrating an overall process of a moving object image classification processing method based on artificial intelligence learning according to the present invention. The process of FIG. 1 is generally performed by a video management server of a video control system.

Step S100: First, the CCTV shot image is analyzed to identify and track a plurality of first moving objects. The CCTV photographed image used in this process is preferably a large amount of image files stored in the image management server.

At this time, techniques for identifying and tracking moving objects in a compressed image can be variously adopted. It is also possible to receive the image clips related to the plurality of first moving objects prepared in advance from the outside and to extract the image clips related to the first moving object from the CCTV shooting image for a long time. In the latter case, a method of analyzing the difference component by decoding a compressed image as in the prior art is also possible. However, the applicant's patent application No. 10-2017-0107580 "Syntax-based Moving Object Region Extraction "Is advantageous in terms of processing speed. Particularly, regarding the tracking technology of moving objects, the applicant's patent application No. 10-2017-0176597 entitled " Method of tracking a syntax-based object on a compressed image "

Then, using the object tracking result, information indicating how to move each of the first moving objects on the CCTV shot image, that is, 'object motion information' is constructed.

The presentational dynamics of a specific object in an image include a position in a CCTV shot image, a size of a bounding box that optimally surrounds the moving object, and an instant speed. To see how a particular object moves within an image, you should not look at just one of these dynamic exposure properties, but rather look at several things, changing the time. In this specification, object motion information is constructed by combining time information and dynamic exposure attributes and extracting and grouping a plurality of them according to time.

The configuration of object motion information in the present invention will be described later in detail with reference to FIG.

Step S110: The degree of similarity is calculated based on the object motion information for the plurality of first moving objects, and the plurality of object clusters are generated from the plurality of first moving objects by grouping the first moving objects having high similarity . For example, 100 first moving objects are classified into seven object clusters. It is based on the assumption that objects with similar motions are likely to be of the same kind. As described above, since the present invention performs object clustering based on the motion of an object, it is not necessary to decode a compressed image and recognize what each object is, as in the prior art. As a result, the present invention enables high-speed processing as compared with the prior art.

The object clustering process will be described later in detail with reference to FIG. 2 and FIG. 7 in the present invention.

Step S120: For each of the generated plurality of object clusters, the tag information is input based on the object thumbnail image related to the first moving object belonging to the corresponding cluster, and the category is designated accordingly.

In the above process, although a plurality of first moving objects are created by grouping objects estimated to be similar, an object cluster is not identified. This is because the object clustering process is performed based on the motion, not the object content, in the present invention.

In this case, a thumbnail image is generated for a plurality of first moving objects, and displayed on the display unit according to the belonging object clusters. Then, corresponding to the thumbnail images, one or more tag information (For example, people, cars, bicycles). Then, the category is designated according to the tag information for each of the plurality of object clusters. Preferably, the tag input in step S120 is performed on an object cluster basis, and a plurality of first moving objects belonging to the same object cluster have the same tag.

Step S130: The artificial intelligence (AI) -based object classification module is artificially intelligently learned by using a plurality of combinations of object motion information and object cluster attribution information obtained from a plurality of first moving objects as described above. The object classification module repeatedly performs supervised learning through a plurality of combination data on the object clusters to which the object motion information is attributed so that the object classification module indicates such a cluster classification tendency. At this time, DNN (Deep Neural Network) technology can be applied as an artificial intelligence learning technique.

The artificial intelligence learning of the object classification module will be described later in detail with reference to FIG. 8 in the present invention.

Step S140: A process of defining an object cluster using a plurality of first moving objects and performing artificial intelligence learning on the object classification module through steps S100 to S130 is described. Hereinafter, a process of quickly determining which object cluster belongs to a newly identified moving object using the object classification module learned above, i.e., an object classification process will be described.

When the second moving object is identified in the photographed image generated from the CCTV camera in the video monitoring system, the object moving information for the second moving object is extracted and input to the learned object classification module. At this time, the identification of the second moving object is not performed manually by the control personnel but is automatically performed by the applicant's patent application No. 10-2017-0107580 entitled " Syntax-Based Moving Object Region Extraction Method for Compressed Image " .

The object moving information is extracted for the thus-identified second moving object, as described above in step S100. When the object motion information for the second moving object is input to the object classification module, the object classification module performs a series of operations according to the multiple layers and outputs the result. The output value provided by the object classification module according to the artificial intelligence learning corresponds to the judgment result about what object cluster the second moving object belongs to.

A plurality of second moving objects are identified from the CCTV photographed image. This process is performed for each second moving object to determine what object clusters each belongs to, preferably in real time.

In the present invention, cluster classification of moving objects using the object classification module will be described later in detail with reference to FIG.

Steps S150 and S160: The object clustering process performed in the above steps S100 and S110 is very rough since it is determined based on the motion form without viewing the image contents. Accordingly, the AI learning result of the object classification module made in step S130 is also completely unreliable, and the cluster classification result for the second moving object in step S140 may also be in error.

Hereinafter, when an error is found in the cluster classification result for the second moving object made in step S140, the process of re-learning the object classification module by artificial intelligence is described.

First, it identifies the attribution change setting of the object cluster for a specific moving object (referred to as a 'third moving object') among the plurality of second moving objects determined to belong to the object cluster. For example, the controller classifies the object clustering result performed by the object classification module with a thumbnail image or the like, finds that the object is incorrect, and manually changes the cluster classification of the moving object. At this time, the object classification module is artificially intelligently learned by using the combination of the object motion information of the third moving object and the changed object cluster attribution information.

Steps S170 and S180: The video control system manages and stores image clips for a plurality of second moving objects found from the CCTV shot image for each object cluster, and the control personnel manipulates the monitor terminal to generate keywords (e.g., Car, bicycle) is input, a video clip having a category corresponding to the keyword is selected and displayed on the monitor screen of the control personnel.

FIG. 2 is a flowchart illustrating a process of generating a plurality of object clusters according to a result of calculating a degree of similarity based on motion information for a plurality of moving objects obtained from a CCTV shot image in the present invention. The process of FIG. 2 corresponds to step S110 in the entire process of FIG.

In operation S200, an object motion vector is generated by vectorizing object motion information for each of a plurality of first moving objects. The object motion information is composed of a series of time information and dynamic exposure information. For the mathematical operation, an object motion vector is generated from the object motion information. In this specification, an object motion vector is composed of a one-dimensional vector having elements of a series of dynamic exposure attributes related to the same moving object, which will be described later in detail with reference to FIG.

Step S210: The Euclidean distance between a plurality of object motion vectors derived from the plurality of first moving objects is calculated. Euclidean distance is a concept that represents the geometric distance between two points in a multidimensional space. In addition to Euclidean distance, other types of distance concept such as Manhattan Distance, Great Distance, Mahalanobis Distamce, Time Distance, etc. may be adopted.

Step S220: Among the plurality of first moving objects, moving objects having the calculated Euclidean distance equal to or less than a preset threshold value are grouped to generate a plurality of object clusters. In the present invention, the closer the Euclidean distance is, the more similar the movement form of the moving object is regarded as being more similar. Therefore, moving objects that are sufficiently similar in shape to movements are likely to be objects of a similar nature, and are attributed to the same object cluster.

3 is a conceptual diagram illustrating an initial learning process for an object classification module in the present invention. The initial learning process of FIG. 3 corresponds to steps S100 to S130 in the entire process of FIG.

When a CCTV shot image is provided from a real-time or storage server, the object tracking module extracts a plurality of moving objects from the search image and extracts motion information (time, size, position, and instantaneous speed).

The object clustering module clusters similar moving objects with similar object motion information, and preferably displays thumbnail images of moving objects to the controller at a time.

The control agent sees a number of thumbnail images and inputs meaningful tags such as people, cars, bicycles, etc. for the moving objects. Preferably, thumbnail images are classified and provided according to object clusters. The input tag is input as tag information to a corresponding object cluster to which the moving objects belong. Preferably, the moving objects belonging to the same object cluster have the same tag.

The object motion information for each of a number of moving objects obtained from the CCTV shot image and each object cluster attribution information (tag information) are provided to an artificial intelligence based object classification module and utilized in artificial intelligence learning (eg, DNN).

4 is a diagram conceptually illustrating a category classification process for a moving object in the present invention. The category classification process of FIG. 4 corresponds to step S140 in the entire process of FIG.

When the object tracking module identifies a new moving object from the CCTV photographed image, it provides object motion information on the object to the object classification module.

When the object motion information is inputted through the process of [FIG. 3], the object classification module is learned such that the attribution information of the object cluster is outputted so as to conform to the cluster classification tendency of the plurality of first moving objects. Accordingly, when the object movement module for the new moving object is input, the object classification module outputs the result of determining the object cluster to which the new moving object belongs.

Meanwhile, the object tracking module also provides a thumbnail image for a new moving object. The thumbnail image will be used in the continuous learning process for the object classification module, which will be described later with reference to FIG.

5 is a diagram conceptually showing a continuous learning process for an object classification module in the present invention. The continuation learning process of FIG. 5 corresponds to step S150 and step S160 in the entire process of FIG.

Through the category classification process, the object classification model determines the attribution of object clusters to a plurality of new moving objects. The controller classifies the object classification data based on the thumbnail image and modifies the cluster classification of the moving object when it is determined that the object classification data is erroneously classified.

When the cluster classification is modified as described above, the object classification module is re-artificially intelligently learned using the object motion information about the moving object and the modified object cluster attribution information.

This continuing learning process can be continuously revised by the control personnel whenever the category classification is modified.

6 is a diagram illustrating a concept in which object motion information is extracted for a moving object in the present invention.

In the present invention, when a moving object is found from a CCTV photographed image, a series of dynamic exposure attributes according to a time change is extracted from the image and grouped to form object motion information. The dynamic exposure properties of the moving object at each time point Tk include the position in the image, the size of the bounding box optimally storing the moving object, and the instantaneous speed. To see how a particular object moves within an image, you should not look at just one of these dynamic exposure properties, but rather look at several things, changing the time. Accordingly, the object motion information A for one moving object consists of a group of dynamic exposure attributes extracted from N times (T1, T2, ..., TN).

In FIG. 6, N pieces of [Tk, Lk, Bk, Sk] are shown as an example of the object motion information (A). At this time, the interval of the sampling time (for example, T2 - T1) and the number N of the total time samples are appropriately set according to the design purpose of the video control system. In particular, as the number of samples N increases, the amount of process computation increases. Therefore, the value of N can be set in consideration of system performance and target throughput. In addition, a thumbnail image is extracted for all or part of the moving object, and the number of thumbnail images is set to a degree suitable for the controller to identify the object category.

7 is a diagram illustrating the concept of an object cluster in the present invention.

7 shows a concept of assigning M (e.g., 100) moving objects extracted from the CCTV shot image to L (e.g., 7) object clusters. In the present invention, object clusters are formed by grouping objects having similar motion shapes in relation to a plurality of moving objects found in an image.

Referring to FIG. 7, the moving objects A1, A3, A12, A31, and A33 show similar motion patterns, and they may be, for example, persons. Also, the moving objects A2, A9, A11, A21, and A32 have similar motion patterns, and these may be, for example, automobiles. In CCTV images, human beings and automobiles define different object clusters because they represent a generally distinct form in terms of location, size, and instantaneous speed.

8 is a diagram illustrating a concept of generating an object motion vector and artificial intelligence learning of an object classification module in the present invention.

In the present invention, a plurality of combinations of object motion information and object cluster attribution information obtained from an image in relation to a plurality of moving objects, i.e., a first moving object and a third moving object, are applied to an object classification module, Artificial intelligence learning is performed. However, FIG. 8 shows a process of learning an object classification module with respect to one moving object on one side.

The object motion information A for a moving object is a group of dynamic exposure attributes (L1, B1, S1; L2, B2, S2, ...) extracted from N times (T1, T2, ..., TN). LN, BN, SN). In the present invention, an object motion vector (C) is generated from object motion information for a mathematical operation. The motion motion vector may be a one-dimensional vector having a series of dynamic exposure attributes grouped as elements. At this time, since the position information is composed of the x-coordinate and the y-coordinate, the object motion vector may be composed of a one-dimensional vector having a length of 4 * N, as shown at the top of FIG.

Also, when the number of object clusters is L, the object cluster attribution information D for the moving object can be composed of a one-dimensional vector of length L. [ Preferably, only the element corresponding to the object cluster to which the moving object belongs is represented by 1, and the remaining elements are set to 0.

In the present invention, the object classification module is composed of a multi-layer neural network based on artificial intelligence. If an object motion vector is provided as an input layer of an object classification module and an object cluster attribution information is provided to an output layer of an object classification module, an artificial intelligent learning process (eg, DNN) Adjust slightly to fit the output. By repeating this process for a number of moving objects, the neural network is set to show the desired object cluster classification characteristics.

9 is a diagram illustrating a concept of classifying a newly discovered moving object through the object classification module in the present invention.

When a moving object is newly found in a CCTV shot image, it is necessary to classify which object cluster belongs to which object cluster. To do this, a new object is traced from the image to extract the object motion information (NA), and from this, an object motion vector (NC) for the new object is constructed.

When an object motion vector (NC) is provided to an input layer of an object classification module and a series of operations are performed to propagate information through multiple layers of a neural network, a classification result for a new object is output in the output layer . The output result is a one-dimensional vector of length L. Preferably, only the element corresponding to the object cluster to which the new object belongs is represented by 1, and the remaining elements are set to zero. Through this process, classification results for new objects are obtained.

10 is a view showing an overview of features of a moving object image classification processing technique based on artificial intelligence learning according to the present invention.

First, since the present invention deals with a moving object based on motion information (meta information) of an object in terms of object classification, it is possible to quickly classify a plurality of objects identified in various channels by using a light object tracking module, Indicates classification performance. The prior art uses a very complicated object analysis engine based on image features, so that it can not accommodate many channels because of its low classification performance.

In the aspect of object learning, the present invention has an advantage that burden of artificial intelligence learning is greatly reduced because tagging similar objects by clustering in advance and simultaneously tagging a plurality of objects and reflecting them in learning. In the conventional artificial intelligence learning technology, artificial intelligence learning was very burdensome because it was a method of learning objects one by one and learning the characteristics of objects accurately.

If the video control system is operated according to the present invention, the learning data generated in the system installation environment is learned during the initial construction process, the object classification module automatically classifies the objects in the system operation process, Continue learning only if there is an error.

It is not necessary to individually learn the object classification module for a plurality of CCTV cameras belonging to the video control system. It can be expected that the result of artificial intelligence learning of the object classification module for the captured image of any one CCTV camera can be applied to the captured image of another CCTV camera of similar environment and it can be verified through several test images .

According to the present invention, the control personnel working in the video control system can monitor the video very effectively by further specifying the type of the moving object after specifying the CCTV camera in case of emergency. This feature can be advantageously used in the process of searching a large-scale shot image stored in the storage device.

Meanwhile, the present invention can be embodied in the form of computer readable code on a computer-readable non-volatile recording medium. Such a non-volatile recording medium includes all kinds of storage devices for storing computer-readable data such as a hard disk, an SSD, a CD-ROM, a NAS, a magnetic tape, a web disk, a cloud disk, The code may be distributed and stored in the storage device of the computer.

Claims (6)

A first step of tracking a plurality of first moving objects through image analysis and extracting object motion information for each of the first moving objects;
A second step of determining a plurality of object clusters for attributing one or more first moving objects to the plurality of first moving objects according to a result of calculating similarity based on the object motion information;
A third step of artificial intelligence learning of the object classification module using a plurality of combinations of object motion information and object cluster attribution information obtained from the plurality of first moving objects;
A fourth step of determining an object cluster to which each second moving object belongs by extracting object motion information of each of the second moving objects for a plurality of second moving objects identified in the image and inputting the extracted object motion information to the object classification module ;
A fifth step of managing and storing image clips for the plurality of second moving objects for each object cluster;
A moving object image classification processing method based on an artificial intelligence learning using motion information including a moving object image including motion information;
The method according to claim 1,
The method of claim 1,
When the identification of the attribution change setting of the object cluster with respect to a specific third moving object among the plurality of second moving objects is identified, Artificial intelligence learning module;
A moving object image classification processing method based on artificial intelligence learning using motion information.
The method according to claim 1,
And performing the second step and the third step,
Generating thumbnail images for the plurality of first moving objects and displaying and displaying the thumbnail images according to the belonging object clusters;
Receiving at least one tag information for the object cluster corresponding to the thumbnail image;
Designating a category according to the tag information for each of the plurality of object clusters;
Further comprising:
The method according to claim 1,
Identifying a specific keyword in response to a user operation;
Displaying one or more image clips designated by the category corresponding to the keyword on the display;
A moving object image classification processing method based on artificial intelligence learning using motion information.
The method according to claim 1,
The second step comprises:
Generating an object motion vector by vectorizing the object motion information for each of the plurality of first moving objects;
Calculating an Euclidean distance between the object motion vectors;
Creating a plurality of object clusters by grouping a plurality of first moving objects whose Euclidean distance is equal to or less than a predetermined threshold value;
And a moving object image classification processing method based on artificial intelligence learning using motion information.
The method of claim 4,
Wherein the object motion information is configured by grouping temporal dynamic exposure attributes identified in a series of image frames for the same first moving object,
Wherein the object motion vector comprises a one-dimensional vector having elements as a series of dynamic exposure attributes grouped for the same first moving object,
Wherein the artificial intelligence learning repeatedly applies object motion information and object cluster attribution information of the first moving object or the third moving object to the input layer and the output layer of the object classification module, Artificial intelligence learning based moving object image classification processing method.
A computer-readable nonvolatile recording medium storing a program for causing a computer to execute a moving object image classification processing method based on artificial intelligence learning using motion information according to any one of claims 1 to 5.

Images