KR20210056050A - Apparaus and method of identifying multi class objects - Google Patents

Abstract

Disclosed are a device and a method for identifying a multi-class object. The method for identifying a multi-class object includes the steps of: inputting an image to the learned artificial intelligence neural network; performing subclass non-maximum suppression (NMS); and performing group class NMS.

Description

Device and method for identifying multi-class objects {APPARAUS AND METHOD OF IDENTIFYING MULTI CLASS OBJECTS}

The present disclosure relates to a multi-class object identification apparatus and method, and more particularly, to a multi-class object identification apparatus and method for identifying an object included in an image using a learned artificial intelligence neural network.

Due to the recent development of deep learning technology, the reliability of multi-class object detection has increased a lot, and it is being used as a core technology in various industrial fields. Early object detection methods such as VJ (Viola Jones) and HOG (Histogram of Oriented Gradient) were more used in the field of detecting only one object such as a person or vehicle rather than multi-class object detection. On the other hand, the object detection method based on deep learning can simultaneously detect various multi-class objects such as people, vehicles, motorcycles, traffic lights, and traffic signs with little increase in complexity.

As shown in FIG. 1, when an input image is input to a pre-learned object detection network, a general deep learning-based object detection technology creates many bounding boxes around detection target objects. In addition, a general deep learning-based object detection technology removes other boxes while leaving only one bounding box based on a non-maximum suppression (NMS) algorithm. In general, deep learning-based object detection technology connects instances of each object every frame through modules such as tracking from the next frame and data association for objects detected in the current frame when images are continuously input, such as videos. By doing so, it is possible to estimate the location and motion information of the objects.

Existing object detection technology is capable of classifying sub-classes such as sedans, vans, vans (SUVs), etc. even within a specific object class, for example, a vehicle class. Existing object detection technology has high classification accuracy for classes that include features that can be clearly distinguished, such as humans and vehicles. However, when classifying classes having similar characteristics such as sedans, vans, vans, etc., there is a high possibility that an object recognition error may occur.

That is, as shown in FIG. 1B, the existing object detection technology incorrectly classifies an object class, or is recognized as two or more different classes by several class boxes created in an object as shown in FIG. 1C. Occurs. Existing object detection technologies trace initialization failure or misrecognized multiple objects in the multi-object tracking step due to a recognition error generated in the object detection step, and cause a problem of activating false detection tracking.

As shown in FIG. 1D, when a misrecognition occurs during a tracking initialization process, the existing object detection technology determines that a new object has been detected, so that tracking cannot be initialized and tracking cannot be activated. And, as shown in Figs. 1e and 1f, the existing object detection technology detects the same object as different objects during a plurality of frames in succession and correlates them with each other, so that two different objects are initialized and tracked. Activates. That is, since the existing multi-object tracking technology determines that tracking is activated for an erroneous detection object (green box, corresponding to a van) generated in the object detection step, a problem in which detection errors are propagated by the object tracking module occurs.

Accordingly, there is a need for a technology capable of accurately classifying classes having similar characteristics and preventing erroneous detection and detection errors.

The present disclosure is to solve the above-described problems, and is to provide a multi-class object identification apparatus and method capable of accurately identifying objects and tracking objects.

According to an embodiment of the present disclosure for achieving the above object, a method for identifying a multi-class object includes inputting an image to a learned artificial intelligence neural network, and an object included in the image based on the learned artificial intelligence neural network. Performing a subclass Non-maximum suppression (NMS) for identifying at least one subclass corresponding to and generating at least one subclass bounding box in units of the identified at least one subclass, and each subclass Calculate a probability for a class, identify a group class including the identified at least one subclass based on a group class in which a plurality of subclasses corresponding to each of a plurality of homogeneous objects are grouped, and the generated at least one And performing a group class NMS of removing the subclass bounding box of and generating one group class bounding box based on the identified group class.

Meanwhile, the method of identifying a multi-class object may further include recognizing a subclass having a highest probability among the calculated probabilities for each subclass as the object.

In addition, the method of identifying a multi-class object may further include displaying the generated group class bounding box and the subclass having the highest probability.

Meanwhile, the image is a video including a plurality of frames, the subclass NMS and the group class NMS are performed for each of the plurality of frames, and the multiclass object identification method includes the plurality of frames. When the group class bounding box generated in each of the frames of is the same group class bounding box and is continuously generated more than a preset number of times, activating object tracking may be further included.

In addition, the multi-class object identification method includes an object tracking step of accumulatively averaging the probabilities of each subclass calculated from a preset number of frames, and recognizing and tracking the subclass having the highest accumulated average probabilities as an object. It may contain more.

According to an embodiment of the present disclosure for achieving the above object, a multi-class object identification apparatus includes a processor including an input unit for receiving an image and a learned artificial intelligence neural network, wherein the processor includes the learned artificial intelligence Perform a subclass NMS operation for identifying at least one subclass corresponding to an object included in the image based on a neural network, and generating at least one subclass bounding box in units of the identified at least one subclass, and , Calculating a probability for each subclass, and identifying a group class including the identified at least one subclass based on a group class in which a plurality of subclasses corresponding to each of a plurality of homogeneous objects are grouped, and the A group class NMS operation is performed for removing at least one generated subclass bounding box and generating one group class bounding box based on the identified group class.

In addition, the processor may recognize a subclass having the highest probability among the calculated probabilities for each subclass as the object.

Meanwhile, the apparatus for identifying a multi-class object may further include a display displaying the generated group class bounding box and the subclass with the highest probability.

Meanwhile, the image is a video including a plurality of frames, and the processor performs the subclass NMS operation and the group class NMS operation for each of the plurality of frames, and the group class generated in each of the plurality of frames If the bounding box is the same group class bounding box and is continuously generated more than a preset number of times, object tracking may be activated.

In addition, the processor may cumulatively average the probabilities of each subclass calculated from a preset number of frames, and track the subclass having the highest cumulative average probability as an object.

As described above, the apparatus and method for identifying a multi-class object can accurately distinguish an object having similar characteristics.

In addition, the multi-class object identification apparatus and method can improve the accuracy and reliability of object tracking.

The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1A is a diagram illustrating a detection result of an existing object detection technology.
1B is a diagram illustrating an example of a recognition error in an existing object detection technology.
1C is a diagram illustrating an example of an error in recognizing a single object as a plurality of objects in the existing object detection technology.
1D is a diagram illustrating an example of a tracking deactivation error in an existing object detection technology.
1E is a diagram illustrating an example of an error in detecting multiple objects for a single object in the existing object detection technology.
1F is a diagram showing an example of propagation of a detection error in an existing object detection technology.
2A is a block diagram of an apparatus for identifying a multi-class object according to an embodiment of the present disclosure.
2B is a block diagram of an apparatus for identifying a multi-class object according to another embodiment of the present disclosure.
3A is a diagram illustrating a first embodiment of a group class according to the present disclosure.
3B is a diagram illustrating a second embodiment of a group class according to the present disclosure.
3C is a diagram illustrating a third embodiment of a group class according to the present disclosure.
4 is a diagram illustrating a process of detecting a multi-class object according to an embodiment of the present disclosure.
5A is an exemplary embodiment illustrating an object detection process using a probability distribution.
5B is an embodiment illustrating an object tracking process using a probability distribution.
6 is a flowchart of a method of identifying a multi-class object according to an embodiment of the present disclosure.

Hereinafter, various embodiments will be described in more detail with reference to the accompanying drawings. The embodiments described herein may be variously modified. Certain embodiments may be depicted in the drawings and described in detail in the detailed description. However, specific embodiments disclosed in the accompanying drawings are only intended to facilitate understanding of various embodiments. Therefore, the technical idea is not limited by the specific embodiments disclosed in the accompanying drawings, it is to be understood as including all equivalents or substitutes included in the spirit and scope of the invention.

Terms including ordinal numbers such as first and second may be used to describe various elements, but these elements are not limited by the above-described terms. The above-described terms are used only for the purpose of distinguishing one component from other components.

In the present specification, terms such as "comprises" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof does not preclude in advance. When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle.

Meanwhile, a "module" or "unit" for a component used in the present specification performs at least one function or operation. In addition, the "module" or "unit" may perform a function or operation by hardware, software, or a combination of hardware and software. In addition, a plurality of "modules" or a plurality of "units" excluding "module" or "unit" to be performed in specific hardware or performed by at least one processor may be integrated into at least one module. Singular expressions include plural expressions unless the context clearly indicates otherwise.

In the description of the present invention, the order of each step is to be understood without limitation, unless the preceding step must be performed logically and temporally prior to the subsequent step. That is, except for the above exceptional cases, even if a process described as a subsequent step is performed prior to a process described as a preceding step, the essence of the invention is not affected, and the scope of rights should also be defined regardless of the order of the steps. And in the present specification, the term "A or B" is defined to mean not only selectively indicating any one of A and B, but also including both A and B. In addition, in the present specification, the term "comprising" has the meaning of encompassing a further including other elements in addition to the elements listed as including.

In the present specification, only essential components necessary for the description of the present invention are described, and components not related to the essence of the present invention are not mentioned. In addition, it should not be interpreted as an exclusive meaning including only the mentioned components, but should be interpreted as a non-exclusive meaning that may include other components as well.

In addition, in describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be abbreviated or omitted. Meanwhile, each embodiment may be implemented or operated independently, but each embodiment may be implemented or operated in combination.

2A is a block diagram of an apparatus for identifying multi-class objects according to an embodiment of the present disclosure, and FIG. 2B is a block diagram of an apparatus for identifying multi-class objects according to another embodiment of the present disclosure. An apparatus for identifying a multi-class object will be described with reference to FIGS. 2A and 2B.

Referring to FIG. 2A, the multi-class object identification apparatus 100 may include an input unit 110 and a processor 120.

The input unit 110 receives an image. The received image may be a still image or a moving image including a plurality of frames. The input unit 110 may be configured to receive an image. For example, the input unit 110 may be implemented as a camera, and the camera may capture an image and input the captured image to the multi-class object identification device 100. Alternatively, the input unit 110 may be implemented as a communication module, and the communication module may receive an image captured using a wired/wireless communication method from an external device and input it to the multi-class object identification device 100. Alternatively, the input unit 110 may be implemented as a storage medium, and the storage medium may input a stored image to the multi-class object identification apparatus 100.

The processor 120 may perform a subclass non-maximum suppression (NMS) operation process and a group class NMS operation process based on the learned artificial intelligence neural network. The NMS operation process refers to a process of identifying a subclass corresponding to an object included in an image and generating a subclass bounding box in units of the identified subclass. There may be one or more subclasses.

The group class NMS operation process calculates the probability for each subclass, identifies a group class containing a plurality of subclasses, removes the subclass bounding box, and binds one group class based on the identified group class. It refers to the process of creating a box.

A detailed description of the group class and subclass will be described later.

In addition, the processor 120 may recognize a subclass having the highest probability among the calculated probabilities for the subclass as an object. For example, when one subclass is identified, the processor 120 may recognize the identified subclass as an object. When the processor 120 identifies three (a, b, c) subclasses, a matching probability of each subclass may be calculated. If the matching probability of subclass a is 0.3, subclass b is 0.2, and subclass c is 0.5, the processor 120 may recognize subclass c as an object.

Meanwhile, the multi-class object identification device may further include a display. Referring to FIG. 2B, a block diagram of an apparatus for identifying a multi-class object including a display 130 is shown. The display 130 may display the generated group class bounding box and the subclass with the highest probability.

As described above, the image input to the multi-class identification device may be a video. The multi-class identification apparatus accurately classifies and identifies an object by accumulating and averaging probabilities for subclasses based on each frame of an input video, and improves accuracy and reliability of object tracking.

Due to the improvement of the object detection accuracy based on deep learning, there is an increasing demand for a technology capable of subdividing and detecting various object classes. For example, users are demanding a technology that detects objects by subdividing them into details such as sedans, SUVs, and vans, rather than simply detecting a vehicle.

The present disclosure detects an object by accumulating averaging a probability distribution of an object tracked every frame, rather than detecting an object only with an independent detection result for each frame. In the following, group classes and subclasses will be described, and a method of identifying and tracking objects using the cumulative average probability will be described.

3A to 3C are diagrams illustrating various embodiments of a group class according to the present disclosure. The class will be described with reference to FIGS. 3A to 3C.

Subclass means individual objects. In addition, the group class refers to a group of a plurality of homogeneous objects or objects having a similar shape. 3A to 3C, sedans, vans, SUVs, etc. can be grouped into vehicle groups, motorcycles, bicycles, etc. can be grouped into two-wheeled vehicle groups, and adults, children, etc. can be grouped into people group. have. That is, each of the sedan, the van, and the SUV may be a subclass, and the vehicle group may be a group class. Each of the motorcycle and bicycle may be a subclass, and the motorcycle group may be a group class. Each of an adult and a child may be a subclass, and a group of people may be a group class. The above-described example is an embodiment, and each object is defined as a subclass, and a group obtained by grouping a plurality of homogeneous objects or objects having a similar shape may be defined as a group class.

4 is a diagram illustrating a process of detecting a multi-class object according to an embodiment of the present disclosure.

Referring to FIG. 4, a process of performing a subclass NMS and a process of performing a group class NMS are shown. When a detection network for detecting an object included in an image is trained, learning is performed by dividing into subclass units, and when inferring, bounding boxes and class probabilities are estimated by subclass units.

When the image 11 is input to the pre-learned detection network, many class bounding boxes are generated around the object in the output map. The NMS algorithm can be used to remove many class bounding boxes around the object. In the present disclosure, a two-step NMS process may be performed in order to minimize detection errors due to ambiguity in subclass classification.

The subclass NMS execution step acquires an object detection result for the subclass. In the subclass NMS execution step, several subclass bounding boxes may be generated for the same object included in the image 11, and a group class NMS process may be performed to remove the generated subclass bounding boxes.

The bounding box finally estimated in the group NMS execution step is represented by a group class, and probability distribution information for each subclass may be calculated. The probability calculated for each subclass in the group NMS execution step is the probability that each subclass has after performing the subclass NMS.

That is, when the image 11 is input, at least one bounding box may be generated around the object in the image 11 based on the learned artificial intelligence neural network. When the subclass NMS process is performed, at least one subclass corresponding to an object included in an image is identified, and a subclass bounding box may be generated in units of the identified at least one subclass. In addition, when the group class NMS process is performed, a group class may be identified based on the generated subclass bounding box, and one group class bounding box may be generated based on the identified group class. In this case, the previously generated subclass bounding box is removed, and a probability for each subclass may be calculated based on a probability possessed by each subclass when subclass NMS is performed.

As an embodiment, as shown in FIG. 4, when subclass NMS and group class NMS are performed on a vehicle included in the image 11, the multi-class object identification device generates a group class bounding box corresponding to the vehicle group. The subclass probability of the sedan is 0.52, the subclass probability of the SUV is 0.97, and the subclass probability of the van is 0.68. In addition, the multi-class object identification device may recognize the object with the highest probability of the SUV.

FIG. 5A is an embodiment showing an object detection process using a probability distribution, and FIG. 5B is an embodiment showing an object tracking process using a probability distribution. This will be described with reference to FIGS. 5A and 5B.

As described above, the image input to the multi-class object identification device may be a video including a plurality of frames. The multi-class object identification device may track an object included in a video. The apparatus for identifying a multi-class object may associate and track an object in units of a group class, and may determine a sub-class based on an accumulated average probability of a sub-class in a current frame. The process of tracking an object by the multi-class object identification device can be divided into an object detection step and an object tracking step. The object detection step may include performing the above-described subclass NMS and group class NMS for each frame.

That is, as illustrated in FIG. 5A, in the object detection step, a probability of a subclass of a detected object is calculated, and a group class is identified to generate one group class bounding box.

In addition, as shown in FIG. 5B, when objects of the same group are detected and associated more than a preset number of times, the multi-class object identification apparatus may initialize object tracking and activate object tracking. As an embodiment, the preset number of times in FIG. 5B is 3, and object tracking may be activated in the third frame, the k+2 frame. The apparatus for identifying a multi-class object may identify a sub-class and recognize an object based on the cumulative average probability of the sub-class calculated in the object detection step. Referring to FIG. 5B, since the subclass cumulative average probability of a van is the highest up to k+2 frames, the multi-class identification apparatus may recognize an object being tracked as a van. However, since the cumulative average probability of the SUV is highest in the k+3 frame, the object being tracked can be recognized as an SUV.

As described above, since the present disclosure recognizes the object being tracked using the cumulative average probability, if the object is continuously tracked, the average probability according to the accumulated frames may be accumulated. Accordingly, in the present disclosure, since the object being tracked is recognized based on the accumulated average probability, reliability may be increased, and object classification accuracy may be improved.

As described above, since the existing object identification/tracking technology classifies a class having a similar shape as a hard decision in the detection step, an error of determining that the same object is a different class every frame occurs. Accordingly, it may be difficult to initialize tracking due to a class classification mismatch in consecutive frames, or a problem such as erroneous detection tracking activation may occur due to the detection of multiple subclasses for the same object.

In the present disclosure, objects that are difficult to distinguish in the object detection step are configured as group classes, and objects are detected in units of group classes. The detected group class has probability distribution information for the subclass, and the subclass is estimated by accumulating and averaging probability distribution information for the subclass for each frame while tracking is performed. Accordingly, according to the present disclosure, the longer the object is tracked, the higher the reliability of the cumulative average probability distribution of the subclass and the higher the object classification accuracy.

When the present disclosure is used in a field in which objects having a similar shape, such as a vehicle type, must be classified and tracked, the accuracy and reliability of multi-object tracking may be improved. In particular, as tracking continues, the reliability of object classification increases, and thus classification accuracy for objects whose classification is ambiguous can be improved. For example, the present disclosure may be utilized in the field of detecting and tracking various objects such as CCTV, automobile, robot, and game.

So far, various embodiments of a multi-class object identification device have been described. The following describes how to identify multi-class objects.

6 is a flowchart of a method of identifying a multi-class object according to an embodiment of the present disclosure.

6, the multi-class object identification apparatus includes a learned artificial intelligence neural network and receives an image through the learned artificial intelligence neural network (S610). The received image may be a still image or a moving image including a plurality of frames.

The multi-class object identification device performs a sub-class NMS process (S620). That is, the multiclass object identification apparatus identifies a subclass corresponding to an object included in an image based on the learned artificial intelligence neural network, and generates a subclass bounding box in units of the identified subclass.

The multi-class object identification device performs a group class NMS process (S630). That is, the multi-class object identification apparatus calculates a probability for each subclass and identifies a group class. In addition, the multi-class object identification apparatus removes the generated sub-class bounding box, and generates one group class bounding box based on the identified group class.

The apparatus for identifying a multi-class object may recognize a subclass having the highest probability among the calculated probabilities for each subclass as an object, and display the generated group class bounding box and the subclass having the highest probability.

The method for identifying a multi-class object according to the various embodiments described above may be provided as a computer program product. The computer program product may include the S/W program itself or a non-transitory computer readable medium in which the S/W program is stored.

The non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short moment, such as a register, a cache, and a memory. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, or the like.

In addition, although the preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention claimed in the claims. In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

100, 100a: multi-class object identification device 110: input unit
120: processor 130: display

Claims (10)

Inputting an image to the learned artificial intelligence neural network;
A subclass for identifying at least one subclass corresponding to an object included in the image based on the learned artificial intelligence neural network, and generating at least one subclass bounding box in units of the identified at least one subclass Performing non-maximum suppression (NMS); And
Calculate a probability for each subclass, identify a group class including the identified at least one subclass based on a group class in which a plurality of subclasses corresponding to each of a plurality of homogeneous objects are grouped, and generate the And performing a group class NMS of removing at least one subclass bounding box and generating one group class bounding box based on the identified group class. The method of claim 1,
Recognizing a subclass having the highest probability among the calculated probabilities for each subclass as the object; and further comprising a multi-class object identification method. The method of claim 2,
Displaying the generated group class bounding box and the highest probability subclass. The method of claim 1,
The video is a video including a plurality of frames,
The performing of the subclass NMS and the performing of the group class NMS are performed for each of the plurality of frames,
If the group class bounding box generated in each of the plurality of frames is the same group class bounding box and is continuously generated more than a preset number of times, activating object tracking; and further comprising. The method of claim 4,
An object tracking step of accumulating averaging the probabilities of each subclass calculated from a preset number of frames, and recognizing and tracking the subclass having the highest accumulative average probability as an object; further comprising multi-class object identification Way. An input unit for receiving an image; And
Including; a processor including the learned artificial intelligence neural network,
The processor,
A subclass for identifying at least one subclass corresponding to an object included in the image based on the learned artificial intelligence neural network, and generating at least one subclass bounding box in units of the identified at least one subclass Perform NMS operation,
Calculate a probability for each subclass, identify a group class including the identified at least one subclass based on a group class in which a plurality of subclasses corresponding to each of a plurality of homogeneous objects are grouped, and generate the A multi-class object identification device that removes at least one subclass bounding box and performs a group class NMS operation to generate one group class bounding box based on the identified group class. The method of claim 6,
The processor,
A multi-class object identification apparatus for recognizing a subclass having the highest probability among the calculated probabilities for each subclass as the object. The method of claim 7,
The multi-class object identification apparatus further comprising a; display for displaying the generated group class bounding box and the subclass of the highest probability. The method of claim 6,
The video is a video including a plurality of frames,
The processor,
The subclass NMS operation and the group class NMS operation are performed for each of the plurality of frames, and the group class bounding box generated in each of the plurality of frames is the same group class bounding box and is continuously generated more than a preset number of times. In case, object tracking is activated, multi-class object identification device. The method of claim 9,
The processor,
A multi-class object identification apparatus for accumulating averaging the probabilities of each of the subclasses calculated from a preset number of frames, and recognizing and tracking the subclass having the highest accumulative average probability as an object.

Images