KR102566614B1 - Apparatus, method and computer program for classifying object included in image - Google Patents

Abstract

An object classification apparatus for classifying objects included in an image includes an image input unit that receives an image, a foreground object extractor that extracts a foreground object from the input image, and a segmentation class activation map from the extracted foreground object based on a deep learning algorithm ( It includes a classification area extraction unit that generates a segmentation-class activation map (S-CAM) and extracts a classification area using the S-CAM, and an object classification unit that classifies objects included in the extracted classification area.

Description

Apparatus, method and computer program for classifying objects included in images

The present invention relates to an apparatus, method, and computer program for classifying objects included in an image.

Surveillance systems using video are being used in various fields such as security management and traffic analysis. However, the performance of the video surveillance system may be significantly degraded due to various causes existing in the real environment, such as weather changes, lighting changes, and foreign substances such as insects.

In a current video surveillance system, an event detection method based on motion detection is mainly used. According to this, there is a problem in that unnecessary cost and time are consumed because an object classifier must be separately used to determine whether an object whose motion is detected corresponds to a person or a specific object.

Recently, object detection and classification methods based on convolutional neural networks (CNNs) of deep learning have been widely used. However, deep learning-based algorithms have limitations in that they require a lot of operational resources and take a relatively long processing time.

In addition, the conventional object classification method is shadow. There is a problem in that an object cannot be classified due to occlusion between objects or cannot be classified when a plurality of objects exist.

Korean Patent Publication No. 2016-0037643 discloses a configuration for setting an object candidate region for object recognition.

It is intended to provide a device, method, and computer program for classifying objects by clearly distinguishing between objects and non-objects in an image. In addition, it is intended to provide a device, method, and computer program capable of classifying each of a plurality of objects when a plurality of objects are included in an image.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problems described above, and other technical problems may exist.

As a means for achieving the above-described technical problem, an embodiment of the present invention is an object classification apparatus for classifying objects included in an image, an image input unit for receiving an image, and a foreground object extracting from the input image. Foreground object extraction unit, based on a deep learning algorithm, generates a segmentation class activation map (S-CAM, Segmentation-Class Activation Map) from the extracted foreground object, and extracts a classification area to extract a classification area using the S-CAM It is possible to provide an object classification apparatus including a unit and an object classification unit for classifying objects included in the extracted classification area.

In an embodiment, the classification area extraction unit may include a CAM generation unit generating a class activation map (CAM) using a channel corresponding to at least one category in which the object is to be classified.

In one embodiment, the classification area extraction unit may further include an S-CAM generation unit generating the S-CAM by dividing and processing the generated CAM.

In one embodiment, the S-CAM generation unit divides the CAM into a plurality of sections, derives a weight variance for each of the plurality of sections, derives a section having a minimum value among the derived weight distributions, and derives The S-CAM may be generated by binary-classifying the CAM of the section.

In one embodiment, the S-CAM generating unit may derive a maximum value and a minimum value of the CAM, and divide an interval between the derived maximum value and the derived minimum value into the plurality of intervals.

In one embodiment, the classification area extraction unit may separate and extract a plurality of classification areas corresponding to each of the plurality of objects using the S-CAM when a plurality of objects exist in the input image.

In an embodiment, the object classification unit may classify the object by determining a category of the object by determining that the object included in the classification area is one of a person, a vehicle, and an animal.

In an embodiment, a storage unit for storing metadata of objects classified by the object classification unit may be further included.

In another embodiment of the present invention, in an object classification method for classifying objects included in an image, the step of receiving an image, the step of extracting an object region from the input image, the extracted object region based on a deep learning algorithm Generating a class activation map (CAM) from, generating a segmentation CAM based on the generated CAM, extracting a classification area using the segmentation CAM, and object in the extracted classification area An object classification method including performing classification may be provided.

Another embodiment of the present invention is a computer program stored in a medium including a sequence of instructions for classifying objects included in an image, wherein the computer program, when executed by a computing device, extracts an object region from the image, and deep A computer stored in a medium including a sequence of instructions for generating a Segmentation-Class Activation Map (S-CAM) from the extracted object region based on a learning algorithm and classifying objects included in an image using the S-CAM program can be provided.

The above-described means for solving the problems is only illustrative and should not be construed as limiting the present invention. In addition to the exemplary embodiments described above, there may be additional embodiments described in the drawings and detailed description.

According to any one of the above-described problem solving means of the present invention, it is possible to provide a device, method, and computer program for classifying an object by accurately extracting a classification area.

In addition, it is possible to provide an object classification apparatus, method, and computer program for separating a plurality of objects and extracting a classification area corresponding to each object.

In addition, cost and time required for object classification can be reduced.

1 illustratively shows a problem occurring in a conventional object classification method.
2 is a block diagram of an object classification apparatus according to an embodiment of the present invention.
3 illustratively illustrates a process in which an object classification apparatus according to an embodiment of the present invention performs object classification.
4 exemplarily illustrates problems that occur when a classification region is extracted using only a class activation map (CAM).
5 exemplarily illustrates another problem that occurs when a classification region is extracted using only a class activation map (CAM).
6 illustratively illustrates a classification area extracted by the object classification apparatus according to an embodiment of the present invention using a segmentation class activation map (S-CAM).
7 illustratively shows a classification area extracted by the object classification apparatus according to an embodiment of the present invention using a segmentation class activation map (S-CAM).
8 illustratively illustrates a process of extracting a classification area for one object by an object classification apparatus according to an embodiment of the present invention.
9 exemplarily illustrates a process of separating and extracting classification areas for a plurality of objects by the object classification apparatus according to an embodiment of the present invention.
10 illustratively shows a classification region extracted using a class activation map (CAM) and a classification region extracted using a division class activation map (S-CAM).
11 exemplarily shows a service that can be used by the object classification apparatus according to an embodiment of the present invention.
12 is a flowchart of an object classification method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware. On the other hand, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or configured to reproduce one or more processors. Therefore, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal or device may be performed instead by a server connected to the terminal or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal or device connected to the corresponding server.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 illustratively shows a problem occurring in a conventional object classification method. A conventional object classification method performs object classification by extracting a region in which an object is estimated to exist based on motion in an image.

1A and 1B show a case in which a large number of non-object parts are included in an area where object classification is performed according to a conventional method. Since motion is detected not only in areas where objects (people) exist in the image, but also in areas where shadows exist, it appears that the area where object classification is performed includes many background parts, not objects. Therefore, as a result of object classification for the corresponding region, a problem occurred in that an object included in the image was not classified as a person.

1C shows a case in which a plurality of objects cannot be distinguished according to a conventional method when a plurality of objects exist in an image. When a plurality of objects exist in the image, it is not possible to determine whether the image includes a plurality of objects, and object classification cannot be performed for each of the plurality of objects.

2 is a block diagram of an object classification apparatus according to an embodiment of the present invention. Referring to FIG. 2 , the object classification apparatus 200 may include an image input unit 210, a foreground object extraction unit 220, a classification area extraction unit 230, and an object classification unit 240.

The object classification apparatus 200 may include a server, a desktop computer, a laptop computer, a KIOSK, a smartphone, and a tablet PC. However, the object classification apparatus 200 is not limited to those exemplified above. That is, the object classification apparatus 200 may include any device equipped with a processor that performs a method of classifying an object included in an image, which will be described later.

The object classification apparatus 200 may classify objects included in an image. In one embodiment, the object classification apparatus 200 may clearly distinguish between an object and a non-object part in an image. In an embodiment, when a plurality of objects are included in an image, the object classification apparatus 200 may separate and detect a plurality of objects, and perform object classification for each object.

The image input unit 210 may receive an image. For example, the image input unit 210 may receive an image from an external device such as a user terminal. The image input unit 210 may receive an image through communication with an external server. The image input unit 210 may perform preprocessing such as point noise removal on the input image.

The foreground object extractor 220 may extract a foreground object from an input image. The foreground object extractor 220 may extract a foreground object, which is an area in which an object is estimated to exist, from the input image.

The foreground object extractor 220 may extract a foreground object from an input image using a background subtraction extraction method. The difference image extraction method may use, for example, one or more algorithms of K-Nearest Neighbor (KNN), Mixture of Gaussian (MOG), and Global Minimum with a Guarantee (GMG), but is not limited thereto.

(a) of FIG. 3 illustrates a process of extracting a foreground object by the foreground object extractor 220 as an example. As shown in (a) of FIG. 3 , if the background 302 is removed from the input image 301 using a difference image extraction algorithm, the foreground object 303 can be extracted.

The classification area extractor 230 may generate a segmentation-class activation map (S-CAM) from the extracted foreground object based on a deep learning algorithm and extract the classification area using the S-CAM. .

Referring back to FIG. 2 , the classification area extraction unit 230 may include a CAM generation unit 231 . The CAM generator 231 may generate a Class Activation Map (CAM) from the foreground object extracted by the foreground object extractor 220 . The position of the object may be estimated using the class activation map (CAM) generated by the CAM generation unit 231 .

A typical CNN (Convolutional Neural Networks) algorithm is composed of a feature extraction part and a classification part. In general, the feature extraction part of CNN is composed of stacks by alternating convolutional layers and pooling layers, and the classification part consists of fully connected layers and finally It is configured to include a Softmax layer in the output layer.

In one embodiment, unlike the CNN algorithm described above, the CAM generation unit 231 may generate a class activation map (CAM) using a global average pooling (GAP) layer without using a fully connected layer. there is.

Also, in one embodiment, the CAM generation unit 231 may generate a class activation map (CAM) using a convolution layer having the same number of channels as the number of categories into which objects are to be classified. That is, the CAM generating unit 231 may generate a class activation map (CAM) using a channel corresponding to at least one category into which an object is to be classified.

For example, in the case of a device that classifies objects into categories 1 to 5, the CAM generation unit 231 may use a convolution layer having 5 channels corresponding to each category. Accordingly, a feature map derived after the convolution layer may have 5 channels. An average value derived on the basis of each channel in the five channels may be a value corresponding to each category, and object classification may be performed based on the category having the largest value.

The CAM generation unit 231 may generate a class activation map (CAM) using Equation 1 below.

where x, y are coordinate values, c is the category (discriminant class) to which the object is classified, k is each channel, M is the class activation map (CAM), w is the weight of the discriminant layer for each channel, and f may be a feature map. The CAM generation unit 231 may generate a class activation map (CAM) by multiplying an n x n matrix that has passed through the convolution layer and the pooling layer by a weight of a final discrimination layer.

(b) of FIG. 3 shows a class activation map (CAM) 312 generated by the CAM generation unit 231 from the extracted foreground object 311 as an example.

When object classification is directly performed using the class activation map (CAM) generated by the CAM generation unit 231, a problem in that the classification area is not properly extracted may occur.

(a) to (d) of FIG. 4 exemplarily illustrate problems that occur when a classification region is extracted using only a class activation map (CAM). When a plurality of objects are included in the input image, especially when the interval between the plurality of objects is narrow as shown in (a) of FIG. 4, as shown in (b) and (c) of FIG. A class activation map (CAM) that does not separate objects of can be created. Accordingly, as shown in (d) of FIG. 4 , a plurality of classification areas corresponding to each of a plurality of objects may not be separated and extracted.

(a) to (d) of FIG. 5 exemplarily illustrate other problems that occur when a classification region is extracted using only a class activation map (CAM). When an image as shown in (a) of FIG. 5 is input, a class activation map (CAM) that misclassifies parts that are not objects as objects can be generated as shown in the results of (b) and (c) of FIG. 5 . . As a result, as shown in (d) of FIG. 5 , the misclassified part may be extracted as a classification area.

In order to solve a problem that occurs when a classification region is extracted using only the class activation map (CAM) as described above, in an embodiment, a split class activation map (S-CAM) may be used. Referring back to FIG. 2 , the classification area extraction unit 230 may further include an S-CAM generation unit 232 .

The S-CAM generation unit 232 may generate S-CAMs by dividing and processing the generated CAMs. The S-CAM generator 232 may generate a split class activation map (S-CAM) from the class activation map (CAM) using a split algorithm.

The S-CAM generation unit 232 may divide the class activation map (CAM) into a plurality of sections. The S-CAM generator 232 may derive the maximum and minimum values of the class activation map (CAM), and divide a section between the derived maximum value and the derived minimum value into a plurality of sections.

The S-CAM generation unit 232 may derive a weight distribution for each of a plurality of sections. The S-CAM generation unit 232 may derive a weight distribution for each section of numbers using Equation 2 below.

Here, σ _w ² is a weight variance for any one of a plurality of sections, and a and b may respectively mean any one of binary classification. For example, a may be the foreground and b may be the background. W _a is the weight of a, μ _a is the mean of a, and σ _a ² may be the variance of a. W _b is the weight of b, μ _b is the average of b, and σ _b ² may be the variance of b.

For example, when a plurality of intervals is a total of 10 intervals, when deriving the weight distribution σ _w ² of s intervals, ego, ego, can be also, ego, ego, can be X _i may be the number of values belonging to the i-th section when the real value of the class activation map (CAM) is divided into a plurality of sections based on the minimum and maximum values. For example, X ₄ may be 3 when all real values of CAM belonging to 4 intervals (0.4 to 0.5) are obtained and 0.44, 0.41, and 0.46 are obtained.

The S-CAM generating unit 232 may derive a section having a minimum value among weight distributions among a plurality of sections. The S-CAM generation unit 232 may binary classify the class activation map (CAM) of the derived section to generate a split class activation map (S-CAM).

For example, when the minimum value of the class activation map (CAM) is 0.0 and the maximum value is 1.0, an interval from 0.0 to 1.0 of the class activation map (CAM) may be divided by an interval of 0.1. Accordingly, the class activation map (CAM) may be divided into a total of 10 sections, such as a first section from 0.0 to 0.1 and a second section from 0.1 to 0.2. In this case, weight variance for each of a plurality of sections can be derived as shown in Table 1 below.

In Table 1, the weight variance appears to have a minimum value in the fourth interval. Accordingly, the S-CAM generation unit 232 may binary classify the class activation map (CAM) of the fourth section to generate a split class activation map (S-CAM).

(c) of FIG. 3 shows split class activation maps (S-CAMs) 321, 322, and 323 generated by the S-CAM generation unit 232 as an example. Reference numerals 321, 322, and 323 denote split class activation maps (S-CAMs) generated by binary classification of class activation maps (CAMs) of different intervals, respectively. Referring to (c) of FIG. 3 , it can be seen that the split class activation maps (S-CAMs) generated from each of a plurality of sections of the class activation map (CAM) are different.

The classification area extractor 230 may extract a classification area using a segmentation class activation map (S-CAM).

For example, as shown in (d) of FIG. 3 , the classification area extraction unit 230 places a box in the classification area 332 from the segmentation class activation map (S-CAM) 331 of the section having the minimum weight variance. marked part) can be extracted.

When a plurality of objects exist in the input image, the classification region extractor 230 may separate and extract a plurality of classification regions corresponding to each of the plurality of objects using a segmentation class activation map (S-CAM). .

(a) of FIG. 6 is a split class activation map (S-CAM) generated by binary classification of the class activation map (CAM) of a section with a weight variance of 0.750, and (b) of FIG. 6 is a split class activation map (S-CAM) of (a). Classification regions extracted using a class activation map (S-CAM) are shown.

(c) of FIG. 6 is a split class activation map (S-CAM) generated by binary classification of the class activation map (CAM) of a section with a weight variance of 0.409, and (d) of FIG. 6 is a split class activation map (S-CAM) of (c). Classification regions extracted using a class activation map (S-CAM) are shown.

(e) of FIG. 6 is a split class activation map (S-CAM) generated by binary classification of the class activation map (CAM) of a section with a weight variance of 2.280, and (f) of FIG. 6 is a split class activation map (S-CAM) of (e). Classification regions extracted using a class activation map (S-CAM) are shown.

Referring to FIG. 6, in the case of sections (c) and (d) in which the weight variance has the minimum value in an image including a plurality of objects, a plurality of classification regions corresponding to each of the plurality of objects are most appropriately separated and extracted. It became. That is, it can be seen that the problem of not being able to separate a plurality of objects when a classification area is extracted using only the class activation map (CAM) shown in FIG. 4 is solved by using the split class activation map (S-CAM). .

(a) of FIG. 7 is a split class activation map (S-CAM) generated by binary classification of the class activation map (CAM) of a section with a weight variance of 0.579, and (b) of FIG. 7 is a split class activation map (S-CAM) of (a). Classification regions extracted using a class activation map (S-CAM) are shown.

(c) of FIG. 7 is a split class activation map (S-CAM) generated by binary classification of a class activation map (CAM) of a section with a weight variance of 0.345, and (d) of FIG. 7 is a split class activation map (S-CAM) of (c). Classification regions extracted using a class activation map (S-CAM) are shown.

(e) of FIG. 7 is a split class activation map (S-CAM) generated by binary classification of the class activation map (CAM) of a section with a weight variance of 2.067, and (f) of FIG. 7 is a split class activation map (S-CAM) of (e). Classification regions extracted using a class activation map (S-CAM) are shown.

Referring to FIG. 7 , in the case of (c) and (d), where the weight variance has the minimum value, the object and non-object part are clearly distinguished, and the most appropriate classification area without noise is extracted. That is, it can be confirmed that the problem of not removing noise when extracting a classification region using only the class activation map (CAM) shown in FIG. 5 is solved by using the split class activation map (S-CAM).

The object classification unit 240 may classify objects included in the extracted classification area. The object classification unit 240 may determine whether an object included in the input image corresponds to one category. The object classification unit 240 may determine at once whether an object included in an input image corresponds to any one of a plurality of categories.

The object classifier 240 may determine the category of the object using a Convolutional Neural Network (CNN) classifier. The convolutional neural network classifier may include a part for extracting features of an image and a part for determining (classifying) a class of an object.

The part that extracts the feature of the image includes a plurality of convolution layers and a plurality of pooling layers, and the part that determines the class of the object is a layer for image classification, for example, a fully connected It may include a layer (Fully Connected Layer). Since the CAM generation unit 231 generates a class activation map (CAM) based on the value obtained by multiplying the weight (w) of the discrimination layer for each channel and the feature map (f) (see Equation 1), the convolutional neural network classifier and The class activation map (CAM) must have the same structure as the convolutional layer and previous layers.

For the convolutional neural network classifier having the structure described above, images of objects of a plurality of categories may be trained. For example, if a convolutional neural network classifier is trained on each image of a person, car, or animal, the trained convolutional neural network classifier derives a probability corresponding to each category for an object included in the classification area, and obtains the highest probability value. Objects can be classified based on

In an embodiment, the object classification unit 240 may classify the object by determining the object category by determining the object included in the classification area as one of a person, a vehicle, and an animal. In another embodiment, the object classification unit 240 may derive a probability that an object included in the classification area corresponds to any one category among a plurality of categories.

For example, the object classification unit 240 may determine whether an object included in the classification area as shown in (e) of FIG. 3 is a human, and derive a 98% probability that the object corresponds to a human. there is. Accordingly, the object may be classified as a person.

8 (a) to (d) exemplarily illustrate a process of extracting a classification region for one object by the object classification apparatus 200 according to an embodiment of the present invention. The image input unit 210 may receive an image as shown in (a) of FIG. 8 , and the foreground object extractor 220 may extract a foreground object from the input image. The classification region extraction unit 230 may generate a division class activation map (S-CAM) as shown in (b) of FIG. 8 and extract a classification region as shown in (c) of FIG. The object classification unit 240 may classify an object included in the classification area shown in (d) of FIG. 8 as a person.

9(a) to (d) exemplarily illustrate a process of extracting classification regions for a plurality of objects by the object classification apparatus 200 according to an embodiment of the present invention. The image input unit 210 may receive an image including a plurality of objects as shown in (a) of FIG. 9 , and the foreground object extractor 220 may extract a foreground object from the input image. The classification area extractor 230 generates a split class activation map (S-CAM) as shown in FIG. Areas can be separated and extracted. The object classification unit 240 may classify each object included in each classification area shown in (d) of FIG. 9 as a person.

FIG. 10(a) shows a classification area extracted using only a class activation map (CAM), and FIG. 10(b) shows a classification area extracted using a split class activation map (S-CAM). For example, the object classification unit 240 may derive that the probability that the object included in the classification area of FIG. 10 (a) is a person is 56%, but the objects included in each classification area of FIG. 10 (b) It can be derived that the probability that the object is a person is 98%.

The object classification apparatus 200 may further include a storage unit (not shown) for storing metadata of objects classified by the object classification unit 240 . In another embodiment, the object classification device 200 may transmit metadata of objects classified by the object classification unit 240 to an external device.

11(a) to (c) illustrate services that can be provided using the object classification apparatus 200 according to an embodiment of the present invention.

As shown in (a) of FIG. 11 , the object classification apparatus 200 may be used to provide a service for detecting intrusion of an object from an image. It can detect the position of a moving object in a specific area and classify what the object is. Objects included in images captured using a plurality of cameras or real-time images may be classified. Intrusion can be determined by classifying pets, cars, and people.

As shown in (b) of FIG. 11 , the object classification apparatus 200 may be used to provide a service for determining whether a specific object exists in an image. For example, the number of cars present in the parking lot may be derived or whether there is a parking space may be determined.

As shown in (c) of FIG. 11 , the object classification apparatus 200 may be used to determine the number of objects included in an image, particularly population density. Also, the object classification apparatus 200 may be used to detect the density of irregular objects such as water and fog.

12 is a flowchart of an object classification method according to an embodiment of the present invention. The method 1200 of classifying an object performed by the apparatus 200 shown in FIG. 12 includes steps processed time-sequentially by the apparatus 200 according to the embodiment shown in FIG. 2 . Therefore, even if the contents are omitted below, they are also applied to the method of classifying objects performed in the apparatus 200 according to the embodiment shown in FIG. 2 .

In step S1201, the device 200 may receive an image.

In step S1202, the device 200 may extract an object area from the input image.

In step S1203, the device 200 may generate a Class Activation Map (CAM) from the extracted object region based on the deep learning algorithm.

In step S1204, the apparatus 200 may generate a segmented CAM based on the generated CAM.

In step S1205, the apparatus 200 may extract a classification area using segmented CAM.

In step S1206, the device 200 may perform object classification in the extracted classification area.

In step S1204 of generating the segmentation CAM, the segmentation process of the CAM may generate the segmentation CAM that clarifies object estimation by separating objects and removing noise.

In the above description, steps S1201 to S1206 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Also, some steps may be omitted as needed, and the order of steps may be switched.

The method of classifying objects in the object classification apparatus described with reference to FIGS. 1 to 12 may be implemented in the form of a computer program stored in a medium executed by a computer or a recording medium including instructions executable by a computer. Also, the method of classifying objects in the object classification apparatus described with reference to FIGS. 1 to 12 may be implemented in the form of a computer program stored in a medium executed by a computer.

Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

200: object classification device
220: foreground object extraction unit
230: classification area extraction unit
240: object classification unit

Claims (16)

An object classification apparatus for classifying objects included in an image,
an image input unit that receives an image;
a foreground object extraction unit extracting a foreground object from the input image;
Based on a deep learning algorithm, a class activation map (CAM) is generated from the extracted foreground object, and a segmentation-class activation map (S-CAM) is generated based on the generated CAM, , a classification area extraction unit extracting a classification area using the S-CAM; and
An object classification unit for classifying objects included in the extracted classification area.
including,
The classification area extraction unit further includes an S-CAM generation unit configured to divide and process the generated CAM to generate the S-CAM,
The S-CAM generation unit divides the CAM into a plurality of sections, derives a weight distribution for each of the plurality of sections based on the variance of the foreground object and the background included in each of the plurality of sections, and the derived Generating the S-CAM based on a section having a minimum value among weight distributions;
The section having the minimum value corresponds to a plurality of classification areas corresponding to each of a plurality of objects or a section in which a classification area from which noise is removed is extracted through classification of the object and a non-object part, .
According to claim 1,
The object classification apparatus of claim 1 , wherein the classification area extraction unit includes a CAM generation unit generating the CAM using a channel corresponding to at least one category into which the object is to be classified.
delete According to claim 1,
Wherein the S-CAM generation unit generates the S-CAM by binary classifying a CAM of a section having a minimum value among the derived weight distributions.
According to claim 4,
The S-CAM generation unit derives the maximum value and the minimum value of the CAM, and divides a section between the derived maximum value and the derived minimum value into the plurality of sections.
According to claim 1,
wherein the classification area extraction unit separates and extracts a plurality of classification areas corresponding to each of the plurality of objects using the S-CAM when a plurality of objects exist in the input image.
According to claim 1,
wherein the object classification unit classifies the object by determining a category of the object by determining an object included in the classification area as one of a person, a vehicle, and an animal.
According to claim 1,
The object classification apparatus further comprising a storage unit for storing metadata of objects classified by the object classification unit.
An object classification method for classifying objects included in an image,
Receiving an image;
extracting an object area from the input image;
generating a class activation map (CAM) from the extracted object region based on a deep learning algorithm;
generating a division CAM based on the generated CAM;
extracting a classification area using the segmented CAM; and
Performing object classification in the extracted classification area
including,
The step of generating the segmented CAM,
Dividing the CAM into a plurality of sections;
deriving a weight variance for each of the plurality of sections based on the variance of the electric object and the background included in each of the plurality of sections;
deriving a section having a minimum value among the derived weight distributions; and
Generating the divided CAM based on the interval having the minimum value;
The section having the minimum value corresponds to a plurality of classification areas corresponding to each of a plurality of objects or a section in which a classification area from which noise is removed is extracted through classification of the object and a non-object part, object classification method .
According to claim 9,
Wherein the generating of the CAM uses a channel corresponding to at least one category into which the object is to be classified.
According to claim 10,
The object classification method of claim 1, wherein the generating of the split CAM generates the split CAM that clarifies object estimation by dividing and processing the CAM to separate objects and remove noise.
According to claim 11,
The step of generating the segmented CAM is
Further comprising the step of generating the segmented CAM by binary classifying the CAM of the derived section.
According to claim 11,
The step of extracting the classification area comprises separating and extracting a plurality of classification areas corresponding to each of the plurality of objects using the split CAM when a plurality of objects exist in the input image. .
According to claim 9,
Wherein the step of classifying the object classifies the object by determining a category of the object by determining an object included in the classification area as one of a person, a vehicle, and an animal.
According to claim 9,
The object classification method further comprising the step of storing metadata of the object classified in the step of classifying the object.
A computer program stored in a medium containing a sequence of instructions for classifying an object included in an image,
When the computer program is executed by a computing device,
extracting an object region from an image;
Based on a deep learning algorithm, a Class Activation Map (CAM) is generated from the extracted object region, and a Segmentation-Class Activation Map (S-CAM) is generated based on the generated CAM,
A sequence of instructions for classifying an object included in an image using the S-CAM,
Dividing the CAM into a plurality of sections, deriving a weight distribution for each of the plurality of sections based on the variance of the foreground object and the background included in each of the plurality of sections, and having a minimum value among the derived weight distributions Further comprising a sequence of instructions to generate the S-CAM based on an interval;
The interval having the minimum value corresponds to a plurality of classification areas corresponding to each of a plurality of objects or a period in which a classification area from which noise is removed is extracted through classification of the object and a non-object part, stored in the medium. computer program.