KR20230083130A - Apparatus and method for identifying weapon objects in military-related images and videos - Google Patents

Abstract

The present invention relates to an object recognition technique using artificial intelligence technology, wherein disclosed are an apparatus and a method for identifying a weapon object included in an image and a video related to military. In accordance with one embodiment, the apparatus for identifying a weapon object included in an image and a video related to military, includes: a deep learning model generation unit labeling a military image including weapon objects, and a representative image selected from a military video, generating a plurality of models for learning the objects included in the images, and performing learning on a model, to which the objects belong, among the plurality of models to generate a deep learning model; and an identification unit, when an image or video to be identified is inputted, identifying a weapon object included in the image or video to be identified, by referring to the generated deep learning model. Therefore, the present invention is capable of efficiently improving the recognition performance of the whole system even in a limited GPU learning resource situation.

Description

Apparatus and method for identifying weapon objects in military-related images and videos}

The present invention relates to object recognition technology using artificial intelligence technology, and relates to an apparatus and method for identifying weapon objects included in military-related images and videos.

Artificial intelligence, machine learning, and deep learning technologies are used in various fields, and accordingly, they are widely used in the image processing field, showing high recognition accuracy. For example, in ImageNet's ILSVRC (Large Scale Visual Recognition Challenge) competition, technologies that show accuracy that exceeds human recognition accuracy have already been introduced, and are useful in various application fields such as autonomous vehicles, satellite image reading, and plant disease detection. It is being utilized.

Meanwhile, this image processing technology can also be applied to military-related images or videos, and can be used to recognize weapon objects included in images or videos. However, these military-related images or videos do not have high recognition accuracy compared to general objects, and often lack learning data due to security issues, so it is not easy to increase the accuracy of the learning model through learning with a sufficient amount of data.

According to an embodiment, by applying deep learning technology to the field of military-related image recognition, an apparatus and method for identifying weapons objects in military-related images and videos optimized therefor are proposed.

In addition, we propose a system capable of labeling the collected data, classifying objects according to the characteristics of each object, generating a plurality of learning models, and verifying them.

An apparatus for identifying weapons objects in military-related images and videos according to an embodiment performs labeling on military images including weapon objects and representative images selected from military-related videos, and assigns labels to the labeled images. a deep learning model generation unit that generates a plurality of models for learning included objects, and performs learning on a model to which the object belongs among the plurality of models to generate a deep learning model; and an identification unit for identifying an inorganic object included in the identification target image or video by referring to the generated deep learning model when an identification target image or video is input.

The deep learning model generation unit may include a data collection unit that collects military images including weapon objects or selects and collects representative images for labeling of the videos from military-related videos including weapon objects; a labeling performing unit that detects objects included in the collected images and performs labeling according to a plurality of predetermined classification items; and a multi-model learning performing unit generating a plurality of models for learning of the labeled object and performing learning in each model.

And, after performing learning on the model with the added image to update the model, it may further include a model verification unit for verifying the result.

The data collection unit may divide the video into shot units and select an image having a representative characteristic of an object included in each shot as a representative image.

a multi-model learning unit generating a plurality of models based on the classification items of the labeled object; and a learning performing unit that performs learning in a corresponding model to which the object belongs among the plurality of models.

In addition, the data collection unit may generate various images by changing the collected data to additionally secure learning data.

Meanwhile, a method for identifying a weapon object in a military-related image and video according to another embodiment performs labeling on a military image including a weapon object and a representative image selected from the military-related video, and labels the labeled image. A deep learning model generation step of generating a plurality of models for learning of an object included in, and generating a deep learning model by performing learning in a model to which the object belongs among the plurality of models; and an weapon object identification step of identifying an weapon object included in the identification target image or video by referring to the generated deep learning model when an identification target image or video is input.

The deep learning model generation step may include a data collection step of collecting military images including weapon objects or selecting and collecting representative images for labeling of the videos from military-related videos including weapon objects; A labeling step of detecting objects included in the collected images and performing labeling according to a plurality of predetermined classification items; and a multi-model learning execution step of generating a plurality of models for learning the labeled object and performing learning in each model.

According to an embodiment, by creating and learning multiple learning models instead of a single learning model, the performance of the learning model can be improved more efficiently when new training data is added, and the entire system can be efficiently implemented in a limited GPU learning resource situation. Recognition performance can be improved.

In addition, even when training data is insufficient due to security issues, such as military-related images, the limit of the number of data can be overcome by increasing the number by changing the training data in various ways and using it for learning.

1 is a configuration diagram of a weapon object identification device in military-related images and videos according to an embodiment of the present invention;
2 is a detailed configuration diagram of a deep learning model generation unit according to an embodiment of the present invention;
3 is a diagram for explaining a process of selecting a representative image from video data according to an embodiment of the present invention;
4 is an exemplary table of classification of weapon objects for labeling;
5 is a diagram illustrating an example of a result of performing labeling for each object in an image according to an embodiment;
6 is a diagram for explaining criteria and examples for generating multi-learning models in the classification example table of FIG. 4;
7 is a diagram for explaining transfer learning according to an embodiment of the present invention;
8 is a diagram showing an example of increasing the number of data by changing the data collected by the data collection unit;
9 is a diagram for explaining identification of an inorganic object in an input image by an identification unit;
10 is a diagram for explaining that an identification unit identifies an inorganic object in an input video;
11 is a flowchart of a weapon object identification method in military-related images and videos according to an embodiment of the present invention.

Advantages and features of the present invention, and methods for achieving them, will become clear with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numbers designate like elements throughout the specification.

In describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted, and the terms described later will be used in the embodiments of the present invention. These terms are defined in consideration of the functions of and may vary depending on the user's or operator's intention or custom. Therefore, the definition should be made based on the contents throughout this specification.

Combinations of each block of the accompanying block diagram and each step of the flowchart may be executed by computer program instructions (execution engine), and these computer program instructions are executed by a processor of a general-purpose computer, special-purpose computer, or other programmable data processing device. Since it can be mounted, the instructions executed through the processor of a computer or other programmable data processing device create means for performing the functions described in each block of the block diagram or each step of the flowchart.

These computer program instructions may also be stored in a computer usable or computer readable memory that can be directed to a computer or other programmable data processing device to implement functionality in a particular way, such that the computer usable or computer readable memory The instructions stored in are also capable of producing an article of manufacture containing instruction means for performing the functions described in each block of the block diagram or each step of the flow chart.

And since the computer program instructions can also be loaded on a computer or other programmable data processing device, a series of operational steps are performed on the computer or other programmable data processing device to create a computer-executed process to create a computer or other programmable data processing device. It is also possible that the instructions for performing the data processing apparatus provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

In addition, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical functions, and in some alternative embodiments may refer to blocks or steps. It should be noted that it is also possible for functions to occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may be performed in the reverse order of their corresponding functions, if necessary.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention exemplified below may be modified in many different forms, and the scope of the present invention is not limited to the embodiments described below. The embodiments of the present invention are provided to more completely explain the present invention to those skilled in the art.

1 is a block diagram of an apparatus for identifying weapons objects in military-related images and videos according to an embodiment of the present invention.

The inorganic object identification device includes a deep learning model generator 110, a deep learning model 120 generated by the deep learning model 110, and an identification unit 130. The deep learning model generation unit 110 performs labeling on a military image including a weapon object and a representative image selected from a military-related video, and performs labeling on a number of objects included in the labeled image for learning. A model is created, and the deep learning model 120 is created by performing learning in a model to which the object belongs among the plurality of models.

When an identification target image or video is input, the identification unit 130 refers to the deep learning model 120 to identify the weapon object included in the identification target image or video. A detailed operation of the identification unit 130 will be described later with reference to FIGS. 9 and 10 .

2 is a detailed configuration diagram of a deep learning model generation unit according to an embodiment of the present invention.

The deep learning model generator 110 includes a data collection unit 210, a labeling unit 220, and a multi-model learning unit 230, and may further include a model verification unit 240.

The data collection unit 210 collects military images including weapon objects or selects and collects representative images for labeling of videos from military videos including weapon objects.

Military-related data used for weapon object identification may be image data and video data. Such image data and video data may be collected from the Internet, photographed directly using photographing equipment, or collected through aerial photographing using a drone or the like. In the case of an image, it can be used immediately in the labeling step, but in the case of a video, a representative image is extracted from the video and used in the labeling step.

In the case of a video, the data collection unit 210 may divide the video into shot units and select an image having a representative characteristic of an object included in each shot as a representative image. A specific example of selecting a representative image from a video will be described later in detail with reference to FIG. 3 .

The labeling performer 220 detects objects included in the collected images and performs labeling according to a plurality of predetermined classification items. Examples of classification items will be described later in detail with reference to FIG. 4 .

The multi-model learning unit 230 generates a plurality of models for learning of labeled objects and performs learning on each model. Examples of classification items will be described later in detail with reference to FIG. 6 . The multi-model learning unit 230 may include a multi-model generation unit that generates a plurality of models based on classification items of labeled objects, and a learning unit that performs learning on a corresponding model to which an object belongs among a plurality of models. can

The model verification unit 240 performs learning on the above model with the added image, updates the model, and verifies the result. A detailed procedure of verification will be described later in detail with reference to FIG. 7 .

3 is a diagram for explaining a process of selecting a representative image from video data according to an embodiment of the present invention.

When the video data 310 is input, the video data 310 is divided into shot units to select a representative image, and then a representative image is determined from among the divided images in each shot unit to form a weapon. ) is selected as the representative image of the object.

More specifically, the input video data 310 is composed of, for example, a total of 171 shots, and it is made into an automatically divided storyboard and shown to the user, and the user selects the 5th shot ( shot) 320 is selected. When the user double-clicks the selected shot 320, a list of images constituting the shot can be viewed, and among them, an image 330 showing representative characteristics of the corresponding object can be selected as the representative image.

4 is an exemplary table of classification of inorganic objects for labeling.

To perform labeling, classification items must be defined first. In this embodiment, for example, military-related image objects are classified into country names, major classifications, and intermediate classifications. In addition, in the case of the K1 tank series and the T55 series, subclassification was subdivided, and finally, classification with the goal of recognizing even the subclass code is shown.

5 is a diagram illustrating an example of a result of labeling per object in an image according to an exemplary embodiment.

For example, as shown in FIG. 4 , an object is found in a representative image according to a defined classification item, a bounding box is drawn, and corresponding information is input. Then, bounding boxes 510, 520, and 530 of objects in the given representative image are displayed, and labeled information about each object is shown.

6 is a diagram for explaining criteria and examples for generating multi-learning models in the classification example table of FIG. 4 .

In general, deep learning models are based on one model. However, according to the classification in the embodiment of the present invention, it has 19 sub-category items. Therefore, a learning model built using the training data generated by this classification outputs one of these 19 classification codes for a given input. On the other hand, when the learning model is created and used as a single model, it is not a big problem when the number of classification codes is small and the number of images is also small. However, if the number of classification codes increases and the number of images increases when GPU resource resources are limited, configuring a single model increases the time required to learn the entire training data, which can reduce the performance of the entire system.

On the other hand, in view of the characteristics of military-related images and videos to be identified in the present invention, the amount of training data increases due to similar images (K1, K1A1, K1A2, K1E1, K2 tank or T55, T59, Cheonma, etc.) of the same major classification. many. At this time, it is more effective in terms of quality control of models to learn individual models by dividing the model into several semantic units rather than learning the entire training data. When images of K1, K1A1, K1A2, K1E1, and K2 tanks are added, it is more efficient to update only the tank learning model than to reconstruct the tank model. The reason why there are so many similar images in military-related images is that in the case of the South Korean Army, a tank with gradually improved performance based on the K1 tank was deployed in actual combat, and in the case of the North Korean Army, a tank with gradually improved performance based on the T55 was deployed in actual combat. because it has been

Therefore, in an embodiment of the present invention, when similar images with the same major classification are often added in a situation where GPU resources are insufficient, multi-model learning that can partially improve performance is used. In the case of learning with a single model as in the prior art, only one learning model is created for all of the major classification, intermediate classification, and small classification. However, in the embodiment of the present invention, multiple models are created by dividing into a large category 610 of tanks and armored vehicles, a middle category and small category 620 corresponding to tanks, and a middle category and small category 630 corresponding to armored vehicles. Each of the plurality of models 610, 620, and 630 generated in this way may be used to identify a large category item or may be used to identify a middle category (or small category code) within a large category.

7 is a diagram for explaining transfer learning according to an embodiment of the present invention.

In the case of general model training, one base model is created for the entire model in the initial training stage. For example, when 100 tank images are newly added, the existing base model is updated by performing transfer learning using these images in the base model. In some cases, it is also possible to create a new base model by including additional data in the original training data.

On the other hand, in the multi-model learning according to the embodiment of the present invention, the initial learning unit 710 performs learning to create three models, for example, a base model 720, a tank model 721, and an armored vehicle model 722. . The base model 720 generated at this time is used as a model for identifying major category items (tank, armored vehicle), and the tank model 721 and the armored vehicle model 722 identify the subclass code of the actual tank and the subclass code of the armored vehicle. used to If 100 additional tank images are added, these images are applied to the base model 720 to create a new base model 740 in the transfer learning performing unit #1 730, and are also applied to the tank model 721 for transfer learning. Performer #2 (731) creates a new tank model (741). In some cases, additional data may be included in the original training data to create new base models and tank models.

Meanwhile, when a new base model 740 is created from the base model 720 through the transfer learning performing unit #1 730 or a new tank model 741 is created from the existing tank model 721, the new model To determine whether to use, for example, a mean average precision (mAP) value may be used. mAP refers to the average precision for all classes, and is a measure for comparing performance evaluations between models. For example, if the mAP value of the old base model is 0.85 and the mAP value of the new base model is 0.90, since the mAP value has increased, the performance of the new base model can be considered to be superior to that of the old base model, so the new base model can be used. However, if the mAP value of the new base model drops further, since there is garbage in the training data or other environmental problems, the cause must be found and resolved, and therefore the new base model is not used.

In determining whether to use a new model, mAP is also a major criterion, but there are other information that can be considered along with it. Such information includes, for example, model version name, learning related time, evaluation scale information, physical learning data, created learning model file, settings and related files, pre-trained file, and the like. Therefore, it is necessary to manage the version of the deep learning model so that the model once created can be recreated and the problem can be identified when the model's performance deteriorates. Table 1 below is an example of such information.

item explanation example model version name Version of generated model T-TANK-v1.0 learning start time Start time information 2021:11:18 20:00 learning end time Information on the time when learning ended 2021:11:19 20:00 class training time 24h learning data Data used for training TANK-20211118-LEARN.txtTANK-20211118-VAL.txt
TANK-20211118-TEST.txt rating scale AP, mAP, etc. AP, mAP information by class training model file Physical learning model name T-TANK-V10.weights Settings and related files Environment information used when learning T-TANK-V10.cfg T-TANK-V10.data pre-trained files In case of transfer learning, used weight file T-TANK-V09.weights

8 is a diagram illustrating an example of increasing the number of data by changing the data collected by the data collection unit.

In general, the performance of artificial intelligence learning depends on how much quality data is obtained. However, in reality, it is not easy to secure high-quality training images because military-related images have security restrictions. Therefore, after augmenting the data by artificially processing the original data, these data can be used for learning.

That is, the data collection unit 210 may generate various images by changing the collected data to additionally secure learning data. As illustrated with reference to FIG. 8 , images 820 and 821 to which weather effects have been applied are created by using various filters such as fog, snow, and rain on an original image, brightness and darkness of the original image are adjusted, or An image 822 in which various cover objects are inserted into the original image may be created. In addition to this, in the case of a low-quality image, a gamma filter is applied to convert the image 823 into a high-quality image, and training data may be augmented.

9 is a diagram for explaining identification of an inorganic object in an input image by an identification unit.

For example, when an image is input, the identification unit 130 first determines a large classification item using a base model and obtains a final classification code using a detailed classification model using the result. As shown in the example shown in FIG. 9, the input image obtained a total of major classification items and a reliability of 0.99 through the base model. From this result, it can be seen that the tank model was selected to obtain the subclass item, the final subclass code was K2 tank, and the reliability was 0.91.

10 is a diagram for explaining identification of an inorganic object in an input video by an identification unit.

For example, when a video is input, the identification unit 130 divides the video into shot units, performs object identification on all images in each shot, and displays the identified objects when playing the video. do.

In addition, objects appearing in the video can be easily identified in which time band they appeared in the video for each class through the appearance timestamp graph for each class. The timestamp graph is linked with the video object player, and when a specific object name in the timestamp graph is clicked, the video object player moves to the first appearing position of the corresponding object and is played. Also, if a random section within the time stamp is clicked, the video object player moves to the corresponding time and is played.

11 is a flowchart of a weapon object identification method in military-related images and videos according to an embodiment of the present invention.

The weapon object identification method performs labeling on military images containing weapon objects and representative images selected from military-related videos, and generates multiple models for learning objects included in the labeled images. In addition, a deep learning model creation step of generating a deep learning model by performing learning in a model to which a specific object belongs among a plurality of models, and when an image or video to be identified is input, referring to the deep learning model, and a weapon object identification step of identifying weapon objects included in the video.

More specifically, military images including weapon objects are collected, or representative images for labeling are selected and collected from military-related videos including weapon objects (1110). Here, in the data collection step, the video may be divided into shot units, and an image having a representative characteristic of an object included in each shot may be selected as a representative image. Further, learning data may be additionally secured by generating various images by changing the collected data.

Then, objects included in the collected images are detected, and labeling is performed according to a plurality of predetermined classification items (1120).

Next, multiple models are generated for learning of the labeled object, and learning is performed on each model (1130). This multi-model learning step may include a multi-model creation step of generating a plurality of models based on the classification of labeled objects, and a learning step of performing learning on a corresponding model to which a specific object belongs among a plurality of models. there is.

After updating the model by performing learning on the model with the added image, the result is verified (1140).

Meanwhile, in this embodiment, for example, only weapons used in the army have been described, but the same can be applied to various weapon systems of the navy and air force, such as airplanes and battleships.

So far, the present invention has been looked at mainly by its embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

Claims (12)

Labeling is performed on military images including weapon objects and representative images selected from military-related videos, and multiple models are created for learning objects included in the labeled images. a deep learning model generation unit for generating a deep learning model by performing learning on a model to which the object belongs; and
An apparatus for identifying weapon objects in military-related images and videos, comprising an identification unit for identifying a weapon object included in the image or video to be identified by referring to the generated deep learning model when an image or video to be identified is input. The method of claim 1, wherein the deep learning model generator,
a data collection unit that collects military images including weapon objects or selects and collects representative images for labeling of the videos from military-related videos including weapon objects;
a labeling performing unit that detects objects included in the collected images and performs labeling according to a plurality of predetermined classification items; and
An apparatus for identifying weapons objects in military-related images and videos, including a multi-model learning unit for generating a plurality of models for learning of the labeled objects and performing learning in each model. According to claim 2,
Weapon object identification device in military-related images and videos, further comprising a model verification unit that performs learning on the model with the added image to update the model and then verifies the result. The method of claim 2, wherein the data collection unit,
An apparatus for identifying weapon objects in military-related images and videos that divides the video into shot units and selects an image having a representative characteristic of an object included in each shot as a representative image. The method of claim 2, wherein the multi-model learning unit
a multi-model generating unit generating a plurality of models based on the classification items of the labeled object; and
A device for identifying weapon objects in military-related images and videos, including a learning unit for performing learning in a corresponding model to which the object belongs, among the plurality of models. The method of claim 2, wherein the data collection unit
A weapon object identification device in military-related images and videos that creates various images by changing the collected data to additionally secure learning data. Labeling is performed on military images including weapon objects and representative images selected from military-related videos, and multiple models are created for learning objects included in the labeled images. a deep learning model generation step of generating a deep learning model by performing learning in a model to which the object belongs; and
Weapon object identification in military-related images and videos including a weapon object identification step of identifying a weapon object included in the identification target image or video by referring to the generated deep learning model when an identification target image or video is input method. The method of claim 7, wherein the deep learning model generation step,
A data collection step of collecting military images including weapon objects or selecting and collecting representative images for labeling of the videos from military-related videos including weapon objects;
A labeling step of detecting objects included in the collected images and performing labeling according to a plurality of predetermined classification items; and
A method for identifying weapon objects in military-related images and videos, comprising a multi-model learning step of generating a plurality of models for learning of the labeled object and performing learning on each model. According to claim 8,
A method for identifying weapon objects in military-related images and videos, further comprising a model verification step of performing learning on the model with the added image, updating the model, and verifying the result. The method of claim 8, wherein the data collection step,
A method for identifying weapon objects in military-related images and videos by dividing the video into shot units and selecting an image having a representative characteristic of an object included in each shot as a representative image. The method of claim 8, wherein the multi-model learning step
a multi-model generation step of generating a plurality of models based on the classification items of the labeled object; and
A weapon object identification method in military-related images and videos comprising a learning execution step of performing learning in a corresponding model to which the object belongs among the plurality of models. The method of claim 8, wherein the data collection step
A weapon object identification method in military-related images and videos that creates various images by changing the collected data to additionally secure learning data.

Images