KR20240029239A - Image learning processing system for object estimation - Google Patents

Abstract

The present invention relates to an image learning processing system for object estimation. The image learning processing system of the present invention trains an object estimation segment model for one or more first input images, and uses the object estimation segment model to determine the first input image. A learning unit that trains the object estimation segment model to classify objects included in an input image and extract corresponding object information, and an object estimated through the object estimation segment model learned by the learning unit for one or more second input images. A test unit that generates correction information that represents classification and object information with actual data, and compares the correction information with ground truth for the one or more second input images and the correction information corresponding thereto, and performs additional learning according to the comparison result. It includes a model estimation unit that collects images classified as training data for an additional learning repository and retrains the object estimation segment model using the data set of images collected in the additional learning repository.

Description

Image learning processing system for object estimation}

The present invention relates to an image learning processing system, and in particular, to an image learning processing system for efficiently estimating objects included in images and corresponding object information from aerial drone images, etc.

There have been significant advances in modern remote sensing technology in recent years, including the development of small, commercial, intelligent and inexpensive satellites and now the widespread availability of unmanned aerial vehicles (UAVs). Remote sensing involves the use of high-resolution optical devices (cameras or sensors) to obtain information remotely. It typically involves satellites, unmanned aerial vehicles and a collection of airborne sensors. However, advances in remote sensing technology have led to a rapid increase in the number of remotely sensed images and videos, which are changing significantly in terms of spatial resolution, data quality, and coverage of usable areas, locations, and scenes. Remotely sensed images and video can help you efficiently control, monitor, and gather valuable information.

Increasing advancements in efficient remote sensing technologies offer excellent opportunities for diverse applications such as urban management, land change monitoring, and traffic monitoring. Among various applications, object detection and segmentation from high-resolution images and videos are being considered more in the field of remote sensing. However, efficient classification, detection, and segmentation of various objects in remote sensing images is challenging in research and development due to various factors including camera height, object appearance, and various background and environmental conditions.

Therefore, the present invention was created to solve the above-mentioned problems, and the purpose of the present invention is to recognize objects included in input images such as aerial drone images and use a learning model to estimate the corresponding object information to identify various objects in the image. The goal is to provide an image learning processing system to efficiently classify and obtain detailed information.

First, to summarize the features of the present invention, the image learning processing system according to one aspect of the present invention for achieving the above object trains an object estimation segment model for one or more first input images, and the object estimation segment model a learning unit that classifies objects included in the first input image and trains the object estimation segment model to extract corresponding object information; a test unit that generates correction information that represents classification of an object and object information estimated through the object estimation segment model learned by the learning unit for one or more second input images and actual data; And for the one or more second input images and the corresponding correction information, compare the correction information with ground truth and collect images classified as learning data for additional learning according to the comparison result in an additional learning storage, It includes a model estimation unit that retrains the object estimation segment model using a data set of images collected in an additional learning repository.

According to the comparison result, the model estimator is configured to calculate the false positive (FP) image and the false negative (FN) image, excluding the true positive (TP) image and the true negative (TN) image, among the second input images. It can be classified as learning data for further learning.

The model estimation unit calculates a harmonic average value (F1-score) of the recall value (Rec) and the precision value (Prec) for the second input image according to the following equations, and calculates the harmonic mean value (F1-score) of the recall value (Rec) and the precision value (Prec) for the second input image. Images that are below the average value (F1-score) are classified as training data for further learning,

Rec = NTP/(NTP+ NFN)

Prec = NTP/(NTP+ NFP)

Here, NTP is the area of one or more objects in the image estimated to be a true positive (TP) image, NFN is the area of one or more corresponding objects in the image estimated to be a false negative (FN) image, and NFP is a false positive (FP) image. It may be the area of one or more objects within the image, which is estimated to be an image.

The object information may include one or more of object classification information, object size, shooting location information, altitude, azimuth, or color.

And, according to another aspect of the present invention, a recording medium recording a computer-readable code for estimating an object and object information included in a learning-based image learns an object estimation segment model for one or more first input images. A learning function for classifying objects included in the first input image using the object estimation segment model and training the object estimation segment model to extract corresponding object information; a test function for generating correction information representing classification of an object and object information estimated through the object estimation segment model learned for one or more second input images and actual data; And for the one or more second input images and the correction information corresponding thereto, compare the correction information with ground truth and collect images classified as learning data for additional learning according to the comparison result in an additional learning storage, It may include a recording medium for implementing a model estimation function that retrains the object estimation segment model using a data set of images collected in the additional learning storage.

According to the image learning processing system according to the present invention, objects included in input images such as aerial drone images are recognized and an object estimation segment model is trained to estimate the corresponding object information to efficiently classify various objects in the image and provide detailed information. An image learning processing system can be provided to enable acquisition.

The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide embodiments of the present invention and explain the technical idea of the present invention along with the detailed description.
FIG. 1 is a diagram illustrating an image learning processing system 100 for deep learning-based object and detailed information estimation according to an embodiment of the present invention.
Figure 2 illustrates images of augmented data sets generated by the data augmentation unit 120 according to an embodiment of the present invention.
Figure 3 is a diagram for explaining the operation of the model estimation unit 150 according to an embodiment of the present invention.
Figure 4 is an example of an input original image, a predicted image based on an object estimation segment model, and a true label mask.
Figure 5 is a diagram for explaining an example of a method of implementing the image learning processing system 100 according to an embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the attached drawings. At this time, the same components in each drawing are indicated by the same symbols whenever possible. Additionally, detailed descriptions of already known functions and/or configurations will be omitted. The content disclosed below focuses on the parts necessary to understand operations according to various embodiments, and descriptions of elements that may obscure the gist of the explanation are omitted. Additionally, some components in the drawings may be exaggerated, omitted, or shown schematically. The size of each component does not entirely reflect the actual size, and therefore the content described here is not limited by the relative sizes or spacing of the components drawn in each drawing.

In describing the embodiments of the present invention, if it is determined that a detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification. The terminology used in the detailed description is only for describing embodiments of the present invention and should in no way be limiting. Unless explicitly stated otherwise, singular forms include plural meanings. In this description, expressions such as “comprising” or “comprising” are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and one or more than those described. It should not be construed to exclude the existence or possibility of any other characteristic, number, step, operation, element, or part or combination thereof.

In addition, terms such as first, second, etc. may be used to describe various components, but the components are not limited by the terms, and the terms are used for the purpose of distinguishing one component from another component. It is used only as

FIG. 1 is a diagram illustrating an image learning processing system 100 for deep learning-based object and detailed information estimation according to an embodiment of the present invention.

Referring to FIG. 1, the image learning processing system 100 includes a preprocessor 110, a data augmentation unit 120, a learning unit 130, a test unit 140, and a model estimation unit 150.

The preprocessor 110 performs image resizing, shuffling (e.g., adjusting pixel data positions for obfuscation), and normalization (e.g., brightness and contrast) for the collected data set and the corresponding true value images. Perform preprocessing such as adjusting etc. The collected data set may, for example, consist of data collected from aerial images using drones, and for learning purposes, the collected images include true images as reference actual images and labeling data ( For example, the correspondence is based on object classification information, object size, shooting location information, altitude, azimuth, color, etc.). Collected images (e.g., 400 images, resolution 6000 x 4000 pixels) and true images can be vectorized by embedding using an image embedding algorithm. The preprocessor 110 may perform preprocessing on the vectorized collected images.

The data augmentation unit 120 performs data augmentation on the preprocessed images (data set) using a data augmentation algorithm to generate an augmented data set and uses it for learning, thereby providing a sufficient level of architecture through the diversity of the data set. It ensures dispersion and robustness and prevents excessive fitting to improve system efficiency.

Figure 2 illustrates images of augmented data sets generated by the data augmentation unit 120 according to an embodiment of the present invention.

As shown in FIG. 2, for example, with respect to the original preprocessed images, the data augmentation unit 120 generates images with different altitudes (vertical shift) and images with different horizontal positions (Herizontal shift). ) images, images that are symmetrical (flipped) to the horizontal or vertical axis of the photo, or rotated images, etc., can be augmented and used for learning.

The learning unit 130 performs deep learning-based learning on some of the augmented data sets from the data augmentation unit 120 (e.g., 200 randomly selected images out of 400 collected images) to create an object estimation segment model. learn Here, for certain applications, for example, machine learning devices such as VGG-16, ResNet 50 and MobileNet for classification of objects in images, and machine learning devices such as U-Net for segmentation of objects in images may be used. there is.

The object estimation segment model classifies and distinguishes objects in the image (e.g., roofs, cars, trees, people, roads, warehouses, mountains, rivers, etc.) from the input image, and uses labeling data (e.g., object classification information, object size ( Area), shooting location information, altitude, azimuth, color, etc.) are compared with the true value images to determine corresponding object information (e.g., object classification information for objects, object size (area), shooting location information, altitude, azimuth, As a model for extracting (color, etc.), the learning unit 130 receives the augmented data set from the data augmentation unit 120, classifies the objects included in the input image using an object estimation segment model for each, and The object estimation segment model is trained to extract the corresponding object information and the necessary learning parameters are updated.

The object classification information is classification information for objects such as roofs, cars, trees, people, roads, warehouses, mountains, and rivers. The shooting location information may be GPS (Global Positioning System) information indicating the shooting location.

The test unit 140 uses the learning unit 130 for the remaining data (e.g., 200 of the 400 collected images excluding the 200 used for learning) of the augmented data set from the data augmentation unit 120. Correction information is generated by performing a test on the object estimation segment model learned from.

For example, the test unit 140 inputs the test data set (test image) into the object estimation segment model learned by the learning unit 130 to classify objects included in the image (e.g., roof, car, trees, people, roads, warehouses, mountains, rivers, etc.) and whether the estimation of corresponding object information is normal, the difference from actual data (e.g., actual object to be classified and corresponding object information) can be generated as correction information. there is.

The model estimation unit 150 reflects the correction information from the test unit 140 to the learning parameters of the learned object estimation segment model, and provides a corrected object estimation segment that can precisely estimate the object and object information for the input image. The learned object estimation segment model is trained to complete the model.

Figure 3 is a diagram for explaining the operation of the model estimation unit 150 according to an embodiment of the present invention.

Referring to FIG. 3, the model estimation unit 150 generates a predicted image (see FIG. 4) generated through image processing of each of the test data sets, that is, test images, using an algorithm such as feature point extraction. Based on this, objects can be classified and object information can be estimated, and this is compared with true images (True labe mask) (see FIG. 4) corresponding to ground truth (correct answer/true value).

That is, the model estimation unit 150 converts the correction information into corresponding true values for the test data set, that is, each test image and the corresponding correction information based on the classification and object information (predicted results) of the estimated object. According to the result of the difference compared to the corresponding ground truth, the test image is classified into four types, namely, true positive (TP) image 210, true negative (TN) image 210, and true negative (TN) image 210. They are classified into images (220), false positive (FP) images (230), and false negative (FN) images (240).

The true positive (TP, True Positive) image 210 has object information estimated by the learned object estimation segment model, and if the information matches the ground truth within a predetermined range, the true negative (TN, True Negative) image 220 determines that there is no object estimated by the learned object estimation segment model, but if there is an object based on ground truth, the false positive (FP) image 230 is determined by the learned object estimation segment model. If it is determined that there is object information estimated by the object estimation segment model, but there is no corresponding object based on the ground truth, the false negative (FN, False Negative) image 240 is generated using the object information estimated by the learned object estimation segment model. It is determined that there is no object, but based on the ground truth, there is an object and there is information about the object.

The model estimation unit 150 generates a true positive (TP) image 210 and a true negative (TN) image (True Negative) whose difference is within a predetermined range for the images of the test data set. Except for 220), the false positive (FP) image 230 and the false negative (FN) image 240 are classified as learning data for further training of the learned object estimation segment model. , the test images can be collected in a designated additional learning storage such as memory. This means that when there are multiple objects (e.g., 20 roofs, etc.) in the test image, even for one object, a false positive (FP) image 230 and a false negative (FN) image 230 are used as above. This may include cases of incorrect judgment (240). The model estimation unit 150 may retrain the object estimation segment model using a data set of image(s) collected in the additional learning storage. Here, the deep learning-based learning algorithm of the learning unit 110 may be used.

Furthermore, the model estimation unit 150 calculates the ground truth and true positive (TP) images 210 according to [Equation 1] for the images of the test data set and sets a recall value (Rec) (the difference Calculate the ratio of objects predicted by the model to be true among objects with actual true values within a predetermined range) and calculate the precision value (Prec) according to [Equation 2] (the difference among objects classified by the model as actual true values) Calculate the ratio of objects that are actual true values within a predetermined range) and calculate the harmonic mean (F1-score) according to [Equation 3], and In relation to this, the test images can be classified as learning data for additional learning of the learned object estimation segment model, and collected in a predetermined additional learning storage such as memory. Likewise. The model estimation unit 150 may retrain the object estimation segment model using a data set of image(s) collected in the additional learning storage. Here, the deep learning-based learning algorithm of the learning unit 110 may be used.

[Equation 1]

Rec = NTP/(NTP+ NFN)

[Equation 2]

Prec = NTP/(NTP+ NFP)

[Equation 3]

Here, NTP is the (pixel) area of the corresponding object(s) (e.g., estimated object) in the test image estimated to be a true positive (TP) image 210, and NFN is estimated to be a false negative (FN) image 240. (pixel) area of the corresponding object(s) (e.g., real object) within the estimated test image, NFP is the false positive (FP) image 230. is the (pixel) area of.

Figure 5 is a diagram for explaining an example of a method of implementing the image learning processing system 100 according to an embodiment of the present invention.

Referring to FIG. 5, the image learning processing system 100 according to an embodiment of the present invention may be comprised of hardware, software, or a combination thereof. For example, the image learning processing system 100 of the present invention may be implemented in the form of a server on the Internet or a computing system 1000 as shown in FIG. 4 having at least one processor for performing the above functions/steps/processes. there is.

Computing system 1000 includes at least one processor 1100, memory 1300, user interface input device 1400, user interface output device 1500, storage 1600, and network connected through bus 1200. It may include an interface 1700. The processor 1100 may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in the memory 1300 and/or storage 1600. Memory 1300 and storage 1600 may include various types of volatile or non-volatile storage media. For example, the memory 1300 may include a read only memory (ROM) 1310 and a random access memory (RAM) 1320.

In addition, the network interface 1700 is a communication module such as a modem that supports wired Internet communication in user terminals such as smartphones, laptop PCs, and desktop PCs, wireless Internet communication such as WiFi and WiBro, and mobile communication such as WCDMA and LTE, or a short-distance communication module. It may include a communication module such as a modem that supports wireless communication (e.g., Bluetooth, ZigBee, Wi-Fi, etc.).

Accordingly, steps of a method or algorithm described in connection with the embodiments disclosed herein may be implemented directly in hardware, software modules, or a combination of the two executed by processor 1100. The software module is a storage/recording medium (i.e., memory 1300 and Alternatively, it may reside in storage 1600). An exemplary storage medium is coupled to a processor 1100, which can read information (code) from and write information (code) to the storage medium. Alternatively, the storage medium may be integrated with processor 1100. The processor and storage medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

As described above, according to the image learning processing system 100 according to the present invention, objects included in input images such as aerial drone images are recognized and an object estimation segment model for estimating the corresponding object information is learned to identify various objects in the image. An image learning processing system can be provided to efficiently obtain classification and detailed information.

As described above, the present invention has been described with specific details such as specific components and limited embodiments and drawings, but this is only provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , those of ordinary skill in the field to which the present invention pertains will be able to make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the spirit of the present invention should not be limited to the described embodiments, and the scope of the patent claims described later as well as all technical ideas that are equivalent or equivalent to the scope of this patent claim are included in the scope of the rights of the present invention. It should be interpreted as

Preprocessing unit (110)
Data augmentation unit (120)
Learning Department (130)
Test section (140)
Model estimation unit (150)

Claims (8)

An object estimation segment model is trained for one or more first input images, and the object estimation segment model is trained to classify objects included in the first input image and extract corresponding object information using the object estimation segment model. wealth;
a test unit that generates correction information that represents classification of an object and object information estimated through the object estimation segment model learned by the learning unit for one or more second input images and actual data; and
For the one or more second input images and the correction information corresponding thereto, the correction information is compared with ground truth, and images classified as learning data for additional learning are collected in an additional learning storage according to the comparison result, and the additional learning storage is stored. A model estimation unit that retrains the object estimation segment model using a data set of images collected in the learning storage.
An image learning processing system including. According to paragraph 1,
The model estimation unit,
According to the comparison result, training for the additional learning is performed on false positive (FP) images and false negative (FN) images, excluding true positive (TP) images and true negative (TN) images, among the second input images. An image learning processing system that classifies data. According to paragraph 1,
The model estimation unit calculates a harmonic average value (F1-score) of the recall value (Rec) and the precision value (Prec) for the second input image according to the following equations, and calculates the harmonic mean value (F1-score) of the recall value (Rec) and the precision value (Prec) for the second input image. Images that are below the average value (F1-score) are classified as training data for further learning,
Rec = NTP/(NTP+ NFN)
Prec = NTP/(NTP+ NFP)

Here, NTP is the area of one or more objects in the image estimated to be a true positive (TP) image, NFN is the area of one or more corresponding objects in the image estimated to be a false negative (FN) image, and NFP is a false positive (FP) image. An image learning processing system in which the area of one or more objects in an image is estimated to be an image. According to paragraph 1,
The object information includes one or more of object classification information, object size, shooting location information, altitude, azimuth, or color. A recording medium recording a computer-readable code for estimating objects and object information included in learning-based images,
An object estimation segment model is trained for one or more first input images, and the object estimation segment model is trained to classify objects included in the first input image and extract corresponding object information using the object estimation segment model. function;
a test function for generating correction information representing classification of an object and object information estimated through the object estimation segment model learned for one or more second input images and actual data; and
For the one or more second input images and the correction information corresponding thereto, the correction information is compared with ground truth, and images classified as learning data for additional learning are collected in an additional learning storage according to the comparison result, and the additional learning storage is collected. Model estimation function that retrains the object estimation segment model using the data set of images collected in the learning storage.
A recording medium for implementing. According to clause 5,
In the model estimation function,
According to the comparison result, training for the additional learning is performed on false positive (FP) images and false negative (FN) images, excluding true positive (TP) images and true negative (TN) images, among the second input images. A recording medium classified as data. According to clause 5,
In the model estimation function, the harmonic mean value (F1-score) of the recall value (Rec) and the precision value (Prec) is calculated for the second input image according to the following equations, and the harmonic mean value (F1-score) of the recall value (Rec) and the precision value (Prec) is calculated for the second input image Images that are less than the harmonic mean value (F1-score) are classified as learning data for further learning,
Rec = NTP/(NTP+ NFN)
Prec = NTP/(NTP+ NFP)

Here, NTP is the area of one or more objects in the image estimated to be a true positive (TP) image, NFN is the area of one or more corresponding objects in the image estimated to be a false negative (FN) image, and NFP is a false positive (FP) image. A recording medium that is the area of one or more objects within an image assumed to be an image. According to clause 5,
The object information includes one or more of object classification information, object size, shooting location information, altitude, azimuth, or color.