RU2747044C1 - Hardware-software complex designed for training and (or) re-training of processing algorithms for aerial photographs of the territory for detection, localization and classification up to type of aviation and ground equipment - Google Patents

Abstract

FIELD: image data processing.

SUBSTANCE: invention relates to a complex and a method of forming a training sample intended for training and/or re-training of processing algorithms for aerial photographs of the terrain for detection, localization and classification up to type of aviation and ground equipment. According to the method, large-format aerial photographs presented as color or grayscale digital images are imported by means of an import unit; the imported data from the data import unit is received by means of a neural network core, and neural network models are selected from the catalog of neural network models; the original large-format aerial photographs are marked by means of a neural network marking unit using the selected neural network model; the obtained neural network marking is corrected by means of a neural network marking correction unit; the volume of images for training samples is automatically expanded by means of an automatic augmentation module; the corrected marked images are transferred to the training and test sample database with expert marking, providing continuous updates of the database, wherein the training samples are balanced by the number of objects of interest and the number of background elements by means of the training and test sample database with expert marking; the neural network models from the catalog of neural network models are re-trained by means of a re-training module based on the obtained training samples formed in the training sample database with expert marking; reports are generated by means of an analytics and report-generating module based on the data obtained at the previous stages of the method, wherein said reports allow controlling the training and/or re-training process.

EFFECT: improved quality of training processing algorithms for aerial terrain photographs.

3 cl, 2 dwg

Description

Technology area

The invention relates to computer systems based on biological models, and in particular to methods of training and / or additional training of convolutional and / or full-convolutional neural networks designed to detect, localize and classify objects to the type of aviation and land equipment in aerospace images of the terrain.

State of the art

Known methods of training convolutional and full-convolutional neural networks using deep learning libraries using the capabilities of highly parallel computing on graphics accelerators (GPGPU-General-purpose computing for graphics processing units, non-specialized computing on graphics processing units [1]). Well-known deep learning libraries such as Caffe [2] and Tensorflow [3] operate under the assumption that fixed-size images are used for training [4]. This simplification allows you to significantly simplify and speed up the learning process.

Training of neural networks in the tasks of detection and localization is carried out on the basis of sets of reference data, on which the objects of interest are marked with high accuracy by an expert. At the same time, to solve the problems of detection and localization, it is sufficient to select the object of interest with a rectangular frame, and to solve the classification problem, it is necessary to highlight the shape of the object in more detail. There is a way to determine the type of aircraft object on the selected images of objects of interest, based on segmentation of objects with subsequent detection of key points of objects and comparison with templates [5]. However, samples of aerospace images of the terrain on which it is necessary to carry out detection, localization and classification to the type of aviation and land equipment may have different sizes, including the need for processing large-format images.

The marking of training samples consists in image segmentation - the separation of objects of interest from the background. Known approaches to performing segmentation of aerospace images differ in the quality of segmentation and the required labor costs. For example, the use of the GrabCut algorithm [6] does not allow fully automatic segmentation and is time consuming.

Known methods for preparing training samples using augmentation based on the formation of new images by inserting objects of interest on various background images [7]. These methods make it possible to form training samples for images with localized objects of interest. On large-format images, the number of background elements significantly exceeds the number of elements of objects of interest, which greatly complicates the learning process.

Well-known frameworks that allow you to markup images for training a neural network (Amazon SageMaker, https://handl.ai/ Handl, Google Data Labeling https://cloud.google.com/data-labeling/docs/). The disadvantage of the known solutions is that they do not support training and additional training.

Disclosure of the essence

The problem to be solved by the claimed invention is training and / or additional training of convolutional and / or full-convolutional neural networks designed to detect, localize and classify objects to the type of aviation and land equipment on aerospace images of the terrain. The technical result is the expansion in comparison with analogues (for example, GrabCut, SegNet [10]) of functionality and improvement of the quality of training and / or additional training of algorithms for processing aerospace images of terrain in order to detect, localize and classify to the type of aviation and land equipment.

FIG. 1 and FIG. 2 shows diagrams of two variants of a software and hardware complex intended for training and / or additional training of algorithms for processing aerospace images of terrain in order to detect, localize and classify to the type of aviation and land equipment.

FIG. 1 the designations are given:

1 - database of training and test samples with expert markup;

2 - additional training module;

3 - import block;

4 - neural network core;

5 - catalog of neural network models;

6 - block of neural network markup;

7 - block for correcting neural network markup;

8 - expert;

9 - module for analytics and reporting.

The proposed hardware and software complex consists of a database of training and test samples with expert markup 1, an import block 3, a neural network core 4, a retraining module 2, a catalog of neural network models 5, a neural network markup block 6, a neural network markup correction unit 7, an analytics and formation module reports 9, while the output of the import block 3 is connected to the first input of the neural network core 4, the output of the database of training and test samples with expert markup 1 is connected to the input of the additional training module 2, the first output of the additional training module 2 is connected to the input of the analytics and reporting module 9, the output of the analytics and reporting module 9 is designed to control the learning outcomes or additional training by an expert 8, the second output of the additional training module 2 is connected to the input of the catalog of neural network models 5, the output of the catalog of neural network models 5 is connected to the second input of the neural network core 4, the output of the neural network core 4 is connected to the input block of neural network markup 6, output block and the neural network markup 6 is connected to the first input of the neural network markup correction unit 7, the second input of the neural network markup correction unit 7 is intended for the expert 8 to correct the training data, the output of the neural network markup correction unit 7 is connected to the input of the training and test samples database with expert markup 1, and the neural network markup unit 6 is configured to automatically mark training data on large-format aerospace images using neural networks, the neural network markup correction unit 7 is configured to correct the markup by an expert 8, the database of training and test samples with expert markup 1 is made with the possibility of continuous replenishment and expansion of the training sampling of the additional training module 2 with the possibility of balancing the number of objects of interest and the number of background elements, which consists in the fact that the number of objects of interest and / or the number of background elements is added to the training set with the aim of achieving comparable (balanced) numbers of objects of interest and background elements, the additional training module 2 is made with the possibility of expanding the area of application of neural network models to new classes of objects and background target environment, which accelerates the marking process.

In accordance with another embodiment, an automatic augmentation module 10 has been added to the software and hardware complex (see Fig. 2) and it is made with the possibility of computer synthesis of 3D models of objects of interest and placing them in automatic mode on the images of the underlying surface, taking into account the permitted areas, with the ability to automatically generate images with different conditions of the atmosphere, the angle of illumination by the light source of the shooting angle.

The method of forming a training sample on large-format aerospace images using neural networks with the possibility of adjusting the markup by an expert and the composition of the proposed software and hardware complex ensures the following tasks:

• import and continuous accumulation of training data for training and (or) additional training of algorithms that provide detection, localization and classification to the type of objects of aviation and land equipment on aerospace images in the visible range of lengths;

• the ability to add new classes of recognition objects;

• improvement of algorithms, deciphering aerospace information;

• expanding the training sample, taking into account the balancing of the number of objects of interest and the number of background elements.

An increase in the volume of the training sample is carried out by artificially expanding the training sample manually or automatically using augmentation of object images using distortions in the form of adding noise, shifts, random rotations in the plane and "out of the plane" within 360 °, compression and stretching along the abscissa axes and ordinate with a coefficient of 0.1; scaling by a factor of 0.1; color changes; contrast; brightness, as well as by inserting objects of interest on various background images.

In accordance with another embodiment of the invention, an increase in the volume of the training sample is carried out using computer synthesis of images of objects of interest. Preliminarily, on real background images, the area is marked in which the object of interest can be placed. After that, in automatic mode, a 3D model of the object of interest is placed on the underlying surface, taking into account the allowed areas. Provides the ability to automatically generate images with different conditions of the atmosphere, angle of the light source, shooting angle.

As a result, the formation of training samples, taking into account the balancing of the number of objects of interest and the number of background elements, leads to the fact that the probability of getting background elements and objects of interest in the sample image will be the same.

In addition, the expert has the ability to manually indicate the value or relevance of each object for training, which makes it possible to reduce the likelihood of adding grouped objects to the training set, taking into account the fact that for training detection and localization, a greater effect can be achieved by increasing the number of images in the training set separately. - standing objects of interest or objects of interest located in unusual places, and reducing images containing groups of nearby objects of interest.

In the algorithm for creating a training sample with closely spaced objects of interest in the automatic mode, the neural network markup correction unit 7 selects an object of interest depending on the presence of other objects nearby, in order to exclude an equiprobable selection of objects and groups of closely spaced objects of interest to reduce the likelihood of adding groups of closely spaced objects to the training sample. interest.

To train the algorithms for processing aerospace images of the terrain for the detection task, reference heat maps are created, which are color maps of the probabilities of a pixel belonging to an object of interest. The color of each element of the heat map corresponds to the degree of confidence of the algorithm that an object of interest is located in the region of the input image corresponding to this element. The more the color shift on the heatmap towards red on a scale from blue to red, the greater the degree of confidence. In the process of training, the heat map obtained at the output of the algorithm for processing aerospace images of the terrain is compared with the reference heat map. As a result, the loss function is calculated, as well as the gradient of this function over the weights of the neurons of the neural network in accordance with the principle of back propagation of the error [8]. On large-format images, the number of background elements significantly exceeds the number of objects of interest. Successful training in this case is possible provided that the background elements are counted less in comparison with the elements belonging to the object. To solve the problem, a coefficient is introduced that reduces the value of the loss function, which is used to determine the discrepancy between the true data and the received responses of the neural network for background elements. The introduction of a decreasing coefficient for balancing objects leads to the appearance of wide "spots" of a high probability value around the centers of objects at the output of the neural network. The occurrence of "spots" with a high probability value makes it difficult to determine the exact position of the center of the object. To reduce the size of these "spots" and improve the localization of the object, a gradual increase in the reduction factor is provided. The value of the decreasing coefficient around the object is determined by a smooth, monotonically decreasing (nuclear [9]) function of the distance to the center of the object of interest. Due to the small number of objects, such a change in the reduction factor has practically no effect on the background balancing, but it significantly reduces the areas of high probability values and improves the localization of objects.

Implementation of the invention

The essence of the invention is illustrated by a diagram of a software and hardware complex intended for training and / or additional training of algorithms for processing aerospace images of terrain in order to detect, localize and classify to the type of aviation and land equipment shown in Fig. 1 and FIG. 2. The original large-format aerospace images presented in the format of color or grayscale digital images enter the import block 3 and are transferred to the neural network core 4 with the possibility of choosing a neural network model from the catalog of neural network models 5, in the neural network markup block 6 the initial large-format aerospace images are marked using the selected neural network, the neural network markup correction unit 7 provides the expert 8 to correct the neural network markup, if necessary, the automatic augmentation module 10 (see Fig. 2) can automatically expand the volume of images for training samples. The corrected marked images are fed to the database of training and test samples with expert markup 1, which ensures its continuous replenishment. The training samples formed in the training sample database with expert markup 1 are sent to the additional training module 2, which provides additional training of neural network models from the catalog of neural network models 5. In the analytics and reporting module 9, reports are generated that allow expert 8 to control the learning process and / or additional training ...

The technical result is provided by the use of import and continuous accumulation of training data with automatic marking on large-format aerospace images using neural networks with the possibility of adjusting the marking by an expert, the possibility of increasing the volume of the training sample manually and automatically due to various augmentation of images of objects of interest, due to computer synthesis of realistic 3D models of objects of interest and placing them in automatic mode on the images of the underlying surface, taking into account the allowed areas, the ability to automatically generate images with different conditions of the atmosphere, the angle of the light source, the shooting angle, using the balancing of training samples by the number of objects of interest and the number of background elements.

Literature

1. Fung J., Tang F., Mann S. Mediated Reality using Computer Graphics Hardware for Computer Vision // Proceedings of the International Symposium on Wearable Computing 2002 (ISWC2002). Seattle, Washington, USA. P. 83-89.

2. Caffe [Electronic resource]. URL: http://caffe.berkeleyvision.org/ (accessed: 10/20/2018).

3. Tensorflow [Electronic resource]. URL: https://www.tensorflow.org/ (accessed: 10/20/2018).

4. Crawford C. An Introduction to Deep Learning [Electronic resource]. URL: https://blog.algorithmia.com/introduction-to-deep-learning/ (accessed: 13.07.2018).

5. Zuo, J., Xu, G., Fu, K., Sun, X., Sun H. Aircraft Type Recognition Based on Segmentation With Deep Convolutional Neural Networks // IEEE Geosci. Remote Sens. Lett. 2018. Vol. 15, no. 2. P. 282-286.

6. Grabcut [Electronic resource]. URL: https://chocopoule.github.io/grabcutweb/ (accessed: 22.02.2019).

7. Dvornik, N., Mairal, J., Schmid C. On the Importance of Visual Context for Data Augmentation in Scene Understanding // ArXiv Prepr. ArXiv1809.02492. 2018.

8. Goodfellow I, Bengio Y, Courville A. Deep learning. MIT press; 2016 Nov 10

9. Rosenblatt M. Remarks on Some Nonparametric Estimates of a Density Function // The Annals of Mathematical Statistics. - 1956. - T. 27, no. 3.- doi: 10.1214 / aoms / 1177728190.

10. Badrinarayanan Vijay, Kendall Alex, Cipolla Roberto. Segnet: A deep convolutional encoder-decoder architecture for image segmentation // IEEE transactions on pattern analysis and machine intelligence. - 2017. - Vol. 39, no. 12. - P. 2481-2495.

Claims (12)

1. A method of forming a training sample intended for training and / or additional training of algorithms for processing aerospace images of terrain in order to detect, localize and classify objects to the type of aviation and land equipment, characterized in that in the claimed method: large-format aerospace images, presented in the format of color or grayscale digital images, are imported through the import unit; receive imported data from the data import unit by means of the neural network core and select neural network models from the catalog of neural network models; carry out the marking of the original large-format aerospace images by means of a neural network marking unit using the selected neural network model; correcting the obtained neural network markup by means of the neural network markup correction unit; perform automatic expansion of the volume of images for training samples by means of the automatic augmentation module; transfer the corrected tagged images to the database of training and test samples with expert markup, providing continuous replenishment of the database, while balancing the training samples according to the number of objects of interest and the number of background elements by means of a database of training and test samples with expert markup; performing additional training of neural network models from the catalog of neural network models by means of the additional training module based on the received training samples formed in the training sample database with expert markup; using the analytics and reporting module, based on the data obtained at the previous stages of the method, reports are generated that allow you to control the learning and / or retraining process. 2. A software and hardware complex designed to implement the method according to claim 1, consisting of a database of training and test samples with expert markup, an import block, a neural network core, characterized by the fact that the complex contains an additional training module, a catalog of neural network models, a neural network markup block , neural network markup correction unit, automatic augmentation module, analytics and reporting module, while the output of the import block is connected to the first input of the neural network core, the output of the training sample database with expert markup is connected to the input of the additional training module, the first output of the additional training module is connected to the input the analytics and reporting module, the output of the analytics and reporting module is designed to control the learning outcomes or additional training by an expert, the second output of the additional training module is connected to the input of the neural network models catalog, the output of the neural network models catalog is connected to the second input of the neural network core, the output of the neural network the core is connected to the input of the neural network markup block, the output of the neural network markup block is connected to the first input of the neural network markup correction block, the second input of the neural network markup correction block is designed to correct the training data by an expert, the output of the neural network markup correction block is connected to the input of the automatic augmentation module, the output of the automatic augmentation module connected to the input of the training sample database with expert markup, and the neural network markup unit is configured to automatically mark the training data on large-format aerospace images using neural networks, the neural network markup correction unit is configured to correct the markup by an expert, the training and test samples database with expert markup made with the possibility of continuous replenishment and expansion of the training sample, with the possibility of balancing training samples by the number of objects of interest and the number of background elements, m The additional training module is made with the possibility of expanding the area of application of neural network models to new classes of objects and background target environment and provides acceleration of the marking process. 3. The hardware and software complex according to claim 2, characterized in that the automatic augmentation module is made with the possibility of computer synthesis of realistic 3D models of objects of interest and their placement in automatic mode on the images of the underlying surface, taking into account the permitted areas, with the possibility of automatically generating images with different the state of the atmosphere, the angle of the light source, the shooting angle.

Images