RU2771442C9 - Method for processing images by convolutional neural networks - Google Patents

Abstract

FIELD: image data processing.

SUBSTANCE: invention relates to the field of image recognition, namely to technology for detecting and classifying objects in images. The method includes the stages of obtaining sets of full-scale images of N>2 objects and three-dimensional models of objects, creating N initial sets of images of objects using a computing system, wherein each image from the initial set of images is the result of rendering a three-dimensional model of the object, creating an augmented set of images of each object by introducing at least one change to each image from the initial set of images and the set of full-scale images of the corresponding object, creating training data samples containing multiple images selected from the augmented sets of images of objects, training a convolutional neural network using the corresponding training data samples, and applying the trained convolutional neural network to the examined image in order to detect and classify the objects depicted therein.

EFFECT: increase in the probability of detecting and classifying objects in images of various types, including images of the visible range, UHF range, and IR range.

19 cl, 14 dwg

Description

The claimed technical solution relates to the field of image recognition, namely, to a technique for detecting and classifying objects in images using 3D modeling and a convolutional neural network.

The claimed invention can be used to increase the probability of detecting and classifying objects in various types of images, including images of the visible range, microwave range and infrared range.

The use of a convolutional neural network trained on a large number of images, including many options for shooting conditions for objects, the variability of the objects themselves, and interference affecting the shooting process, can significantly increase the ability to detect and classify the desired objects in the images under study. Recognizable objects may include various objects of both artificial and natural origin, such as various vehicles and technical means, buildings and building structures, infrastructure elements, space objects, terrain elements, etc.

Modern means of obtaining images make it possible to form images of objects in various ranges of electromagnetic waves, including images of the visible range, microwave range and infrared range. These types of images differ significantly in their characteristics. For example, radar images (RLI) obtained in the microwave range have a number of features that significantly distinguish them from images of the visible range, as described in [1]. The specificity of the scattering of electromagnetic waves in the microwave range leads to the fact that images of objects with a higher electrical conductivity, mainly objects of artificial origin, have greater brightness on radar images. In this case, the images (images) of objects are a set of local scattering centers, which are formed as a result of a single or multiple re-reflection of the probing signal from the elements of the object. The location and brightness of such centers is determined by the viewing angle of the object, which leads to a high variability of the images observed on radar images. Also, images in the IR range have their own characteristics.

The conditions for obtaining images can be very diverse, such as filming from the board of mobile technical means, including manned and unmanned aerial vehicles, ground and sea manned and robotic systems, etc. In this case, the formed images are affected by various conditions, such as distortions caused by the movement of the carrier of the recording means or the subject itself, changes in illumination, shooting distance, exposure to noise interference of various physical origins, etc. These factors lead to the fact that in order to train a convolutional neural network capable of confidently detecting and classifying the desired objects in the studied images, it is necessary to use a large number of images of each object obtained at different times of the day, year, from different imaging tools, from different angles, made with different resolutions and taking into account all kinds of influences. However, this is difficult due to the high cost of obtaining full-scale images of objects in various conditions, legally and ethically difficult for medical images, or physically impossible, for example, for many space objects due to their low availability for observation, etc.

These conditions require the use of new technical solutions, such as the creation of electronic three-dimensional models of objects using computing tools and the formation of a set of synthetic photorealistic images of these objects from them, where synthetic photorealistic images are indistinguishable from natural images used by a convolutional neural network in training in the interests of detection and classification these objects.

A known method of indexing and searching for digital images according to the patent of the Russian Federation No. 2510933, described in [2]. The method relates to systems for indexing and searching for digital images contained in files of various graphic formats. The technical result consists in reducing the time of automatic and semi-automatic indexing of images and in increasing the speed of the procedure for searching for images containing similar predominant colors in the database. A color is found that has the smallest Manhattan distance from the found dominant color in the selected color coordinate system from a predetermined set of primary colors classified by brightness, saturation, and hue. The resulting color is used as an identifier to organize the search procedure in the image database. Color associations are determined from a basic set of primary colors to form characteristics of visual similarity of colors and visual contrast of colors. An image index is formed according to the presented primary color in RGB format and/or the name and/or color tag. Searching for images in the information database. As a result of the search, a list of images is formed with indices matching the presented index and/or indices, in which primary colors are present in the visual similarity association list or in the color contrast list for the primary color in the image search index.

The disadvantage of this analog is the limitation of the method by detecting and classifying objects only by the criterion of the color of the object and only on images of the visible range.

There is also a method of computationally efficient multi-class image recognition using sequential analysis of neural network features according to RF patent No. 2706960, described in [3]. This method consists in the fact that by means of a convolutional neural network a vector of deep features of the input image is obtained, a principal component transformation is applied to this vector to obtain a sequence of principal components of the input image, the sequence of principal components is divided into a predetermined number of adjacent parts, each of which belongs to to a different level of granularity, the part of the sequence related to this level of granularity is attached to the initially empty subsequence of the main components of the input image, the distance between the subsequence and the corresponding subsequences of the main components of the templates from the set of candidate solutions is calculated, the ratios of the minimum distance to other distances are estimated, etalons with ratios less than a threshold are excluded from the set of candidate solutions, and if the set of candidate solutions is included in the etalons of only one class, the input one image as belonging to this class.

The disadvantage of this analog is the instability of image recognition when exposed to the studied images of various distorting influences in the formation of images.

The closest in its technical essence to the claimed method of image processing by a convolutional neural network is the method of image processing by trained convolutional neural networks according to RF patent No. 2709661, described in [4]. The method - a prototype of image processing by a convolutional neural network includes the creation by a computing system of an initial set of images, where each image from the original set of images contains a rendering of a text image, the creation of an augmented set of images by processing the initial set of images to add at least one image to each image from the original set of images. of a simulated defect in the emulation of the image capture process, creating a training data set containing a plurality of image pairs, in which each image pair contains a first image selected from the original image set and a second image selected from the augmented image set, and training using the training data set , one or more convolutional neural networks for image processing by activating a convolutional neural network for each image from the training data sample, preprocessing each specified image ing through the respective convolutional neural network layers, reducing each specified image by a given scaling factor, enlarging the image by a given scaling factor through the respective transposed convolution layers, and applying one or more trained convolutional neural networks to process one or more images.

A feature of the prototype method is that for training a convolutional neural network, a large set of character images is created with various modifications, such as darkening part of the character image, blurring the character image, superimposing interfering noise on the character images, etc., which allows you to train the convolutional neural network. network on a large set of different variants of images of the same text message, and to ensure the detection and recognition of the visible range of text characters in images that are resistant to such modifications using a trained convolutional neural network.

However, in this prototype image processing method, a convolutional neural network performs detection and classification only on visible range images and only alphanumeric inscriptions, and the conditions for obtaining such images are limited to photographing images from aboard ground vehicles, such as a car.

Thus, the disadvantage of the closest analogue (prototype) of the image processing method by a convolutional neural network is the relatively low probability of detecting and classifying objects in images, which are various vehicles and technical means, buildings and building structures, infrastructure elements, terrain elements, etc., when obtaining images of various physical nature, including images of the visible range, microwave range and infrared range, using mobile technical means, including manned and unmanned aerial vehicles and ground and sea manned and robotic systems.

The technical result of the proposed solution is the development of a method for processing images by a convolutional neural network, which provides an increase in the probability of detecting and classifying objects in images of various types, including images of the visible range, microwave range and infrared range.

The specified technical result in the claimed method of image processing by a convolutional neural network is achieved by the fact that in the known method of image processing by a convolutional neural network, including the creation of a computing system of initial sets of images, the creation of an augmented set of images, the creation of training data samples containing a plurality of images, training using the appropriate training sample data of a convolutional neural network, the use of a trained convolutional neural network, additionally, sets of natural images of N> 2 objects and three-dimensional models of objects are preliminarily obtained, moreover, sets of natural images of objects are preliminarily formed in the visible range, or in the microwave range, or in the infrared range .

N initial sets of images of objects are created, where each image from the original set of images is the result of rendering a three-dimensional model of the object, and each image from the original set of images is obtained by changing the viewing angle of the three-dimensional model of this object in azimuth and elevation.

An augmented set of images of each object is created by introducing at least one change into each image from the original set of images and the set of full-scale images of the corresponding object. Making changes to each image from the original set of images and a set of full-scale images of the corresponding object is performed by changing the image scale, by hiding part of the image of this object, by decontrasting all or part of the image of this object, by changing the illumination of all or part of the image of this object, by defocusing the entire or part of the image of this object, by blurring the image of this object, imitating the movement of an object or imaging device, by applying a noise signal to all or part of the image of this object, by changing the reflective properties of all or part of the image of this object, by covering part of the image of this object with an image another object. Also, to make changes to each image in the IR range from the original set of images and the set of full-scale images of the corresponding object, it is performed by changing the radiative properties of all or part of the image of this object.

Training data samples are created containing a plurality of images selected from augmented object image sets.

The training of a convolutional neural network using the corresponding training data set is performed by calculating a set of feature vectors using the convolutional neural network, with each feature vector of the set of feature vectors corresponding to an image from the training data set in the image feature space, calculating the value of the loss function for the training data set , displaying a set of probabilities, where each probability from the set of probabilities characterizes a hypothesis that associates an image from the training data sample with a class associated with this image, while the loss function additionally displays a set of distance values, where each distance value is calculated in the feature space of images between the feature vector, representing an image from the training data sample, and the center of the class associated with this image in accordance with the training data sample, setting one or more parameters convolutionally th neural network based on the value of the loss function.

After training, the convolutional neural network is applied to the image under study to detect and classify the objects depicted on it, and the appropriately trained convolutional neural network detects and classifies objects in the images of the visible range, or the microwave range, or the infrared range.

In the proposed set of actions, sets of full-scale images of N>2 objects and three-dimensional models of these objects are first obtained. The formation of a set of synthetic photorealistic images from the initial set of images by rendering a three-dimensional model of the corresponding object by changing the viewing angle of the three-dimensional model of this object in azimuth and elevation allows you to get a large number of various images of each object, which is important for training a convolutional neural network that can provide a high probability of detection and classification of the depicted object. An important point is also that from a three-dimensional model of an object it is possible to form an arbitrarily large set of synthetic photorealistic images of this object in the visible, microwave and infrared ranges of the electromagnetic spectrum in conditions where obtaining large sets of natural images of objects is difficult or impossible due to the high cost. obtaining full-scale images of objects, their low availability for observation, etc.

In addition, augmented sets of images of each object are created by introducing into each image from the original set of images and the set of full-scale images of the corresponding object a set of changes characteristic of the studied images obtained from the board of mobile technical means, including manned and unmanned aerial vehicles, ground and sea controlled vehicles and robotic systems, taking into account the many distortions that affect such conditions. Training a convolutional neural network using the training data set thus formed allows, using the trained network, to stably detect and classify objects in the further studied images that have a large variability. The described steps for creating a training data sample are performed separately for images of the visible range, microwave range and IR range, and are used for the corresponding training of a convolutional neural network, which ensures that the actions of detecting and classifying objects are adapted to the features of the images of the corresponding range of electromagnetic waves.

Therefore, the specified new set of actions when performing image processing by a convolutional neural network makes it possible to increase the probability of detecting and classifying objects in images of various types, including images of the visible range, microwave range and infrared range.

The claimed method is illustrated by drawings, which show:

- in Fig. 1 - an image processing system with a convolutional neural network;

- in Fig. 2 - algorithm for image processing by a convolutional neural network;

- in Fig. 3 - examples of full-scale and model images of the aircraft on images of the microwave range from different angles;

- in Fig. 4 - examples of full-scale and model images of the aircraft in the IR range, during the day, in winter conditions;

- in Fig. 5 - examples of visualization of a three-dimensional model of the T-90 tank;

- in Fig. 6 is an illustration of the rendering method of throwing rays;

- in Fig. 7 - an example of hiding a part of the image of objects of the type of construction and transport equipment by trees and building structures;

- in Fig. 8 - an example of decontrasting the image of objects such as vehicles and infrastructure elements during aerial photography;

- in Fig. 9 - an example of changing the illumination of a part of the image of an object of the helipad view, top view;

- in Fig. 10 is an example of blurring of an image of a car due to the movement of an object;

- in Fig. 11 - an example of superimposing a noise signal on the image of objects of the form of a parking lot of vehicles and adjacent trees;

- in Fig. 12 - an example of changing the reflective properties of the image of vehicles in the visible light range;

- in Fig. 13 - an example of covering a part of the image of some vehicles with images of other vehicles;

- in Fig. 14 - examples of changes in the radiative properties of a running engine of a passenger car and a conveyor with caterpillars heated due to friction in the IR range images.

The implementation of the claimed method is presented on the example of an image processing system with a convolutional neural network, shown in Fig. 1. Raider block 1 receives pre-created 3D object models. Rendering unit 1, which is a computing system, creates an initial set of images of the corresponding object from each three-dimensional model. In the block for creating an augmented set of images 2 from the original sets of images of objects and previously created sets of full-scale images of the same objects, the images are subjected to various changes in various combinations. As a result, a large augmented set of images of each object is created, from which a smaller training data sample is selected in the training data sample creation block 3, which is used to train the convolutional neural network in the convolutional neural network training block 4. Training can be performed repeatedly, each time changing part of the augmented image sets used as the current version of the training data set. Training ends when, with changes in the training data set, the increase in the probability of detecting and classifying recognizable objects using the trained convolutional neural network stops. Further, in the detection and classification block 5, built on the basis of the trained convolutional neural network, on the image under study arriving at its input, the search and identification of objects whose images are possibly present in the image under study is carried out. The result of the work is the detection and classification by N types of depicted objects or, in their absence, a solution of the form "no objects found". If there are two or more images of objects on the studied image, then their detection and classification are performed independently of each other. The presented image processing system describes the use of a convolutional neural network for object detection and classification in relation to images of the visible range, microwave range, IR range, respectively.

The method implements the following sequence of actions.

The image processing algorithm for a convolutional neural network is shown in Figure 2.

Methods for preliminary obtaining sets of natural images of N>2 objects are known, natural images of objects are obtained, for example, using electronic cameras for visible images, using radar stations for microwave images, using infrared sensors for infrared images. It is advisable for each object to obtain a plurality of natural images that differ in the shooting point, the angle of the object, the image scale, climatic conditions, etc. For example, figure 3 shows full-scale images of the aircraft in the microwave range from different angles (azimuth angle 79 degrees in the upper left part of the figure and 350 degrees in the upper right part of the figure), and figure 4(a) - full-scale image of the aircraft in IR range, during the day, in winter conditions.

Due to the possible high costs of obtaining sets of natural images or the physical complexity of obtaining them, for example, for objects in space or under water, the number of available natural images of some objects is usually limited.

Methods for preliminary obtaining three-dimensional models of objects are known; a number of programs have been developed for their construction, such as 3ds Max, Maya, Cinema 4D, Blender. Three-dimensional models of various objects are created as a result of their 3D modeling. To build a 3D model of an object, its geometric dimensions are measured and its shape is estimated. The resulting 3D models of objects must meet the following requirements: a high degree of realism, visual similarity of the geometric shapes of the created models to the objects themselves, compliance of the overall dimensions of the models with the dimensions of the objects, a sufficient level of detail of the models, high accuracy of the shapes, position and dimensions of individual structural elements. For example, modeling of objects represented by images in the microwave range is described in [5], and modeling of objects represented by images in the visible range is described in [6].

When creating three-dimensional models, the most common way is to build three-dimensional figures from a grid of polygons described by vertices, edges, and faces. A three-dimensional model of an object, consisting of many polygons, is a polygonal mesh. For example, figure 5 shows examples of visualization of a three-dimensional model of the T-90 tank from the computer game "World of Tanks". To ensure photorealistic images, the surfaces of a three-dimensional model of an object are given the appearance of real materials, such as metal, wood, plastic, etc. The surface, if necessary, becomes transparent or mirror. To do this, for example, use the Material Editor feature of the Material Editor view in 3ds Max.

For each object, using a computing system, a corresponding set of images of the object is created, where each image from the original set of images is the result of rendering a three-dimensional model of this object. To do this, each image from the original set of images is obtained by changing the viewing angle of the three-dimensional model of this object in azimuth and elevation. For example, in FIG. 3 shows images of the aircraft formed from a three-dimensional model with different angles (azimuth angle of 79 degrees in the lower left part of the figure and 350 degrees in the lower right part of the figure) in the microwave range, and in Fig. 4(6) - an image of an aircraft in the IR range formed from a three-dimensional model.

When rendering from a virtually observed side of a three-dimensional model of an object, a two-dimensional image of the object is created using a computing system. For example, one of the rendering methods is the ray tracing method [7]. With this method, the generated image is considered as being observed from a certain point. Rays are directed from the observation point to the 3D model of the object, with the help of which the intensity and color of the displayed pixel in the 2D image are determined. The rays stop their propagation when they reach the surface of the 3D model or the background surface being used. An illustration of the ray tracing method for raiding is shown in FIG. 6.

The three-dimensional model of the object existing in the form of an electronic image in the computer system is virtually rotated by the specified azimuth angle and elevation angle and an observable view of the object image is formed. The values of the azimuth angles and elevation angles of the three-dimensional model of the object are selected based on the possible angles of physical observation of this object, such as front, rear, side, horizontal to the earth's surface or at an angle, including a top view. The step of changing the azimuth and elevation angles is chosen no more than a few degrees to increase the number of images in the set for each object, which will increase the probability of detecting and recognizing these objects. For example, in FIG. 3 (lower part of the figure) shows images of the aircraft in the microwave range, when observing its three-dimensional model at various angles in azimuth.

From the created initial set of images and a set of full-scale images of each object, an augmented set of images of this object is created by making at least one change to each image of these sets. These changes are made to electronic images by known electronic editing techniques. When making such changes to the original set of images and the set of natural images, possible shooting conditions and the effect of various distortions obtained in real conditions of observing objects, taking into account the impact of various physical, climatic and other factors, are taken into account.

Images can be modified, for example, by rescaling the image. To do this, for the studied images, on which it is planned to perform the operations of detection and classification of objects, the possible variants of the image scale are determined. Based on the obtained options for the scale of the images under study, for each image from the augmented set of images of the object, its scale is changed so that the scale obtained in at least one modified image does not differ by more than 20 ... 30% from one of the scale options for the images under study. The rescaling of an electronic image may be implemented by a corresponding function of an electronic image editor such as Adobe Photoshop.

Images can be changed, for example, by hiding part of the image of the object. Such distortion can be caused by various reasons, including the impossibility of observing any parts of the object, hiding part of the object by the surrounding landscape, and so on. Methods for hiding a part of an image of an object consist in selecting the corresponding parts on the electronic image and replacing them, for example, with a background image. Selecting parts of the electronic image and replacing them with a background image can be implemented by the corresponding function of the electronic image editor, such as Adobe Photoshop. For example, in FIG. 7 shows an example of hiding part of the images of objects of the type of construction and transport equipment by trees and building structures.

Images can be changed, for example, by decontrasting all or part of the image of this object. When decontrasting all or part of an image of an object, the contrast of this image is reduced. The decontrasting of all or part of the image of an object in practice is caused by haze and precipitation along the electromagnetic signal propagation path. In the proposed method, the computer system simulates the decontrasting of the entire or part of the image of an object by performing an amplitude transformation in the corresponding part of the image, which leads to a narrowing of the dynamic range of brightness corresponding to the physical conditions of shooting and the features of imaging devices. Filtering all or part of an image may be implemented by an appropriate function of an electronic image editor such as Adobe Photoshop. For example, in FIG. 8 shows an example of decontrasting the image of objects such as vehicles and infrastructure elements during aerial photography.

Images can also be changed, for example, by changing the illumination of all or part of the image of the object. Methods for changing the illumination of the entire or part of the image consist in the formation in the computer system of a model of a light source, for example, simulating a solar luminary, determining its intensity and location relative to the image plane of the object and, in accordance with the angles of incidence of the simulated sunlight on the image surface of the object, changing the degree of illumination of these surfaces. The brightness value of the corresponding pixels of the electronic image being edited is changed proportionally to the change in the illuminance using a function of an electronic image editor such as Adobe Photoshop. For example, in FIG. Figure 9 shows an example of changing the illumination of a part of the image of an object of the heliport view, top view.

The images can also be changed, for example, by blurring the image of that object due to movement of the object or imaging device. Objects detected in images may move at some speed and/or imaging devices may be mounted on various vehicles, including aircraft, which in practice causes the image of the object to shift during exposure time. In the proposed method, the computer system imitates the blurring of the image of this object, due to the movement of the object or imaging device, by filtering the image, for example, with a one-dimensional Gaussian filter in the direction of the supposed mutual change in the location of the object or imaging device, and with filtering parameters determined by possible speed values their mutual movement. An appropriate function of an electronic image editor such as Adobe Photoshop can be used to blur the image. For example, in FIG. 10 shows an example of blurring an image of a car due to the movement of an object.

The images can also be modified, for example by applying a noise signal to all or part of the image of that object. The noise signal in the images of the object can be caused, for example, by the noise of the sensor of the acquisition tool, contamination and non-ideality of the optical system of the lenses of the acquisition tool, atmospheric conditions during shooting, lossy compression when recording images, noise in the image transmission channel, etc. In the proposed method, the computer system superimposes a noise signal on all or part of an object image by generating a noise signal matrix with the appropriate intensity and noise distribution and summing the noise signal matrix with the image pixel brightness matrix. Such a summation (mixing) of the two matrices can be realized by an appropriate function of an electronic image editor such as Adobe Photoshop. For example, in FIG. Figure 11 shows an example of overlaying a noise signal on an image of objects of the form of a parking lot of vehicles and adjacent trees.

Images can also be changed, for example, by defocusing all or part of the image of the object. In the proposed method, the computer system simulates defocusing of the entire or part of the object image by performing the operation of interpolating the brightness values of neighboring pixels of the corresponding part of the object image. In this case, the true value of the brightness of the current pixel is replaced by the arithmetic mean value of the brightness of neighboring pixels. The degree of defocusing can be controlled by the number of iterations of the interpolation operation. To defocus all or part of an image of an object, the function of interpolating the brightness values of neighboring pixels in an electronic image editor such as Adobe Photoshop can be used.

Images can also be modified, for example by changing the reflective properties of all or part of the object's image. The quality of the resulting images, for example, radar images, is significantly affected by the reflective properties of this object, which are determined by the characteristics of the structural materials of the object and its components. Three-dimensional models of an object created in the form of an electronic image in a computer system suggest the possibility of changing the reflective properties of the entire or part of the image of the object. For images in the visible range, the reflective properties vary depending on the surface material of the object, the degree of surface smoothness, the characteristics of the paint coating, etc. For images in the microwave range, the reflective properties vary depending on the type of object material: metal, wood, plastic, stone, etc. By changing the type of material, the degree of surface smoothness and characteristics of the coating with known values of the reflection coefficient in the constituent parts of the three-dimensional model of the object, it is possible to change the reflective properties of the entire or part of the image of the object. For example, in FIG. 12 shows an example of changing the reflective properties of an image of vehicles in the visible light range.

Images can also be changed, for example, by covering part of the image of an object with an image of another object. In practice, a part of an image of an object is often covered by an image of another object, for example, an image of an observed building can be partially covered by trees, an image of an aircraft or a space body by clouds, etc. Methods for covering a part of an image of an object with an image of another object consist in highlighting some supposedly unobservable parts on the electronic image of the object and replacing them with an image of other objects that are possible in accordance with the shooting conditions. Selecting parts on the electronic image and replacing them with the image of other objects can be implemented by the corresponding function of the electronic image editor, such as Adobe Photoshop. For example, in FIG. 13 shows an example of covering a part of the image of some vehicles with images of other vehicles.

IR images can also be modified, for example by changing the radiative properties of all or part of that object. The radiative properties of objects are due to a number of factors, such as the illumination (heating) of objects by the sun, internal heat sources of objects, for example, due to heating due to the operation of the engine or due to friction in the axes of mechanisms, as well as the characteristics of radiation by the object and the surrounding background, as described, for example , at 8]. In the rendered images of the IR range, when the radiative properties of the entire or part of the object change, the brightness of the representation pixels of the corresponding parts of the object changes proportionally. For example, in FIG. 14 shows examples of changes in the radiative properties of a running engine of a passenger car and a conveyor with caterpillars heated due to friction in the IR range images.

The listed changes to images can be performed in any order with the choice of an arbitrary set of changes on different images from the original set of images and a set of natural images of each object. As a result, an augmented set of images is created for each object, consisting of many hundreds - thousands of images, reflecting various shooting conditions of this object separately for images of the visible range, microwave range and IR range.

The creation of training data samples containing a plurality of images selected from augmented object image sets is performed as follows. Separately for images of the visible range, microwave range and IR range, a part of the images is randomly selected from the augmented sets of images of objects, for example, 60-90% of the augmented sets of images, as recommended in [9]. In this case, the selected number of images of each object can be either equal for all objects or proportional to the frequency of occurrence of such an object in practice. It is also possible to create several training data samples for images from groups of objects, for example, to divide the entire set of objects into groups into categories such as aircraft, ground vehicles, buildings and building structures, etc. This will allow further, knowing the shooting conditions of the desired objects, to increase the probability of detecting and recognizing objects within a group of objects.

To detect and classify objects, a convolutional neural network is used, in which only a limited small-sized weight matrix is used for the convolution operation, which is shifted over the entire processed layer, forming after each shift an activation signal for the neuron of the next layer with a similar position. That is, for different neurons of the output layer, the same weight matrix is used, which is called the convolution kernel. It is interpreted as a graphic encoding of some feature, for example, the presence of an oblique line at a certain angle. Then the next layer, resulting from the convolution operation by such a weight matrix, shows the presence of this feature in the processed layer and its coordinates, forming the so-called feature map. Accordingly, a convolutional neural network uses a number of sets of weights that encode image elements (eg, lines and arcs at different angles). At the same time, such convolution kernels are not laid down by the researcher in advance, but are formed at the stage of network training in various ways, described, for example, in [9].

The training of a convolutional neural network using the training data sample is performed, for example, as follows. A convolutional neural network is trained separately to detect and classify objects in visible, microwave, and infrared images. To train the network, one of the created training data samples is used. For example, as proposed in [10], for each image from the training data sample, using a convolutional neural network, a feature vector is calculated in the image feature space, for the training data sample, the value of the loss function is calculated, which displays a set of probabilities, where each probability from the set of probabilities characterizes the hypothesis A that associates an image from the training data set with the class associated with this image. At the same time, the loss function additionally displays a set of distance values, where each distance value is calculated in the image feature space between the feature vector representing the image from the training data sample and the center of the class associated with this image in accordance with the training data sample. Based on the calculated value of the loss function, one or more parameters of the convolutional neural network are tuned. This sequence of actions is iteratively performed many times in order to minimize the value of the loss function. If at the next iterations there is no further decrease in the value of the loss function, then for a given training data set, the training of a convolutional neural network to detect and classify objects in images of the corresponding range is considered completed. Checking the achievement of the maximum achievable probability of detecting and classifying objects for the trained convolutional neural network is performed on the remainder of the set of images selected from the augmented sets of object images, from which the training data sample used for network training is excluded. Having memorized the obtained values of the probability of detection and classification of objects, in the manner described above, a new training data sample is created from the augmented sets of images of objects, the convolutional neural network is retrained on it, for the newly trained convolutional neural network, the values of the probability of detection and classification of objects are obtained on the corresponding remainder of the set of images , selected from the augmented sets of images of objects, from which the training data set used for training is excluded. If the newly obtained values of the probability of detection and classification of objects do not exceed the previously obtained values, then the training of the convolutional neural network using this training data sample is completed.

Also, to increase the probability of detecting and classifying objects in the images under study, convolutional neural networks can be trained on the corresponding training data samples for images from groups of objects, for example, divided into groups of objects by categories such as aircraft, ground vehicles, buildings and building structures, etc. .P.

The application of a trained convolutional neural network to the image under study to detect and classify the objects depicted on it is performed as follows. To process the studied image obtained in the corresponding range of electromagnetic waves, a convolutional neural network trained for this range is used. The result of the detection and classification of the object depicted on it can be solutions of the form "objects detected and classified" with the localization of their place in the image under study or "the desired objects were not found in this image", as shown in Fig. 1. There can be several desired objects of the same or different types on the studied image. To detect them in the image, a search for objects of various types is performed in parallel.

Also, to increase the probability of detecting and classifying objects in the studied images obtained in known spatial environments, such as outer space, underwater space, etc., a convolutional neural network trained on the corresponding training data samples for images from groups of possible observed objects can be used. .

Verification of the theoretical prerequisites of the claimed method of image processing by a convolutional neural network was carried out by developing and testing an experimental sample that implements the proposed method.

For dozens of types of objects, which are vehicles and construction equipment, the corresponding sets of full-scale images of the visible range were obtained and three-dimensional models were developed for which augmented sets of images of objects with a volume of at least 300 for each object were created, and a convolutional neural network was trained with the following detection characteristics and classification of objects on the studied images:

probability of detecting objects 96.1%:

the probability of missing objects is 3.9%;

the probability of correct classification of objects is 98.5%.

The conducted studies confirm that when using the proposed method, an increase in the probability of detecting and classifying objects in images of various types, including images of the visible range, microwave range and infrared range, is provided.

Literature

[1] Radar characteristics of objects. Research methods / Ed. CM. Nestorov. - M.: Ed. Radio engineering. 2015. - 312 p.

[2] A method for indexing and searching digital images. RF patent No. 2510933 dated 09/23/2011, IPC G06F 17/30 (2006.01).

[3] A method for computationally efficient multiclass image recognition using sequential analysis of neural network features. Patent of the Russian Federation No. 2706960 dated 01/25/2019, IPC G06K 9/62 (2006.01).

[4] Method of image processing by trained convolutional neural networks. RF patent No. 2709661 dated September 19, 2018, IPC G06N 3/08 (2006.01).

[5] Afinogenov A.Yu. Mathematical modeling of radar portraits of distributed objects of complex shape and some of its applications: diss…cand. physical - Mathematical sciences. - M.: 1996. - 275 p.

[6] Garbul A.A., Zhdanov D.D., Potemkin I.S., Sokolov V.G. Computer modeling of images of complex three-dimensional scenes formed by models of real optical systems: // Journal of Scientific Visualization. MEPhI. 2013, volume 5, number 4. - S. 88-117.

[7] Ulyanov A.Yu., Kotyuzhansky L.A., Ryzhkova N.G. Ray tracing method as the main technology of photorealistic rendering: //Journal of Basic Research. UFU. 2015, No. 11. - S. 1124-1128.

[8] Mikheev S.V. Fundamentals of infrared technology. - St. Petersburg: Ed. Saint Petersburg State University of Information Technologies, Mechanics and Optics. 2017. - 127 p.

[9] Deep learning. Yann LeCun, Yoshua Bengio, Geoffrey Hinton. Nature 521, 2015.

[10] A method for training a neural network. RF patent No. 2707147 dated October 31, 2018, IPC G06N 3/08 (2006.01).

Claims (19)

1. A method for processing images by a convolutional neural network, including the creation by a computing system of initial sets of images, the creation of an augmented set of images, the creation of training data samples containing a plurality of images, training using the corresponding training data sample of a convolutional neural network, the use of a trained convolutional neural network, characterized in that , which previously receive sets of natural images of N> 2 objects and three-dimensional models of objects, create N initial sets of images of objects, where each image from the original set of images is the result of rendering a three-dimensional model of the object, create an augmented set of images of each object by introducing into each image from the original a set of images and a set of natural images of the corresponding object of at least one change, create training data samples containing a plurality of images selected from augmented image sets objects, after training the convolutional neural network, it is applied to the image under study to detect and classify the objects depicted on it. 2. The method according to p. 1, characterized in that the convolutional neural network detects and classifies objects in the images of the visible range. 3. The method according to claim. 1, characterized in that the convolutional neural network detects and classifies objects in the images of the microwave range. 4. The method according to claim 1, characterized in that the convolutional neural network detects and classifies objects in IR images. 5. The method according to p. 1, characterized in that the sets of full-scale images of objects are pre-formed in the visible range. 6. The method according to p. 1, characterized in that the sets of full-scale images of objects are pre-formed in the microwave range. 7. The method according to p. 1, characterized in that the sets of full-scale images of objects are pre-formed in the IR range. 8. The method according to claim 1, characterized in that when creating N initial sets of images of objects, each image from the initial set of images is obtained by changing the viewing angle of a three-dimensional model of this object in azimuth and elevation. 9. The method according to claim 1, characterized in that changes are made to each image from the original set of images and the set of full-scale images of the corresponding object by changing the scale of this image. 10. The method according to claim. 1, characterized in that the modification of each image from the original set of images and a set of full-scale images of the corresponding object is performed by hiding part of the image of this object. 11. The method according to claim 1, characterized in that the modification of each image from the original set of images and the set of full-scale images of the corresponding object is performed by decontrasting all or part of the image of this object. 12. The method according to claim 1, characterized in that changes are made to each image from the original set of images and the set of full-scale images of the corresponding object by changing the illumination of all or part of the image of this object. 13. The method according to claim 1, characterized in that changes are made to each image from the original set of images and the set of full-scale images of the corresponding object by defocusing all or part of the image of this object. 14. The method according to claim. 1, characterized in that the introduction of changes in each image from the original set of images and a set of full-scale images of the corresponding object is performed by blurring the image of this object, simulating the movement of an object or imaging device. 15. The method according to claim 1, characterized in that the modification of each image from the original set of images and the set of full-scale images of the corresponding object is performed by applying a noise signal to all or part of the image of this object. 16. The method according to claim 1, characterized in that changes are made to each image from the original set of images and the set of full-scale images of the corresponding object by changing the reflective properties of all or part of the image of this object. 17. The method according to claim 1, characterized in that changes are made to each image from the original set of images and the set of full-scale images of the corresponding object by covering a part of the image of this object with an image of another object. 18. The method according to claim 1, characterized in that changes are made to each IR image from the original set of images and the set of full-scale images of the corresponding object by changing the radiative properties of all or part of the image of this object. 19. The method according to claim 1, characterized in that the training of the convolutional neural network using the corresponding training data sample is performed by calculating a set of feature vectors using the convolutional neural network, with each feature vector from the set of feature vectors corresponding to an image from the training data set in in the image feature space, calculating for the training data sample the value of the loss function that displays the set of probabilities, where each probability from the set of probabilities characterizes the hypothesis that associates the image from the training data sample with the class associated with this image, while the loss function additionally displays the set of distance values, where each distance value is computed in the feature space of the images between the feature vector representing the image from the training data sample and the center of the class associated with this image in accordance with the training data sample, set derivation of one or more parameters of the convolutional neural network based on the value of the loss function.

Images