RU2785821C1 - Method for detecting objects in the image of the plan-scheme of the construction object - Google Patents

Abstract

FIELD: invention relates to the field of means for identifying objects on the plan-scheme of a construction site.

SUBSTANCE: proposed method includes the identification of conditional images of construction elements on images of real estate, which is carried out using convolutional neural networks, classifying the identified objects into single-level and multi-level. Further produce comparison, taking into account the level mark, among themselves of objects located at adjacent levels and classified as multi-level, combine objects located at adjacent levels, having the same location of the conditional image of the “stairs” construction element, into one object.

EFFECT: technical result is to provide the possibility of identifying multi-level objects on the plan-scheme of the construction object.

5 cl

Description

Technical field

This invention relates to means for identifying objects on the plan-scheme of a construction site based on determining the presence of symbol elements and can be used, in particular, in the analysis of various construction projects.

State of the art

From the prior art, the application CN 108764022 A, published on 11/06/2018, is known. This application discloses an image recognition method, in particular building plans, which involves the use of a trained neural network to create an image classifier. After entering the image into the classifier, it is marked with the corresponding class. All conditional classes can be classified into four categories. At the output of the specified classifier, estimates of the reliability of referring this or that symbol to a certain class are formed. If the confidence index exceeds a predetermined confidence threshold, it is concluded that the recognized designation belongs to the class assigned to it. This solution is aimed at improving the efficiency of processing symbols images on building plans.

Also known from the prior art is US 2019/0243928 A1, published on August 8, 2019, which discloses a method for semantic segmentation of 2d building plans using a pixel classifier. This method involves the semantic segmentation of the image of the layout of the room with its classification based on the individual pixels of the image using a trained neural network. Semantic segmentation provides the ability to identify several semantically significant parts in an image and classify each of these parts according to one of predetermined categories. Classification is carried out for a set of classes, including two classes of walls, doors and windows. The classifiers contain a probability distribution over a set of classes, with each pixel assigned a certain probability number. The method described above provides an increase in the accuracy of processing two-dimensional building plans in comparison with similar previous solutions.

The closest analogue of the claimed invention is a method for automated identification and use of information about the floor plan of a building, disclosed in the application US 2022/0092227 A1, published on 03/24/2022. Said method discloses performing operations for identifying floor plans of buildings having objects that are of interest to the user. In the process of implementing this method, the relative position of objects is determined, for example, rooms, walls, windows of buildings, and a corresponding graph is built. The identification of these objects on the building plans is carried out using a trained neural network. After analyzing the plans and identifying the available objects, those that meet the given criteria are determined, for example, apartments that have a given number of rooms with certain size ranges. Through the use of graphs that describe the relative position of objects, a search is provided by criteria such as the relationship between rooms. The invention described above makes it possible to provide fast and efficient identification of objects on the building plan under study.

The main disadvantage of the analogues of the claimed invention described above is the impossibility of identifying and identifying multi-level objects on the plan-scheme of the construction object.

The proposed invention is aimed at eliminating this disadvantage.

As a technical result achieved in the implementation of the invention, there is the possibility of identifying multi-level objects on the plan-scheme of the construction object.

The essence of the invention

The specified technical result and the task set are achieved / solved due to the fact that the method for identifying real estate objects in the image of a multi-level plan-scheme of a construction object, including:

loading an image of a multi-level plan-scheme of a construction object;

assignment of level labels to each level of the plan-scheme of the construction object,

identification using a convolutional neural network of the first type on images of levels of real estate objects with assignment of a level label to them,

using a convolutional neural network of the second type, recognition of conditional images of construction elements on images of identified real estate objects and classification of identified objects into multi-level and single-level, while identifying a conditional image of a construction element “stairs”, the object is classified as multi-level,

taking into account the level mark, comparison between objects located at adjacent levels and classified as multilevel,

combining objects located at adjacent levels, having the same location of the conditional image of the “staircase” construction element, into one object,

identifying and counting identical conditional images of construction elements corresponding to each object, as well as determining their location relative to each other;

setting object filtering parameters;

based on the data obtained on the identified conditional images of construction elements, their number, as well as their location relative to each other, identifying objects corresponding to the filtering parameters on a multi-level plan-scheme of the construction object,

at the same time, the neural network of the first type is trained on a set of data arrays, which includes images of floor plans of real estate objects,

the neural network of the second type is trained on a set of data arrays, which includes conditional images of building elements, and a set of data arrays, which includes images of floor plans of single/multi-level real estate objects.

In one of the options, the image of the multi-level plan-scheme of the construction site can be in RGB color format.

In one of the options, the conversion of a color image to black and white is carried out, which includes two stages:

converting a color image to a grayscale image;

converting a grayscale image to a black and white image.

In one of the options, the image of the level of the multi-level plan-scheme of the construction object is an image of the floor of the house.

In one embodiment, the property is an apartment or office.

Identification using a convolutional neural network of the first type on images of levels of real estate objects with assignment of a level label to them, recognition using a convolutional neural network of the second type on images of identified real estate objects of conditional images of construction elements and classification of identified objects into multi-level and single-level, comparison taking into account the level label between objects located at adjacent levels and classified as multi-level, the combination of objects located at adjacent levels, having the same location of the symbolic image of the construction element "ladder", into one object, provides the ability to identify multi-level objects on the plan of the construction object.

Implementation of the invention

Embodiments of the invention address the above and other disadvantages by providing mechanisms for identifying multi-level objects on a construction site plan.

The present disclosure describes a method for identifying real estate objects in a layered plan image of a construction site.

At the first stage, an image of a multi-level plan-scheme of the construction object is loaded. An image of a multi-level plan-scheme of a construction site can be presented in RGB color format. An image presented in RGB format can be converted from a color image to black and white. This transformation may include two steps:

converting a color image to a grayscale image; converting a grayscale image to a black and white image.

Each level image of a multi-level plan-scheme of a construction object is assigned a level label.

The image of the level of a multi-level plan-scheme of the construction object may be an image of the floor of the house.

Further, using a convolutional neural network of the first type, real estate objects are identified on the level images. Detected objects are assigned level labels showing on which level (floor) the detected object is located.

With the help of a convolutional neural network of the second type, conditional images of construction elements are recognized on the images of identified real estate objects and the identified objects are classified into multi-level and single-level.

The object is classified as multi-level when a conditional image of the construction element "stairs" is identified on the image of the real estate object.

Multilevel objects located on adjacent levels are compared with each other, taking into account the level label. Objects located at adjacent levels, having the same location of the conditional image of the "stair" construction element, are combined into one object.

Next, the identification and calculation of identical conditional images of construction elements corresponding to each object is carried out, as well as determining their location relative to each other.

After setting the parameters for filtering objects, based on the data obtained on the identified conditional images of construction elements, their number, as well as their location relative to each other, the objects corresponding to the filtering parameters are identified on the multilevel plan-scheme of the construction object.

Conditional images of building elements can, in particular, correspond to doorways between rooms, external doorways, windows, stairs, and others.

Filtration parameters can, in particular, be understood as parameters indicating the type of construction element (for example, a window, door, or other), the number of these elements, their location relative to each other, and others.

In some embodiments, the implementation of the claimed method can be performed on one or more computing systems connected by a network. The network may be, for example, the Internet or a private network. In addition, the network may include various types of wired and/or wireless networks.

Computing systems may include various hardware components such as processors and storage devices.

Information about the image plan-scheme of the construction site and/or additional related information may be obtained from one or more external sources.

The convolutional neural network used is a unidirectional (without feedback) multilayer neural network with alternating convolution layers and subsampling layers (subsampling layers or pooling layers, subsampling layers).

The use of convolutional neural networks is due to the fact that they provide partial resistance to scale changes, displacements, rotations, angle changes and other distortions. Convolutional neural networks combine three architectural ideas to provide invariance to scale, rotation, shift, and spatial distortion:

- local receptor fields (provide local two-dimensional connectivity of neurons);

- general synaptic coefficients (ensure the detection of some features anywhere in the image and reduce the total number of weight coefficients);

- hierarchical organization with spatial subsamples.

At the moment, the convolutional neural network and its modifications are considered the best algorithms for finding objects in the scene in terms of accuracy and speed.

In an ordinary perceptron, which is a fully connected neural network, each neuron is connected to all the neurons of the previous layer, and each connection has its own personal weight coefficient. In a convolutional neural network, the convolution operation uses only a limited weight matrix of a small size, which is “moved” over the entire processed layer (at the very beginning - directly over the input image), forming after each shift an activation signal for the neuron of the next layer with a similar position. That is, for different neurons of the output layer, the same weight matrix is used, which is also called the convolution kernel. It is interpreted as a graphic encoding of some feature, for example, the presence of an oblique line at a certain angle. Then the next layer, resulting from the convolution operation by such a weight matrix, shows the presence of this feature in the processed layer and its coordinates, forming the so-called feature map. Naturally, in a convolutional neural network, the set of weights is not one, but a whole range that encodes image elements (for example, lines and arcs at different angles). At the same time, such convolution kernels are not laid down by the researcher in advance, but are formed independently by training the network using the classical method of error backpropagation. Passing each set of weights generates its own feature map instance, making the neural network multichannel (many independent feature maps on one layer). It should also be noted that when iterating over a layer by the weight matrix, it is usually moved not by a full step (the size of this matrix), but by a small distance. So, for example, when the dimension of the weight matrix is 5 × 5, it is shifted by one or two neurons (pixels) instead of five, so as not to “step over” the desired feature.

The subsampling operation (eng. subsampling, eng. pooling, also translated as “subsampling operation” or pooling operation) performs a reduction in the dimension of the generated feature maps. In this network architecture, it is considered that information about the fact of the presence of the desired feature is more important than the exact knowledge of its coordinates, therefore, from several neighboring neurons of the feature map, the maximum one is selected and taken as one neuron of the compacted feature map of a smaller dimension. Due to this operation, in addition to speeding up further calculations, the network becomes more invariant to the scale of the input image.

A convolutional neural network consists of a large number of layers. After the initial layer (input image), the signal passes through a series of convolutional layers, in which the actual convolution and subsampling (pooling) alternate.

Alternating layers allows you to make "feature maps" from feature maps, on each next layer the map decreases in size, but the number of channels increases. In practice, this means the ability to recognize complex feature hierarchies. Usually, after passing through several layers, the feature map degenerates into a vector or even a scalar, but there are hundreds of such feature maps. At the output of the convolutional layers of the network, several layers of a fully connected neural network (perceptron) are additionally installed, at the input of which end feature maps are fed.

The convolutional layer is the main building block of a convolutional neural network. The convolution layer includes its own filter for each channel, the convolution kernel of which processes the previous layer by fragments (summing up the results of the element-wise product for each fragment). The weights of the convolution kernel (small matrix) are unknown and are set during training.

A feature of the convolutional layer is a relatively small number of parameters that are set during training. So, for example, if the original image has a dimension of 100 × 100 pixels in three channels (which means 30,000 input neurons), and the convolutional layer uses filters with a 3 × 3 pixel kernel with an output of 6 channels, then only 9 weights are determined in the learning process kernel, however, for all combinations of channels, that is, 9 × 3 × 6 = 162, in this case, this layer requires finding only 162 parameters, which is significantly less than the number of required parameters of a fully connected neural network.

The pooling layer (otherwise subsampling, subsampling) is a non-linear compaction of a feature map, while a group of pixels (usually 2 × 2 in size) is compacted to one pixel, undergoing a non-linear transformation. In this case, the maximum function is most commonly used. The transformations affect non-overlapping rectangles or squares, each of which is compressed into one pixel, and the pixel with the maximum value is selected. The pooling operation can significantly reduce the spatial volume of the image. Pooling is interpreted as follows: if some features were already identified during the previous convolution operation, then such a detailed image is no longer needed for further processing, and it is compacted to a less detailed one. In addition, filtering out unnecessary details helps not to retrain. The pooling layer is typically inserted after the convolution layer, before the next convolution layer.

After several passes of image convolution and pooling compression, the system is rebuilt from a specific grid of high-resolution pixels to more abstract feature maps, as a rule, on each subsequent layer, the number of channels increases and the image dimension decreases in each channel. In the end, what remains is a large set of channels that store a small amount of data (even a single parameter) that are interpreted as the most abstract concepts revealed from the original image.

This data is combined and transmitted to a conventional fully connected neural network, which can also consist of several layers. At the same time, fully connected layers already lose the spatial structure of pixels and have a relatively small dimension (in relation to the number of pixels in the original image).

The learning method used can be the method of learning with a teacher (on labeled data) - the method of back propagation of an error and its modifications.

A convolutional neural network of the first type is used to identify the levels of construction objects of objects related to a real estate object on general plan diagrams. To solve this problem, a first type convolutional neural network is trained on datasets that include images of floor plans of real estate objects, which are positive examples, and images of floor plans of common areas, which are negative examples. The ratio of positive to negative examples is 4 to 1, 8000 positive and 2000 negative.

After processing the input data containing the general plan-scheme of the level of the construction object with the level label, the convolutional neural network of the first type, the output will be a set of plans-schemes of individual real estate objects, which is the input data set for the convolutional neural network of the second type.

A convolutional neural network of the second type can be trained on a dataset that includes conditional images of building elements, such as windows, doors, walls, and a dataset that includes images of layouts of single / multi-level objects. real estate.

The steps of the method described above make it possible to identify multi-level objects on the plan-scheme of the construction object.

Claims (18)

1. A method for detecting images of real estate objects on an image of a multi-level diagram of a construction object, including: loading an image of a multi-level plan-scheme of a construction object; assignment of level labels to each level of the plan-scheme of the construction object, identification using a convolutional neural network of the first type on images of levels of real estate objects with assignment of a level label to them, using a convolutional neural network of the second type, recognition of conditional images of construction elements on images of identified real estate objects and classification of identified objects into multi-level and single-level, while identifying a conditional image of a construction element "stairs" classifies the object as multi-level, taking into account the level mark, comparison between objects located at adjacent levels and classified as multilevel, combining objects located at adjacent levels, having the same location of the conditional image of the “staircase” construction element, into one object, identification and calculation of identical conditional images of construction elements corresponding to each object, as well as determining their location relative to each other; setting object filtering parameters; based on the data obtained on the identified conditional images of construction elements, their number, as well as their location relative to each other, identifying objects corresponding to the filtering parameters on a multi-level plan-scheme of the construction object, at the same time, the neural network of the first type is trained on a set of data arrays, which includes images of plans-schemes of real estate objects, the neural network of the second type is trained on a set of data arrays, which includes conditional images of construction elements, and a set of data arrays, which includes images of plans-diagrams of single/multi-level real estate objects. 2. The method according to claim 1, characterized in that the image of the multi-level plan-scheme of the construction object can be in RGB color format. 3. The method according to claim 2, characterized in that the color image is converted to black and white, which includes two stages: converting a color image to a grayscale image; converting a grayscale image to a black and white image. 4. The method according to claim 1, in which the image of the level of the multi-level plan-scheme of the construction object is an image of the floor of the house. 5. The method according to claim 1, in which the property is an apartment or office.