KR102210614B1 - Method and server for differentiating space state information of structure based on machine learning - Google Patents

Abstract

Disclosed are a method for discriminating spatial state information of a building based on machine learning to objectively discriminate the spatial state information based on data, and a spatial state information discrimination server using the same. According to the present invention, the method comprises the following steps of: (a) when a real image representing a space of the building is acquired from a user terminal, allowing the spatial state information discrimination server to input the real image to a feature extraction block so that the feature extraction block can analyze the real image, extract a feature from the real image, and output a feature vector; and (b) allowing the spatial state information discrimination server to input the feature vector to a spatial state classification block so that the spatial state classification block can perform a neural network calculation on the feature vector to output the spatial state information of the building on the real image, and to transmit the spatial state information to the user terminal, thereby supporting the user terminal to display the spatial state information.

Description

A method for determining spatial state information of a building based on machine learning, and a spatial state information determination server using it {METHOD AND SERVER FOR DIFFERENTIATING SPACE STATE INFORMATION OF STRUCTURE BASED ON MACHINE LEARNING}

The present invention relates to a method for determining spatial state information of a building based on machine learning and a spatial state information determination server using the same.

Conventionally, in order to check the type of construction applied to the building, compliance with the safety rules, and whether defects or defects have occurred, experts had to personally visit numerous individual spaces on the construction site and go through the inspection process, so the construction site or the building subject to maintenance There was a problem in that the number of manpower to be put into operation was excessive.

In addition, a method of integrated management of a construction site using a construction site management application is disclosed in Korean Patent Registration No. 10-1829169, but in such a method, the existing stored construction progress status, compliance with safety rules, and whether defects or defects occur. Since only can be checked, there is still a problem that it is difficult to check the above items in real time.

Therefore, there is a need for an improvement method to solve the above problems.

KR 10-1829169 B1

An object of the present invention is to solve all of the above-described problems.

In addition, the present invention objectively provides spatial state information based on data so as not to erroneously determine the status of construction work, compliance with safety rules, and occurrence of defects or defects based on subjective standards of construction site officials or those concerned with buildings to be maintained. Discrimination is for another purpose.

In addition, another object of the present invention is to enable a large amount of work related to a construction site or a maintenance target building to be processed with a small number of manpower.

In addition, another object of the present invention is to reduce the cost input to a construction site or a maintenance target building by minimizing the relevant manpower and work steps.

In addition, another object of the present invention is to enable anyone who has not acquired related knowledge to specifically check the status of construction work, compliance with safety rules, and whether defects or defects have occurred.

In addition, another object of the present invention is to increase the safety of persons concerned by allowing persons concerned to grasp the status of construction work without contacting the structure at a construction site or a building subject to maintenance.

In order to achieve the object of the present invention as described above, and to realize the characteristic effects of the present invention described later, the characteristic configuration of the present invention is as follows.

According to an aspect of the present invention, in a method of determining spatial state information of a building based on machine learning, (a) when a live image representing a space of a building is obtained from a user terminal, the spatial state information determination server, Inputting the real-life image to a feature extraction block, causing the feature-extraction block to analyze the real-life image and extract a feature on the real-world image to output a feature vector; And (b) the spatial state information determination server inputs the feature vector as a spatial state classification block to cause the spatial state classification block to calculate the feature vector by a neural network, so that the spatial state information of the building on the real image image. And supporting the user terminal to display the spatial state information by outputting the spatial state information and transmitting the spatial state information to the user terminal.

As an example, the spatial state classification block includes a fully connected layer, and in step (b), the spatial state information determination server inputs the feature vector to the FC layer to cause the FC layer By performing a neural network operation on the feature vector, a specific construction class corresponding to the space of the building on the real-life image is determined among preset construction classes, and then the spatial state information corresponding to the specific construction class is calculated. A method is disclosed, characterized in that the output is made.

As an example, the feature extraction block includes a convolution layer, a region proposal network (RPN), and a pooling layer, the spatial state classification block includes an FC layer, and the step (a) In the spatial state information determination server, by inputting the real-life image to the convolution layer, the convolutional layer performs a convolution operation on the real-life image at least once, so that a spatial feature corresponding to the real-life image A map is output, and the spatial feature map is input to the RPN to cause the RPN to output proposal boxes corresponding to an area in which an operator is located on the spatial feature map, and the pooling Allows a layer to output the feature vectors corresponding to the pooled feature maps by pooling regions on the spatial feature maps respectively corresponding to the proposal boxes, and in step (b), the spatial state information determination server , The spatial state information corresponding to the safety state after inputting the feature vector to the FC layer and causing the FC layer to check the safety equipment worn by the worker to determine whether the worker is in a safe state. Disclosed is a method characterized in that to output.

As an example, the convolutional layer and the FC layer are in a state in which training data is used to output an arbitrary spatial feature map and arbitrary spatial state information, respectively, and an image for training-the image for training is an arbitrary space. When is acquired, (i) the training video image is input to the convolution layer, causing the convolution layer to perform convolution calculation of the training video image at least once. A training spatial feature map corresponding to the training video image is output, and (ii) the training spatial feature map is input to the RPN to cause the RPN to learn a proposal corresponding to an area where a training worker is located on the training spatial feature map. Boxes are output, and the pooling layer causes the areas on the learning spatial feature map corresponding to the learning proposal boxes to be pooled to output the pooled feature maps for learning, and the pooled feature maps for learning are the learning features. After being converted to a vector, the learning feature vector is input to the FC layer, causing the FC layer to check the safety equipment worn by the learning worker to determine whether the learning worker is in a safe state, and then the safety. The learning spatial state information corresponding to the state is output, and (iii) the loss layer makes the learning spatial state information and the corresponding GT (Ground Truth) reference to generate a loss, and the loss is A method is disclosed, wherein at least one of the parameters of the FC layer and the convolution layer is in a learned state by a back propagation process used.

As an example, the feature extraction block includes a convolution layer, an RPN, and a pooling layer, the spatial state classification block includes an FC layer, and in step (a), the spatial state information determination server, the real image By inputting an image to the convolutional layer, the convolutional layer performs a convolution operation on the real-life image at least once to output a spatial feature map corresponding to the real-life image, and the spatial feature map is the RPN Input to the RPN to output the proposal boxes corresponding to the region determined to have defects or defects on the spatial feature map, and cause the pooling layer to output the spatial feature maps respectively corresponding to the proposal boxes The image regions are pooled to output the feature vector corresponding to the pooled feature maps, and in step (b), the spatial state information determination server inputs the feature vector to the FC layer to cause the FC layer Determine whether the defect or the defect actually exists in the space, and if the defect or the defect actually exists, classify the type of the defect or the defect and then respond to the type of the defect or the defect. Disclosed is a method characterized in that the spatial state information is output.

As an example, the feature extraction block includes a convolution layer, an RPN, and a pooling layer, and the spatial state classification block includes a first FC layer and a second FC layer, and in step (a), the spatial state information The determination server causes the convolutional layer to convolve the real-life image image at least once by inputting the real-life image image to the convolution layer to output a spatial feature map corresponding to the real-life image, By inputting the spatial feature map to the RPN, the RPN outputs proposal boxes corresponding to an area in which a worker is located on the spatial feature map, and the pooling layer causes the space corresponding to each of the proposal boxes. The regions on the feature map are pooled to output a first feature vector corresponding to the pooled feature maps. In step (b), the spatial state information determination server comprises: (i) the spatial output from the convolution layer By inputting a second feature vector corresponding to a feature map as a first FC layer and causing the first FC layer to calculate the second feature vector with a neural network, the structure of the building on the real-life image image among preset classes After determining a specific public type class corresponding to the space, specific public type information corresponding to the specific public type class is output, and (ii) the second FC layer by inputting the first feature vector to the second FC layer The layer checks the safety equipment worn by the worker to output specific safety equipment wearing state information, and (iii) the reference safety equipment wearing state information set in correspondence with the specific work type information output from the first FC layer Generates worker safety status information indicating whether the worker wears safety equipment corresponding to the specific work type information by referring to the specific safety equipment wearing status information output from the second FC layer, and the worker safety status information Disclosed is a method comprising supporting the user terminal to display the worker safety status information by transmitting it to the user terminal.

As an example, the convolutional layer, the first FC layer, and the second FC layer are in a state in which training data is used to output an arbitrary spatial feature map and arbitrary spatial state information, respectively, and the training video image-the The training video image is a video image photographed in an arbitrary space-When is obtained, by the learning device, (i) the training video image is input to the convolution layer, causing the convolution layer to have at least the training video image A convolution operation is performed once so that a learning spatial feature map corresponding to the learning video image is output, and the learning spatial feature map is input to the RPN to cause the RPN to enter the area where the learning worker is located on the learning spatial feature map. Corresponding learning proposal boxes are output, and regions on the learning spatial feature map corresponding to the learning proposal boxes are pooled by the pooling layer, so that a learning first feature vector corresponding to the pooled feature maps for learning is obtained. And (ii-1) a second learning feature vector corresponding to the learning spatial feature map output from the convolutional layer is input to the first FC layer, causing the first FC layer to cause the learning second feature vector By allowing the neural network to be calculated, a specific construction class corresponding to the space of the building on the learning image image among preset construction classes is determined, and then specific construction type information for learning corresponding to the specific construction class is output. , (ii-2) The first feature vector for learning is input to the second FC layer so that the second FC layer checks the safety equipment worn by the learning worker, so that information on the wearing state of specific safety equipment for learning is output. And, (iii-1) the first loss is generated by allowing the loss layer to refer to the specific construction type information for learning and the corresponding first GT (Ground Truth). And (iii-2) a second loss is generated by allowing the loss layer to refer to the training specific safety equipment wearing state information and a corresponding second GT (ground truth), and the first loss And at least one of parameters of the first FC layer, the second FC layer, and the convolution layer is learned by a back propagation process using the second loss. The method is disclosed.

As an example, in a state in which spatial information, which is information about individual spaces within the building specified by at least one of the building's building, building, floor, and room, is obtained, in step (b), the spatial state information is determined. The server checks the specific spatial information in which the user terminal is located among the spatial information by referring to the location information where the user terminal is located, and transmits the specific spatial information to the user terminal, causing the user terminal to Disclosed is a method comprising supporting to display the spatial state information and the specific spatial information.

As an example, in the step (b), the spatial state information determination server transmits the spatial state information to another user terminal, thereby supporting the other user terminal to display the spatial state information. Is initiated.

As an example, in the step (b), the spatial state information determination server transmits the spatial state information and the live-action image to the other user terminal, and (c) randomly selects the live-action image from the other user terminal. When the indication information for a specific area among the areas of is obtained, the spatial state information determination server transmits the indication information to the user terminal to cause the user terminal to render a markup corresponding to the indication information, Disclosed is a method further comprising: supporting the markup to be displayed in the specific area of the live-action image image.

According to another aspect of the present invention, there is provided a method for determining spatial state information of a building based on machine learning, comprising: at least one memory for storing instructions; And at least one processor configured to execute the instructions, wherein the processor includes: (1) when a real-life image representing a space of a building is obtained from a user terminal, the real-life image is input to a feature extraction block, A process of causing a feature extraction block to analyze the real-life image, extract a feature on the real-life image, and output a feature vector; And (2) inputting the feature vector as a spatial state classification block to cause the spatial state classification block to calculate the feature vector with a neural network to output spatial state information of the building on the real-life image, and the spatial state Disclosed is a spatial state information determination server that executes the instructions for performing a process of supporting the user terminal to display the spatial state information by transmitting information to the user terminal.

As an example, the spatial state classification block includes a fully connected layer, and the processor, in the (2) process, inputs the feature vector to the FC layer to cause the FC layer to generate the feature vector. By performing a neural network calculation, after determining a specific construction class corresponding to the space of the building on the real-life image among preset construction classes, outputting the spatial state information corresponding to the specific construction class. A characterized spatial state information determination server is disclosed.

As an example, the feature extraction block includes a convolution layer, a region proposal network (RPN), and a pooling layer, the spatial state classification block includes an FC layer, and the processor includes: (1) In the process, by inputting the real-life image to the convolution layer, the convolutional layer causes the real-life image to be convolved at least once, and a spatial feature map corresponding to the real-life image is performed. ), and inputs the spatial feature map to the RPN to cause the RPN to output proposal boxes corresponding to an area in which a worker is located on the spatial feature map, and the pooling layer causes the Regions on the spatial feature map corresponding to each of the proposal boxes are pooled to output the feature vector corresponding to the pooled feature maps, and in the (2) process, the feature vector is input to the FC layer and the Spatial state information discrimination, characterized in that the FC layer checks the safety equipment worn by the worker to determine whether the worker is in a safe state, and then outputs the spatial state information corresponding to the safety state. The server is started.

As an example, the convolutional layer and the FC layer are in a state in which training data is used to output an arbitrary spatial feature map and arbitrary spatial state information, respectively, and an image for training-the image for training is an arbitrary space. When is acquired, (i) the training video image is input to the convolution layer, causing the convolution layer to perform convolution calculation of the training video image at least once. A training spatial feature map corresponding to the training video image is output, and (ii) the training spatial feature map is input to the RPN to cause the RPN to learn a proposal corresponding to an area where a training worker is located on the training spatial feature map. Boxes are output, and the pooling layer causes the areas on the learning spatial feature map corresponding to the learning proposal boxes to be pooled to output the pooled feature maps for learning, and the pooled feature maps for learning are the learning features. After being converted to a vector, the learning feature vector is input to the FC layer, causing the FC layer to check the safety equipment worn by the learning worker to determine whether the learning worker is in a safe state, and then the safety. The learning spatial state information corresponding to the state is output, and (iii) the loss layer makes the learning spatial state information and the corresponding GT (Ground Truth) reference to generate a loss, and the loss is A spatial state information determination server, characterized in that at least one of the parameters of the FC layer and the convolution layer is learned by a back propagation process used.

As an example, the feature extraction block includes a convolutional layer, an RPN, and a pooling layer, the spatial state classification block includes an FC layer, and the processor includes, in the (1) process, the real-life image By inputting into the lusion layer, the convolutional layer causes the real-life image to be convoluted at least once to output a spatial feature map corresponding to the real-life image, and the spatial feature map is input to the RPN and the Causes the RPN to output proposal boxes corresponding to regions determined to have defects or defects on the spatial feature map, and cause the pooling layer to pull regions on the spatial feature map corresponding to the proposal boxes respectively To output the feature vector corresponding to the pooled feature maps, and in the (2) process, input the feature vector to the FC layer to cause the FC layer to determine whether the defect or the defect actually exists in the space. To determine whether the defect or the defect actually exists, classify the type of the defect or the defect, and output the spatial state information corresponding to the type of the defect or the defect. The spatial state information determination server is started.

As an example, the feature extraction block includes a convolution layer, an RPN, and a pooling layer, the spatial state classification block includes a first FC layer and a second FC layer, and the processor, in the (1) process, By inputting the real-life image image to the convolution layer, the convolutional layer causes the real-life image to be convolved at least once to output a spatial feature map corresponding to the real-life image, and the spatial feature map Input into the RPN to cause the RPN to output proposal boxes corresponding to an area in which a worker is located on the spatial feature map, and to cause the pooling layer to output an area on the spatial feature map corresponding to each of the proposal boxes. Are pooled to output a first feature vector corresponding to the pooled feature maps, and in the (2) process, (i) a second feature vector corresponding to the spatial feature map output from the convolution layer is first By inputting into an FC layer and causing the first FC layer to calculate the second feature vector, a specific construction class corresponding to the space of the building on the real-life image among preset construction classes is determined. Thereafter, the specific work type information corresponding to the specific work class is output, and (ii) the first feature vector is input to the second FC layer, and the second FC layer checks the safety equipment worn by the worker. To output specific safety equipment wearing state information, and (iii) reference safety equipment wearing state information set in response to the specific work type information output from the first FC layer and the specific safety equipment output from the second FC layer By referring to wearing state information, worker safety state information indicating whether the worker wears safety equipment corresponding to the specific work type information is generated, and the worker safety state information is transmitted to the user terminal. Disclosed is a spatial state information determination server, which supports a user terminal to display the worker safety state information.

As an example, the convolutional layer, the first FC layer, and the second FC layer are in a state in which training data is used to output an arbitrary spatial feature map and arbitrary spatial state information, respectively, and the training video image-the The training video image is a video image photographed in an arbitrary space-When is obtained, by the learning device, (i) the training video image is input to the convolution layer, causing the convolution layer to have at least the training video image A convolution operation is performed once so that a learning spatial feature map corresponding to the learning video image is output, and the learning spatial feature map is input to the RPN to cause the RPN to enter the area where the learning worker is located on the learning spatial feature map. Corresponding learning proposal boxes are output, and regions on the learning spatial feature map corresponding to the learning proposal boxes are pooled by the pooling layer, so that a learning first feature vector corresponding to the pooled feature maps for learning is obtained. And (ii-1) a second learning feature vector corresponding to the learning spatial feature map output from the convolutional layer is input to the first FC layer, causing the first FC layer to cause the learning second feature vector By allowing the neural network to be calculated, a specific construction class corresponding to the space of the building on the learning image image among preset construction classes is determined, and then specific construction type information for learning corresponding to the specific construction class is output. , (ii-2) The first feature vector for learning is input to the second FC layer so that the second FC layer checks the safety equipment worn by the learning worker, so that information on the wearing state of specific safety equipment for learning is output. And, (iii-1) the first loss is generated by allowing the loss layer to refer to the specific construction type information for learning and the corresponding first GT (Ground Truth). And (iii-2) a second loss is generated by allowing the loss layer to refer to the training specific safety equipment wearing state information and a corresponding second GT (ground truth), and the first loss And at least one of parameters of the first FC layer, the second FC layer, and the convolution layer is learned by a back propagation process using the second loss. The spatial state information determination server is started.

As an example, in a state in which spatial information, which is information about individual spaces within the building specified by at least one of the building's wing, arc, floor, and room, is obtained, the processor, in the process (2), Among the spatial information, specific spatial information in which the user terminal is located is checked with reference to the location information in which the user terminal is located, and the specific spatial information is transmitted to the user terminal, so that the user terminal causes the spatial state information. And a spatial state information determination server that supports to display the specific spatial information.

As an example, the processor supports the other user terminal to display the spatial state information by transmitting the spatial state information to another user terminal in the (2) process. Is initiated.

As an example, the processor, in the (2) process, transmits the spatial state information and the live-action image image to the other user terminal, and (3) a specific area of the live-action image image from the other user terminal. When the indication information for the region is obtained, the spatial state information determination server transmits the indication information to the user terminal to cause the user terminal to render a markup corresponding to the indication information, and perform the markup. The spatial state information determination server further comprises a process of supporting the display of the real-life image on the specific area.

In addition to this, a computer-readable recording medium for recording a computer program for executing the method of the present invention is further provided.

The present invention objectively determines spatial state information based on data so that the status of construction work, compliance with safety rules, and occurrence of defects or defects, etc. are not erroneously determined based on subjective standards of construction site officials or those concerned with buildings to be maintained. It works.

In addition, the present invention has the effect of allowing a large amount of work related to a construction site or a maintenance target building to be processed with a small number of manpower.

In addition, the present invention has the effect of reducing the cost input to the construction site or the maintenance target building by minimizing the relevant manpower and work steps.

In addition, the present invention has the effect of allowing anyone who has not acquired related knowledge to specifically check the status of construction types, compliance with safety rules, and whether defects or defects have occurred.

In addition, the present invention has the effect of increasing the safety of the person concerned by allowing the person concerned to grasp the status of the work type without contacting the structure at the construction site or the maintenance target building.

The accompanying drawings, which are attached to be used in the description of the embodiments of the present invention, are only some of the embodiments of the present invention, and those of ordinary skill in the technical field to which the present invention pertains Other drawings can be obtained based on these drawings without any further work being done.
1 schematically illustrates a spatial state information determination server, a user terminal, and other user terminals for determining spatial state information of a building based on machine learning according to an embodiment of the present invention.
2 schematically illustrates a process of determining spatial state information of a building according to an embodiment of the present invention,
3 schematically shows a state of determining a specific public type class according to an embodiment of the present invention,
4 schematically shows a state of determining a safety state according to an embodiment of the present invention,
5 schematically shows a state of determining whether a defect or defect exists and its type according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION The detailed description of the present invention to be described below refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced in order to clarify the objects, technical solutions and advantages of the present invention. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the present invention.

Further, throughout the detailed description and claims of the present invention, the word "comprises" and variations thereof are not intended to exclude other technical features, additions, components or steps. Other objects, advantages, and features of the present invention to those skilled in the art will appear, partly from the present disclosure, and partly from the practice of the present invention. The examples and drawings below are provided by way of example and are not intended to limit the invention.

Moreover, the present invention covers all possible combinations of the embodiments indicated herein. It is to be understood that the various embodiments of the present invention are different from each other, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all scopes equivalent to those claimed by the claims. In the drawings, like reference numerals refer to the same or similar functions over several aspects.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable those of ordinary skill in the art to easily implement the present invention.

1 schematically shows a spatial state information determination server 100, a user terminal 200, and another user terminal 300 for determining spatial state information of a building based on machine learning according to an embodiment of the present invention. Referring to FIG. 1, the spatial state information determination server 100 corresponds to a memory 110 storing instructions for determining spatial state information of a building and instructions stored in the memory 110 based on machine learning. Accordingly, it may include a processor 120 that performs an operation of determining spatial state information of a building based on machine learning.

Specifically, the spatial state information determination server 100 is typically a computing device (e.g., a device that may include a computer processor, a memory, a storage, an input device and an output device, and other components of a conventional computing device; a router, a switch). Electronic communication devices such as; using a combination of electronic information storage systems such as network attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that make the computing device function in a specific way) To achieve the desired system performance.

In addition, the processor of the computing device may include a hardware configuration such as a micro processing unit (MPU) or a central processing unit (CPU), a cache memory, and a data bus. In addition, the computing device may further include an operating system and a software configuration of an application that performs a specific purpose.

However, it is not excluded that the computing device includes an integrated processor in which a medium, a processor, and a memory are integrated for implementing the present invention.

On the other hand, the user terminal 200 and the other user terminal 300, respectively, each memory (210, 310) that stores instructions for displaying the spatial state information of the building and instructions stored in each of the memories (210, 310). Each of the processors 220 and 320 may be configured to perform an operation of displaying spatial state information of a building in response to the corresponding structure.

Specifically, each of the user terminal 200 and the other user terminal 300 typically includes a computing device (e.g., a computer processor, memory, storage, input device and output device, and other components of an existing computing device). Devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network attached storage (NAS) and storage area networks (SANs) and computer software (i.e., instructions that cause the computing device to function in a specific way) ) May be used to achieve the desired system performance.

In addition, the processor of the computing device may include a hardware configuration such as a micro processing unit (MPU) or a central processing unit (CPU), a cache memory, and a data bus. In addition, the computing device may further include an operating system and a software configuration of an application that performs a specific purpose.

However, it is not excluded that the computing device includes an integrated processor in which a medium, a processor, and a memory are integrated for implementing the present invention.

A method for determining spatial state information of a building based on machine learning using the spatial state information determination server 100, the user terminal 200 and the other user terminal 300 according to an embodiment of the present invention configured as described above. It is as follows.

For reference, the convolution layer 410, the RPN 420, the pooling layer 430, the first FC layer 510 and the second FC layer 520 shown in FIG. 2 are the spatial state information determination server 100 It may be included in, but is not limited thereto, and may be configured as a separate independent device. In addition, the first FC layer 510 and the second FC layer 520 may obtain a classification score through a classifier, but are not limited thereto, for example, a league through a regressor. It is also possible to obtain additional lesson information. A detailed description of this will be described later.

When the overall process of outputting spatial state information and supporting it to be displayed on the user terminal 200 is described, when a live-action image representing the space of the building is obtained from the user terminal 200, the spatial state information determination server 100 , By inputting the real image image to the feature extraction block 400, the feature extraction block 400 may analyze the real image image and extract a feature on the real image image to output a feature vector.

Here, the spatial status information is information on the type of work applied to a specific part of the building space, information on the safety status of workers working in the building space, information on defects or defects occurring in a specific part of the building space And the like.

In addition, the spatial state information determination server 100 inputs a feature vector into the spatial state classification block 500 and causes the spatial state classification block 500 to calculate the feature vector in a neural network, The state information is output, and the spatial state information is transmitted to the user terminal 200 to enable the user terminal 200 to display the spatial state information.

First, a case of outputting spatial state information about a construction type applied to a specific part of a building space will be described in detail with reference to FIG. 2.

Specifically, when the spatial state classification block 500 includes a fully connected layer, the spatial state information determination server 100 inputs a feature vector as an FC layer and causes the FC layer to compute the feature vector in a neural network. By doing so, it is possible to determine a specific construction class corresponding to the space of the building on the live-action image among the preset construction class classes, and then output spatial state information corresponding to the specific construction class class. Here, the preset construction type classes may include all construction types applicable to a construction site or a maintenance target building.

As an example, in FIG. 3, the spatial state information corresponding to the masonry construction is transmitted to the user terminal 200, and the construction type shown in the live-action image is a masonry construction and is displayed on the lower end of the user terminal 200. have.

Next, a specific process will be described with reference to FIG. 2 again for the case of outputting spatial state information on the safety state of a worker working in the space of the building.

Specifically, when the feature extraction block 400 includes a convolution layer, a region proposal network (RPN), and a pooling layer, and the spatial state classification block 500 includes an FC layer, The spatial state information determination server 100 inputs a real-life image to the convolution layer 410 to cause the convolution layer 410 to convolve the real-life image at least once, thereby providing a spatial feature corresponding to the real-life image. A map (feature map) is output, and the spatial feature map output from the convolution layer 410 is input to the RPN 420, causing the RPN 420 to a proposal box corresponding to the area where the operator is located on the spatial feature map. (proposal boxes) may be output, and the pooling layer 430 may be able to output feature vectors corresponding to the pooled feature maps by pooling regions on a spatial feature map corresponding to each of the proposal boxes.

In addition, the spatial state information determination server 100 inputs the feature vector as an FC layer to allow the FC layer to check the safety equipment worn by the worker to determine whether the worker is in a safe state, and then respond to the safety state. It is possible to output spatial state information.

As an example, a description will be made of a case in which a live-action image image indicating a state in which a worker is wearing safety shoes but not wearing a hard hat is acquired at a construction site. That is, the spatial state information determination server 100 may allow the FC layer to determine that the worker is not currently wearing a hard hat by allowing the FC layer to check the safety equipment worn by the corresponding worker, and the safety state of the worker. When the corresponding spatial state information is output and the corresponding spatial state information is transmitted to the user terminal 200, as shown in FIG. 4, the operator shown in the live image image is not currently wearing a hard hat, and the user terminal 200 Can be displayed on.

In addition, the spatial state information determination server 100 may support not only whether a worker is currently wearing a hard hat, but also a bounding box corresponding to the worker as shown in FIG. 4 to be displayed on the user terminal 200.

In addition, in the above, as a safety state corresponding to the spatial state information, the current state of wearing safety equipment is described as an example, but the present disclosure is not limited thereto. Spatial state information corresponding to the probability of occurrence may be displayed.

As an example, when an elevator installation work is in progress at a construction site, and an elevator door is not yet installed on each floor, when a live-action image of the elevator door is obtained, the space state information determination server 100 ) May cause the FC layer to output spatial state information indicating the possibility of a fall safety accident, through the neural network computation process described above, and transmit the spatial state information to the user terminal 200, thereby allowing the user terminal ( 200) may be supported to display a notification window warning that a fall safety accident may occur.

On the other hand, the convolution layer 410 and the second FC layer 520 are in a state in which training data is used and are in a state in which they are trained to output arbitrary spatial feature maps and arbitrary spatial state information, respectively. When is acquired, (i) the training video image is input to the convolution layer 410, causing the convolution layer 410 to receive the training video image at least once. Convolution operation is performed so that the learning spatial feature map corresponding to the learning image is output, and (ii) the learning spatial feature map is inputted to the RPN 420 to cause the RPN 420 to have a learning operator on the learning spatial feature map. The learning proposal boxes corresponding to the regions are output, and the pooling layer 430 causes the regions on the learning spatial feature map corresponding to the learning proposal boxes to be pooled to output the pooled feature maps for learning, and pooling for learning. After the converted feature maps are converted into learning feature vectors, the learning feature vectors are input to the FC layer, and the FC layer allows the FC layer to check the safety equipment worn by the learning worker to determine whether the learning worker is in a safe state. The spatial state information for learning corresponding to the state is output, and (iii) the loss layer is made to refer to the spatial state information for learning and the corresponding GT (Ground Truth) to generate a loss. At least one parameter among the parameters of the FC layer and the convolution layer 410 may be learned by a back propagation process.

Next, a detailed process will be described with reference to FIG. 2 again for the case of outputting spatial state information about defects or defects occurring in a specific part of the building space.

Specifically, when the feature extraction block 400 includes a convolution layer 410, an RPN 420, and a pooling layer 430, and the spatial state classification block 500 includes an FC layer, spatial state information is determined The server 100 outputs a spatial feature map corresponding to the real image image by inputting the real image image to the convolution layer 410 so that the convolution layer 410 performs a convolution operation on the real image image at least once. The spatial feature map is inputted to the RPN 420 so that the RPN 420 outputs proposal boxes corresponding to the region determined to have a defect or defect on the spatial feature map, and the pooling layer 430 By pooling regions on a spatial feature map corresponding to each of the proposal boxes, it is possible to output a feature vector corresponding to the pooled feature maps.

In addition, the spatial state information determination server 100 inputs the feature vector to the FC layer to allow the FC layer to determine whether a defect or defect actually exists in the space, and when the defect or defect actually exists, the defect or defect After classifying the types of defects, spatial state information corresponding to the types of defects or defects may be output.

As an example, a description will be given of a case in which a live-action image image containing a leak defect occurred on the ceiling of a specific space in a building to be maintained is acquired.

In this case, the spatial state information determination server 100 may determine whether a defect or defect actually exists by analyzing the corresponding real-life image image, and display only the type of the defect or defect through the user terminal 200. However, the present invention is not limited thereto, and the location of the defect or defect may be displayed together with the type of the defect or defect.

That is, the spatial state information determination server 100 may analyze a real-life image containing a leak defect through the image processing process described above to calculate the classification scores and regression information. Then, the classification score is referenced to determine that the type of the defect is a leak defect, and the regression information is referred to determine the location of the leak defect in the live image, and spatial state information corresponding to the leak defect is output, When the spatial state information corresponding to the leak defect is transmitted to the user terminal 200, as shown in FIG. 5, the spatial state information determination server 100, the type of the defect observed on the live image is a leak defect. The occurrence location of is corresponding to a predetermined location on the image and may be supported to be displayed on the user terminal 200.

Meanwhile, the spatial state information determination server 100 may determine whether or not the current worker wears safety equipment corresponding to the specific work type by referring to information on a specific work type and safety equipment wearing state information of the worker.

That is, the feature extraction block 400 includes a convolutional layer 410, an RPN 420, and a pooling layer 430, and the spatial state classification block 500 includes a first FC layer 510 and a second FC layer. When including the layer 520, the spatial state information determination server 100, by inputting the real image image into the convolution layer 410, the convolutional layer 410 performs a convolution operation of the real image image at least once So that the spatial feature map corresponding to the real-life image is output, and the spatial feature map is input to the RPN 420 so that the RPN 420 outputs the proposal boxes corresponding to the area where the operator is located on the spatial feature map. In addition, the pooling layer 430 may pool regions on a spatial feature map corresponding to each of the proposal boxes to output a first feature vector corresponding to the pooled feature maps.

Then, the spatial state information determination server 100, (i) inputs the second feature vector corresponding to the spatial feature map output from the convolution layer 410 to the first FC layer 510 to the first FC layer ( 510) to determine a specific construction class corresponding to the space of the building in the real-life image among preset construction classes by performing a neural network operation on the second feature vector, and then determine specific construction type information corresponding to the specific construction type class. And (ii) inputting the first feature vector to the second FC layer 520 to cause the second FC layer 520 to check the safety equipment worn by the worker and output specific safety equipment wearing state information. , (iii) The operator is specified by referring to the standard safety equipment wearing state information set in response to the specific work type information output from the first FC layer 510 and the specific safety equipment wearing state information output from the second FC layer 520. It is possible to support the user terminal 200 to display the worker safety status information by generating worker safety status information indicating whether safety equipment corresponding to the work type information is worn, and transmitting the worker safety status information to the user terminal 200. .

For example, in order to install epoxy on the floor of a building, workers must wear a gas mask, and information on the wearing state of standard safety equipment may be set.

In the above situation, when a photorealistic image of the workers installing the epoxy without wearing a gas mask is acquired, the spatial state information determination server 100 performs the above-described image analysis processing, (i) Specific construction type information corresponding to the epoxy construction work and specific safety equipment wearing status information indicating the state of workers currently wearing safety equipment are output, and (ii) the standard safety set in response to the epoxy construction work by referring to the specific construction type information It is possible to check the equipment wearing state information, and (iii) compare the standard safety equipment wearing state information with the specific safety equipment wearing state information to generate worker safety state information indicating that the corresponding workers are not wearing gas masks.

Accordingly, it is possible to determine the state of wearing the safety equipment for each specific type of work without uniformly determining the state of wearing the safety equipment of workers.

Here, the convolutional layer 410, the first FC layer 510, and the second FC layer 520 are in a state in which training data is used to output an arbitrary spatial feature map and arbitrary spatial state information, respectively, The training video image-The training video image is a video image photographed in an arbitrary space-When is acquired, by the learning device, (i) the training video image is inputted to the convolution layer 410 to be converted to the convolution layer 410. The training video image is subjected to a convolution operation at least once so that a training spatial feature map corresponding to the training video image is output, and the training spatial feature map is input to the RPN 420 to allow the RPN 420 to perform a training spatial feature map. The learning proposal boxes corresponding to the region where the learning worker is located are output, and the areas on the learning spatial feature map corresponding to the learning proposal boxes are pooled by the pooling layer 430 to correspond to the pooled feature maps for learning. (Ii-1) The second feature vector for learning corresponding to the learning spatial feature map output from the convolution layer 410 is inputted to the first FC layer 510 and the first FC By allowing the layer 510 to calculate the second feature vector for learning, a neural network is used to determine a specific construction class corresponding to the space of the building on the learning image among preset construction classes, and then for learning corresponding to the specific construction class. Specific construction type information is output, and (ii-2) the first feature vector for learning is input to the second FC layer 520 to allow the second FC layer 520 to check the safety equipment worn by the learning worker for learning. (Iii-1) The first loss is generated by allowing the specific safety equipment wearing state information to be output, and (iii-1) the loss layer to refer to the specific work type information for learning and the corresponding first GT (Ground Truth), (iii-2) To the loss layer The second loss is generated by referring to the training specific safety equipment wearing state information and the corresponding second GT (Ground Truth), and by a back propagation process using the first loss and the second loss. At least one of parameters of the first FC layer 510, the second FC layer 520, and the convolution layer 410 may be in a learned state.

Meanwhile, as shown in the upper part of FIGS. 3 to 5, specific spatial information in which the user terminal 200 is located may be additionally displayed.

That is, in a state in which spatial information, which is information about individual spaces within the building specified by at least one of the building's building, building, floor, and room, is acquired, the spatial state information determination server 100 Among the spatial information, specific spatial information in which the user terminal 200 is located is checked with reference to the location information in which the user terminal 200 is located, and specific spatial information is transmitted to the user terminal 200 to cause the user terminal 200 It can support to display not only spatial state information about safety states, defects or defects, but also specific spatial information. Through this, even if it is a building with a complex structure, it is possible to accurately grasp where a specific type of construction, a specific worker, a specific defect or defect is located.

Meanwhile, the spatial state information determination server 100 may support the other user terminal 300 to display the spatial state information by transmitting the spatial state information to the other user terminal 300.

Here, the other user may include an indicator, a supervisor, etc., instructing a series of work to a worker in addition to a worker who works on a construction site or a building to be maintained.

In addition, the spatial state information determination server 100 may transmit not only the spatial state information, but also the live image image to the other user terminal 300, and from the other user terminal 300 to a specific region of the live image image. When the instruction information is obtained, the instruction information is transmitted to the user terminal 200 so that the user terminal 200 renders the markup corresponding to the instruction information, and displays the markup in a specific area of the live image. You can apply.

In this way, through the markup display, an instruction to perform a supplementary work for an inappropriate construction state in which a defect or defect has occurred, and an instruction on the work order for each type of work can be made.

Further, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks. media), and a hardware device specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

In the above, the present invention has been described by specific matters such as specific elements and limited embodiments and drawings, but this is provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , Anyone with ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention is limited to the above-described embodiments and should not be defined, and all modifications that are equally or equivalent to the claims as well as the claims to be described later fall within the scope of the spirit of the present invention. I would say.

Claims (20)

In a method of determining spatial state information of a building based on machine learning,
(a) When a real-life image representing the space of the building is acquired from the user terminal, the spatial state information determination server inputs the real-life image to the feature extraction block, causing the feature extraction block to analyze the real-life image and Extracting a feature on the real-life image and outputting a feature vector; And
(b) The spatial state information determination server inputs the feature vector as a spatial state classification block, and causes the spatial state classification block to calculate the feature vector by a neural network, so that the spatial state information of the building on the real-life image Outputting and transmitting the spatial state information to the user terminal to enable the user terminal to display the spatial state information;
Including,
The feature extraction block includes a convolution layer, an RPN, and a pooling layer,
The spatial state classification block includes a first FC layer and a second FC layer,
In step (a),
The spatial state information determination server generates a spatial feature map corresponding to the real image image by inputting the real image image to the convolution layer to cause the convolution layer to convolve the real image image at least once. Output, and input the spatial feature map to the RPN so that the RPN outputs proposal boxes corresponding to the area where the worker is on the spatial feature map, and causes the pooling layer to each of the proposal boxes The corresponding regions on the spatial feature map are pooled to output a first feature vector corresponding to the pooled feature maps,
In step (b),
The spatial state information determination server (i) inputs a second feature vector corresponding to the spatial feature map output from the convolutional layer to a first FC layer, and causes the first FC layer to generate the second feature vector. By performing a neural network operation, after determining a specific construction class corresponding to the space of the building on the real-life image among preset construction classes, the specific construction type information corresponding to the specific construction class is output, and ( ii) by inputting the first feature vector to the second FC layer, causing the second FC layer to check the safety equipment worn by the worker and output specific safety equipment wearing state information, (iii) the first Safety equipment in which the worker corresponds to the specific work type information by referring to reference safety equipment wearing state information set in response to the specific work type information output from the FC layer and the specific safety equipment wearing state information output from the second FC layer. And generating worker safety status information indicating whether or not the worker is wearing, and transmitting the worker safety status information to the user terminal, thereby supporting the user terminal to display the worker safety status information. delete delete delete delete delete The method of claim 1,
The convolutional layer, the first FC layer, and the second FC layer are in a state in which training data is used to output an arbitrary spatial feature map and arbitrary spatial state information, respectively,
When the training video image-the training video image is a video image photographed in an arbitrary space-is obtained, by the learning device, (i) the training video image is input to the convolution layer, causing the convolution layer The training video image is subjected to a convolution operation at least once so that a training spatial feature map corresponding to the training video image is output, and the training spatial feature map is input to the RPN to cause the RPN to learn on the training spatial feature map. The learning proposal boxes corresponding to the area where the operator is located are output, and the pooling layer causes the areas on the learning spatial feature map corresponding to the learning proposal boxes to be pooled to be used for learning corresponding to the pooled feature maps for learning. A first feature vector is output, and (ii-1) a second learning feature vector corresponding to the learning spatial feature map output from the convolution layer is input to the first FC layer, and the first FC layer causes the By allowing the second feature vector for learning to be computed with a neural network, a specific construction class corresponding to the space of the building on the learning image is determined among preset construction classes, and then a specific construction type for learning corresponding to the specific construction class. Information is output, and (ii-2) the first feature vector for learning is input to the second FC layer so that the second FC layer confirms the safety equipment worn by the learning worker to wear specific safety equipment for learning. The state information is output, (iii-1) the loss layer is made to refer to the specific work type information for learning and the first GT (Ground Truth) corresponding thereto to generate a first loss, and (iii- 2) A second loss is generated by allowing the loss layer to refer to the training specific safety equipment wearing state information and a corresponding second GT (Ground Truth). At least one of the parameters of the first FC layer, the second FC layer, and the convolution layer is learned by a back propagation process using the first loss and the second loss. Method, characterized in that in the state. The method of claim 1,
In a state in which spatial information, which is information about individual spaces within the building specified by at least one of the building's wing, arc, floor, and room, is obtained,
In step (b),
The spatial state information determination server checks the specific spatial information in which the user terminal is located among the spatial information by referring to the location information in which the user terminal is located, and transmits the specific spatial information to the user terminal, And supporting the user terminal to display the spatial state information and the specific spatial information. The method of claim 1,
In step (b),
And the spatial state information determination server transmits the spatial state information to another user terminal to support the other user terminal to display the spatial state information. The method of claim 9,
In step (b),
The spatial state information determination server transmits the spatial state information and the live-action image image to the other user terminal,
(c) When instruction information for a specific region among a certain region of the live-action image is obtained from the other user terminal, the spatial state information determination server transmits the instruction information to the user terminal to the user terminal. Rendering the markup corresponding to the indication information and supporting the markup to be displayed in the specific area of the real-life image;
The method further comprising a. In a spatial state information determination server that determines spatial state information of a building based on machine learning,
At least one memory for storing instructions; And
Comprising at least one processor configured to execute the instructions,
The processor includes: (1) When a real-life image representing a space of a building is obtained from a user terminal, the feature extraction block analyzes the real-life image by inputting the real-life image into a feature extraction block, A process of extracting a feature on the image and outputting a feature vector; And (2) inputting the feature vector as a spatial state classification block to cause the spatial state classification block to calculate the feature vector with a neural network to output spatial state information of the building on the real-life image, and the spatial state Executing the instructions for performing a process of supporting the user terminal to display the spatial state information by transmitting information to the user terminal,
The feature extraction block includes a convolution layer, an RPN, and a pooling layer,
The spatial state classification block includes a first FC layer and a second FC layer,
The processor,
In the above (1) process,
By inputting the real-life image image to the convolution layer, the convolutional layer causes the real-life image to be convolved at least once to output a spatial feature map corresponding to the real-life image, and the spatial feature map Input into the RPN to cause the RPN to output proposal boxes corresponding to an area in which a worker is located on the spatial feature map, and to cause the pooling layer to output an area on the spatial feature map corresponding to each of the proposal boxes. Are pooled to output a first feature vector corresponding to the pooled feature maps,
In the above (2) process,
(i) By inputting a second feature vector corresponding to the spatial feature map output from the convolutional layer into a first FC layer, and causing the first FC layer to calculate the second feature vector, a neural network is performed, Among the classes, a specific construction class corresponding to the space of the building on the real-life image is determined, and then specific construction type information corresponding to the specific construction class is output, and (ii) the first feature vector is By inputting to the second FC layer, the second FC layer checks the safety equipment worn by the worker and outputs specific safety equipment wearing state information, and (iii) the specific construction type output from the first FC layer. Worker safety status information indicating whether the worker has worn safety equipment corresponding to the specific work type information by referring to reference safety equipment wearing status information set in response to the information and the specific safety equipment wearing status information output from the second FC layer And, by transmitting the worker safety status information to the user terminal, the user terminal supports the display of the worker safety status information. delete delete delete delete delete The method of claim 11,
The convolutional layer, the first FC layer, and the second FC layer are in a state in which training data is used to output an arbitrary spatial feature map and arbitrary spatial state information, respectively,
When the training video image-the training video image is a video image photographed in an arbitrary space-is obtained, by the learning device, (i) the training video image is input to the convolution layer, causing the convolution layer The training video image is subjected to a convolution operation at least once so that a training spatial feature map corresponding to the training video image is output, and the training spatial feature map is input to the RPN to cause the RPN to learn on the training spatial feature map. The learning proposal boxes corresponding to the area where the operator is located are output, and the pooling layer causes the areas on the learning spatial feature map corresponding to the learning proposal boxes to be pooled to be used for learning corresponding to the pooled feature maps for learning. A first feature vector is output, and (ii-1) a second learning feature vector corresponding to the learning spatial feature map output from the convolution layer is input to the first FC layer, and the first FC layer causes the By allowing the second feature vector for learning to be computed with a neural network, a specific construction class corresponding to the space of the building on the learning image is determined among preset construction classes, and then a specific construction type for learning corresponding to the specific construction class. Information is output, and (ii-2) the first feature vector for learning is input to the second FC layer so that the second FC layer confirms the safety equipment worn by the learning worker to wear specific safety equipment for learning. The state information is output, (iii-1) the loss layer is made to refer to the specific work type information for learning and the first GT (Ground Truth) corresponding thereto to generate a first loss, and (iii- 2) Even if the second loss is generated by allowing the loss layer to refer to the training specific safety equipment wearing state information and the corresponding second GT (Ground Truth) And at least one of the parameters of the first FC layer, the second FC layer, and the convolution layer is learned by a back propagation process using the first loss and the second loss. Space status information determination server, characterized in that in the state. The method of claim 11,
In a state in which spatial information, which is information about individual spaces within the building specified by at least one of the building's wing, arc, floor, and room, is obtained,
The processor,
In the above (2) process,
Among the spatial information, specific spatial information in which the user terminal is located is checked with reference to the location information in which the user terminal is located, and the specific spatial information is transmitted to the user terminal, so that the user terminal causes the spatial state Spatial state information determination server, characterized in that supporting to display information and the specific spatial information. The method of claim 11,
The processor,
In the above (2) process,
The spatial state information determination server, characterized in that by transmitting the spatial state information to another user terminal to support the other user terminal to display the spatial state information. The method of claim 19,
The processor,
In the above (2) process,
Transmit the spatial state information and the live-action image to the other user terminal,
(3) When instruction information for a specific region among a certain region of the live-action image is obtained from the other user terminal, the spatial state information determination server transmits the instruction information to the user terminal to the user terminal. A process of rendering the markup corresponding to the indication information and supporting the display of the markup in the specific area of the live image;
Space state information determination server, characterized in that it further comprises.

Images