WO2024038551A1

WO2024038551A1 - Building interior structure recognition system and building interior structure recognition method

Info

Publication number: WO2024038551A1
Application number: PCT/JP2022/031247
Authority: WO
Inventors: 徹伊藤; 康文福間; ザイシンマオ; 央塚田
Original assignee: 株式会社 Sai
Priority date: 2022-08-18
Filing date: 2022-08-18
Publication date: 2024-02-22

Abstract

This building interior structure recognition system comprises: a first machine learning model generation unit that uses BIM data and a first site image to generate a first machine-learned model; a second machine learning model generation unit that uses a second site image containing an image of a noise component to carry out relearning with respect to the first machine-learned model; a scanning unit that scans the interior of a building while determining whether scanning of a structure inside the building is successful, and acquires a third site image and 3D point cloud data of a structure inside the building; a noise component removal unit that extracts an image of a noise component; a third machine learning model generation unit uses the image of the noise component to carry out relearning with respect to the second machine-learned model; a building interior structure recognition unit that uses the third machine-learned model to recognize a structure inside the building; and a point group data output unit that extracts and outputs point cloud data of the structure inside the building.

Description

Building structure recognition system and building structure recognition method

The present invention relates to an in-building structure recognition system and an in-building structure recognition method, and in particular, to an in-building structure recognition system that recognizes a structure placed inside a building such as a building using deep learning using a neural network. The present invention relates to a recognition system, a method and program for recognizing structures inside buildings.

Conventionally, methods for checking the construction status of buildings such as buildings that are currently under construction include using two-dimensional construction drawings, etc., and checking by direct measurement by humans using instruments etc. at the construction site, for example. Comparisons are being made with BIM (Building Information Modeling) models using remote sensing technology that can measure distance using reflected light, such as LiDER (Light Detection and Ranging).

However, when measuring using LiDER, etc., it is necessary to measure multiple locations at the construction site depending on the site situation based on experience, and there is a problem that the accuracy of the data obtained varies depending on the skill level of the measurer. was there. Further, there were problems in that it took time and effort to register the obtained point cloud data, and to manually identify structures in the building such as pipes and measure their positions and sizes. Additionally, there were problems with the accuracy of the captured point cloud data and the data processed from it, and the difficulty of reusing the data.

It is difficult to realistically adopt measurements at all points on the construction site with an emphasis on data accuracy because the amount of information would be enormous. If the measurer is highly skilled, it is possible to measure only the necessary points based on his or her own experience, but automation of measurement is required to prevent variations due to skill level and to improve measurement efficiency. .

When considering automating the identification of the area of structures placed at the construction site and the recognition of the structure in order to compare the construction status of the construction site in the middle of construction with the completed form. It is expected that trained models using deep learning using neural networks will be used for this purpose.

In order to create a trained model for automating the recognition of structures in images, a sufficient number of construction site images are required as input data for learning. Further, as the correct answer data for learning, an annotation for the structure included in the image, that is, the result of recognition of the structure in the image, indicating which part of the image is what, is required. However, it is difficult to collect a large number of photographic images of actual construction sites that can be used for learning as input data, and to annotate a huge number of structures for use as correct answer data.

Another idea is to perform machine learning and create a trained model using a rendered image of a completed 3D model of the construction site that closely resembles the actual appearance, rather than photos of the actual construction site. It will be done. However, rendered images are mainly created for commercial purposes of buildings, and their production costs are high, making it difficult to prepare a sufficient number of rendered images as learning images. Further, the work of annotating structures included in rendered images becomes enormous and requires time and effort to perform manually.

Therefore, it is required to be able to prepare a sufficient number of learning images related to construction sites for learning, and to automate the annotation of structures included in the learning images. Furthermore, the trained model created thereby is required to be able to recognize structures with high accuracy.

Furthermore, at actual construction sites, in addition to the structures to be measured, there are various items such as wire mesh, protective nets and sheets, temporarily installed iron fences and poles, trash, and materials. Additionally, people may be unintentionally captured during the scanning process at the site. These noise components interfere with the recognition of the structure that is the object of measurement, and affect the recognition accuracy of the structure.

Therefore, the system is capable of recognizing with high accuracy site images that include noise components such as wire mesh, protective nets and sheets, temporarily installed iron fences and poles, garbage, and materials existing at the construction site. is required. In addition, at actual sites, the existence status of such noise components changes from moment to moment as they are moved or added to suit the construction situation, so it is necessary to It is recommended to use a model. In this case, it is desirable to be able to minimize the time and cost of regenerating a model tailored to the site.

Non-Patent Document 1 discusses the problem of the huge amount of point cloud data in as-built modeling, which creates 3D models based on 3D measurements of existing large-scale equipment, and describes the problem of ``Measurement used in as-built modeling of large-scale equipment''. It should be noted that the measurement principle of this device is different from that of point cloud measurement devices for small parts.For point cloud measurement of small parts, triangulation is generally performed using a laser output device and a CCD camera. However, with this method, as the size of the object increases, the equipment becomes larger.Also, when measuring small parts, the measured point cloud is often only a few million points at most, but in the case of large equipment, It has been pointed out that "a large number of point clouds are required for modeling."

For example, in Patent Document 1, "digitized data of existing parts of a building acquired from existing drawings is converted into 3D CAD data, and created from point cloud data acquired by a 3D laser scanner or the point cloud data. Existing partial survey means is stored together with various field survey data including the 3D polygon model that has been constructed, and newly constructed part objects are selected from member objects stored in the member library in advance for the 3D polygon model. A construction member design means for arranging a member object, and the member object obtained by reading, with an ID reader, an electronic tag attached to a member precut at a parts factory according to the member object arranged by the construction member design means. a CPU functioning as a member construction position output means for searching and outputting a member object corresponding to a unique ID together with its construction position information from a three-dimensional CAD model designed by the construction member design means; and an automatic position pointing device that indicates the construction position of the member in the existing part based on the construction position information of the member object output by the member construction position output means.

Further, in Patent Document 2, "an image acquisition unit that acquires an input image generated by imaging a real space using an imaging device, and based on the position of one or more feature points reflected in the input image, a recognition unit that recognizes a relative position and orientation between the real space and the imaging device; an application unit that provides an augmented reality application using the recognized relative position and orientation; and the recognition unit. and a display control unit that superimposes a guiding object that guides a user operating the imaging device on the input image according to the distribution of the feature points so that the recognition processing performed by the image processing device is stabilized. is disclosed.

However, although Patent Documents 1 and 2 both disclose techniques for grasping a three-dimensional space or an object in a three-dimensional space, in particular, three-dimensional point clouds in large-scale facilities such as buildings and factories are used. It did not solve the problem of huge amounts of data, and it was not suitable for automating the recognition of structures in images in order to quickly understand the situation at a construction site in the middle of construction. .

In addition, both Patent Documents 1 and 2 also apply to site images that include noise components such as wire mesh, protective nets and sheets, temporarily installed iron fences and poles, garbage, and materials existing at the construction site. There was no consideration given to recognizing structures with high accuracy or retraining the model to match the latest site conditions.

Japanese Patent Application Publication No. 2013-149119 JP2013-225245A

Therefore, the present invention solves the above-mentioned problems, and also handles site images that include noise components such as wire mesh, protective nets and sheets, temporarily installed iron fences and poles, garbage, and materials existing at the construction site. The present invention provides an in-building structure recognition system and an in-building structure recognition method that can recognize target structures with high accuracy and can recognize structures with high precision in accordance with the latest site conditions. be.

The present invention also provides a program for causing a computer to execute each step of the method for recognizing structures inside a building.

In order to solve the above problems, the present invention provides an in-building structure recognition system for recognizing in-building structures using a machine learning model, which uses BIM (Building Information Modeling) data and a first site image. A first machine learning model generation unit that performs machine learning using a first machine learning model to generate a first machine learned model, and an image of a noise construct that does not have BIM data for the first machine learned model a second machine learning model generation unit that performs relearning using a second on-site image containing images to generate a second machine learned model; a scanning unit that scans the interior of the building and acquires three-dimensional point cloud data and a third on-site image; and a noise component that removes an image of the noise constituent from the third on-site image acquired by the scanning unit. Retraining the second machine learned model using the removing unit and the image from which the noise components obtained by the noise constituent removing unit have been removed to generate a third machine learned model; Obtained by a building structure recognition unit and a scanning unit that recognize structures in the building included in the third site image using the third machine learning model generation unit and the third machine learned model. and a point cloud data output unit that extracts and outputs point cloud data of structures in the building recognized by the building structure recognition unit from the three-dimensional point cloud data. Provides a structure recognition system.

In the intra-building structure recognition system according to an aspect of the present invention, the first machine learning model generation unit uses an image generated from BIM data as correct data and an image generated by rendering the BIM data as a first site image. The method is characterized in that machine learning is performed using an image obtained by processing using information obtained from observation data as observation data to generate a first machine learned model.

In the building structure recognition system according to an aspect of the present invention, the second machine learning model generation unit is configured to generate correct data and observation data for the second site image that includes an image of a noise component that does not have BIM data. The method is characterized in that the set is input to a first machine-learned model and re-learning is performed to generate a second machine-learned model.

In the building structure recognition system according to an aspect of the present invention, the scanning unit acquires images inside the building, and at least one corresponding reference point or reference structure exists between successive frames. If at least one corresponding reference point or reference structure does not exist, an alert prompting rescanning is notified.

In the building structure recognition system according to an aspect of the present invention, the noise component removal unit extracts an image of the noise construct from the third on-site image acquired by the scanning unit by stereo matching, and generates a mask image of the noise construct. The method is characterized in that it generates an image, reconstructs the image so as to interpolate a portion of the mask image, and generates an image from which noise components have been removed.

In the building structure recognition system according to an aspect of the present invention, the third machine learning model generation unit generates a correct answer for the image from which noise components have been removed from the third on-site image obtained by the noise component removal unit. The method is characterized in that a set of data and observed data is input to a second machine-learned model and re-learning is performed to generate a third machine-learned model.

In the in-building structure recognition system according to an aspect of the present invention, the in-building structure recognition unit inputs a third on-site image to a third machine-learned model to identify inside buildings included in the third on-site image. The feature is that it recognizes the structure of.

Further, the present invention provides an in-building structure recognition method for recognizing structures in a building using a machine learning model, which performs machine learning using BIM (Building Information Modeling) data and a first site image. and retraining the first machine learned model using a second site image that includes an image of a noise composition that does not have BIM data. scanning the building while determining the success or failure of scanning the structures inside the building, and generating 3D point cloud data of the structures inside the building and images inside the building. a scanning step of acquiring a noise component, a step of removing an image of the noise component from the image inside the building acquired by the scanning unit, and a second re-learning the machine learned model to generate a third machine learned model; recognizing structures in the building using the third machine learned model; and a scanning unit A method for recognizing a structure inside a building is provided, which includes a step of extracting and outputting point cloud data of a structure inside a building recognized by an inside structure recognition unit from the three-dimensional point cloud data acquired in the above.

In the method for recognizing structures inside a building according to an aspect of the present invention, the scanning step includes acquiring an image inside the building and determining whether or not there is at least one corresponding reference point or reference structure. The method is characterized in that an alert prompting rescanning is notified if at least one or more corresponding reference points or reference structures do not exist.

Furthermore, the present invention provides a program that causes a computer to execute each step of the above method for recognizing structures inside a building.

In the present invention, "BIM (Building Information Modeling) data" refers to data of a three-dimensional model of a building reproduced on a computer.

According to the present invention, it is possible to recognize with high accuracy even site images that include noise components such as wire mesh, protective nets and sheets, temporarily installed iron fences and poles, garbage, and materials existing at the construction site. It has the effect of being able to.

In addition, the accuracy of structure recognition can be improved by relearning the model to match the latest site conditions in response to noise components that change moment by moment.

Furthermore, it is possible to notify the user of the success or failure of scanning during on-site scanning, and to prompt the user to re-scan on the spot if the scanning is defective, thereby avoiding having to re-do the scanning work at a later date. can.
Other objects, features and advantages of the invention will become apparent from the following description of embodiments of the invention, taken in conjunction with the accompanying drawings.

FIG. 1 is a schematic diagram showing the entire structure recognition system in a building according to the present invention. FIG. 2 is a diagram showing the flow of each process of the building structure recognition system according to the present invention. FIG. 3 is a schematic diagram showing the first machine learning model generation section of the present invention. FIG. 4 is a schematic diagram showing the second machine learning model generation section of the present invention. FIG. 5 is a schematic diagram showing the scanning section of the present invention. FIG. 6 is a schematic diagram illustrating the noise construct removal section of the present invention. FIG. 7 is a schematic diagram showing the third machine learning model generation section of the present invention. FIG. 8 is a schematic diagram showing the intra-building structure recognition section of the present invention. FIG. 9 is a diagram showing the overall flow of the method for recognizing structures inside a building according to the present invention.

FIG. 1 is a schematic diagram showing the entire building structure recognition system 1 according to the present invention.
The in-building structure recognition system 1 according to the present invention includes a first machine learning model that performs machine learning using BIM (Building Information Modeling) data and a first site image to generate a first machine learned model. The generation unit 11 performs relearning on the first machine learned model M1 using a second site image including an image of a noise component that does not have BIM data, and generates a second machine learned model M2. A second machine learning model generation unit 12 that generates 3D point cloud data inside the building and a third on-site image are obtained by scanning the inside of the building while determining the success or failure of scanning the structures inside the building. From the third on-site image obtained by the scanning unit 20 and the noise constituent removal unit 30 and the noise constituent removal unit, which removes the image of the noise constituent from the image inside the building acquired by the scanning unit 20. a third machine learning model generation unit 13 that performs relearning on the second machine learned model using the image from which noise components have been removed, and generates a third machine learned model M3; The building structure recognition unit 40 uses the third machine learned model M3 to recognize the structures in the building, and the building structure recognition unit recognizes the structures from the three-dimensional point cloud data acquired by the scanning unit. and a point cloud data output unit 50 that extracts and outputs point cloud data of structures in the building.

The first machine learning model generation unit 11 performs machine learning using BIM (Building Information Modeling) data and the first site image to generate a first machine learned model. Specifically, the first machine learning model generation unit 11 uses the image generated from the BIM data as correct data, and processes the image generated by rendering the BIM data using the information obtained from the first site image. Machine learning is performed using the image obtained by this as observation data to generate a first machine learned model.

The second machine learning model generation unit 12 performs relearning on the first machine learned model M1 using a second site image including an image of a noise component that does not have BIM data, and A machine learned model M2 is generated. Specifically, the second machine learning model generation unit 12 generates a set of correct data and observed data for the second site image including an image of a noise component that does not have BIM data, using the first machine learning model generation unit 12. A second machine-learned model is generated by inputting the input into the trained model and performing re-learning.

The scanning unit 20 scans the inside of the building while determining the success or failure of scanning the structures inside the building, and acquires three-dimensional point cloud data inside the building and a third on-site image. The scanning unit 20 acquires an image inside the building, determines whether or not there is at least one corresponding reference point or reference structure, and determines whether there is at least one or more corresponding reference point or reference structure. If a structure does not exist, an alert will be sent prompting a rescan.

The noise constituent removal unit 30 extracts an image from which noise constituents have been removed from the image inside the building acquired by the scanning unit 20. The noise constituent removing unit 30 extracts an image of the noise constituent from the third scene image acquired by the scanning unit 20 by stereo matching, generates a mask image of the noise constituent, and interpolates the portion of the mask image. The image is reconstructed to generate an image from which noise components have been removed.

The third machine learning model generation unit 13 performs relearning on the second machine learning model using the image from which the noise components extracted by the noise component removal unit have been removed, and generates a third machine learning model. Generate trained model M3. Specifically, the third machine learning model generation unit 13 generates a set of correct data and observed data for the image from which noise components have been removed from the third on-site image, obtained by the noise component removal unit 30. , and performs relearning by inputting it into the second machine learned model to generate a third machine learned model.

The in-building structure recognition unit 40 recognizes the structures in the building using the third machine learned model M3. The intra-building structure recognition unit 40 inputs the third on-site image to the third machine-learned model and recognizes the intra-building structures included in the third on-site image.

The point cloud data output unit 50 extracts and outputs point cloud data of the structures inside the building recognized by the building structure recognition unit from the three-dimensional point cloud data acquired by the scanning unit.

FIG. 2 is a diagram showing the flow of each process of the building structure recognition system according to the present invention.
Figure 2 shows the relationships between machine learning model generation processing, data acquisition/scanning processing, noise component removal processing, and building structure recognition processing among the processes executed in the building structure recognition system. ing. The machine learning model generation process is executed by the first machine learning model generation unit 11, the second machine learning model generation unit 12, or the third machine learning model generation unit 13. The data acquisition/scanning process is executed by the scanning section 20 or the point cloud data output section 50. The pre-processing before scanning may be performed by any external imaging device or scanning device (not shown). The noise constituent removal process is executed by the noise constituent removal unit 30. The building structure recognition process is executed by the building structure recognition section 40.

The overall processing flow is roughly divided into pre-processing before scanning and processing after scanning. First, we will explain the preprocessing before performing actual scanning on site. The pre-processing before scanning includes generating a first machine learned model and relearning the first machine learned model to generate a second machine learned model. First, BIM data and a first site image for generating a first machine learned model are acquired. A first machine learned model is generated using the acquired BIM data and the first site image. The first machine learned model is created assuming an ideal site, and can be used universally for various sites. However, in the actual site, there may be items (wire mesh, protective nets and sheets, temporarily installed iron fences and poles, trash, materials, etc.) that are not included in the BIM data. In addition to such objects, unintended persons may also be reflected. Therefore, in order to recognize the structures inside the building, we acquired a second site image that includes such noise components that become noise, and added it to the first machine learned model in order to learn the noise components as noise. Re-learning will be performed. By performing such re-learning, a second machine-learned model matching the actual site situation is obtained. The second site image is acquired when inspecting the site before performing actual scanning at the site.

Next, we will explain the processing after the actual scanning at the site. Processing after scanning involves scanning, removing noise components, generating a third machine learned model, recognizing structures inside the building, and acquiring point cloud data. It includes doing. First, the inside of the building, which is the site, is scanned to obtain a third site image and three-dimensional point cloud data. At this time, while scanning is performed, the success or failure of scanning is determined, and if it is necessary to rescan, an alert is sent to prompt rescanning. If an alert prompting a rescan is received, a rescan will be performed. Repeat this until all objects in the building are scanned. Noise components are then removed from the scanned third scene image. Noise components present at the scene often change from moment to moment, and the third scene image may contain new noise components that are not included in the second scene image acquired through pre-processing before scanning. may be included. Further, noise components included in the second scene image may have been moved to another location. Unintended reflections of people can also be noise components. Therefore, in order to remove the noise components from the scanned image and further learn the image from which the noise components have been removed, relearning is performed on the second machine learned model. By performing such re-learning, a third machine-learned model matching the latest situation at the actual site is obtained. Next, using the third machine learned model, recognition of the structures inside the building is performed on the third site image. Finally, three-dimensional point cloud data corresponding to the recognized structures inside the building is obtained.

FIG. 3 is a schematic diagram showing the first machine learning model generation section of the present invention.
The first machine learning model generation unit 11 uses the image generated from the BIM data as correct data (correct image), and processes the image generated by rendering the BIM data using the information obtained from the first site image. Machine learning is performed using the image obtained by this as observation data (observed image) to generate a first machine learned model. The correct image generated from the BIM data shows the structure in the image in a way that distinguishes it from the background. The correct image may be one that is manually generated, such as by manually filling in parts of the structure within the image. The correct image may be, for example, a binarized image in which a structure part and a background part can be distinguished. Observation images are also generated from BIM data. First, a rendered image is generated by rendering BIM data. Next, using information such as textures extracted from the first site image, textures, etc. are added to the rendered image to generate an image that is more similar to the actual photograph of the site, and this is used as an observation image. . A machine learning model generation process is performed using such a set of the correct image and observed image to generate a first machine learned model M1. The first machine learned model M1 can be used as a general-purpose machine learned model for recognizing structures inside a building. In particular, when scanning a building in an ideal environment where no noise components exist and recognizing structures within the building, the first machine-learned model M1 can be used.

FIG. 4 is a schematic diagram showing the second machine learning model generation section of the present invention.
The second machine learning model generation unit 12 inputs a set of correct data and observation data regarding the noise component image that does not have BIM data included in the second site image into the first machine learned model M1. Then, relearning is performed to generate a second machine learned model M2. When generating the first machine learned model M1, since the correct image and observed image are generated based on BIM data and used for machine learning, the first machine learned model M1 is free of noise components. Suitable for scanning ideal building environments that do not exist. On the other hand, in actual sites, noise components that do not have BIM data often exist. Therefore, by additionally learning the second site image containing noise components to the first machine learned model M1, it is possible to accurately recognize structures even in the building environment including noise components. A second machine learned model M2 that is capable of is generated. A set of correct images and observed images to be used for relearning the first machine learned model M1 is generated from the second on-site image. The correct image of the second scene image shows the structure in the image so as to be distinguished from the background. The correct image of the second site image may be one that is manually generated, such as by manually filling in a portion of the structure in the image. The correct image of the second site image may be, for example, a binarized image in which a structure part and a background part can be distinguished. As the observed image of the second on-site image, the second on-site image including noise components may be used as is. Further, as the observed image of the second on-site image, an image obtained by pre-processing the second on-site image including noise components as necessary may be used. Using such a set of the correct image of the second on-site image and the observed image, a relearning process is performed on the first machine learned model M1 to generate a second machine learned model M2.

FIG. 5 is a schematic diagram showing the scanning section of the present invention.
The scanning unit 20 acquires an image inside the building, determines whether or not there is at least one corresponding reference point or reference structure, and determines whether there is at least one or more corresponding reference point or reference structure. If a structure does not exist, an alert will be sent prompting a rescan. The scanning unit 20 may include a scanning success/failure determination unit 203, an alert notification unit 204, and a rescan processing unit 205 for each function.

The scanning success/failure determination unit 203 determines whether there is at least one corresponding reference point or reference structure. Here, the "reference point" is a point that serves as a reference for matching consecutive frames, and for example, a marker or the like may be attached to a structure in a building to serve as the reference point. In addition, a "reference structure" is a structure that serves as a reference for matching consecutive frames, and for example, a structure that has a straight part, such as a corner of a column, is a structure that serves as a reference. You may also do so.

The alert notification unit 204 notifies an alert prompting rescanning when at least one corresponding reference point or reference structure does not exist. The alert is for notifying the user that rescanning is necessary, and includes the display of an icon or message on the display screen of a scanning device such as the distance measuring scanner 201 or the imaging device 202, a warning sound, etc. It may be.

The rescan processing unit 205 receives a rescan instruction from the user and performs the rescan.

The processing in the scanning success/failure determination unit 203, alert notification unit 204, and rescan processing unit 205 is repeated until all scanning of the scan target is completed, and when the scanning is completed, the third site image and the three-dimensional point cloud data are is obtained. The third on-site image and three-dimensional point cloud data acquired by the scanning unit 20 may include information on noise components.

FIG. 6 is a schematic diagram illustrating the noise construct removal section of the present invention.
The noise constituent removing unit 30 extracts an image of the noise constituent from the third scene image acquired by the scanning unit 20 by stereo matching, generates a mask image of the noise constituent, and interpolates the portion of the mask image. The image is reconstructed to generate an image from which noise components have been removed. The noise component removal unit 30 may include a stereo matching unit 301, a mask image generation unit 302, and an image reconstruction unit 303 for each function.

The stereo matching unit 301 adds two images of the same object taken from different viewpoints (typical (right image and left image) are input, three-dimensional depth is estimated for each pixel, and a distance image is generated in which the estimated depth for each pixel is represented by a gradation. Existing machine learned models for performing stereo matching, for example, calculate the parallax that represents the disparity between two images of the same object taken from different viewpoints (typically a right image and a left image). It may be a machine-learned model in which a mapping function for obtaining an image is learned using a convolutional neural network (CNN). Additionally, existing machine learned models for performing stereo matching may use recursive refinement to update disparity from coarse to fine, or a hierarchical network with a layered cascade architecture for inference.

In another aspect, the stereo matching unit 301 may use an existing stereo matching method that does not use a machine learning model instead of the above method that uses an existing machine learned model for stereo matching. In this case, for example, the three-dimensional depth of each pixel is estimated using two images (typically the right image and the left image) taken from two points, and the estimated depth of each pixel is A distance image represented by a gradation may be generated.

The mask image generation unit 302 performs threshold processing on the distance image generated by the stereo matching unit 301 to generate a mask image of the noise component. For example, if a wire mesh that is a noise component exists in front of a structure to be recognized, a mask image of the wire mesh portion that is a noise component is generated.

The image reconstruction unit 303 removes the mask portion of the mask image generated by the mask image generation unit 302 from the original image, and obtains a reconstructed image by interpolating the mask image portion. The image reconstruction unit 303 adds the third on-site image containing noise components and the mask image of the noise components generated by the mask image generation unit 302 to an existing machine learned model for image reconstruction. A reconstructed image may be obtained such that regions of the mask image are removed and portions of the mask image are interpolated. The existing machine learned model for reconstructing an image may be an existing machine learned model generated by deep learning using a neural network. Thereby, the image reconstruction unit 303 obtains an image from which noise components have been removed from the third on-site image.

FIG. 7 is a schematic diagram showing the third machine learning model generation section of the present invention.
The third machine learning model generation unit 13 generates a set of correct data and observed data for the image from which noise components have been removed from the image inside the building extracted by the noise component removal unit 30. The input is input to the trained model and relearning is performed to generate a third machine learned model. When generating the second machine-learned model M2, the correct image of the second on-site image containing noise components and the observed image obtained in pre-processing before scanning are used for re-learning. However, in order to accommodate new noise components that were not present at the time of pre-processing before scanning, noise components were removed from the third field image that contained the noise components and was acquired during the actual scanning. By additionally learning the images, a third machine-learned model M3 is generated that is capable of recognizing structures further suited to the target site. The set of correct images and observed images to be used for relearning the second machine learned model M2 is generated from the image obtained by the noise component removal unit 30 from which noise components have been removed from the third on-site image. be done. The correct image of the image from which the noise components have been removed is one that shows the structures in the image in a way that distinguishes them from the background. The correct image of the image from which the noise components have been removed may be one that is manually generated, for example, by manually filling in portions of the structure in the image. The correct image of the image from which noise components have been removed may be, for example, a binarized image in which a structure part and a background part can be distinguished. As the observed image of the image from which the noise components have been removed, the image from which the noise components have been removed from the third on-site image containing the noise components may be used as is. In addition, the observed image of the image from which the noise components were removed was processed by preprocessing as necessary on the image from which the noise components were removed from the third on-site image containing the noise components. Images may also be used. Using the set of the correct image and observed image from which such noise components have been removed, relearning processing is performed on the second machine learned model M2, and the third machine learned model M3 is generate.

FIG. 8 is a schematic diagram showing an intra-building structure recognition section and a point cloud data output section of the present invention.
The intra-building structure recognition unit 40 inputs the third on-site image to the third machine-learned model M3 and recognizes the intra-building structures included in the third on-site image. The output from the third machine learned model M3 shows the structure inside the building in the image so as to distinguish it from the background. The output from the third machine-learned model M3 may be a binarized image that can distinguish between the components inside the building and the background. The third scene image may include noise constructs that were present during the scanning of the scene. The third machine learned model M3 is obtained by additionally learning an image in which noise components are removed from a third on-site image that includes noise components, and when the third on-site image includes noise components, Structures inside a building can be recognized with high accuracy even when

The point cloud data output unit 50 is a scanning unit that generates three-dimensional point cloud data of a portion corresponding to the recognized structure in the building based on the image information of the structure in the building recognized by the structure recognition unit 40 in the building. It is extracted from the three-dimensional point cloud data scanned in step 20 and output as point cloud data. As a result, 3D point cloud data of the structures inside the building is obtained, and by performing rendering etc. on the obtained 3D point cloud data, it is used to generate 3D CAD data of the inside of the building including the structure, etc. It becomes possible to use it for.

A method for recognizing structures inside a building using a machine learning model will be described below.
FIG. 9 is a diagram showing the overall flow of the method for recognizing structures inside a building according to the present invention.
The indoor structure recognition method according to the present invention includes step S901 of performing machine learning using BIM (Building Information Modeling) data and a first site image to generate a first machine learned model; Step S902 of performing re-learning on the learned model M1 using a second site image including images of noise components that do not have BIM data to generate a second machine-learned model M2; A scanning step S903 in which the building is scanned while determining the success or failure of scanning of the structure in the building, and three-dimensional point cloud data of the structure in the building and an image inside the building are obtained; Step S904 of removing the image of the noise component from the image of Step S905 of generating the third machine learned model M3; Step S906 of recognizing structures in the building using the third machine learned model M3; and S906, from the three-dimensional point cloud data acquired by the scanning unit. An in-building structure recognition method is provided, which includes step S907 of extracting and outputting point cloud data of an in-building structure recognized by an in-building structure recognition unit.

In the method for recognizing structures inside a building according to an aspect of the present invention, the scanning step S903 acquires an image inside the building and determines whether there is at least one corresponding reference point or reference structure. If at least one corresponding reference point or reference structure does not exist, an alert prompting rescanning is sent.

Hereinafter, as a second embodiment, an example in which the indoor structure recognition system 1 described in the above embodiment 1 does not include the second machine learning model generation unit 12, and an example in which the indoor structure recognition method described in the above embodiment 1 is , an example that does not include the step of generating a second machine learned model will be described. Points not particularly described below are the same as the building structure recognition system 1 of the first embodiment.

An in-building structure recognition system for recognizing structures in a building using a machine learning model in Example 2 performs machine learning using BIM (Building Information Modeling) data and a first site image, and The first machine learning model generation unit 11 generates the first machine learned model M1, scans the building while determining the success or failure of scanning the structures in the building, and generates three-dimensional point cloud data and data in the building. A scanning section 20 that acquires a third on-site image; a noise component removal section 30 that removes an image of noise components from the image inside the building acquired by the scanning section 20; a third machine learning model generation unit 13 that performs relearning on the first machine learned model M1 using the image from which noise components have been removed, and generates a third machine learned model M3; The building structure recognition unit 40 recognizes the structures in the building using the third machine learned model M3, and the three-dimensional point cloud data acquired by the scanning unit 20 recognizes the structures in the building. The present invention is characterized by comprising a point cloud data output section 50 that extracts and outputs point cloud data of the structures in the building recognized by the section 40.

The difference from the building structure recognition system 1 of Example 1 is that the first machine-learned model is re-trained using a second site image that includes images of noise components that do not have BIM data. It does not include a second machine learning model generation unit 12 that generates a second machine learned model. In addition, in the second embodiment, the third machine learning model generation unit 13 performs relearning on the first machine learned model M1 instead of the second machine learned model M2, and Generate trained model M3. That is, in the second embodiment, the third machine learning model generation unit 13 calculates the difference between the correct data and the observed data for the image from which the noise components extracted from the building image extracted by the noise component removal unit 30 are removed. The set is input to the first machine learned model M1 and relearning is performed to generate the third machine learned model M3.

A set of correct images and observed images to be used for relearning the first machine-learned model M1 is generated from an image obtained by the noise component removal unit 30 from which noise components have been removed from the third on-site image. be done. The correct image of the image from which the noise components have been removed is one that shows the structures in the image in a way that distinguishes them from the background. The correct image of the image from which the noise components have been removed may be one that is manually generated, for example, by manually filling in portions of the structure in the image. The correct image of the image from which noise components have been removed may be, for example, a binarized image in which a structure part and a background part can be distinguished. As the observed image of the image from which the noise components have been removed, the image from which the noise components have been removed from the third on-site image containing the noise components may be used as is. In addition, the observed image of the image from which the noise components were removed was processed by preprocessing as necessary on the image from which the noise components were removed from the third on-site image containing the noise components. Images may also be used. Using a set of the correct image from which such noise components have been removed and the observed image, relearning processing is performed on the first machine learned model M1, and the third machine learned model M3 is generate.

The in-building structure recognition method for recognizing structures in a building using a machine learning model in Example 2 is the in-building structure recognition method for recognizing structures in a building using a machine learning model. A step S901 of performing machine learning using BIM (Building Information Modeling) data and a first site image to generate a first machine learned model, and determining the success or failure of scanning the structure in the building. scanning step S903, in which the inside of the building is scanned to obtain three-dimensional point cloud data of structures inside the building and an image inside the building, and an image of noise components is acquired from the inside image acquired by the scanning unit. Step S905 of performing re-learning on the first learned model to generate a third machine-learned model using the removing step S904 and the image from which the noise components obtained in the removing step have been removed. Step S906 recognizes the structures inside the building using the third machine learned model, and from the three-dimensional point cloud data acquired by the scanning section, Step S907 of extracting and outputting point cloud data of the structure.

The difference from the building structure recognition method of Example 1 is that the first machine learned model is retrained using a second site image that includes images of noise components that do not have BIM data. , does not include step S902 of generating the second machine learned model. In addition, in the second embodiment, step S905 of generating the third machine learned model performs relearning on the first machine learned model M1 instead of the second machine learned model M2, and 3 machine learned model M3 is generated. That is, in Example 2, the set of correct data and observed data for the image from which noise components have been removed from the third on-site image containing noise components obtained by the noise component removal unit 30 is It is input to the machine learned model M1 and relearning is performed to generate a third machine learned model M3.

According to the building structure recognition system and the building structure recognition method according to the present invention described above, it becomes possible to measure the shape and position of noteworthy members at a construction site, thereby improving accuracy and speed. can be improved. Furthermore, the amount of components to be managed at a construction site can be reduced, and accordingly, the amount of data handled by the construction site component management system can be significantly reduced. In addition, according to the building structure recognition system and building structure recognition method according to the present invention, wire mesh, protective nets and sheets, temporarily installed iron fences and poles, garbage, materials, etc. existing at the construction site can be used. Even on-site images containing noise components can be recognized with high accuracy. In addition, the accuracy of structure recognition can be improved by relearning the model to match the latest site conditions in response to noise components that change moment by moment. Furthermore, it is possible to notify the user of the success or failure of scanning during on-site scanning, and to prompt the user to re-scan on the spot if the scanning is defective, thereby avoiding having to re-do the scanning work at a later date. can. As described above, according to the present invention, it is possible to minimize the time and cost for regenerating a model tailored to the actual site by responding to various situations unique to the site. .
Although the above description has been made regarding the embodiments, it will be apparent to those skilled in the art that the present invention is not limited thereto, and that various changes and modifications can be made within the scope of the principles of the present invention and the appended claims.

1 Building structure recognition system 11 First machine learning model generation section 12 Second machine learning model generation section 13 Third machine learning model generation section 20 Scanning section 30 Noise component removal section 40 Building structure recognition section 50 Point cloud data output section

Claims

An in-building structure recognition system for recognizing in-building structures using a machine learning model, the system comprising:
a first machine learning model generation unit that performs machine learning using BIM (Building Information Modeling) data and the first site image to generate a first machine learned model;
re-learning the first machine-learned model using a second site image including an image of a noise component that does not have BIM data to generate a second machine-learned model; a machine learning model generation unit,
a scanning unit that scans the inside of the building while determining the success or failure of scanning the structure inside the building, and acquires three-dimensional point cloud data and a third on-site image inside the building;
a noise component removal unit that removes an image of noise components from the third on-site image acquired by the scanning unit;
relearning the second machine learned model using the image from which the noise constituents obtained by the noise constituent removal unit have been removed to generate a third machine learned model; 3 machine learning model generation unit,
an in-building structure recognition unit that recognizes structures in the building using the third machine-learned model;
and a point cloud data output unit that extracts and outputs point cloud data of the structure inside the building recognized by the building structure recognition unit from the three-dimensional point cloud data acquired by the scanning unit. A building structure recognition system characterized by:
An in-building structure recognition system for recognizing in-building structures using a machine learning model, the system comprising:
a first machine learning model generation unit that performs machine learning using BIM (Building Information Modeling) data and the first site image to generate a first machine learned model;
a scanning unit that scans the inside of the building while determining the success or failure of scanning the structure inside the building, and acquires three-dimensional point cloud data and a third on-site image inside the building;
a noise component removal unit that removes an image of noise components from the third on-site image acquired by the scanning unit;
re-learning the first machine-learned model using the image from which the noise constituents obtained by the noise constituent removal unit have been removed to generate a third machine-learned model; 3 machine learning model generation unit,
an in-building structure recognition unit that recognizes structures in the building using the third machine-learned model;
and a point cloud data output unit that extracts and outputs point cloud data of the structure inside the building recognized by the building structure recognition unit from the three-dimensional point cloud data acquired by the scanning unit. A building structure recognition system characterized by:
The first machine learning model generation unit may process the image generated from the BIM data as correct data and the image generated by rendering the BIM data using information obtained from the first site image. 3. The building structure recognition system according to claim 1, wherein a first machine learned model is generated by performing machine learning using images obtained by the above as observation data.
The second machine learning model generation unit generates a set of correct data and observation data for the second site image, which includes an image of a noise component that does not have the BIM data, into the first machine learning model. The building structure recognition system according to claim 1, wherein the second machine learned model is generated by inputting and relearning.
The scanning unit acquires an image inside the building, and determines whether there is at least one corresponding reference point or reference structure between consecutive frames, and 3. The in-building structure recognition system according to claim 1, wherein an alert prompting rescanning is provided when there is no corresponding reference point or reference structure.
The noise component removing unit extracts an image of the noise component from the third on-site image acquired by the scanning unit by stereo matching, generates a mask image of the noise component, and removes a portion of the mask image. 3. The building structure recognition system according to claim 1, wherein the system reconstructs the image so as to interpolate the noise components and generates an image from which the noise components are removed.
The third machine learning model generation section generates a set of correct data and observation data for the image from which the noise components have been removed from the third on-site image obtained by the noise component removal section. 2. The building structure recognition system according to claim 1, wherein the third machine-learned model is generated by inputting the second machine-learned model and performing re-learning.
The third machine learning model generation unit generates a set of correct data and observed data for the image from which the noise constituents have been removed from the third on-site image in the noise constituent removal unit. 3. The building structure recognition system according to claim 2, wherein a third machine-learned model is generated by inputting the information into a machine-learned model and performing re-learning.
The in-building structure recognition unit inputs the third on-site image to the third machine-learned model and recognizes the structures in the building included in the third on-site image. The building structure recognition system according to claim 1 or 2.
An in-building structure recognition method for recognizing in-building structures using a machine learning model, the method comprising:
performing machine learning using BIM (Building Information Modeling) data and the first site image to generate a first machine learned model;
Re-learning the first machine-learned model using a second site image including an image of a noise component that does not have BIM data to generate a second machine-learned model;
a scanning step of scanning the building while determining the success or failure of scanning the structure inside the building, and acquiring three-dimensional point cloud data of the structure inside the building and a third on-site image;
removing an image of noise components from the image inside the building obtained in the scanning step;
retraining the second machine learned model using the image from which the noise components obtained in the removing step are removed to generate a third machine learned model;
Recognizing structures within the building using the third machine learned model;
A method for recognizing a structure in a building, the method comprising: extracting and outputting point cloud data of the structure in the building recognized in the recognizing step from the three-dimensional point cloud data acquired in the scanning step. .
The scanning step includes acquiring an image inside the building, determining whether there is at least one corresponding reference point or reference structure between successive frames, and determining whether or not there is at least one corresponding reference point or reference structure between successive frames; 11. The method for recognizing a structure in a building according to claim 10, further comprising: notifying an alert prompting rescanning when the corresponding reference point or reference structure does not exist.
An in-building structure recognition method for recognizing in-building structures using a machine learning model, the method comprising:
performing machine learning using BIM (Building Information Modeling) data and the first site image to generate a first machine learned model;
a scanning step of scanning the building while determining the success or failure of scanning the structure inside the building, and acquiring three-dimensional point cloud data of the structure inside the building and an image inside the building;
removing an image of noise components from the image inside the building obtained in the scanning step;
Using the image from which the noise components obtained in the removing step have been removed, the first machine learned model is retrained to generate a third machine learned model;
Recognizing structures within the building using the third machine learned model;
A method for recognizing a structure in a building, the method comprising: extracting and outputting point cloud data of the structure in the building recognized in the recognizing step from the three-dimensional point cloud data acquired in the scanning step. .
The scanning step acquires an image inside the building, determines whether there is at least one corresponding reference point or reference structure between successive frames, and determines whether or not there is at least one corresponding reference point or reference structure between successive frames. 13. The method for recognizing a structure in a building according to claim 12, further comprising: notifying an alert prompting rescanning when there is no corresponding reference point or reference structure.
A program for causing a computer to execute each step of the method according to any one of claims 10 to 13.