JP7274782B1 - STRUCTURE RECOGNITION SYSTEM IN BUILDING AND STRUCTURE RECOGNITION METHOD IN BUILDING - Google Patents

Abstract

An object of the present invention is to provide an in-building structure recognition system capable of recognizing structures in a building with high accuracy even at a construction site where there are a large number of similar feature points, such as a construction site in the middle of construction. . A building structure recognition system according to the present invention includes at least one ranging sensor, at least one imaging device, and a SLAM (Simultaneous Localization and Mapping) measuring device, the SLAM measuring device comprising: A reference recognition unit that recognizes a reference member using a first learned model, a self-position recognition unit that recognizes the self-position of the imaging device using a second learned model, and a third learned model. and an object-to-be-measured recognition unit that recognizes the object to be measured in the building using the [Selection drawing] Fig. 1

Description

TECHNICAL FIELD The present invention relates to an intra-building structure recognition system, and more particularly, an intra-building structure recognition system and an intra-building structure that recognize structures placed inside a building such as a building using deep learning using a neural network. It relates to recognition methods.

SLAM (Simultaneous Localization and Mapping) is a technology that performs self-location estimation and environment map creation at the same time. At the same time, it estimates its own position within the environment map.

SLAM technology is also expected to be used for sensing the environment inside a building and obtaining data on structures inside the building in the construction field. In the method using mobile SLAM technology, the inside of the building is continuously measured while moving the camera and 3D scanner within the building, and the image information obtained from the camera and the 3D information obtained from the 3D scanner are continuously collected. to obtain. By linking the obtained continuous information in real time or after all the data measurement work is completed, the three-dimensional space inside the building and the structures arranged therein can be grasped.

However, when the conventional mobile SLAM technology is used to grasp the situation of the construction site during construction, there is a problem that it is difficult to obtain the expected accuracy. At construction sites in the middle of construction, a large number of components are arranged in an intricate manner, so similar patterns may continue, and unpredictable movement due to poor scaffolding may cause a misalignment of the line of sight. As a result, erroneous recognition is likely to occur due to the lack of matching. Once such an erroneous recognition occurs, the next recognition is performed based on the erroneously recognized point, so the influence of the deviation and the erroneous recognition increases one after another, making it difficult to return to the original correct state.

Another conventional method for sensing the environment inside a building is to perform all-sky measurements at multiple fixed points, and connect the measurement data at each fixed point after the data measurement work at all fixed points is completed. be.

However, all-sky measurement requires knowledge of surveying, and in order to obtain higher accuracy, it is necessary to rely on the experience of experts. In addition, there is a problem that the measurement of the whole sky type takes time, and the post-processing after the measurement also takes time and effort.

Therefore, there is a demand for an in-building structure recognition system that can solve the problem of accuracy due to erroneous recognition while taking advantage of the mobile SLAM technology that enables measurement in a short period of time.

As an image processing device for recognizing a real space using SLAM technology, for example, Patent Document 1 describes an image acquisition unit that acquires an input image generated by capturing an image of a real space using an imaging device; a recognition unit that recognizes a relative position and orientation between the real space and the imaging device based on the positions of one or more feature points appearing in the input image; and the recognized relative position. and an application unit that provides an augmented reality application using posture; and a guidance object that guides a user operating the imaging device according to the distribution of the feature points so that recognition processing performed by the recognition unit is stabilized. and a display control unit that superimposes on the input image” is disclosed.

In addition, as a method of acquiring data inside a building under construction using a camera and displaying a composite image, for example, in Patent Document 2, "a composite image display system composed of a mobile imaging terminal and a data server A three-dimensional computer graphics image of a building under construction is written in advance and stored in the data server, and the portable imaging terminal captures an image of the interior of the building and outputs it as a captured image. a display unit for displaying the captured image; coordinates of the same feature point in the captured image and a virtual image captured corresponding to the position and the capturing direction of the captured image in the computer graphic image; a position control unit that compares positions, and performs coordinate transformation in which the coordinate positions of the feature points are superimposed on the basis of the comparison result of the coordinate positions of the feature points, thereby superimposing the virtual image on the captured image and displaying the virtual image. A composite image display system is disclosed, which includes an image processing unit for displaying in a unit.

However, both Patent Documents 1 and 2 are based on feature points, and they function sufficiently in construction sites where there are a large number of similar feature points, such as construction sites in the middle of construction. It wasn't.

JP 2013-225245 A JP 2014-2645 A

Therefore, the present invention solves the above problems, and can recognize structures in a building with high accuracy even at a construction site where there are a large number of similar feature points, such as a construction site in the middle of construction. A building structure recognition system and a building structure recognition method are provided.

In addition, in the present invention, AI does not simply extract feature points from an image, but recognizes what can be used as a reference, connects a lot of measurement image data based on that, and grasps and measures the entire interior of the building. An object of the present invention is to provide an in-building structure recognition system and an in-building structure recognition method.

In order to solve the above problems, the present invention includes at least one ranging sensor, at least one imaging device, and a SLAM (Simultaneous Localization and Mapping) measuring device, wherein the SLAM measuring device is a first trained model A reference recognition unit that recognizes the reference member using the second learned model, a self-position recognition unit that recognizes the self-position of the imaging device using the second learned model, and a third learned model inside the building Provided is an in-building structure recognition system characterized by comprising an object-to-be-measured recognition unit for recognizing an object to be measured.
In the present invention, an AI that has been trained to recognize what can be used as a reference at a construction site recognizes what should be used as a reference at the construction site, and performs coordinate transformation in which the coordinate positions overlap based on the position of the object that should be used as a reference. Thus, by superimposing a plurality of captured images, the entire data inside the building under construction is obtained.

In the building structure recognition system according to one aspect of the present invention, the reference recognition unit selects a point or area that can be used as a reference for self-position estimation and environment map creation in SLAM among the data obtained from the ranging sensor. Recognize as a reference member.

In the building structure recognition system according to one aspect of the present invention, the first trained model is generated by rendering the BIM data using a correct image generated from BIM (Building Information Modeling) data as the correct data. It is generated by performing machine learning using virtual observation images as observation data.

In the system for recognizing structures in a building according to one aspect of the present invention, the reference recognition unit recognizes a reference member using image data obtained from an imaging device and distance data obtained from a range sensor as input data, and recognizes a reference member. Three-dimensional feature data of the reference member of the member is output as output data.

In the intra-building structure recognition system according to one aspect of the present invention, the self-position recognition unit sequentially recognizes the self-position of the imaging device when the imaging device moves while imaging the object to be measured.

In the building structure recognition system according to one aspect of the present invention, the second trained model uses the current three-dimensional coordinates and the attitude angle of the imaging device obtained from the BIM data as correct data, a) current and past It is generated by performing machine learning using virtual observed images, b) current and past virtual reference features, and c) past three-dimensional coordinates and attitude angles of the imaging device as observed data.

In the building structure recognition system according to an aspect of the present invention, the self-position recognition unit includes a) image data obtained from the imaging device, b) three-dimensional feature data of the reference member output from the reference recognition unit, c) By inputting the past three-dimensional coordinates and the attitude angle of the imaging device as input data into the second trained model, the self-position of the imaging device is recognized, and the current three-dimensional coordinates and the attitude angle of the imaging device are recognized. as the output data.

In the in-building structure recognition system according to one aspect of the present invention, the object recognition unit recognizes structures and objects in the building as objects to be measured.

In the building structure recognition system according to one aspect of the present invention, the third trained model renders the BIM data using the correct image generated from the BIM data and the data indicating the type of the object to be measured as correct data. It is generated by performing machine learning using the virtual observation image generated by and the virtual range image generated from the BIM data as observation data.

In the building structure recognition system according to one aspect of the present invention, the measured object recognition unit inputs image data obtained from the imaging device and distance data obtained from the range sensor into the third trained model. By doing so, the object to be measured is recognized, and the type of the object to be measured and the shape data of the object to be measured are output as output data.

The object recognition unit recognizes the object to be measured from the shape data of the object to be measured in the camera coordinate system of the image pickup device and the current three-dimensional coordinates and attitude angle of the image pickup device output from the self-position recognition unit. Calculate the self-position in the spatial coordinate system inside the building.

In the building structure recognition system according to an aspect of the present invention, the ranging sensors include a ToF (Time Of Flight) camera, a ToF sensor, a LiDAR (Light Detection And Ranging) sensor, a RADAR (RADio Detection And Ranging) sensor, an ultrasonic wave It may be a sensor or an infrared sensor.

Further, in the present invention, the step of recognizing the reference member using the first trained model, the step of recognizing the self-position of the imaging device using the second trained model, and the step of recognizing the self-position of the imaging device using the third trained model. and recognizing the object to be measured using the method.

Further, the present invention provides a program characterized by causing a computer to execute each step of the above-described indoor structure recognition method.

In the present invention, "BIM data" refers to data of a three-dimensional model of a building reproduced on a computer. BIM data generally includes information on the three-dimensional structure of a building, and also considers building materials for each part as an object. Information other than drawings can be included. By rendering BIM data, an image of the three-dimensional space can be obtained.

When recognizing structures in images of a building site, it is not enough to simply extract feature points. It is necessary to recognize objects or parts that can be used as references, and connect multiple measurement images based on those objects. , connect. This is similar to the work that humans do in an actual site by measuring the relative positions of many things from a good reference, such as a pillar, and grasping the whole.
According to the present invention, it is possible to use AI to achieve the same effects as the work performed by humans based on their own knowledge, thereby improving the efficiency of the work and improving the structure of the building. can be realized with high accuracy.
Other objects, features and advantages of the present invention will become apparent from the following description of embodiments of the invention with reference to the accompanying drawings.

FIG. 1 is a schematic diagram showing the entire structure recognition system in a building according to the present invention. FIG. 2 is a diagram illustrating generation of a first trained model according to the present invention. FIG. 3 is a diagram showing a reference recognition section of the SLAM measuring device according to the present invention. FIG. 4 is a diagram illustrating generation of a second trained model according to the present invention. FIG. 5 is a diagram showing the self-position recognition unit of the SLAM measuring device according to the present invention. FIG. 6 is a diagram illustrating generation of a third trained model according to the present invention. FIG. 7 is a diagram showing the object recognition unit of the SLAM measurement device according to the present invention. FIG. 8 is a flow chart showing the flow of reference recognition processing in the reference recognition section according to the present invention. FIG. 9 is a flow chart showing the flow of self-location recognition processing in the self-location recognition section according to the present invention. FIG. 10 is a flow chart showing the flow of object recognition processing in the object recognition unit according to the present invention.

FIG. 1 is a schematic diagram showing the entire structure recognition system 1 in a building according to the present invention.
A building structure recognition system 1 according to the present invention includes at least one distance measuring sensor 20, at least one imaging device 30, and a SLAM (Simultaneous Localization and Mapping) measurement device 10. The SLAM measurement device 10 includes: A reference recognition unit 101 that recognizes the reference member using the first trained model M1, a self-position recognition unit 102 that recognizes the self-position of the imaging device using the second trained model M2, and a third and an object-to-be-measured recognition unit 103 that recognizes the object to be measured in the building using the learned model M3.

The ranging sensor 20 is used to measure the distance between a structure or object in the building and the ranging sensor 20 . The ranging sensor 20 is preferably a ToF (Time Of Flight) camera, but is not limited thereto. It may be any ranging sensor such as a sensor.

The imaging device 30 is used to acquire an image inside the building. The imaging device 30 is preferably a 360-degree camera, but is not limited to this, and may be any imaging means such as a omnidirectional camera, a hemispherical camera, or a CCD camera mounted on a mobile terminal.

The SLAM measurement device 10 includes a reference recognition section 101 , a self-position recognition section 102 and a measurement object recognition section 103 .

The reference recognition unit 101 recognizes the reference member using the first trained model M1. The reference recognition unit 101 recognizes, as a reference member, a point or area that can be used as a reference for self-position estimation and environment map creation in SLAM among the data obtained from the ranging sensor.

A self-position recognition unit 102 recognizes the self-position of the imaging device using the second trained model. The self-position recognition unit 102 sequentially recognizes the self-position of the imaging device when the imaging device moves while imaging the object to be measured.

The measured object recognition unit 103 recognizes the measured object in the building using the third learned model. The object recognition unit 103 recognizes structures and objects in the building as objects to be measured. The object to be measured may be any object placed in the building, such as pillars, pipes, ducts, and furniture such as desks and shelves in the building.

The first to third trained models M1, M2, M3 are generated in advance by the machine learning model generation device 40 before measurement by the SLAM measurement device 10. FIG. A machine learning model generation device 40 that generates a machine learning model includes a correct data generation unit 401 that generates a correct image from BIM data, a virtual observation data generation unit 402 that generates virtual observation data from BIM data, and a correct data generation unit. A learning model generation unit 403 for generating a machine learning model by performing machine learning using the data generated in 401 as correct data and the data generated in the virtual observation data generation unit 402 as observation data. The correct data generated by the correct data generation unit 401 and the virtual observation data generated by the virtual observation data generation unit 402 are different depending on the learning model to be generated, and as a result, different first to third trained models M1 , M2 and M3 are generated.

FIG. 2 is a diagram illustrating generation of the first trained model M1 according to the present invention.
The first trained model M1 uses a correct image generated from BIM (Building Information Modeling) data as correct data, and performs machine learning using a virtual observation image generated by rendering BIM data as observation data. generated. The first trained model M1 is generated in advance by the machine learning model generation device 40 before measurement by the SLAM measurement device 10 .

When the machine learning model generation device 40 generates the first trained model M1, the correct data generation unit 401 generates a correct image from the BIM data. A virtual observed data generation unit 402 generates a virtual observed image by rendering BIM data. The correct image may be a masked image with masked areas showing structures. The correct image may be, for example, a binarized image generated from BIM data, as shown in FIG. In the example of FIG. 2, the pipe region, which is the structure inside the building, is represented in white, and the other portions are represented in black. The correct image is not limited to the example shown in FIG. 2, and may be an image of another format depending on the structure to be recognized. The learning model generating unit 403 performs machine learning using the correct image generated by the correct data generating unit 401 as correct data and the virtual observation image generated by the virtual observed data generating unit 402 as observation data, and performs machine learning to obtain a first learned model. Generate model M1.

FIG. 3 is a diagram showing the reference recognition section 101 of the SLAM measurement device 10 according to the present invention.
The reference recognition unit 101 recognizes a reference member using image data obtained from the imaging device 30 and distance data obtained from the distance measuring sensor 20 as input data, and outputs three-dimensional feature data of the reference member as output data. do. Here, the three-dimensional feature data output as a result of the reference recognition processing in the reference recognition unit 101 is data in which reference members such as columns, pipes, and ducts are represented by features such as straight lines, points, and planes.

FIG. 4 is a diagram illustrating generation of the second trained model M2 according to the present invention.
The second trained model M2 uses the current three-dimensional coordinates and the attitude angle of the imaging device 30 obtained from the BIM data as correct data, a) current and past virtual observation images, b) current and past virtual reference It is generated by performing machine learning using the features and c) the three-dimensional coordinates and attitude angles of the imaging device 30 in the past as observation data. The second trained model M2 is generated in advance by the machine learning model generation device 40 before measurement by the SLAM measurement device 10. FIG.

When the machine learning model generation device 40 generates the second trained model M2, the correct data generation unit 401 generates the current three-dimensional coordinates and attitude angle of the imaging device 30 as correct data from the BIM data. By rendering the BIM data, the virtual observation data generation unit 402 generates a) current and past virtual observation images, b) current and past virtual reference features, and c) past three-dimensional coordinates of the imaging device 30. Attitude angle is generated as observation data. The learning model generating unit 403 uses the current three-dimensional coordinates and the attitude angle of the imaging device 30 generated by the correct data generating unit 401 as correct data, and a) present and past generated by the virtual observation data generating unit 402. , b) the current and past virtual reference features, and c) the past three-dimensional coordinates and attitude angles of the imaging device 30 as observation data to perform machine learning to generate a second learned model M2. .

FIG. 5 is a diagram showing the self-position recognition unit 102 of the SLAM measuring device 10 according to the present invention.
The self-position recognition unit 102 uses a) image data obtained from the imaging device, b) three-dimensional feature data of the reference member output from the reference recognition unit 101, and c) past three-dimensional coordinates of the imaging device 30. By inputting the posture angle as input data to the second trained model M2, the self position of the imaging device is recognized, and the current three-dimensional coordinates and the posture angle of the imaging device 30 are output as output data. Here, among the three-dimensional coordinates and the posture angle output as a result of the self-position recognition processing in the self-position recognition unit 102, the three-dimensional coordinates (X coordinate, Y coordinate, Z coordinate) are 3 shows the current three-dimensional position of the imaging device 30 in the building when moving within the building while imaging the . The attitude angles (roll angle, pitch angle, yaw angle) indicate the rotation angles of the three-dimensional coordinates of the imaging device 30 with respect to the X axis, the Y axis, and the Z axis, respectively.

FIG. 6 is a diagram illustrating generation of the third trained model M3 according to the present invention.
The third trained model M3 is generated from a virtual observation image generated by rendering the BIM data and the BIM data, using the correct data generated from the BIM data and the data indicating the type of the object to be measured as the correct data. It is generated by performing machine learning using the virtual range image as observed data. The third learned model M3 is generated in advance by the machine learning model generation device 40 before measurement by the SLAM measurement device 10. FIG.

When the machine learning model generation device 40 generates the third trained model M3, the correct data generation unit 401 generates a correct image and correct data from the BIM data. A virtual observed data generation unit 402 generates a virtual observed image by rendering BIM data, and generates a virtual range image from the BIM data. The correct image may be a mask image having a mask area showing the object to be measured. The correct image may be, for example, a binarized image generated from BIM data, as shown in FIG. In the example of FIG. 6, the pillar area, which is the structure inside the building, is expressed in white, and the other parts are expressed in black. The correct image is not limited to the example shown in FIG. 6, and may be an image of another format depending on the structure to be recognized. Further, the correct data is data indicating the type of the object to be measured (for example, pillar, pipe, duct, etc.). The type of the object to be measured is included in the BIM data, and the type of the object to be measured corresponding to the object to be measured indicated by the correct image is extracted from the BIM data and used as correct data. A virtual observed image is generated by rendering the BIM data. A virtual range image is generated from the three-dimensional coordinates of the BIM data. The virtual distance image expresses the distance from the camera to the surface of the object to be measured using, for example, shading. The learning model generating unit 403 performs machine learning using the correct image generated by the correct data generating unit 401 as correct data and the virtual observation image and virtual range image generated by the virtual observation data generating unit 402 as observation data. 3, a trained model M3 is generated.

FIG. 7 is a diagram showing the object recognition unit 103 of the SLAM measurement device 10 according to the present invention.
The measured object recognition unit 103 recognizes the measured object by inputting the image data obtained from the imaging device 30 and the distance data obtained from the distance measuring sensor to the third trained model M3. The type of the object to be measured and the shape data of the object to be measured are output as output data. Here, the type of the object to be measured is, for example, a column, a pipe, a duct, etc. The type of the object to be measured may be indicated by the name of the object to be measured, the category name, the ID, or the like. Further, the shape data of the object to be measured output from the third trained model M3 is shape data in the camera coordinate system of the imaging device 30 .

Next, the object-to-be-measured recognition unit 103 recognizes the shape data (three-dimensional point group) of the object to be measured in the camera coordinate system of the imaging device 30 and the current position of the imaging device 30 output from the self-position recognition unit 102 . From the three-dimensional coordinates and attitude angle, the self-position of the object to be measured in the space coordinate system inside the building is calculated. That is, the measured object recognizing unit 103 uses the current three-dimensional coordinates and attitude angle of the imaging device 30 output from the self-position recognizing unit 102 to determine the position of the imaging device 30 output from the third trained model M3. The shape data of the object to be measured in the camera coordinate system are coordinate-transformed into the shape data of the object to be measured in the space coordinate system within the floor or inside the building. As a result of this, the type of the object to be measured and the shape data (that is, three-dimensional point group) of the object to be measured in the spatial coordinate system within the floor or inside the building are obtained as recognition results in the object recognition unit 103 .

The flow of processing of the method for recognizing structures in a building according to the present invention will be described below with reference to FIGS.
A building structure recognition method according to the present invention comprises steps of recognizing a reference member using a first learned model, recognizing a self-position of an imaging device using a second learned model, and recognizing the object to be measured using the trained model of.

The step of recognizing the reference member includes obtaining image data and distance data, inputting the image data and distance data into the first trained model, and outputting the reference member as output data from the first trained model. and outputting three-dimensional feature data of the member.

The step of recognizing the self-position includes acquiring a) image data, b) three-dimensional feature data of the reference member, c) past three-dimensional coordinates and attitude angles of the imaging device, and a) image data. b) the three-dimensional feature data of the reference member, and c) the past three-dimensional coordinates and attitude angle of the imaging device as input data into the second trained model; and outputting the current self-position of the imaging device in the building as output data.

The step of recognizing the object to be measured includes a step of acquiring image data and distance data, a step of inputting the image data and distance data into a third trained model, and output data from the third trained model, a step of outputting the type of the object to be measured and the shape of the object to be measured in the camera coordinate system; a step of calculating the self-position of the object by using the current self-position in the building of the imaging device; outputting the type of the object to be measured, the self-position of the object to be measured in the spatial coordinate system inside the building, and the shape of the object to be measured in the spatial coordinate system;

FIG. 8 is a flow chart showing the flow of reference recognition processing in the reference recognition unit 101 according to the present invention.
First, in step S801, image data and distance data are obtained. Image data is obtained from the imaging device 30, and distance data is obtained from the ranging sensor 20 (S801).

Next, in step S802, reference recognition is performed. Reference recognition is performed using the first trained model M1. As input data for the first trained model M1, the image data acquired from the imaging device 30 in step S801 and the distance data acquired from the distance measuring sensor 20 are inputted. As a result, prediction using the first trained model M1 is performed.

Next, in step S803, three-dimensional feature data of the reference member is output as output data from the first trained model M1. The three-dimensional feature data of the reference member is data expressing the reference member by features such as straight lines, points, and planes. As a result of the prediction using the first trained model M1, data expressing the reference member with features such as straight lines, points, and planes can be obtained. The three-dimensional feature data of the reference member, which is the result of this reference recognition processing, is later used as input data for the second trained model M2 in step S901 of self-position recognition processing in FIG.

FIG. 9 is a flow chart showing the flow of self-location recognition processing in the self-location recognition unit 102 according to the present invention.
First, in step S901, a) current and past image data, b) current and past three-dimensional feature data of the reference member, and c) past self-position (three-dimensional coordinates and posture angle) in the building are obtained. . a) Current and past image data are obtained from the imaging device 30 . Current and past image data acquired from the imaging device 30 are stored in a memory, a database (not shown), or the like in the imaging device 30 at any time. b) The three-dimensional feature data of the current and past reference members are the results R1 of the reference recognition processing in FIG. c) The past self-position (three-dimensional coordinates and attitude angle) inside the building is saved in the memory or database (not shown) in the self-position recognition unit 102 as needed. The past self-position (three-dimensional coordinates and attitude angle) inside the building is acquired during the second and subsequent self-position recognition processes, and the current self-position (three-dimensional coordinates and attitude angle) inside the building is recognized. used to

Next, in step S902, recognition of the self position of the imaging device 30 is performed. Recognition of the self-position of the imaging device 30 is performed using the second trained model M2. As input data for the second trained model M2, a) current and past image data, b) current and past three-dimensional feature data of the reference member, and c) past self A position (three-dimensional coordinates and attitude angle) is input. As a result, prediction using the second trained model M2 is performed. Regarding c) past self-position (three-dimensional coordinates and attitude angle) inside the building, in the first self-position recognition process, the initial value of the past self-position inside the building is set to the origin of the camera at the start of measurement. Three-dimensional coordinates and attitude angles of other reference points, or preset three-dimensional coordinates and attitude angles, etc., may be set.

Next, in step S903, the current self-position (three-dimensional coordinates and posture angle) of the imaging device 30 inside the building is output. As a result of the prediction using the second trained model M2, the current self-position (three-dimensional coordinates and attitude angle) of the imaging device 30 inside the building is obtained. The result R2 of the self-position recognition process is later used to calculate the self-position of the object to be measured in step S1004 of the object recognition process in FIG.

FIG. 10 is a flow chart showing the flow of object recognition processing in the object recognition unit 103 according to the present invention.
First, in step S1001, image data and distance data are obtained. Image data is obtained from the imaging device 30, and distance data is obtained from the distance measuring sensor 20 (S1001).

Next, in step S1002, the object to be measured in the building is recognized. The objects to be measured are structures and objects in the building, and may be arbitrary objects arranged in the building, such as pillars, pipes, ducts, and furniture such as desks and shelves in the building. Recognition of the object to be measured is performed using the third trained model M3. As input data for the third trained model M3, the image data acquired from the imaging device 30 in step S1001 and the distance data acquired from the distance measuring sensor 20 are inputted. As a result, prediction using the third trained model M3 is performed.

Next, in step S1003, the type of the object to be measured (pipe, duct, etc.) and the shape of the object to be measured (three-dimensional point group) in the camera coordinate system are output. As a result of the prediction using the third trained model M3, the type of the object to be measured (pipe, duct, etc.), the self-position of the object to be measured in the spatial coordinate system inside the building, and the space inside the building A shape (three-dimensional point group) of the object to be measured in the coordinate system is obtained.

Next, in step S1004, the self-position of the object to be measured is calculated. In step S1004, the result R2 of the self-position recognition process output in step S903 of FIG. 9 is used to calculate the self-position of the object to be measured inside the building.

Next, in step S1005, the type of the object to be measured (pipe, duct, etc.) and the position and shape of the object to be measured inside the building (three-dimensional point group) are output.

According to the building structure recognition system 1 according to the present invention described above, structures in a building can be recognized with high accuracy even at a construction site where there are a large number of similar feature points, such as a construction site in the middle of construction. It is possible to recognize In addition, according to the present invention, AI does not simply extract feature points from an image, but recognizes what is good as a reference, connects a lot of measured image data with that as a reference, grasps the entire inside of the building, It is possible to measure In addition, according to the present invention, at an actual construction site, AI can be used to achieve the same effect as the work of measuring many things relative to a good reference such as a pillar and grasping the whole. This makes it possible to improve work efficiency and achieve highly accurate recognition of structures in the building.

Further, according to the present invention, the reference recognition processing by the reference recognition unit 101, the self-position recognition processing by the self-position recognition unit 102, and the object recognition processing by the object-to-be-measured recognition unit 103 can be performed by Measurement can be performed in real time while synchronizing such that the error is sufficiently reduced in consideration of the moving speed of the imaging device such as the camera. Conventional SLAM processing uses time-varying information as the camera or other measuring device moves, so errors in measurement results due to changes in posture and positional information during movement can cause errors. On the other hand, according to the present invention, information can be completed at each position during movement of an imaging device such as a distance measuring sensor or a camera, so errors can be kept small. In particular, errors caused by movement can be reduced.

In addition, according to the present invention, when finding a potential reference using AI, in addition to the visual appearance of the reference, information about the relevance between features in the image such as adjacent features can be obtained. By using it as a reference, it is possible to improve the distinguishability of what can be used as a reference (for example, a vertical member of a steel frame, etc.).

Further, in the above embodiment, one ranging sensor 20 and one imaging device 30 are shown as examples, but the number of ranging sensors and imaging devices to be used is not limited to this. , and one imaging device 30, or a configuration using one distance measuring sensor 20 in the moving direction and three on each side in the direction perpendicular to the moving direction. By using the distance measuring sensor 20 and the imaging device 30 together, for example, when the edge color is similar to the background color, or when color information cannot be handled and the shapes are similar but the members are separate. Accuracy can be maintained even in the field where errors are likely to occur with only one of the imaging device and the distance measuring sensor.

Further, according to the present invention, by performing stereo measurement, it is possible to obtain necessary measurement information by one measurement operation without analyzing time-continuous information. In addition, since the measurement is performed while recognizing the reference point, it is possible to obtain the same accuracy by fusing the information of different measurement points in the measurement area having the same reference point.

In addition, although the present invention assumes measurement while moving, when higher measurement accuracy is required, focus on the object to be measured, temporarily stop moving, and move the sensor or camera to the object to be measured. It is also possible to place it facing the direction and then perform the measurement. In this case, by using a camera or the like having a certain angle of view in the direction of movement, it is possible to obtain the same accuracy without stopping the movement and directing the camera or sensor toward the object to be measured. .

In the present invention, an object to be used as a reference is recognized while moving, and image information and distance information are connected using the object to be used as a reference, so that the certainty of measurement is increased. Also, it is possible to notice an error in the middle of the measurement, and it is possible to minimize the influence on the acquired data when an error occurs.

In addition, it is possible to simulate whether or not the reference member can always be seen during the measurement on the floor plan before the measurement. can also be placed. The reference mark is preferably placed on a wall or pillar near the object to be measured, but may also be placed on the floor.

Furthermore, as another application of the present invention, it is also possible to increase the accuracy of the trained model by using the expert's knowledge used at the construction site together and giving the expert's knowledge as input data at the time of learning. be.
Although the above description has been made of examples, it will be apparent to those skilled in the art that the present invention is not limited thereto and that various changes and modifications can be made within the principles of the invention and the scope of the appended claims.

1 Building Structure Recognition System 10 SLAM Measurement Device 101 Reference Recognition Unit 102 Self Position Recognition Unit 103 Object Recognition Unit M1 First Trained Model M2 Second Trained Model M3 Third Trained Model 20 Distance Measurement Sensor 30 Imaging device 40 Machine learning model generation device 401 Correct data generation unit 402 Virtual observation data generation unit 403 Learning model generation unit

Claims (16)

An indoor structure recognition system,
at least one ranging sensor;
at least one imaging device;
SLAM (Simultaneous Localization and Mapping) measurement device,
The SLAM measurement device is
a reference recognition unit that recognizes a reference member using the first learned model;
a self-position recognition unit that recognizes the self-position of the imaging device using the second trained model;
an object recognition unit that recognizes an object to be measured in the building using the third trained model;
with
The first trained model performs machine learning using a correct image generated from BIM (Building Information Modeling) data as correct data and a virtual observation image generated by rendering the BIM data as observation data. A building structure recognition system characterized by being generated by 2. The reference recognition unit according to claim 1, wherein the reference recognition unit recognizes, as the reference member, a point or area that can be used as a reference for self-position estimation and environment map creation in SLAM among the data obtained from the ranging sensor. building structure recognition system. The reference recognition unit recognizes the reference member by inputting image data obtained from the imaging device and distance data obtained from the range sensor as input data into the first trained model, 3. The intra-building structure recognition system according to claim 1 , wherein three-dimensional feature data of said reference member is output as output data. 4. The self-position recognizing unit sequentially recognizes the self-position of the imaging device when the imaging device moves while imaging the object to be measured. The building structure recognition system described in . The second trained model uses the current three-dimensional coordinates and the attitude angle of the imaging device obtained from the BIM data as correct data, a) current and past virtual observation images, b) current and past reference members and c) the three- dimensional coordinates and attitude angles of the imaging device in the past are generated by performing machine learning as observation data. 1. The building structure recognition system according to 1. The self-position recognition unit includes a) image data obtained from the imaging device, b) three-dimensional feature data of the reference member output from the reference recognition unit, and c) past three-dimensional coordinates of the imaging device. and attitude angle as input data into a second trained model to recognize the self-position of the imaging device, and to output the current three-dimensional coordinates and attitude angle of the imaging device as output data. The intra-building structure recognition system according to any one of claims 1 to 5 . The in-building structure recognition system according to any one of claims 1 to 6 , wherein the measured object recognition section recognizes structures and objects in the building as the measured objects. The third trained model uses a correct image generated from BIM data and data indicating the type of the object to be measured as correct data, and a virtual observation image generated by rendering the BIM data and from the BIM data The indoor structure recognition system according to any one of claims 1 to 7 , characterized in that it is generated by performing machine learning using the generated virtual range image as observation data. The measured object recognition unit recognizes the measured object by inputting image data obtained from the imaging device and distance data obtained from the distance measuring sensor into a third trained model. , the type of the object to be measured and shape data of the object to be measured are output as output data. The object-to-be-measured recognition unit
the shape data in the camera coordinate system of the imaging device of the object to be measured;
The self-position of the object to be measured in a spatial coordinate system inside the building is calculated from the current three-dimensional coordinates and attitude angle of the imaging device output from the self-position recognition unit. 9. The building structure recognition system according to 9 . The ranging sensor is a ToF (Time Of Flight) camera, a ToF sensor, a LiDAR (Light Detection And Ranging) sensor, a RADAR (Radio Detection And Ranging) sensor, an ultrasonic sensor, or an infrared sensor. Item 11. The indoor structure recognition system according to any one of items 1 to 10 . A building structure recognition method comprising:
recognizing a reference member using the first trained model;
recognizing the self-location of the imaging device using the second trained model;
and recognizing the object under test using the third trained model ,
The first trained model performs machine learning using a correct image generated from BIM (Building Information Modeling) data as correct data and a virtual observation image generated by rendering the BIM data as observation data. A building structure recognition method, characterized in that it is generated by Recognizing the reference member comprises:
obtaining image data and distance data;
inputting the image data and the distance data into the first trained model;
13. The indoor structure recognition method according to claim 12 , further comprising a step of outputting three-dimensional feature data of a reference member as output data from said first trained model. The step of recognizing the self-location includes:
obtaining a) the image data, b) the three-dimensional feature data of the reference member, and c) the past three-dimensional coordinates and attitude angles of the imaging device;
a) inputting the image data, b) the three-dimensional feature data of the reference member, and c) the past three-dimensional coordinates and attitude angle of the imaging device as input data into a second trained model;
14. The method for recognizing structures in a building according to claim 13 , further comprising the step of: outputting the current self-position of the imaging device within the building as the output data from the second trained model. . The step of recognizing the object to be measured includes:
obtaining image data and distance data;
inputting the image data and the distance data into the third trained model;
a step of outputting the type of the object to be measured and the shape of the object to be measured in the camera coordinate system as output data from the third trained model;
calculating the self-position of the object to be measured using the current self-position of the imaging device in the building;
and outputting the type of the object to be measured, the self position of the object to be measured in the spatial coordinate system inside the building, and the shape of the object to be measured in the spatial coordinate system. 15. The intra-building structure recognition method according to 14 . A program for causing a computer to perform the steps of the method according to any one of claims 12-15 .

Images