KR102618069B1 - Method and apparatus for analyasing indoor building disaster information using point cloud data and visual information from ground survey robot - Google Patents

Abstract

The present invention relates to a method for analyzing disaster information of a facility executed by a computer device connected to an investigation robot, which is provided on the investigation robot and includes at least one of a LiDAR sensor, an Inertial Measurement Unit (IMU) sensor, and one or more vision sensors. moving the location of the investigation robot to a first location for analyzing information on the facility using a sensor module including; Obtaining sensing information corresponding to the first location based on Simultaneous Localization And Mapping (SLAM); Identifying a facility segment in an indoor space based on the sensing information; Corresponding to the facility segment, acquiring visual crack identification information analyzed from vision image information of the sensing information and crack depth change information determined by depth sensing information corresponding to the visual crack identification information; estimating the degree of damage to the facility segment according to the presence or absence of a crack and the size of the crack obtained based on the visual crack identification information and the crack depth change information; and configuring facility information analysis result data of the facility segment based on the degree of damage, and outputting it to one or more devices.

Description

Method and device for analyzing indoor building disaster information using point cloud data and visual information of ground survey robot {METHOD AND APPARATUS FOR ANALYASING INDOOR BUILDING DISASTER INFORMATION USING POINT CLOUD DATA AND VISUAL INFORMATION FROM GROUND SURVEY ROBOT}

Embodiments of the present invention relate to a disaster information analysis method and device, and more specifically, to an indoor building disaster information analysis technology using point cloud data and visual information of a ground survey robot used in multi-sensing-based indoor positioning.

Objective cause identification and damage investigation at the scene of a disaster accident are very important in preventing and responding to disasters.

To objectively determine the cause and investigate damage at the scene of such a disaster, cutting-edge technology equipment such as drones, robots, and special vehicles are being used. In particular, orbital ground investigation robots are used to acquire indoor investigation data on earthquake and facility collapse sites and the risk of facility collapse. It is being usefully used as a disaster investigation robot that can replace investigation missions in the region.

The investigation robot that investigates these disaster sites is equipped with a positioning module, LiDAR sensor, camera sensor, and communication module, and plays a role in predicting and reporting the risk of facility collapse through indoor and outdoor positioning and LiDAR sensing. .

However, a LiDAR sensor is a sensor that measures the distance and position of an object by emitting light (laser beam) and receiving signals reflected from the object. Accurate measurement is possible only when stability against vibration applied to the sensor module is guaranteed. Therefore, there is a problem that it is not easy to measure in a restricted field.

In addition, when analyzing cracks or fissures, which are the most important elements in determining damage to facilities, the method used to date is to extract the feature points of the camera image as a point cloud and identify the overall shape and outline of the crack. However, these methods cannot identify the starting or ending point of the crack itself, which makes it difficult to accurately determine whether it is an actual crack or the exact location and size of the crack.

In particular, in an actual disaster environment, the risk of real-time collapse of indoor facilities or the risk of additional collapse during recovery work can cause significant damage, so accurate identification and scale analysis, as well as technology to quickly provide analysis results, are needed.

The present invention was developed to solve the problems described above, and identifies cracks by segmenting the sensing information observed from the sensor module of the investigation robot according to the visual characteristics of the vision sensor information, and detects the corresponding lidar sensor-based depth. The purpose is to provide a disaster information analysis method and device that can visualize and output the more accurate location, presence or absence of cracks, and the size of each crack by calculating changes as changes are determined.

The problem to be solved by this specification is not limited to the above, and can be expanded to various matters that can be derived by the embodiments of the invention described below.

The method according to an embodiment of the present invention for solving the above-described problem is a method of analyzing disaster information of a facility executed by a computer device connected to a survey robot, and is provided in the survey robot, and includes a lidar sensor, an IMU Using a sensor module including at least one of an (Inertial Measurement Unit, inertial measurement sensor) sensor and one or more vision sensors, moving the position of the survey robot to a first position for analyzing information on the facility; Obtaining sensing information corresponding to the first location based on Simultaneous Localization And Mapping (SLAM); Identifying a facility segment in an indoor space based on the sensing information; Corresponding to the facility segment, acquiring visual crack identification information analyzed from vision image information of the sensing information and crack depth change information determined by depth sensing information corresponding to the visual crack identification information; estimating the degree of damage to the facility segment according to the presence or absence of a crack and the size of the crack obtained based on the visual crack identification information and the crack depth change information; and configuring facility information analysis result data of the facility segment based on the degree of damage, and outputting it to one or more devices.

Additionally, the method according to an embodiment of the present invention for solving the above-mentioned problem may be implemented as a computer program stored in a computer-readable medium in order to execute the method on a computer.

In addition, a device according to an embodiment of the present invention for solving the problems described above is provided in a research robot including a computing device, and is provided in the research robot, a lidar sensor, an inertial measurement sensor, and one or more vision sensors. a survey robot driving unit that controls the driving of the mobile device to move the position of the survey robot to a first position for analyzing information on the facility, using a sensor module including at least one of the following; and acquire sensing information corresponding to the first location based on SLAM (Simultaneous Localization And Mapping), identify a facility segment of the indoor space based on the sensing information, and, in response to the facility segment, create a vision of the sensing information. Obtaining visual crack identification information analyzed from image information and crack depth change information determined by depth sensing information corresponding to the visual crack identification information, and obtained based on the visual crack identification information and the crack depth change information. A crack analysis unit that estimates and calculates the degree of damage to the facility segment according to the presence or absence of a crack and the size of the crack; and an investigation robot that includes an output unit that configures facility information analysis result data of the facility segment based on the degree of damage and outputs it to one or more devices.

According to an embodiment of the present invention, the position of the investigation robot for analyzing facility information is moved to the first position, visual crack identification information of the facility segment is extracted from vision image information, and visual crack identification information corresponding to the visual crack identification information is extracted. By obtaining crack depth change information determined by depth sensing information, the presence or absence of a crack and the size of the crack can be determined, and the degree of damage to the facility segment can be estimated and calculated based on this.

Accordingly, the present invention can improve crack prediction accuracy while providing visualized facility damage levels and crack analysis results that can be easily recognized by workers in an intuitive manner.

It should be understood that the effect of the present specification is not limited to the matters described above, and can be expanded to various contents that can be derived from the detailed description of the embodiments of the invention below.

1 is a diagram illustrating an example of an operating environment of a system according to an embodiment of the present specification.
FIG. 2 is a block diagram for explaining the internal configuration of the computing device 200 according to an embodiment of the present specification.
Figure 3 is a diagram for explaining the hardware configuration of a survey robot connected to a computing device according to an embodiment of the present invention.
Figure 4 is an exemplary diagram of a sensor module according to an embodiment of the present invention.
Figure 5 is an exemplary diagram of a damage information processing unit according to an embodiment of the present invention.
Figure 6 is a block diagram illustrating in more detail a crack analysis unit according to an embodiment of the present invention.
Figure 7 is a block diagram showing the service processing unit in more detail according to an embodiment of the present invention.
Figure 8 is a flowchart for explaining the operation of a computing device according to an embodiment of the present invention.
Figure 9 is a flowchart to explain in more detail the damage detection algorithm according to an embodiment of the present invention.
Figure 10 is a diagram for explaining an implementation example of a process for providing damage information according to an embodiment of the present invention.
Figure 11 is an analysis diagram to explain the crack skeletonization and size estimation method according to an embodiment of the present invention.
Figures 12 and 13 are experimental examples in which crack detection performance according to an embodiment of the present invention was tested according to each viewing angle and distance from the crack.

In describing the embodiments of the present specification, if it is determined that a detailed description of a known configuration or function may obscure the gist of the embodiments of the present specification, the detailed description thereof will be omitted. In addition, in the drawings, parts that are not related to the description of the embodiments of the present specification are omitted, and similar parts are given similar reference numerals.

In the embodiments of the present specification, when a component is said to be “connected,” “coupled,” or “connected” to another component, this refers not only to a direct connection relationship, but also to an indirect relationship where another component exists in between. Connection relationships may also be included. In addition, when a component is said to "include" or "have" another component, this does not mean excluding the other component, but may further include another component, unless specifically stated to the contrary. .

In the embodiments of this specification, terms such as first, second, etc. are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of components unless specifically mentioned. No. Therefore, within the scope of the embodiments of the present specification, the first component in an embodiment may be referred to as a second component in another embodiment, and similarly, the second component in the embodiment may be referred to as the first component in another embodiment. It may also be called.

In the embodiments of the present specification, distinct components are intended to clearly explain each feature, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to form one hardware or software unit, or one component may be distributed to form a plurality of hardware or software units. Therefore, even if not specifically mentioned, such integrated or distributed embodiments are also included in the scope of the embodiments of the present specification.

In this specification, a network may be a concept that includes both wired and wireless networks. At this time, the network may refer to a communication network in which data exchange between devices, systems, and devices can be performed, and is not limited to a specific network.

Embodiments described herein may have aspects that are entirely hardware, partly hardware and partly software, or entirely software. In this specification, “unit,” “device,” or “system” refers to computer-related entities such as hardware, a combination of hardware and software, or software. For example, in this specification, a part, module, device, or system refers to a running process, processor, object, executable, thread of execution, program, and/or computer. It may be a (computer), but is not limited thereto. For example, both an application running on a computer and the computer may correspond to a part, module, device, or system in the present specification.

Additionally, in this specification, the device may be a mobile device such as a smartphone, tablet PC, wearable device, and HMD (Head Mounted Display), as well as a fixed device such as a PC or a home appliance with a display function. Additionally, as an example, the device may be a cluster within a vehicle or an Internet of Things (IoT) device. That is, in this specification, a device may refer to devices capable of operating an application, and is not limited to a specific type. In the following, for convenience of explanation, the device on which the application runs is referred to as a device.

In this specification, the network communication method is not limited, and connections between each component may not be connected through the same network method. The network may include not only a communication method utilizing a communication network (for example, a mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.), but also short-range wireless communication between devices. For example, a network can include objects and any communication method through which objects can be networked, and is not limited to wired communication, wireless communication, 3G, 4G, 5G, or other methods. For example, wired and/or networks include Local Area Network (LAN), Metropolitan Area Network (MAN), Global System for Mobile Network (GSM), Enhanced Data GSM Environment (EDGE), High Speed Downlink Packet Access (HSDPA), W-CDMA (Wideband Code Division Multiple Access), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), Bluetooth, Zigbee, Wi-Fi, VoIP (Voice over Internet Protocol), LTE Advanced, IEEE802.16m, WirelessMAN-Advanced, HSPA+, 3GPP Long Term Evolution (LTE), Mobile WiMAX (IEEE 802.16e), UMB (formerly EV-DO Rev. C), Flash-OFDM, iBurst and It may refer to a communication network using one or more communication methods selected from the group consisting of MBWA (IEEE 802.20) systems, HIPERMAN, Beam-Division Multiple Access (BDMA), Wi-MAX (World Interoperability for Microwave Access), and ultrasonic communication. However, it is not limited to this.

Components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, embodiments consisting of a subset of the elements described in the embodiments are also included in the scope of the embodiments of the present specification. In addition, embodiments that include other components in addition to the components described in the various embodiments are also included in the scope of the embodiments of the present specification.

Hereinafter, embodiments of the present specification will be examined in detail with reference to the drawings.

1 is a diagram illustrating an example of the operating environment of a system according to an embodiment of the present specification. Referring to FIG. 1, a user device 110 and one or more servers 120, 130, and 140 are connected through a network 1. Figure 1 is an example for explaining the invention, and the number of user devices or servers is not limited as in Figure 1.

The user device 110 may be a fixed terminal implemented with a computer system or a mobile terminal. The user device 110 includes, for example, a smart phone, a mobile phone, a navigation device, a computer, a laptop, a digital broadcasting terminal, a Personal Digital Assistant (PDA), a Portable Multimedia Player (PMP), a tablet PC, or a game console. consoles, wearable devices, IoT (internet of things) devices, VR (virtual reality) devices, AR (augmented reality) devices, etc. For example, in embodiments, user device 110 may be one of a variety of physical computer systems capable of communicating with other servers 120 - 140 over network 1 using substantially wireless or wired communication methods. It can mean.

Each server may be implemented as a computer device or a plurality of computer devices that communicate with the user device 110 and the network 1 to provide commands, codes, files, content, services, etc. For example, the server may be a system that provides each service to the user device 110 connected through the network 1. As a more specific example, the server may provide a service (for example, provision of information, etc.) targeted by the application to the user device 110 through an application as a computer program installed and running on the user device 110. As another example, the server may distribute files for installing and running the above-described application to the user device 110, receive user input information, and provide a corresponding service.

Figure 2 is a block diagram for explaining the internal configuration of the computing device 200 in one embodiment of the present specification. This computing device 200 can be applied to the user device 110 or the server 120-140 described above with reference to FIG. 1, and each device and server is configured by adding or excluding some components to have the same or similar internal structure. It can have a configuration.

Referring to FIG. 2 , the computing device 200 may include a memory 210, a processor 220, a communication module 230, and a transceiver 240. The memory 210 is a non-transitory computer-readable recording medium, and is a non-perishable large capacity device such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It may include a permanent mass storage device. Here, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the above-mentioned device or server as a separate persistent storage device that is distinct from the memory 210. In addition, the memory 210 stores an operating system and at least one program code (for example, code for a browser installed and running on the user device 110, etc., or an application installed on the user device 110 to provide a specific service, etc.). It can be. These software components may be loaded from a computer-readable recording medium separate from the memory 210. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards.

In another embodiment, software components may be loaded into the memory 210 through the communication module 230 rather than a computer-readable recording medium. For example, at least one program is a computer program (for example, installed by files provided through the network 1) by developers or a file distribution system (for example, the server described above) that distributes the installation file of the application. It may be loaded into the memory 210 based on the above-described application).

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 220 by the memory 210 or the communication module 230. For example, processor 220 may be configured to execute received instructions according to program code stored in a recording device such as memory 210.

The communication module 230 may provide a function for the user device 110 and the servers 120 - 140 to communicate with each other through the network 1, and the device 110 and/or the servers 120 - 140, respectively. This can provide functions for communicating with other electronic devices.

The transceiver unit 240 may be a means for interfacing with an external input/output device (not shown). For example, external input devices may include devices such as a keyboard, mouse, microphone, and camera, and external output devices may include devices such as a display, speaker, and haptic feedback device.

As another example, the transceiver 240 may be a means for interfacing with a device that integrates input and output functions, such as a touch screen.

Additionally, in other embodiments, computing device 200 may include more components than those of FIG. 2 depending on the nature of the device to which it is applied. For example, when the computing device 200 is applied to the user device 110, it is implemented to include at least some of the above-described input/output devices, or includes a transceiver, a GNSS (Global Navigation Satellite System), a camera, various sensors, It may further include other components such as a database, etc. As a more specific example, if the user device is a smartphone, the smartphone generally includes various devices such as acceleration sensors, gyro sensors, camera modules, various physical buttons, buttons using a touch panel, input/output ports, and vibrators for vibration. It can be implemented to include more components.

The following description will be made based on a computing device operating based on FIGS. 1 and 2. As an example, in the following, a user may communicate with the robot 300 based on the computing device 200. Additionally, as described below, the robot 300 may be a device that is connected to one computing device 200 and communicates with the outside. As a specific example, the robot 300 communicates with another device or server through a network and may be configured with components included in the computing device 200 of FIG. 2 . Additionally, the robot 300 may include a sensor for identifying surrounding images or images, which will be described later.

Figure 3 is a diagram for explaining the hardware configuration of a survey robot connected to a computing device according to an embodiment of the present invention.

Referring to FIG. 3, the robot 300 has multiple sensors 320 including at least one sensor combined by a header 330 capable of fine adjustment and light source formation, and the multiple sensors 320 are connected separately. By being connected to the robot arm 310, indoor structure analysis can be performed.

Here, the robot arm 310 is designed to be able to rotate 360 degrees and may be provided in a form capable of folding and extending.

In addition, the bottom of the robot 300 may be provided with a control unit 340 that controls the hardware, power, and communication of the robot, and a moving unit 350 that controls the means of movement according to movement, so that it can be used as a structure in indoor environments such as disasters. Allows movement for damage analysis. As an example, the moving unit 350 may be a wheel or other component that performs the function of moving in an indoor space, and is not limited to a specific form.

In this way, the robot 300 is configured to be able to move in an indoor space and may be equipped with a communication function to collect various information and transmit it to another device or server. Additionally, the robot 300 can acquire sensing information by combining with multiple sensors 320. Here, the multi-sensor 320 may include a lidar sensor, an IMU (Inertial Measurement Unit) sensor, a vision camera, a depth camera, a gyro sensor, and other various sensors for sensing surrounding information, and is not limited to a specific sensor. .

A LIDAR/LiDAR sensor may be a sensor that measures the location coordinates of a reflector by shooting a laser pulse and measuring the time it takes for it to reflect and return. Additionally, the IMU sensor is a motion sensor and may be a sensor that measures the direction, acceleration, and position of the device by measuring speed, direction, and magnetism through a speedometer, gyroscope, and magnetometer. In addition, a vision camera can sense video information from surrounding images, and a depth camera is a sensor that measures or senses 3D images (i.e. depth images) of surrounding objects based on stereo or ToF (time of flight) methods. It can be. Other sensors may include sensors for temperature, pressure, and other ambient sensing, and are not limited to specific embodiments.

As an example, the robot 300 may include a LiDAR sensor (or module) for indoor positioning and mapping. Here, the lidar sensor can be manufactured in a small size and combined with the robot 300. In particular, the lidar sensor operates based on SLAM (Simultaneous Localization And Mapping) and can generate indoor maps in real time, through which indoor structures can be recognized.

Additionally, as an example, a Global Navigation Satellite System (GNSS) can be used for indoor and outdoor positioning at a disaster site. However, in cases where communication and radio wave connections from the outside are not smooth due to building shielding and damage indoors at a disaster site, there is a need to recognize the location and perform mapping on its own in the indoor space. For this purpose, the robot 300 uses multiple sensors ( 320) and may include cameras and lidar. At this time, as an example, the LiDAR sensor may be sensitive to vibration caused by external factors because it measures time based on the ToF method. In consideration of the above, the multi-sensor 320 may include an IMU sensor, through which values for movement and vibration of the robot 300 can be sensed, and position movement considering vibration and analysis accuracy, etc. By performing this, stable crack detection and damage information analysis can be performed.

As such, the multiple sensors 320 may include sensors for the robot 300 to smoothly perform sensing in an indoor structure collapsed by an earthquake, and the sensing information is not limited to a specific embodiment. In the following, for convenience of explanation, the robot 300 is coupled with multiple sensors 320, and the robot 300 performs sensing through the multiple sensors 320 to analyze the indoor structure. . That is, the robot 300 described below may be a robot 300 equipped with multiple sensors 320, and may not be limited to a specific embodiment.

Figure 4 is an exemplary diagram of a sensor module mounted on the robot 300 according to an embodiment of the present invention, and is a diagram showing the multiple sensors 320 in hardware.

Referring to FIG. 4, the multiple sensors 320 according to an embodiment of the present invention include a camera sensor 321 located at the top, an IMU sensor 322 axially coupled to the bottom, and a lidar axially coupled to the bottom. It may include a sensor 323.

Here, each sensor module of the multiple sensor 320 is combined in hardware, and is combined around one main axis 320a that passes vertically through each module, so that the three-dimensional space sensing coordinate axes can be synchronized. there is.

According to the synchronization of the coordinate axes of the multiple sensors 320, positioning and rotation information due to the movement of the robot 300 and entry into an inclined section, etc. can be processed in real time and reflected when acquiring sensing information of the multiple sensors 320. For example, the output of each sensor module acquired according to coordinate axis synchronization can be mapped with position information centered on the synchronized main coordinate axis information, and the image information, depth sensing information, and rotation information obtained by synchronization from each sensor module. , angle information, acceleration information, and vibration information are mapped with the location information and can be used to track the location of the robot 300 and detect damage to indoor space facilities.

Additionally, in order to facilitate processing of sensing information from these multiple sensors 320, it is desirable to utilize data processing based on the known SLAM (Simultaneous Localization And Mapping) technology. SLAM is a technology that uses various sensors for an unknown environment to estimate location and create a 3D map of the environment. As computer processing speeds improve and sensor technologies such as cameras and lidar develop, it is being used in various fields. This is a technology that is being applied, especially for indoor positioning and mapping in cases where there is no existing map, such as at a disaster site. It builds spatial information through lidar and camera-based scanning, data structuring, and 3D indoor map visualization. Let's do it.

Accordingly, the robot 300 can construct a 3D indoor map by converting information sensed from the multiple sensors 320 into data, and perform crack analysis on the estimated facilities inside the constructed indoor map and the degree of damage to each facility. Based on this, it can be detected and shared. For example, the robot 300 can be inserted into an indoor structure that collapsed due to an earthquake, build spatial information based on the robot's position indoors, analyze the degree of damage to each facility, and output it.

At this time, as an example, an indoor structure that collapsed due to an earthquake may be in a rough terrain environment, and the accuracy of sensing can be increased by using data from the rough terrain conditions preset below. That is, the robot 300 can be introduced indoors and perform sensing through a sensor, taking into account the environment of indoor structures collapsed by an earthquake. Here, the rough terrain conditions may be, for example, classification conditions such as flat land, bumps, gravel, etc. detected according to average acceleration and maximum acceleration sensing.

The robot 300 and the multiple sensors 320 configured in this way can be controlled by the computing device 200 according to an embodiment of the present invention. As an example, the robot 300 can perform indoor positioning, mapping, and damage analysis using SLAM based on the above-described multiple sensors 320, and control of the robot 300 is performed by the connected computing device 200. It can be performed by As an example, the robot 300 and the computing device 200 may exchange information through a mutual communication connection. At this time, the robot 300 can perform sensing in an indoor space based on the control operation of the computing device 200, and the computing device 200 determines the positions of the multiple sensors 320 to increase the accuracy of crack detection. The movement of the moving unit 350 can be controlled in real time.

For example, vibration for the sensitivity of the LiDAR sensor or light source environment for camera shooting may vary depending on the situation at the disaster site. Therefore, distortion may occur in the sensing information, and various computing processes to minimize this can be processed in the computing device 200 and configured as a control signal for the robot 300.

As an example, the internal position and rotation of the multiple sensors 320 provided in the robot 300 can be controlled, and this can be achieved by controlling the operation of the robot arm 310 by the computing device 200.

Additionally, as an example, the computing device 200 may control the operation of the header 330 combined with the multiple sensors 320, which includes fine-tuning the positions of sensor modules provided in the multiple sensors 320, It performs a role of enabling more accurate sensing operations by controlling the light source of the LED lighting attached to the header 330.

Furthermore, the computing device 200 can control the moving position and moving speed of the robot 300, and the moving position and moving speed are adjusted to prevent distortion or movement by taking into account vibration sensing due to changes in the indoor environment and the recognition rate of the vision sensor. It can be optimized to prevent errors.

Figure 5 shows in more detail the service processing unit 250 of the computing device 200 for driving the robot 300 according to an embodiment of the present invention to output facility analysis information corresponding to 3D analysis of indoor space. It is a block diagram, and FIG. 6 is a block diagram showing the crack analysis unit in more detail, and FIG. 7 is a block diagram showing the analysis information output unit in more detail.

Referring to FIG. 5, the computing device 200 according to an embodiment of the present invention may further include a service processing unit 250. The service processing unit 250 is a processor module that performs various service processes for crack analysis, damage detection, and robot 300 control data processing according to an embodiment of the present invention among the logic and input/output operations processed by the processor 220. It can be composed of: The computing device 200 configures the service processing unit 250 inside the processor 220, or the service processing unit 250 performs service processing using data processed from a device equipped with an external processor, thereby creating a service. Processing results may be provided to the user device 110 or one or more servers 120, 130, and 140.

According to this processing, the user device 110 or one or more servers 120, 130, and 140 receive the analysis information formed by the operation of the service processing unit 250 according to an embodiment of the present invention and display or output it through a display device. , can be used to send and receive data through an external network.

More specifically, referring to FIG. 5 , the service processing unit 250 includes a robot driving unit 251, a sensor module driving unit 253, a crack analysis unit 255, and an analysis information output unit 257.

The robot driver 251 generates a control signal to perform hardware drive control and transmits it to the robot 300 in order to control the position movement and sensing operation of the robot 300 shown in FIGS. 3 and 4 described above. Then, the response is received and drive control is performed.

Here, the robot driving unit 251 performs driving control of each of the multiple sensors 320, header 330, robot arm 310, control unit 340, and moving unit 350 of the robot 300 described above. Accordingly, the movement of the robot 300 and the internal position and sensing control of the multiple sensors 320 can be performed.

In addition, the sensor module driver 253 acquires the sensor signal of each sensor provided in the multiple sensor 320, configures it with sensing information synchronized for each location based on SLAM (Simultaneous Localization And Mapping), and crack analysis unit. Perform data processing transmitted to (255).

Here, the sensor signal may include a sensor signal from the lidar sensor 323, a sensor signal from the IMU sensor 322, and a vision sensing signal from the camera sensor 321. In addition, the sensor module driver 253 mutually maps and converts the signals obtained from each sensor module of the multiple sensor 320 to configure SLAM data, and the composed SLAM data can be transmitted to the crack analysis unit 255. there is.

Here, the SLAM data may include not only sensor signals from the basic LiDAR sensor 323, IMU sensor 322, and camera sensor 321, but also information converted to be converted into three-dimensional spatial data.

For example, SLAM data includes indoor location coordinate information calculated from the IMU sensor 322, RAW image mapping image obtained from the camera sensor 321, and camera and camera data calculated from the IMU sensor 322. It may further include Ida's pose information and point cloud information obtained from the original video image.

In order to improve the sensing accuracy of each sensor information, the sensor module driver 253 moves the position of the robot 300 or precisely controls the position of the multiple sensors 320 to the robot driver 251. ) can also be requested.

In addition, the crack analysis unit 255 acquires the information converted and processed by the above-described sensor module driving unit 253, and processes the construction of a 3D indoor map and detection of facility damage using the obtained sensing information. Here, the crack analysis unit 255 may identify a facility segment in the indoor space based on the sensing information for damage detection processing, and, in response to the facility segment, analyze the vision image information of the sensing information. It is possible to obtain visual crack identification information and crack depth change information determined by depth sensing information corresponding to the visual crack identification information, and the presence or absence of cracks obtained based on the visual crack identification information and the crack depth change information. And depending on the size of the crack, the degree of damage to the facility segment can be estimated.

Accordingly, the analysis information output unit 257 may configure facility information analysis result data of the facility segment based on the degree of damage, combine it with the constructed 3D indoor map information, and output it to one or more devices. Such one or more devices may be, for example, a user device 110, or may include one or more servers 120, 130, 140.

More specifically, referring to FIG. 6, the crack analysis unit 255 includes a facility segment identification unit 2551, a visual crack segment analysis unit 2533, a depth change information mapping unit 2535, and a damage information detection unit 2537. Includes.

The facility segment identification unit 2551 divides facility segments from vision image information obtained from sensing information and performs identification and classification processing.

Here, the facility segment identification unit 2551 uses a pre-built artificial neural network deep learning-based image classification model to classify and process the images of each facility included in the vision image information captured by the robot 300, and the classified Facility images can be separated into individual facility segments.

For example, vision video images can be divided into major categories, medium categories, minor categories, and detailed categories depending on each facility. For example, specific facilities in an image can be classified into detailed categories such as architecture (major category), structure (middle category), door (small category), and window (detailed category), and the portion of the facility image classified into each category is extracted from the original image. They can be separated into individual facility segments and labeled with classification information.

In addition, the visual crack segment analysis unit 2533 acquires visual crack identification information from the vision image information obtained from the sensing information, and uses the obtained visual crack identification information and the facility segment identified in the facility segment identification unit 2551 to By mapping, visual crack identification information can be constructed for each facility segment.

Here, visual crack identification information can be detected by inputting the raw image applied from the sensor module driver 253 to an artificial neural network-based learning model in which facility images and crack detection results are pre-trained. Here, the learning model is a model that preferentially separates and configures crack segments, especially those caused by cracks, among damage to the facility. An artificial neural network learning model such as DCNN may be used for such learning.

In addition, the visual crack segment analysis unit 2533 can map the crack segment and facility segment extracted from the original image in this way, and the crack segment is composed of crack segments for each facility and can be used to represent the crack of the facility. there is.

Here, when performing analysis on a group of existing image feature points, the visual crack segment analysis unit 2533 according to an embodiment of the present invention performs analysis of pixel information corresponding to the original image captured at high resolution and selects pixels. By constructing vector image information of connecting cracks, it is possible to define the start and end points of cracks more accurately than the shape data for existing point groups.

Accordingly, when detecting a crack segment as visual crack identification information, the visual crack segment analysis unit 2533 detects a visual crack object area estimated to be a crack from a vision image, and provides start and end point information for the corresponding object area. It can be configured as included object information. This object information can be configured as 3D object information using 3D spatial coordinate information sensed from the multiple sensors 320, and can be used to construct 3D model data representing the crack segment.

In addition, the crack segment may be composed of a color image visualized by mapping color information to each pixel, and the visual crack segment analysis unit 2533 uses geometric transformation information corresponding to the visual crack object to determine each pixel location. A color image that has undergone color conversion can be mapped to the crack segment.

Here, the geometric transformation information may be obtained according to rotation information of the IMU sensor 322 corresponding to the visual crack object and distance movement information of the robot 300. For example, the camera's 2D RGB information can be corrected according to 3D rotation information and distance movement information, which allows for intuitive visualization by having color information more similar to reality by considering the light source location, etc. do.

Additionally, the visual crack segment analysis unit 2533 may request various drives of the sensor module driver 253 to increase the accuracy of crack segment detection.

For example, when the prediction accuracy of the detected crack segment is less than the first threshold and more than the second threshold, the visual crack segment analysis unit 2533 controls the camera zoom-in or zoom-out of the multiple sensors 320 by the sensor module driver 253. You can request it with . Accordingly, the presence and size of cracks in the area estimated to have cracks can be more accurately determined.

In addition, the visual crack segment analysis unit 2533 moves to the first position according to the control of the moving unit 350 of the robot 300 when the prediction accuracy of the detected crack segment is less than the first threshold and more than the second threshold. may be requested from the robot driving unit 251. This is to accurately determine the presence and size of a crack. In cases where the object presumed to be a crack is not accurate because it is visible from a distance, or when the depth error estimate due to vibration sensing is high, the above product can acquire stable and high-quality images taking this into account. It can be moved to position 1. Accordingly, the presence and size of a crack can be more accurately determined by moving the position of the area where the crack is estimated to exist.

Additionally, for three-dimensional analysis corresponding to the presence or absence and size of a crack, the depth change information mapping unit 2535 may perform visualization of the crack segment corresponding to the crack object and mapping of the depth change information.

Here, the depth change information mapping unit 2535 can acquire a crack area image corresponding to the visual crack object, and depth change information for each pixel obtained from the multiple sensor 320 in response to the crack area image. It can be acquired and mapped.

Here, the depth change information may be determined from distance information of the LiDAR sensor 323 or distance information using a depth camera.

Here, in the case of a depth camera, it may be two or more cameras that simultaneously photograph the same object from two or more locations. In this case, the first camera may be a front camera provided for movement of the robot 300, and the second camera may preferably be comprised of a camera sensor provided in the multi-sensor 320.

Additionally, in the case of a depth camera, it may be configured as a regular camera that moves to two or more different locations, sequentially photographs them, and uses the difference to detect depth. In this case, the internal position of the multiple sensors 320 or the positional movement of the robot 300 are controlled so that depth images corresponding to the crack area images are sequentially photographed from two or more different positions, and depth information is calculated from the depth images. It can be handled.

Meanwhile, the crack analysis unit 255 is configured to obtain the visual crack identification information and the crack depth change information corresponding to the crack when an object presumed to be the crack is identified from the vision sensor of the multi-sensor 320. You can search for the optimal location. In other words, the crack analysis unit 255 can check in advance from a distance a suitable location for the robot 300 to determine the presence and size of a crack, and the optimal location for determining this (for example, the closest location with less vibration) expected location) can be determined, and a movement control request to move to the corresponding location and perform sensing can be transmitted to the robot driver 251.

For example, the crack analysis unit 255 may determine the visual crack identification information of the object tracked by the crack among the movable positions of the investigation robot calculated according to the sensing stability prediction of the sensor module of the multiple sensor 320. And one or more locations predicted to be able to identify the crack depth change information above a threshold may be searched as movement candidate locations. And, the robot driving unit 251 can move the investigation robot to the first location determined from among the searched locations.

Meanwhile, the damage information detection unit 2537 estimates and calculates the degree of damage to the facility segment according to the presence or absence of a crack and the size of the crack obtained based on the visual crack identification information and the crack depth change information. Accordingly, the crack analysis unit 255 may construct crack analysis information including the calculated degree of damage and transmit it to the analysis information output unit 257. Accordingly, the analysis information output unit 257 may configure facility information analysis result data of the facility segment based on the degree of damage and output it to one or more devices.

More specifically, referring to FIG. 7 , the damage information detection unit 2537 includes a 3D positioning model configuration unit 2537a and a damage analysis information visualization unit 2537b.

Here, the 3D positioning model constructor 2537a constructs a 3D positioning model of the indoor space using SLAM data-based sensing information obtained from the multiple sensors 320.

In addition, the damage analysis information visualization unit 2537b displays the estimated damage degree of the facility segment on the three-dimensional positioning model based on the visual crack identification information and the crack depth change information. can be performed.

More specifically, the damage analysis information visualization unit 2537b extracts crack skeletonization information forming a crack in response to a crack segment determined as a crack object, and cracks estimated from the extracted crack skeletonization information. Using the average length and thickness information, the size of the crack can be determined, and the size of the determined crack can be displayed on the three-dimensional positioning model.

Figure 8 is a flowchart for explaining the operation of a computing device according to an embodiment of the present invention.

Referring to FIG. 8, the computing device 200 including the service processing unit 250 according to an embodiment of the present invention first determines the location of the survey robot 300 for analyzing information on the facility through the robot driving unit 251. Move to the first position (S101).

Then, the computing device 200 obtains sensing information corresponding to the first location through the sensor module driver 253 based on Simultaneous Localization And Mapping (SLAM) (S103).

Thereafter, the computing device 200 identifies a facility segment in the indoor space based on the sensing information through the crack analysis unit 255 (S105).

Then, the computing device 200 acquires visual crack identification information analyzed corresponding to the facility segment through the crack analysis unit 255 (S107), and acquires crack depth change information corresponding to the visual crack identification information. Do it (S109).

Accordingly, the computing device 200 calculates the presence or absence and size of a crack based on the visual crack identification information and the crack depth change information through the crack analysis unit 255 (S111), and then determines the presence or absence of a crack. And the degree of damage to the facility segment is estimated and calculated based on the scale (S113).

Thereafter, the computing device 200 configures facility information analysis result data using the estimated damage information of the facility segment through the analysis information output unit 257 and outputs it to one or more devices (S115).

Figure 9 is a flowchart to explain in more detail the damage detection algorithm according to an embodiment of the present invention.

Figure 9 relates to a detailed damage detection algorithm. First, the computing device 200 builds a pre-associative learning model based on an artificial neural network between facility segments extracted from sensor video images and actual crack detection information ( S200).

Here, for association learning for extracting and classifying facility segments, an image classification learning method based on artificial neural network deep learning such as DeepLab can be used as an example, and for association learning for crack detection, crack segment images and cracks can be extracted from the facility based on ResNet. An example may be a DCNN-based artificial neural network learning method that learns to extract objects. Accordingly, a first learning model for detecting facility segments and a second learning model for extracting crack segments may each be configured in advance.

When sensing information is input to the crack analysis unit 255 (S201), the computing device 200 first divides and processes the facility segment image of the indoor space using the above-described first learning model (S203).

Thereafter, the computing device 200 detects the above-described crack location using the second learning model through the crack analysis unit 255 (S205), and divides and processes the crack segment image into a crack segment image for each facility segment (S207).

In addition, the computing device 200 estimates the degree of damage for each crack segment through the damage degree detection unit 2537 of the crack analysis unit 255 (S209), and comprehensively visualizes the degree of damage for each facility in the indoor space. Output (S211).

Figure 10 is a diagram for explaining an implementation example of a process for providing damage information according to an embodiment of the present invention.

As shown in FIG. 10, when raw image information is acquired from the sensor module driver 253, the crack analysis unit 255 performs geometric conversion on the color information of the original image and detects crack segments corresponding to each facility segment. You can perform crack analysis to indicate this. Each of these analysis results can be visualized as shown in FIG. 10, and the analysis results can be configured as a three-dimensional model and provided to each user device 110 or one or more servers 120, 130, and 140. .

In particular, as shown in FIG. 10, the damage location and degree of damage for each recognized facility are configured as 3D position coordinates and can be mapped onto the visualized 3D analysis result data on the facility segment model 401. A crack detection area 402 may be displayed, and crack object location coordinates 403 detected as a crack object may be mapped within the crack detection area 402. This detection information may be mapped as detection log data and each individual crack segment image data and stored and managed in the computing device 200.

Figure 11 is an analysis diagram to explain the crack skeletonization and size estimation method according to an embodiment of the present invention.

As shown in Figure 11, the detected crack object information can be converted to determine the degree of damage by measuring the average length and thickness of the crack, and a skeletonization method may be preferable as the conversion method. there is. This allows the presence and size of cracks to be identified more accurately than the original crack image, and the degree of damage can be quickly identified using the estimated value. To estimate the size of such cracks, binary conversion and vectorization processes of crack segment images can be used, and algorithms such as trend line prediction can be used.

Figures 12 and 13 are experimental examples in which crack detection performance according to an embodiment of the present invention was tested according to each viewing angle and distance from the crack.

Referring to Figure 12, it can be seen that the shape of the crack segment is detected more accurately as the distance becomes closer. Referring to Figure 13, as the gaze angle is adjusted to look directly at the object, the shape of the crack segment is detected and You can see that color conversion mapping is performed more accurately. In this way, the computing device 200 determines the optimized first position when the crack estimation object is detected, considering the position where vibration is minimized while detecting the shape of the crack segment more accurately, and moves the robot 300 to that position. ) can be moved, and as a result, more accurate crack segments are detected, and more accurate crack object configuration and damage level analysis based on this becomes possible.

The embodiments described above can be implemented at least partially as a computer program and recorded on a computer-readable recording medium. Computer-readable recording media on which programs for implementing the embodiments are recorded include all types of recording devices that store data that can be read by a computer. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, and optical data storage devices. Additionally, computer-readable recording media may be distributed across computer systems connected to a network, and computer-readable codes may be stored and executed in a distributed manner. Additionally, functional programs, codes, and code segments for implementing this embodiment can be easily understood by those skilled in the art to which this embodiment belongs.

Although the above-described specification has been described with reference to the embodiments shown in the drawings, these are merely illustrative examples, and those skilled in the art will understand that various modifications and modifications of the embodiments are possible therefrom. However, such modifications should be considered within the scope of technical protection of this specification. Therefore, the true technical protection scope of the present specification should be determined to include other implementations, other embodiments, and those equivalent to the claims according to the technical spirit of the appended claims.

Claims (15)

In the indoor building investigation robot disaster information analysis method using the point cloud data and visual information of the ground investigation robot,
It is provided in the investigation robot and uses a sensor module that includes at least one of a LiDAR sensor, an IMU (Inertial Measurement Unit) sensor, and one or more vision sensors to determine the location of the investigation robot for information analysis of the facility. moving to position 1;
Obtaining sensing information corresponding to the first location based on Simultaneous Localization And Mapping (SLAM);
Identifying a facility segment in an indoor space based on the sensing information;
Corresponding to the facility segment, acquiring visual crack identification information analyzed from vision image information of the sensing information and crack depth change information determined by depth sensing information corresponding to the visual crack identification information;
estimating the degree of damage to the facility segment according to the presence or absence of a crack and the size of the crack obtained based on the visual crack identification information and the crack depth change information; and
Comprising the step of configuring facility information analysis result data of the facility segment based on the degree of damage and outputting it to one or more devices,
The moving step is,
When an object presumed to be the crack is identified from the vision sensor, searching for an optimal location for obtaining the visual crack identification information and the crack depth change information corresponding to the crack; and
It includes moving the investigation robot to the first location determined among the searched locations,
The step of searching for the location is,
Among the movable positions of the investigation robot calculated according to the sensing stability prediction of the sensor module, one predicted to be able to identify the visual crack identification information and the crack depth change information of the object tracked by the crack above a threshold value Comprising the step of searching the above locations as movement candidate locations,
Disaster information analysis method.
According to paragraph 1,
The visual crack area identification information is,
Containing start and end point information of a visual crack object obtained by performing pixel information analysis corresponding to the facility segment using one or more camera images included in the sensing information of the vision sensor
Disaster information analysis method. According to paragraph 2,
The visual crack area identification information includes a crack area image corresponding to the visual crack object,
The crack area image is,
Contains a color image in which color conversion for each pixel position has been processed using geometric conversion information corresponding to the visual crack object,
The geometric transformation information is obtained based on at least one of rotation information of the IMU sensor corresponding to the visual crack object and distance movement information to the first position.
Disaster information analysis method. According to paragraph 3,
The crack depth change information is,
Contains depth change information detected according to depth sensing information corresponding to the crack area image,
The depth sensing information is calculated from the lidar sensor, a stereo camera, or a plurality of camera images taken at different locations and mapped onto the crack area image.
Disaster information analysis method. According to paragraph 1,
The step of estimating the degree of damage is,
Extracting crack skeletonization information consisting of crack connection lines based on the visual crack identification information and the crack depth change information; and
Comprising the step of determining the size of the crack using the average length and thickness information of the crack estimated from the crack skeletonization information.
Disaster information analysis method. delete According to paragraph 1,
The step of searching for the location is,
Among the movable positions of the investigation robot calculated according to the sensing stability prediction of the sensor module, two are predicted to be able to identify the visual crack identification information and the crack depth change information of the object tracked by the crack above a threshold value. Including the step of searching for more than one location as a candidate location for movement.
Disaster information analysis method. According to paragraph 1,
The visual crack identification information is,
Crack detection result information corresponding to pre-classified facility images is obtained by dividing the facility segment image of the indoor space using a pre-trained learning module based on an artificial neural network learning algorithm.
Disaster information analysis method. A computer program stored in a computer-readable medium to execute the method according to any one of claims 1 to 5 and 7 on a computer. In a research robot including a computing device,
Using a sensor module provided in the survey robot and including at least one of a lidar sensor, an inertial measurement sensor, and one or more vision sensors, the position of the survey robot is moved to a first position for analyzing information on the facility. A survey robot driving unit that controls the operation of the mobile device to do so; and
Obtain sensing information corresponding to the first location based on SLAM (Simultaneous Localization And Mapping), identify a facility segment in an indoor space based on the sensing information, and generate a vision image of the sensing information in response to the facility segment. Acquire visual crack identification information analyzed from information and crack depth change information determined by depth sensing information corresponding to the visual crack identification information, and obtain cracks based on the visual crack identification information and the crack depth change information. A crack analysis unit that estimates and calculates the degree of damage to the facility segment according to the presence or absence of and the size of the crack; and
An output unit that configures the facility information analysis result data of the facility segment based on the degree of damage and outputs it to one or more devices.
Contains,
The visual crack area identification information is,
Contains start and end point information of a visual crack object obtained by performing pixel information analysis corresponding to the facility segment using one or more camera images included in the sensing information of the vision sensor,
The visual crack area identification information includes a crack area image corresponding to the visual crack object,
The crack area image is,
Contains a color image in which color conversion for each pixel position has been processed using geometric conversion information corresponding to the visual crack object,
The geometric transformation information is obtained based on at least one of rotation information of the IMU sensor corresponding to the visual crack object and distance movement information to the first position,
The survey robot driving unit,
When an object estimated to be the crack is identified from the vision sensor, the optimal location for obtaining the visual crack identification information and the crack depth change information corresponding to the crack is searched, and the first location determined from among the searched locations Outputting a control signal to move the survey robot to a location,
The survey robot driving unit,
Among the movable positions of the investigation robot calculated according to the sensing stability prediction of the sensor module, one predicted to be able to identify the visual crack identification information and the crack depth change information of the object tracked by the crack above a threshold value Searching the above locations as movement candidate locations,
Investigation robot. According to clause 10,
The geometric transformation information is obtained based on at least two of rotation information of the IMU sensor corresponding to the visual crack object and distance movement information to the first position.
Investigation robot. In article 10,
The crack depth change information is,
Contains depth change information detected according to depth sensing information corresponding to the crack area image,
The depth sensing information is calculated from the lidar sensor, a stereo camera, or a plurality of camera images taken at different locations and mapped onto the crack area image.
Investigation robot. According to clause 11,
The crack analysis unit,
Based on the visual crack identification information and the crack depth change information, crack skeletonization information consisting of crack connection lines is extracted, and the average length and thickness information of the crack estimated from the crack skeletonization information is used to determine the size of the crack. determining the size
Investigation robot. delete According to clause 10,
The survey robot driving unit,
Among the movable positions of the investigation robot calculated according to the sensing stability prediction of the sensor module, two are predicted to be able to identify the visual crack identification information and the crack depth change information of the object tracked by the crack above a threshold value. Search for the above locations as candidate locations for movement
Investigation robot.