KR20220032920A - Maintenance method for underground pipeline using point cloud data and augmented reality - Google Patents

Abstract

The present invention relates to an underground pipeline maintenance method using point cloud data and augmented reality. According to the present invention, an underground pipeline is exposed to the outside for the installation of a novel pipeline or maintenance of an existing pipeline. In this state, the three-dimensional coordinates of multiple points on the surface of the underground pipeline are measured using a point cloud generator, underground pipeline point cloud data is generated, and an IT infrastructure is built. For maintenance such as underground pipeline repair, a user approaches the underground pipeline with a smart terminal network-connected to a server. The smart terminal grasps posture information including current location information, azimuth, and angle. In addition, the location coordinates of a region photographed by the camera of the smart terminal are calculated using the location information and the posture information of the smart terminal. The location coordinates of the underground pipeline point cloud data and the location coordinates of the photographing region are compared to detect points included in the photographing region. Using the detected underground pipeline point cloud data, a rendered image of the underground pipeline is superimposed on an actual image of the smart terminal and expressed in augmented reality. The user can accurately grasp and visually confirm where the underground pipeline is buried before ground excavation for pipeline maintenance.

Description

Method for maintenance of underground pipelines using point cloud data and augmented reality

The present invention relates to a method for maintaining and managing underground pipelines, such as gas pipes, water pipes, and communication lines, and more particularly, to a method for allowing the location of underground pipelines to be identified on land.

The Daegu gas explosion accident in 1995 was an accident in which leaked gas exploded as a gas pipe was damaged during subway excavation work. After the Daegu subway accident, the location of underground lifelines such as water and sewage pipes, gas pipes, and power cables is checked in advance, and much attention is paid to the excavation work, but damage to underground facilities still occurs frequently.

In order to prevent damage to underground facilities during excavation work, it is important to know exactly where the underground facilities are buried. In addition, even if the location has been identified, identification marks are requested so that workers on the ground can visually check and pay attention to the location of the burial during construction.

Currently, paper drawings or electronic drawings are used to locate underground facilities. The location information of pipelines identified by each underground facility operating organization is received and utilized in the form of paper drawings or 2D electronic drawings. However, regardless of the accuracy of the location information provided by each operating organization, it is not easy to apply this location information to the field. Recently, electronic tags (RFID, QR codes) are attached to ground facilities to recognize computerized location information provided by each operating organization on site, and when the user approaches the tag, the location information is displayed on the screen of the terminal. system has been developed. However, such a marker-based underground facility recognition system has the inconvenience of manufacturing a marker and attaching it to an underground facility.

Meanwhile, a method for implementing an underground pipeline in a mobile phone or tablet terminal using augmented reality (AR) has been proposed. Augmented reality technology is a technology that combines the real world seen by the user and the virtual world with additional information in real time as a single image. For example, by using the location and direction information of the portable terminal to determine the approximate location, and by comparing object information such as information about surrounding buildings and the actual image information input according to the movement of the camera, the service desired by the user is identified and It is a technology that provides relevant information. In the case of using the underground pipeline to identify the location, the underground pipeline is synthesized and displayed in the actual image captured by the camera of the mobile phone terminal using the location information of the underground pipeline.

One problem when displaying an underground pipeline on a terminal using augmented reality is that the terrain cannot be reflected. In general, location information for an underground pipeline includes only planar coordinates (latitude and longitude) and depth information of the pipeline (depth from the surface). Therefore, in the sloping terrain, the underground pipeline in the augmented reality image is located at a completely different location from the actual location of the pipeline.

1 is to explain a state implemented in augmented reality when a pipeline is buried in a sloping terrain with a difference in elevation, and information about the underground pipeline includes only planar location coordinates and depth from the surface. Referring to FIG. 1 , the ground surface is inclined to the right and there is a curvature, and the underground pipeline is disposed along the ground surface as a whole. However, only the depth of burial from the ground at each point is slightly different. In such a sloping terrain, an underground pipeline appears different from reality on a screen implemented in augmented reality on a terminal in the prior art. This is because altitude information is not included in topography, that is, pipeline information. Conversely, it can be explained that the elevation is not included in the surface coordinates.

Terrain such as contour lines, digital elevation model (DEM: Digital Elevation Model), digital terrain elevation model (DTM, Digital Terrain Model), digital surface model (DSM, Digital Surface Model), etc. Models and 3D objects were also created by modeling using a 3D engine such as a height map technique, a quadtree technique, and a frustum culling technique. However, the 3D modeling process is complicated and time consuming, so there is a limit to actual commercialization.

The present invention is to solve the above problems, and by acquiring the underground pipeline information as point cloud data, the underground pipeline includes elevation information in addition to the planar coordinates, so that the buried position of the underground pipeline is augmented to correspond to the actual location even in a sloping terrain The purpose is to provide a method that can be expressed in reality.

On the other hand, other objects not specified in the present invention will be additionally considered within the range that can be easily inferred from the following detailed description and effects thereof.

The maintenance method of the underground pipeline according to the present invention for achieving the above object,

(a) For the installation of a new pipeline or maintenance of an existing pipeline, the three-dimensional coordinates of multiple points on the surface of the underground pipeline are measured with a point cloud generator in the state that the underground pipeline is exposed to the outside, and the underground pipeline point cloud data is generated to do; (b) measuring the position information of the smart terminal connected through the server and the communication network, and the posture information including the azimuth and angle; (c) calculating the position coordinates of the photographing area photographed by the camera of the smart terminal by using the location information and the posture information of the smart terminal; and (d) comparing the position coordinates of the underground pipeline point cloud data with the position coordinates of the photographing area to detect points included in the photographing area, and rendering the underground pipeline using the detected underground pipeline point cloud data. It is characterized in that it includes; superimposing the image on the actual image of the smart terminal and expressing it in augmented reality.

According to the present invention, the method comprising the steps of: setting at least one reference point in an adjacent area through which the underground pipe passes in a buried state while covering the upper part of the underground pipe, and acquiring the actual position coordinates of the reference point; detecting the reference point through image analysis on the actual screen of the smart terminal; and compensating the azimuth value of the smart terminal by comparing the coordinate values of the reference point identified in the smart terminal with the actual position coordinates.

In particular, the reference point is a shape of a specific pattern, and the reference image for the specific pattern is pre-stored in the image detection algorithm of the smart terminal. The reference point can be detected by comparison.

Alternatively, the three-dimensional coordinates of a plurality of feature points on a nearby surface may be measured with a point cloud generating device in a state in which the underground pipeline is buried, and the plurality of feature points may be set as the reference point.

In the present invention, specific point cloud data is obtained for a plurality of points on the outskirts (surface) of the underground pipeline as three-dimensional coordinates, and the point cloud is rendered and expressed in augmented reality on the actual screen of the smart terminal. Since the point cloud data is specified by three-dimensional coordinates of latitude, longitude, and elevation, the elevation of the underground pipeline is reflected unlike the existing two-dimensional planar data. Accordingly, there is an advantage that the underground pipeline expressed in augmented reality is expressed in accordance with the reality even on sloping terrain.

Also, by rendering the point cloud data and adding color to each point, the underground pipeline is three-dimensionally expressed. It does not require additional processes such as 3D object styling for the underground pipeline.

As point cloud data is used, additional processes such as 3D modeling or object styling are all unnecessary, so there is an advantage that faster processing is possible as the operation speed is improved in a smart terminal or server.

On the other hand, in the present invention, in order to correct an error in azimuth sensing in the smart terminal, a reference point that knows the actual location coordinates is set in an area adjacent to the underground pipeline buried so that the underground pipeline matches the actual location in augmented reality on the display of the smart terminal. It has the advantage that it can be implemented.

On the other hand, even if it is an effect not explicitly mentioned herein, it is added that the effects described in the following specification expected by the technical features of the present invention and their potential effects are treated as described in the specification of the present invention.

1 is for explaining a problem when the underground pipeline is expressed in augmented reality when the underground pipeline is buried in the inclined terrain.
2 is a schematic flowchart of an underground pipeline maintenance method according to an embodiment of the present invention.
3 is
※ It is revealed that the accompanying drawings are exemplified as a reference for understanding the technical idea of the present invention, and the scope of the present invention is not limited thereby.

In the description of the present invention, if it is determined that related known functions are obvious to those skilled in the art and may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

The present invention is performed by a computing device including a tablet, a smart phone, a computer, and the like, wherein the computing device such as a smart phone or a computer is loaded with software capable of performing the following specific functions. Here, software can be used interchangeably with various names such as logic, logic block, algorithm, program, and application.

In addition, in the present specification, the server, database, and user terminal are divided for convenience of description, but this is only for function description. It may exist separately, and may have a structure in which a plurality of user terminals communicate with a server to exchange data. That is, the system for implementing the method according to the present invention should be understood as a functional unit that implements a specific function, rather than focusing on physical independence. That is, all elements performing a specific function may be included in the user terminal, and the user terminal may only serve as a display, and processing and processing of all information may be performed in the server. Alternatively, it should be clarified in advance that the system can be configured in various ways, such as the user terminal and the server can divide and take charge of specific functions.

Hereinafter, a method for maintaining an underground pipeline using point cloud data and augmented reality according to an embodiment of the present invention will be described in more detail with reference to the accompanying drawings.

2 is a flowchart of an underground pipeline maintenance method according to an embodiment of the present invention.

In the present invention, first, point cloud data for an underground pipeline is generated and stored in a database. A point cloud is a set of points belonging to a certain coordinate system. In a three-dimensional coordinate system, a point is usually defined by X, Y, and Z coordinates and can be used to represent the exterior (surface) of an object. Point cloud data can be obtained through Lidar mounted on a point cloud generating device, for example, various means of transportation (drone, airplane, vehicle), or video images shot from multiple angles from drones/aircraft/vehicles, etc. can be obtained through CV analysis. . Lidar-based point cloud data or open CV analysis-based point cloud data is acquired. The point cloud data is specified as three-dimensional coordinates for a plurality of points on the object surface. Although the present invention does not exclude that the three-dimensional coordinate system is separately set, the three-dimensional coordinates are set by latitude/longitude/altitude.

The location information of the existing underground pipeline was specified by two-dimensional planar coordinates (latitude, longitude) and depth from the surface. Since the surface has a height depending on the terrain, when the Z-axis coordinate of the underground pipe is specified only by the depth of the underground, the same problem as described in the prior art appears. The present invention uses a point cloud technology to overcome this problem.

In the present invention, the surface of the underground pipeline by using the point cloud generating device as described above in a state in which an underground pipeline is newly buried or the surface is excavated for maintenance of an existing pipeline, that is, in a state in which the pipeline is installed and exposed to the outside Point cloud data is generated by acquiring the three-dimensional coordinates of a plurality of points of Point cloud data is stored in the DB of the server.

In the present invention, it is planned that a specific pipeline, for example, a nationwide network of gas pipelines, is created as point cloud data as described above and stored in a database to be utilized for maintenance of gas pipelines in the future. Even if it is not a nationwide deployment network, it is desirable to generate and use point cloud data for maintenance for deployment networks in urban areas where various lifelines and underground infrastructure are dense and excavation works are frequent.

When the point cloud database for the underground pipeline is established, the maintenance of the underground pipeline becomes possible on a daily basis using augmented reality implemented on the display screen of the smart terminal.

As the user terminal, a smart terminal such as a smartphone or a tablet PC may be used. In this embodiment, the smart terminal is equipped with a GPS module, an IMU sensor, an AR module, a video camera module, and a display panel.

The GPS module detects the current location of the user terminal, that is, location information in real time, and transmits it to the AR module. Of course, if the GPS module installed by default in the user terminal is not precise, a precise GPS module may be separately mounted near the user terminal and location information may be transmitted through wired/wireless communication with the user terminal.

The IMU sensor detects posture information including the azimuth and vertical angle of the user terminal. The IMU sensor consists of an acceleration sensor that detects movement in three axes of front, back, up, down, left and right in three-dimensional space, and a gyroscope sensor that detects three-axis rotation of pitch, roll, and yaw. . Posture information such as the direction and angle of the front of the user terminal is detected using the IMU sensor and transmitted to the AR module in real time.

In addition, the video camera module of the user terminal acquires the actual screen in real time. In the AR module, location coordinates for the shooting area of the actual screen shot in the smart terminal can be calculated using location information, posture information, and angle of view information of the camera of the smart terminal.

When the location coordinates of the photographing area of the smart terminal are obtained, the location coordinates of the underground pipeline point cloud data recorded in the database of the server are compared with the location coordinates of the photographing area. Through comparison, the point cloud of the underground pipeline located within the shooting area is detected.

A virtual image of a rendering of an underground pipeline is generated using the detected point loud data, and the virtual image is superimposed on the actual image of the smart terminal to realize augmented reality.

The AR module that implements the above-described augmented reality is generally installed in the user terminal in the form of a computer program such as an app, and the augmented reality implementation technology is a well-known technology, and further detailed description will be omitted.

The present invention is characterized in that the underground pipeline is created as point cloud data and used for maintenance. Existing underground pipelines are specified only with the location coordinates of latitude and longitude, and the elevation coordinates are specified only with depth from the surface. Accordingly, when realizing augmented reality of underground pipelines in sloping terrain, there was a problem in that the underground pipelines were expressed horizontally instead of following the slopes in a completely different form from reality. Using point cloud data can solve the above problem. That is, it can be realized that the underground pipeline is arranged along the slope as in reality. This is possible because the point cloud data contains three-dimensional coordinates, especially elevation information. If the point cloud is rendered in three-dimensional coordinates without going through complex processing such as terrain model or 3D modeling, the underground pipeline is placed along the slope. do.

Another advantage of using point cloud data is that 3D object styling of the underground pipeline is not required when implementing the underground pipeline in augmented reality. Existing underground pipeline information has only location information or sometimes includes attribute information. The attribute information indicates the size and material of the pipe. Accordingly, in the AR module, 3D object styling of the underground pipeline was performed arbitrarily or based on attribute information, and the pipeline was displayed in a three-dimensional form when implementing augmented reality. However, as in the present invention, since the point cloud itself reflects the shape of the actual underground pipeline, there is an advantage that an additional process such as object styling is not required.

As above, if additional processes are involved, such as transforming two-dimensional coordinates into three dimensions through modeling or performing three-dimensional styling of objects using attribute information, processing speed in smart terminals may become a problem. There is an advantage that all of these problems can be solved by using the point cloud data as in the present invention.

On the other hand, one weakness when using a smart terminal is an error in orientation. Compared to the location information obtained through the GPS module in the smart terminal and the angle information obtained through the IMU sensor, the azimuth has a relatively high error. If there is an error in the orientation, there is a problem in that the underground pipeline is displayed in a location different from the actual location when the AR module implements a virtual image using the point cloud data.

In order to solve this problem, the present invention provides a method capable of correcting the azimuth value of the smart terminal.

Point cloud data were acquired while the underground pipeline was exposed to the outside. After that, after the upper part of the underground pipe is covered and buried again, one reference point is set on the adjacent ground surface, and the actual position coordinates for this reference point are acquired. The location coordinates may use various methods such as a GPS module and surveying. The location coordinates of the reference point are stored in the server's database.

In this embodiment, a marker may be used as a reference point. A marker is a kind of mark in which a specific pattern is embodied. For example, you can attach a QR code like a sign and use it. Even if it is not a QR code, a cover using figures, numbers, letters, etc. alone or in combination can be used. Alternatively, you can use the surrounding terrain. For example, in an urban area, various topographical features such as manhole covers, rain gutters, road signs, and road boundary stones can be used. The reference image of the shape and pattern of the marker is stored in the server. That is, the server stores the coordinates of the position of the marker as the reference point and the reference image in the server.

In order to maintain the underground pipeline, the user approaches the perimeter of the underground pipeline and takes a picture of the marker with a smart terminal. In the image detection algorithm of the smart terminal, the marker can be detected by comparing the image of the marker displayed on the actual screen of the smart terminal with the reference image received from the server to check the similarity. When the marker is detected, the azimuth value of the smart terminal may be corrected using the location information of the marker recognized by the smart terminal.

The location information of the marker recognized by the smart terminal and the actual location information of the marker may appear different from each other. This is mainly caused by an error in the azimuth in the smart terminal. Accordingly, the orientation sensed by the smart terminal (eg, the true north direction) may be corrected using the location information of the actual marker. If there is an error in the azimuth measurement of the smart terminal, the underground pipeline is inevitably expressed in augmented reality at a location different from the actual location on the screen of the smart terminal. In order to solve the above problem, in the present invention, an error in azimuth sensing of a smart terminal is corrected by using a marker having an accurate position.

Meanwhile, in another example of the present invention, a point cloud may be used instead of a marker as a reference point for correcting an azimuth sensing error of the smart terminal. Recently, as the IT infrastructure begins to be built rapidly, there are attempts to secure point cloud data not only in downtown areas but also throughout the country. Norway already has point cloud data for all territories.

In this way, if the point cloud data is public data that anyone can access, the point cloud data can be used as a reference point without using a marker. That is, feature points, for example, corners and vertices of buildings, may be detected from the point cloud data. And when a building is expressed on the actual screen of the smart terminal, the characteristic points (corners, vertices) of the building can be specified through an image analysis algorithm. The location coordinates of the feature points of the building detected from the point cloud data are compared with the location coordinates of the feature points of the building specified on the actual screen of the smart terminal. Since the point cloud data is an actual coordinate, it is enough to calibrate the azimuth value or the position coordinate value of the smart terminal based on this. That is, in this example, the position coordinate value or the azimuth value of the smart terminal can be corrected by using the feature point as a reference point in the point cloud data for the adjacent area in which the underground pipeline is buried. The feature point may be one or may be plural. In addition, when detecting a plurality of feature points, in addition to their coordinate values, an interval between the plurality of feature points, an interior angle of a triangle connecting three feature points, etc. may be used as a reference. For reference, the point cloud data of the surface area adjacent to the underground pipeline buried may be public data, or may be obtained through a separate survey for maintenance of the underground pipeline.

As described above, in the present invention, a plurality of points on the outskirts (surface) of an underground pipeline are secured with specific point cloud data as three-dimensional coordinates, respectively, and the point cloud is rendered and expressed in augmented reality on the actual screen of the smart terminal.

Since the point cloud data is specified by three-dimensional coordinates of latitude, longitude, and elevation, the elevation of the underground pipeline is reflected unlike the existing two-dimensional planar data. Accordingly, there is an advantage that the underground pipeline expressed in augmented reality is expressed in accordance with the reality even on sloping terrain.

Also, by rendering the point cloud data and adding color to each point, the underground pipeline is three-dimensionally expressed. It does not require additional processes such as 3D object styling for the underground pipeline.

As point cloud data is used, additional processes such as 3D modeling or object styling are all unnecessary, so there is an advantage that faster processing is possible as the operation speed is improved in a smart terminal or server.

On the other hand, in the present invention, in order to correct an error in azimuth sensing in the smart terminal, a reference point that knows the actual location coordinates is set in an area adjacent to the underground pipeline buried so that the underground pipeline matches the actual location in augmented reality on the display of the smart terminal. It has the advantage that it can be implemented.

For reference, the underground pipeline maintenance method according to the present invention may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be provided by being stored in a non-transitory computer readable medium. Here, the non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, etc., and can be read by a device. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

The protection scope of the present invention is not limited to the description and expression of the embodiments explicitly described above. In addition, it is added once again that the protection scope of the present invention cannot be limited due to obvious changes or substitutions in the technical field to which the present invention pertains.

Claims (4)

(a) For the installation of a new pipeline or maintenance of an existing pipeline, the three-dimensional coordinates of multiple points on the surface of the underground pipeline are measured with a point cloud generator in the state that the underground pipeline is exposed to the outside, and the underground pipeline point cloud data is generated to do;
(b) measuring the position information of the smart terminal connected through the server and the communication network, and the posture information including the azimuth and angle;
(c) calculating the position coordinates of the photographing area photographed by the camera of the smart terminal by using the location information and the posture information of the smart terminal; and
(d) An image obtained by comparing the location coordinates of the underground pipeline point cloud data with the location coordinates of the photographing area to detect points included in the photographing area, and using the detected underground pipeline point cloud data to render the underground pipeline Superimposed on the actual image of the smart terminal to express in augmented reality; Maintenance method of the underground pipeline comprising a. The method of claim 1,
setting at least one reference point in an adjacent area through which the underground pipe passes in a buried state while covering the upper part of the underground pipe, and acquiring the actual positional coordinates of the reference point;
detecting the reference point through image analysis on the actual screen of the smart terminal; and
Compensating the azimuth value of the smart terminal by comparing the coordinate value of the reference point identified in the smart terminal with the actual position coordinate; 3. The method of claim 2,
The reference point is that a specific pattern is shaped,
The reference image taken by the reference point is stored in the server,
Using the image detection algorithm of the smart terminal, when the reference point is photographed on the actual screen of the smart terminal, the reference point is detected by comparing the captured image with the reference image transmitted from the server maintenance method. 3. The method of claim 2,
3D coordinates of a plurality of feature points on a nearby surface in a state in which the underground pipeline is buried are measured with a point cloud generating device, and the plurality of feature points are set as the reference point.

Images