KR102077934B1 - Method for generating alignment data for virtual retrofitting object using video and Terminal device for performing the same - Google Patents

Abstract

Provided are an alignment data creation method for virtually retrofitting an object with a video and a terminal device therefor. The method includes the following steps of: aligning pre-obtained three-dimensional global point cloud data of the object and real time-obtained three-dimensional partial point cloud data about a specific part of the object; creating first alignment data by converting the aligned point cloud data into a two-dimensional format; extracting a two-dimensional frame corresponding to the three-dimensional partial point cloud data from a real time-obtained camera image; creating second alignment data by aligning the first alignment data and the extracted two-dimensional frame; and creating third alignment data by aligning the second alignment data and a pre-created CAD model of the object. Meanwhile, an operation for the virtual retrofitting is performed based on the third alignment data.

Description

Method for generating alignment data for virtual retrofitting object using video and terminal device for performing the same

Embodiments of the present invention relate to a method for generating alignment data for performing virtual retrofitting of an object using video, and a terminal device for performing the same. More specifically, to reduce the amount of computation and to quickly arrange for virtual retrofitting. A terminal device and method for generating data.

In industrial plants, such as thermal, petrochemical and nuclear power plants, the planning of routes for transporting pipes, equipment and other specific objects to areas within the plant is generally difficult. In addition, factory maintenance, re-installation and upgrade work must take the risk of unreliable coupling caused by the complex layout in the plant. In addition, the use of the 3D model in the field in the existing way is likely to be inconsistent with the actual model.

While techniques using current laser scanning techniques can obtain detailed information of the actual model surface, the task of retrofitting plant equipment has the disadvantage of being a time consuming and tedious manual task.

In order to solve the problems of the prior art as described above, the present invention is to propose a method for reducing the amount of calculation using the video obtained through the camera and quickly generating alignment data for virtual retrofitting, and a terminal device for performing the same. do.

Other objects of the present invention may be derived by those skilled in the art through the following examples.

According to a preferred embodiment of the present invention to achieve the above object, a terminal device for performing alignment data generation for virtual retrofitting of an object, the terminal device comprising: a memory storing a computer-readable command; And a processor implemented to execute the instructions, wherein the processor is configured to align 3D global point cloud data of the object obtained in advance with 3D partial point cloud data for a specific portion of the object obtained in real time; And generating first alignment data by converting the aligned point cloud data in two dimensions, extracting a 2D frame corresponding to the 3D partial point cloud data from an image of a camera obtained in real time, and extracting the extracted 2D frame. The second alignment data is generated by aligning the first alignment data, and the third alignment data is generated by aligning the CAD model of the object and the second alignment data, which have been previously generated, using the third alignment data. An operation for virtual repeating can be performed.

The 3D partial point cloud is obtained through a laser scanner,

The camera and the laser scanner are arranged to face in the same direction in one device, and the processor is configured to calibrate the point cloud data of the laser scanner and the pixels of the camera to generate the second alignment data. Can be performed.

The processor generates the first alignment data by projecting the aligned point cloud data onto an image plane defined by the parameter of the camera, wherein the parameter of the camera is the focal length of the camera according to the 2D frame, the 2D It can include the width and height of the frame.

The projection process may be performed based on the following equation.

Here, (x, y) is coordinates projected in two dimensions, (X, Y, Z) is coordinates of three-dimensional point cloud data, (X ₀ , Y ₀ , Z ₀ ) is among three-dimensional point cloud data The coordinates of one cloud data, ω, φ, and κ, mean roll value, yaw value and pitch value for rotation.

In addition, according to another embodiment of the present invention, in the alignment data generation method for the repeating of the object performed by the processor, 3D global point cloud data of the object obtained in advance, and specifying the object obtained in real time Sorting 3D part point cloud data for the part; Generating first alignment data by converting the aligned point cloud data in two dimensions; Extracting a 2D frame corresponding to the 3D partial point cloud data from an image of a camera acquired in real time; Generating second alignment data by aligning the extracted 2D frame with the first alignment data; And generating a third alignment data by aligning the CAD model of the object and the second alignment data generated in advance. 3. There is provided a method of generating sorted data.

According to the present invention, there is an advantage of reducing the amount of computation during the virtual repeating operation and quickly generating alignment data.

In addition, the effect of this invention is not limited to the above-mentioned effect, It should be understood that it includes all the effects which can be deduced from the structure of the invention described in the detailed description of this invention or a claim.

1 is a view showing a schematic configuration of a terminal device according to an embodiment of the present invention.
2 is a conceptual diagram illustrating a method of generating alignment data for virtual retrofitting of an object according to an embodiment of the present invention.
3 is a flowchart illustrating a method of generating alignment data for virtual retrofitting of an object according to an embodiment of the present invention.
4 to 9 are diagrams for explaining a process of generating alignment data according to the present invention.

As used herein, the singular forms "a", "an" and "the" include plural forms unless the context clearly indicates otherwise. In this specification, terms such as “consisting of” or “comprising” should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or some steps It should be construed that it may not be included or may further include additional components or steps. In addition, the terms "... unit", "module", etc. described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. .

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing a schematic configuration of a terminal device according to an embodiment of the present invention.

Referring to FIG. 1, a terminal device 100 according to an embodiment of the present invention includes a memory 110, a processor 120, a display 130, and a user interface 140.

The memory 110 may be a volatile and / or nonvolatile memory, and stores instructions or data related to at least one other component of the terminal device 100. In particular, the memory unit 110 may store instructions or data related to a computer program, an application, and a recording medium for performing an operation of virtual repeating.

The processor 120 may include one or more of a central processing unit, an application processor, or a communication processor. For example, the processor 120 may execute control and / or calculation or data processing of at least one other component of the terminal device 100. In particular, the processor 120 may execute instructions related to the execution of the computer program or application.

The display 130 may include a liquid crystal display (LCD), a light emitting diode display (LED), an organic light emitting diode display (OLED), and the like, and is a device capable of displaying an image or an image frame to a user. In particular, the display 130 may be a head mounted display.

The user interface 140 receives an input signal of a user required to perform virtual repeating. For example, the user interface 140 may be a device including a keyboard, a mouse, and a motion tracking sensor.

Hereinafter, a method of generating alignment data for virtual retrofitting of an object, which is an operation performed by the terminal device 100, in particular, the processor 120, will be described in detail with reference to the following drawings.

2 is a conceptual diagram illustrating a method of generating alignment data for virtual retrofitting of an object according to an embodiment of the present invention.

Referring to FIG. 2, the present invention relates to i) aligning 3D global point cloud data of a previously acquired object with 3D partial point cloud data for a specific part of an object obtained in real time, and aligning the aligned point cloud data in two dimensions. Generating a first alignment data by converting the data into a second image; and ii) extracting a 2D frame corresponding to the 3D partial point cloud data from an image of a camera acquired in real time, and aligning the extracted 2D frame and the first alignment data. And generating the third alignment data by aligning the CAD model of the pre-generated object and the second alignment data. At this time, an operation for virtual retrofitting is performed using the third alignment data.

3 is a flowchart illustrating a method of generating alignment data for virtual retrofitting of an object according to an embodiment of the present invention.

Hereinafter, a process performed for each step will be described in detail.

First, in step 310, the 3D global point cloud data of the object obtained in advance is aligned with the 3D partial point cloud data of a specific part of the object obtained in real time. At this time, the object may be an industrial plant.

More specifically, the global point cloud data of the object can be obtained in three dimensions in advance by a laser scanner. In one example, the laser scanner may include a Lidar sensor and an IMU sensor.

In this case, the global point cloud data may be preprocessed in advance, and the preprocessed global point cloud data may be global point cloud data from which noise is removed. On the other hand, by using the laser scanner's calibration position and the corresponding orientation, it acquires an array of depth values that can be transformed into a three-dimensional position in the scanner coordinate system, and four-position and orientation scanner calibration is required to obtain a global point cloud model. .

Hereinafter, preprocessed global point cloud data is used for convenience of description. 4 (a) shows a sample environment, and FIG. 4 (b) shows preprocessed global point cloud data.

In addition, the partial point cloud data of the object may be obtained in three dimensions by a laser scanner in real time. In FIG. 4C, partial point cloud data is illustrated.

Through the above description, in step 310, alignment point cloud data including geometric information is generated by aligning pre-processed point cloud data with partial point cloud data of an object obtained in real time.

More specifically, to align the preprocessed point cloud data in the real model, you need to know the relative position and orientation of the real model in the laser scanner coordinates. The position and orientation of the actual model from the laser scanner coordinate system can be estimated using the partial point cloud data of the object obtained in the laser scanner in real time.

At this time, consistently aligning the views of the various three-dimensional point cloud data into a complete model is called "registration", and the preprocessed point cloud data is aligned with the relative position and orientation of the global dimension physical model.

One of the most commonly used sorting methods for sorting processed point cloud data is an Iterative Closest Point (ICP) algorithm. This algorithm minimizes the distance error metric and finds the best conversion between two data sets through repeated attempts. The ICP assumes that all points match and proceeds with the alignment using the 3D points closest to the set of source and target models. Since the ICP algorithm is an iterative approach, the time complexity of the algorithm depends on the number of iterations needed to find the optimal transformation matrix.

Therefore, in order to reduce the time complexity of the ICP algorithm, in step 310, an optimized GICP (Optimized GICP), which is an alignment algorithm based on a generalized iterative close point (GICP), may be used. This will be described in more detail as follows.

The GICP algorithm is a sorting algorithm used to align preprocessed point cloud data and partial point cloud data. In this case, the GICP algorithm applies a probabilistic model in a step of minimizing a cost function (transfer function) for estimating the attitude relationship between three-dimensional point cloud data. First, the GICP algorithm estimates nearest neighbor correspondences, and the preprocessed global point cloud data and real-time partial point cloud data are indexed with corresponding points having similar index values. At this time, points that do not correspond are deleted.

According to an embodiment of the present invention, the global point cloud data is sampled to reduce the number of point cloud data included in the global point cloud data, and the aligned point cloud data using the sampled global point cloud data and the partial point cloud data. Can be generated. In this case, the aligned point cloud data may be generated by applying the sampled global point cloud data and the partial point cloud data to an optimal GICP algorithm.

According to an embodiment of the present invention, the processor 120 may randomly select the point cloud data included in the global point cloud data and perform sampling. This may be expressed as Equation 1 below.

Here, P ₀ is sampled global point cloud data, P _P is preprocessed global point cloud data, and randomsample () denotes a random sampling function.

After performing the sampling, the processor 120 performs an alignment process using an optimal GICP algorithm. In this case, the preprocessed global point cloud data is not used, but the sampled global point cloud data is used, thereby reducing the time required for sorting the point cloud data.

According to an embodiment of the present invention, the processor 120 calculates a transformation matrix for alignment of the point cloud data using the sampled global point cloud data and the partial point cloud data, and uses the transformation matrix. A homogeneous transformation matrix may be calculated and alignment point cloud data may be generated using the homogeneous transformation matrix and global point cloud data.

In more detail, the transformation matrix is a cost function according to the GICP algorithm, and may be expressed as Equation 2 below.

는 a_i의 주변의 포인트 클라우드의 분포 형태를 설명하는 공분산 행렬,

는 b_i의 주변의 포인트 클라우드의 분포 형태를 설명하는 공분산 행렬을 각각 의미한다. Where T is the transformation matrix, i is the index of the point cloud, a _i is the point cloud data included in the sampled global point cloud data, b _i is the point cloud data included in the partial point cloud data,

Is a covariance matrix describing the distribution form of the point cloud around a _i ,

Denotes a covariance matrix describing the distribution form of the point cloud around b _i .

The homogeneous transformation matrix may be expressed by Equation 3 below. At this time, the homogeneous transformation matrix is for performing optimal rotation and parallel movement.

는 호모지니어스 변환 행렬, c는 cos 함수, s는 sin 함수,

,

,

각각은 회전에 대한 롤 값, 요 값, 피치 값,

,

,

각각은 변환 행렬의 x, y, z 값을 의미한다. here,

Is the homogenous transformation matrix, c is the cos function, s is the sin function,

,

,

Each of the roll value, yaw value, pitch value,

,

,

Each means the x, y, and z values of the transformation matrix.

That is, the transformation matrix is calculated by considering the optimal preprocessed point cloud data P ₀ and real time partial point cloud data P _r using Equation 3.

In addition, the alignment point cloud data generated using the homogeneous transformation matrix and the global point cloud data may be expressed by Equation 4 below.

는 정렬 포인트 클라우드 데이터,

는 전역 포인트 클라우드 데이터,

는 호모지니어스 변환 행렬을 각각 의미한다. 이 때,

는 앞서 언급한 바와 같이 샘플링된 전역 포인트 클라우드 데이터에 포함된 포인트 클라우드 데이터와 b_i는 부분 포인트 클라우드 데이터에 포함된 포인트 클라우드 데이터의 정렬에 따라 획득될 수 있다. here,

Sorting point cloud data,

Global point cloud data,

Denotes homogeneous transformation matrices, respectively. At this time,

As described above, point cloud data included in the sampled global point cloud data and b _i may be obtained according to the alignment of the point cloud data included in the partial point cloud data.

5 shows a flow of a process of generating alignment point cloud data.

Next, in step 320, the 2D frame corresponding to the 3D partial point cloud data is extracted from the image of the camera obtained in real time.

More specifically, the present invention simultaneously acquires 3D partial point cloud data and camera images using a single device. This is as shown in FIG. In this case, the camera and the laser scanner may be disposed to face the same direction in one device to generate image frames and point cloud data continuously at a plurality of viewpoints. In operation 320, the 2D frame corresponding to the 3D partial point cloud data is extracted from the frames of the camera image.

In operation 320, a camera parameter is calculated based on the plurality of frames extracted from the camera image. In this case, the parameters of the camera may include a focal length of the camera according to the 2D frame, a width and a height of the 2D frame.

Subsequently, in step 330, the aligned point cloud data is transformed in two dimensions to generate first alignment data.

At this time, according to an embodiment of the present invention, in step 330, the processor 120 may generate the first alignment data by projecting the alignment point cloud data on the image plane defined by the parameters of the camera. This will be described in more detail as follows.

Point cloud data acquired from the LiDAR sensor is defined in the world coordinate system. In order to generate 3D RGB images by converting 3D point cloud data in two dimensions, the present invention projects 3D points on an image plane defined by camera parameters.

The center of the lens of the camera is regarded as the origin of the camera coordinate system as shown in FIG. The Z axis direction is perpendicular to the XY plane of the camera coordinate system and has a positive direction towards the object to be imaged. 3D point cloud data is then projected onto the image plane defined by the camera's sensor. At this time, as shown in Figure 6, the LiDAR sensor and the camera sensor is located at the same distance from the object. That is, the distance of the LiDAR from the object corresponds to the focal length of the camera at the origin of the camera coordinate system. Further adjustments to the difference between the projection parameters of the LiDAR sensor and the camera are described below.

Also, as shown in Fig. 7, the projection origin of the camera coordinate system is generated when the acquired image is defined at the center (X, Y, f). The camera focal length f, width w and height h of the generated 2D frame image are used to define the image plane for the projection of the point cloud data. FIG. 8 (a) shows the 3D point cloud data of the pipeline, and FIG. 8 (b) shows the 2D point cloud data of the pipeline projected from the 3D point cloud data.

According to an embodiment of the present invention, the projection process may be performed based on Equation 5 below.

Here, (x, y) is coordinates projected in two dimensions, (X, Y, Z) is coordinates of three-dimensional point cloud data, (X ₀ , Y ₀ , Z ₀ ) is among three-dimensional point cloud data Coordinates, ω, φ, and κ of one cloud data mean roll value, yaw value, and pitch value for rotation.

The processor 120 may perform an operation of calibrating the pixel of the camera and the point cloud data of the laser scanner. This will be described in detail below.

Calibration between the LiDAR sensor and the camera is necessary to accurately align the point cloud data in the image as described below. This is because the image acquired through the camera is an RGB image, which is different from the image generated from the point cloud data acquired through the LiDAR sensor.

Calibrating the camera with LiDAR is used to integrate the image and point cloud data, confirming the match between the pixel in the camera image and the point on the LiDAR sensor. Marker-based correction methods efficiently align cameras with LiDARs. The geometrical displacement of the data from the two sensors is adjusted through the marker positions detected at the two sensors.

The direction of the scanning platform is obtained from an IMU sensor placed adjacent to the camera (see FIG. 6), and the IMU sensor outputs the 3D direction using various representations such as unit quaternion, Euler angles and rotation matrix. For the initial alignment of preprocessed point cloud data for the scanner platform, we considered unit quaternion output to the IMU sensor. This is expressed as in Equation 6.

는 IMU 센서를 이용한 정렬 후의 포인트 클라우드 데이터,

는 정렬 전의 포인트 클라우드 데이터,

는 카메라의 초점 거리를 가지는 카메라 투영 행렬,

는 보정 매트릭스,

및

는 이미지의 가로 및 세로의 길이,

,

,

는 LiDAR와 카메라의 변위 벡터를 각각 의미한다.here,

Is the point cloud data after alignment using the IMU sensor,

Is the point cloud data before sorting,

Is the camera projection matrix with the focal length of the camera,

Is the correction matrix,

And

Is the width and height of the image,

,

,

Denotes displacement vectors of LiDAR and camera, respectively.

Based on the above description, in step 340, the extracted 2D frame and the first alignment data are aligned to generate second alignment data. 9 illustrates an example of aligning the 2D frame and the first alignment data to generate second alignment data.

In this case, the intensity value of the point cloud image may be updated to the camera frame using Equation 7 below.

Here, w is the width of the 2D frame image, h is the width of the 2D frame image, C _I (i, j) means the pixels of the image of the point cloud data, respectively.

The intensity of the F _I (i, j) pixel of the 2D frame image is then replaced by the intensity of the C _I (i, j) pixel of the image of the corresponding point cloud data.

In operation 350, the 3D CAD model of the pre-generated object and the second alignment data are aligned to generate third alignment data. At this time, an operation for virtual retrofitting may be performed using the third alignment data.

In short, the present invention is a framework of a new four-to-one virtual retrofitting operation aimed at virtually retrofitting and upgrading industrial plant facilities. This work begins by organizing the partial point cloud data of the physical environment obtained by laser scanning and the entire point cloud model previously measured and combined with the same coordinates. Photorealistic images are superimposed based on the sorted data, virtual repeating operations are performed on the generated CAD model, and the entire data is effectively visualized in a virtual reality device through the display 130 and the user interface 140. Proceeds. This procedure completes the construction of a proactive refitting model that allows you to understand and manage existing site plant information for final refitting in the field.

In addition, embodiments of the present invention may be implemented in the form of program instructions that may be executed by various computer means to be recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions recorded on the media may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks such as floppy disks. Examples of program instructions such as magneto-optical, ROM, RAM, flash memory, etc. may be executed by a computer using an interpreter as well as machine code such as produced by a compiler. Contains high-level language codes. The hardware device described above may be configured to operate as one or more software modules to perform the operations of one embodiment of the present invention, and vice versa.

In the present invention as described above has been described by the specific embodiments, such as specific components and limited embodiments and drawings, but this is provided to help the overall understanding of the present invention, the present invention is not limited to the above embodiments, Various modifications and variations can be made by those skilled in the art to which the present invention pertains. Therefore, the spirit of the present invention should not be limited to the described embodiments, and all of the equivalents and equivalents of the claims, as well as the appended claims, will belong to the scope of the present invention. .

Claims (6)

A terminal device for performing alignment data generation for virtual retrofitting of an object, the terminal device comprising:
Memory for storing computer-readable instructions; And
A processor implemented to execute the instructions,
The processor,
The 3D global point cloud data of the object obtained in advance and the 3D partial point cloud data for a specific part of the object obtained in real time are aligned, and the aligned point cloud data is converted into two dimensions to obtain the first alignment data. Create,
Extracting a 2D frame corresponding to the 3D partial point cloud data from an image of a camera obtained in real time, generating second alignment data by aligning the extracted 2D frame with the first alignment data,
The third alignment data is generated by aligning the CAD model of the object and the second alignment data generated in advance.
And performing the operation for the virtual repeating using the third alignment data. The method of claim 1,
The 3D partial point cloud is obtained through a laser scanner,
The camera and the laser scanner are arranged to face in the same direction in one device,
And the processor is configured to calibrate the pixel of the camera and the point cloud data of the laser scanner to generate the second alignment data. The method of claim 1,
The processor generates the first alignment data by projecting the aligned point cloud data onto an image plane defined by the parameter of the camera, wherein the parameter of the camera is the focal length of the camera according to the 2D frame, the 2D Terminal device comprising a width and height of the frame. The method of claim 3,
The projection process is performed based on the following equation.

Here, (x, y) is coordinates projected in two dimensions, (X, Y, Z) is coordinates of three-dimensional point cloud data, (X ₀ , Y ₀ , Z ₀ ) is among three-dimensional point cloud data The coordinates of one cloud data, ω, φ, and κ, mean roll value, yaw value and pitch value for rotation. A method of generating alignment data for virtual retrofitting of an object performed by a processor,
Aligning 3D global point cloud data of the object obtained in advance with 3D partial point cloud data for a specific part of the object obtained in real time;
Generating first alignment data by converting the aligned point cloud data in two dimensions;
Extracting a 2D frame corresponding to the 3D partial point cloud data from an image of a camera acquired in real time;
Generating second alignment data by aligning the extracted 2D frame with the first alignment data; And
And arranging the CAD model of the object and the second alignment data generated in advance, to generate third alignment data.
And performing an operation for virtual retrofitting of the object using the third alignment data. A computer-readable recording medium having recorded thereon a program for performing the method of claim 5.

Images