KR20210083437A - Method and system for combining image textures into point cloud - Google Patents

Abstract

The present invention relates to a method and system for combining an image texture with a point cloud. The method of the present invention comprises the steps of: interpolating a road surface point cloud to configure at least one triangular surface; calculating three points in an aerial photograph, which correspond to three points constituting the surface; and allocating a triangular texture composed of the three points in the aerial photograph as a texture of the surface. In accordance with the present invention, a method and system for combining a point cloud and an aerial photograph through a texture, not through points, are provided to solve problems which a conventional mobile mapping system (MMS) has, such as a mismatching phenomenon, a hiding phenomenon, and the like. In addition, a problem that there is no information on color and brightness between points in the conventional MMS can be solved.

Description

METHOD AND SYSTEM FOR COMBINING IMAGE TEXTURES INTO POINT CLOUD

The present invention relates to a method and system for combining an image texture with a point cloud, and more particularly, to a method and system for combining an aerial photographed image texture with a previously acquired road surface point cloud to create an immersive three-dimensional road model is about

Recently, as the need for a smart city and a three-dimensional city model increases worldwide, a technology for generating a three-dimensional road model, which is an essential element constituting a city, has been widely studied. The construction of 3D road models mainly uses MMS (Mobile Mapping System), which integrates various sensors such as cameras, GPS, inertial measurement devices, and laser scanners and mounts them on vehicles, It is a system that simultaneously acquires location information and image information of geographical features. MMS is advantageous in terms of cost and efficiency compared to existing field surveys, and various studies including Korean Patent No. 10-2017-0115778 are being conducted because it can shorten the revision and update cycle of numerical maps or 3D data.

MMS uses LiDAR to emit lasers forward, backward, and to the side, and collects the distribution of surrounding objects in the form of a point cloud using the reflected return time. Then, color information, brightness information, and the like are inputted to each point (point) of the point cloud from the camera image of the road taken from the vehicle. Then, there is no color information between each point of the point cloud, but there is no color information, but since each point is very dense, it looks like an image with no empty space to the human eye. However, the MMS data formed in this way has limitations as immersive images that have been recently demanded, and in particular, when enlarged, the limitations become prominent.

In addition, in the case of MMS data, “matching distortion” occurs between the image and the point cloud due to the perspective that exists in the image taken while facing the front, and some points in the point cloud are caused by the “occlusion phenomenon” by surrounding vehicles, trees, etc. There are many cases in which color information is not input to the display (see FIG. 1). In order to solve these problems, a plurality of vehicle-captured images are required. In this case, there is a significant difference in color and brightness values between the various images, which causes another problem in which correction must be performed separately.

KR 10-2017-0115778 A.

The present invention was devised to solve the problems of the prior art as described above, and by providing a method and system for combining road surface point clouds and aerial photographed images acquired from different platforms, the matching distortion phenomenon of conventional MMS data And it aims to solve problems such as occlusion phenomenon.

In addition, by providing a method and system for combining a point cloud and an aerial photographed image with a texture rather than a point, the conventional MMS solves the problem that there is no color and brightness information between the point and the realistic three-dimensional road model. Another purpose is to create

According to an embodiment of the present invention for achieving the above technical problem, the step of interpolating the road surface point cloud to construct at least one triangular surface, three points on the aerial shot image corresponding to the three points constituting the surface There is provided a method of combining an image texture to a point cloud, comprising the steps of calculating , and allocating a triangular texture composed of three points on the aerial shot image as the texture of the surface.

The calculating includes constructing a sensor model from the external expression element and the inner expression element of the image, and inversely projecting three points constituting the surface onto the image, wherein the inverse projection is the sensor model. Available.

The interpolation may use a Poisson Reconstruction or a triangular network construction technique.

The point cloud and the aerial photographed image may be acquired from different platforms.

The sensor model may use a collinear conditional expression or direct linear transformation (DLT).

Further, according to another preferred embodiment, there is provided a computer program stored in a computer-readable recording medium for executing the method according to each method described above.

In addition, according to another preferred embodiment, there is provided a computer-readable recording medium in which a program for executing each of the above-described methods is recorded.

According to another embodiment of the present invention for achieving the above technical problem, an interpolator constituting at least one triangular surface by interpolating a road surface point cloud, an aerial captured image corresponding to three points constituting the surface A system for combining an image texture to a point cloud is provided, comprising: an image point calculator for calculating three points on the image; and a texture allocator for allocating a triangular texture composed of three points on the aerial shot image as a texture of the surface.

The image point calculation unit may construct a sensor model from the external expression element and the inner expression element of the image, and use the sensor model to back-project three points constituting the surface to the image.

The interpolation may use a Poisson technique or a triangular network construction technique.

The point cloud and the aerial photographed image may be acquired from different platforms.

As described above, according to the present invention, by providing a method and system for combining road surface point clouds and aerial photographed images acquired from different platforms, problems such as matching distortion and occlusion of conventional MMS data are solved. has the effect of

In addition, by providing a method and system for combining a point cloud and an aerial photographed image with a texture rather than a point, there is an effect of solving the problem that the conventional MMS does not have any color and brightness information between the points.

1 is a diagram illustrating an occlusion phenomenon (a portion indicated by an ellipse) due to a vehicle in MMS data.
2 is a flowchart illustrating a specific example of a method of combining an image texture to a point cloud according to an embodiment of the present invention.
3 is a diagram illustrating a point cloud (a) and an aerial photographed image (b).
4 is a block diagram illustrating a process of interpolating a point cloud using the Poisson technique.
5 is a diagram illustrating an interpolated point cloud according to an embodiment of the present invention.
6 is a diagram illustrating a triangle texture on a corresponding aerial photographed image (b) by back-projecting a triangle of an interpolated point cloud (a) to an aerial photographed image according to an embodiment of the present invention.
7 is a diagram illustrating a result of combining an image texture with a point cloud according to an embodiment of the present invention.
8 is a block diagram showing the configuration of a system for combining an image texture to a point cloud according to another embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily carry out the present invention. In the following description, only the parts necessary for understanding the operation according to the embodiment of the present invention are shown and described, and the illustration and description of other parts are omitted so as not to obscure the gist of the present invention. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein.

In addition, the terms or words used in the present specification and claims described below should not be construed as being limited to conventional or dictionary meanings, and meanings consistent with the technical spirit of the present invention so that the present invention can be most appropriately expressed. and should be interpreted as a concept.

For simplicity of explanation, one or more methods are shown and described herein as a series of steps, by way of example, by way of example, in the form of a flowchart or flow chart, but the invention is not limited by the order of steps, since the invention is not limited thereto. It will be appreciated that, in accordance with the present disclosure, it may be performed in a different order or concurrently with other steps than those shown and described herein. Moreover, not all illustrated steps may be required to implement a method in accordance with the present invention.

In terms of terms used herein, the singular expression should be understood to include the plural expression unless the context clearly dictates otherwise, and terms such as "comprises" include the embodied feature, number, step, operation, element. , parts or combinations thereof are to be understood, but not to exclude the possibility of the presence or addition of one or more other features or numbers, step operation components, parts or combinations thereof.

Prior to a detailed description of the drawings, it is intended to clarify that the classification of the constituent parts in the present specification is merely a division according to the main function that each constituent unit is responsible for. That is, two or more components to be described below may be combined into one component, or one component may be divided into two or more for each more subdivided function. In addition, each of the constituent units to be described below may additionally perform some or all of the functions of the other constituent units in addition to the main function it is responsible for, and may additionally perform some or all of the functions of the other constituent units. Of course, it may be carried out by being dedicated to it.

Throughout the specification, the point cloud and aerial shot images used in the present invention are acquired from different platforms while the MMS simultaneously acquires the point cloud (location information) and image information on the same platform. That is, they were acquired from different moving objects or from different viewpoints. For example, if the point cloud is acquired by a ground scanning method from a car, the aerial photographed image may be acquired from an aircraft or a UAV. As another example, if the point cloud is acquired from the aircraft, the aerial photographed image may be acquired from the aircraft at a time different from the time point at which the point cloud was acquired.

And, if the MMS is a system that simultaneously acquires location information and image information and inputs the color and brightness values of an image to the location information in units of “points,” the system according to the present invention can “ It is a system that combines textures in units of “surface”.

On the other hand, the reason for using the aerial photographed image instead of using the ground photographed image in the present invention is that the ground photographed image acquired from the vehicle has a very narrow area that can be covered compared to the aerial photographed image, so a large number of vehicle photographed images are required. This is because another problem arises that correction must be performed separately due to a large difference in color and brightness values between multiple images. In addition, in ground-captured images, “matching distortion” occurs between the image and the point cloud due to perspective, and color information cannot be input to some areas due to “occlusion” by surrounding vehicles, trees, etc. to be.

Hereinafter, the above-described contents will be described in detail with reference to the drawings.

2 is a flowchart illustrating a specific example of a method of combining an image texture to a point cloud according to an embodiment of the present invention.

Referring to FIG. 2 , the system 10 for combining an image texture to a point cloud according to the present invention may construct at least one triangular surface by interpolating a road surface point cloud in step S110.

Point clouds are often acquired by ground scanning in vehicles, but can also be acquired from aircraft or UAVs. 3( a ) shows an example of a point cloud.

As a method of interpolating the point cloud, a triangular network construction method or a Poisson method may be used.

A surface made of numerous triangles can be constructed by interpolating the point cloud using the triangular network construction technique or the Poisson technique. A general flowchart of the Poisson technique is shown in FIG. 4 . A detailed description of the triangular network construction technique or the Poisson technique itself will be omitted.

When a point cloud is interpolated using a triangular network construction technique or a Poisson technique, the interpolated point cloud is composed of numerous “surfaces” of triangles, which are closely connected with each other.

Referring to FIG. 5 , as an “interpolated point cloud” according to an embodiment of the present invention, the point cloud of FIG. 3(a) is interpolated by the above-described triangular network construction technique or Poisson technique.

When the point cloud is composed of the surface of numerous triangles by the above-described interpolation, steps S120 and S130 to search for a triangle corresponding (corresponding) to the surface of each triangle in the aerial shot image (see Fig. 3(b)). will go through

In step S120, a sensor model may be constructed from the external expression element and the inner expression element of the aerial photographed image.

The sensor model is a model representing the relationship between a photographed image, a photographing sensor (camera), and a coordinate system on the ground (longitude and latitude coordinate system, etc.). It is a formula that can obtain column and row coordinates of an image using ground coordinates. An action such as obtaining the ground coordinates corresponding to each pixel of the image or obtaining the image coordinates corresponding to the ground coordinates can be freely performed using the equation.

In general, calculating the ground coordinates from the image coordinates is called forward transformation, and conversely, calculating the image coordinates from the ground coordinates is called inverse transformation. Therefore, in the present invention, since the corresponding image coordinates are obtained from the ground coordinates of the vertices of each triangle of the interpolated point cloud, the inverse transformation formula may be used.

A sensor model constructed on the basis of a collinear conditional expression that an arbitrary point A in space, a point a of an image corresponding to it, and a photographing center L of a camera that has taken an image must exist on the same line is as shown in Equation 1 above.

는 공간상의 임의의 점A의 3차원 지상 좌표이고,

는 임의의 점 A에 대응되는 영상 상의 점 a의 영상 좌표이고,

는 영상을 촬영한 카메라의 투영 중심 L의 3차원 지상 좌표이다.In <Equation 1>

is the three-dimensional ground coordinates of an arbitrary point A in space,

is the image coordinate of the point a on the image corresponding to the arbitrary point A,

is the three-dimensional ground coordinates of the projection center L of the camera that captured the image.

In addition, the sensor model may be constructed using a DLT (Direct Linear Transformation) method. The DLT expresses the relationship between the three-dimensional coordinates and the photographic coordinates as an expression, and is as in <Equation 2>.

는 영상 좌표이고,

는 정규화를 위한 보정계수이고,

는 지상 좌표이고, M은 4 x 3 크기의 변환행렬이다.In <Equation 2>

is the image coordinate,

is the correction factor for normalization,

is the ground coordinates, and M is a transformation matrix with a size of 4 x 3.

In step S130, using the sensor model configured in step S120, three points constituting the triangular surface of the point cloud may be back-projected onto the aerial photographed image to calculate three points on the aerial photographed image. That is, the image coordinates can be obtained from the ground coordinates by using <Equation 1> or <Equation 2>, which are inverse transformation equations for inverse projection.

Referring to FIG. 6 , (a) is an interpolated point cloud, (b) is an aerial shot image, and a triangle on the image corresponding to one triangle of the interpolated point cloud (indicated by an arrow) is shown. The three points of the triangle of FIG. 6(b) may be obtained by back-projecting from the three points of the triangle of FIG. 6(a) in steps S120 and S130.

In step S140 , a triangular texture composed of three points on the aerial captured image may be assigned as a texture of a triangular surface of the interpolated point cloud.

If the triangle texture of FIG. 6(b) is assigned (applied or copied) to the triangle of FIG. 6(a), and this process is performed for all triangles of the interpolated point cloud, color and brightness, etc. This input 3D road model is completed.

Since the road model created through the above process is textured on the “face”, it becomes a much better realistic model than the general MMS model in which color information is input at “points”. In addition, compared to the MMS model in which there is no information between the points of the point cloud, the model of the present invention has an effect that there is no empty space even when enlarged because the points are filled with texture.

8 is a block diagram showing the configuration of a system 10 for combining an image texture to a point cloud according to another embodiment of the present invention.

Referring to FIG. 8 , the system 10 for combining an image texture to a point cloud may include an interpolation unit 11 , an image point calculation unit 13 , and a texture allocator 15 .

The interpolator 11 may construct at least one triangular surface by interpolating the road surface point cloud 20 .

When the point cloud 20 is interpolated by the triangular network construction technique or the Poisson technique, the interpolated point cloud is composed of a number of “surfaces” of triangles, which are closely and seamlessly connected.

The image point calculation unit 13 may construct a sensor model from the external expression element and the inner expression element of the aerial photographed image 30 .

The sensor model may be configured based on the collinear condition expression as in <Equation 1>, or may be constructed using the DLT method as in <Equation 2>.

In addition, the image point calculation unit 13 reverse-projects three points constituting the triangular surface of the point cloud 20 on the aerial photographed image 30 using the sensor model to calculate the three points on the aerial photographed image 30 . can be calculated That is, the image coordinates can be obtained from the ground coordinates by using <Equation 1> or <Equation 2>, which are inverse transformation equations for inverse projection.

The texture allocator 15 may allocate a triangular texture composed of three points on the aerial captured image 30 as a texture of the triangular surface of the interpolated point cloud 20 . When this allocation process is performed for all triangles of the interpolated point cloud 20, a three-dimensional road model in which color and brightness are inputted as shown in FIG. 7 is completed.

As described above, according to the present embodiments, problems such as matching distortion and occlusion of conventional MMS data can be solved by combining road surface point clouds and aerial photographed images acquired from different platforms.

In addition, by combining a point cloud and an aerial photographed image with a texture rather than a point, it is possible to solve the problem that the conventional MMS does not have any color and brightness information between the points.

In addition, the embodiment of the method for combining the image texture to the point cloud described above may be implemented in the form of computer program instructions that may be executed through various computer components. In addition, the implemented computer program may be recorded in a computer-readable recording medium. The mentioned recording medium may be, but is not necessarily limited to, a ROM, a magnetic disk or compact disk, an optical disk, and the like.

As described above, the present invention has been described with reference to the embodiment shown in the drawings, but this is only exemplary, and various modifications and equivalent other embodiments are possible therefrom by those skilled in the art. will understand Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

10: A system for combining image textures with point clouds
11: Interpolation
13: image point calculator
15: Texture Allocator
20: Point Cloud
30: aerial shot video

Claims (11)

interpolating the road surface point cloud to construct at least one triangular surface;
calculating three points on an aerial photographed image corresponding to three points constituting the surface; and
allocating a triangular texture composed of three points on the aerial photographed image as a texture of the surface;
A method of combining an image texture to a point cloud, comprising:
According to claim 1,
The calculating step is
constructing a sensor model from the external expression element and the internal expression element of the image; and
back-projecting three points constituting the surface onto the image;
including,
The method of combining an image texture to a point cloud, characterized in that the back-projection uses the sensor model.
According to claim 1,
The interpolation is a method of combining an image texture to a point cloud, characterized in that using a Poisson method (Poisson Reconstruction) or a triangular network construction method.
According to claim 1,
The point cloud and the aerial shot image, characterized in that the obtained from different platforms, the method of combining image textures to the point cloud.
3. The method of claim 2,
The sensor model is a method of combining an image texture to a point cloud, characterized in that using a collinear expression or DLT (Direct Linear Transformation).
A program stored in a computer-readable recording medium for executing the method according to any one of claims 1 to 5.
A computer-readable recording medium in which a program for executing the method according to any one of claims 1 to 5 is recorded.
an interpolator interpolating the road surface point cloud to construct at least one triangular surface;
an image point calculation unit for calculating three points on an aerial photographed image corresponding to three points constituting the surface; and
a texture allocator for allocating a triangular texture composed of three points on the aerial shot image as a texture of the surface;
A system for combining an image texture to a point cloud, comprising:
9. The method of claim 8,
The image point calculation unit constructs a sensor model from the external expression element and the inner expression element of the image, and uses the sensor model to reversely project three points constituting the surface to the image, point cloud A system for combining image textures into .
9. The method of claim 8,
The interpolation is a system for combining an image texture to a point cloud, characterized in that using a Poisson technique or a triangular network construction technique.
9. The method of claim 8,
The system for combining an image texture to a point cloud, characterized in that the point cloud and the aerial shot image are acquired on different platforms.

Images