KR102558095B1 - Panoramic texture mapping method with semantic object matching and the system thereof - Google Patents

Abstract

The present invention relates to a panoramic texture mapping method and system through semantic object matching, which includes extracting semantic information from a street view image, rendering a G-buffer for a camera view image according to a user's movement, performing panoramic image mapping to map each pixel from the G-buffer to the street view image, and using the semantic information and the G-buffer to perform a depth test, a semantic object matching test, and 3D inpainting for the camera view image and the street view image. and performing panoramic texture mapping.

Description

Panoramic texture mapping method and system through semantic object matching

The present invention relates to a panoramic texture mapping method and system through semantic object matching, and more particularly, to a rendering technique based on panoramic texture mapping that reproduces a street view image in real time.

Photorealistic reproductions of real-world urban scenes have been instrumental in powering a variety of virtual reality (VR) applications that mirror the physical world, including virtual touring, geotagged social media, and information visualization. As a key technique to maximize the realism of these experiences, Image-Based Rendering (IBR) is one of the most active and important research topics in both computer graphics and vision. In particular, according to IBR's geometric image continuum, it is common to adopt a physics-based approach that maps captured real-world images as textures to corresponding virtual proxy geometries, often synthesizing novel views with a wider range of user-free gaze.

However, these traditional IBR methods are difficult to apply to large-scale urban scenes, as they generally require decades or hundreds of adjacent images from the user's point of view to cover a limited area. In addition, previous methods for synthesizing near new views depended on local geometry information, and those methods were limited in terms of scalability of realistically synthesized new views by reconstructing high-fidelity and dense meshes of local proxy geometries from depth images or point clouds. As a result, such an approach could not support users with complete degrees of freedom (DoF) including transformation and rotation for a free walking experience in larger scenes.

With the development of global data acquired by leading companies and research organizations, advanced techniques for handling large-scale scenes have recently been proposed. In particular, with the vast amount of street-level panoramic images provided by Google Street View and the corresponding camera parameters, a wide range of urban scenes could be reconstructed from vision-based algorithms such as Structure from Motion (SfM) available for IBR. Nonetheless, such methods still required more resource images and computational time proportional to the scene scale, and reconstructed point clouds were too numerous to be used in real-time IBR systems.

Google Earth is a more interactive way to support users with highly realistic fly-over experiences in world-scale urban scenes, with nearly all global scenes pre-reconstructed into textures using captured aerial photography and rendered in real time. Nonetheless, this massive project focused primarily on rendering the global scene from a bird's eye view, so the quality of street-level geometry and textures was noticeably reduced. Geollery builds local proxy geometries by leveraging efficient transformations of dense spherical meshes based on Google Street View's depth maps as a more appropriate method for walking experiences on city streets. However, since this method relied on a single nearby street view and local depth information, frequent resource updates were required depending on the user's location, resulting in temporary instability.

In addition, the proposed method to cover various urban scenes relies on global street view images from Google Street View and textures of simplified 3D models readily available from open databases, and maps these street view images onto the scene geometry as view-dependent textures (View-Dependent Texture Mapping (VDTM)). In particular, since the present invention uses sparsely sampled street-view images containing high-resolution, omnidirectional scene information at camera positions, there is no need to use sufficient image resources to synthesize new adjacent views or frequently sample street-view images as in the previous method. This feature provides temporarily stable rendering quality regardless of the dynamically changing user perspective and brings low computational cost.

However, an important problem still remains how to deal with the gap between the simplified scene model and the actual geometric model captured in the street view image. This gap causes incorrect texture mapping and perceptually undesirable rendering results.

R. Szeliski. Computer vision: algorithms and applications. Springer Science & Business Media, 2010.

An object of the present invention is to improve both mapping accuracy and performance time for a street-view image by extracting semantic information from a given street-view image and using it at an appropriate stage.

A panoramic texture mapping method through semantic object matching according to an embodiment of the present invention includes extracting semantic information from a street view image, rendering a G-buffer for a camera view image according to a user's motion, performing panoramic image mapping in which each pixel in the G-buffer is mapped to the street view image, and using the semantic information and the G-buffer to perform a depth test, a semantic object matching test, and 3D inpainting, the panorama for the camera view image and the street view image. and performing texture mapping.

A panoramic texture mapping system through semantic object matching according to an embodiment of the present invention includes a pre-processing unit that extracts semantic information from a street view image, an image mapping unit that renders a G-buffer for a camera view image according to a user's movement and maps each pixel in the G-buffer to the street view image, and a camera view image and the street view through a depth test, a semantic object matching test, and 3D inpainting using the semantic information and the G-buffer. and a texture mapping performing unit that performs panoramic texture mapping on the image.

According to an embodiment of the present invention, more efficient and accurate panoramic texture mapping can be provided by utilizing inferred semantic information of a scene in an appropriate intermediate stage of the proposed pipeline.

In addition, according to an embodiment of the present invention, effective real-time inpainting utilizing both 3D geometry and semantic information can support dynamically changing new view synthesis without visual holes due to complex occlusion.

1 is an operational flowchart of a panoramic texture mapping method through semantic object matching according to an embodiment of the present invention.
2 is an image showing a process of extracting semantic information from a street view image according to an embodiment of the present invention.
3 illustrates a panoramic texture mapping process through semantic object matching according to an embodiment of the present invention.
4 illustrates a camera pose improvement result according to an embodiment of the present invention.
5 is a diagram to explain panoramic image projection according to an embodiment of the present invention.
6A and 6B illustrate a visualized panorama texture mapping process according to an embodiment of the present invention.
7 illustrates the effect of a texture filtering test according to an embodiment of the present invention.
8 shows a result of weighted blending according to an embodiment of the present invention.
9 illustrates low pass filtering results according to an embodiment of the present invention.
10 is a diagram to explain semantic 3D Pixmix inpainting according to an embodiment of the present invention.
11 shows the result of using the inpainting method according to an embodiment of the present invention.
12 is a block diagram illustrating a detailed configuration of a panoramic texture mapping system through semantic object matching according to an embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only these embodiments make the disclosure of the present invention complete, and those skilled in the art are provided to fully inform the scope of the invention, and the present invention is only defined by the scope of the claims.

Terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, “comprises” and/or “comprising” does not preclude the presence or addition of one or more other components, steps, operations, and/or elements in which a stated component, step, operation, and/or element is present.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. The same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

Embodiments of the present invention have a panoramic texture mapping-based rendering technology capable of reproducing a large-scale city scene at a street level in real time as identical to a photograph.

The present invention proposes a novel instant texture mapping system using sampled panoramic street view images and an open 3D model of a large-scale global city scene for the ultimate purpose of experiencing highly realistic city streets in virtual reality (VR).

Hereinafter, a panoramic texture mapping method and system through semantic object matching will be described in detail with reference to FIGS. 1 to 12 .

1 is an operation flowchart of a panoramic texture mapping method through semantic object matching according to an embodiment of the present invention, FIG. 2 is an image showing a process of extracting semantic information from a street view image according to an embodiment of the present invention, and FIG. 3 illustrates a panoramic texture mapping process through semantic object matching according to an embodiment of the present invention.

4 shows a camera pose improvement result according to an embodiment of the present invention, FIG. 5 is shown to explain panoramic image projection according to an embodiment of the present invention, and FIGS. 6A and 6B show a visualized panorama texture mapping process according to an embodiment of the present invention.

In addition, FIG. 7 shows the effect of a texture filtering test according to an embodiment of the present invention, FIG. 8 shows a weighted blending result according to an embodiment of the present invention, and FIG. 9 shows a result of low pass filtering according to an embodiment of the present invention.

In addition, FIG. 10 is a diagram to explain semantic 3D Pixmix inpainting according to an embodiment of the present invention, and FIG. 11 shows a result using an inpainting method according to an embodiment of the present invention.

The method of FIG. 1 is performed by a panoramic texture mapping system through semantic object matching according to an embodiment of the present invention shown in FIG. 12 .

Referring to FIG. 3, the proposed system according to an embodiment of the present invention is largely composed of two parts: a preliminary process 310 for obtaining input resources and constructing a 3D scene to be rendered, and a real-time process 320 including panoramic texture mapping and semantic 3D inpainting. The present invention is characterized by matching semantic information extracted from street view images at appropriate intermediate stages such as Google Street View, Semantic Object Matching Test, and Semantic 3D Inpainting.

Referring to Figures 1 and 3, in the preliminary process 310, step 110 extracts semantic information.

Step 110 may extract semantic information of an RGB panoramic street view image (PI), a city model (M), an object unit semantic image (PS), a depth image (PD), and a camera parameter (C) from a street view image. At this time, the RGB panoramic street view image (PI) is a wide street level panoramic image around the world, and represents an image resource provided by Google Street View of the Google Maps platform. In addition, the city model M is virtual model data of city buildings and is obtained from an open database of 3D geographic information. In addition, the object unit semantic image (PS) is an object unit semantic information image extracted from street view images using a deep learning-based method, DeppLabV3+, and the camera parameter (C) is data calibrated by matching the boundary lines of buildings using the city model (M) and the object unit semantic image (PS).

As described above, the RGB panoramic street view image (PI) and city model (M) are obtained from open data, and the object unit semantic image (PS) can be obtained from the RGB panoramic street view image (PI) by a deep learning-based method. In addition, the camera parameter (C) can be subdivided using the object unit semantic image (PS) inferred from the image model registration between the RGB panoramic street view image (PI) and the city model (M). Finally, a depth image (PD) for depth test in real-time texture mapping can be rendered using a city model (M) and camera parameters (C).

Step 110 uses DeepLabv3+, which is one of the deep learning-based approaches in Google Street View, and ASPP (Atrous Spatial Pyramid Pooling) to encode multiscale context information to extract an object unit semantic image (PS). The boundary of the segmented object can be segmented. At this time, since the present invention uses 5000 large-scale city scene images with high-quality high-density pixel annotations of 30 classes suitable for the target scene, the object unit semantic image (PS), as shown in the first row of FIG.

Step 110 may also obtain appropriate camera parameters (C) used to capture resource textures for more accurate panoramic image projection. More specifically, step 110 can improve the 6 DoF camera parameters (C) using effective image 3D model registration based on contour alignment. This method has the advantage of being able to perform optimization for individual cameras, unlike camera pose estimation or batch adjustment using SfM (Structure from Motion). The image registration method optimizes the energy function to reach a zero value when the projected uk at the pworld point of the 3D model is exactly superimposed on the sampled panoramic street-view image (PI), resulting in the outline of the 3D model aligned with the sampled panoramic street-view image (PI) model. To realize this, the present invention uses the original camera pose of Google Street View as an initial value to minimize the loss function of [Equation 1] below.

[Equation 1]

Here, the outline of the 3D model can be efficiently rendered through a function L using the camera index k in the Unity shader (pink line in Fig. 4), and the boundary Lk of the sampled panoramic street view image (PI) can be extracted using the previously acquired sampled object unit semantic image (PS, cyan line in Fig. 4). At this time, each pixel of the contour image has a binary value (0: non-contour pixel, 1: contour pixel), and [Equation 1] can obtain the optimal sampled camera parameter (C). To find the best sampled camera parameters (C), the present invention deploys particle swarm optimization (PSO) using an evolutionary sampling method in which each sampled particle searches for a sample of better sampled camera parameters (C) by taking into account the locally found optimal particle and the globally found optimal particle shared by all other particles. Accordingly, as shown in FIG. 4 , as a result of using the improved camera pose, it can be seen that the qualitative alignment is better than that of the original camera pose of Google Street View.

Step 110 may also filter texture pixels using a depth test when mapping textures to 3D surface points. Specifically, in step 110, the sampled depth image PD may be rendered into the city model M, photographed in advance with the sampled camera parameters C, and placed in the sampled depth image PD for depth testing.

In addition, step 110 can effectively manage the street view image using a sparse sampling technique based on regularly segmented global space similar to the global geometry. For example, in pre-process 310, the X-Z plane is divided into smaller tiles called Districts according to the width and height selected by the user. Among the street view cameras located in the area, a camera closest to the central position of the area indicated by a yellow circle in the refined 3D scene data is sampled. For each user's current zone, up to eight adjacent zones can be placed in the real-time texture mapping process, and if the tile size of a zone is small, a closer distance view image can be sampled, but more texture updates are required depending on the user's frequently changing zone.

In step 120, in real-time process 320, a G-buffer is rendered for a camera view image according to a user's movement, and panoramic image mapping is performed to map each pixel in the G-buffer to a street view image.

In step 120, a G-buffer of a position buffer, a segmentation buffer, and a normal buffer may be rendered according to the camera view image according to the user's movement using the city model M of the street view image, a panoramic coordinate system conversion process may be performed based on adjacent camera information in the position buffer, and each pixel of the position buffer may be mapped to street view images.

In step 120, a position buffer (Gp), a segmentation buffer (Gs), and a G-buffer of a normal buffer (Gn) may be rendered according to the camera view image using the city model M of the street view image for real-time panoramic texture mapping. The G-buffer is a space that renders and stores various geometric information about the scene currently viewed by the user, and may include a location buffer that stores topographical location information in virtual space, a general buffer that stores normal vectors, and a segmentation buffer that contains semantic information for each object.

In step 120, in the real-time process 320, a 3D surface point may be projected (Panoramic Image Projection) to a corresponding pixel position in the street view image along with a camera index k using the sampled camera parameter C. Referring to FIG. 5 , step 120 may sequentially perform spherical and equirectangular projections. here, and denotes the axis angle expression and rotation, respectively. 3D point in world coordinates is the view matrix through [Equation 2] below point in camera coordinates via is converted to

[Formula 2]

Assuming the virtual 360° camera model performs a spherical projection Esphere, can be projected to point sk on the unit sphere at the origin of camera space. According to the 2D spherical coordinate system Is expressed as [Equation 3], [Equation 4] and [Equation 5] below.

[Formula 3]

[Formula 4]

[Formula 5]

here, and denotes the azimuth and polar angle, respectively, and rk is at the center position of the unit sphere represents the Euclidean distance to In addition, assuming that left-handed coordinates are used as in the Unity engine in which the system according to the embodiment of the present invention was developed, pixels in the sk of the panoramic image If mapped as , equirectangular projection as shown in [Equation 6] and [Equation 7] below can be done with

[Formula 6]

[Formula 7]

here, and represents the st image coordinate value between 0.0 and 1.0. finally, The projection function E from to uk can be defined by [Equation 8].

[Formula 8]

Step 130 includes a first step (Depth Test) of performing a depth test to minimize projection errors for pixels projected on the camera view image, a second step (Semantic Object Matching Test) of performing an object matching test through semantic information on the camera view image in which the projection error is minimized, and a third step (Weighted Blending) of applying blending weights to the camera view image to interpolate candidate texture colors to synthesize the final color, and a semantic 3D rendering of a non-sky area in the camera view image. A fourth step (Semantic 3D Inpainting) of completing panorama texture mapping by inting may be included.

Step 120 is all surface pixels in position buffer Gp using view matrix Mk and projection E from [Equation 8]. is the corresponding pixel location of the panoramic street view image (PI) sampled by the camera index k. map to Subsequently, step 120 may assign a texture color to a possible pixel of the current view in the sampled panoramic distance view image PI as shown in FIG. 6A through a mapping relationship. However, as shown in Fig. 7(a), not all of the colors may be used to generate the final color of a particular surface pixel due to possible projection errors.

Accordingly, step 130 according to an embodiment of the present invention selects usable candidate texture colors from the panoramic street view image PI sampled through the first to fourth steps, and calculates an appropriate weight for mixing. In order to effectively improve texture mapping accuracy, the present invention not only excludes erroneously projected texture colors at blockage surface points using sampled depth images (PD), but also selects contextually correct candidates using sampled per-object semantic images (PS), and then uses weighted blending to render a conceptually natural result, as shown in Fig. 6b.

In the first step, projection error can be minimized by removing RGB values due to occlusion in the current user's view using the rendered depth image PD.

The purpose of the depth test is to address the penetration problem of projection texture mapping. An error through the k-th camera used to capture the sampled panoramic street-view image PI may appear as a pixel color of the sampled panoramic street-view image PI. Therefore, in the first step according to the embodiment of the present invention, since the sampled depth image PD of the 3D scene model is rendered according to the sampled camera parameter C, the camera position and The distance between the sampled pixels of the depth image (PD) Any 3D point in the position buffer (Gp) by comparing with the depth value of It can be simply determined whether is visible to the k-th camera.

However, although the depth test can eliminate most of the projection errors, it is not an optimal solution because misregistration between the actual geometry of the sampled panoramic distance view image (PI) and the synthetic scene geometry occurs due to the incomplete sampled depth image (PD) that simplifies the 3D model and calibration errors in the sampled camera parameters (C). As a result, we can still find that the texture color is projected onto the semantically incorrect target object, as shown in Fig. 7(b). Therefore, step 120 according to an embodiment of the present invention precisely removes the remaining errors by performing the second step, the third step, and the fourth step.

In the second step, when each pixel of the location buffer is mapped to the RGB panoramic street view image PI using the object unit semantic image PS, an object matching test may be performed to check matching of the RGB panoramic street view image PI with an object, and a color may be assigned to the target object in the sampled panoramic street view image PI.

Placing the semantic information of each object in the current scene can accurately assign a texture color to each. For this reason, in the second step, by performing an object matching test using the object unit semantic image (PS) sampled in the real-time process 320, a color may be assigned to the correct target object in the sampled panoramic street view image (PI). For example, the sampled object-by-object semantic image (PS) estimated in DeepLabV3+ in the pre-process 310 can classify each pixel of the sampled panoramic street view image (PI) based on the pre-trained object class and various city scenes in the CityScapes dataset, so that accurate and clear boundaries of different object areas can be obtained. Accordingly, assuming that the 3D model has known labels defined in DeepLabV3+, the second step is to determine the surface points on the model. Estimation of labels and sampled object unit semantic images (PS) of Labels of classes can be equated, so we can filter out incorrect texture colors projected on non-matching objects. Accordingly, as shown in FIG. 7(c), the removed texture color of the building (target object) projected on the sky can be represented.

In the third step, RGB values of a final color synthesized by interpolating candidate texture colors through a blending method may be allocated to each pixel of the location buffer.

The final color must be synthesized by interpolating the selected candidate texture color from the sampled panoramic street view image (PI) passed through the previous steps. Therefore, in the third step, when N candidate colors are given, the output color oi is a standardized weight as shown in [Equation 9] below is interpolated using

[Formula 9]

here, is the surface point of the sampled panoramic distance view image (PI) with camera index k through the projection function E in [Equation 8] Indicates the projected pixel position in . At this time, assigns a pixel color to

The third step uses the new blend weights motivated by the underlying energy transfer equation to calculate the intensity of incident light on the surface based on the inverse square law between energy and distance. In addition, the proposed weight is the texture camera parameter (C) and The geometric relationship between the sampled camera parameter C and the current location of the user is considered as shown in [Equation 10].

[Formula 10]

where ni is the surface point denotes a normal vector at , and vk is represents the camera vector from to the k-th camera position tk. α represents the smoothness condition (α=4). Also, disti-k and distuser-k are respectively and tk, and the distance between the user's location and tk. As shown in Fig. 8, the main advantage of the proposed blending weight is that it can calculate a color more suitable for the user's location while observing the three main factors described for blending.

In the fourth step, a real-time inpainting method is used to process pixels whose RGB values are not filled in the position buffer, low-pass filtering is performed on the sky area in the camera view image, and the non-sky area is geometrically and semantically matched. The panorama texture mapping can be completed by filling the RGB values of the texture color. The fourth step of step 130 according to an embodiment of the present invention can efficiently reduce calculation time by using an inpainting method that separates and processes the sky region and the non-sky region in the camera view image. To this end, a fourth step may apply a low-pass filter to the sky areas, handle untextured holes and errors simultaneously, and fill the non-sky areas with geometrically or semantically matching texture colors.

More specifically, the fourth step may correct the sky area through low-pass filtering. Referring to FIG. 9 as an example, in the fourth step, the sky area and non-sky area are classified, and the sky area without texture may be filled with the average sky color. Then, scale with resolution using nearest-neighbor downsampling, and double the image width by upsampling using low-pass filtering until the image size is original. As a result, as shown in FIG. 9(f), it can be confirmed that the present invention shows convincing results without a visual layer, projection error, and pixels without texture, including meaningful color changes such as the upper right bright part.

Also, the fourth step may use a semantic 3D Pixmix inpainting method. Compared to the sky area, the non-textured non-sky area must be smoothly filled with a high-frequency, semantically appropriate texture color to create a realistic urban scene. Therefore, the present invention adopts an advanced inpainting technique considering 3D geometric and semantic information of the current scene based on a high-quality 2D inpainting method called Pixmix.

Pixmix iteratively from the target pixel p in the non-text area T to the source pixel in the already textured area S, as shown in FIG. shows the result of finding . If an appropriate mapping function f is found for every pixel p, the color of each pixel p is replaced with the source color determined in the textured region S for the final image. Accordingly, Pixmix solved the problem of global minimization of the total cost function consisting of two types of costs: a spatial function and an apparent cost, Costa, in order to obtain the best mapping function f.

[Equation 11]

[Equation 12]

here, denotes a pixel location at T, denotes the relative position of adjacent pixels near p, which is A vector of elements from p to adjacent pixels, except for includes As in the original work, patch sizes of 3 × 3 and 5 × 5 were used for Ns and Na, respectively, and weighted is basically uniformly weighted indicates However, in the case of wa, pixels adjacent to the boundary of T and S have a large weight that can selectively affect a high impact on Costa, so the border effect can be reduced. Thus, squared Euclidean distances may be used for the distance functions ds and da. Intuitively, minimizing costs means that the neighborhood of the target pixel p closely matches the surrounding structure of f(p) while maintaining similar surrounding structures. When h(p) represents the pixel color of p, minimizing Costa allows p to have similar pixel colors to f(p).

Pixmix also showed convincing image inpainting results in real time, but struggled to fill holes with non-planar backgrounds and lacked semantic matching between target and source pixels. As a solution to this, the fourth step according to an embodiment of the present invention extends Pixmix to arrange 3D geometric information and semantic information of the current scene using a total cost function as shown in [Equation 13] below.

[Equation 13]

Here, the coincident Costm(p) has 0 when the semantic label of p and f(p) are equal, whereas it has infinity otherwise. The label of the target pixel p can be obtained from the known class of the 3D model to which P belongs, but the label of the source pixel f(p) is obtained from the sampled per-object semantic image (PS) of the sampled panoramic street view image (PI), which is the most influential candidate texture selected in the previous third step. This helps to find semantically appropriate source pixels more accurately than using the 3D model's semantic labels for both p and f(p). In order to further consider the already known 3D geometry in the position buffer (Gp) and Calculate the space cost using the 3D surface position of and the 3D distance function ds. control parameter Use 0.5 to set

A major advantage of using semantic information according to an embodiment of the present invention is that the search range of f(p) can be effectively reduced by sampling contextually matching source pixels.

Referring to FIG. 11, FIG. 11(a) shows a captured camera view image, FIG. 11(b) shows a state where no depth test, semantic object test, and inpainting are applied, and FIG. 11(c) shows a state where a depth test is added. In addition, FIG. 11(d) shows a state in which the semantic object test has been added, and FIG. 11(e) shows the final result of the present invention in a state in which inpainting has been performed. 11(f) and 11(g) show the result of applying the existing inpainting method. It can be seen that the present invention fills a large hole with a non-planar background better than previous work because it places a 3D geometry as shown in FIG. 11 .

12 is a block diagram illustrating a detailed configuration of a panoramic texture mapping system through semantic object matching according to an embodiment of the present invention.

Referring to FIG. 12 , a panoramic texture mapping system through semantic object matching according to an embodiment of the present invention proposes a rendering technique based on panoramic texture mapping that reproduces a street view image in real time.

To this end, the panoramic texture mapping system 1200 through semantic object matching according to an embodiment of the present invention includes a pre-processing unit 1210 and a mapping performing unit 1220.

The pre-processing unit 1210 extracts semantic information.

The pre-processing unit 1210 may extract semantic information of an RGB panoramic street view image (PI), a city model (M), an object unit semantic image (PS), a depth image (PD), and a camera parameter (C) from a street view image. At this time, the RGB panoramic street view image (PI) is a wide street level panoramic image around the world, and represents an image resource provided by Google Street View of the Google Maps platform. In addition, the city model M is virtual model data of city buildings and is obtained from an open database of 3D geographic information. In addition, the object unit semantic image (PS) is an object unit semantic information image extracted from street view images using a deep learning-based method, DeppLabV3+, and the camera parameter (C) is data calibrated by matching the boundary lines of buildings using the city model (M) and the object unit semantic image (PS).

As described above, the RGB panoramic street view image (PI) and city model (M) are obtained from open data, and the object unit semantic image (PS) can be obtained from the RGB panoramic street view image (PI) by a deep learning-based method. In addition, the camera parameter (C) can be subdivided using the object unit semantic image (PS) inferred from the image model registration between the RGB panoramic street view image (PI) and the city model (M). Finally, a depth image (PD) for depth test in real-time texture mapping can be rendered using a city model (M) and camera parameters (C).

The image mapping unit 1220 renders a G-buffer for a camera view image according to a user's movement and performs panoramic image mapping to map each pixel in the G-buffer to a street view image.

The image mapping unit 1220 may render a G-buffer of a position buffer, a segmentation buffer, and a normal buffer according to a camera view image according to a user's movement using a city model M of a street view image, perform a panoramic coordinate system conversion process based on adjacent camera information in the position buffer, and map each pixel of the position buffer to street view images.

The image mapping unit 1220 may render a G-buffer of a position buffer (Gp), a segmentation buffer (Gs), and a normal buffer (Gn) according to a camera view image using a city model M of a street view image for real-time panoramic texture mapping. The G-buffer is a space that renders and stores various geometric information about the scene currently viewed by the user, and may include a location buffer that stores topographical location information in virtual space, a general buffer that stores normal vectors, and a segmentation buffer that contains semantic information for each object.

The texture mapping performer 1230 performs panoramic texture mapping on the camera view image and the street view image through a depth test, a semantic object matching test, and 3D inpainting using semantic information and a G-buffer.

The texture mapping performer 1230 performs a first step (Depth Test) of performing a depth test for minimizing projection errors on pixels projected on a camera view image, a second step (Semantic Object Matching Test) of performing an object matching test through semantic information on the camera view image in which the projection error is minimized, and a third step (Weighted Blending) of synthesizing a final color by interpolating candidate texture colors by applying blending weights to the camera view image. A fourth step (Semantic 3D Inpainting) of completing panorama texture mapping by 3D inpainting may be included.

Although the description is omitted in the system of FIG. 12, it is apparent to those skilled in the art that the system according to the present invention may include all of the contents described in FIGS. 1 to 11.

The system or apparatus described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may be implemented using one or more general purpose or special purpose computers, such as, for example, a processor, controller, arithmetic logic unit (ALU), digital signal processor, microcomputer, field programmable gate array (FPGA), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will recognize that the processing device may include a plurality of processing elements and/or multiple types of processing elements. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to operate as desired, or may independently or collectively direct a processing device. Software and/or data may be permanently or temporarily embodied in any tangible machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, to be interpreted by, or to provide instructions or data to, a processing device. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions such as ROM, RAM, and flash memory. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, even if the described techniques are performed in an order different from the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or replaced or substituted by other components or equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (11)

Extracting semantic information of an RGB panoramic street view image (PI), a city model (M), an object unit semantic image (PS), a depth image (PD), and a camera parameter (C) from a street view image constituting 3D scene data;
rendering a G-buffer for a camera view image according to a user's movement and performing panoramic image mapping to map each pixel in the G-buffer to the street view image; and
performing panoramic texture mapping on the camera view image and the street view image through a depth test, a semantic object matching test, and 3D inpainting using the semantic information and the G-buffer;
including,
The step of performing the panoramic image mapping
Rendering a G-buffer of a position buffer, a segmentation buffer, and a normal buffer according to the camera view image according to the user's movement using the city model M of the street view image;
The step of performing the panoramic texture mapping
performing a depth test to minimize projection errors for pixels projected on the camera view image;
performing a semantic object matching test by performing an object matching test on the camera view image in which the projection error is minimized;
synthesizing a final color by interpolating candidate texture colors by applying a blending weight to the camera view image; and
Completing panoramic texture mapping by performing semantic 3D inpainting on a non-sky area in the camera view image
including,
The step of performing the semantic object matching test
When each pixel of the location buffer is mapped to the RGB panoramic street view image (PI) using the object-unit semantic image (PS), the object match test for checking matching with an object in the RGB panoramic street view image (PI) Characterized in that,
Completing the panoramic texture mapping
In order to process pixels not filled with RGB values in the camera view image, a real-time inpainting method of separating and processing a sky region and a non-sky region in the camera view image is used. Panoramic texture mapping method through semantic object matching. delete delete According to claim 1,
The step of performing the panoramic image mapping
Panoramic texture mapping method through semantic object matching, wherein a panoramic coordinate system conversion process is performed based on adjacent camera information in the position buffer, and each pixel of the position buffer is mapped to the street view images. delete According to claim 1,
The depth testing step is
A panoramic texture mapping method through semantic object matching, wherein projection error is minimized by removing RGB values due to occlusion from a current user's view using the rendered depth image (PD). delete According to claim 1,
The synthesizing step is
A panoramic texture mapping method through semantic object matching, wherein RGB values of a final color synthesized by interpolating candidate texture colors through a blending method are allocated to each pixel of the location buffer. delete According to claim 1,
Completing the panoramic texture mapping
In the camera view image, low-pass filtering is performed on the sky area, and the panoramic texture mapping is completed by filling the non-sky area with RGB values of texture colors that match geometrically and semantically. Panorama texture mapping method through semantic object matching. A pre-processing unit for extracting semantic information of an RGB panoramic street view image (PI), a city model (M), an object unit semantic image (PS), a depth image (PD), and a camera parameter (C) from a street view image constituting 3D scene data;
an image mapping unit rendering a G-buffer for a camera view image according to a user's motion and performing panoramic image mapping to map each pixel in the G-buffer to the street view image; and
A texture mapping processor performing panoramic texture mapping on the camera view image and the street view image through a depth test, a semantic object matching test, and 3D inpainting using the semantic information and the G-buffer
including,
The image mapping unit
Rendering a G-buffer of a position buffer, a segmentation buffer, and a normal buffer according to the camera view image according to the user's movement using the city model M of the street view image;
The texture mapping unit
a first step of performing a depth test to minimize a projection error for a pixel projected on the camera view image; a second step of performing a semantic object matching test by performing an object matching test on the camera view image in which projection error is minimized; a third step of synthesizing a final color by interpolating candidate texture colors by applying a blending weight to the camera view image; and a fourth step of completing panoramic texture mapping by performing semantic 3D inpainting on a non-sky area in the camera view image,
The second step is
When each pixel of the location buffer is mapped to the RGB panoramic street view image (PI) using the object-unit semantic image (PS), the object match test for checking matching with an object in the RGB panoramic street view image (PI) Characterized in that,
The fourth step is
Panoramic texture mapping system through semantic object matching, characterized by using a real-time inpainting method that separates and processes a sky region and a non-sky region in the camera view image to process pixels not filled with RGB values in the camera view image.

Images