KR102526189B1 - Apparatus and method for modeling three dimensional image - Google Patents

Abstract

The present invention relates to a device and a method for modeling a three-dimensional image, which model a two-dimensional building front image into a three-dimensional building image. According to an embodiment of the present invention, the method comprises the steps of: generating a second image from which noise is removed from a first image including a two-dimensional real building; generating a third image in which a plurality of components configuring the front of a building are classified from the second image; separating a layer from the third image, and generating a shape grammar defining a morphological structure for the components included in the layer; generating a three-dimensional mesh for the front of the building using the floor obtained based on a grammar analysis result of a grammar interpreter using the shape grammar as input and coordinate information on the components included in the floor; and generating a three-dimensional building modeling result by applying the three-dimensional mesh to a shape file. Therefore, a manual work and processing time of a worker can be reduced during modeling into a three-dimensional building image.

Description

3D image modeling apparatus and method {APPARATUS AND METHOD FOR MODELING THREE DIMENSIONAL IMAGE}

The present invention relates to a 3D image modeling apparatus and method for modeling a 2D front image of a building into a 3D building image.

In general, modeling tools based on building drawings and actual images, modeling tools based on point clouds using 3D scanners, and modeling tools based on multiple images are used as tools for modeling 3D buildings. However, for 3D building modeling, supply and demand of building drawings, manual texturing skills of experienced users, and handling of various modeling tools are required, so there is a problem in that various times are required depending on the user's skill level.

The foregoing background art is technical information that the inventor possessed for derivation of the present invention or acquired during the derivation process of the present invention, and cannot necessarily be said to be known art disclosed to the general public prior to filing the present invention.

Domestic Patent Publication No. 10-2012-0071276 (2012.07.02)

One object of the present invention is to reduce the user's manual work and processing time when modeling a 2D building front image into a 3D building image.

The problem to be solved by the present invention is not limited to the above-mentioned problems, and other problems and advantages of the present invention that are not mentioned can be understood by the following description and more clearly understood by the embodiments of the present invention. It will be. In addition, it will be appreciated that the problems and advantages to be solved by the present invention can be realized by the means and combinations indicated in the claims.

A 3D image modeling method according to the present embodiment is a 3D image modeling method performed by a processor of a 3D image modeling apparatus, and generates a second image from which noise is removed from a first image including a 2D real-life building. generating a third image in which a plurality of components constituting the front of the building are classified from the second image; separating layers from the third image; morphological analysis of a plurality of components included in the layer; Step 3 for the front of the building using the step of generating a form grammar that defines the structure and the coordinate information for the layer and a plurality of components included in the layer obtained based on the grammar analysis result of the grammar analyzer using the form grammar as an input. It may include generating a 3D mesh and generating a 3D building modeling result by applying the 3D mesh to a shape file.

The 3D image modeling apparatus according to the present embodiment includes a processor and a memory operably connected to the processor and storing at least one code executed by the processor, and when the memory is executed through the processor, the processor performs a two-dimensional real image. Creating a second image from which noise is removed from the first image including the building, generating a third image in which a plurality of components constituting the front of the building are classified from the second image, separating the layers from the third image, , Create a form grammar defining the morphological structure of a plurality of components included in the layer, and generate a form grammar based on the grammar analysis result of the grammar analyzer using the form grammar as an input for the layer and the plurality of components included in the layer It is possible to generate a 3D mesh for the front of the building using the coordinate information for the building, and store a code that causes the 3D building modeling result to be generated by applying the 3D mesh to a shape file.

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to the present invention, when modeling a 2D building front image into a 3D building image, a user's manual work and processing time can be reduced.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is an exemplary diagram of a 3D image modeling environment according to an exemplary embodiment.
2 is a block diagram schematically illustrating the configuration of a 3D image modeling apparatus according to an exemplary embodiment.
FIG. 3 is a block diagram schematically illustrating the configuration of an image processing unit in the 3D image modeling device of FIG. 2 .
4 to 10 are exemplary views illustrating 3D image modeling according to the present embodiment.
11 is a block diagram schematically illustrating a configuration of a 3D image modeling apparatus according to another embodiment.
12 is a flowchart illustrating a 3D image modeling method according to an exemplary embodiment.

Advantages and features of the present invention, and methods for achieving them will become clear with reference to the detailed description of embodiments in conjunction with the accompanying drawings. However, it should be understood that the present invention is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all conversions, equivalents, and substitutes included in the spirit and scope of the present invention. . The embodiments presented below are provided to complete the disclosure of the present invention and to fully inform those skilled in the art of the scope of the invention to which the present invention belongs. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded. Terms such as first and second may be used to describe various components, but components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

In addition, in this application, “unit” may be a hardware component such as a processor or a circuit, and/or a software component executed by a hardware component such as a processor.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof are omitted. I'm going to do it.

In the following embodiments, terms such as first and second are used for the purpose of distinguishing one component from another component without limiting meaning.

In the following embodiments, singular expressions include plural expressions unless the context clearly indicates otherwise.

In the following embodiments, terms such as include or have mean that features or elements described in the specification exist, and do not preclude the possibility that one or more other features or elements may be added.

When an embodiment is otherwise embodied, a specific process sequence may be performed differently from the described sequence. For example, two processes described in succession may be performed substantially simultaneously, or may be performed in an order reverse to the order described.

1 is an exemplary diagram of a 3D image modeling environment according to an exemplary embodiment. Referring to FIG. 1 , a 3D image modeling environment 1 may include a 3D image modeling device 100 , a user terminal 200 and a network 300 .

The 3D image modeling apparatus 100 may generate a second image from which noise is removed from the first image including the 2D photorealistic building received from the user terminal 200 . The 3D image modeling apparatus 100 may pre-process the first image to generate a second image from the first image, and apply an artificial intelligence algorithm (eg, a generative adversarial network (GAN)) to the pre-processed first image. )) may be applied to generate the second image.

The 3D image modeling apparatus 100 may generate a third image in which a plurality of components constituting the front of the building are classified from the second image. In this embodiment, the plurality of components constituting the front of the building may include windows, walls, doors, pillars, shops, window frames, and the like. The 3D image modeling apparatus 100 may generate a second image by applying an artificial intelligence algorithm (eg, a high-resolution network (HRNet)) to the second image.

The 3D image modeling apparatus 100 may separate a floor from the third image and generate a shape grammar defining a morphological structure of a plurality of components included in the layer. Shape grammar is a concept derived from the inspiration of Christopher Alexander's Pattern Language, and it can suggest an automated method of grammaticalizing geometric patterns. By allowing a computer to capture various shape patterns existing inside a complicated geometric shape, it is possible to automatically extract various shape patterns that are difficult for humans to find. Using shape grammar, it is possible to objectively extract shape patterns that appear in common among various shape designs. In addition, if a commonly used shape pattern is grammatically grammaticalized according to a certain logic, it is possible to automatically generate several similar shapes that have not yet been derived.

The 3D image modeling apparatus 100 creates a 3D mesh for the front of the building by using coordinate information about a layer and a plurality of components included in the layer obtained based on the grammar analysis result of the grammar analyzer taking the form grammar as an input. (mesh) can be created. The mesh structure is a method of expressing a graphic model using a plurality of triangular elements and vertices, and can be widely used to examine physical characteristics of a specific object by constructing a graphic model. In this embodiment, when generating a 3D mesh, a Delaunay triangulation method may be used. The Delaunay triangulation may refer to a division in which, when a space is divided by connecting points on a plane with triangles, the minimum value of the interior angles of these triangles is maximized. Triangles generated by Delaunay triangulation are calculated close to equilateral triangles, so they are suitable for use as a mesh structure for graphic models.

The 3D image modeling apparatus 100 may generate a 3D building modeling result by applying a 3D mesh to a shape file. A shape file is a common standard format file used not only by ArcView but also by other programs such as GIS, 3D programs, and AutoCad map. shx file, a .dbf file that is a dBASE file that provides attribute information (one of Point, Line, or Polygon) of a geographic map, .sbn file that stores a spatial index of a geographic map, spatial join, etc. It consists of a .sbx file, which is a necessary file when performing the function of or creating an index for shpe fields, and the files with the above 5 extensions can be executed as a bundle. These shape files enable quick viewing of spatial information in View, and easy editing, addition, and deletion of spatial mapping and attributes.

In this embodiment, the 3D image modeling device 100 may exist independently in the form of a server or may implement functions provided by the 3D image modeling device 100 in the form of an application and be installed in the user terminal 200 .

The user terminal 200 may receive a 3D image modeling service by accessing a 3D image modeling application and/or a 3D image modeling site provided by the 3D image modeling apparatus 100 .

The user terminal 200 may include a communication terminal capable of performing functions of a computing device (not shown), and in addition to a desktop computer 201, a smartphone 202, and a laptop computer 203 operated by a user, Tablet PCs, smart TVs, mobile phones, PDAs (personal digital assistants), media players, micro servers, GPS (global positioning system) devices, e-readers, digital broadcasting terminals, navigation devices, kiosks, MP3 players, digital cameras, home appliances and It may be other mobile or non-mobile computing devices, but is not limited thereto. Also, the user terminal 200 may be a wearable terminal such as a watch, glasses, hair band, and ring having communication functions and data processing functions. The user terminal 200 is not limited to the above, and a terminal capable of web browsing may be borrowed without limitation.

The network 300 may serve to connect the 3D image modeling device 100 and the user terminal 200 . Such a network 300 may be, for example, a wired network such as a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), an integrated service digital network (ISDN), a wireless LAN (WLAN), or a code network (CDMA). -division multiple access), satellite communication, etc., but the scope of the present invention is not limited thereto. Also, the network 300 may transmit and receive information using short-range communication and/or long-distance communication. Here, short-range communication may include Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra-wideband (UWB), ZigBee, and Wi-Fi technologies, and long-distance communication may include code-division multiple access (CDMA). ), frequency-division multiple access (FDMA), time-division multiple access (TDMA), orthogonal frequency-division multiple access (OFDMA), and single carrier frequency-division multiple access (SC-FDMA) technologies.

The network 300 may include connections of network elements such as hubs, bridges, routers, and switches. Network 300 may include one or more connected networks, such as a multiple network environment, including a public network such as the Internet and a private network such as a secure corporate private network. Access to network 300 may be provided via one or more wired or wireless access networks.

The network 300 may include connections of network elements such as hubs, bridges, routers, and switches. Network 300 may include one or more connected networks, such as a multiple network environment, including a public network such as the Internet and a private network such as a secure corporate private network. Access to network 300 may be provided via one or more wired or wireless access networks.

Furthermore, the network 300 includes CAN (controller area network) communication, V2I (vehicle to infrastructure) communication, V2X (vehicle to everything) communication, wave (wireless access in vehicular environment) communication technology, things, etc. It can support an Internet of Things (IoT) network and/or 5G communication that exchanges and processes information between distributed components.

2 is a block diagram schematically illustrating the configuration of a 3D image modeling apparatus according to an exemplary embodiment. In the following description, descriptions of overlapping portions with those of FIG. 1 will be omitted. Referring to FIG. 2 , the 3D image modeling apparatus 100 includes a communication unit 110, a storage medium 120, a program storage unit 130, a database 140, an image processing unit 150 and a control unit 160. can do.

The communication unit 110 may provide a communication interface necessary to provide a transmission/reception signal between the 3D image modeling device 100 and the user terminal 200 in the form of packet data in conjunction with the network 300 . Furthermore, the communication unit 110 may serve to receive a predetermined information request signal from the user terminal 200 and may serve to transmit information processed by the image processing unit 150 to the user terminal 200. . Here, the communication network is a medium that serves to connect the 3D image modeling device 100 and the user terminal 200, and information after the user terminal 200 accesses the 3D image modeling device 100 It may include a path that provides an access path to transmit and receive. In addition, the communication unit 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals to and from other network devices through wired or wireless connections.

The storage medium 120 performs a function of temporarily or permanently storing data processed by the controller 160 . Here, the storage medium 120 may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto. The storage medium 120 may include built-in memory and/or external memory, and may include volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, and flash ROM. , NAND flash memory, or non-volatile memory such as NOR flash memory, SSD. It may include a compact flash (CF) card, a flash drive such as an SD card, a Micro-SD card, a Mini-SD card, an Xd card, or a memory stick, or a storage device such as an HDD.

The program storage unit 130 generates a second image from which noise is removed from the first image including the two-dimensional real-life building, and generates a third image in which a plurality of components constituting the front of the building are classified from the second image. Generation task, task of separating a floor from the third image and generating a morphological structure defining the morphological structure of a plurality of components included in the layer, grammar analysis result of the grammar analyzer with the morphological grammar as an input Create a 3D mesh for the front of the building using coordinate information on the layer and multiple components included in the layer obtained based on It can be equipped with control software that performs tasks such as

The database 140 may include a management database that stores various tools for generating 3D building modeling results from 2D building images. For example, various image processing algorithms and artificial intelligence algorithms for generating a second image from a first image may be stored in the management database. An artificial intelligence algorithm for classifying a plurality of components constituting a building facade may be stored in the management database. An algorithm for generating a morphological grammar to define a morphological structure for a plurality of elements may be stored in the management database. An algorithm for generating a 3D mesh may be stored in the management database. A shape file for generating a 3D building modeling result may be stored in the management database.

In addition, the database 140 may include a user database for storing information of a user to be provided with a 3D image modeling service. Here, the information of the user includes basic information about the user such as the user's name, affiliation, personal information, gender, age, contact information, e-mail, address, image, etc., and user information such as ID (or e-mail) and password. It may include information about authentication (login), country of access, access location, information about the device used for access, and information related to access, such as the connected network environment.

In addition, the user database includes the user's unique information, information and/or category history provided by the user who accessed the 3D image modeling application or 3D image modeling site, environment setting information set by the user, resource usage information used by the user, Billing and payment information corresponding to the user's resource usage may be stored.

The image processor 150 may generate a second image from which noise is removed from the first image including the 2D real-life building received from the user terminal 200 . The image processing unit 150 may generate a third image in which a plurality of components constituting the front of the building are classified from the second image. The image processing unit 150 may separate a floor from the third image and generate a shape grammar defining a morphological structure of a plurality of components included in the layer. The image processing unit 150 generates a 3D mesh for the front of the building using coordinate information about a plurality of components included in the layer and the layer obtained based on the grammar analysis result of the grammar analyzer taking the shape grammar as an input. can The image processing unit 150 may generate a 3D building modeling result by applying the 3D mesh to a shape file.

The controller 160, as a kind of central processing unit, can control the entire operation of the 3D image modeling device 100 by driving control software loaded in the program storage unit 130. The controller 160 may include all types of devices capable of processing data, such as a processor. Here, a 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or instructions included in a program, for example. As an example of such a data processing device built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit), field programmable gate array (FPGA), etc., but the scope of the present invention is not limited thereto.

FIG. 3 is a block diagram for schematically explaining the configuration of an image processing unit in the 3D image modeling apparatus of FIG. 2 , and FIGS. 4 to 10 are exemplary views illustrating 3D image modeling according to the present embodiment. In the following description, descriptions of portions overlapping those of FIGS. 1 and 2 will be omitted. 3 to 10, the image processing unit 150 includes a first generator 151, a second generator 152, a third generator 153, a fourth generator 154, and a fifth generator. may include section 155 .

The first generator 151 may generate a second image from which noise is removed from the first image including the 2D real-life building. As shown in 410 of FIG. 4 , the first generation unit 151 receives a first image including a 2D photorealistic building from the user terminal 200 so that the front of the building included in the first image faces the front. You can edit to see.

The first generator 151 may apply a K-means clustering algorithm to the first image corrected so that the front of the building faces the front, and generate a cartoon image obtained by averaging the centeroid value of each pixel. there is. In this embodiment, cartooning of an image may include a process of obtaining simplified colors and shapes from the image. Simplified images reduce information overload so they can be easily understood and recognized. 420 of FIG. 4 shows a cartoon image generated through predetermined processing of the first image. A cartoon image such as 420 in FIG. 4 has simple colors, no noise in shading, and a silhouette of an object (building) can be expressed by a shading effect. A luminance quantization technique may be used as a technique used in non-photorealistic rendering (NPR) to generate a cartoon effect. Brightness quantization is widely used to simplify the representation of shadows and can be cost effective in terms of time. A currently used brightness quantization technique may use a method of changing the brightness value of each pixel of an image to a representative value. The resulting image simplified by this method can have only a small number of brightness values.

The first generation unit 151 may convert the cartoonized image into a gray scale image. In this embodiment, a gray scale image may be converted into an approximately 8-bit gray scale image. The first generator 151 may generate an 8-bit gray scale image in which the brightest image information is 255 and the darkest image information is 0.

The first generator 151 may generate a contrast-enhanced image by performing histogram smoothing on the gray scale image. 430 of FIG. 4 shows a contrast enhancement image generated through a histogram smoothing process for a gray scale image. The first generator 151 generates a histogram based on the intensity values of each pixel included in the gray scale image, and smooths the histogram as the histogram is modified according to a preset monotonic transformation model to generate a contrast-enhanced image. can do.

The first generator 151 may generate a noise-removed second image by applying the contrast-enhanced image to a generative adversarial network (GAN). 5 shows a detailed block diagram of the first generator 151, which includes a preprocessor 151-1, a generator 151-2 for executing a generative adversarial network (GAN), and a discriminator. , 151-3). Referring to FIG. 5, the pre-processor 151-1 performs signal processing from the above-described first image 510 (same as 410 in FIG. 4) to generating a contrast-enhanced image 520 (same as 430 in FIG. 4). can be performed.

A generative adversarial network (GAN) is an artificial intelligence algorithm used for unsupervised learning. When the generator 151-2 generates fake data, the discriminator 151-3 uses real data based on It may refer to an algorithm that configures the generator 151-2 to generate simulated data almost similar to real data by repeatedly learning a process of probabilistically examining whether simulated data is real or fake.

In this embodiment, the generator 151-2 may generate and output a second image 530 from which noise is removed from the contrast-enhanced image 520 as the contrast-enhanced image 520 is input. In this embodiment, the noise-removed second image may be the above-described simulated data. When the second image 530 from which noise has been removed is input, the identifier 151-3 may determine whether the second image is authentic through comparison between the second image 530 and the target image. Here, the target image may include an image from which noise generated by the learner is removed. Parameters (eg, weights of the neural network) of the generator 151-2 and the identifier 151-3 may be adjusted according to the authenticity. In this embodiment, parameters of the generator 151-2 and the identifier 151-3 may be adjusted by the identifier 151-3.

The second generator 152 may generate a third image in which a plurality of components constituting the front of the building are classified from the noise-removed second image.

The second generation unit 152 may classify components of the building facade from the second image by using a deep neural network model previously trained to classify components of the building facade by inputting an image including the building facade. Here, the deep neural network model may be a neural network model pre-trained using training data including images including the front of a building and labels for components of the front of the building for each image. In this embodiment, the deep neural network model is a model based on HRNet, trained by a supervised learning method using training data including images including the front of a building and labels for components of the front of the building for each image. It can be.

The second generator 152 may generate a third image by including the result of classification in the second image.

6 shows a detailed block diagram of the second generating unit 152, which may include the HRNet 152-1. Referring to FIG. 6, the HRNet 152-1 may generate a third image 620 in which a plurality of components constituting the front of the building are classified from the second image 610 (same as 530 in FIG. 5). . The HRNet (152-1) may extract a feature map for classifying a plurality of components constituting the front of a building through effective convergence of a bottom-up path and a top-down path by taking the second image as an input.

HRNet (152-1) consists of a total of 4 sequential steps, and in each step, a feature map of 1/2 the size of the smallest resolution feature map is additionally created, and a residual block is formed by 4 times can be applied. Thereafter, as the last process of each step, an exchange process may be performed to fuse feature maps of all resolutions in a fully-connected manner. The exchange process in each stage consists of repeating the exchange unit several times, and it can be repeated once in stage 2, 4 times in stage 3, and 3 times in stage 4. In the exchange process, up sampling is performed using 1Х1 convolution and Nearest-Neighbor Interpolation in the bottom-up path, and down-sampling is performed using 3Х3 convolution with a stride of 2 in the bottom-up path. down sampling). On the other hand, in the parallel path, resolution can be maintained by using 1Х1 convolution.

The third generator 153 may separate a floor from the third image and generate a shape grammar defining a morphological structure of a plurality of components included in the layer.

The third generator 153 may convert the third image into a binary image divided into a first value (eg, 0) and a second value (eg, 1). Here, the threshold for conversion to a binary image can be set to 0.7. That is, the third generator 153 may generate a binary image by converting a pixel value included in the third image into a first value if the pixel value is less than 0.7, and converting it into a second value if the pixel value is greater than 0.7.

The third generation unit 153 generates a quadrangular area based on the first line and the second line generated by traversing each pixel included in the binarized image in the first and second directions, and pixels in the quadrangular area. A maximum quadrangular area may be created through a process of merging adjacent quadrangular areas having the same value. Here, the first direction may indicate a direction from left to right, and the second direction may indicate a direction from top to bottom. Also, the first line may represent a horizontal line, and the second line may represent a vertical line.

That is, the third generation unit 153 traverses (N×M) each pixel included in the binarized image from left to right, generates a horizontal line at the boundary between 0 and 1, and selects each pixel included in the binarized image from above. A vertical line is created at the boundary between 0 and 1 while traversing downward (N×M), and a quadrangular area can be created for the intersection area of each generated horizontal line and vertical line. Subsequently, when the pixel value 1 exists inside the quadrangular area, the third generation unit 153 sets the quadrangular area as a candidate area and performs a sweeping method to generate the maximum quadrangular area. can do.

710 of FIG. 7 shows an example in which horizontal lines and vertical lines are generated in the binarized image. 720 of FIG. 7 shows an example in which a quadrangular area is generated for the generated intersection area of a horizontal line and a vertical line. 730 of FIG. 7 shows an example in which the maximum quadrangular area is created through an operation of merging a quadrangular area having a pixel value of 1 therein and an adjacent quadrangular area.

The third generation unit 153 may generate coordinate information of the maximum quadrangular area based on (0,0) the lower left corner of the building where the maximum quadrangular area is created, and record it as text. In this embodiment, the third generation unit 153 generates the maximum squared area in the same way for the plurality of components described above, that is, windows, walls, doors, columns, shops, window frames, etc., and coordinates of the maximum squared area. Information can be created and recorded as text.

The third generation unit 153 visualizes the entire coordinate information of the maximum quadrangular area recorded as text for each component, performs an operation to merge vertical adjacent quadrangular areas within a threshold value of 0.8, and merges the combined quadrangular areas below. You can do the work of separating floors by traversing from top to bottom.

The third generation unit 153 may generate a form grammar for each of the separated layer and a plurality of components included in the separated layer and convert it into text. FIG. 8 shows an example in which a form grammar is generated from a layered image (810, the same as 740 in FIG. 7) and converted into text.

The fourth generation unit 154 generates a 3D mesh for the front of the building by using coordinate information about a plurality of components included in the layer and the layer obtained based on the grammar analysis result of the grammar analyzer taking the form grammar as an input. can create

The fourth generation unit 154 reads the text of the form grammar, puts it into a prepared grammar analyzer (not shown), and stores layer information and information about a plurality of elements included in the layer in a tree structure in memory (FIG. 11). 180) can be stored.

In this embodiment, the left child node and the right node of the tree may be arranged in order. Conventional type grammar trees have a general tree structure, but in this embodiment, a B+ tree structure can be selected to facilitate insertion, deletion, and modification of components.

The fourth generation unit 154 generates repetition values of x (width), y (length), w (width), and h (height) for a layer and a plurality of components included in the layer based on the grammar analysis result of the grammar analyzer. By triangulating, a 3D mesh for the front of the building can be created. 820 of FIG. 8B shows an example of generating a 3D mesh for the front of a building using the Delaunay triangulation method.

The fifth generator 155 may generate a 3D building modeling result by applying the 3D mesh to a 2D shape file.

The fifth generation unit 155 may receive a polygon selection signal to locate the 3D mesh among a plurality of polygons (faces) among attribute information included in the shape file. A 2D shape file is shown on the left side of FIG. 9, and any one polygon 910 among a plurality of polygons included in the shape file can be selected. An example of enlarging a partial region including the selected polygon 910 is shown on the right side of FIG. 9 .

The fifth generation unit 155 matches the coordinate information of the 3D mesh to the coordinate information of the selected polygon 910 (p1, p2, p3, p4 included in the left image of FIG. 9), and generates the selected polygon 910. It is possible to scale the size of the 3D mesh matched to the coordinate information for .

In order to fit the 3D mesh to the selected polygon 910, the fifth generation unit 155 calculates direction vectors p1 and p2 (L1 = p1-p2) for the selected polygon 910, and generates the selected polygon 910. After rotating the 3D mesh according to the angle of the direction vector L1 for , a 3D building modeling result can be generated by locating the rotated 3D mesh on the selected polygon 910 .

Specifically, the fifth generator 155 normalizes “? The fifth generator 155 may determine whether the first vector V1 intersects other polygons in the shape file, and determines whether the second vector V2 intersects other polygons in the shape file. there is.

The fifth generation unit 155 determines that a vector that does not intersect with other polygons in the shapefile, that is, the first vector V1 is an outward direction, and then calculates the angle between the normal of the 3D mesh and the first vector V1. and rotate the 3D mesh according to the calculated angle. Here, the normal of the 3D mesh may represent a result of processing such that the front direction of the 3D mesh matches the direction the camera is looking at. That is, the fifth generator 155 may rotate the 3D mesh by calculating an angle so that the direction of the selected polygon 910 and the 3D mesh coincide.

The fifth generation unit 155 maps the lower left corner of the 3D mesh to p1 of the selected polygon 910 to move the 3D mesh to which scaling and rotation are applied, and p2 of the selected polygon 910 to move the 3D mesh. You can create new coordinates by mapping the lower right corner. 10 shows a result of modeling the 3D mesh 1010 in a shape file.

In this way, the 3D image modeling apparatus 100 can reduce the user's manual work and processing time when modeling the 3D building image through the above-described processing of the front image of the 2D building.

11 is a block diagram schematically illustrating a configuration of a 3D image modeling device 100 according to another embodiment. In the following description, descriptions of portions overlapping those of FIGS. 1 to 10 will be omitted. Referring to FIG. 12 , a 3D image modeling apparatus 100 according to another embodiment may include a processor 170 and a memory 180.

In this embodiment, the processor 170 includes the communication unit 110, the storage medium 120, the program storage unit 130, the database 140, the image processing unit 150 and the control unit 160 disclosed in FIGS. 2 and 3. It can handle the functions it performs.

The processor 170 may control the entire operation of the 3D image modeling device 100 . Here, a 'processor' may refer to a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or instructions included in a program, for example. As an example of the data processing device built into the hardware, processing devices such as microprocessors, central processing units, processor cores, multiprocessors, ASICs, and FPGAs may be covered, but the scope of the present invention is not limited thereto. .

The memory 180 is operatively connected to the processor 170 and may store at least one code associated with an operation performed by the processor 170 .

In addition, the memory 180 may perform a function of temporarily or permanently storing data processed by the processor 170 and may include data built into the database 140 . Here, the memory 180 may include a magnetic storage medium or a flash storage medium, but the scope of the present invention is not limited thereto. Such memory 180 may include internal memory and/or external memory, volatile memory such as DRAM, SRAM, or SDRAM, OTPROM, PROM, EPROM, EEPROM, mask ROM, flash ROM, NAND flash memory, or NOR It may include a non-volatile memory such as a flash memory, a flash drive such as an SSD, a CF card, an SD card, a Micro-SD card, a Mini-SD card, an xD card, or a memory stick, or a storage device such as an HDD.

12 is a flowchart illustrating a 3D image modeling method according to an exemplary embodiment. In the following description, descriptions of portions overlapping with those of FIGS. 1 to 11 will be omitted. The 3D image modeling method according to the present embodiment will be described assuming that the 3D image modeling apparatus 100 is performed by the processor 170 with the help of peripheral components.

Referring to FIG. 12 , in step S1210, the processor 170 may generate a second image from which noise is removed from the first image including the 2D real-life building. In this embodiment, when a first image including a 2D photorealistic building is received from the user terminal 200, the processor 170 may modify the front of the building included in the first image to face the front. The processor 170 may generate a cartoon image by applying a K-means clustering algorithm to the first image and averaging the centeroid value of each pixel. The processor 170 may convert the cartoonized image into a gray scale image. The processor 170 may generate a contrast-enhanced image by performing histogram smoothing on the gray scale image. The processor 170 may generate a second image from which noise is removed by applying the contrast enhancement image to a generative adversarial network (GAN). Here, when generating the noise-removed second image using the generative adversarial network (GAN), the processor 170 generates the noise-removed second image from the contrast-enhanced image as the contrast-enhanced image is input to the generator. can be printed out. When the second image is input to the identifier, the processor 170 may determine authenticity of the second image through comparison between the second image and the target image. The processor 170 may adjust the parameters of the generator and identifier according to authenticity.

In step S1220, the processor 170 may generate a third image in which a plurality of components constituting the front of the building are classified from the second image. In this embodiment, the processor 170 may classify components of the building facade from the second image by using a deep neural network model previously trained to classify components of the building facade by inputting an image including the building facade. . Here, the deep neural network model may be a neural network model pre-trained using images including the front of a building and training data including labels for components of the front of the building for each image. For example, the deep neural network Models may include HRNet. Then, the processor 170 may generate a third image including the classification result in the second image.

In step S1230, the processor 170 may separate a floor from the third image and generate a shape grammar defining a morphological structure of a plurality of components included in the layer. In this embodiment, the processor 170 may convert the third image into a binary image divided into a first value and a second value. The processor 170 may generate a quadrangular area based on the first line and the second line generated by traversing each pixel included in the binarized image in the first and second directions. The processor 170 may generate a maximum quadrangular area through a process of merging adjacent quadrangular areas having the same pixel value in the quadrangular area. The processor 170 may generate and visualize a plurality of coordinate information based on the lower left corner of the building including the maximum quadrangular area, and then separate the floors. The processor 170 may generate a form grammar for the separated layer and a plurality of elements included in the separated layer.

In step S1240, the processor 170 creates a 3D mesh for the front of the building by using coordinate information on the layer and a plurality of components included in the layer obtained based on the grammar analysis result of the grammar analyzer taking the form grammar as an input. (mesh) can be created.

In step S1250, the processor 170 may generate a 3D building modeling result by applying the 3D mesh to a shape file. In this embodiment, the processor 170 may receive a polygon selection signal to locate a 3D mesh among a plurality of polygons included in the shape file. The processor 170 may match the coordinate information of the 3D mesh to the coordinate information of the selected polygon, and scale the size of the 3D mesh matched with the coordinate information. The processor 170 may rotate the 3D mesh according to the "? direction vector? * angle of the selected polygon, and then position the rotated 3D mesh on the selected polygon.

Embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed on a computer through various components, and such a computer program may be recorded on a computer-readable medium. At this time, the medium is a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM hardware devices specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. An example of a computer program may include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification of the present invention (particularly in the claims), the use of the term "above" and similar indicating terms may correspond to both singular and plural. In addition, when a range is described in the present invention, it includes an invention in which individual values belonging to the range are applied (unless there is a description to the contrary), and each individual value constituting the range is described in the detailed description of the invention Same as

The steps constituting the method according to the present invention may be performed in any suitable order unless an order is explicitly stated or stated to the contrary. The present invention is not necessarily limited according to the order of description of the steps. The use of all examples or exemplary terms (eg, etc.) in the present invention is simply to explain the present invention in detail, and the scope of the present invention is limited due to the examples or exemplary terms unless limited by the claims. it is not going to be In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments and should not be determined, and all scopes equivalent to or equivalently changed from the claims as well as the claims described below are within the scope of the spirit of the present invention. will be said to belong to

100: 3D image modeling device
200: user terminal
300: network

Claims (8)

A 3D image modeling method performed by a processor of a 3D image modeling device,
generating a second image from which noise is removed from a first image including a 2D real-life building;
generating a third image in which a plurality of components constituting the front of a building are classified from the second image;
separating a floor from the third image and generating a shape grammar defining a morphological structure of a plurality of elements included in the layer;
A 3D mesh for the front of the building is generated using coordinate information on the floor and a plurality of components included in the floor obtained based on the grammar analysis result of the grammar analyzer using the shape grammar as an input. doing; and
Generating a 3D building modeling result by applying the 3D mesh to a shape file,
The step of generating the shape grammar,
converting the third image into a binary image divided into a first value and a second value;
generating a quadrangular area based on a first line and a second line generated by traversing each pixel included in the binarized image in a first direction and a second direction;
generating a maximum quadrangular area through a process of merging adjacent quadrangular areas identical to pixel values in the quadrangular area;
Separating the floors after generating and visualizing a plurality of coordinate information based on the lower left corner of the building including the maximum quadrangular area; and
Generating a form grammar for the separated layer and a plurality of components included in the separated layer,
3D image modeling method. According to claim 1,
Generating the second image,
generating a cartoon image by applying a K-means clustering algorithm to the first image and averaging the centeroid value of each pixel;
converting the cartoonized image into a gray scale image;
generating a contrast-enhanced image by performing a histogram smoothing process on the gray scale image; and
Applying the contrast enhancement image to a generative adversarial network (GAN) to generate a second image from which the noise is removed,
3D image modeling method. According to claim 2,
Generating a second image from which the noise is removed,
generating and outputting a second image from which noise is removed from the contrast-enhanced image as the contrast-enhanced image is input to a generator;
If the second image is input to the identifier, determining whether the second image is authentic through comparison between the second image and the target image; and
Adjusting the parameters of the generator and the identifier according to the authenticity,
3D image modeling method. According to claim 1,
Generating the third image,
classifying components of the building facade from the second image by using a deep neural network model previously trained to classify components of the building facade by inputting an image including the building facade; and
generating a third image including the result of the classification in the second image;
The deep neural network model,
A neural network model pre-trained using training data including images including the front of a building and labels for components of the front of the building for each image,
3D image modeling method. delete According to claim 1,
The step of generating the 3D building modeling result,
Receiving a polygon selection signal to locate the 3D mesh among a plurality of polygons included in the shape file;
matching the coordinate information of the 3D mesh with the coordinate information of the selected polygon, and scaling the size of the 3D mesh matched with the coordinate information; and
After rotating the 3D mesh according to the angle of the direction vector for the selected polygon, locating the rotated 3D mesh on the selected polygon,
3D image modeling method. A computer-readable recording medium storing a computer program for executing the method of any one of claims 1 to 4 and 6 using a computer. As a three-dimensional image modeling device,
processor; and
a memory operatively connected to the processor and storing at least one code executed by the processor;
When the memory is executed by the processor, the processor generates a second image from which noise is removed from the first image including the 2D real-life building;
Creating a third image in which a plurality of components constituting the front of the building are classified from the second image;
Separating a floor from the third image and generating a shape grammar defining a morphological structure of a plurality of elements included in the layer;
A 3D mesh for the front of the building is generated using coordinate information on the floor and a plurality of components included in the floor obtained based on the grammar analysis result of the grammar analyzer using the shape grammar as an input. do,
Stores a code that causes a 3D building modeling result to be generated by applying the 3D mesh to a shape file;
The memory causes the processor to:
When generating the shape grammar, converting the third image into a binary image divided into a first value and a second value;
Creating a quadrangular area based on a first line and a second line generated by cycling each pixel included in the binarized image in a first direction and a second direction;
Creating a maximum quadrangular area through a process of combining adjacent quadrangular areas identical to pixel values in the quadrangular area;
After generating and visualizing a plurality of coordinate information based on the lower left corner of the building including the maximum quadrangular area, the floors are separated,
Storing code that causes to generate a form grammar for the separated layer and a plurality of components included in the separated layer,
3D image modeling device.

Images