KR20240046399A - Apparatus and method for modeling three dimensional image - Google Patents

Abstract

The present invention relates to a 3D image modeling device and method for modeling a 2D building front image into a 3D building image. The three-dimensional image modeling method according to this embodiment includes generating a second image from which noise is removed from a first image including a two-dimensional photo-realistic building, and generating a plurality of components constituting the front of the building from the second image. A step of generating a classified third image, separating a layer from the third image, and generating a shape grammar defining a morphological structure for a plurality of components included in the layer, and a grammar using the shape grammar as input. A step of creating a 3D mesh for the front of the building using coordinate information about the floor and a plurality of components included in the floor obtained based on the grammar analysis result of the interpreter, and applying the 3D mesh to the shape file. Generating dimensional building modeling results may be included.

Description

3D image modeling device and method {APPARATUS AND METHOD FOR MODELING THREE DIMENSIONAL IMAGE}

The present invention relates to a 3D image modeling device and method for modeling a 2D building front image into a 3D building image.

In general, tools for modeling 3D buildings include modeling tools based on building drawings and actual images, point cloud-based modeling tools using 3D scanners, and modeling tools based on multiple images. However, 3D building modeling requires the supply of building drawings and manual texturing skills from experienced users, and requires knowledge of various modeling tools, so there is a problem in that it takes varying amounts of time depending on the user's skill level.

The above-mentioned background technology is technical information that the inventor possessed for deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be known art disclosed to the general public before filing the application for the present invention.

Domestic Patent Publication No. 10-2012-0071276 (2012.07.02)

One object of the present invention is to reduce the user's manual work and processing time when modeling a 2D building front image into a 3D building image.

The problem to be solved by the present invention is not limited to the problems mentioned above, and other problems and advantages of the present invention that are not mentioned can be understood through the following description and can be understood more clearly through the examples of the present invention. It will be. In addition, it can be seen that the problems and advantages to be solved by the present invention can be realized by the means and combinations thereof indicated in the patent claims.

The 3D image modeling method according to this embodiment is a 3D image modeling method performed by a processor of a 3D image modeling device, and generates a second image from which noise is removed from a first image including a 2D photorealistic building. A step of generating a third image in which a plurality of components constituting the front of the building are classified from the second image, separating a layer from the third image, and morphologically determining the plurality of components included in the floor. The step of generating a shape grammar that defines the structure, and the coordinate information for the floor and a plurality of components included in the floor obtained based on the grammar analysis result of the grammar interpreter with the shape grammar as input, 3 for the front of the building It may include generating a dimensional mesh and applying the 3D mesh to a shape file to generate 3D building modeling results.

The three-dimensional image modeling device according to the present embodiment includes a processor and a memory operably connected to the processor and storing at least one code executed by the processor, and the memory is configured to, when executed through the processor, cause the processor to create a two-dimensional photo image. Generate a second image with noise removed from the first image including the building, generate a third image in which a plurality of components constituting the front of the building are classified from the second image, and separate the floors from the third image. , generate a morphological grammar that defines the morphological structure of a plurality of components included in the layer, and apply the morphological grammar to the layer and the plurality of components included in the layer obtained based on the grammar analysis results of a grammar interpreter with the morphological grammar as input. You can use the coordinate information to create a 3D mesh for the front of the building and save the code that generates 3D building modeling results by applying the 3D mesh to the shape file.

In addition, another method for implementing the present invention, another system, and a computer-readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages in addition to those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to the present invention, the user's manual work and processing time can be reduced when modeling a 2D building front image into a 3D building image.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is an exemplary diagram of a 3D image modeling environment according to this embodiment.
Figure 2 is a block diagram schematically illustrating the configuration of a 3D image modeling device according to this embodiment.
FIG. 3 is a block diagram schematically illustrating the configuration of an image processing unit in the 3D image modeling device of FIG. 2.
4 to 10 are exemplary diagrams explaining 3D image modeling according to this embodiment.
FIG. 11 is a block diagram schematically illustrating the configuration of a 3D image modeling device according to another embodiment.
Figure 12 is a flowchart for explaining the 3D image modeling method according to this embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but may be implemented in various different forms, and should be understood to include all conversions, equivalents, and substitutes included in the spirit and technical scope of the present invention. . The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof. Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by these terms. The above terms are used only for the purpose of distinguishing one component from another.

Additionally, in this application, a “part” may be a hardware component, such as a processor or circuit, and/or a software component executed by the hardware component, such as a processor.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components are assigned the same drawing numbers and duplicate descriptions thereof are omitted. I decided to do it.

In the following embodiments, terms such as first and second are used not in a limiting sense but for the purpose of distinguishing one component from another component.

In the following examples, singular terms include plural terms unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean the presence of features or components described in the specification, and do not exclude in advance the possibility of adding one or more other features or components.

In cases where an embodiment can be implemented differently, a specific process sequence may be performed differently from the described sequence. For example, two processes described in succession may be performed substantially at the same time, or may be performed in an order opposite to that in which they are described.

1 is an exemplary diagram of a 3D image modeling environment according to this embodiment. Referring to FIG. 1 , the 3D image modeling environment 1 may include a 3D image modeling device 100, a user terminal 200, and a network 300.

The 3D image modeling apparatus 100 may generate a second image from which noise is removed from a first image including a 2D photorealistic building received from the user terminal 200. The three-dimensional image modeling apparatus 100 may preprocess the first image to generate a second image from the first image, and apply an artificial intelligence algorithm (e.g., GAN (generative adversarial network) to the first image that has completed preprocessing. )) can be applied to create a second image.

The 3D image modeling apparatus 100 may generate a third image in which a plurality of components constituting the front of the building are classified from the second image. In this embodiment, the plurality of components constituting the front of the building may include windows, walls, doors, pillars, stores, window frames, etc. The 3D image modeling apparatus 100 may generate a second image by applying an artificial intelligence algorithm (eg, high-resolution network (HRNet)) to the second image.

The 3D image modeling apparatus 100 may separate a floor from the third image and generate a shape grammar that defines the morphological structure of a plurality of components included in the layer. Shape grammar is a concept derived from Christopher Alexander's Pattern Language, and can propose an automated method for grammaticalizing geometric patterns. By having the computer capture various shape patterns that exist inside complex geometric shapes, various shape patterns that are difficult for humans to find can be automatically extracted. Using shape grammar, it is possible to objectively extract shape patterns that commonly appear among various shape designs. Additionally, by grammatically analyzing the logic by which commonly used shape patterns are used repeatedly, it is possible to automatically generate several shapes that are similar but have not yet been derived.

The 3D image modeling device 100 creates a 3D mesh for the front of the building using coordinate information about the floor and a plurality of components included in the floor obtained based on the grammar analysis result of the grammar interpreter with shape grammar as input. (mesh) can be created. The mesh structure is a method of expressing a graphic model using a plurality of triangular elements and vertices, and can be widely used to construct a graphic model of a specific object and examine the physical characteristics of the object. In this embodiment, the Delaunay triangulation method can be used when creating a 3D mesh. Delaunay triangulation may mean dividing space by connecting points on a plane into triangles so that the minimum value of the interior angles of these triangles is maximum. Triangles created according to Delaunay triangulation are calculated to be close to equilateral triangles, so they are suitable for use as a mesh structure for graphic models.

The 3D image modeling device 100 may generate 3D building modeling results by applying a 3D mesh to a shape file. A shape file is a common standard format file used not only in ArcView but also in other GIS programs, 3D programs, AutoCad maps, etc., and includes a .shp file that stores the geometric information of the geographic map, and a .shp file that stores the index of the geometric information of the geographic map. shx file, .dbf file, which is a dBASE file that provides attribute information of the geographic map (either Point, Line, or Polygon), .sbn file that stores the spatial index of the geographic map, spatial join, etc. It consists of a .sbx file, which is a file necessary when performing the function of or creating an index for a shpe field, and files with the above five extensions can be executed as a bundle. These shape files allow you to quickly view spatial information, etc. in View, and easily edit, add, and delete spatial information and properties.

In this embodiment, the 3D image modeling device 100 may exist independently in the form of a server, or the functions provided by the 3D image modeling device 100 may be implemented in the form of an application and mounted on the user terminal 200.

The user terminal 200 may receive a 3D image modeling service by accessing a 3D image modeling application and/or a 3D image modeling site provided by the 3D image modeling device 100.

This user terminal 200 may include a communication terminal capable of performing the functions of a computing device (not shown), and in addition to the desktop computer 201, smartphone 202, and laptop 203 operated by the user, Tablet PCs, smart TVs, mobile phones, PDAs (personal digital assistants), media players, micro servers, GPS (global positioning system) devices, e-readers, digital broadcasting terminals, navigation, kiosks, MP3 players, digital cameras, home appliances, and It may be, but is not limited to, any other mobile or non-mobile computing device. Additionally, the user terminal 200 may be a wearable terminal such as a watch, glasses, hair band, or ring equipped with communication functions and data processing functions. This user terminal 200 is not limited to the above-described content, and any terminal capable of web browsing may be used without limitation.

The network 300 may serve to connect the 3D image modeling device 100 and the user terminal 200. This network 300 may be, for example, a wired network such as a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or an integrated service digital network (ISDN), a wireless LAN (WLAN), or a code division multiple access (CDMA) network. -division multiple access), and may encompass wireless networks such as satellite communication, but the scope of the present invention is not limited thereto. Additionally, the network 300 may transmit and receive information using short-range communication and/or long-distance communication. Here, short-range communication may include Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra-wideband (UWB), ZigBee, and Wi-Fi technology, and long-distance communication may include code-division multiple access (CDMA). ), frequency-division multiple access (FDMA), time-division multiple access (TDMA), orthogonal frequency-division multiple access (OFDMA), and single carrier frequency-division multiple access (SC-FDMA) technology.

Network 300 may include connections of network elements such as hubs, bridges, routers, and switches. Network 300 may include one or more connected networks, including public networks such as the Internet and private networks such as secure enterprise private networks, such as a multi-network environment. Access to network 300 may be provided through one or more wired or wireless access networks.

Network 300 may include connections of network elements such as hubs, bridges, routers, and switches. Network 300 may include one or more connected networks, including public networks such as the Internet and private networks such as secure enterprise private networks, such as a multi-network environment. Access to network 300 may be provided through one or more wired or wireless access networks.

Furthermore, the network 300 includes CAN (controller area network) communication, V2I (vehicle to infrastructure) communication, V2X (vehicle to everything) communication, wave (wireless access in vehicular environment) communication technology, objects, etc. It can support IoT (Internet of Things) networks and/or 5G communications that exchange and process information between distributed components.

Figure 2 is a block diagram schematically illustrating the configuration of a 3D image modeling device according to this embodiment. In the following description, parts that overlap with the description of FIG. 1 will be omitted. Referring to FIG. 2, the 3D image modeling device 100 includes a communication unit 110, a storage medium 120, a program storage unit 130, a database 140, an image processing unit 150, and a control unit 160. can do.

The communication unit 110 may work with the network 300 to provide a communication interface necessary to provide transmission and reception signals between the 3D image modeling device 100 and the user terminal 200 in the form of packet data. Furthermore, the communication unit 110 may serve to receive a predetermined information request signal from the user terminal 200 and transmit information processed by the image processing unit 150 to the user terminal 200. . Here, the communication network refers to a medium that connects the 3D image modeling device 100 and the user terminal 200, and the user terminal 200 receives information after accessing the 3D image modeling device 100. It may include a path that provides a connection path to transmit and receive. Additionally, the communication unit 110 may be a device that includes hardware and software necessary to transmit and receive signals such as control signals or data signals through wired or wireless connections with other network devices.

The storage medium 120 functions to temporarily or permanently store data processed by the control unit 160. Here, the storage medium 120 may include magnetic storage media or flash storage media, but the scope of the present invention is not limited thereto. This storage medium 120 may include internal memory and/or external memory, volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, etc. , non-volatile memory such as NAND flash memory, or NOR flash memory, SSD. It may include a flash drive such as a compact flash (CF) card, SD card, Micro-SD card, Mini-SD card, Xd card, or memory stick, or a storage device such as an HDD.

The program storage unit 130 generates a second image from which noise has been removed from a first image including a two-dimensional photo-realistic building, and generates a third image from the second image in which a plurality of components constituting the front of the building are classified. Generating work, separating a floor from a third image, creating a shape grammar that defines the morphological structure of a plurality of components included in the layer, grammar analysis result of a grammar interpreter using the shape grammar as input The task of creating a 3D mesh for the front of the building using the coordinate information for the floor and multiple components included in the floor obtained based on , and generating 3D building modeling results by applying the 3D mesh to the shape file. It can be equipped with control software that performs certain tasks.

The database 140 may include a management database that stores various tools for generating 3D building modeling results from 2D building images. For example, various image processing algorithms and artificial intelligence algorithms for generating a second image from a first image may be stored in the management database. The management database may store an artificial intelligence algorithm for classifying multiple components that make up the front of a building. An algorithm for generating a morphological grammar to define a morphological structure for a plurality of components may be stored in the management database. An algorithm for generating a 3D mesh may be stored in the management database. Shape files for generating 3D building modeling results may be stored in the management database.

Additionally, the database 140 may include a user database that stores information on users who will be provided with a 3D image modeling service. Here, the user's information includes basic information about the user such as the user's name, affiliation, personal information, gender, age, contact information, email, address, and image, and the user's ID (or email) and password. It may include information related to the connection, such as information about authentication (login), country of connection, location of connection, information about the device used for connection, and the connected network environment.

In addition, the user database includes the user's unique information, information and/or category history provided by users who access the 3D image modeling application or 3D image modeling site, environment setting information set by the user, resource usage information used by the user, Billing and payment information corresponding to the user's resource usage may be stored.

The image processing unit 150 may generate a second image from which noise has been removed from the first image including a two-dimensional photorealistic building received from the user terminal 200. The image processing unit 150 may generate a third image in which a plurality of components constituting the front of the building are classified from the second image. The image processing unit 150 may separate a floor from the third image and generate a shape grammar that defines the morphological structure of a plurality of components included in the layer. The image processing unit 150 generates a three-dimensional mesh for the front of the building using coordinate information about the floor and a plurality of components included in the floor obtained based on the grammar analysis result of the grammar interpreter with the shape grammar as input. You can. The image processing unit 150 may generate 3D building modeling results by applying a 3D mesh to a shape file.

The control unit 160 is a type of central processing unit and can control the entire operation of the three-dimensional image modeling device 100 by driving control software mounted on the program storage unit 130. The control unit 160 may include all types of devices that can process data, such as a processor. Here, 'processor' may mean, for example, a data processing device built into hardware that has a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of data processing devices built into hardware include a microprocessor, central processing unit (CPU), processor core, multiprocessor, and application-specific integrated (ASIC). circuit) and FPGA (field programmable gate array), etc., but the scope of the present invention is not limited thereto.

FIG. 3 is a block diagram schematically illustrating the configuration of an image processing unit in the 3D image modeling device of FIG. 2 , and FIGS. 4 to 10 are diagrams illustrating 3D image modeling according to this embodiment. In the following description, parts that overlap with the description of FIGS. 1 and 2 will be omitted. 3 to 10, the image processing unit 150 includes a first generation unit 151, a second generation unit 152, a third generation unit 153, a fourth generation unit 154, and a fifth generation unit. It may include part 155.

The first generator 151 may generate a second image from which noise is removed from a first image including a 2D photorealistic building. As shown at 410 in FIG. 4, when the first image including a two-dimensional photorealistic building is received from the user terminal 200, the first generator 151 generates the front of the building included in the first image to face the front. You can edit it to see it.

The first generator 151 may apply the K-means clustering algorithm to the first image modified so that the front of the building faces the front, and generate a cartoon image averaged with the centerroid value of each pixel. there is. In this embodiment, cartooning an image may include a process of obtaining simplified colors and shapes from the image. Simplified images reduce information overload and can be easily understood and recognized. At 420 in FIG. 4, a cartoonized image generated through predetermined processing of the first image is shown. A cartoon image such as 420 in FIG. 4 has simple colors, no noise in the shaded portion, and the silhouette of an object (building) can be expressed by a shading effect. To create a cartoon effect, luminance quantization can be used as a technique used in non-photorealistic rendering (NPR). Brightness quantization is widely used to simplify the representation of shadows and can be efficient in terms of time cost. The brightness quantization technique currently used can be used to change the brightness value of each pixel in the image to a representative value. The resulting image simplified by this method can have only a small number of brightness values.

The first generator 151 may convert the cartoonized image into a gray scale image. In this embodiment, the gray scale image can be converted to an approximately 8-bit gray scale image. The first generator 151 may generate an 8-bit gray scale image in which the brightest image information is set to 255 and the darkest image information is set to 0.

The first generator 151 may generate a contrast-enhanced image by performing histogram equalization processing on the gray scale image. At 430 in FIG. 4, a contrast-enhanced image generated through histogram equalization processing for a gray scale image is shown. The first generator 151 generates a histogram based on the intensity values of each pixel included in the gray scale image, and smoothes the histogram by modifying the histogram according to a preset monotonic transformation model to generate a contrast-enhanced image. can do.

The first generator 151 may apply the contrast-enhanced image to a generative adversarial network (GAN) to generate a second image from which noise has been removed. Figure 5 shows a detailed block diagram of the first generator 151, and includes a preprocessor 151-1, a generator 151-2 for executing a generative adversarial network (GAN), and a discriminator. , 151-3). Referring to FIG. 5, the preprocessor 151-1 performs signal processing from the above-described first image (510, same as 410 in FIG. 4) to generating a contrast-enhanced image (520, same as 430 in FIG. 4). can be performed.

Generative adversarial network (GAN) is an artificial intelligence algorithm used in unsupervised learning. When the generator 151-2 generates fake data, the identifier 151-3 uses real data based on the real data. This may refer to an algorithm that ultimately configures the generator 151-2 to generate simulated data that is almost similar to real data by repeatedly learning the process of probabilistically examining whether the simulated data is real or fake.

In this embodiment, the generator 151-2 may generate and output a second image 530 from which noise is removed from the contrast-enhanced image 520 as the contrast-enhanced image 520 is input. In this embodiment, the second image from which noise has been removed may be the simulated data described above. When the second image 530 from which noise has been removed is input, the identifier 151-3 can determine whether the second image is authentic by comparing the second image 530 and the target image. Here, the target image may include an image from which noise generated by the learner has been removed. Depending on the authenticity, the parameters (eg, weights of the neural network) of the generator 151-2 and the identifier 151-3 may be adjusted. In this embodiment, parameter adjustment of the generator 151-2 and the identifier 151-3 may be performed in the identifier 151-3.

The second generator 152 may generate a third image in which a plurality of components constituting the front of the building are classified from the second image from which noise has been removed.

The second generator 152 may input an image including the front of the building and classify the components of the front of the building from the second image using a deep neural network model that has been previously trained to classify the components of the front of the building. Here, the deep neural network model may be a neural network model trained in advance using images including the front of the building and training data including labels for components of the front of the building for each image. In this embodiment, the deep neural network model is a model based on HRNet, and is trained in a supervised learning method using images including the front of the building and training data including labels for components of the front of the building for each image. It can be.

The second generator 152 may generate a third image that includes the classification result in the second image.

FIG. 6 shows a detailed block diagram of the second generator 152, which may include an HRNet 152-1. Referring to FIG. 6, the HRNet 152-1 may generate a third image 620 in which a plurality of components constituting the front of the building are classified from the second image 610 (same as 530 in FIG. 5). . HRNet 152-1 can extract a feature map for classifying a plurality of components constituting the front of a building through effective fusion of a bottom-up path and a top-down path using the second image as input.

HRNet (152-1) consists of a total of 4 sequential steps, and in each step, a feature map of 1/2 the size of the smallest resolution feature map is additionally created, and then residual blocks are created. It can be applied once. Afterwards, as the final process of each step, an exchange process can be performed to fuse feature maps of all resolutions in a fully-connected manner. The exchange process at each stage consists of multiple repetitions of exchange units, which can be repeated once in stage 2, 4 times in stage 3, and 3 times in stage 4. In the exchange process, the bottom-up path uses 1Х1 convolution and nearest-neighbor interpolation for up sampling, and the top-down path uses 3Х3 convolution with a stride of 2 to downsample ( down sampling) is possible. Meanwhile, in the parallel path, resolution can be maintained by using 1Х1 convolution.

The third generator 153 may separate a floor from the third image and generate a shape grammar that defines the morphological structure of a plurality of components included in the layer.

The third generator 153 may convert the third image into a binarized image divided into a first value (eg, 0) and a second value (eg, 1). Here, the threshold for conversion to a binarized image can be set to 0.7. That is, the third generator 153 can generate a binarized image by converting the pixel value included in the third image to a first value if it is less than 0.7, and converting it to a second value if the pixel value is more than 0.7.

The third generator 153 generates a squared area based on the first and second lines generated by traversing each pixel included in the binarized image in the first and second directions, and pixels within the squared area The maximum rectangular area can be created through the process of combining adjacent rectangular areas with the same value. Here, the first direction may represent a left to right direction, and the second direction may represent a top to bottom direction. Additionally, the first line may represent a horizontal line, and the second line may represent a vertical line.

That is, the third generator 153 traverses each pixel included in the binarized image from left to right (N It traverses downward (N Subsequently, if a pixel value of 1 exists inside the squared area, the third generator 153 sets the squared area as a candidate area and performs a sweep method to merge it with adjacent squared areas to generate the maximum squared area. can do.

710 in FIG. 7 shows an example of horizontal and vertical lines being created in a binarized image. 720 in FIG. 7 shows an example in which a squared area is created in the intersection area of the generated horizontal and vertical lines. 730 in FIG. 7 shows an example in which the maximum rectangular area is created by merging a rectangular area with a pixel value of 1 inside and an adjacent rectangular area.

The third generator 153 may generate coordinate information of the maximum square area using the lower left corner of the building where the maximum square block area is created as a reference (0,0) and record it as text. In this embodiment, the third generation unit 153 generates the maximum square area in the same manner for the plurality of components described above, that is, windows, walls, doors, pillars, stores, window frames, etc., and coordinates the maximum square area. Information can be generated and recorded as text.

The third generator 153 visualizes the entire coordinate information of the maximum rectangular area recorded as text for each component, performs the task of merging vertically adjacent rectangular areas within a threshold of 0.8, and places the combined rectangular areas below. You can perform the task of separating floors by traversing from top to bottom.

The third generation unit 153 may generate a shape grammar for each of the separated layer and a plurality of components included in the separated layer and convert it into text. FIG. 8 shows an example of generating a shape grammar from an image with separated layers (810, same as 740 in FIG. 7) and converting it into text.

The fourth generator 154 creates a three-dimensional mesh for the front of the building using coordinate information about the floor and a plurality of components included in the floor obtained based on the grammar analysis result of the grammar interpreter with the shape grammar as input. can be created.

The fourth generator 154 reads the text of the morphological grammar, puts it into a prepared grammar interpreter (not shown), and stores layer information and information about a plurality of components included in the layer in a tree structure (tree) (see FIG. 11). 180).

In this embodiment, the nodes from the left child node to the right node of the tree may be arranged in order. The form grammar tree used previously has a general tree structure, but in this embodiment, a B+ tree structure can be selected to facilitate insertion, deletion, and modification of components.

The fourth generator 154 generates x (width), y (vertical), w (width), and h (height) repetition values for the layer and a plurality of components included in the layer based on the grammar analysis result of the grammar interpreter. By triangulating, you can create a 3D mesh for the front of the building. 820 in FIG. 8B shows an example of generating a 3D mesh for the front of a building using the Delaunay triangulation method.

The fifth generation unit 155 may generate a 3D building modeling result by applying the 3D mesh to a 2D shape file.

The fifth generator 155 may receive a polygon selection signal in which to position the 3D mesh among a plurality of polygons (faces) among the attribute information included in the shape file. A two-dimensional shape file is shown on the left side of FIG. 9, and any one polygon 910 among a plurality of polygons included in the shape file can be selected. On the right side of FIG. 9, an example of enlarging a partial area including the selected polygon 910 is shown.

The fifth generator 155 matches the coordinate information of the 3D mesh to the coordinate information (p1, p2, p3, p4 included in the left image of FIG. 9) for the selected polygon 910, and The size of the 3D mesh matched to the coordinate information can be scaled.

The fifth generator 155 calculates the direction vectors (L1=p1-p2) of p1 and p2 for the selected polygon 910 in order to fit the 3D mesh to the selected polygon 910, and After rotating the 3D mesh according to the angle of the direction vector (L1), 3D building modeling results can be generated by placing the rotated 3D mesh on the selected polygon 910.

Specifically, the fifth generator 155 normalizes the direction vector (L1) for the selected polygon 910 and then obtains the first vector (V1) and the second vector (V2) rotated by 90 degrees -90 degrees. You can. The fifth generator 155 may determine whether the first vector (V1) intersects other polygons in the shape file and determine whether the second vector (V2) intersects other polygons in the shape file.

The fifth generator 155 determines that the vector that does not intersect with other polygons in the shape file, that is, the first vector (V1), is in the outward direction, and then calculates the angle between the normal of the 3D mesh and the first vector (V1). And the 3D mesh can be rotated according to the calculated angle. Here, the normal of the 3D mesh can represent the result of processing the front direction of the 3D mesh to match the direction the camera is looking. That is, the fifth generator 155 may rotate the 3D mesh by calculating an angle so that the direction of the selected polygon 910 matches the 3D mesh.

In order to move the 3D mesh to which scaling and rotation have been applied, the fifth generator 155 maps the lower left corner of the 3D mesh to p1 of the selected polygon 910, and maps the bottom left corner of the 3D mesh to p2 of the selected polygon 910. You can create new coordinates by mapping the bottom right corner. Figure 10 shows the result of modeling the 3D mesh 1010 in a shape file.

In this way, the 3D image modeling device 100 can reduce the user's manual work and processing time when modeling a 3D building image through the above-described processing on the 2D building front image.

FIG. 11 is a block diagram schematically illustrating the configuration of a 3D image modeling device 100 according to another embodiment. In the following description, parts that overlap with the description of FIGS. 1 to 10 will be omitted. Referring to FIG. 12, a 3D image modeling device 100 according to another embodiment may include a processor 170 and a memory 180.

In this embodiment, the processor 170 includes the communication unit 110, storage medium 120, program storage unit 130, database 140, image processing unit 150, and control unit 160 disclosed in FIGS. 2 and 3. The function it performs can be processed.

This processor 170 can control the entire operation of the 3D image modeling device 100. Here, 'processor' may mean, for example, a data processing device built into hardware that has a physically structured circuit to perform a function expressed by code or instructions included in a program. Examples of data processing devices built into hardware include processing devices such as microprocessors, central processing units, processor cores, multiprocessors, ASICs, and FPGAs, but the scope of the present invention is not limited thereto. .

The memory 180 is operatively connected to the processor 170 and can store at least one code in association with an operation performed by the processor 170.

Additionally, the memory 180 may perform a function of temporarily or permanently storing data processed by the processor 170 and may include data established as the database 140. Here, the memory 180 may include a magnetic storage medium or a flash storage medium, but the scope of the present invention is not limited thereto. Such memory 180 may include internal memory and/or external memory, such as volatile memory such as DRAM, SRAM, or SDRAM, OTPROM, PROM, EPROM, EEPROM, mask ROM, flash ROM, NAND flash memory, or NOR. It may include non-volatile memory such as flash memory, flash drives such as SSD, CF card, SD card, Micro-SD card, Mini-SD card, xD card, or memory stick, or storage devices such as HDD.

Figure 12 is a flowchart for explaining the 3D image modeling method according to this embodiment. In the following description, parts that overlap with the description of FIGS. 1 to 11 will be omitted. The 3D image modeling method according to this embodiment will be described assuming that the 3D image modeling device 100 is performed by the processor 170 with the help of peripheral components.

Referring to FIG. 12 , in step S1210, the processor 170 may generate a second image from which noise has been removed from the first image including the 2D photorealistic building. In this embodiment, when the processor 170 receives a first image including a two-dimensional photorealistic building from the user terminal 200, the processor 170 may modify the front of the building included in the first image to face the front. The processor 170 may apply the K-means clustering algorithm to the first image to generate a cartoon image averaged with the centerroid value of each pixel. The processor 170 may convert the cartoonized image into a gray scale image. The processor 170 may generate a contrast-enhanced image by performing histogram equalization processing on the gray scale image. The processor 170 may generate a second image from which noise has been removed by applying the contrast-enhanced image to a generative adversarial network (GAN). Here, when generating a second image from which noise has been removed using a generative adversarial network (GAN), the processor 170 generates a second image from which noise has been removed from the contrast-enhanced image as the contrast-enhanced image is input to the generator. You can print it out. When the second image is input to the identifier, the processor 170 can determine whether the second image is authentic by comparing the second image and the target image. Processor 170 may adjust parameters of the generator and identifier depending on authenticity.

In step S1220, the processor 170 may generate a third image in which a plurality of components constituting the front of the building are classified from the second image. In this embodiment, the processor 170 inputs an image including the front of the building and classifies the components of the front of the building from the second image using a deep neural network model that has been pre-trained to classify the components of the front of the building. . Here, the deep neural network model may be a neural network model pre-trained using images containing the front of the building and training data containing labels for components of the front of the building for each image, for example, a deep neural network The model may include HRNet. Thereafter, the processor 170 may generate a third image including the classification result in the second image.

In step S1230, the processor 170 may separate a floor from the third image and generate a shape grammar that defines the morphological structure of a plurality of components included in the layer. In this embodiment, the processor 170 may convert the third image into a binarized image divided into first and second values. The processor 170 may generate a squared area based on the first and second lines generated by traversing each pixel included in the binarized image in the first and second directions. The processor 170 may generate the maximum quadrangle area through a process of combining adjacent quadrangle areas that have the same pixel value within the quadrangle area. The processor 170 may generate and visualize a plurality of coordinate information based on the lower left corner of the building including the maximum rectangular area, and then separate the floors. The processor 170 may generate a shape grammar for a separated layer and a plurality of components included in the separated layer.

In step S1240, the processor 170 creates a three-dimensional mesh for the front of the building using coordinate information about the floor and a plurality of components included in the floor obtained based on the grammar analysis result of the grammar interpreter with the shape grammar as input. (mesh) can be created.

In step S1250, the processor 170 may generate a 3D building modeling result by applying the 3D mesh to a shape file. In this embodiment, the processor 170 may receive a polygon selection signal to position the 3D mesh among a plurality of polygons included in the shape file. The processor 170 may match the coordinate information of the 3D mesh to the coordinate information for the selected polygon and scale the size of the 3D mesh with the matched coordinate information. The processor 170 may rotate the 3D mesh according to the angle of the direction vector for the selected polygon and then position the rotated 3D mesh on the selected polygon.

Embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. At this time, the media includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROM. , RAM, flash memory, etc., may include hardware devices specifically configured to store and execute program instructions.

Meanwhile, the computer program may be designed and configured specifically for the present invention, or may be known and available to those skilled in the art of computer software. Examples of computer programs may include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification (particularly in the claims) of the present invention, the use of the term “above” and similar referential terms may refer to both the singular and the plural. In addition, when a range is described in the present invention, the invention includes the application of individual values within the range (unless there is a statement to the contrary), and each individual value constituting the range is described in the detailed description of the invention. It's the same.

Unless there is an explicit order or statement to the contrary regarding the steps constituting the method according to the invention, the steps may be performed in any suitable order. The present invention is not necessarily limited by the order of description of the above steps. The use of any examples or illustrative terms (e.g., etc.) in the present invention is merely to describe the present invention in detail, and unless limited by the claims, the scope of the present invention is limited by the examples or illustrative terms. It doesn't work. Additionally, those skilled in the art will recognize that various modifications, combinations and changes may be made depending on design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

100: 3D image modeling device
200: user terminal
300: Network

Claims (6)

A three-dimensional image modeling method performed by a processor of a three-dimensional image modeling device, comprising:
Generating a second image from which noise has been removed from a first image including a two-dimensional photo-realistic building;
generating a third image in which a plurality of components constituting the front of the building are classified from the second image;
separating a floor from the third image and generating a shape grammar defining a morphological structure for a plurality of components included in the layer;
Creating a three-dimensional mesh for the front of the building using coordinate information about the floor and a plurality of components included in the floor obtained based on the grammar analysis result of a grammar interpreter using the shape grammar as input. steps; and
Generating a 3D building modeling result by applying the 3D mesh to a shape file,
The step of generating the 3D building modeling results is,
Receiving a polygon selection signal to position the 3D mesh among a plurality of polygons included in the shape file;
Matching coordinate information of the 3D mesh to coordinate information for the selected polygon, and scaling the size of the 3D mesh with the matched coordinate information; and
Including rotating the three-dimensional mesh according to the angle of the direction vector for the selected polygon, and then positioning the rotated three-dimensional mesh in the selected polygon.
3D image modeling method. According to claim 1,
The step of generating the second image is,
Applying a K-means clustering algorithm to the first image to generate a cartoon image averaged with the centerroid value of each pixel;
Converting the cartoonized image into a gray scale image;
generating a contrast-enhanced image by performing histogram equalization on the gray scale image; and
Applying the contrast-enhanced image to a generative adversarial network (GAN) to generate a second image from which the noise has been removed,
3D image modeling method. According to claim 2,
The step of generating a second image from which the noise has been removed is,
When the contrast-enhanced image is input to a generator, generating and outputting a second image from which noise has been removed from the contrast-enhanced image;
When the second image is input to the identifier, determining whether the second image is authentic by comparing the second image and the target image; and
Including adjusting parameters of the generator and the identifier according to the authenticity,
3D image modeling method. According to claim 1,
The step of generating the third image is,
Inputting an image including the front of a building and classifying the components of the front of the building from the second image using a deep neural network model previously trained to classify the components of the front of the building; and
Generating a third image that includes the results of the classification in the second image,
The deep neural network model is,
A neural network model pre-trained using images containing the front of the building and training data containing labels for components of the front of the building for each image,
3D image modeling method. A computer-readable recording medium storing a computer program for executing the method of any one of claims 1 to 4 using a computer. A three-dimensional image modeling device,
processor; and
a memory operably connected to the processor and storing at least one code to be executed by the processor;
When the memory is executed through the processor, the processor generates a second image with noise removed from the first image including a two-dimensional photorealistic building,
Generate a third image in which a plurality of components constituting the front of the building are classified from the second image,
Separating a floor from the third image and generating a shape grammar that defines a morphological structure for a plurality of components included in the layer,
Creating a three-dimensional mesh for the front of the building using coordinate information about the floor and a plurality of components included in the floor obtained based on the grammar analysis results of a grammar interpreter using the shape grammar as input. do,
storing code that causes the 3D mesh to be applied to a shape file to generate a 3D building modeling result,
The memory allows the processor to:
When generating the 3D building modeling result, receive a polygon selection signal to position the 3D mesh among a plurality of polygons included in the shape file,
Matching the coordinate information of the 3D mesh to the coordinate information for the selected polygon, scaling the size of the 3D mesh with the matched coordinate information,
After rotating the three-dimensional mesh according to the angle of the direction vector for the selected polygon, storing a code that causes the rotated three-dimensional mesh to be located in the selected polygon,
3D image modeling device.

Images