KR20220069145A - method and system for building DB of 3D cultural heritage by combining data - Google Patents

Abstract

The present invention relates to a method and a system for constructing a database of a three-dimensional (3D) cultural asset through data combination, in which even 3D cultural asset spatial information with high-quality is realized. The method of the present invention comprises: a cultural asset image data acquisition step; a contour processing step; a 3D image restoration step; and a 3D DB utilization web service environment establishing step.

Description

Method and system for building DB of 3D cultural heritage by combining data

The present invention relates to a method and system for constructing a database of three-dimensional cultural properties through data combination, and a three-dimensional It is possible to have a vast terabyte-level DB of cultural assets in the virtual reality of It relates to a database construction method and system of

According to the announcement of the 'Guidelines for Building 3D Scanned Data for Cultural Heritage' announced by the Cultural Heritage Administration, which manages cultural assets, it is announced that 3D scan data can only be constructed using lidar.

As such, constructing 3D scan data using a method using lidar consumes a lot of equipment, time, and manpower, and the data capacity is also huge, so it is not easy to use it.

Of course, in addition to this method using lidar, objects of cultural heritage (cultural heritage, buildings and large sculptures, historic sites and excavations) can be 3D aerially photographed at an appropriate altitude using the video camera and altimeter mounted on the drone. Scan data may be acquired and secured.

However, the amount of data of the aerial data of the cultural object acquired from the drone is huge, and furthermore, even the ground data of the cultural object can be merged with these aerial data, but the DB processing speed due to the vast DB of the merged data Due to the delay, the DB is not utilized properly and is neglected.

In addition, virtual experience technology for cultural objects in 3D reality using such a vast DB is also being distributed, but it does not reach the level of realistic immersiveness. can't be done

Patent Document 001: Registered Patent No. 10-1863188 (Registration Date 2018. 05. 25)

The present invention for solving the above-mentioned problems, in providing a method and system for building and managing cultural objects in a virtual three-dimensional reality, along with improving the processing speed of the DB due to the system load due to the vast DB holding The purpose of this is to provide a method and system for constructing a database of 3D cultural properties through data combination to realize the efficient use of memory of the system and the possession and construction of a massive terabyte-level image processing DB through distributed processing technology.

The present invention for achieving the above objects is a drone in which aerial data, which is an aerial image image of cultural property taken from the drone 100, and ground data, which is a ground image image of cultural property, taken from the ground camera 200 are obtained; The object recognition algorithm 310 recognizes the object of the image as an outline based on the aerial data and ground data of the cultural property acquired in the step 10 of acquiring image data of the cultural property using a ground camera and selected as a test bed in the step 10 3D restoration interlocked in the artificial intelligence server 300 based on the aerial data and ground data in which the image object was recognized as an outline in the processing step 20 of the outline based on the acquired data to be processed, and the processed image object in the step 20 3D image restoration step 30 using artificial intelligence to restore 3D image through program 410, and 3D immersive content that is compatible with the 3D cultural property restored in step 30 can be displayed on the website An example of a method of constructing a database of three-dimensional cultural properties through data combination consisting of a configuration including the step 40 of constructing a web service environment using the 3D DB that is built and provided through the web viewer program 510 installed in the web server 500 in the form has the characteristics of

The terrestrial data photographed from the terrestrial camera 200 in the step (10) is carried out in such a way as to be photographed at the same distance, a gradual photographing distance is secured, and while photographing at 360° in different directions, overlapping images of at least 70% or more On the other hand, the aerial data photographed from the drone 100 in the step 10 maintains a redundancy of at least 80% in length and breadth using a cross-pass photographing technique, and builds a database of three-dimensional cultural properties through combining the photographed data The method has exemplary features.

Aviation data and ground data recognized and processed by the object recognition algorithm 310 in the step 20 are processed in the segmental labeling method for each category through the image segmentation algorithm 311, while the segmental labeling method by category Validation of image matching for aerial data and ground data processed with , has an example of a method for constructing a database of three-dimensional cultural properties through data combination made through the image matching algorithm 320 .

Between the above steps (10) and (20), the obtained aerial data and ground data are projected into a three-dimensional space, and errors or unnecessary data generated from the 3D scan data are optimized in a way that the cloud server 1000 organizes them. There is an example of a method for constructing a database of three-dimensional cultural assets through data combination in which a low-capacity cloud creation step 15 to create a low-capacity cloud is further performed.

Between the steps 30 and 40, the 3D cultural heritage restored in the step 30 can be displayed on a website compatible with the 3D editing tool 1100 installed in the cloud server 1000. A 3D viewer and a method of constructing a database of 3D cultural assets through data combination in which the 3D viewer production step 35 is further performed is an example of a 3D viewer and compatibility of producing VR and AR immersive contents in the format.

The 3D editing tool 1100 inserts a 3D viewer on the canvas created on the website and connects the low-capacity cloud while providing a free viewpoint through the user's input according to the creation of a virtual camera while utilizing the low-capacity cloud. An example of a method for constructing a database of three-dimensional cultural properties through data combination to produce three-dimensional sensory content synthesized in real time that can react with the surrounding environment has a characteristic.

On the other hand, the present invention for achieving the above objects is a drone 100 equipped with a video camera 110 for photographing an object of cultural property in the air, a ground camera 200 for photographing an object of cultural property on the ground, a drone and an artificial intelligence server 300 that performs object recognition processing of aerial data acquired from a ground camera and an image of ground data, which is interlocked with the artificial intelligence server and converts images of the data (aviation data and ground data) into three-dimensional 3D A three-dimensional restoration program 410 that processes to restore an image, and a web server 500 that provides a three-dimensionally restored 3D cultural property in a form of three-dimensional immersive content displayable and compatible on a website. Another example is the database construction system of 3D cultural properties through data combination.

According to the present invention as described above, by merging the point cloud obtained from the aerial data and the ground data together with the combined processing of aerial data for cultural objects photographed from a drone and ground data photographed from a ground camera, one 3D As a result can be produced with a model, and a construction method is provided for this 3D model to be displayed on the website, it not only uses the system's memory efficiently and has a terabyte-level image processing DB through distributed processing technology, but also provides a high-quality 3D model. The effect of building spatial information for dimensional cultural properties can be expected.

1 is a block diagram illustrating a processing sequence for a method for constructing a database of three-dimensional cultural properties through data combination according to an embodiment of the present invention.
2 is a diagram illustrating a method of photographing an object (cultural property) using the ground camera 200 as an example concept.
3 is a diagram illustrating a method of photographing an object (cultural property) using the drone 100 as an example concept.
4 is a diagram illustrating a method of photographing an object (cultural property) using the drone 100 as another example concept.
5 is a diagram illustrating a method of photographing an object (cultural property) using the drone 100 as another example concept.
6 is a diagram conceptually illustrating a process of merging the aerial data acquired from the drone 100 and the ground data acquired from the ground camera 200 .
7 is a diagram conceptually illustrating an outline recognition processing method of the object recognition algorithm 310 for processing of aerial data and ground data.
8 is a diagram conceptually illustrating a segmental labeling processing method for each category of the image segmentation algorithm 311 for aerial data and ground data processing recognized and processed by the object recognition algorithm 310 .
9 shows the processing of aerial data and ground data recognized and processed by the object recognition algorithm 310. It is a diagram conceptually illustrating matching techniques such as Image Couple, Non-rigid registration method, and SIFT according to the identification of data characteristics of the image matching algorithm 320 .
10 is a diagram for processing of aviation data and ground data. It is a diagram conceptually illustrating the encoder and decoder processing of the image matching algorithm 320 .
11 is a diagram illustrating a captured 3D cultural heritage restored through a three-dimensional restoration program 410 based on aerial data and ground data of the cultural heritage.
12 is a diagram illustrating an initial screen for entering (entrying) for virtual reality experience for 3D immersive content that can be displayed through the web viewer program 510 installed in the web server 500. Referring to FIG.
13 is a diagram illustrating a state in which optimal texture quality is calculated through an image projection technique as an example of data optimization processing performed in the cloud server 1000 .
14 is a diagram in which a result matched with one mesh is generated by a low-capacity cloud method, and a state in which a thing for each unit object is objectified according to each property is captured.
15 is a diagram illustrating a state in which a wider range of 3D content production is performed through the 3D editing tool 1100 .
FIG. 16 is a diagram schematically illustrating the configuration of a database construction system for a three-dimensional cultural property through data combination according to an embodiment of the present invention in a simplified block.

The present invention should be interpreted in a way that includes the scope of rights that reach the technical idea through various modified embodiments, only these embodiments make the publication of the present invention complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims.

Moreover, the drawings attached to the present invention are attached as a way to help the understanding of the description of the present invention to the last, and the drawings attached to the present invention do not represent the technical idea of the present invention, The idea should not be understood as a limited interpretation by the accompanying drawings.

As shown in FIG. 1, the method for constructing a database of three-dimensional cultural assets through data combination according to an embodiment according to the present invention includes a step 10 of acquiring image data of cultural assets using a drone and a ground camera, based on the acquired data. It can be performed as a process including the processing step 20 of the outline, the 3D image restoration step 30 using artificial intelligence, and the 3D DB utilization web service environment construction step 40 .

Image data acquisition step of cultural assets using drones and ground cameras (10)

In the step (10), in order to acquire high-quality three-dimensional data on cultural assets, a preliminary investigation (pre-investigation of an object for data acquisition) necessary for acquiring aerial photographing data of cultural assets from a drone, input of equipment required for photographing an object, and shooting Preliminary construction procedures such as method determination (drone model and shooting plan establishment) and main survey (reference point surveying and shooting using drones) can be performed. Preliminary construction procedures such as pre-investigation of the object for acquisition), equipment input required for photographing the object and determination of the shooting method (establishing a DLSR shooting plan), and main investigation (surveying and shooting a reference point using a ground camera) can be performed. .

An image camera 110 for aerial photographing of cultural properties may be mounted on the aircraft body of the drone 100 and installed. In this way, aerial image capturing of cultural assets in the air can be performed through the image camera 110 installed in the body of the drone 100 . Of course, on the ground, the ground camera 200 may be used to capture the ground image of the cultural property.

In particular, the object of cultural property and the photographed ground data photographed by the ground camera 200 are photographed at the same distance as in FIG. % or more overlapping methods can be used.

In addition, photographing an object of cultural property using the drone 100 can be photographed at different heights, for example, as shown in FIG. 3, focusing on the POI (Point of Interest) photographing technique required for 3D DB construction. High-quality point cloud acquisition is possible because it can be carried out in a high-overlapping method through movement.

Furthermore, when photographing an object of cultural property using the drone 100, for example, as shown in FIG. 4, the corner of the building can be photographed in a high-overlapping method so that three walls can be photographed at the same time, so that it is necessary for the construction of a three-dimensional DB. , in the case of indoor shooting, the opposite side of the wall is photographed, and bulky objects can be photographed according to the height.

In this way, in accordance with the need for 3D DB construction, the overlapping degree of photographing objects of cultural properties is set to 70% or more and filming is taken so that no closed areas are generated. Shooting can be performed while maintaining at least 80% overlap.

Of course, the above-mentioned aerial data and ground data are, for example, key points and image matching, bundle adjustment, and point clouds based on the raw photos acquired in the process of going through the steps to be described later. Cloud), a wire frame, a textured frame, and a 3D model (3D model) can be processed in order. In this way, the aerial data acquired from the drone 100 and the ground data acquired from the ground camera 200 may be merged as shown in FIG. 6 .

Processing step of the contour based on the acquired data (20)

In the step 20, a process of recognizing the outline of an object based on the aerial data acquired from the drone 100 and the ground data acquired from the ground camera 200 is performed.

That is, the accuracy verification and improvement of the aerial data and ground data of the cultural property selected as the test bed can be performed through the object recognition algorithm 310, and the object recognition algorithm 310 is the aerial data of the cultural property selected as the test bed. And based on the ground data, it is possible to perform test verification that recognizes the object of the image as an outline.

In particular, the object recognition algorithm 310 can process in a manner that recognizes only the outline that distinguishes the object, not all the lines for the aerial data and the ground data, which is shown in FIG. 7 . You can refer to Of course, the object recognition algorithm 310 may be processed in a manner that recognizes various and high-performance contour lines such as LSD, AFM, Wireframe, L-CNN, Ground Truth, and the like.

In the processing of such various contour recognition, various threshold and parameter techniques can be applied in an optimized state to the object recognition algorithm 310, and actual classification lines through the optimized techniques are, for example, LSD, AFM, It can be processed in ways such as Wireframe, L-CNN, Ground Truth, etc.

For example, in L-CNN, a convolution operation using a filter is repeatedly performed to detect image features, and the structure is, for example, a convolution layer (feature extraction; feature extraction) and a pooling layer (feature extraction). , it can be configured as a Fully-connected layer (classification). Of course, in such an L-CNN, processing methods such as filter (kernel), padding, stride, pooling, ReLu activation function, Drop-Out, Fully-Connected layer, etc. may be used.

In particular, the aerial data and ground data recognized and processed by the object recognition algorithm 310 can be processed and processed in a category-by-category segmental labeling method through the image segmentation algorithm 311, and are processed in this category-specific segmental labeling method. Figure 8 You can refer to The above image segmentation algorithm 311 is a segmental labeling method for each category that removes noise such as grass from an input photo, for example, as shown in the drawing, based on aerial data and ground data processed by high-performance contour recognition. can be processed with

In the process of processing the Segmetic Labels in this input photo, for example, 1 (Person), 2 (Purse), 3 (Plants/Grass), 4 (Sidewalk), 5 (Building/Structures), etc. For example, 1 (Person) is red, 2 (Purse) is purple, 3 (Plants/Grass) is green, 4 (Sidewalk) is gray, and 5 (Building/Structures) is yellow.

Accordingly, among the 1 (Person), 2 (Purse), 3 (Plants/Grass), 4 (Sidewalk), and 5 (Building/Structures) identified by color, for example, grass [3 (Plants/Grass)] Noise can be easily removed.

Moreover, verification of image matching for aerial data and ground data from which noise can be removed in this way can be made through the image matching algorithm 320, and the image matching algorithm 320 is test-verified aerial data and ground data. Image matching verification can be performed by comparing the performance of SIFT to which a mathematical algorithm is applied and an algorithm to which AI is applied based on the data.

Of course, the object recognition algorithm 310 and the image matching algorithm 320 can be installed and operated in the artificial intelligence server 300, and the image matching algorithm 320 identifies the characteristics of the data and each It can be processed by a matching method of an ensemble technique that combines the advantages of

Furthermore, in the image matching algorithm 320, convolution and pooling processing are performed in the encoder process for the data as shown in FIG. 10, for example, in the processing of the above techniques. In the process of being decoded through a compressed representation layer, convolution processing of the transpose function may be performed.

3D image restoration stage using artificial intelligence (30)

In the step 30, recognition is processed by the object recognition algorithm 310, and the 3D image is restored according to image processing based on the aerial data and ground data matched from the image matching algorithm 320. is a process in which

That is, the 3D image restoration according to the image processing of the aerial data and the ground data can be made through the 3D restoration program 410 linked in the artificial intelligence server 300, and the 3D restoration program 410 can perform a process of restoring the image to a three-dimensional 3D image based on the matching-processed aerial data and ground data.

The above-mentioned three-dimensional restoration program 410 is a combination of a 2D image restoration algorithm and a 3D algorithm based on GAN in a test through application of a 3d restoration algorithm to cultural assets. can be restored

3D DB utilization web service environment construction stage (40)

In the step 40, the structural environment design of the website suitable for displaying the 3D cultural property restored in the step 30 in the website is constructed. For example, the restored 3D cultural property is 3D compatible on the website. It can be produced in the form of viewer and VR and AR immersive content.

In other words, by using Webgl, a 3D viewer is inserted on the canvas created on the web and a low-capacity cloud is connected, and a free viewpoint can be provided through user input according to the creation of a virtual camera. It is possible to produce responsive, real-time synthesized AR content, and it can also be implemented in an Android environment using ARcore, which can be demonstrated through the web viewer program 510 installed in the web server 500 .

Of course, the 3D viewer and VR and AR contents may be produced in a design suitable for the structural environment design of the website, and may be registered and posted in a web publishing format using CSS and HTML.

Accordingly, the 3D viewer and VR and AR immersive contents for a specific cultural property may be implemented as shown in FIG. 12 on the website, and a virtual reality experience created through entry (entry) may be possible. Moreover, the above-mentioned website can be provided to users through the establishment and verification of the environment of the web server 500 that meets the requirement for the number of concurrent users.

Such 3D viewer and VR and AR immersive content can be made through the compatibility 3D viewer production step 35 between the steps 30 and 40, and of course, the low-capacity cloud can be created in the step (10). ) and the low-capacity cloud generation step 15 that can be performed between the steps 20, in some cases, can also be made in the compatibility 3D viewer production step 35. Of course, the low-capacity cloud processing in the low-capacity cloud generation step 15 and the production of 3D viewers and VR and AR immersive content can be referred to through the detailed description to be described later.

The low-capacity cloud generation step 15 is a process of optimizing the data by arranging errors or unnecessary data generated from the 3D scan data by projecting the aerial data and ground data acquired in the step 10 into a three-dimensional space. .

Of course, the processing contents involved in the above data optimization processing include image distortion removal, image matching and three-dimensional camera coordinate derivation, camera coordinate data export, camera coordinate data import and image connection, axis alignment and operation view creation, Mesh generation, UV generation, mapping, texture generation and compression, and more.

Of course, such data optimization processing may be performed in the cloud server 1000, and for example, an optimal texture quality may be calculated through an image projection technique as shown in FIG. 13 .

That is, in order to improve the texture stretching or overlapping phenomenon that occurs when a low-capacity cloud mesh is created, one or more original images can be selected and projected according to the user's selection to improve the degraded portion.

In addition, the result matched with one mesh may be generated as a low-capacity cloud, for example, as shown in FIG. 14 , and objects may be objectified and stored for each unit object according to each property.

The compatibility 3D viewer production step 35 is a process of making the 3D cultural heritage restored in the step 30 compatible with a 3D viewer and VR and AR immersive content format suitable for displaying on a website. , these 3D viewers and VR and AR sensory content can be produced through the 3D editing tool 1100 installed in the cloud server 1000 .

That is, the 3D editing tool 1100 inserts a 3D viewer on the canvas created on the website and connects the low-capacity cloud while providing a free viewpoint through the user's input according to the creation of the virtual camera, It is possible to create AR content that is synthesized in real time that can react with the surrounding environment using the low-capacity cloud.

Of course, the 3D viewer and VR and AR contents can be produced in a design suitable for the structural environment design of the website, and can be registered and posted in a web publishing format using CSS and HTML.

The 3D editing tool 1100 can also produce content that is easy to customize, and can upload a separate 3D model desired by the user in addition to the low-capacity cloud, so that a wider range of 3D content can be produced, for example, as shown in FIG. 15 , It may be linked with the platform of the cloud server 1000 .

On the other hand, the database construction system for three-dimensional cultural heritage through data combination according to an embodiment of the present invention is a drone 100 equipped with an image camera 110 for photographing an object of cultural heritage in the air as shown in FIG. 16 and A ground camera 200 for shooting on the ground, an artificial intelligence server 300 that performs object recognition processing of images for aerial data and ground data acquired from drones and ground cameras, and the artificial intelligence server and interworking with the above data A three-dimensional restoration program 410 that processes images of (aviation data and ground data) to restore three-dimensional 3D images, and a three-dimensional immersive content format that can be displayed on the website and is compatible with the three-dimensionally restored 3D cultural assets It may be configured to include a web server 500 that provides

In particular, the artificial intelligence server 300 is an object recognition algorithm 310 necessary for object recognition processing of images for aerial data and ground data acquired from a ground camera, and by category for object recognition-processed aerial data and ground data. The image segmentation algorithm 311 necessary for the processing of the segmental labeling process, and the image matching algorithm 320 necessary for the image matching verification process for the aerial data and the ground data subjected to the segmental labeling may be configured to further include.

In addition, the three-dimensional restoration program 410 may process images of data (aviation data and terrestrial data) to be restored into three-dimensional 3D images in a manner linked to the artificial intelligence server 300 .

The cloud server 1000 may be configured to further include a 3D viewer capable of displaying the restored 3D cultural heritage while being compatible with the website, and a 3D editing tool 1100 capable of producing VR and AR immersive content.

As described above, by merging the point cloud obtained from the aerial data and the ground data together with the combined processing of the aerial data on the cultural objects photographed from the drone and the ground data photographed from the ground camera, the result is calculated as a single 3D model This is possible, and as it provides a construction method for this 3D model to be displayed on the website, it not only has a terabyte-level image processing DB through efficient memory usage of the system and distributed processing technology, but also high-quality three-dimensional cultural property spatial information. Construction can also be implemented.

Drone (100) Video Camera (110)
Ground Camera (200) Artificial Intelligence Server (300)
Object Recognition Algorithm (310) Image Segmentation Algorithm (311)
Image Matching Algorithm (320) Cloud Server (1000)
3D editing tool (1100) 3D restoration program (410)
Web Server (500) Web Viewer Program (510)

Claims (7)

Image data acquisition of cultural assets using a drone and ground camera from which aerial data that is an aerial image image of a cultural property object photographed from the drone 100 and ground data that is a ground image image of a cultural property object photographed from the ground camera 200 are obtained step (10);
Contour processing step ( 20);
In the step (20), the image object is recognized as an outline and based on the processed aerial data and ground data using artificial intelligence to restore the 3D image through the 3D restoration program 410 that is linked in the artificial intelligence server 300. 3D image restoration step (30); and
3D DB utilization web service built and provided through the web viewer program 510 installed in the web server 500 in a compatible 3D immersive content format in which the 3D cultural heritage restored in step 30 can be displayed on the website environment construction step (40);
A method of constructing a database of three-dimensional cultural properties through data combination, characterized in that it consists of a configuration comprising a. According to claim 1,
The terrestrial data photographed from the terrestrial camera 200 in the step (10) is carried out in such a way that it is photographed at the same distance, a gradual photographing distance is secured, and while photographing at 360° in different directions, the photographing is repeated at least 70% or more Meanwhile,
Aerial data photographed from the drone 100 in the step (10) is a database of three-dimensional cultural properties through data combination, characterized in that it is photographed while maintaining at least 80% of vertical and horizontal redundancy using a cross-pass photographing technique construction method. 3. The method of claim 2,
Aviation data and ground data recognized and processed by the object recognition algorithm 310 in the step 20 are processed and processed in a segmental labeling method for each category through the image segmentation algorithm 311,
A method of constructing a database of three-dimensional cultural properties through data combination, characterized in that the verification of image matching for aerial data and ground data processed by the segmental labeling method for each category is made through an image matching algorithm (320). According to claim 1,
Between the above steps (10) and (20), the obtained aerial data and ground data are projected into a three-dimensional space, and errors or unnecessary data generated from the 3D scan data are optimized in a way that the cloud server 1000 organizes them. A method of constructing a database of three-dimensional cultural properties through data combination, characterized in that the low-capacity cloud generation step (15) for generating a low-capacity cloud is further performed. According to claim 1,
Between the steps 30 and 40, the 3D cultural heritage restored in the step 30 is suitable for displaying through the 3D editing tool 1100 installed in the cloud server 1000 so that it can be compatible on the website. A method of constructing a database of three-dimensional cultural properties through data combination, characterized in that the 3D viewer and the compatible 3D viewer production step (35) for producing in the form of VR and AR immersive content is further performed. 6. The method of claim 5,
The 3D editing tool 1100 inserts a 3D viewer on the canvas created on the website and connects the low-capacity cloud while providing a free viewpoint through the user's input according to the creation of a virtual camera while utilizing the low-capacity cloud. A method of constructing a database of three-dimensional cultural properties through data combination, characterized in that it produces three-dimensional immersive content that is synthesized in real time that can react with the surrounding environment. a drone 100 equipped with an image camera 110 for photographing objects of cultural properties in the air;
a ground camera 200 for photographing objects of cultural properties on the ground;
An artificial intelligence server 300 that performs object recognition processing of images for aerial data and ground data acquired from drones and ground cameras;
A three-dimensional restoration program 410 that is interlocked with an artificial intelligence server and processes the images of the data (aviation data and ground data) to be restored into a three-dimensional 3D image; and
a web server 500 that provides 3D reconstructed 3D cultural assets in a form of 3D immersive content that can be displayed on a website and is compatible with;
A database construction system of three-dimensional cultural properties through data combination, characterized in that it consists of a configuration comprising a.

Images