KR102578580B1 - Apparatus and method for generating 3d data - Google Patents

Abstract

This disclosure relates to an apparatus and method for generating 3D data. A method according to an embodiment of the present disclosure includes receiving data from the outside, generating first point cloud data based on the received data, detecting an occluded area based on the first point cloud data, and detecting an occluded area based on the first point cloud data. Generating second point cloud data based on the occluded area, generating three-dimensional mesh and texture information based on the first and second point cloud data, and generating object unit information based on the three-dimensional mesh and texture information. It includes a generating step and a step of generating visualization data based on object unit information. According to the present disclosure, data in a partial area corresponding to the occluded area can be used as input data, and can be reprocessed on an object basis to generate lightweight and visualized 3D data.

Description

Apparatus and method for generating 3D data {APPARATUS AND METHOD FOR GENERATING 3D DATA}

The present disclosure relates to a 3D data generation device and method, and more specifically, to a 3D data generation device and method that follows a pipeline from data acquisition to 3D data generation suitable for virtual space services.

Recently, interest in 3D restoration data is increasing due to the development of 3D space restoration technology and the revitalization of the AR/VR market. In particular, the size of the AR/VR market is growing rapidly, and the demand for 3D restoration data, which is essential for it, is increasing.

In order to generate 3D data, an image acquisition process, a process of calculating 3D information from the acquired image and creating a 3D space, and a visualization process to utilize and service the 3D space are required. A typical 3D data restoration tool takes an image as input and provides the ability to restore the entire space.

In fact, in order for users to provide services, a method is needed to create and modify 3D data to suit the service scenario. However, the existing technology has a problem in that it cannot restore only some areas according to the user's needs or cannot utilize only some objects among the restored results.

In addition, general 3D data restoration tools have a problem in that they do not consider both the quality and weight reduction of restoration results depending on the type of service scenario. In order to solve this problem, the user manually edits the 3D data, which takes a lot of time and effort, resulting in an increase in the service production cost. Furthermore, with general 3D data restoration tools, even if there is an occluded area in the restored data, only some areas cannot be restored, so the entire area must be restored again.

The purpose of the present disclosure is to provide a 3D data generation device and method that follows a pipeline from data acquisition to 3D data generation suitable for virtual space services in order to restore and utilize 3D space.

A 3D data generation method according to an embodiment of the present disclosure includes receiving at least one of image data and video data from an external source, calculating 3D spatial information based on the received data, and generating first point cloud data. Detecting an occluded area based on the first point cloud data, receiving data of a partial area corresponding to the occluded area based on the detected occluded area, and generating second point cloud data, Generating 3D mesh and texture information based on the first point cloud data and the second point cloud data, classifying into object units based on the 3D mesh and texture information and generating reprocessed object unit information. and generating visualization data based on the object unit information.

According to the present disclosure, 3D data can be generated by rephotographing only the image or video corresponding to the occluded area of the restoration result. In addition, 3D data can be generated on an object basis using object classification and reconstruction functions that are not provided by general 3D data restoration tools.

Furthermore, according to the present disclosure, data can be lightened and visualized to suit the service scenario, and thus can be used for various services.

FIG. 1 is a block diagram schematically showing a 3D data generating device 100 according to an embodiment of the present disclosure.
FIG. 2 is a block diagram showing the configuration of the 3D reconstruction unit 120 shown in FIG. 1 in more detail.
FIG. 3 is a block diagram showing the configuration of the reconstruction unit 130 shown in FIG. 1 in more detail.
FIG. 4 is a block diagram showing the configuration of the service work unit 140 shown in FIG. 1 in more detail.
Figure 5 is a diagram showing an occluded area according to an embodiment of the present disclosure.
Figure 6 is a diagram showing object classification results according to an embodiment of the present disclosure.
Figure 7 is a flowchart showing a method for generating 3D data according to an embodiment of the present disclosure.

Below, embodiments of the present disclosure will be described clearly and in detail so that a person skilled in the art can easily practice the present disclosure.

The components described with reference to terms such as unit, module, block, ~or, ~er, etc. used in the detailed description and the functional blocks shown in the drawings are It may be implemented in the form of software, hardware, or a combination thereof. By way of example, software may be machine code, firmware, embedded code, and application software. For example, the hardware may include an electrical circuit, an electronic circuit, a processor, a computer, an integrated circuit, integrated circuit cores, a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), a passive component, or a combination thereof. .

FIG. 1 is a block diagram schematically showing a 3D data generating device 100 according to an embodiment of the present disclosure. The 3D data generating device 100 may include a data acquisition unit 110, a 3D restoration unit 120, a reconstruction unit 130, a servitization unit 140, and a storage unit 150.

The 3D data generating device 100 can generate and store 3D data required for online services based on input data ( _DIN ). For example, an online service may involve providing users with content that requires 3D spatial information, such as movies, TV commercials, games, or AR/VR. For example, the 3D data generating device 100 may generate and store 3D data based on object unit information desired by the user. Furthermore, when the 3D data generation device 100 detects an area that has not been restored to 3D space, that is, an occluded area, during the 3D data generation process, the 3D data is reacquired by reacquiring the input data ( _DIN ) corresponding to the occluded area. can be created.

The data acquisition unit 110 may generate processed data (PD) necessary for generating 3D data based on the input data (D _IN ). For example, the input data D _IN may include at least one of image data and video data captured by an aircraft, satellite, or drone. Although not shown, the 3D data generating device 100 may further include a transmission/reception module (not shown) or a network adapter (not shown). For example, a transmit/receive module (not shown) may include an amplifier, mixer, bandwidth filter, and transmit/receive antenna. The transmitting and receiving module (not shown) can physically transmit and receive data through wireless communication with an aircraft, satellite, or drone. For example, a network adapter (not shown) may communicate with an external network to receive image data and video data captured by an aircraft, satellite, or drone. For example, the data acquisition unit 110 may receive input data D _IN from a transmission/reception module (not shown) or a network adapter (not shown).

Furthermore, when the data acquisition unit 110 receives an occlusion region feedback signal (ORS), the data acquisition unit 110 uses aircraft, satellites, drones, etc. to acquire new input data ( _DIN ) corresponding to the occlusion region. A re-shoot request signal (RRS) requesting re-shooting may be transmitted using a transmission/reception module (not shown) or a network adapter (not shown).

Additionally, the data acquisition unit 110 may generate processed data (PD) through a preprocessing process based on the input data (D _IN ), if necessary. For example, the preprocessing process may include at least one of size adjustment, format conversion, fusion, and image quality correction of image or video data. As an example, when receiving two or more input data D _IN , the data acquisition unit 110 may generate processed data PD through a preprocessing process of fusing each input data. For example, the processed data (PD) may be the input data (D _IN ) itself or data that has undergone a preprocessing process based on the input data (D _IN ).

The 3D restoration unit 120 may generate 3D mesh and texture information (MTI) based on the processed data (PD). The 3D restoration unit 120 may detect an area that has not been completely restored as a result of restoration, that is, an occluded area. Accordingly, when an occluded area is detected, the 3D reconstruction unit 120 may transmit an occluded area feedback signal (ORS) requesting new input data (D _IN ) to the data acquisition unit 110. For example, the occluded area feedback signal (ORS) includes information about the occluded area (e.g., coordinate value, direction value, and phase value, etc.), required movement value information of the rephotograph tool, and required illuminance value information. It can be included. For example, information about the occluded area may be information corresponding to the entire area including the occluded area or information corresponding to the occluded area. Furthermore, the 3D restoration unit 120 can fuse the rephotographed data through feedback with the existing restored data stored in an external memory (not shown). The specific configuration and operation of the 3D reconstruction unit 120 will be explained with reference to FIG. 2.

The reconstruction unit 130 may generate object information (OBJI) based on 3D mesh and texture information (MTI). For example, the reconstruction unit 130 may classify 3D mesh and texture information (MTI) into object units (e.g., cars, buildings, trees, etc.) and reconstruct the classified objects according to the user's request. can do. The specific configuration and operation of the reconfiguration unit 130 will be explained with reference to FIG. 3.

The service work unit 140 may generate visualized data (VD) based on object unit information (OBJI). The service work unit 140 can be visualized as three-dimensional data with the quality and capacity desired by the user. For example, in the case of a building object, the service work unit 140 can define various characteristics that can represent the building and visualize the building only with the defined characteristics. For example, the service processing unit 140 may process (for example, make it lightweight) 3D data according to the user's request or the specifications of the service device. If lightweighting of the 3D data is not required, the service work unit 140 may not lighten the 3D data. The specific configuration and operation of the servitization work unit 140 will be explained with reference to FIG. 4.

The storage unit 150 may be configured to store visualization data (VD). For example, the storage unit 150 may be composed of random access memory (RAM), a general storage device, or flash memory. However, the storage unit 150 is not limited to this configuration and may be composed of any memory for storage, and may be part of an external memory (not shown).

FIG. 2 is a block diagram showing the configuration of the 3D reconstruction unit 120 shown in FIG. 1 in more detail. The 3D reconstruction unit 120 may include a spatial information calculator 121, a 3D spatial restorer 122, and a mesh/texture generator 123.

The spatial information calculator 121 may generate 3D spatial information (SI) based on the processed data (PD). For example, the spatial information calculator 121 may calculate 3D spatial information (SI) to restore a 3D space using 2D processed data (PD). For example, the spatial information calculator 121 may extract 3D spatial features based on one or more processed data (PD) and calculate 3D spatial information (SI) using a transformation function.

The 3D space restorer 122 may generate point cloud data (PCD) based on 3D spatial information (SI). Each point in the point cloud may have 3D spatial coordinates, color values, and normal values. For example, point cloud data (PCD) may include a region that is not restored by 3D spatial information (SI), that is, an occlusion region. The 3D space restorer 122 can detect the occluded area. When an occluded area is detected, the spatial restorer 122 may transmit an occluded area feedback signal (ORS) to the data acquisition unit 110. This is to enable the 3D restoration unit 120 to 3D restore the processed data (PD) corresponding to the occluded area.

Furthermore, the 3D space restorer 122 may temporarily store 3D space information (SI) in an external memory (not shown). For example, the 3D space restorer 122 combines the 3D spatial information (SI) newly generated by the occlusion region feedback signal (ORS) with the existing 3D spatial information (SI) stored in an external memory (not shown). can be fused with As a result, if the occlusion area is not detected, the 3D space restorer 122 may generate point cloud data (PCD) using fused or unfused 3D spatial information (SI).

The mesh/texture generator 123 may generate 3D mesh and texture information (MTI) based on point cloud data (PCD). For example, the mesh/texture generator 123 may remove points with noise information from the point cloud. Furthermore, the mesh/texture generator 123 can generate more intuitive 3D mesh and texture information (MTI) through operations such as flattening.

FIG. 3 is a block diagram showing the configuration of the reconstruction unit 130 shown in FIG. 1 in more detail. The reconstruction unit 130 may include an object classifier 131 and an object processor 132.

The object classifier 131 may generate classified objects (O _SEG ) based on 3D mesh and texture information (MTI). For example, the object classifier 131 may perform a segmentation process that separates 3D mesh and texture information (MTI) generated in one continuous form into individual object units. For example, segmented objects (O _SEG ) may include objects such as buildings, roads, trees, or cars that are classified based on the object's appearance.

The object processor 132 may generate object unit information (OBJI) based on the classified objects (O _SEG ). For example, the object processor 132 may create object unit information (OBJI) by using the classified objects (O _SEG ) to create new or reorganized objects in a form desired by the user.

FIG. 4 is a block diagram showing the configuration of the service work unit 140 shown in FIG. 1 in more detail. The service work unit 140 may include a lightweight work unit 141, a visualization work unit 142, and a data buffer 143.

The lightweight work unit 141 may generate lightweight data based on object unit information (OBJI). For example, when lightweighting of data is required in a service scenario, the lightweighting task unit 141 performs data lightweighting using the trade-off relationship between data capacity and quality, and provides data lightweighting for specific object data according to the user's request. Lightweight data can be created through a lightweighting operation that includes at least one of selective lightweighting and data lightweighting using compression technology such as removal of background areas excluding objects. The lightweight work unit 141 may temporarily store the lightweight data in the data buffer 143 before the lightweight data is visualized by the visualization work unit 142, which will be described later.

The visualization task unit 142 may generate visualization data (VD) based on object unit information (OBJI) or lightweight data. For example, when lightweighting of data is not required in a service scenario, the visualization unit 142 may directly receive object unit information (OBJI) and generate visualization data (VD). On the other hand, when lightweight data is required in a service scenario, the visualization task unit 142 may generate visualization data (VD) based on the lightweight data.

Furthermore, the visualization task unit 141 may check whether the visualization data (VD) has been created with data of an appropriate capacity for the service scenario. If the capacity is inadequate, additional lightweighting work is required, so additional lightweighting work can be requested from the lightweighting work unit 141. The lightweight work unit 141, which has received a request for lightweight work, may additionally reduce the lightweight data stored in the data buffer 143 and transmit it to the visualization work unit 142. The service work unit 140 may repeat lightweighting and visualization tasks until visualization data (VD) is created with an appropriate amount of data for the service scenario.

Figure 5 is a diagram showing an occluded area according to an embodiment of the present disclosure. Hereinafter, description will be made with reference to FIGS. 1 and 2. In the 3D reconstruction unit 120, the 3D space restorer 122 may detect the occluded area through 3D spatial information (SI). As shown in FIG. 5, 3D spatial information (SI) may include a portion that is obscured by another object and cannot be restored, that is, an occluded area. When an occluded area is detected, the 3D space restorer 122 may request new input data (D _IN ) corresponding to the occluded area by transmitting an occluded area feedback signal (ORS) to the data acquisition unit 110.

Figure 6 is a diagram showing object classification results according to an embodiment of the present disclosure. Hereinafter, description will be made with reference to FIGS. 1 and 3. As shown in FIG. 6, the object classifier 131 of the reconstruction unit 130 can separate the 3D mesh and texture information (MTI) generated in one continuous form into object units. Classified objects O _SEG shown in FIG. 6 may be modified or reorganized through the object processor 132.

Figure 7 is a flowchart showing a method for generating 3D data according to an embodiment of the present disclosure. Hereinafter, description will be made with reference to FIGS. 1 to 3.

In step S110, the data acquisition unit 110 may receive at least one of image data and video data captured by an aircraft, satellite, or drone. When receiving the occluded area feedback signal (ORS), the data acquisition unit 110 may request re-photography of some areas corresponding to the occluded area from an aircraft, satellite, or drone, and the re-photographed image data and At least one of the video data can be input.

In step S120, the data acquisition unit 110 may acquire input data (D _IN ) to generate processed data (PD), and when two or more input data (D _IN ) are acquired, input data (D _IN) ) can be fused to generate processed data (PD).

In step S130, the spatial information calculator 121 may calculate 3D spatial information (SI) of the input processed data (PD) and transmit it to the 3D spatial restorer 122. The 3D space restorer 122 can restore data in the form of a point cloud using 3D spatial information (SI) and store it in an external memory (not shown).

In step S140, the 3D space restorer 122 may detect an occluded area in data restored to 3D space. If an occluded area is detected (Yes), the procedure moves to step S110. In this case, the 3D space restorer 122 may transmit the occluded area feedback signal (ORS) to the data acquisition unit 110. The data acquisition unit 110 that receives the occluded area feedback signal (ORS) may receive at least one of image data and video data for a partial area corresponding to the occluded area. On the other hand, if the occlusion area is not detected (No), the procedure moves to step S150.

In step S150, when the occluded area feedback signal (ORS) is transmitted at least once, the 3D space restorer 122 stores the point cloud data (PCD) and the occluded area feedback signal (ORS) stored in an external memory (not shown). ), the newly created point cloud data (PCD) can be fused. The mesh/texture generator 123 can generate 3D mesh and texture information (MTI) using point cloud data (PCD).

In step S160, the object classifier 131 may separate one continuous 3D mesh and texture information (MTI) into object units. The object processor 132 may create a new object or reorganize it using the classified objects (O _SEG ).

In step S170, the object processor 132 may check object classification suitability, boundary estimation suitability between objects, etc. If the value quantifying the suitability does not reach the threshold arbitrarily set at the user's request, the procedure moves to S160 to create or reorganize a new object, and if it exceeds the threshold, it moves to S180.

In step S180, the service work unit 140 may lighten and/or visualize the 3D data to suit the service scenario.

In step S190, the servitization task unit 140 may check whether 3D data of a capacity suitable for the service scenario has been generated. If the capacity is suitable (Yes), the storage unit 150 stores the 3D visualization data (VD) and the entire procedure is completed. If the capacity is inadequate (No), the procedure moves to S180.

The 3D data generating device and method according to embodiments of the present disclosure have been described above. According to the present disclosure, the entire pipeline from data acquisition to generation of 3D data suitable for virtual space services is performed. If an occluded area occurs due to insufficient input, 3D data can be generated by rephotographing only a portion of the data, and 3D data can be utilized on an object basis by using object classification and reconstruction functions that are not provided by general 3D data restoration tools. . Furthermore, 3D data can be lightweight and visualized to suit the service scenario.

The above-described contents are specific embodiments for carrying out the present disclosure. The present disclosure will include not only the above-described embodiments, but also embodiments that are simply designed or can be easily changed. In addition, the present disclosure will also include techniques that can be easily modified and implemented using the embodiments. Therefore, the scope of the present disclosure should not be limited to the above-described embodiments, but should be determined by the claims and equivalents of this disclosure as well as the claims described later.

100: 3D data generation device 110: Data acquisition unit
120: 3D restoration unit 121: Spatial information calculator
122: 3D space restorer 123: Mesh/texture generator
130: Reconstruction unit 131: Object classifier
132: Object processor 140: Servitization work unit
141: Lightweight work department 142: Visualization work department
143: data buffer 150: storage unit

Claims (10)

In the 3D data generation method,
Receiving at least one of image data and video data from an external source;
generating first point cloud data by calculating 3D spatial information based on the received data;
detecting an occluded area based on the first point cloud data;
generating second point cloud data by receiving data of a partial area corresponding to the occluded area based on the detected occluded area;
Generating 3D mesh and texture information based on the first point cloud data and the second point cloud data;
Classifying into object units based on the 3D mesh and texture information and generating reprocessed object unit information; and
Including generating visualization data based on the object unit information,
A method including the step of determining whether the classification of the object unit is appropriate after the step of generating the reprocessed object unit information. According to claim 1,
Generating the first and second point cloud data includes performing at least one of size adjustment, format conversion, fusion, and image quality correction of each of the received data and the received occluded area data. . According to claim 1,
Detecting the occluded area includes requesting re-photographing of the occluded area by transmitting an occluded area feedback signal to obtain new data for the occluded area of the received data. According to claim 3,
The method wherein the occluded area feedback signal includes information on a coordinate value regarding the occluded area, a required movement value of the rephotograph tool, and a required illuminance value of the rephotograph tool. According to claim 3,
The method of generating the second point cloud data includes fusing 3D spatial information generated by the occlusion area feedback signal with the calculated 3D spatial information. According to claim 1,
Each point of the first and second point cloud data includes at least one of three-dimensional space coordinate values, color values, normal values, and noise information. According to claim 6,
The method of generating the 3D mesh and texture information includes removing points having noise information from the first and second point cloud data. delete According to claim 1,
If the method further includes the step of lightening data generated based on the object unit information before generating the visualization data, the visualization data is generated based on the lightened data. According to clause 9,
The method further includes determining a data capacity after generating the visualization data, and reducing the weight of the visualization data according to the determination result.