KR102536096B1 - Learning data generation method and computing device therefor - Google Patents

Abstract

A computing device according to one technical aspect of the present application includes a memory for storing one or more instructions and a processor for executing the one or more instructions stored in the memory, wherein the processor executes the one or more instructions, thereby performing basic RGB An image and a basic depth map image corresponding thereto are provided, and setting information of a basic spherical virtual image generated by performing spherical transformation based on the basic RGB image and the basic depth map image is changed to obtain a plurality of learning RGB images and a plurality of learning RGB images. A depth map image can be created. The plurality of training RGB images and the plurality of training depth map images may be matched 1:1, respectively.

Description

Learning data generation method and computing device therefor

This application relates to a method for generating learning data and a computing device therefor.

In recent years, virtual space realization technology has been developed that allows users to experience as if they are in a real space without directly visiting the real space by being provided with an online virtual space corresponding to the real space.

In order to implement such a virtual space, it is necessary to acquire a plane image photographed for a real space to be implemented, and create a three-dimensional virtual image based on this to provide the virtual space.

In the case of such a prior art, a virtual image is provided based on a flat image, but there is a limit in that a sense of reality and three-dimensional information are lacking because distance information is not known in a conventional virtual space.

US Patent Application No. 16/880143

One technical aspect of the present application is to solve the above problems of the prior art, and, according to an embodiment disclosed in the present application, aims to provide distance information with respect to a virtual space.

According to an embodiment disclosed in the present application, an object is to generate various learning data sets using one RGB image and a distance map image for the RGB image.

According to an embodiment disclosed in the present application, an object is to generate a depth map image from an RBG image based on learning using a neural network model.

The tasks of the present application are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

One technical aspect of the present application proposes a computing device. The computing device includes a memory for storing one or more instructions and a processor for executing the one or more instructions stored in the memory, wherein the processor executes the one or more instructions to perform a basic RGB image and a basic depth corresponding thereto. A map image is provided, and a plurality of training RGB images and a plurality of learning depth map images may be generated by changing setting information of a basic spherical virtual image generated by spherical transformation based on the basic RGB image and the basic depth map image. there is. The plurality of training RGB images and the plurality of training depth map images may be matched 1:1, respectively.

Another technical aspect of the present application proposes a method for generating learning data. The learning data generation method is a learning data generation method performed in a computing device, comprising: receiving a basic RGB image and a basic depth map image corresponding to the basic RGB image by executing the one or more instructions; and generating a plurality of training RGB images and a plurality of training depth map images by changing setting information of a basic spherical virtual image generated by spherical conversion based on the map image. The plurality of training RGB images and the plurality of training depth map images may be matched 1:1, respectively.

Another technical aspect of the present application proposes a storage medium. The storage medium is a storage medium storing computer readable instructions. The instructions, when executed by a computing device, cause the computing device to receive a basic RGB image and a basic depth map image corresponding thereto, and perform spherical transformation based on the basic RGB image and the basic depth map image to generate the result. and generating a plurality of training RGB images and a plurality of training depth map images by changing setting information of the basic spherical virtual image. The plurality of training RGB images and the plurality of training depth map images may be matched 1:1, respectively.

The means for solving the problems described above do not enumerate all the features of the present application. Various means for solving the problems of the present application will be understood in more detail with reference to specific embodiments of the detailed description below.

According to the present application, there are one or more of the following effects.

According to an embodiment disclosed in the present application, there is an effect of providing distance information with respect to a virtual space.

According to an embodiment disclosed in the present application, there is an effect of generating a plurality of training data sets using one RGB image and a distance map image thereof.

According to an embodiment disclosed in the present application, a depth map image may be generated from an RBG image based on learning using a neural network model, and a virtual space including depth information may be provided using only the RGB image. There are possible effects.

According to an embodiment disclosed in the present application, in calculating a loss used for learning a neural network, by using a plurality of functions in combination, the loss range can be reduced to a minimum. .

The effects of the present application are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is an exemplary diagram for explaining a system for providing a spherical virtual image based on a depth map image according to an embodiment disclosed in the present application.
2 is a block diagram illustrating a computing device according to an exemplary embodiment disclosed in the present application.
3 is a diagram illustrating an architecture for generating learning data according to an embodiment disclosed in the present application.
4 is a diagram illustrating an equirectangular projection image and a spherical virtual image generated using the equirectangular projection image according to an embodiment disclosed in the present application.
5 is a diagram explaining a method of generating learning data according to an embodiment disclosed in the present application.
6 is a diagram for explaining an example of generating a large amount of training data sets based on basic RGB and basic depth maps according to an example.
7 is a diagram for explaining an example of a neural network architecture according to an embodiment disclosed in the present application.
8 is a diagram for explaining another example of a neural network architecture according to an embodiment disclosed in the present application.
9 is a diagram explaining a training method using a neural network according to an embodiment disclosed in the present application.
10 is a diagram illustrating a difference between an equirectangular projection image according to an embodiment disclosed in the present application and a spherical virtual image generated using the same.
11 is a diagram explaining a training method using a training module according to an embodiment disclosed in the present application.
12 is a diagram explaining a loss calculation method according to an embodiment disclosed in the present application.
13 is a diagram illustrating learning RGB, a learning depth map, an estimated depth map, and a difference image between the estimated depth map and the learning depth map according to an example.
14 is a diagram illustrating an architecture for generating a spherical virtual image according to an embodiment disclosed in the present application.
15 is a diagram explaining a method of providing a spherical virtual image to a user according to an embodiment disclosed in the present application.
16 is a diagram for explaining spherical transformation according to an embodiment disclosed in the present application.

Hereinafter, preferred embodiments of the present application will be described with reference to the accompanying drawings.

However, the embodiments of the present application may be modified in many different forms, and the scope of the present application is not limited to the embodiments described below. In addition, the embodiments of the present application are provided to more completely explain the present application to those skilled in the art.

Various embodiments of the present application and terms used therein are not intended to limit the technical features described in the present application to specific embodiments, and should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this application, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "A, Each of the phrases such as "at least one of B or C" may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may simply be used to distinguish that component from other corresponding components, and may refer to that component in other aspects (e.g., importance or order). A (e.g., first) component is "coupled" to another (e.g., second) component, with or without the terms "functionally" or "communicatively". When referred to as "connected" or "connected", it means that the certain component can be connected to the other component directly or through a third component.

The term "module" used in this application means a unit that processes at least one function or operation, which may be implemented as software or a combination of hardware and software.

Various embodiments of the present application are software (eg, user terminal 100 or computing device 300) including one or more instructions stored in a storage medium readable by a machine (eg, the user terminal 100 or the computing device 300). For example, it can be implemented as a program). For example, the processor 301 may call at least one command among one or more commands stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term means that data is stored semi-permanently in the storage medium. It does not distinguish between the case and the case of temporary storage.

According to embodiments, methods according to various embodiments disclosed in this application may be included in and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CDROM)), or through an application store (eg Play Store™) or on two user devices. It can be distributed (eg, downloaded or uploaded) online, directly between (eg, smart phones). In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the components described above may include singular or plural entities. According to various embodiments, one or more components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by modules, programs, or other components are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

Although various flow charts have been disclosed to describe the embodiments of the present application, this is for convenience of explanation of each step, and each step is not necessarily performed in the order of the flowchart. That is, each step in the flowchart may be performed simultaneously with each other, in an order according to the flowchart, or in an order reverse to that in the flowchart.

1 is an exemplary diagram for explaining a system for providing a spherical virtual image based on a depth map image according to an embodiment disclosed in the present application.

The system 10 for providing a spherical virtual image based on a depth map image may include a user terminal 100 , an image acquisition device 200 and a computing device 300 .

The user terminal 100 is an electronic device that can be used by a user to access the computing device 300, for example, a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, and a personal digital assistant (PDA). digital assistants), PMP (portable multimedia player), navigation, personal computer (PC), tablet PC (tablet PC), ultrabook, wearable device (e.g., smartwatch), glasses It covers smart glass, HMD (head mounted display), etc. However, other than that, the user terminal 100 may include electronic devices used for virtual reality (VR) and augmented reality (AR).

The image acquisition device 200 is a device for generating an RGB image and/or a depth map image, which is used to generate a spherical virtual image.

In the illustrated example, the image acquisition device 200 is shown divided into a distance measurement device 210 and an imaging device 220, but this is exemplary, and distance measurement and Imaging may also be performed.

The imaging device 220 is a portable electronic device having a photographing function, and generates an RGB image expressed in color with respect to a subject area - that is, an area captured in an RGB image.

That is, in the specification of the present application, an RGB image encompasses all images expressed in color, and is not limited to a specific expression method. Therefore, images expressed in CMYK (Cyan Magenta Yellow Key) are also collectively referred to as RGB images in the present application.

The imaging device 220 may be, for example, a mobile phone, a smart phone, a laptop computer, a personal digital assistant (PDA), a tablet PC, an ultrabook, or a wearable device. (wearable device, for example, a glass type terminal (smart glass)) and the like.

The distance measuring device 210 is a device capable of generating a depth map image by generating depth information about a subject area.

In the present application specification, a depth map image encompasses an image including depth information with respect to a subject space. That is, the depth map image refers to an image represented by distance information from an imaging point to each point with respect to each point in the photographed subject space.

The distance measurement device 210 may include a predetermined sensor for distance measurement, for example, a LIDAR sensor, an infrared sensor, an ultrasonic sensor, and the like. Alternatively, the distance measurement imaging device 220 may include a stereo camera, a stereoscopic camera, a 3D depth camera, and the like that can measure distance information by replacing the sensor. there is.

An image generated by the imaging device 220 is referred to as a basic RGB image, and an image generated by the distance measuring device 210 is referred to as a basic depth map image. Since the basic RGB image generated by the imaging device 220 and the basic depth map image generated by the distance measuring device 210 are generated for the same subject area under the same conditions (eg, resolution, etc.), each other is 1: matches with 1

The computing device 300 may receive the basic RGB image and the basic depth map image and perform learning. Here, the basic RGB image and the basic depth map image may be delivered through a network.

The computing device 300 may generate a spherical virtual image based on the progressed learning. In addition, the computing device 300 provides the generated spherical virtual image to the user terminal 100 . Here, the provision of the spherical virtual image may be made in various forms, and as an example, a spherical virtual image is provided to be driven in the user terminal 100, or as another example, the user for the spherical virtual image implemented in the computing device 300 It includes providing an interface for

The provision of a spherical virtual image from the computing device 300 to the user terminal 100 may also be provided through a network.

The computing device 300 may generate a plurality of training RGB images and training depth map images by converting the basic RGB image and the basic depth map image. This is by using a characteristic environment using a spherical virtual image, and after sphericalizing the basic RGB image and the basic depth map image, a plurality of training RGB images and training depth map images can be generated through slight adjustment.

Hereinafter, various embodiments of components constituting the system 10 for providing a spherical virtual image will be described with reference to FIGS. 2 to 15 .

2 is a block diagram illustrating a computing device according to an exemplary embodiment disclosed in the present application.

The computing device 300 may include a processor 301 , a memory 302 and a communication unit 303 .

The processor 301 controls the overall operation of the computing device 300 . For example, processor 301 may perform the functions of computing device 300 described in this disclosure by executing one or more instructions stored in memory 302 .

The processor 301 may generate a spherical virtual image based on the basic RGB image transmitted from the imaging device 220 and the basic depth map image input from the distance measuring device 210 .

The processor 301 includes a training data generation module 310 that generates various training data based on the basic RGB image and the basic depth map image, a neural network module 320 that performs learning based on the training data, and an estimated depth map. and a training module 330 for training the neural network module 320 by comparing the learning depth map with the training module 330 and a virtual image providing module 340 for generating a spherical virtual image and providing distance information of the Pisa area to the user terminal. can

The training data generation module 310 may generate a plurality of training data, that is, a training RGB image and a training depth map image by performing spherical transformation on the basic RGB image and the basic depth map image and adjusting them.

For example, the learning data generating module 310 may transform the basic RGB image transmitted from the imaging device 220 and the basic depth map image transmitted from the distance measuring device 210 into a sphere. Various training data can be acquired while changing the rotation angle of the converted image based on several axes of the spherical image. In this case, the training RGB image refers to an RGB image provided to the neural network module 320 for learning, and the training depth map image refers to a depth map image provided to the neural network module 320 for learning. Accordingly, the training RGB image is an image generated from the basic RGB image, and the training depth map image is an image generated from the basic depth map image.

The neural network module 320 learns based on a learning RGB image for learning and a learning depth map image corresponding thereto. For example, the training depth map image is associated 1:1 with the training RGB image. The learning depth map image is created by actually measuring the distance using a lidar sensor for the Pisa area where the learning RGB image was created - including a distance estimation method using a stereo camera - so it is a ground truth depth map. . After learning based on the learning RGB image and the learning depth map image, the neural network module 320 may generate an estimated depth map image for the input RGB image based on the learned content.

The training module 330 may train the neural network module 320 based on the accuracy of the estimated depth map generated by the neural network module 320 .

For example, the training module 330 compares the estimated depth map generated by the neural network module 320 for the training RGB image with the training depth map, which is a measured depth map, and compares the estimated depth map with the training depth map. The neural network module 320 may be continuously trained to reduce the difference.

The neural network module 320 receives the query RGB image and generates an estimated depth map. The virtual image providing module 340 may generate a spherical virtual image based on the query RGB image and the estimated depth map. The spherical virtual image may be an image provided to the user terminal 100 from the computing device 300, for example, a virtual space that can be implemented in the user terminal 100.

The memory 302 may store programs for processing and control of the processor 301 and may store data input to or output from the computing device 300 . For example, the memory 302 may be a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD memory, etc.) ), RAM (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (ROM, Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory , a magnetic disk, and an optical disk may include at least one type of storage medium.

The communication unit 303 enables communication between the computing device 300 and another electronic device, for example, the user terminal 100 or the image acquisition device 200, or between the computing device 300 and a network where the other electronic device is located. It may contain one or more modules that

3 is a diagram illustrating an architecture for generating learning data according to an embodiment disclosed in the present application.

The basic RGB image shown in FIG. 3 is transmitted from the imaging device 220 and the basic depth map image is transmitted from the distance measuring device 210 .

Here, the basic RGB image and the basic depth map image may be equirectangular projection images used for omnidirectional virtual reality. Various types of RGB images and depth map images described below may be equirectangular projection images used to generate an omnidirectional virtual space.

4 is a diagram illustrating an equirectangular projection image and a spherical virtual image generated using the equirectangular projection image according to an embodiment disclosed in the present application.

As in the example shown in Figure (a), the equirectangular projection image can be converted into a spherical omnidirectional virtual image (hereinafter referred to as a 'spherical virtual image') providing a spherical viewpoint by the computing device 300 do.

Figure (b) shows an example of converting the equirectangular projection image of Figure (a) into a spherical virtual image. The spherical virtual image shown in Figure (b) is a spherical virtual image generated based on an RGB equirectangular projection image, and relates to an example in which distance values of each pixel are equally set. Meanwhile, figure (b) shows a spherical virtual image outside of the spherical virtual image, but is for convenience of description. Accordingly, the spherical virtual image may provide a virtual image in all directions of 360 degrees left and right and 360 degrees up and down inside the illustrated spherical virtual image.

Hereinafter, a panoramic image (eg, a 2:1 panoramic image, etc.) will be described as an example of an equirectangular projection image. A panoramic image has the advantage of being able to derive an omnidirectional image of a space by taking a single shot, and being able to convert it easily in a spherical transformation.

However, since the image delivered from the image acquisition device 200 is composed of a panoramic image as an example for implementing the present application, the image delivered from the image acquisition device 200 is a general image taken according to the user's use and convenience. image(s). And the computing device 300 may convert these general images into equirectangular projection images.

The learning data generation module 310 may receive a basic RGB image and a basic depth map image, and generate a basic spherical virtual image by performing spherical transformation on them.

The basic depth map image may be a panoramic depth map image including distance information about the Pisa region. The basic depth map image is matched 1:1 with the basic RGB image having the same subject area.

The training data generation module 310 may transform the basic spherical virtual image in various ways to generate a plurality of training data, that is, a training RGB image and a training depth map image. The training data generation module 310 may provide the generated training RGB image to the neural network module 320 and the training depth map image to the training module 330 .

That is, the training data generation module 310 generates a basic spherical virtual image using the basic RGB image and the basic depth map image, changes the setting information of the basic spherical virtual image to generate a plurality of training spherical images, and then generates a plurality of training spherical images. Based on this, various training data sets can be created.

An embodiment of the learning data generation module 310 will be further described with reference to FIGS. 5 and 6 .

5 is a diagram for explaining a method for generating learning data according to an embodiment disclosed in the present application, and FIG. 6 is a reference diagram for explaining the method for generating learning data in FIG. 5 .

Referring to FIG. 5 , the training data generation module 310 may generate a basic spherical virtual image using the basic RGB image and the basic depth map image (S501).

For example, the learning data generation module 310 may generate a basic spherical virtual image by performing spherical conversion on the basis of the basic RGB image and the basic depth map image.

This is illustrated in step (a) of FIG. 6 . That is, the basic spherical virtual image may be one basic spherical virtual image obtained from one basic RGB image and one basic depth map image.

In an embodiment of generating a basic spherical virtual image, the training data generation module 310 generates a basic spherical virtual image using the basic RGB image, and converts depth information corresponding to each pixel of the basic spherical virtual image into a basic depth map image. Based on , it is possible to generate a basic spherical virtual image.

For example, the learning data generating module 310 may generate a basic spherical virtual image in which the distance of each pixel is expressed as an equal distance by performing spherical conversion on the basic RGB image. The learning data generation module 310 may store distance information corresponding to each pixel of the basic RGB image in association with the basic spherical virtual image by using the basic depth map image. For example, distance information corresponding to each pixel of the RGB image may be stored as a table including identification information and distance information for each pixel.

In such an embodiment, when the setting information for the basic spherical virtual image is changed, the learning data generation module 310 may change the storage of distance information in response to such a change. For example, when a basic spherical virtual image is rotated in a specific direction and at a specific angle with respect to a specific rotation axis, distance information may be obtained from a table by reflecting a position change of a pixel changed by such rotation.

In another embodiment of generating the basic spherical virtual image, the training data generating module 310 may generate spherical virtual images for each of the basic RGB image and the basic depth map image.

For example, the learning data generation module 310 performs spherical transformation on the basic RGB image to generate a first basic spherical virtual image in which the distance of each pixel is expressed as an equal distance, and spherical transformation on the basic depth map image so that each pixel is a distance A second basic spherical virtual image represented by information may be generated.

In this embodiment, the learning data generation module 310 changes the setting information identically for a pair of the first basic spherical virtual image and the second basic spherical virtual image, and - the pair of first and second basic spherical virtual images in which the setting information is converted. The second basic spherical virtual image corresponds to the training spherical virtual image, and a pair of training RGB images and a training depth map image may be generated by performing a planar transformation on the training spherical virtual image.

In another embodiment of generating the basic spherical virtual image, the training data generation module 310 may generate a 3D stereoscopic basic depth map image by reflecting both color information and distance information to one pixel. That is, in the above-described embodiments, as in the example shown in FIG. 6, since the distance of each pixel is set to be constant, the shape of the basic spherical virtual image is displayed as a round rectangle. Therefore, it is displayed as a three-dimensional shape in a three-dimensional space, not a round sphere.

For example, the learning data generation module 310 obtains color information for each pixel from the basic RGB image and obtains distance information for each pixel from the basic depth map image, and obtains color information and distance for each pixel. information can be set. The learning data generation module 310 may generate a basic spherical virtual image by expressing color information and distance information for each set pixel in 3D coordinates. This basic spherical virtual image is expressed as a three-dimensional shape displayed on a three-dimensional space rather than a circular shape.

The training data generation module 310 may generate a plurality of training spherical images by changing the setting information of the basic spherical virtual image (S502). For example, the setting information may include a rotation axis, a rotation direction, or a rotation angle of the spherical image.

For example, the training data generating module 310 may generate a plurality of training sphere images from the basic sphere virtual image by changing at least one of the rotation axis, rotation direction, and rotation angle of the basic sphere virtual image.

Step (b) of FIG. 6 shows an example of generating a plurality of training spherical images by changing the setting information of the basic spherical virtual image.

The learning data generation module 310 plane-converts the plurality of training spherical images again to obtain a plurality of training data sets -a training data set means a training RGB image and a pair of training depth map images 1:1 matched thereto- can be generated (S503). Here, the planar transformation is the inverse transformation of the spherical transformation, and a set of training RGB images and training depth map images may be generated by performing a planar transformation of one training spherical image.

In this way, generating a plurality of training spherical images by changing basic spherical virtual image setting information provides an effect of generating a large amount of training data from one basic spherical image. That is, although the accurate calculation capability of the neural network module 320 is based on a lot of training data, it is difficult to actually secure a lot of training data. However, in one embodiment of the present application, a plurality of training spherical images can be generated by applying various transformations based on the basic spherical virtual image, and a large amount of training data set can be easily secured by inverse transformation. there is.

The plurality of training RGB images and training depth map images generated in this way may be provided to the neural network module 320 and used as training information.

7 is a diagram for explaining an example of a neural network architecture according to an embodiment disclosed in this application, and FIG. 8 is a diagram for explaining another example of a neural network architecture according to an embodiment disclosed in this application. .

For ease of explanation, it is described that the neural network module 320 shown in FIGS. 7 and 8 is implemented using the computing device 300 described in FIG. 2 . That is, it may be implemented by executing at least one instruction performed by the memory 302 and the processor 301 of the computing device 300 . However, in addition, the neural network module 320 may be used in any other suitable device(s) and any other suitable system(s). Further, the neural network module 320 is described as being used to perform image processing related tasks. However, the neural network module 320 may be used to perform any other suitable task in conjunction, including non-image processing tasks.

The neural network module 320 learns based on a learning RGB image for learning and a depth map image for the learning RGB image.

The neural network module 320 is a deep learning-based image conversion learning model, and may generate an estimated depth map image based on conversion of an input training RGB image through a learning neural network.

The neural network module 320 may be expressed as a mathematical model using nodes and edges. The neural network module 320 may be an architecture of a deep neural network (DNN) or n-layers neural networks. DNN or n-layer neural networks include Convolutional Neural Networks (CNN), Convolutional Neural Networks (CNN) based on HRNet (Deep High-Resolution Network), and Recurrent Neural Networks (RNN). ), Deep Belief Networks, and Restricted Boltzman Machines.

For example, as in the example shown in FIG. 7 , the neural network module 320 may receive a training RGB image and generate an estimated depth map image for the received RGB image. In this example, since there is no learned content in the initial operation, the neural network module 320 may generate an estimated depth map image based on a random value in each node of the neural network. The neural network module 320 may improve the accuracy of the estimated depth map by repeatedly performing feedback training on the generated estimated depth map image.

As another example, as shown in FIG. 8 , the neural network module 320 receives a training RGB image and a training depth map set matching the training RGB image, and learns a correlation between the RGB image and the depth map image based on learning therefrom. and, based on such correlation, an estimated depth map image for the input learning RGB can be generated. Even in this example, the accuracy of the estimated depth map may be improved by repeatedly performing feedback training on the estimated depth map image.

FIG. 9 is a diagram for explaining a training method using a neural network according to an embodiment disclosed in the present application, which will be further described with reference to FIG. 9 .

The neural network module 320 receives the training RGB image and generates a predicted depth map image based on the learned content (S901).

A training RGB image refers to an RGB image provided to neural network module 320 for training. A training depth map image refers to a depth map image provided to the neural network module 320 or the training module 330 for learning. The training depth map image is related 1:1 with the training RGB image. The learning depth map image is a Ground Truth Depth map because it is generated by actually measuring a distance using a Lidar sensor or the like with respect to the Pisa area where the learning RGB image is generated.

The neural network module 320 is a deep learning-based image conversion learning model, and may generate an estimated depth map image based on conversion of an input training RGB image through a learning neural network.

Thereafter, the neural network module 320 performs learning through a training process described later (S902). As described above, the neural network module 320 can perform learning on a plurality of training RGB images and training depth maps generated by the training data generation module 310, so that its accuracy can be easily increased.

The estimated depth map image is a depth map generated by the trained neural network module 320 . This estimated depth map image is different from the learning depth map image, which is a ground truth depth map generated by actually measuring a distance using a Lidar sensor or the like. Accordingly, the neural network module 320 may be trained so that the difference between the estimated depth map image and the training depth map (actually measured depth map) image is small, and learning of the neural network module 320 is performed by the training module 330. is performed by

The training module 330 may compare the estimated depth map generated by the neural network module 320 and the learning depth map, and train the neural network module 320 based on the difference.

In one embodiment, training module 330 may perform training based on a spherical transformation. For example, the training module 330 performs spherical transformation on the estimated depth map and the training depth map, respectively, and then trains the neural network module 320 based on the difference between the spherically transformed estimated depth map and the spherically transformed learning depth map. there is.

In this embodiment, the training RGB image, the estimated depth map image, and the training depth map image may all be equirectangular projection images. That is, since this equirectangular projection image is used in a spherical transformed state, in order to more accurately determine the difference between the estimated depth map image and the learning depth map image in the use state, the training module 330 uses the estimated depth map and the learning depth. After transforming the maps into spherical shapes, they are compared to perform training. This will be further described with reference to FIG. 10 .

10 is a diagram illustrating a difference between an equirectangular projection image according to an embodiment disclosed in the present application and a spherical virtual image generated using the same.

Area A (1010) and area B (1020) have the same area and shape in a spherical virtual image, but when converted to an equirectangular projection image, area A' (1011) and area B' (1021) are different from each other. and have a shape. This is due to conversion between a spherical virtual image and a planar equirectangular projection image (panoramic image).

Accordingly, the training module 330 may perform training after performing spherical transformation on the estimated depth map and the learning depth map, respectively, thereby increasing the accuracy of training and, accordingly, the accuracy of the estimated depth map image.

11 is a diagram for explaining a training method using a training module according to an embodiment disclosed in the present application, which will be described with reference to FIGS. 8 and 11 below.

As shown in FIG. 8 , the training module 330 may include a spherical transformation module 331 , a loss calculation module 332 and an optimization module 333 .

Referring further to FIG. 11 , the spherical transformation module 331 performs spherical transformation so that the equirectangular projection image corresponds to the spherical transformation. The spherical transform module 331 may receive an estimated depth map image and a learning depth map image, and perform spherical transform on each of them (S1101).

In FIG. 8 , the learning depth map image subjected to spherical transformation by the spherical transformation module 331 is displayed as 'learning depth map*', and the estimated depth map image subjected to spherical transformation is indicated as 'estimated depth map*'.

In one embodiment, the rectangle transformation module 331 may perform rectangle transformation using Equation 1 below.

[Equation 1]

A detailed description of Equation 1 can be easily understood with reference to the description shown in FIG. 16 .

The loss calculation module 332 may calculate a loss between the spherical transformed estimated depth map (estimated depth map*) and the spherically transformed learning depth map (learning depth map*) (S1102).

That is, the loss calculation module 332 may quantify (loss value) the difference between the spherically transformed estimated depth map and the spherically transformed learning depth map. For example, the loss value determined by the loss calculation module 332 may be determined to be in the range of 0 to 1.

The optimizing module 333 may receive the calculated loss from the loss calculating module 332 and perform optimization by changing parameters of the neural network in response to the loss (S1103).

For example, the optimization module 333 may perform optimization by adjusting the weight parameter W of the neural network. As another example, the optimization module 333 may perform optimization by adjusting at least one of the weight parameter W and the bias b of the neural network.

Optimizing methods of various types can be applied to the optimization module 333 . For example, the optimization module 333 may perform batch gradient descent, stochastic gradient descent, mini-batch gradient descent, momentum, Optimization can be performed using Adagrad, RMSprop, etc.

12 is a diagram explaining a loss calculation method according to an embodiment disclosed in the present application.

In the embodiment shown in FIG. 12 , the loss calculation module 332 may determine a loss by applying a plurality of loss calculation methods and calculating a representative value from the resultant values.

Referring to FIG. 12 , the loss calculation module 332 may calculate a first loss function result between the spherical transformed estimated depth map and the spherically transformed learning depth map using a first loss calculation method ( S1201 ).

Equation 2 below is an equation explaining an example of the first loss calculation equation.

[Equation 2]

Here, T is the number of samples, y is the training depth map, and y* is the estimated depth map.

The loss calculation module 332 may calculate a second loss function result between the sphere-transformed estimated depth map and the sphere-transformed learning depth map in a second loss calculation method ( S1202 ).

Equation 3 below is an equation explaining an example of the second loss calculation equation.

[Equation 3]

where T is the number of samples and d is the difference between the training depth map and the estimated depth map in log space.

The loss calculation module 332 may calculate a third loss function result between the sphere-transformed estimation depth map and the sphere-transformed learning depth map using the third loss calculation method (S1203).

Equation 4 below is an equation explaining an example of the third loss calculation equation.

[Equation 4]

Here, y _true means the training depth map and y _predicted means the estimated depth map.

The loss calculation module 332 may calculate a representative value for the result of the first loss function to the result of the third loss function and determine it as a loss (S1204). Here, the average value, the median value, the mode value, etc. can be applied as the representative value.

13 is a diagram illustrating a training RGB image, a training depth map image, an estimated depth map image, and a difference image between the estimated depth map image and the training depth map image according to an example.

Figure (a) shows an example of a training RGB image input to the neural network module 320.

Figure (b) shows an example of an estimated depth map image generated by the neural network module 320 receiving the training RGB image.

Figure (c) shows an example of a training depth map image providing measured depth values corresponding to the training RGB image.

Figure (d) illustrates a difference image between the estimated depth map image and the training depth map image. However, figure (d) is for intuitive explanation, and as described above, in an embodiment disclosed in the present application, the training module 330 performs spherical transformation on the estimated depth map image and the training depth map image. If the difference is calculated after doing so, it may be displayed differently from Figure (d).

14 is a diagram for explaining a spherical virtual image generation architecture according to an embodiment disclosed in the present application, and FIG. 15 is a diagram for explaining a method for providing a spherical virtual image to a user according to an embodiment disclosed in the present application. .

A method of providing a spherical virtual image to a user according to an exemplary embodiment disclosed in the present application will be described with reference to FIGS. 14 and 15 .

When the neural network module 320 receives the query RGB image, it generates an estimated depth map corresponding to the query RGB image using the neural network trained as described above (S1501).

Here, the query RGB image is an RGB image used to generate a spherical virtual image and does not have a ground truth map matched thereto. Accordingly, an estimated depth map is generated using the neural network module 320 and used to generate a spherical virtual image.

The neural network module 320 provides the generated estimated depth map to the virtual image providing module 340 .

The virtual image providing module 340 may generate a spherical virtual image based on the query RGB image and the estimated depth map image provided by the neural network module 320 (S1502).

For example, the virtual image providing module 340 may check the estimated depth map generated by the neural network module 320 and generate a spherical virtual image using the query RGB image and the estimated depth map.

Here, the spherical virtual image collectively refers to an image for providing a virtual space that a user can experience.

For example, the spherical virtual image may generate a spherical virtual image based on the query RGB image, and may include distance information of each pixel included in the estimated depth map image for each pixel of the spherical virtual image. Figure (b) of Figure 4 shows an example of such a spherical virtual image. are showing Even if it is displayed in this way, since distance information for each pixel is included, it is possible to obtain distance information for each pixel in the spherical virtual image.

As another example, the spherical virtual image displays the position and color of each pixel on 3D coordinates using color information about each pixel obtained from the query RGB image and distance information about each pixel obtained from the estimated depth map image. can do. Another example of such a spherical virtual image may be displayed in a three-dimensional space displayed on three-dimensional coordinates.

That is, the spherical virtual image may include distance information about at least one point (eg, pixel) included in the virtual image. Here, distance information is determined based on the estimated depth map.

The virtual image providing module 340 may provide a spherical virtual image to the user. For example, the virtual image providing module 340 may provide the user terminal 100 with a user interface including an access function to a spherical virtual image.

The virtual image providing module 340 may receive a user request from a user through a user interface (S1503). For example, the virtual image providing module 340 may receive a request for checking a distance to at least one dot in the spherical virtual image, that is, a user query. The virtual image providing module 340 may check distance information about dots in the spherical virtual image in response to a user's query, and provide the information to the user. For example, in the spherical virtual image provided to the user terminal 100, the user can set a desired object or position in space, and the virtual image providing module 340 checks distance information for this and returns the user terminal 100 to the user terminal 100. can provide

The present application described above is not limited by the foregoing embodiments and the accompanying drawings, but is limited by the claims to be described later, and the configuration of the present application may be varied within a range that does not deviate from the technical spirit of the present application. Those skilled in the art can easily know that it can be changed and modified accordingly.

100: user terminal
200: image acquisition device
300: computing device

Claims (15)

As a computing device,
a memory that stores one or more instructions; and
a processor to execute the one or more instructions stored in the memory;
The processor, by executing the one or more instructions,
Receiving a basic RGB image and a corresponding basic depth map image;
Creating a plurality of training spherical images from the basic spherical virtual image by changing setting information of a basic spherical virtual image generated by performing spherical transformation based on the basic RGB image and the basic depth map image;
Planar transforming the plurality of training spherical images, respectively, to generate a plurality of training RGB images and a plurality of training depth map images that are matched 1: 1, respectively.
computing device.
According to claim 1,
The learning RGB image and the learning depth map image are equirectangular projection images,
computing device.
The method of claim 1, wherein the setting information,
Information on at least one of a rotation axis, a rotation direction, or a rotation angle for the basic spherical virtual image,
computing device.
delete According to claim 1,
The processor, by executing the one or more instructions,
The basic RGB image is converted into a sphere to generate the basic spherical virtual image in which the distance of each pixel is expressed as an equal distance;
Storing distance information corresponding to each pixel of the basic RGB image in association with the basic spherical virtual image using the basic depth map image
computing device.
According to claim 1,
The processor, by executing the one or more instructions,
Spherical conversion of the basic RGB image to generate a first basic spherical virtual image in which the distance of each pixel is expressed as an equal distance;
Spherical transforming the basic depth map image to generate a second basic spherical virtual image in which each pixel is represented by distance information;
computing device.
According to claim 1,
The processor, by executing the one or more instructions,
obtaining color information for each pixel from the basic RGB image and obtaining distance information for each pixel from the basic depth map image, and setting color information and distance information for each pixel;
Creating the basic spherical virtual image by expressing color information and distance information for each set pixel in three-dimensional coordinates,
computing device.
As a learning data generation method performed on a computing device,
receiving a basic RGB image and a basic depth map image corresponding thereto;
generating a basic spherical virtual image generated by spherical transformation based on the basic RGB image and the basic depth map image;
generating a plurality of training spherical images by changing setting information of the basic spherical virtual image; and
generating the plurality of training RGB images and the plurality of training depth map images that are matched 1:1 by plane-converting the plurality of training spherical images, respectively;
including,
How to generate training data.
According to claim 8,
The learning RGB image and the learning depth map image are equirectangular projection images,
How to generate training data.
The method of claim 8, wherein the setting information,
Information on at least one of a rotation axis, a rotation direction, or a rotation angle for the basic spherical virtual image,
How to generate training data.
delete The method of claim 8, wherein the operation of generating the basic spherical virtual image,
performing spherical conversion on the basic RGB image to generate the basic spherical virtual image in which the distance of each pixel is expressed as an equal distance; and
storing distance information corresponding to each pixel of the basic RGB image in association with the basic spherical virtual image by using the basic depth map image;
including,
How to generate training data.
The method of claim 8, wherein the operation of generating the basic spherical virtual image,
performing spherical conversion on the basic RGB image to generate a first basic spherical virtual image in which distances of respective pixels are expressed as equal distances; and
performing spherical transformation on the basic depth map image to generate a second basic spherical virtual image in which each pixel is represented by distance information;
including,
How to generate training data.
The method of claim 8, wherein the operation of generating the basic spherical virtual image,
obtaining color information for each pixel from the basic RGB image and distance information for each pixel from the basic depth map image, and setting color information and distance information for each pixel; and
generating the basic spherical virtual image by expressing color information and distance information for each set pixel in 3D coordinates;
including,
How to generate training data.
A storage medium storing computer readable instructions,
The instructions, when executed by a computing device, cause the computing device to:
receiving a basic RGB image and a basic depth map image corresponding thereto;
generating a plurality of training spherical images from the basic spherical virtual image by changing setting information of a basic spherical virtual image generated by performing spherical transformation on the basis of the basic RGB image and the basic depth map image; and
generating a plurality of training RGB images and a plurality of training depth map images that are matched 1:1 by plane-converting the plurality of training spherical images, respectively;
to do,
storage medium.

Images