JP2022160382A - Method and system for generating learning data for machine learning - Google Patents

Abstract

To provide a method and system for generating learning data for machine learning, and a method and system for generating a sample image.SOLUTION: A learning data generation method for machine learning includes: a step (S110) of generating a sample image using a target image (including a depth map generated from the target image); a step (S120) of estimating depth values of pixels included in the sample image, to generate a depth map for the sample image; and a step (S130, S140) of generating learning data using the depth map generated from the target image and the depth map generated from the sample image.SELECTED DRAWING: Figure 3

Description

TECHNICAL FIELD The present invention relates to a learning data generation method and system, and more particularly to a learning data generation method for machine learning and a machine learning method using the same.

In three-dimensional computer graphics, a depth map provides information about the distance from the viewpoint to the surface of the object. Three-dimensional information acquired by depth maps is actively used in 3D modeling, robotics, medical, aviation, national defense, autonomous driving, and the like.

On the other hand, artificial intelligence, which is the core of the quaternary industry, has made dramatic progress through deep learning (machine learning in the broad sense), which is a combination of machine learning (machine learning in the narrow sense) and neural networks that mimic the human brain. ing.

With the development of machine learning, the recent depth estimation technology field focuses on using machine learning to perform 3D reconstruction from 2D images.

In this case, machine-learning-based depth estimation techniques leverage sequential images when training depth estimation models on an unsupervised basis. However, in the depth estimation techniques known so far, the static scene assumption is applied that objects in consecutive images should not move, and if there is a dynamic object in the images, the consecutive images are completely reproduced. There was a problem that it could not be used as a learning data.

For example, as shown in FIG. 1, when a camera is attached to a vehicle 100 and images are collected, if the collected images include other cars in motion, they are excluded from the learning data of the depth estimation model. Therefore, there is a problem that the depth estimation result is inaccurate.

In order to solve the above problems and improve inference performance, the present invention provides a new learning data generation method that can fully utilize images containing dynamic objects as learning data in a depth estimation model that utilizes continuous images. Suggest.

Godard, Clement, et al. "Digging Into Self-Supervised Monocular Depth Estimation" ICCV, 2019.

SUMMARY OF THE INVENTION An object of the present invention is to provide a new learning data generation method that can fully utilize images containing dynamic objects as learning data.

The present invention also provides a method and system for generating self-samples for unsupervised machine learning and utilizing them to generate learning data.

More specifically, the present invention relates to a machine learning data generation method and system capable of highly accurate depth estimation from a single image containing dynamic objects.

Further, the present invention relates to a method and system for generating multiple self-samples from a single image for use in unsupervised depth estimation learning.

In order to solve the above problems, the present invention provides a step of generating a sample image using a target image and a depth map of the target image; A learning data generating method for machine learning, comprising the steps of: generating a depth map; and generating at least part of learning data using the depth map of the target image and the depth map generated from the sample image. I will provide a.

Further, the present invention includes a storage unit that stores a target image, and a control unit that estimates depth values of pixels included in the target image and generates a depth map for the target image, wherein the control unit includes: generating a sample image using the target image and the depth map; estimating depth values of pixels included in the sample image to generate a depth map for the sample image; a training data generation system for machine learning, wherein the depth map generated from the sample image and the depth map subjected to the rigid transformation are used to generate at least a portion of the training data; offer.

Further, the present invention is a program executed by one or more processes in an electronic device and storable in a computer-readable recording medium, the program estimating depth values of pixels included in a target image to generating a depth map for an image; generating a sample image using the target image and the depth map; and estimating depth values of pixels included in the sample image to generate a depth map for the sample image. performing rigid transformation on the depth map generated from the target image; and transforming at least part of learning data using the depth map generated from the sample image and the depth map subjected to the rigid transformation. A program storable on a computer-readable recording medium is provided that includes commands for performing the generating steps.

Further, the present invention is a program executable by one or more processes in an electronic device and storable on a computer-readable recording medium, the program comprising the steps of: estimating depth values of pixels included in a target image; mapping the pixels included in the target image onto a three-dimensional space based on the depth values of the pixels included in the target image; A program storable on a computer-readable recording medium, comprising commands for performing rigid transformation with a parameter and projecting the rigidly transformed pixels onto a two-dimensional plane to generate a sample image. do.

As described above, the present invention allows multiple applications of rigid transformation parameters to generate multiple self-samples from a single target image. Therefore, the present invention can reserve an unlimited number of images for depth estimation learning.

Also, since the sample images generated by the self-sampling method according to the present invention consist entirely of static regions, there is no need to filter dynamic regions in the sample images, which reduces depth estimation accuracy. Therefore, the present invention can utilize the entire sample image for learning.

Furthermore, the learning data generation method according to the present invention can calculate losses in various situations by applying various rigid transformation parameters when generating sample images. Specifically, according to the present invention, the loss is calculated in all cases where the camera collecting the target image can move. It can be carried out.

FIG. 4 is a conceptual diagram for explaining a depth estimation method used during autonomous driving; 1 is a conceptual diagram for explaining a system according to the present invention; FIG. 4 is a flowchart for explaining a learning data generation method according to the present invention; FIG. 4 is a conceptual diagram showing a method for executing the learning data generation method according to the present invention; FIG. 4 is a conceptual diagram showing a method for executing the learning data generation method according to the present invention; 4 is a flow chart for explaining a sample image generation method according to the present invention; 1 is a conceptual diagram showing a sample image generation method according to the present invention; FIG. 1 is a conceptual diagram showing a sample image generation method according to the present invention; FIG. 1 is a conceptual diagram showing a sample image generation method according to the present invention; FIG. 1 is a conceptual diagram showing a conventional depth estimation learning method; FIG. 1 is a conceptual diagram showing a conventional depth estimation learning method; FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same or similar components are denoted by the same reference numerals regardless of the drawing numbers, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” used in the following explanation are given or used together to facilitate the preparation of the specification, and themselves have significance and usefulness. does not have In addition, in describing the embodiments of the present invention, detailed descriptions of related known techniques will be omitted if it is determined that they may obscure the gist of the embodiments of the present invention. Furthermore, the accompanying drawings are only for helping understanding of the embodiments of the present invention, and the technical ideas of the present invention are not limited by the accompanying drawings. , equivalents or alternatives.

Terms including ordinal numbers such as "first", "second", etc. are used to describe various components, but the components are not limited by the above terms. The above terms are only used to distinguish one component from another.

When a component is referred to as being "coupled" or "connected" to another component, it may be directly coupled or connected to the other component, with additional components in between. It should be interpreted as something that can be done. In contrast, when a component is referred to as being "directly coupled" or "directly connected" to another component, it should be understood that there are no additional components in between.

References to the singular include the plural unless specifically stated otherwise.

As used herein, terms such as "including" and "having" are intended to specify the presence of features, numbers, steps, acts, components, parts, or combinations thereof described herein. and does not preclude the presence or addition of one or more other features, figures, steps, acts, components, parts or combinations thereof.

The present invention relates to a method of generating self-samples for depth estimation from a single image and using said self-samples to generate machine learning data.

In the present invention, for convenience of explanation, the original image used for depth estimation learning is referred to as a "target image". The target image may be an image collected from a camera. More specifically, the target image may be an image collected from a camera located on vehicle 200 .

On the other hand, a plurality of images generated from the target image and used for unsupervised depth estimation learning are called "sample images". A sample image is an image that is generated based on a target image, but which is not exactly the same as the target image. A plurality of sample images may be generated from the target image, and the plurality of sample images are different images. In the present invention, such a plurality of sample images are also called "self-samples".

The present invention generates learning data using target images and sample images. The "learning data" in this specification is data utilized for machine learning, and includes a target image and a depth map of the target image, a sample image and a depth map of the sample image, a depth map of the target image and the It means loss data calculated based on the depth map of the sample image and data generated in all calculation processes for minimizing the value of the loss data.

"Generating learning data" in this specification includes generating a sample image, generating a depth map from a target image, generating a depth map from a sample image, and calculating loss data using a loss function. , using the calculated loss data to modify the weights needed during depth estimation.

A system according to the present invention generates a plurality of sample images from a target image, and utilizes the target image and the sample images to generate learning data for depth estimation. Prior to specifically describing the present invention, the system according to the present invention will be specifically described.

FIG. 2 is a conceptual diagram for explaining the system according to the present invention.

First, as shown in FIG. 2, the vehicle 200 means all means of transportation running on roads and railroad tracks. Vehicle 200 may include at least one camera 210 for capturing images. Specifically, the vehicle may include multiple cameras that capture images in the same direction, or multiple cameras that capture images in different directions. A "target image" as used herein means a single image captured in a specific direction.

Meanwhile, the depth estimation system 300 according to the present invention includes at least one of a communication unit 310 , a storage unit 320 and a control unit 330 . System 300 may be included in vehicle 200 or may be a separate server located external to vehicle 200 . In this specification, vehicle 200 and system 300 are described separately for convenience of explanation, but system 300 may be included in vehicle 200 .

The communication unit 310 can communicate with at least one of the vehicle 200, an external storage (eg, database 340), an external server, and a cloud server.

Note that the external server or cloud server may be configured to play at least a part of the control unit 330 . That is, the execution of data processing, data calculation, etc. may be performed by an external server or a cloud server, and the present invention does not care about the method.

In addition, the communication unit 310 can support various communication methods in compliance with communication standards of communication targets (for example, electronic equipment, external servers, devices, etc.).

For example, the communication unit 310 supports WLAN (Wireless LAN), Wi-Fi (Wireless Fidelity), Wi-Fi Direct (Wireless Fidelity Direct), DLNA (registered trademark) (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX （Ｗｏｒｌｄ Ｉｎｔｅｒｏｐｅｒａｂｉｌｉｔｙ ｆｏｒ Ｍｉｃｒｏｗａｖｅ Ａｃｃｅｓｓ）、ＨＳＤＰＡ（Ｈｉｇｈ－Ｓｐｅｅｄ Ｄｏｗｎｌｉｎｋ Ｐａｃｋｅｔ Ａｃｃｅｓｓ）、ＨＳＵＰＡ（Ｈｉｇｈ－Ｓｐｅｅｄ Ｕｐｌｉｎｋ Ｐａｃｋｅｔ Ａｃｃｅｓｓ）、ＬＴＥ（Ｌｏｎｇ Ｔｅｒｍ Ｅｖｏｌｕｔｉｏｎ）、ＬＴＥ－Ａ（Ｌｏｎｇ Ｔｅｒｍ Ｅｖｏｌｕｔｉｏｎ－Ａｄｖａｎｃｅｄ）、５Ｇ（５ｔｈ Ｇｅｎｅｒａｔｉｏｎ Mobile Telecommunication), Bluetooth (Registered Trademark), RFID (Radio Frequency Identification), IrDA (Infrared Data Association), UWB (Ultra Wide Band), ZigBeeWireless, NFC (Near Wireless) Wireless Universal Serial Bus) technology may be used to communicate with the communication target.

Next, the storage unit 320 may store various information according to the present invention. In the present invention, the storage unit 320 may be provided in the system 300 itself according to the present invention. Alternatively, at least part of the storage unit 320 may be at least one of a database (DB) 340 and cloud storage (or cloud server). In other words, the storage unit 320 has no physical space limitation as long as it stores information necessary for the system and method according to the present invention. Therefore, hereinafter, the storage unit 320, the database 340, the external storage, and the cloud storage (or cloud server) are not classified and are all referred to as the storage unit 320.

Information stored in the storage unit 320 for sample image generation and depth estimation according to the present invention may include a target image and a plurality of sample images generated from the target image.

Controller 330 is then configured to control the overall operation of system 300 in accordance with the present invention. The control unit 330 may process signals, data, information, etc. input or output from the above components, or may provide or process appropriate information or functions to a user.

The control unit 330 includes at least one central processing unit (CPU) and can perform functions according to the present invention. The control unit 330 can also perform artificial intelligence-based data processing, and can perform sample image generation and depth estimation according to the present invention. Further, the control unit 330 can perform sample image generation and depth estimation according to the present invention by at least one of machine learning and deep learning.

Prior to explaining the learning data generation method according to the present invention, a conventional depth estimation learning method utilizing continuous images will be explained.

Conventionally, two target images (hereinafter also referred to as a first target image and a second target image) photographed from different viewpoints with the same camera are utilized. Here, it is assumed that only the camera is moving and all objects contained in the images are stationary when the two target images are captured. According to this assumption, it is assumed that the pixels contained in the first target image are rigidly transformed to form the second target image. An ego-motion estimator may be applied to the first target image and the second target image to calculate rigid body transformation parameters (or parameters).

A known model can be used as the ego motion estimator. For example, the ego-motion estimator disclosed in Non-Patent Document 1 may be used, but is not limited thereto.

Once the rigid transformation parameters are calculated, an inverse rigid transformation is possible. Specifically, rigid transformation parameters for a three-dimensional image may be calculated in the form of a 4×4 matrix. By multiplying the pixels (pixel vectors) forming the second target image by the inverse matrix of the parameter matrix, an inverse rigid body transformation result is calculated.

A new image is generated by performing an inverse rigid transformation on all the pixels that make up the second target image.

Specifically, as shown in FIG. 7a, assuming that the second target image 740 is the result of rigid transformation of the first target image 710, the pixel p2 forming the first target image 710 is the second target image 740 is rigid-transformed to any one pixel p3.

An ego-motion estimator (G) can compute rigid transformation parameters for the first target image 710 and the second target image 740 .

After that, when inverse rigid transformation (T) is performed on any one pixel p3 constituting the second target image 740 using the calculated rigid transformation parameter, a pixel p3′ subjected to the inverse rigid transformation is generated. be done. An inverse rigid transformation image 740 ′ is generated by performing inverse rigid transformation on all the pixels that make up the second target image 740 . In this specification, an image generated as a result of inverse rigid transformation is referred to as an inverse rigid transformation image.

According to the above assumption, when the inverse rigid body transformation is performed on a specific pixel forming the second target image, the specific pixel is the same as the pixel corresponding to the specific pixel among the pixels forming the first target image. must move into position.

Using the first target image and the inverse rigid-transformed image, the photometric loss is calculated according to Equation 1 below. Photometric loss is the error calculated assuming that all objects in the images are rigid bodies and that the two images are taken while the camera is stationary and only the camera is moving.

（数式１）
上記数式１は、非特許文献１に開示された数式であるので、具体的な説明は省略する。

(Formula 1)
Since the formula 1 is disclosed in Non-Patent Document 1, a detailed description thereof will be omitted.

On the other hand, depth estimation 730 of the first target image 710 produces a depth map 720, as shown in FIG. 7b. The inverse rigid transform image 740 ′ generated in the process of FIG. 7 a is warped (W) with the depth map 720 . Then, the pixel p3'' included in the warped image and the pixels included in the first target image 710 are used to calculate the loss according to Equation 1 above. The machine learning for depth estimation described in FIGS. 7a and 7b learns to minimize the photometric loss.

The depth estimation learning method of the above-described scheme gives inaccurate results when the object in the image is moving. In order to prevent this, objects that move in the image are filtered, but in that case there is a problem that the entire target image cannot be used for learning.

SUMMARY OF THE INVENTION The present invention provides a machine learning data generation method that can utilize a dynamic object in learning even if the target image contains a dynamic object, and generate an unlimited number of consecutive images for learning.

Hereinafter, the machine learning data generating method according to the present invention will be described more specifically with reference to the accompanying drawings together with the above configuration.

FIG. 3 is a flowchart for explaining the machine learning data generation method according to the present invention, and FIGS. 4a and 4b are conceptual diagrams showing a method for executing the machine learning data generation method according to the present invention.

First, a step of generating self-samples using the target image (S110) is performed.

A plurality of sample images are generated, and each of the plurality of sample images is generated with different rigid body transformation parameters. Each of the multiple sample images is utilized for depth estimation learning. This specification describes a method of performing depth estimation learning using one sample image and a target image. A sample image generation method will be described later.

After generating the self-samples, the step of estimating (S120) the depth values of the pixels contained in the self-samples is performed.

A depth map is calculated by applying a depth estimation method to the sample image. At this time, as the depth estimation model, the same model as the depth estimation model applied to the target image when generating the sample image is applied. The depth estimation model and generation of sample images will be described later.

Once the depth estimation step is performed, a depth map is generated. The depth map includes pixel coordinate information and depth value information of each pixel. The pixel coordinate information is the coordinates corresponding to the pixel of the target image, and the pixel depth value is information indicating the depth value calculated for the specific pixel. The depth map may be output as an image. A depth map image includes a plurality of pixels, each pixel including coordinate information and depth information. A depth map image defines a depth value instead of color information matched to each of the pixels contained in the target image.

A depth map calculated from the sample image includes coordinate information and depth information. The coordinate information included in the depth map is the same information as the coordinate information included in the sample image, and the depth information is matched with each piece of coordinate information. The relationship between the sample image and the depth map is the same as the relationship between the target image and the depth map generated from the target image.

Next, a step (S130) of mapping the pixels constituting the depth map calculated from the target image onto a three-dimensional space and then performing rigid body transformation on the mapped pixels is performed.

A rigid transformation means a geometric transformation that preserves the Euclidean distance between all pairs of points. Rigid transformations include translations, rotations, reflections, or combinations thereof. All objects retain the same shape and size after rigid transformations are performed.

Although translations, rotations, or a combination thereof are described herein as one embodiment of performing the rigid transformations, rigid transformations herein include other types of transformations other than translations and rotations. is also included.

On the other hand, each pixel mapped on the three-dimensional space may consist of a vector containing coordinate information. That is, each pixel may consist of a vector containing X-axis coordinate information and Y-axis coordinate information. In this specification, a vector containing coordinate information of each pixel is called a pixel vector.

A rigid transformation may be performed by multiplying the pixel vector by a matrix containing the rigid transformation parameters. In this specification, a matrix containing rigid transformation parameters is referred to as a parameter matrix.

In one embodiment, the rigid transformation parameters include at least one of an X-axis distance value, a Y-axis distance value, and a Z-axis distance value to be moved. Also, the rigid body transformation parameter includes at least one of a rotation angle with respect to the X-axis, a rotation angle with respect to the Y-axis, and a rotation angle with respect to the Z-axis. As mentioned above, when performing a rigid transformation consisting of translation and rotation, the rigid transformation parameters include up to 6 different parameters. However, it is not limited to this, and when other types of rigid body transformations other than the translation and rotation are performed, the rigid body transformation parameters may include parameters other than the six parameters described above.

On the other hand, the parameter matrix used for rigid transformation in two dimensions is 3×3, and the parameter matrix used for rigid transformation in three dimensions is 4×4.

A depth map generated from the target image includes coordinate information and depth information. Here, the coordinate information is coordinate information that defines two-dimensional coordinates. For example, a depth map generated from a target image may include X-axis coordinate information and Y-axis coordinate information.

For rigid transformation, the pixels that make up the depth map may be mapped onto a three-dimensional space. Specifically, a plurality of pixels forming the depth map may be converted into a vector. Here, each pixel making up the depth map is transformed into a three-dimensional vector. Specifically, when generating a three-dimensional vector, both two-dimensional coordinate information and depth information included in pixels are utilized. That is, depth information included in pixels is used as coordinate information about a specific axis.

Rigid body transformation is performed by applying the same rigid body transformation parameters that were applied when the sample image was generated to the three-dimensional vector generated as described above.

After that, any one of the three types of coordinate information included in the vector newly generated by the rigid body transformation is converted into depth information to generate a new pixel. Applying the above process to all pixels contained in the depth map produces a new depth map.

In one embodiment, the depth information contained in the depth map is converted to Z-axis coordinate information to generate a three-dimensional vector. The rigid transformation parameters applied when generating the sample images are in the form of a 4x4 matrix. A new vector is generated by multiplying the 3D vector generated from the depth map by the matrix. A new pixel is generated by converting the Z-axis coordinate information included in the rigid-transformed vector into depth information.

In this specification, for convenience of explanation, the depth map generated by rigid body transformation of the depth map generated from the target image is referred to as the first depth map, and the depth map generated from the sample image is referred to as the second depth map. .

Finally, a step of generating learning data using the depth values of the pixels included in the self-samples and the depth values of the pixels included in the target image subjected to the rigid body transformation is performed (S140).

The first and second depth maps are formed with the same size and shape and contain the same number of pixels. The first and second depth maps contain pixels that contain the same coordinate information as each other.

A loss is calculated using depth information included in pixels having the same coordinate information among pixels included in each of the first and second depth maps. At this time, depth information included in each of all pixels forming the first and second depth maps is used.

Here, some depth values included in the first and second depth maps may not be used to generate learning data. Specifically, the present invention further includes generating a mask map after generating the sample image.

Since the sample image generated from the target image has undergone rigid body transformation, there is a deviation from the target image. Therefore, the sample image may have areas where there are no pixels corresponding to the target image.

In simple terms, the sample image assumes that all objects in the image are rigid bodies and that the captured image is taken after the target image has been captured with only the camera moving while stationary. If the camera moves, there will be areas outside the camera's field of view, so the present invention generates a mask map so that the areas outside the camera's field of view are not used for depth estimation learning.

A mask map is generated based on the target image and the sample image. Specifically, since some pixels are lost when generating a sample image, which will be described later, new pixels containing color information of basic values are generated.

The mask map is generated to have the same size and shape as the target image, and includes coordinate information and filter information. The coordinate information included in the mask map is the same as the coordinate information included in the target image. Filter information is matched to each piece of coordinate information. The filter information is information defined as 0 or 1, and is used to filter a partial region of the sample image during depth estimation learning in the future.

On the other hand, the mask map is formed in the same size and shape as the sample image and contains the same number of pixels as the sample image. The mask map and the sample image each contain pixels that correspond to each other.

When generating the mask map, the filter information included in the mask map is determined according to the types of pixels included in the sample image. Specifically, the sample image includes pixels onto which rigid-transformed 3D pixels are projected and newly generated pixels. Filter information of pixels having the same coordinate information as the projected pixel is set to one. On the other hand, filter information of pixels having the same coordinate information as the newly generated pixel is set to zero. Filter information of all pixels included in the mask map can be defined in the manner described above.

In one embodiment, the mask map M is generated from a target image 410 and a sample image 410', as shown in Figure 4a. The mask map M consists of two regions. The first is a first area M0 that is matched with an area that does not include pixels corresponding to pixels included in the target image 410, among all areas of the sample image 410'. Pixels included in the first region M0 include coordinate information and filter information defined as zero. The second is a second area M1 that is matched to an area including pixels corresponding to pixels included in the target image 410 among all areas of the sample image 410'. Pixels included in the second region M1 include coordinate information and filter information defined in 1.

The mask map described above is used to calculate the loss for depth estimation learning according to the present invention.

Specifically, the depth estimation 430 produces a depth map 420 from the target image 410, as shown in FIG. 4b. Applying a rigid transformation (T) to depth map 420 produces a first depth map 420'. Here, the rigid transformation parameter applied to generate the first depth map 420' is the same as the rigid transformation parameter used to generate the sample image.

Meanwhile, depth estimation 430 produces a second depth map 420'' from the sample image 410'. Here, as the depth estimation model used to generate the second depth map 420 ″, the same model as the depth estimation model used when generating the depth map 420 from the target image 410 is used.

Then, using the first and second depth maps 420' and 420'' and the mask map M, isometric consistency loss is calculated.

In one embodiment, loss calculation using the first and second depth maps is performed as in Equation 2 below.

（数式２）
上記数式２において、Ｄｓｅｌｆ（上付き文字を含む）とは、第２デプスマップを構成するピクセルのうち、特定の座標（ｕ，ｖ）に存在するピクセルにマッチングされた深度値を意味し、Ｄｓｅｌｆ（上付き文字を含まない）とは、第１デプスマップを構成するピクセルのうち、特定の座標（ｕ，ｖ）に存在するピクセルにマッチングされた深度値を意味する。

(Formula 2)
In Equation 2, Dself (including a superscript) means a depth value matched to a pixel existing at specific coordinates (u, v) among pixels constituting the second depth map. "(not including superscript)" means a depth value matched to a pixel existing at specific coordinates (u, v) among pixels constituting the first depth map.

In Equation 2 above, k is the number of self-samples generated from one target image and used for learning, and t is the number of target images used for learning.

Meanwhile, in Equation 2, V is a variable corresponding to the mask map described above, and is a filter information value matched to specific coordinates (u, v). V is 0 or 1;

As mentioned above, the sample image according to the present invention consists entirely of static regions, so there is no need to filter dynamic regions in the sample image, which reduces depth estimation accuracy. Therefore, the present invention can utilize the entire sample image for learning.

The sample image generation method used to generate the learning data described above will be described in more detail below.

FIG. 5 is a flow chart for explaining the sample image generation method according to the present invention, and FIGS. 6a to 6c are conceptual diagrams showing the sample image generation method according to the present invention.

First, as shown in FIG. 5, in the sample image generating method according to the present invention, a step (S210) of estimating depth values of pixels included in the target image is performed.

In the depth estimation step, the distance to the object included in the image is calculated based on the viewpoint of the camera that captures the image. The distance value is calculated for each pixel included in the target image. That is, in the depth estimation step, the distance between the object indicated by the pixels included in the depth target image and the reference viewpoint is calculated.

Once the depth estimation step is performed, a depth map is generated. Various known models can be used as the depth estimation model. For example, the depth estimation model disclosed in Non-Patent Document 1 may be used, but it is not limited to this.

In one embodiment, a depth map 620 is generated by depth estimation 630 for a target image 610, as shown in FIG. 6a. The depth map 620 is an image expressed in different colors according to the depth information matched to the pixels, and visualizes the depth of each object included in the target image. Target image 610 and depth map 620 are utilized to generate a sample image.

Next, a step of mapping the pixels included in the target image onto a three-dimensional space based on the depth values of the pixels included in the target image (S220) is performed.

The target image includes pixel coordinate information and color information that make up the target image. Since the target image is a two-dimensional image, the pixel coordinate information includes coordinate information regarding two axes. For convenience of explanation, the pixel coordinate information included in the target image will be described as including X-axis coordinate information and Y-axis coordinate information.

The depth map generated in step S220 includes pixel coordinate information and depth information. The pixel coordinate information included in the depth map includes coordinate information regarding two axes. For convenience of explanation, the pixel coordinate information included in the target image will be described as including X-axis coordinate information and Y-axis coordinate information.

A depth map generated from a specific target image includes coordinate information of specific pixels included in the target image and depth information of the pixels. In step S220, the depth information included in the depth map is matched to the target image. Each of the target image and the depth map includes the same coordinate information as each other. Depth information corresponding to specific coordinate information is matched to pixels corresponding to the same coordinate information as the specific coordinate information. A three-dimensional image is thus generated.

Meanwhile, a camera calibration process may be performed when generating a 3D image. Specifically, in the process of transforming a two-dimensional image into a three-dimensional image, a matrix having predetermined parameters may be applied to each pixel forming the target image. The predetermined parameter differs depending on the model of the pinhole camera.

Here, the predetermined parameters for camera calibration may include at least one of camera extrinsic parameters and camera intrinsic parameters. Here, the camera intrinsic parameters may include parameters related to at least one of focal length, principal point and skew coefficient. The predetermined parameters for camera calibration may be applied when converting from one of the two-dimensional image and the three-dimensional image to the other.

A 3D image generated by the above method includes pixel coordinate information and color information. Pixel coordinate information contained in a three-dimensional image includes coordinate information regarding three axes. For convenience of explanation, the pixel coordinate information included in the target image will be described as including X-axis coordinate information, Y-axis coordinate information, and Z-axis coordinate information. The X-axis coordinate information and Y-axis coordinate information included in the three-dimensional image are coordinate information included in the target image, and the Z-axis coordinate information is depth information included in the depth map.

When the coordinate information included in the 3D image generated by the method described above is mapped onto the 3D space, each pixel forming the 3D image is mapped onto the 3D space.

In one embodiment, a depth value corresponding to pixel p1 is mapped to pixel p1 contained in target image 610, as shown in FIG. 6a. Therefore, the pixel p1 located on the XY plane is lifted (L) onto the three-dimensional space. By mapping depth values to all pixels contained in target image 610, a three-dimensional image can be generated.

Next, a step (S230) of performing rigid body transformation with preset parameters on the pixels mapped on the three-dimensional space is performed.

Each of the pixels mapped on the three-dimensional space may consist of a vector containing coordinate information. That is, each pixel may consist of a vector containing X-axis coordinate information, Y-axis coordinate information, and Z-axis coordinate information.

A parameter matrix used for rigid body transformation in three dimensions has the form of 4×4. Specifically, the parameter matrix for parallel movement consists of a 4×4 matrix containing distance values in the X-axis direction, Y-axis direction, and Z-axis direction distance values to be moved. On the other hand, the parameter matrix for rotation may include a different matrix for each axis that serves as a reference for rotation.

For example, the parameter matrix for rotation includes a 4×4 matrix containing rotation angle information about the X axis, a 4×4 matrix containing rotation angle information about the Y axis, and a rotation angle information about the Z axis. Contains a 4x4 matrix containing angle information.

A vector including X-axis coordinate information, Y-axis coordinate information, and Z-axis coordinate information may be calculated by multiplying each of the pixel vectors by the parameter matrix in a predetermined order.

A rigid-transformed vector contains new coordinate information for a particular pixel when a rigid transformation is performed on the particular pixel. A new three-dimensional image can be generated by performing a rigid transformation on all the pixels forming the three-dimensional image. The pixels contained in the 3D image generated by the above method contain the same color information and different coordinate information as the pixels contained in the original 3D image.

A parameter matrix used for rigid body transformation in three dimensions has the form of 4×4. Specifically, the parameter matrix for parallel movement consists of a 4×4 matrix containing distance values in the X-axis direction, Y-axis direction, and Z-axis direction distance values to be moved. On the other hand, the parameter matrix for rotation may include a different matrix for each axis that serves as a reference for rotation.

In one embodiment, as shown in FIG. 6b, when a rigid transformation (T) is performed on a pixel p1 located in the three-dimensional space, the coordinates of the pixel p1 in the three-dimensional space are changed. A pixel p1' subjected to rigid body transformation contains the same color information as the existing pixel p1, but contains different coordinate information.

Finally, the step of projecting the rigid-transformed pixels onto a two-dimensional plane to generate self-samples (S240) is performed.

The pixels that have been rigidly transformed are projected onto a preset plane. Specifically, the preset plane may be a plane on which the target image is arranged when the target image is lifted onto the three-dimensional space.

For example, if the target image is placed on the XY plane and depth values are mapped onto the target image, the plane onto which the rigid transformed image is projected may be the XY plane.

Meanwhile, a camera calibration process may be performed in the projection process. Since the camera calibration has been described above, a detailed description will be omitted.

A pixel that has undergone rigid body transformation includes coordinate information and color information that define coordinates in a three-dimensional space. Specifically, the rigid-transformed pixels include X-axis coordinate information, Y-axis coordinate information, and Z-axis coordinate information.

Any one of the X-axis coordinate information, the Y-axis coordinate information, and the Z-axis coordinate information constituting the coordinate information may be deleted when projecting the pixels subjected to the rigid body transformation. For example, when projecting a rigid-transformed image onto the XY plane, the Z-axis coordinate information contained in the rigid-transformed pixels is deleted.

Pixels are arranged on a two-dimensional plane by deleting any of the X-axis coordinate information, Y-axis coordinate information, and Z-axis coordinate information included in the pixels subjected to rigid body transformation. A sample image is generated after projecting all pixels contained in the rigid-transformed image onto a preset plane.

Here, the area onto which the rigid-transformed three-dimensional image is projected differs from the area in which the existing target image exists. A sample image is generated with reference to an area in which an existing target image exists. Information projected outside the existing target image is not used to generate the sample image.

Since the target image and the rigidly transformed 3D image contain the same number of pixels, if some of the pixels contained in the rigidly transformed 3D image are not used to generate the sample image, some pixels are lost. occurs.

Therefore, the sample image consists of two types of pixels. Specifically, the sample image may include pixels generated by projecting pixels that have undergone rigid transformation, and pixels newly generated when the sample image is generated.

A pixel generated by projecting a pixel subjected to rigid body transformation includes color information contained in the existing pixel as it is. If only the projected pixels form the sample image, it will have a different number of pixels than the target image. Therefore, new pixels are formed in areas where no pixels were projected when the sample image was generated. The newly generated pixels contain preset color information (eg, color information corresponding to black or white).

For example, if the target image is placed on the XY plane and placed in the regions 0<X<A and 0<Y<B, among the pixels projected onto the XY plane after rigid body transformation, Only pixels projected into the 0<X<A, 0<Y<B region are used to generate the sample image, and pixels projected outside the region are not used to generate the sample image. New pixels are generated at those points in the area where no pixels are projected.

In one embodiment, as shown in FIG. 6c, the rigidly transformed pixel p1' is projected (P) onto the XY plane where the target image was originally located. Therefore, the two-dimensionally projected pixel p1'' contains the same color information and different coordinate information as the pixels contained in the target image. If all the pixels contained in the rigid transformed 3D image are projected onto the XY plane, a 2D sample image 610' is generated. Here, only the pixels projected onto the area A where the target image is arranged are used to generate the sample image, and the other pixels are not used to generate the sample image.

Different sample images are generated according to the parameters applied during the rigid body transformation in step S230 described above. The present invention applies multiple rigid transformation parameters to a target image to generate multiple sample images.

As described above, the present invention allows multiple applications of rigid transformation parameters to generate multiple self-samples from a single target image. Therefore, the present invention can reserve an unlimited number of images for depth estimation learning.

As described above, the present invention generates a sample image in which all objects in the image are rigid bodies and maintains a stationary state, and utilizes it for generating machine learning data. Generation of learning data for depth estimation learning according to the present invention will be specifically described below.

The isometric consistency loss calculated in the manner described above can be used to set a loss function for depth estimation learning.

In one embodiment, the loss function using the isometric consistency loss calculated by the present invention is set as Equation 3 below.

（数式３）
上記数式３において、Ｌｐは、図７ａ及び図７ｂにおいて説明した方式で算出された測光損失であり、Ｌｓは、滑らかさ損失（Ｓｍｏｏｔｈｎｅｓｓ ｌｏｓｓ）であり、Ｌｉｓｓｇは、本発明による方法で算出された等尺性の一貫性の損失である。測光損失及び滑らかさ損失は公知の損失関数であるので、具体的な説明は省略する。

(Formula 3)
In Equation 3, Lp is the photometric loss calculated by the method described in FIGS. 7a and 7b, Ls is the smoothness loss, and Lissg is calculated by the method according to the present invention. Isometric consistency loss. Since the photometric loss and the smoothness loss are well-known loss functions, a detailed description thereof will be omitted.

Loss data is calculated based on the depth map generated from the target image and the depth map generated from the sample image, and then weights necessary for depth estimation are changed based on the loss data. After changing the weights, using the depth estimation model with the changed weights to regenerate the depth map of the target image, using it to regenerate the sample image, and regenerating the depth map from the sample image. . After that, the loss data is recalculated based on the depth map of the target image and the depth map of the sample image. The above calculation is repeated until the learning data generation count reaches a preset count. Here, the preset number of times must be a number large enough to ensure the reliability of the depth estimation model. By doing so, we can find the optimal weights needed during depth estimation.

As described above, the sample images generated by the present invention are generated from the target image, so all objects included in the sample images are rigid bodies and do not move. Therefore, when depth estimation learning is performed using the sample images generated by the present invention, there is no need to perform filtering for non-rigid objects and moving objects. Therefore, all regions of the target image can be utilized for depth estimation learning.

On the other hand, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored in a computer-readable medium (or recording medium).

Also, the present invention described above can be implemented as computer readable codes or commands on a program recording medium. That is, the present invention can be provided in the form of a program.

A computer-readable medium, on the other hand, includes any type of recording device on which data readable by a computer system is recorded. Examples of computer readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device and the like.

Computer-readable media also includes storage, which may be a server or cloud storage communicatively accessible by the electronic device. In this case, the computer can download the program according to the present invention from a server or cloud storage via wired or wireless communication.

Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, that is, a central processing unit (CPU), and its type is not particularly limited.

On the other hand, the detailed description of the present invention is illustrative and should not be construed as limiting in any respect. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention.

200 vehicle 210 camera 300 depth estimation system 310 communication unit 320 storage unit 330 control unit 340 database (DB)
410 target image 410 ′ sample image 420 depth map 420 ′ first depth map 420 ″ second depth map 430 depth estimation 610 target image 610 ′ two-dimensional sample image 620 depth map 630 depth estimation 710 first target image 730 depth estimation 740 Second target image 740′ Inverse rigid body transformation image A Region where target image is arranged M Mask map M0 First region M1 Second region

Claims (17)

generating a sample image using a target image and a depth map of the target image;
estimating depth values of pixels contained in the sample image to generate a depth map for the sample image;
and generating at least part of learning data using the depth map of the target image and the depth map generated from the sample image. further comprising the step of mapping pixels constituting a depth map generated from the target image onto a three-dimensional space, and then performing a rigid body transformation on the mapped pixels;
generating at least a portion of the learning data,
2. The learning data generation method for machine learning according to claim 1, wherein the depth map generated from the sample image and the depth map subjected to rigid body transformation are used. at least a portion of the learning data includes loss data;
generating at least a portion of the learning data,
calculating the loss data based on depth information included in pixels corresponding to each other among pixels included in each of the depth map generated from the sample image and the rigid-transformed depth map. 3. The learning data generation method for machine learning according to claim 2. The step of generating the sample image includes:
estimating depth values of pixels contained in the target image;
mapping the pixels included in the target image onto a three-dimensional space based on the depth values of the pixels included in the target image;
a step of performing rigid body transformation with preset parameters on the pixels mapped on the three-dimensional space;
4. The learning data generation method for machine learning according to claim 3, further comprising: projecting the rigid-transformed pixels onto a two-dimensional plane to generate the sample image. The step of performing rigid body transformation on the depth map generated from the target image includes:
5. The learning data generation method for machine learning according to claim 4, wherein the same parameters as the preset parameters are used. further comprising generating a mask map using the target image and the sample image;
The loss data is
6. The depth map of the sample image is generated based on a remaining area of the entire area of the depth map subjected to rigid body transformation, excluding an area excluded based on the mask map. 3. The learning data generation method for machine learning according to any one of 1. estimating depth values for pixels in the target image;
mapping the pixels included in the target image onto a three-dimensional space based on the depth values of the pixels included in the target image;
a step of performing rigid body transformation with preset parameters on the pixels mapped on the three-dimensional space;
and projecting the rigid-transformed pixels onto a two-dimensional plane to generate a sample image. The sample image is
formed in the same shape and size as the target image,
8. The method of claim 7, wherein the sample image includes the same number of pixels as the target image. each pixel included in the target image includes color information;
Pixels projected onto the two-dimensional plane are
9. The method of claim 8, wherein color information included in pixels corresponding to pixels projected onto the two-dimensional plane among pixels included in the target image is included. . The sample image is
10. The method of claim 9, wherein the pixels projected onto the two-dimensional plane include only pixels projected onto a preset area. The sample image is
11. The machine learning method according to claim 10, comprising a plurality of pixels formed by projection from the pixels subjected to the rigid transformation, and a plurality of pixels newly generated when the sample image is generated. A sample image generation method for 12. The method of claim 11, wherein the newly generated pixels contain the same color information and preset color information. 13. The sample image generating method for machine learning according to claim 8, wherein said preset parameters include parameters relating to at least one of rotational transformation and parallel transformation. a storage unit for storing target images;
a control unit that estimates depth values of pixels included in the target image to generate a depth map for the target image;
The control unit
generating a sample image using the target image and the depth map;
estimating depth values of pixels included in the sample image to generate a depth map for the sample image;
performing rigid body transformation on the depth map generated from the target image;
A learning data generation system for machine learning, wherein learning data is generated using the depth map generated from the sample image and the depth map subjected to rigid body transformation. a storage unit for storing target images;
estimating depth values of pixels included in the target image; mapping pixels included in the target image onto a three-dimensional space based on the depth values of pixels included in the target image; a controller that performs rigid transformation with preset parameters on the mapped pixels and projects the pixels subjected to the rigid transformation onto a two-dimensional plane to generate a sample image for machine learning. sample image generation system. A computer program comprising a plurality of instructions,
When the instruction is executed
estimating depth values of pixels included in a target image to generate a depth map for the target image;
generating a sample image using the target image and the depth map;
estimating depth values of pixels contained in the sample image to generate a depth map for the sample image;
performing a rigid transformation on a depth map generated from the target image;
and generating training data using the depth map generated from the sample image and the rigid-transformed depth map. a computer program comprising a plurality of instructions which, when executed, estimate depth values for pixels contained in a target image;
mapping the pixels included in the target image onto a three-dimensional space based on the depth values of the pixels included in the target image;
a step of performing rigid body transformation with preset parameters on the pixels mapped on the three-dimensional space;
and projecting the rigid-transformed pixels onto a two-dimensional plane to generate a sample image.

Images